16:58:01 #startmeeting Network team meeting, 25 January 2021 16:58:02 Meeting started Mon Jan 25 16:58:01 2021 UTC. The chair is ahf. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:58:02 Useful Commands: #action #agreed #help #info #idea #link #topic. 16:58:08 hallo hallo hallo network-team o/ 16:58:18 o/ 16:58:21 we have a pad at https://pad.riseup.net/p/tor-netteam-2021.1-keep 16:58:23 o/ 16:58:25 hello 16:58:42 hi 16:58:51 o/ 16:59:13 hi 16:59:22 let's get started 16:59:30 how are folks doing with their boards? 16:59:38 o/ 16:59:47 * asn looking at board 17:01:07 * nickm looking too..... 17:01:46 * ahf good now 17:02:02 mine seems ok 17:02:17 * asn fixing mine 17:02:54 * asn is good 17:03:50 asn, dgoulet: you assigned reviewers? i see we have a lot of MR's with no reviewer on right now but i think that is because the feature arrived recently 17:03:56 hmmm 17:04:01 i thought i assigned everything 17:04:07 i think you might have too 17:04:20 hm, you use assignee or reviewer? 17:04:25 ah because the old stuff have "assignee" and the new stuff have "reviewer" 17:04:32 ya 17:04:34 o/ 17:04:36 should i migrate everythign to "Reviewer"? 17:04:43 or leave the old stuff to eventually become done? 17:04:46 we probably need to untangle that a bit 17:04:52 i think leaving the old things with assignee is fine 17:04:59 but for new stuff let's use reviewer 17:05:05 i did use reviewer today 17:05:08 i assigned 5-6 tickets 17:05:08 the gitlab update over the weekend also gave us the list of things we are expected to review 17:05:09 awesome 17:05:11 thank you! 17:05:27 okay, 0.4.5 17:06:01 * ahf takes tor#24857 out and moves it to 0.4.6 17:06:47 11 tickets in the list? 17:06:54 https://gitlab.torproject.org/dashboard/issues?scope=all&utf8=%E2%9C%93&state=opened&milestone_title=Tor%3A%200.4.5.x-stable 17:06:57 that's what I see... 17:06:59 * asn looks 17:07:21 i dont have any tickets from that list 17:07:28 if someone wants to offload something, feel free to put it on me 17:07:35 tor#40107 is missing an assignee 17:07:35 hi, what is this channel for? 17:07:45 SomeHacker: for meetings, and we are in the middle of one right now :-) 17:08:11 ok im gonna take #40107 and see if i can od osmething useful with it 17:08:39 asn: maybe coordinate with dgoulet there, he's been working on stuff that's related 17:08:51 oh! dgoulet! 17:09:04 (or maybe not related stuff, I'm not 100% sure) 17:09:09 oh? dgoulet? 17:09:15 :) 17:09:20 cool, thanks asn! 17:09:50 no new "For x-team" tickets for us 17:10:05 ok! many discusions items 17:10:16 - [1/25 2021]: Should we backport sandbox feature/fix patches to older releases? 17:10:27 this was added by david and i while we were going over backport tickets last week 17:10:47 it seems a bit like we often put those up for backports, but are unsure about whether we want to do them, so they stick in the queue for a few iterations 17:10:58 do we want to just not backport sandbox features? 17:11:26 I am okay either way; I think that if we decide "no backport" what it means is adding it to the list on CoreTorReleases 17:11:44 the one that starts "Some features and configurations are currently only maintained in the two most recent stable releases" 17:11:57 i think it's a good idea to reduce the set of things we do backports for in general 17:12:09 yeah 17:14:19 ok, if i have to interpret on random things then i go with heads/tails which is what we don't want :-S 17:14:26 asn, dgoulet: what do you think 17:14:49 i will happily update the doc like nick explained above 17:15:22 i like "no backports". perhaps we can do that, and if we get lots of complains, we can revisit? 17:15:23 I'm up for reducing also 17:15:46 ok with me 17:15:53 asn: yes, i think so too - i think we should have a sensible default and of course be open to backporting things if we have some very sad bug :-/ 17:16:01 ok, i will update the doc then after the meeting 17:16:30 [1/25 2021]: See https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/81#note_2722322 . Admin team will not keep supporting gitolite forever, and wants us to figure out which mechanisms we want to use to mitigate threat models related to gitlab's larger attack surface. We should figure out what we actually want and what we're prepared to do to get it. 17:16:41 i *think* this was added by nickm? 17:16:43 y 17:16:59 anarcat asked me what I thought, and I figured I should bring it to the team 17:17:01 * asn reading 17:17:23 ya, i know the browser team is also having some thoughts on this i believe 17:17:35 basically, supposing that there is no longer a gitolite, what do we want to do (if anything) to make it so we all believe in gitlab's repositories? 17:17:40 i do think tor.git can be the very last thing we get rid of, but having a plan and some ideas for what we want to do is a good idea 17:17:55 has the browser team made a decision? 17:18:10 no 17:18:10 i don't think anybody have yet as we haven't started removing things from gitolite yet 17:18:36 i think we have a better and clearer idea and plan on user permissions in gitlab before moving repos there and getting rid of gitolite 17:18:43 i think one thing we do think about is signing commits 17:18:51 i think if we have to look into running more services, that is also a possibility. i have no clue if services for git commit transparency logs and such exists yet 17:18:59 and have a git hook enforcing that the tip is always signed 17:19:00 so the threat model IIUC is "attacker who finds a gitlab exploit". 17:19:01 * anarcat waves 17:19:10 having lunch, but maybe around to answer quick questions 17:20:02 nickm: ya 17:20:13 and there is a lot more surface on gitlab than anything else we have i think 17:20:26 and what we want to make sure is that every commit was either written by or merged by somebody with permission to do so. 17:20:30 correct 17:20:35 security of the runners is also an issue 17:20:44 if we rely on the artifacts produced there 17:20:56 less of an issue for the network team though 17:21:00 it's also an issue now with the jenkins runners, but we tend to have stricter control over there 17:21:01 and browser team have trusted builders 17:21:16 network team don't release binaries 17:21:22 I believe network team doesn't currently redistribute artifacts from runners 17:21:36 i suspect this might change over time, which is why i brought it up ;) 17:21:56 but yes, certainly not an immediate concern then 17:22:01 for runners, I think the only answer is reproducibility 17:22:19 yes 17:22:30 well i'm working on "rootless" and more secure runners that i want to use for TPA, so that could also prove useful 17:22:37 i like this guix signing workflow mentioned by anarcat. seems pretty robust. 17:22:48 you could have runners as a trusted anchor, which could validate OpenPGP signatures from git and not have to trust GitLab as much 17:22:52 asn: i found that one to be very interesting as well 17:22:55 i also think that someone haxing gitlab and inserting code into tor.git as a commit, seems like a pretty poor supply-chain attack 17:23:06 this would obviously require more research, but it's an avenue i'm looking at for using GitLab for critical code in TPA 17:23:32 not many people use tor.git master, and the attack has reasonable chances to be detected in the future by us 17:23:34 "sign all commits" is indeed simple to explain 17:23:47 sign all commits -- and have some sort of whitelist of committers 17:23:52 yep 17:23:55 then we have to be careful with rebasing iirc 17:24:04 i think that's a model browser folks would be happy with 17:24:07 we need to make sure our rebase tools re-sign. 17:24:11 ya 17:24:26 asn: how often do you say "git pull" without double-checking that you got only the expected changes? 17:24:41 asn: that's the attack I'd worry about: somebody adds a commit to tor.git and nobody notices 17:24:59 "Fix some typos" -- Roger Dingledine 17:25:06 right 17:25:06 lol 17:25:12 yes 17:25:13 i agree. that's the attack here. 17:25:21 next time you push a git hook could complain 17:25:26 my attack scenario also includes some less trusted party issuing a Merge request on GitLab, which would trigger a CI pipeline, root the runner, and then backdoor binaries 17:25:35 if you have a strict sign-everything policy 17:25:36 but that might not be part of the scenario for the network team if you don't trust artifacts 17:25:51 I want the ability for volunteers to issue merge requests 17:26:04 that's pretty crucial for our workflow 17:26:07 agreed. 17:26:14 yeah, problem with "everything is signed" is third-party contributions become harder, because unsigned 17:26:15 we don't care about merging MRs via the gitlab UI though 17:26:20 but that can be achieved, by us signing also volunteers' commits, right? 17:26:25 then the trick is to sign merge commits, and not merge via the GitLab UI 17:26:36 asn: then you need to rebase volunteers, kind of a PITA 17:26:40 and messes up their histories 17:26:52 if you sign merge requests, the "tip is signed" rule works 17:27:04 I do like the gerwitz approach, but it needs lots of tooling 17:27:07 tip AKA HEAD in git parlance 17:27:21 what is the gerwith approach? 17:27:25 *gerwitz 17:27:33 see https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/81 17:27:37 we have a hackathon coming up in march in the organisation. i don't know much about it, but i think gaba does 17:27:37 GeKo: ^ 17:27:45 is this something that would be relevant for such thing as a cross-team effort? 17:28:09 ah, yes, i like that blog post :) 17:28:17 anarcat: I think we want _not only_ "head must be signed on push" but also "head must be signed on pull" in that case. 17:28:27 +1 17:28:45 correct 17:28:52 One thing also I'd want is the ability to use less-crucial PGP keys for this. 17:28:54 which is why it's interesting to add such mechanisms on the runners 17:29:04 not sure how practical that is, considering gitlab-runner is this weird black box 17:29:34 Like, not my main "on a yubikey" key, but a separate "nick's key for signing tor" key. 17:29:36 gitlab-runners can be made to help us verify this too with verifying that things are right upon merge 17:29:44 nickm: yes, +1 17:29:47 (we'd need an answer to key lifetimes and revocation) 17:30:14 If we do adopt a version of this, we'll want to make sure that there is not only a "is everything right now" tool we can run, but also a "was everything right at all points in the past" tool 17:30:15 might be a bit tricky to enforce "head must be signed on pull" for developer local checkouts. wouldn't they individually need to enable a local git hook, or the like? 17:30:36 we already have tooling to keep our local git hooks up to date :) 17:30:43 ahf: i'm not sure about that wrt runners - they probably need to trust gitlab if we do those checks in the gitlab-ci.yaml file 17:30:53 see scripts/git/git-install-tools.sh 17:31:00 ah cool 17:32:04 what is the next steps for this? it sounds like there is a lot of unknowns in this 17:32:15 i don't think we'll get a solution today 17:33:23 it sounds like we have an answer-in-principle of "we'd like to do the signed-HEAD rule" 17:33:30 and we need to figure out the tooling to do it 17:33:37 ya 17:33:40 we should probably break down who would be responsible for what tooling 17:33:54 makes sense for somebody in the development world to do the dev-side tooling 17:34:18 yeah 17:34:23 sounds interesting 17:34:30 I don't know how much custom git client-side stuff the TB does? We might be the logical people given what we already have... 17:34:42 my priority with this ticket is to make sure we want to move away from gitolite and are ready to look for solutions 17:34:44 though ideally somebody has already built it 17:34:59 it doesn't need to be immediately it, but if people are open to looking into code signing, i think it's the way forward 17:35:24 and then it gives me a way forward to retiring gitolite, which people have expressed a desire for in the survey (which is why i pinged nickm :) 17:35:33 * anarcat lunch 17:35:37 i hope someone has built it, otherwise i hope it can be done with a simple git hook which validates all commits when we pull 17:36:03 and we can assume that someone of us is gonna pull between the pwnage and the pwned code getting into an honest computer 17:36:19 ok, let's continue this conversation either in another meeting or post the meeting. we still have a few items and we also have the s61 sync 17:36:44 ack 17:36:45 [1/25 2021]: Do we want an Arti hackathon in February? An hour later IMO, but let's do Doodle if yes? ahf didn't get any feedback via email yet. 17:36:55 it would be fine w me 17:36:55 yesss 17:36:56 quick one: yay/nay, if yay i will do the email stuff soon to tor-internal@ 17:37:03 and do it at 14 UTC instead of 13 UTC i think 17:37:07 sounds gooddddd 17:37:13 it is too early with 13 i think 17:37:19 ok good 17:37:22 yeah, hour later would be nice :) 17:37:23 arti hackfests make me excite 17:37:38 qq there for asn and dgoulet: I think some time this week i might be hacking on state. did you get something partial done there that I should look at? 17:37:38 jnewsome: was fun to hack with you and juga last week even if it was just one hour 17:37:42 (in arti) 17:37:47 14:00 would be better for me since I get online ~13:10 usually :P 17:38:04 nickm: did not continue on my side no :( 17:38:05 ya, i hadn't realized what time 13 was for some of you. i thought it was an hour later 17:38:09 asn: likewise! 17:38:16 i would say no to invitations at 13 if i was in your tz 17:38:40 jnewsome: just fwiw, in the session after you left, nick recommended we do this trait approach you suggested. so that's on my TODO list for this week. 17:38:44 it's 50/50 wheether I have to drive my kid to school at 1300 UTC. (depends whether the whether is bad.) It's 8am for me 17:38:56 weather the whether 17:39:04 err, *weather 17:39:09 8am meetings should be illegal :o 17:39:12 asn: oh cool, yeah I was wondering how that shook out 17:39:33 asn: fwiw, wrt the trait thing, I recommend that you try it out at least. I'm not 100% sure it'll work, but you'll definitely learn something more about rust 17:39:42 yes agreed 17:39:50 (about 80% of what I know about rust comes from trying stuff that didn't work) 17:40:51 heh. i'm fairly confident the trait approach will *work*, but there are bigger questions about whether it's a sufficiently effective test strategy, how maintainable it is, etc. but trying it out can help develop a feeling for some of that 17:40:53 what's the next topic? 17:40:58 ya 17:41:00 [1/25 2021]: Should we do stable stable releases soon because of the client (and thus HS) consensus liveness issue that caused availability issues in early 2021? 17:41:10 i think it's from nickm 17:41:18 I don't think that one is from me 17:41:21 though the next one is 17:41:34 my plan has been "do stable releases shortly after 045x is stable 17:41:46 do we think we should aim for sooner than that? 17:42:13 the sooner the better imo since 043 is EOL is ~20 days 17:42:24 so would be wise to do at least one last stable in 043 to fix taht v3 problem 17:42:31 huh, ah, i think i might have added it 17:42:46 yes 17:42:54 it was after dgoulet and i had a sync on DoS mitigation, sorry 17:44:09 we can also put out an 043 then too. Nothing stopping us from doing one last release 17:44:17 dgoulet: is there more that needs to go in than what is there now? 17:44:27 i think we only talked about the liveness patch 17:44:59 yeah as far as I can recall, that is it 17:45:03 an important fix 17:45:26 ok 17:46:55 I'm okay doing stables in early feb instead, if we think that's better 17:47:08 I think this week would be a bit premature though 17:47:29 ok, let's look at it next week then? 17:48:29 ok w me 17:48:43 last item: [1/25 2021]: Will we also want to do another 045x-rc between now and the target stable date (15 Feb?) 17:49:49 our best target if so is feb 1, which is next monday 17:50:16 we should keep an eye on what we're doing in 045, and if we are merging anything that really needs an rc for testing 17:50:24 do we anticipate anything like that landing this week? 17:50:40 i have no strong opinion about this. i have the mystery issue with signals on gentoo hardened under load which may or may not be something fishy in our code, but we have only heard of it from one person 17:50:51 and then the compatibility issue with old glibc/musl and glob() 17:51:05 nickm: sounds fine to me with getting things in this week and then we look next week if we want to do one more? 17:51:27 let's look towards the end of this week? Trying to decide _on_ monday would be hard if monday is the target date :) 17:51:41 ya, we can sync on thursday? 17:51:52 and see if there is something 17:52:29 ack 17:52:35 ok! 17:52:41 that was a lot of stuff, no more discussion items 17:52:45 mikeperry: you wanna do the s61 part? 17:52:58 s61 updates are in the pad, one thing to highlight: 17:53:22 I am talking with multipath tcp folks about coupled congestion control and scheduling algs to minimize out-of-order queues 17:53:29 I got some great answers 17:53:42 but if any9one knows more of these people, I probably will have future questions! 17:53:45 nice 17:53:46 (this is for objective 3?) 17:53:49 yes 17:53:54 for the conflux proposal 17:53:55 awesome 17:54:03 currently in draft in my remote 17:54:18 I am updating it this week based on their input 17:54:22 so i finished the CBT parts of O1.1 on EOY, and I'm kinda task-less wrt s61 right now 17:54:37 not sure what i should be doing next for this sponsor 17:54:53 asn: dgoulet is dying under dirauth dos, but maybe you couold help him with the overload descriptor implementation? 17:55:12 https://gitlab.torproject.org/tpo/core/tor/-/issues/40222 17:55:28 oh! dgoulet! 17:55:46 yeah that is the one in the pipe that I had to put on hold 17:55:49 asn: other option is understand onionperf filters and start looking at https://gitlab.torproject.org/tpo/core/tor/-/issues/40160 17:56:09 ack 17:57:11 ok ok 17:57:17 will check both of those tickets 17:57:20 and get in touch with dgoulet 17:57:26 to see what can be done 17:57:31 kk great 17:58:39 I think that's it unless there are comments or concerns from geko and juga 17:59:19 i think juga had a question 17:59:22 just whether you know if bwauths might be not voting on relays that are in the bandwidth file 17:59:31 cause of dos patches 17:59:50 like the dos patches blocking scanning or something? 18:00:01 not blocking scanning 18:00:03 I saw you got sbws to measure things torflow was not. is this part of that? 18:00:12 just discarding relays in the vote 18:00:23 mikeperry: nop 18:01:04 hrmmm.. I thought torflow was rather patient with *keeping* measurements for relays that disappeared, in case they came back. I am not aware of cases where it would decide to get rid of measurements... 18:01:08 but it has been a long while 18:01:33 the dirauths take the last decission on which relays to vote on 18:02:01 oh and perhaps due to the dos, they are not getting connections from some relays 18:02:03 so far they voted on the ones in the bwfile, but might be slightly different right now 18:02:04 and dropping them 18:02:23 yes, thats the only changed i saw in the scanner itself 18:02:41 relays in that disapear 18:03:05 anyway, probably not important 18:03:31 yes in the past week or so, we could have seen massive drops of relays from some dirauths if they could not accept descriptors 18:03:45 or had to drop some descriptor submission attempts 18:03:48 yeah, that makes sense 18:04:00 ok, i think my question is answered 18:04:35 cool 18:04:46 do we have more things for s61? 18:05:04 <- is fine 18:05:13 not from me 18:05:26 I am groot 18:05:45 hello groot i'm dad 18:06:03 who is the rocket raccoon then!? 18:06:10 ok, thanks all for the slightly long meeting 18:06:16 thanks! 18:06:16 let's continue in #tor-dev and friends 18:06:19 #endmeeting