16:58:01 <ahf> #startmeeting Network team meeting, 25 January 2021
16:58:02 <MeetBot> Meeting started Mon Jan 25 16:58:01 2021 UTC.  The chair is ahf. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:58:02 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:58:08 <ahf> hallo hallo hallo network-team o/
16:58:18 <juga> o/
16:58:21 <ahf> we have a pad at https://pad.riseup.net/p/tor-netteam-2021.1-keep
16:58:23 <mikeperry> o/
16:58:25 <nickm> hello
16:58:42 <GeKo> hi
16:58:51 <jnewsome> o/
16:59:13 <gaba> hi
16:59:22 <ahf> let's get started
16:59:30 <ahf> how are folks doing with their boards?
16:59:38 <asn> o/
16:59:47 * asn looking at board
17:01:07 * nickm looking too.....
17:01:46 * ahf good now
17:02:02 <nickm> mine seems ok
17:02:17 * asn fixing mine
17:02:54 * asn is good
17:03:50 <ahf> asn, dgoulet: you assigned reviewers? i see we have a lot of MR's with no reviewer on right now but i think that is because the feature arrived recently
17:03:56 <asn> hmmm
17:04:01 <asn> i thought i assigned everything
17:04:07 <ahf> i think you might have too
17:04:20 <ahf> hm, you use assignee or reviewer?
17:04:25 <asn> ah because the old stuff have "assignee" and the new stuff have "reviewer"
17:04:32 <ahf> ya
17:04:34 <dgoulet> o/
17:04:36 <asn> should i migrate everythign to "Reviewer"?
17:04:43 <asn> or leave the old stuff to eventually become done?
17:04:46 <ahf> we probably need to untangle that a bit
17:04:52 <ahf> i think leaving the old things with assignee is fine
17:04:59 <ahf> but for new stuff let's use reviewer
17:05:05 <asn> i did use reviewer today
17:05:08 <asn> i assigned 5-6 tickets
17:05:08 <ahf> the gitlab update over the weekend also gave us the list of things we are expected to review
17:05:09 <ahf> awesome
17:05:11 <ahf> thank you!
17:05:27 <ahf> okay, 0.4.5
17:06:01 * ahf takes tor#24857 out and moves it to 0.4.6
17:06:47 <ahf> 11 tickets in the list?
17:06:54 <ahf> https://gitlab.torproject.org/dashboard/issues?scope=all&utf8=%E2%9C%93&state=opened&milestone_title=Tor%3A%200.4.5.x-stable
17:06:57 <nickm> that's what I see...
17:06:59 * asn looks
17:07:21 <asn> i dont have any tickets from that list
17:07:28 <asn> if someone wants to offload something, feel free to put it on me
17:07:35 <ahf> tor#40107 is missing an assignee
17:07:35 <SomeHacker> hi, what is this channel for?
17:07:45 <ahf> SomeHacker: for meetings, and we are in the middle of one right now :-)
17:08:11 <asn> ok im gonna take #40107 and see if i can od osmething useful with it
17:08:39 <nickm> asn: maybe coordinate with dgoulet there, he's been working on stuff that's related
17:08:51 <asn> oh! dgoulet!
17:09:04 <nickm> (or maybe not related stuff, I'm not 100% sure)
17:09:09 <asn> oh? dgoulet?
17:09:15 <nickm> :)
17:09:20 <ahf> cool, thanks asn!
17:09:50 <ahf> no new "For x-team" tickets for us
17:10:05 <ahf> ok! many discusions items
17:10:16 <ahf> - [1/25 2021]: Should we backport sandbox feature/fix patches to older releases?
17:10:27 <ahf> this was added by david and i while we were going over backport tickets last week
17:10:47 <ahf> it seems a bit like we often put those up for backports, but are unsure about whether we want to do them, so they stick in the queue for a few iterations
17:10:58 <ahf> do we want to just not backport sandbox features?
17:11:26 <nickm> I am okay either way; I think that if we decide "no backport" what it means is adding it to the list on CoreTorReleases
17:11:44 <nickm> the one that starts "Some features and configurations are currently only maintained in the two most recent stable releases"
17:11:57 <ahf> i think it's a good idea to reduce the set of things we do backports for in general
17:12:09 <ahf> yeah
17:14:19 <ahf> ok, if i have to interpret on random things then i go with heads/tails which is what we don't want :-S
17:14:26 <ahf> asn, dgoulet: what do you think
17:14:49 <ahf> i will happily update the doc like nick explained above
17:15:22 <asn> i like "no backports". perhaps we can do that, and if we get lots of complains, we can revisit?
17:15:23 <dgoulet> I'm up for reducing also
17:15:46 <nickm> ok with me
17:15:53 <ahf> asn: yes, i think so too - i think we should have a sensible default and of course be open to backporting things if we have some very sad bug :-/
17:16:01 <ahf> ok, i will update the doc then after the meeting
17:16:30 <ahf> [1/25 2021]: See https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/81#note_2722322 .  Admin team will not keep supporting gitolite forever, and wants us to figure out which mechanisms we want to use to mitigate threat models related to gitlab's larger attack surface.  We should figure out what we actually want and what we're prepared to do to get it.
17:16:41 <ahf> i *think* this was added by nickm?
17:16:43 <nickm> y
17:16:59 <nickm> anarcat asked me what I thought, and I figured I should bring it to the team
17:17:01 * asn reading
17:17:23 <ahf> ya, i know the browser team is also having some thoughts on this i believe
17:17:35 <nickm> basically, supposing that there is no longer a gitolite, what do we want to do (if anything) to make it so we all believe in gitlab's repositories?
17:17:40 <ahf> i do think tor.git can be the very last thing we get rid of, but having a plan and some ideas for what we want to do is a good idea
17:17:55 <nickm> has the browser team made a decision?
17:18:10 <GeKo> no
17:18:10 <ahf> i don't think anybody have yet as we haven't started removing things from gitolite yet
17:18:36 <gaba> i think we have a better and clearer idea and plan on user permissions in gitlab before moving repos there and getting rid of gitolite
17:18:43 <GeKo> i think one thing we do think about is signing commits
17:18:51 <ahf> i think if we have to look into running more services, that is also a possibility. i have no clue if services for git commit transparency logs and such exists yet
17:18:59 <GeKo> and have a git hook enforcing that the tip is always signed
17:19:00 <nickm> so the threat model IIUC is "attacker who finds a gitlab exploit".
17:19:01 * anarcat waves
17:19:10 <anarcat> having lunch, but maybe around to answer quick questions
17:20:02 <ahf> nickm: ya
17:20:13 <ahf> and there is a lot more surface on gitlab than anything else we have i think
17:20:26 <nickm> and what we want to make sure is that every commit was either written by or merged by somebody with permission to do so.
17:20:30 <anarcat> correct
17:20:35 <anarcat> security of the runners is also an issue
17:20:44 <anarcat> if we rely on the artifacts produced there
17:20:56 <ahf> less of an issue for the network team though
17:21:00 <anarcat> it's also an issue now with the jenkins runners, but we tend to have stricter control over there
17:21:01 <ahf> and browser team have trusted builders
17:21:16 <ahf> network team don't release binaries
17:21:22 <nickm> I believe network team doesn't currently redistribute artifacts from runners
17:21:36 <anarcat> i suspect this might change over time, which is why i brought it up ;)
17:21:56 <anarcat> but yes, certainly not an immediate concern then
17:22:01 <nickm> for runners, I think the only answer is reproducibility
17:22:19 <ahf> yes
17:22:30 <anarcat> well i'm working on "rootless" and more secure runners that i want to use for TPA, so that could also prove useful
17:22:37 <asn> i like this guix signing workflow mentioned by anarcat. seems pretty robust.
17:22:48 <anarcat> you could have runners as a trusted anchor, which could validate OpenPGP signatures from git and not have to trust GitLab as much
17:22:52 <ahf> asn: i found that one to be very interesting as well
17:22:55 <asn> i also think that someone haxing gitlab and inserting code into tor.git as a commit, seems like a pretty poor supply-chain attack
17:23:06 <anarcat> this would obviously require more research, but it's an avenue i'm looking at for using GitLab for critical code in TPA
17:23:32 <asn> not many people use tor.git master, and the attack has reasonable chances to be detected in the future by us
17:23:34 <nickm> "sign all commits" is indeed simple to explain
17:23:47 <asn> sign all commits -- and have some sort of whitelist of committers
17:23:52 <GeKo> yep
17:23:55 <ahf> then we have to be careful with rebasing iirc
17:24:04 <GeKo> i think that's a model browser folks would be happy with
17:24:07 <nickm> we need to make sure our rebase tools re-sign.
17:24:11 <ahf> ya
17:24:26 <nickm> asn: how often do you say "git pull" without double-checking that you got only the expected changes?
17:24:41 <nickm> asn: that's the attack I'd worry about: somebody adds a commit to tor.git and nobody notices
17:24:59 <nickm> "Fix some typos" -- Roger Dingledine
17:25:06 <asn> right
17:25:06 <ahf> lol
17:25:12 <ahf> yes
17:25:13 <asn> i agree. that's the attack here.
17:25:21 <GeKo> next time you push a git hook could complain
17:25:26 <anarcat> my attack scenario also includes some less trusted party issuing a Merge request on GitLab, which would trigger a CI pipeline, root the runner, and then backdoor binaries
17:25:35 <GeKo> if you have a strict sign-everything policy
17:25:36 <anarcat> but that might not be part of the scenario for the network team if you don't trust artifacts
17:25:51 <nickm> I want the ability for volunteers to issue merge requests
17:26:04 <nickm> that's pretty crucial for our workflow
17:26:07 <asn> agreed.
17:26:14 <anarcat> yeah, problem with "everything is signed" is third-party contributions become harder, because unsigned
17:26:15 <nickm> we don't care about merging MRs via the gitlab UI though
17:26:20 <asn> but that can be achieved, by us signing also volunteers' commits, right?
17:26:25 <anarcat> then the trick is to sign merge commits, and not merge via the GitLab UI
17:26:36 <anarcat> asn: then you need to rebase volunteers, kind of a PITA
17:26:40 <anarcat> and messes up their histories
17:26:52 <anarcat> if you sign merge requests, the "tip is signed" rule works
17:27:04 <nickm> I do like the gerwitz approach, but it needs lots of tooling
17:27:07 <anarcat> tip AKA HEAD in git parlance
17:27:21 <GeKo> what is the gerwith approach?
17:27:25 <GeKo> *gerwitz
17:27:33 <nickm> see https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/81
17:27:37 <ahf> we have a hackathon coming up in march in the organisation. i don't know much about it, but i think gaba does
17:27:37 <nickm> GeKo: ^
17:27:45 <ahf> is this something that would be relevant for such thing as a cross-team effort?
17:28:09 <GeKo> ah, yes, i like that blog post :)
17:28:17 <nickm> anarcat: I think we want _not only_ "head must be signed on push" but also "head must be signed on pull" in that case.
17:28:27 <GeKo> +1
17:28:45 <anarcat> correct
17:28:52 <nickm> One thing also I'd want is the ability to use less-crucial PGP keys for this.
17:28:54 <anarcat> which is why it's interesting to add such mechanisms on the runners
17:29:04 <anarcat> not sure how practical that is, considering gitlab-runner is this weird black box
17:29:34 <nickm> Like, not my main "on a yubikey" key, but a separate "nick's key for signing tor" key.
17:29:36 <ahf> gitlab-runners can be made to help us verify this too with verifying that things are right upon merge
17:29:44 <ahf> nickm: yes, +1
17:29:47 <nickm> (we'd need an answer to key lifetimes and revocation)
17:30:14 <nickm> If we do adopt a version of this, we'll want to make sure that there is not only a "is everything right now" tool we can run, but also a "was everything right at all points in the past" tool
17:30:15 <jnewsome> might be a bit tricky to enforce "head must be signed on pull" for developer local checkouts. wouldn't they individually need to enable a local git hook, or the like?
17:30:36 <nickm> we already have tooling to keep our local git hooks up to date :)
17:30:43 <anarcat> ahf: i'm not sure about that wrt runners - they probably need to trust gitlab if we do those checks in the gitlab-ci.yaml file
17:30:53 <nickm> see scripts/git/git-install-tools.sh
17:31:00 <jnewsome> ah cool
17:32:04 <ahf> what is the next steps for this? it sounds like there is a lot of unknowns in this
17:32:15 <ahf> i don't think we'll get a solution today
17:33:23 <nickm> it sounds like we have an answer-in-principle of "we'd like to do the signed-HEAD rule"
17:33:30 <nickm> and we need to figure out the tooling to do it
17:33:37 <ahf> ya
17:33:40 <nickm> we should probably break down who would be responsible for what tooling
17:33:54 <nickm> makes sense for somebody in the development world to do the dev-side tooling
17:34:18 <ahf> yeah
17:34:23 <asn> sounds interesting
17:34:30 <nickm> I don't know how much custom git client-side stuff the TB does? We might be the logical people given what we already have...
17:34:42 <anarcat> my priority with this ticket is to make sure we want to move away from gitolite and are ready to look for solutions
17:34:44 <nickm> though ideally somebody has already built it
17:34:59 <anarcat> it doesn't need to be immediately it, but if people are open to looking into code signing, i think it's the way forward
17:35:24 <anarcat> and then it gives me a way forward to retiring gitolite, which people have expressed a desire for in the survey (which is why i pinged nickm :)
17:35:33 * anarcat lunch
17:35:37 <asn> i hope someone has built it, otherwise i hope it can be done with a simple git hook which validates all commits when we pull
17:36:03 <asn> and we can assume that someone of us is gonna pull between the pwnage and the pwned code getting into an honest computer
17:36:19 <ahf> ok, let's continue this conversation either in another meeting or post the meeting. we still have a few items and we also have the s61 sync
17:36:44 <nickm> ack
17:36:45 <ahf> [1/25 2021]: Do we want an Arti hackathon in February? An hour later IMO, but let's do Doodle if yes? ahf didn't get any feedback via email yet.
17:36:55 <nickm> it would be fine w me
17:36:55 <asn> yesss
17:36:56 <ahf> quick one: yay/nay, if yay i will do the email stuff soon to tor-internal@
17:37:03 <ahf> and do it at 14 UTC instead of 13 UTC i think
17:37:07 <asn> sounds gooddddd
17:37:13 <ahf> it is too early with 13 i think
17:37:19 <ahf> ok good
17:37:22 <jnewsome> yeah, hour later would be nice :)
17:37:23 <asn> arti hackfests make me excite
17:37:38 <nickm> qq there for asn and dgoulet: I think some time this week i might be hacking on state.  did you get something partial done there that I should look at?
17:37:38 <asn> jnewsome: was fun to hack with you and juga last week even if it was just one hour
17:37:42 <nickm> (in arti)
17:37:47 <dgoulet> 14:00 would be better for me since I get online ~13:10 usually :P
17:38:04 <dgoulet> nickm: did not continue on my side no :(
17:38:05 <ahf> ya, i hadn't realized what time 13 was for some of you. i thought it was an hour later
17:38:09 <jnewsome> asn: likewise!
17:38:16 <ahf> i would say no to invitations at 13 if i was in your tz
17:38:40 <asn> jnewsome: just fwiw, in the session after you left, nick recommended we do this trait approach you suggested. so that's on my TODO list for this week.
17:38:44 <nickm> it's 50/50 wheether I have to drive my kid to school at 1300 UTC. (depends whether the whether is bad.)  It's 8am for me
17:38:56 <asn> weather the whether
17:39:04 <nickm> err, *weather
17:39:09 <ahf> 8am meetings should be illegal :o
17:39:12 <jnewsome> asn: oh cool, yeah I was wondering how that shook out
17:39:33 <nickm> asn: fwiw, wrt the trait thing, I recommend that you try it out at least.  I'm not 100% sure it'll work, but you'll definitely learn something more about rust
17:39:42 <asn> yes agreed
17:39:50 <nickm> (about 80% of what I know about rust comes from trying stuff that didn't work)
17:40:51 <jnewsome> heh. i'm fairly confident the trait approach will *work*, but there are bigger questions about whether it's a sufficiently effective test strategy, how maintainable it is, etc. but trying it out can help develop a feeling for some of that
17:40:53 <nickm> what's the next topic?
17:40:58 <ahf> ya
17:41:00 <ahf> [1/25 2021]: Should we do stable stable releases soon because of the client (and thus HS) consensus liveness issue that caused availability issues in early 2021?
17:41:10 <ahf> i think it's from nickm
17:41:18 <nickm> I don't think that one is from me
17:41:21 <nickm> though the next one is
17:41:34 <nickm> my plan has been "do stable releases shortly after 045x is stable
17:41:46 <nickm> do  we think we should aim for sooner than that?
17:42:13 <dgoulet> the sooner the better imo since 043 is EOL is ~20 days
17:42:24 <dgoulet> so would be wise to do at least one last stable in 043 to fix taht v3 problem
17:42:31 <ahf> huh, ah, i think i might have added it
17:42:46 <ahf> yes
17:42:54 <ahf> it was after dgoulet and i had a sync on DoS mitigation, sorry
17:44:09 <nickm> we can also put out an 043 then too.  Nothing stopping us from doing one last release
17:44:17 <ahf> dgoulet: is there more that needs to go in than what is there now?
17:44:27 <ahf> i think we only talked about the liveness patch
17:44:59 <dgoulet> yeah as far as I can recall, that is it
17:45:03 <dgoulet> an important fix
17:45:26 <ahf> ok
17:46:55 <nickm> I'm okay doing stables in early feb instead, if we think that's better
17:47:08 <nickm> I think this week would be a bit premature though
17:47:29 <ahf> ok, let's look at it next week then?
17:48:29 <nickm> ok w me
17:48:43 <ahf> last item:     [1/25 2021]: Will we also want to do another 045x-rc between now and the target stable date (15 Feb?)
17:49:49 <nickm> our best target if so is feb 1, which is next monday
17:50:16 <nickm> we should keep an eye on what we're doing in 045, and if we are merging anything that really needs an rc for testing
17:50:24 <nickm> do we anticipate anything like that landing this week?
17:50:40 <ahf> i have no strong opinion about this. i have the mystery issue with signals on gentoo hardened under load which may or may not be something fishy in our code, but we have only heard of it from one person
17:50:51 <ahf> and then the compatibility issue with old glibc/musl and glob()
17:51:05 <ahf> nickm: sounds fine to me with getting things in this week and then we look next week if we want to do one more?
17:51:27 <nickm> let's look towards the end of this week?  Trying to decide _on_ monday would be hard if monday is the target date :)
17:51:41 <ahf> ya, we can sync on thursday?
17:51:52 <ahf> and see if there is something
17:52:29 <nickm> ack
17:52:35 <ahf> ok!
17:52:41 <ahf> that was a lot of stuff, no more discussion items
17:52:45 <ahf> mikeperry: you wanna do the s61 part?
17:52:58 <mikeperry> s61 updates are in the pad, one thing to highlight:
17:53:22 <mikeperry> I am talking with multipath tcp folks about coupled congestion control and scheduling algs to minimize out-of-order queues
17:53:29 <mikeperry> I got some great answers
17:53:42 <mikeperry> but if any9one knows more of these people, I probably will have future questions!
17:53:45 <ahf> nice
17:53:46 <nickm> (this is for objective 3?)
17:53:49 <mikeperry> yes
17:53:54 <mikeperry> for the conflux proposal
17:53:55 <nickm> awesome
17:54:03 <mikeperry> currently in draft in my remote
17:54:18 <mikeperry> I am updating it this week based on their input
17:54:22 <asn> so i finished the CBT parts of O1.1 on EOY, and I'm kinda task-less wrt s61 right now
17:54:37 <asn> not sure what i should be doing next for this sponsor
17:54:53 <mikeperry> asn: dgoulet is dying under dirauth dos, but maybe you couold help him with the overload descriptor implementation?
17:55:12 <mikeperry> https://gitlab.torproject.org/tpo/core/tor/-/issues/40222
17:55:28 <asn> oh! dgoulet!
17:55:46 <dgoulet> yeah that is the one in the pipe that I had to put on hold
17:55:49 <mikeperry> asn: other option is understand onionperf filters and start looking at https://gitlab.torproject.org/tpo/core/tor/-/issues/40160
17:56:09 <asn> ack
17:57:11 <asn> ok ok
17:57:17 <asn> will check both of those tickets
17:57:20 <asn> and get in touch with dgoulet
17:57:26 <asn> to see what can be done
17:57:31 <mikeperry> kk great
17:58:39 <mikeperry> I think that's it unless there are comments or concerns from geko and juga
17:59:19 <GeKo> i think juga had a question
17:59:22 <juga> just whether you know if bwauths might be not voting on relays that are in the bandwidth file
17:59:31 <juga> cause of dos patches
17:59:50 <mikeperry> like the dos patches blocking scanning or something?
18:00:01 <juga> not blocking scanning
18:00:03 <mikeperry> I saw you got sbws to measure things torflow was not. is this part of that?
18:00:12 <juga> just discarding relays in the vote
18:00:23 <juga> mikeperry: nop
18:01:04 <mikeperry> hrmmm.. I thought torflow was rather patient with *keeping* measurements for relays that disappeared, in case they came back. I am not aware of cases where it would decide to get rid of measurements...
18:01:08 <mikeperry> but it has been a long while
18:01:33 <juga> the dirauths take the last decission on which relays to vote on
18:02:01 <mikeperry> oh and perhaps due to the dos, they are not getting connections from some relays
18:02:03 <juga> so far they voted on the ones in the bwfile, but might be slightly different right now
18:02:04 <mikeperry> and dropping them
18:02:23 <juga> yes, thats the only changed i saw in the scanner itself
18:02:41 <juga> relays in that disapear
18:03:05 <juga> anyway, probably not important
18:03:31 <mikeperry> yes in the past week or so, we could have seen massive drops of relays from some dirauths if they could not accept descriptors
18:03:45 <mikeperry> or had to drop some descriptor submission attempts
18:03:48 <juga> yeah, that makes sense
18:04:00 <juga> ok, i think my question is answered
18:04:35 <ahf> cool
18:04:46 <ahf> do we have more things for s61?
18:05:04 <GeKo> <- is fine
18:05:13 <juga> not from me
18:05:26 <mikeperry> I am groot
18:05:45 <nickm> hello groot i'm dad
18:06:03 <ahf> who is the rocket raccoon then!?
18:06:10 <ahf> ok, thanks all for the slightly long meeting
18:06:16 <juga> thanks!
18:06:16 <ahf> let's continue in #tor-dev and friends
18:06:19 <ahf> #endmeeting