16:58:50 <ahf> #startmeeting Network team meeting, 21st June 2022 16:58:50 <MeetBot> Meeting started Tue Jun 21 16:58:50 2022 UTC. The chair is ahf. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:58:50 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic. 16:58:54 <ahf> yo yo yo 16:59:05 <nickm> hiya! 16:59:09 <Diziet> Afternoon. 16:59:13 <ahf> welcome to this meeting that will feel like it's still monday the 13th since all days in last week seemed a bit like it was monday 16:59:16 <ahf> our pad is at https://pad.riseup.net/p/tor-netteam-2022.1-keep 16:59:22 <GeKo> o/ 16:59:34 <mikeperry> o/ 16:59:56 <eta> o/ 16:59:58 <dgoulet> o/ 17:00:00 <ahf> okay, how are folks doing with their boards: https://gitlab.torproject.org/groups/tpo/core/-/boards ? 17:00:24 * eta is okay 17:00:39 <nickm> doing okay here too. 17:00:56 <eta> mikeperry: if you could get me some updated RTT test vectors that would be very appreciated, btw 17:01:08 <ahf> do take a moment to notice the much smaller number of tickets open thanks to, especially, nickm's epic triage round thursday+friday 17:01:15 <eta> though I think the next thing I'm going to tackle is probably the ntor handshake 17:01:19 <dgoulet> a bit stalled here while this DoS is ongoing and I keep trying to find ways to stop it 17:01:27 <nickm> ahf: nickm+dgoulet 's epic triage :) 17:01:42 <Diziet> eta: Could trouble you for the rereview on the channel MR ? 17:01:53 <mikeperry> eta: yeah, sorry, I got paralyzed by DoS and stuff; also was a bit wary of updating the spec publicly before release packages were out 17:02:02 * eta nods 17:02:03 <mikeperry> but I can do it this week 17:02:16 <eta> wfm! I don't think I should be that blocked on it for a while 17:02:30 <Diziet> mikeperry: You need to ask me for something you're blocked on, now, I think ? To make it a full circle (triangle) 17:02:45 <ahf> nickm: ya! 17:03:26 <ahf> ok, i don't see anything off 17:03:41 <ahf> dgoulet: on releases, i don't think we have anything here, right? nice work with the emergency release that happened 17:03:42 <Diziet> nickm: I haven't said this yes, so: thanks for digging through tickets! I hope you enjoyed it more than I would have! 17:03:51 <dgoulet> yeah security release on friday... 17:03:55 <mikeperry> Diziet: I guess rust prevents deadlocks so we need to invent them at the team level? keep things spicey! ;) 17:04:01 <nickm> Diziet: I love that kind of thing 17:04:03 <dgoulet> got a bit messy at the end with that CVE waiting time but we got it.... 17:04:04 <dgoulet> that is about it 17:04:13 <Diziet> nickm: I love that I have you on our team, then :-) 17:04:37 <Diziet> mikeperry: (Also Rust totally doesn't prevent deadlocks, sadly...) 17:04:37 <ahf> dgoulet: very nice - thank you! 17:04:43 <ahf> it looks like we don't have anything bad incoming 17:04:45 <nickm> "That's what makes horse races" 17:04:46 <eta> Diziet: I'll push it onto my stack 17:04:52 <Diziet> eta: Ta. 17:05:01 <eta> I've now forgotten whether I was okay with tolerating the ChannelsConfigUpdates thing 17:05:11 <Diziet> You said you'd live with it, iirc. 17:05:18 <eta> that sounds like something I'd say :p 17:05:25 <ahf> lol 17:05:29 <Diziet> https://gitlab.torproject.org/tpo/core/arti/-/merge_requests/586#note_2814258 last para 17:05:47 <Diziet> I was paraphrasing, in fact. I think it may be the kind of thing that I would say. 17:05:57 <eta> I'll go check 17:06:57 <Diziet> That's what I took as a grudging OK. If you still feel it's a premature optimisation and want it reworked I think we'll have to have a conversation about that but not here. 17:07:22 <eta> Diziet: I still think it's a premature optimisation, actually 17:07:47 <eta> (looking back at the scrollback, I think that was where I intended to leave things before the passport office got in the way :p) 17:08:20 <eta> though, sure, we can chat about that — I'm receptive to the idea that you might not want to rework it any more if you've already done a fair amount of that :p 17:08:34 <Diziet> Let's take that to #tor-dev after this meeting? 17:08:39 <eta> wfm 17:08:51 <ahf> ok! 17:09:12 <ahf> we have two topics 17:09:20 <ahf> [2022-06-21] [nickm] Anything to do to prep for hackweek next week? 17:09:35 <ahf> i think the important thing is that people find something they want to work on here 8) 17:10:16 <eta> how does the hackweek work? 17:10:24 <jnewsome> oof, next week. that snuck up on me :) 17:10:29 <eta> (are we dropping everything to focus on the hackweek tasks?) 17:10:33 <Diziet> I see an email from 26th of May which err I seem to have spaced. 17:10:43 <ahf> ya 17:10:47 <ahf> also no meetings 17:10:55 <eta> sweet 17:10:55 <emmapeel> eta: you can present a proposal or join one of the proposals already presented 17:10:59 <ahf> is a good idea to try to team up also with people outside of just the netteam 17:11:09 <emmapeel> https://hackweek.onionize.space/hackweek/schedule/#0 (ignore the 17:11:09 <emmapeel> "time/date") 17:11:51 <Diziet> The CFP deadline https://hackweek.onionize.space/hackweek/cfp is ... quite relaxed. 17:11:56 <eta> perhaps I'll go help richard with gosling :p 17:12:57 <nickm> ahf: I'd like to pick your brain after the meeting about whether any of the ideas I suggested are things I should offer to do 17:13:13 <ahf> ok, sounds like people are now *aware* of it at least, so spend some time this week looking at options or create options here 8) 17:13:28 <ahf> nickm: cool! i think i can give feedback but in the end it is also what you would like to do 17:13:30 <Diziet> Is it possible to see the other proposals or is that what "Schedule" is ? 17:13:39 <ahf> emmapeel: ^^ 17:13:45 <eta> (I think the schedule is indeed a list of other proposals, aiui?) 17:13:48 <nickm> I don't know if the schedule == 100% of the other proposals or not :) 17:13:57 <eta> ah 17:14:23 <emmapeel> yes, the schedule is a list of other proposals AFAIK 17:14:33 <emmapeel> maybe there are more proposals, ggus may know more about this 17:14:36 <ahf> i think people can propose multiple things, but they end up only working on one 17:14:57 <emmapeel> please ignore the date/time 17:15:03 <emmapeel> yes, rhatto was wondering what to do with his 2 proposals as well 17:15:22 <ahf> nice 17:15:27 <ahf> ok, i am gonna move to next item: 17:15:31 <ahf> [2022-06-21] [nickm] Next arti release date? (This friday or next friday?) 17:15:31 <Diziet> That list of proposals is ... quite short given the number of people, I think ? 17:15:43 <Diziet> From which, can I infer that everyone else is as late as me ? 17:15:44 <ahf> Diziet: i think people are, like us, a bit "oh already next week" right now 17:15:46 <emmapeel> but as we can stop work in sponsor deliverables, and meetings, we also have more time 17:15:52 <Diziet> ahf: lol right 17:16:07 <emmapeel> (sorry, lag) 17:16:40 <ahf> nickm: i would say this friday given the above discussion? 17:17:16 <nickm> works for me; it'll come out earlier than it would have otherwise, but it will encourage us to pay attention to the hackweek instead 17:17:38 <Diziet> arti releases don't seem to generate fallout that means we have to put out fires. (I don't think we have that many users yet...) 17:18:06 <ahf> i think everybody should clear their minds for hackweek so they can dive into something new without having things in the back of their heads 17:18:18 <ahf> ... i know this will likely be impossible with the DoS ongoing too, but .. :-/ 17:18:53 <ahf> anything else we need to discuss before we move to s61? 17:19:21 <nickm> all good here :) 17:19:42 <ahf> mikeperry: wanna dive into s61 ? 17:19:45 <mikeperry> ok 17:20:21 <mikeperry> so hiro and i have been looking at different guard and CBT params: https://gitlab.torproject.org/tpo/network-health/analysis/-/issues/37 17:20:49 <mikeperry> we set up new onionperf 7a instances with 2 guards + 70% cbt cutoff, and 3 guards with 66% CBT cutoff 17:21:11 <mikeperry> the theory being that more guards plus lower CBT could avoid relays overloaded due to dos 17:21:39 <mikeperry> 3 guards + 66% CBT seems to result in about a 2X throughput boost 17:21:54 <mikeperry> 2 guards + 70% CBT is maybe 1.25-1.5X.. but it is hard to see 17:22:23 <mikeperry> hiro: I think we're gonna need to get the tornetools workflow going, as per https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/issues/17 17:23:02 <mikeperry> because the days of good vs bad perf from this DoS are all over the place 17:24:32 <mikeperry> if we get all that filtering done, we can cross-reference sbws and see how much changing Guard+Fast cutoffs helps. 17:25:32 <mikeperry> we could change the guard and CBT params with consensus param update right now, and this would help people, but perhaps not as much as actually changing those cutoffs 17:25:49 <mikeperry> though changing the flag cutoffs requires a consensus update.. 17:26:37 <mikeperry> hiro: so I will likely be pinging you trying to get some tornettools workflow soon 17:27:28 <mikeperry> juga was also looking at sbws votes in https://gitlab.torproject.org/tpo/network-health/analysis/-/issues/40 17:27:53 <mikeperry> it looks like two of our bw auths have a throughput bottleneck of around 25Mbit :/ 17:28:32 <mikeperry> I believe sbws is the main thing that is making the DoS not be miserable, all of the time.. but with only one guard, some users are probably still getting unlucky 17:29:42 <mikeperry> we also had the TROVE, of course. we still see no evidence of active exploitation AFIACT 17:29:48 <GeKo> farahavar can do around 35Mbit at least 17:29:49 <GeKo> but yeah 17:29:59 <mikeperry> we would see this evidence in https://gitlab.torproject.org/tpo/network-health/analysis/-/issues/38 17:30:33 <mikeperry> as well as relay logs 17:31:25 <mikeperry> in general the DoS is really fucking up my world. it looks like sbws upgrading to CC was a major missing piece of our bad perf wrt not matching what shadow predicted 17:31:28 <GeKo> *faravahar actually 17:31:33 <mikeperry> but with the DoS, it is very hard to tell now 17:32:35 <jnewsome> need to call a timeout with the DoSers so we can get some measurements... 17:33:41 <mikeperry> do we have a new Exit upgrade rate datapolint for 0.4.7? 17:34:11 <ahf> jnewsome: :s 17:34:49 <GeKo> mikeperry: it's not making any progress anymore 17:35:01 <mikeperry> what is the percentage? 17:35:31 <mikeperry> hiro: We can also filter out non-upgraded Exits for https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/issues/17 17:35:45 <GeKo> 1297: 0.4.7 [81.28 %] [84.34 %] (MAJOR) 17:36:00 <GeKo> the first is consensus weight 17:36:06 <GeKo> the other is adv bw 17:36:20 <GeKo> that's exits fwiw 17:36:29 <mikeperry> yeah ok 17:36:31 <mikeperry> thanks 17:36:36 <GeKo> whole network is 17:36:38 <GeKo> 4210: 0.4.7 [64.99 %] [63.05 %] (MAJOR) 17:37:55 <mikeperry> ok 17:38:41 <mikeperry> so yeah, major action items are https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/issues/17, and trying to work with bastet and Farahavar bwauths to figureout if they can improve their bottleneck 17:39:08 <mikeperry> and I will update the spec and test vectors for the TROVE 17:39:20 <jnewsome> mikeperry: i'm still working on simulating link sharing as well 17:39:50 <GeKo> mikeperry: should i wait looking closer into: https://gitlab.torproject.org/tpo/network-health/analysis/-/issues/38 17:39:52 <GeKo> ? 17:40:14 <GeKo> right now i am wary that this would be a timesink not giving us the data anyway we want 17:40:25 <mikeperry> GeKo: I think looking at the sbws votes for those outliers may be useful, but yeah, it may need several days of datapoints 17:40:37 <jnewsome> i could deprioritize it if you no longer think it's likely to be a major factor, though at this point might as well try to finish the implementation while it's fresh in my mind. https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/issues/16#note_2811688 17:40:44 <mikeperry> none of them get the overload flag? 17:41:09 <GeKo> i could check that but i doubt it 17:41:31 <GeKo> given that only one op*a instance has issues with any of them 17:41:46 <mikeperry> jnewsome: yeah, worth finishing.. but ideally we would compare against a post-sbws upgrade time period.. but all of those are riddled with DoS, so it is hard to say if we're making it closer to reality now, or just closer to the DoS state 17:41:56 <GeKo> so, it seems there should be a different reason for almost any of them showing up on the outliers than general overload 17:43:35 <mikeperry> dgoulet is noticing that the DoS appears to be moving around.. so it may be emphermeral wrt this.. but it might also be worth trying it later in the week once we have several days of data to look through 17:44:07 <GeKo> okay. i'll look a bit more then 17:46:23 <mikeperry> I think that is it for me. any questions? 17:46:54 * ahf is good 17:47:31 <GeKo> me too 17:47:59 <ahf> let's call it 17:48:02 <ahf> thanks folks! 17:48:06 <nickm> peace all! 17:48:06 <ahf> #endmeeting