22:59:21 #startmeeting snowflake network team + friends kick off 28/11 2018 22:59:21 Meeting started Tue Nov 27 22:59:21 2018 UTC. The chair is ahf. Information about MeetBot at http://wiki.debian.org/MeetBot. 22:59:21 Useful Commands: #action #agreed #help #info #idea #link #topic. 22:59:31 who is here? :-) 22:59:41 hi 23:00:03 hello 23:00:05 hihi 23:00:27 so we have a lot of sponsor 19/snowflake tasks on the network team roadmap, which we should begin with in december 23:00:36 at the same time we are closing down the s8 tasks 23:01:02 i think it would be good to figure out what our goals are for december up to our feature freeze for 0.4.0 in january 23:01:20 hey, is there a meeting here right now? 23:01:25 yep 23:01:28 oops, sorry 23:01:33 just started it :-) 23:01:41 I had my scrollback into other meeting in this channel 23:01:44 not sure why :P 23:01:46 ahhh! 23:01:57 do you agree with the lines i wrote above you? 23:02:01 hi 23:02:05 hi teor ! 23:02:06 o/ teor 23:02:28 hi all 23:02:38 hi! yes, we also wants to clarify the works that needs to be done for snowflake, where we are at 23:03:19 have others from the network team had a chance to try running with snowflake so far? tried to build it? tried to use it? maybe looked a bit at the code? 23:03:36 I've read the design document, and the paper, and the thesis chapter... 23:03:43 ok, cool! 23:03:46 and then I got sucked into reading about What Is WebRTC Actually 23:04:11 yeah, that was how i started too. didn't look at the thesis chapter though 23:04:18 cool 23:04:21 https://trac.torproject.org/projects/tor/wiki/doc/Snowflake is a very good site to get references to all of this 23:04:50 I found the "technical writeup" link there very helpful 23:04:55 source code is at https://gitweb.torproject.org/pluggable-transports/snowflake.git/tree/ and follows very standard go installation procedures 23:05:29 https://github.com/keroserene/snowflake seems to be in sync too if you are more into github 23:05:56 i don't think the javascript parts are that important for us to look at in the beginning 23:07:03 the go webrtc component had some troubles on my machines at first with building, but i think that was fixed after an i updated it. i have only tried on linux so far, not on macos 23:07:49 who is going to be interested in working on these things when we get to december ? and who will have time? 23:08:02 i'd quite like to split my december between this and whatever remains with s8 at that time 23:08:08 I might have some time, but I don't know yet. 23:08:11 I hope I do 23:08:25 One more thing that would be good from this meeting is to check on estimations in the issues in the roadmap and add tickets if possible or commit to add tickets to the issues there. 23:08:28 I think I will have to focus on PrivCount, but I might have some leftover time 23:08:45 gaba: agreed, i also think many of them are very abstract 23:08:54 and we might end up going into a rabbithole with some of them 23:09:13 stepping back, I think we should look at each of the roadmapped deliverables and ask two questions: 23:09:19 yep 23:09:21 teor, does the issue "Make sure Snowflake bridges publish the right stats (country count, etc)" is related? 23:09:24 * is this standing in the way of snowflake deployment 23:09:36 yes 23:09:43 * if not, does it help users significantly 23:09:55 and I think maybe we should prioritize snowflake deployment, if that's our goal 23:09:58 yes, good ones 23:10:07 ok 23:10:20 this means we can postpone the alternative PT design spec+implementation tasks till *after* january? 23:10:27 do we agree on that? 23:10:38 ahf: IMO yes, though we can start earlier 23:10:45 I bet we'll get ideas as we think about snowflake 23:10:57 they are marked as 5-20 points and i think they are the most abtract ones. we have the httpsproxy one we could use 23:11:00 yes, i think that too 23:11:22 * arma1 reads backlog 23:11:38 ok, marked them in our roadmap 23:11:46 we have a bridgedb task 23:11:57 bridgedb seems orthogonal to snowflake, to me 23:12:01 which i think could need some love, and i think sysrqb have become the maintainer of that 23:12:23 nickm: what does that mean in this context? that you think that part can wait? 23:12:47 arma1: o/ 23:13:00 I mean that I think that snowflake and bridgedb are independent, IIUC. Improvments to either one will help users. 23:13:04 I think we could work on them in parallel 23:13:18 right 23:13:19 Unless I'm mistaken and snowflake obsoletes bridgedb or something? I don't think it does 23:13:49 I don't think snowflake needs bridgedb 23:13:53 yes, and bridgedb is the other thing for the new anticensorship team 23:14:15 it doesn't need it, no, unless we want to use bridgedb to find alternative brokers 23:14:18 Actually, that was one of my questions 23:14:32 snowflake has its own 'distributor' or 'broker' which is equivalent to its bridgedb 23:14:33 How are we going to make each component of the snowflake system redundant? 23:14:53 arma1: yeah, but is the plan to use bridgedb to find brokers? or? 23:14:56 That's not quite true 23:15:11 ahf: no, the initial plan is to hard-code a broker (or two i guess) into the snowflake client 23:15:36 Ideally, we want multiple brokers, bridges, domain fronts, and STUN/TURN servers in the snowflake system 23:15:45 yeah, that is what you do today if you want to run it, you just specify one and a domain to front against 23:15:57 One thing that concerns me about snowflake is that anybody can sign up as a snowflake, and if you do, you get to enumerate clients and the bridges they use. Is that right? 23:15:59 teor: i agree about that, that has been one of the parts i've thought we should aim for too 23:16:30 nickm: in the current design, there are only a handful of bridges that all the snowflake users head to. 23:16:36 nickm: yeah, you can quickly get a view of the state for the given broker 23:16:48 that is, "many censored users <-> many snowflakes <-> a couple of entry points into the tor network" 23:17:09 But the snowflakes learn client IP addresses 23:17:12 ok. and censored users select these bridges from a list that the broker gives them? 23:17:43 nickm: i'm not sure. i think it might instead be that censored users get passed to a bridge by the snowflake, and they should be happy with the bridge they get. 23:17:50 yes, and rendevouz with them, possibly using the stun/nat punching services 23:18:04 uhh, hang on 23:18:27 ack 23:18:28 I'm not sure why we're using bridges here. Could the snowflake not just connect to the tor network directly? 23:18:32 the tor clients connect to the snowflake proxy, and the snowflake proxy connects to the brisge 23:18:49 nickm: the snowflake cannot make TCP connections 23:18:53 ah 23:18:54 nickm: the snowflake is just some javascript in a browser 23:18:58 nickm: it's a browser, so it uses websocket 23:19:01 until we have a tor client in javascript in a browser, no :) 23:19:10 so how does it connect to the bridge? 23:19:10 or we do a WebSocketORPort 8) 23:19:40 nickm: the bridge runs a server-side pluggable transport, and the snowflake talks to it using a web protocol 23:19:43 via websocket 23:19:47 ah 23:20:00 which is HTTP + a handshake + a framing protocol 23:20:02 clever 23:20:18 yes, it is very neat 23:20:34 So how are we going to make each of the components redundant? 23:20:48 so for deployment we also need to get some people to run these websocket bridges/relays/whatever we call them 23:21:01 ahf: i assume there is one running right now, probably by dcf 23:21:11 we could have multiple brokers by just shipping a list of several of them, and having the snowflake proxies and the clients just pick one at random 23:21:14 What is in a snowflake PT line? 23:21:28 arma1: are you mixing broker with this now? there is a broker running yeah, hopefully there is more than one bridge? 23:21:47 ahf: i would not be surprised if there is one bridge, run by dcf. :) 23:21:47 teor: broker + domain fronting name + stun server 23:22:20 each broker can have its own domain fronting; each broker can dynamically manage a list of STUN servers 23:22:22 Ok, so we can get redundancy on those 3 components by distributing 2 or more bridge lines 23:22:40 (by "can" i mean "could in principle") 23:22:46 and then we make each broker use a separate bridge 23:22:52 ClientTransportPlugin snowflake exec ./client -url https://snowflake-broker.azureedge.net/ -front ajax.aspnetcdn.com -ice stun:stun.l.google.com:19302 -max 3 23:22:55 is how it looks 23:22:56 how do the snowflake proxies know which broker to contact to offer proxy services? 23:22:56 teor: or set of bridges 23:23:11 indeed 23:23:12 catalyst: they specify a broker too i believe 23:23:35 What do we do now domain fronting is dead? 23:23:43 might we run into some scaling problems recruiting proxies if there are multiple brokers? 23:23:43 the snowflakes go to whichever broker is listed in the javascript they get 23:23:52 so, the idea behind this is that domain fronting was a problem because it used a lot of traffic i think 23:24:04 with snowflake it uses not a lot of traffic because it's just to meet one other client that works 23:24:13 then you switch over to speaking webrtc 23:24:18 teor: (a) domain fronting isn't dead, it still works on azure, and also it still secretly works on amazon as long as you're not loud, i hear. (b) there are two ideas for using AMP or amazon-messaging-something instead. 23:24:21 but it is a valid question i think if this gets users 23:24:25 (many users) 23:24:42 2x redundancy is not great, I think we should aim for 3x 23:24:46 I wonder if we want a snowflake proxy implementation that _doesn't_ make you run it in a browser. Personally I'd find that easier to run. 23:24:59 Also easier for testing 23:24:59 nickm: yes, a headless snowflake. there's a ticket for it. 23:25:04 ack 23:25:11 there is a ticket for headless snowflake and a ticket for a browser plugin 23:25:13 a webextension 23:25:32 so that it always runs in your browser if you opt in to it and install the plug-in, instead of having to visit some website with the snowflake proxy js code on 23:25:47 ahf: oh cool 23:25:51 How will we test snowflake as we add changes to it? 23:25:54 yeah; last thing I want is a tab that I'm never allowed to close :) 23:25:57 i absolutely *love* the idea of a webextension 23:26:10 the idea that everyone can contribute almost like a relay with a browser plugin makes me super happy 23:26:23 teor: good question. ahf: do you know what the state of the snowflake tests is now? 23:26:23 i think a webextension makes more sense as a one time effort 23:26:47 nickm, teor: i don't, no, that is not something i've had time to look at 23:26:58 arlo/dcf would be good to ask there 23:27:21 antonela: yep, i think it's mostly having a neat UI that can use some of the info from the browser such as "am i roaming" / "am i on fast network" and then a way to configure it 23:27:23 ok. The engineers who worked on it are all smart responsible people, so I think there are probably going to be tests that we can work with 23:27:25 I think we need a full tor integration test, which means we need chutney or something similar. It's not enough to just have partial integration tests. 23:27:49 teor: good points, that we don't have on our s19 roadmap right now 23:27:51 I agree that one of the first things we should do is to assess the state of testing 23:28:00 tesching chutney about PTs would be handy 23:28:06 looking at testing early does sound smart. 23:28:15 *teaching 23:28:24 (following on the "first write the whole set of tests, and then fill in everything so that the tests work") 23:28:27 and that might also sound like something that could fit well into our december plans which still have a lot of s8 too for some of us 23:28:28 Indeed. And I think we should do it as part of s19, otherwise we never will. 23:28:40 teor: yeah, that is very plausible 23:28:49 do any of you know how much time that could take? 23:29:02 Well, I could spend some quality time with chutney in December, and do the s8 bootstrap stuff, and the s19 PT stuff. 23:29:09 ahf: i can work on that UI - do you have the headless ticket close? 23:29:18 antonela: 2 sec 23:29:23 teor: let;s chat about that. it may work fine 23:29:45 gaba: so, integration tests in general is very open-ended. But teor could probably estimate the time for making chutney handle PTs and/or snowflake 23:29:51 Can a headless snowflake run from the command-line 23:29:55 #23888 is the webextension one 23:30:00 teor: yeah, i think that is the idea 23:30:07 ok 23:30:08 thx - cli too is cool 23:30:31 i think the headless one is called go-proxy, and the browser extension has no name yet 23:30:53 cupcake was the browser extension to do this for flashproxy. in theory it could be extended to be snowflakes too. 23:30:54 https://github.com/keroserene/snowflake/tree/master/proxy-go 23:30:59 gaba: if I do chutney travis tests, s8 bootstrap, and s19 PTs, it's about a week. A few days for each. 23:31:19 https://github.com/glamrock/cupcake/issues/24 23:31:25 Doing the travis tests makes the whole thing faster, because I don't have to test every combination 23:31:27 ok. would that means to let privcount on a side for a little bit, right? 23:31:31 should we move forward with the headless/extension version? or do we want to have the js web client thing first? 23:31:49 gaba, teor: also we should prioritize getting s8 bootstrap 100% done IMO 23:32:00 yes, that is the priority for sure 23:32:04 antonela: i think the js web thing is already there and is working right now? 23:32:05 The temptation to do other stuff will be real 23:32:27 yeah, we need to balance it here for december :-/ 23:32:38 gaba: yes, but I will need the chutney travis changes to test PrivCount as well, and the s8 bootstrap and PT changes will also help with PrivCount 23:32:45 ok 23:32:54 I have another silly question to ask. Is there anything blocking snowflake deployment that is _not_ on our deliverable list? 23:33:05 (chutney can launch PrivCount the same way it launches a PT) 23:33:28 nickm: i think the test point that came up was not on our list of deliverables for s19 23:33:33 nickm: url for deliverable list? 23:33:40 nickm: what is the current state of snowflake fingerprintability? 23:33:47 we are looking at the roadmap and the s19 deliverables 23:33:57 gaba: could you link to those for people? 23:34:00 https://docs.google.com/spreadsheets/d/1Ufrun1khEo5Cwd6OwngERn829wU3W3eskdrriaYfUBQ/edit#gid=856122210 23:34:01 teor: I don't know. 23:34:04 There is WebRTC fingerprintability here: https://trac.torproject.org/projects/tor/wiki/doc/Snowflake/Fingerprinting 23:34:05 teor: snowflake fingerprintability situation is not good. it used the webrtc data channel, which nobody else uses. 23:34:12 s/used/uses/ 23:34:12 But the transport itself might be fingerprintable 23:34:19 teor: there is some info on that on the snowflake trac page. one task could be to look at if the current chrome / firefox webrtc implementation matches that that is being used by go-webrtc right now 23:34:26 that is the roadmap for the network team, (roadmap sheet) 23:34:29 the go-webrtc lib uses the chrome C++ implementation i believe 23:34:56 Has anyone done a full privacy review? 23:35:08 arma1: there is a deliverables tab in that document BUT I think what we do about snowflake is flexible for s19, right? 23:35:10 For example, nickm's question about the snowflake proxies learning the client's IP address 23:35:22 i don't know the answer to that 23:35:47 teor: if we want snowflake to survive the china arms race, we're going to need somebody on-hand to do tweaking to get around blocking, plus also plan the longer term roadmap of making it more like other users of webrtc 23:35:49 yeah, the current tasks in the network team s19 roadmap is whatever we had on the postits in mexico i think? 23:36:17 yes, and is also starting some work that will go into the anticensorship team plate 23:36:45 arma1: tweaking is good, but "enumerate all the clients" and "enumerate all the proxies" are design issues 23:37:38 we may need a guard design here 23:37:58 teor: "spec for facilitator" and "improve robustness of facilitator" look like they're on-topic for that concern 23:38:10 yeah 23:38:26 and yes, the snowflakes can learn about some clients, just like current bridges can learn about some users 23:38:30 yeah, that should be handled by the broker, because it's too late after the broker has leaked your info 23:38:41 but can they learn about *all* clients? 23:38:49 it depends how the broker behaves. 23:39:04 there are a bunch of design choices around how bridgedb behaves, and the broker faces many of the same choices. 23:39:32 this is why i described the broker as snowflake's bridgedb 23:39:35 I honestly think that we could easily forget to look at enumeration in "spec for facilitator" and "improve robustness of facilitator" 23:39:57 we can add additional tasks 23:40:02 yeah, we should. 23:40:04 i don't think any of this is gonna be set in stone 23:40:12 i think this might be the most open tasks we have on our roadmap 23:40:24 we can move them around, add things, maybe not delete things 23:40:41 so, here's my question that has probably been answered before... 23:40:51 what has blocked snowflake roll-out up to this point? 23:40:56 Do we need a pipeline of replacement servers so we can act quickly when one is blocked? 23:41:13 nickm: the main issue i think is that it doesn't build on windows, so it isn't in tor browser. 23:41:25 teor: that would be smart. 23:41:31 huh, ok, i can look at that in december for sure 23:41:37 i'm starting to get used to windows by now 23:41:58 nickm: the secondary issue in the past was that the broker was super naive and gave out snowflakes inefficiently and then there were no snowflakes offered to new users 23:42:01 i think it just didn't build reproducibly on windows? 23:42:25 ahf: i'm not sure. it doesn't build in a way that tor browser finds useful. :) 23:42:39 ok, i can chat with GeKo about that i think 23:42:45 hm, how many web pages offered the snowflake proxy js? 23:42:56 i've put my name on the windows task on our roadmap now 23:42:58 arma1 is there where the web extension will improve snowflake proxy acquisition? 23:42:59 catalyst: very few. so there are very few snowflakes currently. 23:43:02 catalyst: probably not many 23:43:34 (gaba says she got disconnected and is trying to get back in) 23:43:36 i think we can get a bunch of snowflakes pretty quickly just by tweeting or something. it won't be the long term plan but it will get us 'enough' to start 23:43:36 arma1: can a snowflake only help one user at a time? 23:44:08 i think it can handle more than one user at a time 23:44:12 i'm almost sure about that 23:44:14 in that case, how do we "run out" of snowflakes to offer to new users? 23:44:17 like a relay can 23:44:18 nickm: in principle no, but i see this ticket called #25601 that makes me nervous :) 23:44:18 thanks catalyst. I'm back :) (on a driving car between mountains now...) 23:44:21 but it can run out of bw 23:44:29 yikes 23:44:42 nickm: we run out because the broker has a simple algorithm like "give out each snowflake three times and then don't give it out for a while" 23:44:47 ahf: see #25601 above? 23:44:48 #25483 is the ticket for building snowflake on windows 23:44:53 yeah, looking 23:45:22 s/on/for/ 23:45:26 For people with spare mobile devices, can we create an app or background process that snowflakes over wifi? 23:45:27 okay. Making the broker smarter seems do-able. Making a non-multiplexing proxy multiplex seems harder. 23:45:52 teor: yes and in fact nathan is excited to make it part of orbot, so when your device is sitting next to your bed, plugged in, while you sleep, you're helping people 23:46:08 Also FWIW, WebRTC is _huge_. We are not going to audit an entire webrtc implementation unless I'm deeply mistaken. 23:46:09 boklm: yep, that is the one we have in our roadmap too 23:46:21 ok, the #25601 situation is not so good 23:46:22 and for those of us with iOS? 23:46:45 I guess it's enough to have proxy-go on android 23:46:48 teor: no idea, but should be doable. 23:47:35 ("a simple matter of programming") s/programming/overcoming apple's monopolistic practices/ 23:47:38 So IMO within S19 we should prioritize stuff that we know we must do. That's going to be fixing the blockers above, and the testing stuff we talked about 23:47:40 iOS has funny ideas about background tasks. It would have to be a fake "VPN", and Apple might block it from the app store because it's not actually being a VPN 23:48:20 nickm is right. Essentials first. 23:48:35 nickm: that makes sense. another way of looking at it might be: what should the network team do, to make the new anti-censorship team hires be most effective once they start? 23:48:36 +1 23:48:46 yeah, i think doing apps is something we might hear benjamin from the guardian project about as well (for ios) 23:49:13 it sounds a bit to me the windows thing is an issue, testing is an issue, and the #25601 is a pretty bad issue for deployment right now? 23:49:19 If the browser proxy can't multiplex, then it can't accidentally collect lots of client IP addresses 23:49:46 But it could deliberately find lots of clients by failing fast 23:49:55 * Samdney lurks ... 23:50:02 teor: also, the failure mode of people not being able to get online seems bad. 23:50:19 yeah.. 23:50:23 the broker currently has to guess, when it gives out a snowflake to somebody, whether that snowflake is actually still in use 23:50:24 i think it is a scaling concern here 23:50:26 Indeed. We don't want to have to tweet every time snowflake gets low 23:50:29 arma1: is there a ticket for the problem with the broker handing out snowflakes in a silly way? 23:51:22 nickm: hunting 23:51:52 there's #25681 which i guess has it implicitly 23:52:10 but no, i think there is no simpler "just actually make it not dumb" ticket 23:52:17 ok 23:52:40 also the network team roadmap has a "spec for facilitator" line but i think there's no ticket for it yet 23:52:52 imo the spec comes first, because "what does it do now" is a good step zero before making it smarter 23:52:59 +1 23:53:00 * gaba was just looking for the spec ticket but no luck 23:53:16 arma1: in answer to your early "what should the network team to do make the new anti-censorship team hires effective", I think the most helpful thing we'll do will be engaging with the code in the first place. 23:53:28 yep 23:53:43 That way the anticensorship folks will find a crew of people who sort of know the basics of what they're supposed to be working on 23:53:56 speaking of future blockers, there's also #21314 -- at least currently, when my tor goes quiet/dormant, my snowflake keeps on sending traffic 23:54:14 oh, yeah, that will matter. 23:54:39 it's tied together with broader PT bugs, like #21967 23:54:50 yeah, i think the part about being able to work with the new hires with a smooth handover is important 23:54:56 When is the s8 deadline? December 31 2018? 23:54:58 like, if i configure my tor to not longer use bridges or PTs, the PT sticks around 23:55:00 teor: yep 23:55:11 well, killing the PT is one part. But PTs should generally try to be quiet when they aren't used 23:55:18 so that's another part 23:55:24 yep 23:55:40 ahf, teor: Yup, that's the deadline, but if we can declare it finished earlier than that, it will be much better 23:56:05 I'd rather minimize the amount of last-minute scrambling if I can 23:56:22 sure, we should aim for a few days before tor shutdown then 23:56:38 yeah 23:57:24 who on the network-team has actually used Go or JS for anything nontrivial before? 23:57:50 (I haven't, though I've read a bunch and done some tiny things) 23:57:55 I have read a lot of Go and JS, and written a little bit of each 23:57:59 i've done both go and some js before, mostly on the C++ side of javascriptcore in webkit 23:58:23 * Samdney ,too. although she isn't on network-team :) 23:58:41 * catalyst has dug around a bit in the snowflake proxy js, but is not a js expert by any stretch 23:59:01 https://exercism.io seems like a good site for programming language learning 23:59:18 i have some go code still running at my old job which they don't maintain and don't complain about i hear, i think that might be the best go i've done :-P 23:59:26 and that has been ~2 years soon since i left them 00:00:00 Is the snowflake proxy still written in CoffeeScript? 00:00:01 ok. we are at the one-hour mark. What else should we figure out here tonight? 00:00:05 so, we have reached an hour now. i plan on going over the backlog and maybe create some tickets for some tasks we have talked about here, put them into our roadmap sheet 00:00:29 and then at next network team meeting i hope we can maybe get some names on some of the important tasks we have identified here? 00:00:38 yes, or maybe the one after. 00:00:47 (I won't be able to make the next meeting, due to mozilla thing) 00:00:53 ahhh! right, next meeting then 00:00:57 gaba: does that sound OK with you? 00:01:06 i can work on the webextension UI on December cc pili gaba 00:01:41 antonela: do you know griffin? you might try to connect with him, since cupcake might be the closest path to having a working extension 00:02:00 teor: i think so, yep, it gets compiled down to JS 00:02:04 yes, oh cool, will ping him 00:02:21 we have been talking via twitter dms, is he on irc? 00:02:27 antonela: https://github.com/glamrock/cupcake/issues/24 00:02:46 antonela: I dunno; ask him? 00:02:56 antonela: he is sometimes 'saint' on irc but i have not seen him lately 00:03:02 oki, will do 00:03:13 Nickname last seen: Wed 14 Dec 2016 23:29:41 +0000 00:03:39 sorry i got disconnected again (just reading back) 00:03:42 ok, i'm gonna end the meeting log thing and go away from my computer for a bit. i'll do the things i wrote above 00:03:47 and thanks all for coming by o/ 00:03:52 keen. I think we made a good start 00:03:55 #stopmeeting 00:03:57 #endmeeting