15:59:44 #startmeeting snowflake meeting 15:59:44 Meeting started Thu Apr 5 15:59:44 2018 UTC. The chair is asn. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:59:44 Useful Commands: #action #agreed #help #info #idea #link #topic. 15:59:56 hello people 16:00:10 how should we start? 16:00:19 arma4: you wanna give super short intro about sponsor? 16:00:26 or just jump into the snowflake part? 16:00:34 I started an agenda with what I think are the most important things to hit, but a short intro works 16:00:40 asn made a riseup pad. i was thinking step two could be "we spend a few minutes adding things to the agenda that we wanted to make sure we cover" 16:00:54 where step one is the super short intro: here it is, while you're looking at the pad: 16:01:22 we will soon have funding (i hope) to fund a snowflake developer. i hope to also fund an organizer project manager pusher-forward-of-things 16:01:39 there is a bigger picture around the funding, but the snowflake angle is what we're discussing today 16:02:00 there are some things i would love to end this meeting knowing or understanding, 16:02:09 and i'm going to go put them on the pad rather than try to write them here :) 16:02:12 ok 16:02:52 i think one basic thing i'd like to know 16:03:03 is what's the current state of snowflake 16:04:55 ha, i like how somebody put "roger notes" at the bottom of the pad and left the quotes around it while changing the text :) 16:05:10 not me not me 16:05:13 ok, i have put three hopes on the pad 16:05:14 - Get a handle on the current state of Snowflake 16:05:14 - What should be the future architecture for Snowflake? double down on c-go or be ready to switch? 16:05:14 - What do we want most in a Snowflake developer? Skills/background/etc 16:05:23 i think #1 is a great one to start with 16:05:27 yes plz 16:05:32 dcf or arlo, can you give us a snapshot of where it is now? 16:05:35 i think doing the meeting by editing the pad is a bit weird 16:05:40 When you all say "distributor", do you mean what we usually call the "broker"? 16:05:46 dcf1: i think so 16:05:47 [arma4 ergh, me, sorry, but yes, current state] 16:06:11 * Samdney lurks 16:06:17 the thing that hears from users that they need a snowflake, and gives out users to snowflakes 16:06:30 unless that's not even how this broker works :) 16:06:41 yes 16:07:12 current state is 16:07:13 the facilitator 16:07:32 client software is shipping in alpha bundles for linux and mac (not windows yet) 16:07:47 https://blog.torproject.org/tor-browser-80a5-released, choose snowflake from the menu 16:08:12 A small number of actual current users https://metrics.torproject.org/userstats-bridge-transport.html?transport=snowflake 16:08:29 but enough that we get cypherpunks tickets like https://trac.torproject.org/projects/tor/ticket/25600 16:08:40 (cool) 16:09:11 the broker is currently hosted on app engine, though we are working on moving to one that is separate from app engine 16:09:19 a windows build has happened, but it's not reproducible / automated yet 16:09:36 in fact the separate standalone broker already exists and is running, but (backstory) 16:10:04 the appengine broker (and current badge hosting page) are controlled by serene, who hasn't been responsive to requests for a long time 16:10:22 alas 16:10:25 there is a weird dependency chain of things to happen in order to effect the migration, which I have summarized here: 16:10:39 * arma4 adds "pick a name for the long-term snowflake code hostname" to the list 16:10:42 https://trac.torproject.org/projects/tor/ticket/23947#comment:3 16:11:24 (In fact we found from testing something else that there are already some people using the standalone broker; I guess they must have configured it themselves.) 16:11:45 cool 16:11:53 There exists browser-based JS proxy code, as well as a headless standalone proxy (proxy-go) 16:12:16 We're currently running several instances of proxy-go around the clock so that there's always some level of service available; 16:12:27 what's the status of the cgo bugs? you squished the big ones identified from montreal, but...some are remaining? or are your recent freeze mysteries in a different component? 16:12:34 they are on a static IP address though, so they don't contribute to actual blocking resistance, 16:12:56 but we wanted at least that someone who tries snowflake out of curiosity would not be disappointed that it doesn't work. 16:13:20 those are in the standalone proxy-go, not the client (re: cgo deadlocks) 16:13:46 arma4: I am not sure that these are "cgo bugs" exactly; a lot of the difficulty is due to the complexity of the C++ webrtc library, partly independent of cgo 16:13:53 you should read https://trac.torproject.org/projects/tor/ticket/25592#comment:1 16:14:04 arma4: https://trac.torproject.org/projects/tor/ticket/25688 16:14:34 what's proxy-go? 16:14:43 the standalone proxy (bridge)? 16:14:50 i think it's the standalone snowflake 16:14:54 But yes, it's true, until recently there were some memory leak and file descriptor leaks problems, which were especially acute in the client code and made it unstable. 16:14:58 the bridge is a different component 16:15:10 tor browser -> snowflake client -> snowflake volunteer -> bridge -> tor network 16:15:11 the bridge doesn't use webrtc at all 16:15:43 oh, there is probably a bonus item then, between snowflake volunteer and bridge, which unwebrtc's things for the bridge? 16:16:11 that would be the snowflake volunteer itself (browser or proxy-go) 16:16:18 it speaks webrtc on one side and websocket on the other 16:16:22 ah ha 16:16:33 dcf1: arlolra: so basically the current architecture of snowflake (wrt programming language, deployment etc.) is ok? 16:16:38 we shouldn't blame cgo so much? 16:16:41 webrtc for the censored area, websocket because it's simpler and snowflake-to-bridge isn't expected to be interfered with 16:16:46 because there were some bad feelings about cgo during the rome meeting 16:17:09 Could you say more about the feelings about cgo? Because I fear there is a misunderstanding about how the library works. 16:17:30 again, dcf1 wrote something useful here https://trac.torproject.org/projects/tor/ticket/25592#comment:1 16:18:00 arlolra: i read that but im not sure exaflty what it means 16:18:31 What Yawning wrote about Cgo on #25592 (thread performance) is, I think, not whatever you were concerned about in rome 16:18:31 dcf1: i felt that roger was saying that cgo has brought many fundamental problems like deadlocks and undeployability (?) 16:19:15 i think the rust people were also making comments about go, and cgo bindings in particular 16:19:34 i don't do rust or go or cgo much, so i am trying not to be the person with the strong opinions here 16:19:45 yeah i dont know about cgo either... 16:20:00 but, ok. asn, did you get your summary-of-current-state? any more questions there? 16:20:21 what's up with windows, which i understand is missing right now? 16:20:27 i think i've heard two different comments about cgo in the tor community: one is related to some old openssl cgo bindings that had some problems and the other being that if you have a fast-path somewhere with some cgo code, then the transforming the stack format back and forward is visible during profiling 16:20:47 * isabela is around 16:21:14 asn: as i understand it, it builds on windows, but not reproducibly, so tor browser hasn't taken it yet 16:21:31 it builds? neat! 16:21:39 okay, so here's the situation as I see it 16:21:40 or maybe that was "it built once" :) 16:21:51 it built once, with visual studio 16:21:59 oh! 16:22:11 from https://github.com/keroserene/go-webrtc/pull/69 16:22:34 About "hard to deploy", that is almost completely due to libwebrtc itself, not the golang bindings to the library 16:23:02 * GeKo wanders off and reads the backlog later 16:23:05 In fact native compiling of libwebrtc is not that hard, but doing it in a way tat's compatible with tor brwoser's build, cross-compiling from linux, is hard 16:23:25 In fact, cross compiling libwebrtc only because supported upstream about 5 months ago 16:23:39 https://trac.torproject.org/projects/tor/ticket/25483#comment:10 16:23:57 Before that, we were trying to work around and make it happen ourselves, like we did for macos 16:23:58 is libwebrtc under active development, say by the chrome team? 16:24:10 yes 16:24:13 (where cross compiling is also officially not supported but doesn't take too many hacks to activate) 16:24:28 and yes, it seems that for webrtc, libwebrtc from chrome is about the only game in town 16:24:35 if I'm not mistaken, firefox uses that code also 16:25:03 There was an independent OpenWebRTC (https://trac.torproject.org/projects/tor/ticket/22718) but it also did not support windows at all, and isn't maintained anymore 16:25:13 ok 16:25:29 asn: more questions? or all set? 16:25:36 So that problem is this: even if you decide you don't want to write in Go, and you rewrite the client and the standalone proxy in C++, you have almost all the same problems 16:25:42 gstreamer has webrtc support https://gstreamer.freedesktop.org/releases/1.14/ 16:25:45 dcf1: right 16:25:50 That said, cgo does introduce some level of additional complexity 16:25:54 dcf1: because of libwebrtc being weird 16:26:04 arlolra: oh yes, right, good point 16:26:10 dcf1: what level of additional complexity? 16:26:28 there could conceivably be a probject to amek a webrtc library that's easier to work with, but that's another thing 16:26:42 amek? 16:26:44 make 16:26:45 ok 16:26:49 asn: in the sense that there's some intrinsic complexity with any foreign-language function binding 16:26:57 right 16:27:13 like rust in litte-t tor, it doesn't come completely for free 16:27:18 are there aany other projects in this world using libwebrtc with cgo bindings? 16:27:39 if not, who made the cgo bindings for libwebrtc? us? 16:27:49 i would say yes because go-webrtc gets pull requests 16:27:56 It's a https://github.com/keroserene/go-webrtc, and yes, it's by us 16:28:04 there's also node.js binding to libwebrtc, so people are doing this 16:28:18 me and arlolra are maintaining it currently 16:28:20 https://github.com/js-platform/node-webrtc 16:28:41 ok 16:28:59 anything else to say here? 16:29:18 I thought node-webrtc also used the chromium code, though I'm not sure 16:29:31 it does 16:29:35 next on my topic list is: 16:29:36 What should we anticipate for future architecture changes for Snowflake? double down on c-go or be ready to switch? double down on the C++ webrtc lib, or be ready to switch? 16:29:43 it sounds like we answered some of that already 16:30:05 "we're stuck with the chrome webrtc lib, because it's our only hope, not because it's great" 16:30:29 correct, I personally would love to use something else, but we haven't found anything despite searching 16:30:34 but two questions: 16:30:59 (a) is there a way to move things forward where we couple more tightly, or less tightly, to the existing lib? and what are the tradeoffs there? 16:31:12 like, if 90% of the work is in figuring out bugs in the current lib, that effort isn't reusable for a new lib 16:31:53 in an ideal world we'd have our snowflake code, and a little abstraction barrier that calls webrtc things, and inside that abstraction barrier we map our webrtc things to how the current lib wants things to be 16:32:33 i guess an earlier question would be: "how much code is there and how complex is this thing? is it really just some glue and libwebrtc does all the real work?" 16:32:44 by 'this thing' i mean snowflake, not libwebrtc 16:32:54 (client-side and snowflake-side) 16:33:25 I mean, I guess go-webrtc kind of plays the role of an abstraction barrier 16:33:43 It binds both the C++ and the go code, all the other programs call Go functions in the go-webrtc library 16:33:49 aha 16:33:49 great 16:34:03 webrtc defines an api that either the browser or go-webrtc exposes 16:34:56 ok 16:35:04 yeah, so even if we decided to use, say, OpenWebRTC, we would still probably make the adaptation at the go-webrtc level 16:35:04 arma4: you have another question? 16:35:40 But it's lot we've committed to a stable API, other calling code would proabbly have to change somewhat 16:35:43 what do you expect a developer would be spending the most effort/energy on, to (1) get to a viable snowflake in tor browser with lots of users, and to (2) maintain it during the arms race? 16:36:12 (by 'what' i think i mean: snowflake code, go-webrtc code, libwebrtc hacking, or other) 16:37:42 or let me know if i should rephrase the question :) 16:38:42 Check current priorities here: 16:38:45 https://trac.torproject.org/projects/tor/query?status=!closed&component=Obfuscation%2FSnowflake&or&status=!closed&keywords=~snowflake&max=0&col=id&col=summary&col=status&col=owner&col=keywords&col=priority&order=priority 16:39:16 (Except ignore the one about a deb package, I wouldn't consider that so high) 16:39:36 There's basically three big steps to get to your (1), each requiring different skills 16:39:48 IMO 16:40:24 1. move the badge hosting page away from serene's server (#23947), and all the things that are depending on that 16:40:42 (needs sysadmin/server maintenance skills) 16:41:08 2. reproducible build for windows (needs build system hacking, shell script, cross-dev skills) 16:41:51 3. start producing snowflake (incl. browser extension, blog posts, etc.) (needs mainly social skills) 16:42:01 arlolra: what do you think about that characterization? 16:42:39 it's fair, but 16:43:03 not exactly to the point 16:43:09 seems like (1) and (2) will be the job of the snowflake dev, while (3) can be either the job of the snowflake dev or the project manager / cnesorship-lead 16:43:18 depending on the humans we find 16:43:31 asn: let's focus first on what we need, not yet how to get it :) 16:43:38 arma4: agreed 16:43:40 arlolra: say more 16:44:04 * Samdney thinks sounds not really like typical dev job... 16:44:06 the question was what the developer would be spending their energy on 16:44:18 right 16:44:31 and i think you're making good progress on 2. already 16:44:43 (i hope to do #1 tonight) 16:44:52 who is making good progress on #2? 16:45:01 dcf1, and hooman 16:45:09 ack 16:45:19 oi 16:45:32 do you need help with that? 16:45:38 with #2 16:47:26 well I'm still not sure the windows build will be finished anytime soon 16:47:32 a lot of the open tickets are about the broker and proxies, which seem more like golang / js dev 16:48:03 it's true there's recent progress, but a lot is sitll uncertain, and there's an anticipated long tail of issues down the line 16:48:11 but you have a point 16:48:25 i am aksing cuz maybe sukhe could help here 16:48:26 and maybe I shouldn't have been volunteering you for that 16:48:35 regarding arma4's point (2) "maintain it during the arms race", yes, go and js are required skills 16:48:40 ok. so there are three big picture dev roles i'm hearing. there's the client snowflake code, which has some windows build issues but is "mostly supposed to work", 16:48:42 if there is need for help :) 16:48:47 And also I would say there is still a fair amount of sysadmining 16:48:55 there's the snowflake itself, which is..js? 16:49:04 and there's the broker, which is golang? 16:49:08 hi yes, I would like to help with the builds if I can be useful 16:49:08 There's a JS one and a Go (headless one) 16:49:15 broker is golang 16:49:18 bridge is golang 16:49:55 JS is only used in the browser-based proxy 16:50:17 ok. so, pretending we get a new developer but not for a few months. thinking into the future: what will this developer be spending their energy on improving or maintaining or fixing bugs in or reacting based on censor behavior? 16:51:27 it sounds like all the components are designed so some single version of them can be hacked up pretty quickly (barring bugs like "omg libwebrtc on windows is harder than we thought") 16:51:27 various forms of broker DoS is one (empty ticket at #25593) 16:51:45 ok 16:51:58 s/single/simple/ 16:52:09 #22945 would be nice, avoid exposing information to the domain front 16:52:26 * asn ding dang 50 mins in meeting 16:52:46 you're still thinking tasks. i'm trying to think of classes of future tasks we haven't made tickets for yet. :) 16:52:48 #25598, dynamic adjustment of proxy polls will be good once there are more snowflakes 16:52:57 but yes, ok, let's get back to that one, and/or let arlo think about it too 16:52:59 we should talk about next steps after this meeting, as well 16:53:07 next item on my list: What would dcf and arlo like their roles to be over the next year? 16:53:23 arma4: these are all things that, if I were doing them myself, I would consider priorities and probably budget at least 6 months 16:53:34 ok 16:53:35 but I guess I didn't understand what you were asking 16:53:49 no worries. we're making progress on better understanding all around i think. 16:55:01 and while you're thinking about how to answer the future roles question, there's also the "what should serene's role be". unless that one's easy and she's having fun playing her piano and she has gone totally awol. 16:55:02 I was planning on continuing to contribute to this over the next year 16:55:27 unfortunately, I haven't heard from serene in a long while 16:55:51 i heard from her in november 16:55:59 I'm going to see her play piano tomorrow, but I'm not planning to bring up anything about snowflake, I think she's put it behind her 16:56:09 :) 16:56:44 I think we should meet again in a week or two to discuss this further and hammer out the requirements for a job posting 16:56:47 I don't mind continuing to work on snowflake 16:56:56 but would definitely defer to whoever you hire 16:57:22 IMO it was a problem in snowflake development for a long time, unclear leadership, and I wouldn't want to exacerbate that again 16:57:42 hear hear 16:57:43 t0mmy: i like that idea 16:57:45 ok. the timing on getting a good dev will be exciting, since i'm still trying to wrestle the funding out of the funder, which is also a full time job it turns out, but that's a different story :) 16:57:45 ack 16:58:01 ok, sounds good. two short term questions i want to cover in the rest of the meeting: 16:58:02 so 16:58:12 we should have a job draft for the next meeting too 16:58:16 does 17:00 UTC next Thursday work with people? 16:58:20 (1) isa has been brainstorming with sukhbir, and they were thinking that one thing he might be great on is helping with the windows reproducible build stuff 16:58:27 i think we should change the time tho 16:58:38 is that something he can jump in on nowish? he can meet with dcf and hooman and see where things are and whether he thinks he can be helpful? 16:58:45 isabela in an hour later not enough to not conflict? 16:58:49 arma4: tx, yes that is a thing :) 16:59:04 dcf1 arlolra what do you think? 17:00:04 I'll send sukhe the most recent email I sent hooman, with an update of the current status and a howto, and sukhe can see how it strikes him 17:00:09 t0mmy: depends sometimes it will be that we end right at 1pm (now) today we ended earlier - that would not conflict but it would be one right after the other no breaks 17:00:13 dcf1: great 17:00:14 sukhe is very familiar with rbm, so that helps 17:00:25 awesome 17:00:27 that sounds promising 17:00:37 asn would 18:00 UTC be too late? 17:00:44 probably 17:00:53 t0mmy: i think let's try 17:00 utc 17:00:54 let's put it for 18:00 UTC and we see 17:01:01 =) 17:01:04 17:00 UTC is better. 16:00 UTC is also ok. 17:01:09 but 18:00 UTC can also work 17:01:16 17:10 17:01:17 :D 17:01:18 1600 utc makes all the vegas team people have to multi-task 17:01:19 16:00 UTC would work on a day that's not Thursday 17:01:24 10min break from one meeting to another 17:01:33 dcf1, arlolra: ok, and item 2: what name would we like for #23947? 17:01:43 i think we should put it on a foo.torproject.org name 17:01:57 and, this is just static files right? we can just add another static component to our webserver rotation 17:02:06 i even talked to weasel about it yesterday and he suggested this 17:02:07 whatever, put it at 18:00 UTC 17:02:09 i will figure it out 17:02:15 yeah, static files 17:02:23 yes, just static files, we set cookies and stuff, but only through JS. 17:02:24 asn: 17:10? 17:02:30 (Don't know if cookies are a problem for tpo) 17:02:35 isabela: that's also ok. but i'm also good with 18:00 17:02:35 arlolra, dcf1: so, the model would be that you scp stuff over to a tor host, and then log in and run a sync command when you want stuff to go live 17:03:07 lets do 17:10 17:03:21 hmm, cookies! the cookies are... subdomain specific? or.. surely browsers wouldn't offer snowflake cookies to every other torproject.org site they go to? 17:03:24 snowflake.torproject.org I think? 17:03:36 isabela: ok 17:03:38 arma4: I'm not sure. Cookies are to make the opt-in work 17:03:53 ok. if we pick snowflake.tpo, we can also put html on the front of it and it can become the website for the snowflake stuff 17:04:08 and the auto fetched code can be somewhere tucked away that humans don't have to run across first 17:04:17 Picture the embed from https://keroserene.net/snowflake/, and replace "keroserene.net" with the chosen name. That's what we'll be telling people to do. 17:04:18 ok, i will check with weasel about the cookie question 17:04:30 ok 17:04:35 so what should we do until the next meeting? 17:04:44 someone said something about draft job posting? 17:04:58 i did 17:05:05 :) 17:05:11 one of the things i want us to do is to flesh out that roadmap more. that is, what are the categories of things that snowflake needs, to succeed 17:05:20 yes 17:05:23 it's not about identifying specific tasks, though those can be useful to understand context, 17:05:39 asn isabela dcf1 arlolra ok, 18:00 UTC next Thursday? 17:05:40 ok 17:05:41 but it's more about making sure there aren't entire types of things that we're going to say "oh shit" about when we realize nobody has any plans to do them 17:05:53 arma4: ack 17:06:00 t0mmy: yup 17:06:01 yes 17:06:02 so some sort of medium/high-level roadmap planning 17:06:04 t0mmy: i'll do better at 17:10 actually. my thursday afternoon gets bad the later it goes 17:06:14 i want to give another pass on that spreadsheet and update it with tickets and follow ups 17:06:32 can help organize roadmap for snowflake there too 17:06:35 in another tab 17:06:39 dcf1: (please forwad the windows build status, thanks!) 17:06:47 i think it is still unclear what exactly we would want out of a snowflake dev. that is, there are a bunch of small things floating and if they land then we can not get distracted so much by them 17:06:55 i think it is more clear what we want in a snowflake project manager 17:07:02 and maybe that's the thing to focus on first 17:07:07 ok 17:07:10 17:10 then, I'll send an email reminder 17:07:10 we can do that 17:07:12 like, a person to lead and organize and help track progress 17:07:17 t0mmy: tx 17:07:26 and know what's getting done and what *isn't* getting done 17:07:34 and help us make sure all things get done 17:07:36 arma4: yes and this person will help do everything else so is priority :) 17:07:43 hehe 17:08:04 also i secretly have a hope that this person will slowly transition into being our generic second project manager 17:08:07 so isa can have a friend 17:08:22 :D 17:08:23 awh 17:08:29 tx arma4 17:08:44 and in the short term, sukhbir on windows build, and arma on "get them their hostname already", should be useful progress 17:08:53 yes 17:08:55 plus drafting the job posting 17:09:09 yes that too 17:09:20 ok 17:09:30 ok great! thank you all for taking the time; sorry for keeping y'all past the hour 17:09:40 dcf1: arlolra: thanks so much for all the useful info! 17:09:41 awesome. we're ten minutes over the number 60 that somebody might have pointed to. we have a bunch of plans? including a plan to re-convene in a week? 17:10:01 yes sir 17:10:06 which job posting we talking about? :) 17:10:08 thanks t0mmy for organizing it! 17:10:10 both? or just once? 17:10:11 thanks everyone! 17:10:13 what time did we settle on? 17:10:15 17:10 17:10:16 UTC 17:10:17 17:10 utc 17:10:20 ty 17:10:23 asn: team lead first 17:10:27 ok 17:10:33 yep, I'll send a follow-up 17:10:41 sounds good 17:10:43 isabela np =) 17:10:57 i will have time to work on this project again from tuesday and onwards fwiw 17:11:07 im gonna end this meeting! 17:11:10 * asn waits a bit 17:11:27 thanks. i'm going to still be here after the meeting in case anybody wants to ask anything or tell us anything :) 17:11:30 #endmeeting