16:59:38 <phw> #startmeeting anti-censorship weekly checkin 2019-10-17
16:59:38 <MeetBot> Meeting started Thu Oct 17 16:59:38 2019 UTC.  The chair is phw. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:59:38 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:59:47 <phw> hi everyone!
16:59:52 <phw> here's our meeting pad: https://pad.riseup.net/p/tor-censorship-2019-keep
17:00:01 <phw> it's quite the agenda today
17:00:30 <phw> let me start with our first discussion point
17:00:49 <dcf1> I will move two items to the bottom that can be skipped if there's not time today.
17:01:01 <phw> some context: we started supporting an ngo with private obfs4 bridges that it can distribute among its users
17:01:14 <phw> i tried to formalise the process a bit in this wiki page: https://trac.torproject.org/projects/tor/wiki/org/teams/AntiCensorshipTeam/NGOBridgeSupport
17:01:29 <cohosh> nice!
17:01:45 <phw> please add comments/suggestions/etc to the wiki page
17:02:00 <phw> so far, our private bridges exist in a csv file on my laptop. that's not great
17:02:21 <phw> ideally, they are in a shared, private location, and we keep track of who got what bridge
17:02:53 <cohosh> maybe i am misunderstanding this, but are these different from the unallocated bridges in bridgedb?
17:02:57 <phw> i could add the file to a private repository on dip.tpo unless anyone has a better idea
17:03:12 <phw> cohosh: yes, the bridges i'm talking about don't report themselves to bridgedb
17:03:19 <cohosh> ok cool
17:03:33 <cohosh> is the idea to use these two groups of bridges differently?
17:03:57 <phw> the unallocated bridgedb bridges are run by people we don't know and could go offline any time
17:04:20 <phw> the private bridges are run by womble, and are reliable and fast, and a much better fit for an ngo
17:04:32 <cohosh> ok thanks
17:04:47 <cohosh> i think gitlab sounds good
17:04:51 <phw> so far we've been using the unallocated bridges the way we're now using the private bridges
17:05:05 <phw> okay, i'll make the csv more usable and throw them in a repository
17:05:08 <cohosh> this also makes access to them more redundant
17:05:27 <cohosh> so that if something goes wrong with your laptop we can still give them out
17:05:39 <phw> yes
17:06:10 <phw> ok, shared access was the most important thing wrt this discussion item. as for the process: we can improve it as we go
17:06:39 <cohosh> sounds good
17:06:47 <phw> shall we talk about our ipv6 snowflake broker next?
17:07:34 * phw puts the mic on the ground and waits for somebody to pick it up
17:07:37 <dcf1> I've set up a new broker and documented the installation instructions.
17:07:54 <dcf1> https://trac.torproject.org/projects/tor/ticket/29258#comment:11
17:08:20 <dcf1> Figners crossed, I think all that's needed to start using it is to update some DNS records.
17:08:35 <cohosh> thanks for doing that dcf1!
17:08:36 <dcf1> But perhpas we should do a smaller-scale test first.
17:09:08 <dcf1> I mentioned on the ticket a proposal for dealing with concurrent logs; i.e., let them happen concurrently and merge them after we decomission the older broker.
17:09:41 <dcf1> One option is we give the new broker a hostname different than the snowflake-broker ones already in use; that way we can test it ourselves.
17:09:59 <cohosh> we have 3 different broker domains already
17:10:02 <dcf1> Another option is to only set up AAAA records now, so that IPv4 traffic goes to the old broker and IPv6 traffic goes to the new.
17:10:05 <cohosh> bamsoftware, freehaven, and tp.net
17:10:25 <dcf1> Yeah and freehaven in a CNAME to bamsoftware, so really we only need to update bamsoftware and torproject.
17:10:31 <dcf1> *is a CNAME
17:10:39 <cohosh> we could switch tp.net first and test with that
17:10:49 <dcf1> I forgot what names are used where.
17:10:54 <cohosh> since freehaven/bamsoftware is the deployed one
17:11:04 <cohosh> we haven't deployed tp.net in the client or proxies yet
17:11:24 <dcf1> Yeah I guess you're right.
17:11:24 <cohosh> due to concerns that some places (like the UK) are good places for proxies but may block tor project domains
17:11:54 <dcf1> And I guess that snowflake-broker.azureedge.net still points to the bamsoftware one, though I would have to check to be sure.
17:12:19 <dcf1> Okay, that's a good idea cohosh. We need to ask someone to update the torproject.net names to the IP addresses mentioned in the ticket.
17:12:30 <cohosh> i will make a ticket for that
17:13:14 <dcf1> Then we ourselves can test using the client with `-url https://snowflake-broker.torproject.net/' and proxy-go with `-broker https://snowflake-broker.torproject.net/`
17:13:22 <cohosh> or just cc anarcat on that ticket
17:13:58 <dcf1> I can handle making this ticket.
17:14:10 <dcf1> Thanks, unless there are any objections, I think this discussion point is covered.
17:14:24 <phw> thanks for this, dcf1
17:14:32 <cohosh> ok thanks
17:14:53 <phw> next up are some preliminary design considerations for a bridge test service: https://trac.torproject.org/projects/tor/ticket/29258#comment:11
17:15:07 <phw> to remind everyone: we currently have no service that tests a PT bridge
17:15:16 <dcf1> #31874 you mean
17:15:23 <phw> the one thing that gets closest is a simple port scan tool: https://bridges.torproject.org/scan/
17:15:30 <phw> oops, yes
17:16:05 <phw> there are two parties that would benefit from such a service: bridge operators could test their bridges and bridgedb could test what bridges are reachable
17:16:36 <phw> basically, you paste your bridge line in there (which works for vanilla, obfs2, obfs3, ...) and the service tells you if it could bootstrap a tor connection over the given bridge
17:16:59 <phw> i'm curious what y'all think about the points in https://trac.torproject.org/projects/tor/ticket/31874#comment:2
17:17:52 <dcf1> One consideration is misuse: an obfs4 testing service is like a public fuzz-testing service, send garbage to any IP:port on demand.
17:18:35 <dcf1> I suppose it could check that the bridge line matches a bridge already in BridgeDB? I.e., something that's intending to be a bridge?
17:18:43 <phw> dcf1: yes. for bridges.tp.o/scan/ we have a simple rate limiter.
17:19:13 <cohosh> do we need to make it publicly accessible? if bridgedb is automatically doing these tests, maybe that's more convenient that allowing operators to do their own anyway
17:19:24 <cohosh> *than allowing
17:20:03 <phw> we certainly don't need to but "does my bridge work?" is a very common question among new operators and we should have a useful response to it
17:20:34 <phw> i suppose "wait a few hours and check your relay status page" may also be a useful answer
17:20:44 <cohosh> is the intention to log and try to contact operators whose bridges bridgedb detects as being offline?
17:21:07 <phw> cohosh: yes, and also to not hand out bridges that don't work.
17:21:12 <cohosh> cool
17:22:22 <phw> thanks dcf1 and cohosh, these are useful questions and ideas. i'll update the ticket and give it some more thought
17:23:02 <phw> we can move on to the gettor workflow unless anyone has anything else to say
17:24:43 <phw> cohosh, hiro: ^
17:25:11 <cohosh> okay, so hiro has been doing some really awesome work on gettor, but we're not getting a lot of reviews done in time and so code is being merged without review
17:25:24 <cohosh> i'm wondering if we need to be better at communicating
17:25:34 <gaba> we should require reviews before merging
17:25:58 <hiro> yes we can do that
17:26:06 <cohosh> ok, so is the best way to do that moving forward to hand out reviews here?
17:26:14 <hiro> email I think is ok
17:26:34 <phw> gaba: agreed. when i don't review something in time, it's because i either did not realise that i'm supposed to review, or i forgot.
17:26:38 <gaba> we can check reviews in the meeting but if there is a review in the middle of the week people can communicate directly
17:26:39 <cohosh> is there a way to set it up with gitlab as well for us to get notifications for pull requests?
17:26:55 <gaba> yes
17:27:05 <cohosh> i find trac useful for that but gettor is all on dip it seems
17:27:07 <hiro> there is a bug I am trying to solve for which I haven't managed to do PR to the main repo atm
17:27:09 <gaba> we can get a column for a needs review label
17:27:13 <hiro> from my own repo
17:27:26 <cohosh> yeah i found the gettor board a bit confusing
17:28:16 <gaba> yes, the gettor board in gitlab needs more work. I can try to change it so it is useful for all of us
17:28:27 <gaba> we are in a weird space as we still did not migrate to gitlab
17:28:28 <cohosh> that would be really useful, thanks!
17:28:34 <gaba> so trac is the source of truth right now
17:28:39 <cohosh> yup
17:29:00 <cohosh> does all merged code in gettor correspond to trac tickets?
17:29:22 <hiro> well in the beginning it started with a massive refactoring from ilv
17:29:32 <hiro> that I had to merge back into our repository
17:29:41 <hiro> and in that there was some dangling code on the server itslef
17:29:55 <hiro> that I didn't know about until I tried to test
17:30:24 <gaba> it should correspond to trac tickets
17:30:31 <gaba> until we migrate we need to have trac updated
17:30:32 <cohosh> okay
17:30:43 <hiro> track tickets addressed features and some issues
17:31:15 <phw> hmm, if we update both trac and dip, i'm afraid we'll soon run into syncing issues
17:32:13 <phw> gaba: no?
17:32:16 <hiro> I have been closing dip tickets corresponding to trac tickets
17:32:54 <gaba> yes
17:33:09 * gaba will work on syncing both
17:33:18 <gaba> hopefully we will have the migration done in december
17:33:40 * antonela hopes
17:33:51 <phw> ok, so 1) there needs to be a trac ticket for each gettor merge, 2) gitlab and trac will remain in sync, and 3) review requests go out over email?
17:34:08 <gaba> ok
17:34:17 <hiro> sounds good
17:34:18 <cohosh> thank you gaba for all this work
17:35:09 <phw> anything else wrt gettor workflow?
17:35:44 <hiro> I think I am good
17:35:49 <phw> also, please ping me if i'm ever late on an email. reviews are high up on my list of priorities but sometimes i do forget
17:35:49 <cohosh> same, thanks!
17:35:56 <phw> s/email/review/
17:35:59 <hiro> sure!
17:36:13 <phw> ok, let's talk about s28/s30 next
17:36:36 <phw> regarding s28, aka RACE: we sent our part of the quarterly report to micah
17:36:39 <phw> and by "we" i mean gaba :)
17:37:04 <phw> and we have a prototype of our new obfs4 flow obfuscator: https://trac.torproject.org/projects/tor/ticket/30716#comment:16
17:37:23 <phw> (which i've been meaning to send to the traffic-obf@ list)
17:37:55 <phw> our partners are struggling with getting access to a dataset for evaluation
17:38:02 <phw> hopefully, we will sort that out in the coming weeks
17:38:25 <phw> and cohosh has been making steady snowflake progress as well
17:38:26 <gaba> nice
17:39:05 <phw> regarding s30: also steady progress
17:39:23 <phw> with some tickets we are lagging behind, with others we're ahead
17:40:12 <phw> anything specific you'd like to know gaba?
17:40:25 <gaba> just a check-in. thanks!
17:40:36 <phw> alright, next up is snowflake's poll interval
17:41:02 <dcf1> A few weeks ago I tailed the log of the broker, and requests were coming in furiously.
17:41:23 <dcf1> Lately those particular log lines have been removed, so it's not as apparent, but according to https://snowflake-broker.bamsoftware.com/debug there's 500 proxies,
17:41:39 <dcf1> and with a poll interval of 20 s, that's 25 incoming proxy requests per second.
17:41:42 <arma2> i liked the idea of having the broker tell each snowflake when to come back
17:41:59 <dcf1> Something on the order of 1 or 2 per second is probably adequate.
17:42:07 <cohosh> arma2: serna has started on that ticket
17:42:14 <cohosh> i agree
17:42:23 <dcf1> arma2: yeah that's #25598, serna ran into some trouble with that.
17:42:42 <cohosh> we have metrics of how many idle proxies we have: https://metrics.torproject.org/collector/archive/snowflakes/
17:42:57 <cohosh> and it is orders of magnitude more than the the number of client matches
17:43:31 <dcf1> Anyway, I think an interval of around 300 seconds would be workable.
17:43:37 <cohosh> sounds good
17:43:58 <dcf1> (I'm still confused as to why when a client connects and gets an ultra-fresh proxy, the proxy sometimes doesn't respond.)
17:44:25 <dcf1> Another thing we did in flash proxy was make the proxy not connect to the broker immediately, but wait about a minute before contacting the first time.
17:44:49 <dcf1> The idea there was to eliminate the cases where someone goes to the page or activates the extension momentarily just to see what it looks like.
17:44:56 <dcf1> I'm not sure if it really helped.
17:44:59 <cohosh> yeah, i'm doing some dog food and looking into that but it's *really* difficult to track down
17:45:18 <cohosh> it also seems to be 1/3 of all snowfalke proxies the last time i checked that do this
17:45:46 <dcf1> Hmm, yeah it's weird. It almost seems like it must be some systemic thing, like "all proxies on browser X" or something.
17:45:47 <cohosh> which suggests to me that it's probably not one-off user actions like that
17:45:53 <cohosh> yup
17:46:17 <dcf1> ok, I'll make a ticket to increase the poll interval.
17:46:30 <arma2> do the snowflakes offer specifics about themselves? like, can we distinguish cupcakes from snowflakes, at the broker
17:46:31 <cohosh> you know, i haven't updated the chrome version in a while. i'll do that this afternoon
17:46:42 <cohosh> i'm still unsure what the status of cupcake is
17:47:06 <cohosh> arma2: we can distinguish snowflake webextensions from standalones, that's about it right now
17:47:10 <dcf1> Oh I thought the Chrome store was just reall slow at reviewing, I noticed they never listed 0.0.11 before 0.0.12 was available.
17:47:29 <cohosh> i'm working on #29207 at the moment and can add something there
17:47:42 <cohosh> dcf1: no >.< i thought cupcake was our chrome app now
17:47:51 <cohosh> but i think it's best to just keep going with snowflake on chrome in parallel
17:48:09 <dcf1> Oh aha I see, once again I misunderstand the situation :)
17:48:18 <cohosh> i mean, they are slow at reviewing but not that slow
17:48:30 <phw> have we heard anything from griffin since the dev meeting?
17:49:00 <dcf1> No, I updated some tickets he's cced on, but I should sent email, because his email used for trac may not have been working.
17:49:56 <phw> doing both in parallel sounds good, at least until we hear from griffin
17:49:58 <arma2> cohosh: it does seem like "really difficult to track down" and "the snowflake could specify some details about itself when it contacts the broker" could go well together
17:50:11 <cohosh> yeah
17:51:05 <cohosh> alright i'll add that to the changes in #29207
17:51:34 <phw> are we done with the poll interval? if so, dedicated build server are next
17:51:57 <cohosh> yes please on the build server
17:52:12 <dcf1> So for the pion-webrtc tests I rented a VPS because building tor-browser is too much on a personal computer
17:52:14 <cohosh> i have a digital ocean instance that i use but my use is intermittent and not really worth what i pay for it
17:52:29 <dcf1> It's like 100+ GB and 48 hrs+ if starting from scratch.
17:52:42 <dcf1> And yeah, cohosh has been doing the same thing out of necessity.
17:52:59 <dcf1> Devs shouldn't be paying for this IMO.
17:53:13 <phw> yes, agreed
17:53:15 <dcf1> In the past, there was some EC2 server or something that was shared with the TB devs, and that was really nice.
17:53:38 <dcf1> But that got shut down and I'm not aware of any replacement.
17:54:00 <phw> let's file an admin ticket for getting such a vm?
17:54:01 <dcf1> One option would be to try eclips.is, but they have a credits system and we would probably need most of the 200 credits you get by default.
17:54:20 <dcf1> For comparison, the new snowflake broker with 10 GB disk and 2 CPUs is 25 credits.
17:54:29 <cohosh> storage space is a big issue here
17:54:54 <cohosh> 10GB might not be enough
17:55:09 <dcf1> I'm suggesting 200 GB for the build server
17:55:24 <dcf1> I'm saying that what we need is much bigger than the new broker, and the new broker cost 25 credits.
17:55:26 <cohosh> oh yeah that's way better
17:55:41 <dcf1> So eclips.is may not be enough, though I could try it.
17:56:03 <arma2> i think there are idle tor browser build machines currently
17:56:11 <arma2> but i might be wrong. it's worth asking them.
17:56:58 <GeKo> we  have one and i am using it usually to do release and test builds
17:57:15 <GeKo> but we could think about sharing that one more it that's helpful
17:57:28 <GeKo> *if
17:58:10 <dcf1> I think it's helpful, because for me for exmaple, just doing a test for #32076 before putting the ticket in needs_review is pretty cumbersome.
17:58:35 <dcf1> I happened to have a tor-browser-build VPS already set up, but I shut it down right after that ticket because it was costing money.
17:59:12 <arma2> i think sharing the build machines with the browser team makes a lot of sense
17:59:23 <arma2> since that's what you are both building
17:59:32 <cohosh> same here, i don't do incremental builds that often but it's nice to test patches before putting them in needs review
17:59:53 <cohosh> so it wouldn't be a lot of extra load on that machine
18:01:08 <phw> ok, we're out of time but have one item left on our agenda. shall we continue in #tor-dev?
18:01:43 <ggus> ok
18:01:52 <phw> looks like the only "needs review" for this week is #31890. i'll send my weekly reminder to sina
18:02:08 <arma2> phw: feel free to cc me on it if that's useful :)
18:02:23 <phw> #endmeeting