15:58:34 <meskio> #startmeeting tor anti-censorship meeting
15:58:34 <MeetBot> Meeting started Thu Oct 20 15:58:34 2022 UTC.  The chair is meskio. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:58:34 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
15:58:36 <meskio> hello all
15:58:40 <meskio> here is our meeting pad: https://pad.riseup.net/p/tor-anti-censorship-keep
15:58:42 <shelikhoo> Hi~
15:58:43 <meskio> here is our meeting pad: https://pad.riseup.net/p/tor-anti-censorship-keep
15:58:50 <meskio> feel free to add what you've been working on and put items on the agenda
15:59:28 <itchyonion> hello
15:59:54 <meskio> our first point in the agenda is 'blocking by TLS fingerprint in Iran'
16:00:10 <meskio> mmm, I see someone still writting there, let's wait for a bit
16:00:15 <dcf1> I'm writing a summary quick
16:00:21 <meskio> thanks :)
16:01:45 <ggus> hello
16:01:57 <dcf1> I did a lot of investigation into blocking in Iran in the past week
16:02:21 <itchyonion> hello
16:02:37 <dcf1> What started a minor breakthrough was looking at a graph of directory requests across multiple countries, with a few days of context before and after Oct 04
16:02:41 <dcf1> https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40207#note_2844116
16:03:24 <dcf1> my conclusion is that the loss of traffic on Oct 04 *is* due to a block in Iran, that apparent effects in other countries are geoip errors, and that the mechanism of blocking is TLS fingerprinting using at least two identified features: ciphersuite order and minimum supported TLS version
16:04:07 <dcf1> I looked at the available blocking reports, and identified two TLS fingerprint features that have to do with blocking
16:04:35 <dcf1> namely, whether the platform has AES acceleration (affects ciphersuite order) and whether it was compiled with go1.18 or later (affects minimum TLS version)
16:04:58 <dcf1> Of the 4 quadrants in the matrix, only 1 is blocked, which happens to be the fingerprint of released versions of Orbot
16:05:24 <dcf1> As far as I can tell, Tor Browser on desktop and Tor Browser for Android are working in Iran and have been
16:05:59 <dcf1> (btw Orbot is preparing a new release with utls enabled that may fix this: https://github.com/guardianproject/orbot/releases/tag/16.6.3-BETA-2-tor.0.4.7.10)
16:06:27 <dcf1> That's my summary of the TLS fingerprint blocking situation, which leads to the next discussion point
16:06:54 <meskio> nice that orbot is reacting to that
16:06:57 <cohosh> wow nice work dcf1
16:07:06 <meskio> would be nice to have orbot using circumvention settings
16:07:20 <meskio> but that needs some work
16:07:35 <dcf1> it's a bit tricky because orbot (at least currently) does not use bridge lines the same way
16:08:02 <dcf1> custom bridges can only be obfs4; the snowflake configurations are hardcoded in executable code and function parameters
16:08:32 <meskio> ohh, wow
16:09:05 <dcf1> https://github.com/net4people/bbs/issues/131#issuecomment-1272120924
16:09:59 <shelikhoo> Oh! thanks for the workaround in the link
16:12:09 <dcf1> I should say, re the uTLS enabling discussion, that as far as I can tell, the TLS fingerprints of Tor Browser (android and desktop) are not being blocked
16:12:41 <dcf1> They are close to the fp that is being blocked, so it's not something we can depend on long-term, but appears to be true for now
16:12:55 <meskio> yes, let's move to the enxt topic as they are linked...
16:13:03 <dcf1> e.g. https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/96#note_2845302 "Tor Browser for Android + Snowflake works, but Orbot (iOS and Android) with snowflake is not working ... Tor Browser for Android 11.5.4 (without any bridge line changes)."
16:13:40 <dcf1> There's one report saying the custom bridge lines with utls-imitate work: https://github.com/net4people/bbs/issues/131#issuecomment-1284211012 "just tested tor browser on android with ur custom snowflake, and it works, not perfectly or fast, but does connect and brings certain restricted websites."
16:13:49 <meskio> AFAIK you are talking about this discussion: https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/merge_requests/540
16:14:03 <dcf1> yes !540
16:14:18 <meskio> where I see dcf1 advocating to enable uTLS for everybody on snowflake and shelikhoo having some doubts about it
16:15:02 <dcf1> however there's another report that says manually entered utls-imitate bridge lines do not work, but the plain snowflake option with no manual configuration does work: https://github.com/net4people/bbs/issues/125#issuecomment-1285003475 "A friend told me that neither hellochrome_auto nor hellorandomizedalpn was working on his phone. But the snowflake option itself (without custom bridge) was working!"
16:15:14 <dcf1> And we must keep in mind the possibility of user error in any of these reports.
16:15:34 <shelikhoo> Yes. I have stated my point that for a censorship with the ability to customize the censorship tool, utls won't work
16:15:57 <shelikhoo> there are other tools written in golang that have golang fingerprint
16:16:11 <shelikhoo> but only anti-censorship tool uses utls
16:17:49 <shelikhoo> however, I think we can proceed with enable utls by default and see what happens
16:18:06 <shelikhoo> maybe censor don't care to make change to the censor tools
16:18:19 <shelikhoo> (which typically is the case)
16:18:35 <dcf1> I have not heard this concern before; can you elaborate? Are you expecting that utls creates its own identifiable features despite removing identifiable features from client hello?
16:19:29 <dcf1> As far as I know, there's no simple classifier an observer can use to identify the use of utls. I could be wrong; perhaps there are some active attacks or distinctive features later than the handshake; are you thinking of something specific?
16:19:59 <shelikhoo> the client hello sent by utls contain extensions that appear on browser but not on golang's tls implementation
16:20:53 <shelikhoo> for a censor, it may be possible to check whether these extensions are actually implemented
16:21:43 <shelikhoo> but this would require change to the censor's tool, so isn't a immediate concern
16:22:29 <dcf1> Can you give an example? Does the censor have to trick the client into connecting to a server the censor controls, in order to see what happens inside the encrypted record layer?
16:23:05 <dcf1> I can see that it would be easy to test if a *server* were lying about the TLS extensions it supports; it's not so clear it's easy to test a client though.
16:23:06 <meskio> interesting idea, I wonder if that is possible to do without having some TLS certificates to impersonate the domain front
16:23:31 <shelikhoo> sorry, I didn't make specific research about these extensions, I can do that in the future
16:23:41 <shelikhoo> I was just stating an possibility
16:24:06 <meskio> yes, maybe there are extensions that don't can be tested without providing a valid key
16:24:36 <meskio> s/don't//
16:25:05 <meskio> but is a bit more convoluted for the censor, as they need to MitM the connection
16:25:14 <meskio> or as dcf1 says to trick the client to connect to them
16:25:49 <meskio> I'm a bit less worried about this than the block of goTLS fingerprint, as the latter is already happening
16:25:55 <shelikhoo> In the past there are something like https://datatracker.ietf.org/doc/rfc8879/
16:26:18 <shelikhoo> there was some browser that support certificate compression extension
16:26:52 <shelikhoo> and when it is supported by the server, the client is supposed to compress the certificate
16:27:15 <shelikhoo> (or decompress to be specific)
16:27:52 <meskio> will an invalid certificate produce a different notizable error by the server than a client not being able to decompress it?
16:27:55 <meskio> maybe...
16:28:06 <meskio> but I see the point
16:28:08 <shelikhoo> https://gitlab.com/yawning/utls
16:28:20 <shelikhoo> I see this from this fork
16:28:32 <shelikhoo> that try to fix this by implementing this extension
16:28:49 <shelikhoo> but there could be many more extension like this
16:29:01 <meskio> would be nice if the license of yawning allowed us to upstream his changes
16:29:11 <dcf1> It is in recent versions of utls (v1.1.2+) too: https://github.com/refraction-networking/utls/commit/7344e34650c003f29afed2e5228a5f6e662994f1 https://github.com/refraction-networking/utls/pull/95
16:29:24 <meskio> ohh, cool
16:29:28 <shelikhoo> yes, so I can't make a valid example
16:29:34 <shelikhoo> but it is a possibility
16:30:00 <dcf1> (nota bene the version we ship is currently an old version, v1.0.0 https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40217)
16:31:01 <meskio> shelikhoo: do you have a strong opinion against enabling uTLS? I see your concern, but I would personally enable it now and disable it in the future if it becomes a problem
16:31:21 <shelikhoo> no, I don't object it personally
16:31:32 <shelikhoo> but make sure my concern is heard
16:31:50 <meskio> nice, thanks for bringing it, we should pay attention to it
16:32:01 <meskio> so I guess we agree on merging !540
16:32:05 <meskio> anything else on this topic?
16:32:09 <dcf1> thank you shelikhoo
16:32:32 <shelikhoo> no problem
16:32:40 <meskio> snowflake load after rever broker change
16:32:44 <meskio> shelikhoo?
16:32:51 <shelikhoo> yes
16:33:13 <shelikhoo> earlier this week, I have revert the change to the broker
16:33:30 <shelikhoo> do we observe an increased traffic because of that?
16:34:12 <dcf1> I can make a graph of bandwidth at the bridge real quick; I am curious though about polling rates (client and proxy)
16:34:26 * meskio goes to graphana...
16:35:41 <dcf1> as I understand it, reverting the broker change to permit older proxies was a long-shot guess to remedy the problem that I now believe was actually caused by focused blocking in Iran
16:36:06 <meskio> I don't see any clear difference in the failed client requests in the graphs since the change
16:36:22 <shelikhoo> yes, if that is the case, revert is unnecessary
16:36:34 <shelikhoo> let's enable the secondary bridge
16:36:43 <shelikhoo> and distributed snowflake support
16:36:43 <meskio> there is a raise of idle proxy polls since 2 days, so more proxies around, but I don't see the change in the graphs
16:37:05 <shelikhoo> we should be able to find more proxies to replace them.....
16:37:21 <meskio> nice, I'm happy we can move on the distributed snowflake support \o/
16:37:34 <shelikhoo> without auto update, these kind of legacy proxy will always exist
16:37:54 <dcf1> haha no, no change in bridge bandwidth either https://share.riseup.net/#t_CKQ2cMaypFIyv2OfxsCw
16:38:35 <meskio> let's move to the next topic then
16:38:46 <meskio> UDP traffic between Iran and the rest of the inet
16:39:15 <shelikhoo> yes. I will revert the revert, and add the definition of secondary bridge into broker next monday
16:39:35 <meskio> +1
16:39:53 <shelikhoo> yes. i am currently conducting a research on selective block of UDP traffic in Iran
16:40:29 <shelikhoo> It seems it is banning unknown traffic in someway
16:40:49 <shelikhoo> which is another attempt to block looks like nothing proxy
16:41:39 <shelikhoo> I will keep working on this research(and see if it impacts snowflake)
16:41:45 <shelikhoo> over
16:42:30 <meskio> the good thing is that snowflake doesn't seem to be affected by that block
16:42:34 <meskio> isn't it?
16:42:55 <shelikhoo> I have yet to test that...
16:43:03 <meskio> ok
16:43:38 <meskio> anything else on this?
16:43:55 <shelikhoo> nothing from me...
16:44:14 <meskio> obfs4proxy meek utls patches
16:44:41 <meskio> I brought back uTLS into obfs4proxy, as Tor Browser uses it for moat and meek
16:45:07 <meskio> I sent the patch upstream, but as I expect that might not be fast to be merged I'm also getting it reviewed by dcf1
16:45:15 <meskio> https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/obfs4/-/merge_requests/1
16:45:47 <meskio> if everybody is ok with it I'll merge that into our copy of the repo and try to include it in the next TB release
16:45:53 <meskio> due next tuesday
16:46:11 <meskio> so we get the security updats of 0.0.14 included in TB
16:46:25 <meskio> what do you think? am I rushing it too much?
16:46:28 <shelikhoo> no objection from me~
16:47:03 <dcf1> I think it's okay.
16:47:15 <meskio> cool, I'll move that forward
16:47:22 <dcf1> For peace of mind you may want to generate some pcaps and inspect them to see that the fingerprint is really changing.
16:47:47 <meskio> yes, I've being doing that
16:48:06 <dcf1> cool
16:48:13 <meskio> the fp is changing as we are including a newer version of utls with newer firefox fingerprints
16:48:20 <meskio> but that should be good, I hope
16:48:41 <dcf1> There can be compatibility problems with certain fingerprints and certain servers
16:48:44 <meskio> BTW, I'm keeping for now the firefox fingerprint to don't change too many things at once
16:49:01 <dcf1> This is due to what shelikhoo was talking about, the client not actually supporting all the things it claims to support
16:49:15 <dcf1> Example from when I was adding utls to dnstt: https://ntc.party/t/testing-branch-for-utls-support/1593/2
16:49:20 <meskio> dcf1: I've tested it against our meek server and moat, wich are the only two places I expect it to be used to
16:49:33 <dcf1> awesome, I think that is sufficient
16:49:45 <cohosh> in that case, could we alter the configuration of the moat/meek server to fix compatibility issues?
16:49:52 <meskio> we can always change the fingerprint in the bridgeline if we want
16:50:13 <dcf1> cohosh: probably not, since it would depend on the CDN's TLS terminator not us as the origin server
16:50:25 <cohosh> ah right
16:51:44 <meskio> we'll deal with it if it happen, hopefully the CDN's will not change that
16:51:57 <meskio> should we move to 'Testing new PTs'?
16:52:35 <ggus3> yes, i added that
16:53:40 <ggus3> gaba asked me about testing conjure
16:53:59 <cohosh> ah, that needs a bit more work before it's ready
16:54:14 <ggus3> i wonder if we need to point testers to that email that cohosh sent, or should we have a better instructions somewhere. plus where should i report findings
16:54:34 <gaba> it will be ready to test after november in alpha, right?
16:54:37 <cohosh> i have working tor browser integrations, but the conjure station is somewhat unreliable
16:54:54 <cohosh> so i want to make some more client-side improvements to recover from failed connections
16:55:02 <cohosh> ggus3: yes early november sounds right
16:56:06 <ggus3> cohosh: ok, and what kind of testers do we want? folks from a specific region where vanilla tor is blocked?
16:56:28 <ggus3> i thought about russia
16:56:40 <cohosh> honestly at this point any testers, but yeah somewhere with censorship would be good too
16:57:05 <cohosh> it's still in the "just get it working" stage and not quite in the "make it really censorship resistant" phase
16:57:35 <meskio> :)
16:58:01 <shelikhoo> (I was also trying to get webtunnel to build with Tor Browser, but currently postponed by research on iran's censorship)
16:58:35 <meskio> maybe we can get two PTs to test next month :)
16:58:47 <meskio> anything else on this?
16:58:59 <shelikhoo> nothing from me
16:59:00 <ggus3> cohosh: what kind of feedback is needed now?
16:59:02 <cohosh> ggus3: i can follow up with you on this, i got distracted by snowflake development the last few weeks but i really need to get back to getting this working
16:59:23 <ggus3> ok!
16:59:32 <cohosh> ggus3: i'm aware of some pretty annoying issues right now, after i fix those i'll ping you :)
16:59:39 <meskio> too many things happening at once
16:59:47 <ggus3> cohosh: perfect! :)
17:00:09 <meskio> next poing: snowflake in RACE takes too long to transfer messages
17:00:12 <meskio> itchyonion?
17:00:14 <itchyonion> This one is by me. I just discovered it yesterday so I haven't spent too much time on it. The amount of time it takes snowflake to transfer a message in RACE has a huge variance. Sometimes it takes 2 seconds, sometimes it takes longer than 45 seconds. I haven't done any profiling of snowflake, but I'm wondering if it heavily depends on the quality/avialibility of snowflake proxies and is something we have little control of.
17:00:54 <dcf1> Does the time to send a message include initial bootstrapping time, or is it after a connection is already established?
17:01:00 <meskio> for others to understand RACE has their own snowflake setup, they are not using our broker/proxies...
17:01:11 <shelikhoo> I think if it need ICE then it will take a while
17:01:11 <itchyonion> before connection is established
17:01:44 <shelikhoo> the candidate gathering will wait for a time out
17:01:48 <dcf1> In a controlled environment 45 seconds to bootstrap does seem long
17:01:57 <shelikhoo> if there is a STUN server didn't send the reply
17:02:32 <itchyonion> Do we have exsiting metrics or a rough idea of the worst case?
17:02:36 <dcf1> itchyonion: short answer is yes, it can greatly depend on the number and quality of available snowflakes
17:03:02 <cohosh> itchyonion: i don't think metrics from real world snowflake will help here
17:03:05 <dcf1> that's the only source of delay I can think of, several iterations at the beginning to find a proxy that works. There are some hardcoded timeouts.
17:03:20 <itchyonion> hmm OK. There is a 30 seconds timeout set by the RACE test, which is a problem.
17:03:21 <cohosh> since it's so dependent on how many proxies are available
17:04:06 <meskio> does RACE have its own STUN server? could reaching outside STUN servers be problem here?
17:04:38 <dcf1> Some suggestions: make a histogram plot of total time to send a message, see if the times fall into "buckets" or if they are smoothly distributed. Try tuning some of the hardcoded timeouts in the source code.
17:04:43 <cohosh> it's all in its own local environment
17:04:50 <meskio> ack
17:05:05 <cohosh> itchyonion: can you check the logs to see if the broker is saying whether there are proxies available?
17:05:23 <cohosh> if there aren't enough, the broker should say so when the client polls
17:05:24 <itchyonion> ah ok. That's good point. Will do it later today
17:05:48 <dcf1> like if you see spikes at 15s, 30s, 45s, it could mean that something with a timeout of 15s is probabilistically failing
17:06:07 <itchyonion> dcf1 also good point
17:07:52 <meskio> anything else on this?
17:08:00 <itchyonion> that;s all for now
17:08:01 <shelikhoo> nothing from me
17:08:18 <meskio> I leave the last point for next week, as is already late, you can look at the issue if you want
17:08:44 <meskio> #endmeeting