15:58:34 <meskio> #startmeeting tor anti-censorship meeting 15:58:34 <MeetBot> Meeting started Thu Oct 20 15:58:34 2022 UTC. The chair is meskio. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:58:34 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic. 15:58:36 <meskio> hello all 15:58:40 <meskio> here is our meeting pad: https://pad.riseup.net/p/tor-anti-censorship-keep 15:58:42 <shelikhoo> Hi~ 15:58:43 <meskio> here is our meeting pad: https://pad.riseup.net/p/tor-anti-censorship-keep 15:58:50 <meskio> feel free to add what you've been working on and put items on the agenda 15:59:28 <itchyonion> hello 15:59:54 <meskio> our first point in the agenda is 'blocking by TLS fingerprint in Iran' 16:00:10 <meskio> mmm, I see someone still writting there, let's wait for a bit 16:00:15 <dcf1> I'm writing a summary quick 16:00:21 <meskio> thanks :) 16:01:45 <ggus> hello 16:01:57 <dcf1> I did a lot of investigation into blocking in Iran in the past week 16:02:21 <itchyonion> hello 16:02:37 <dcf1> What started a minor breakthrough was looking at a graph of directory requests across multiple countries, with a few days of context before and after Oct 04 16:02:41 <dcf1> https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40207#note_2844116 16:03:24 <dcf1> my conclusion is that the loss of traffic on Oct 04 *is* due to a block in Iran, that apparent effects in other countries are geoip errors, and that the mechanism of blocking is TLS fingerprinting using at least two identified features: ciphersuite order and minimum supported TLS version 16:04:07 <dcf1> I looked at the available blocking reports, and identified two TLS fingerprint features that have to do with blocking 16:04:35 <dcf1> namely, whether the platform has AES acceleration (affects ciphersuite order) and whether it was compiled with go1.18 or later (affects minimum TLS version) 16:04:58 <dcf1> Of the 4 quadrants in the matrix, only 1 is blocked, which happens to be the fingerprint of released versions of Orbot 16:05:24 <dcf1> As far as I can tell, Tor Browser on desktop and Tor Browser for Android are working in Iran and have been 16:05:59 <dcf1> (btw Orbot is preparing a new release with utls enabled that may fix this: https://github.com/guardianproject/orbot/releases/tag/16.6.3-BETA-2-tor.0.4.7.10) 16:06:27 <dcf1> That's my summary of the TLS fingerprint blocking situation, which leads to the next discussion point 16:06:54 <meskio> nice that orbot is reacting to that 16:06:57 <cohosh> wow nice work dcf1 16:07:06 <meskio> would be nice to have orbot using circumvention settings 16:07:20 <meskio> but that needs some work 16:07:35 <dcf1> it's a bit tricky because orbot (at least currently) does not use bridge lines the same way 16:08:02 <dcf1> custom bridges can only be obfs4; the snowflake configurations are hardcoded in executable code and function parameters 16:08:32 <meskio> ohh, wow 16:09:05 <dcf1> https://github.com/net4people/bbs/issues/131#issuecomment-1272120924 16:09:59 <shelikhoo> Oh! thanks for the workaround in the link 16:12:09 <dcf1> I should say, re the uTLS enabling discussion, that as far as I can tell, the TLS fingerprints of Tor Browser (android and desktop) are not being blocked 16:12:41 <dcf1> They are close to the fp that is being blocked, so it's not something we can depend on long-term, but appears to be true for now 16:12:55 <meskio> yes, let's move to the enxt topic as they are linked... 16:13:03 <dcf1> e.g. https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/96#note_2845302 "Tor Browser for Android + Snowflake works, but Orbot (iOS and Android) with snowflake is not working ... Tor Browser for Android 11.5.4 (without any bridge line changes)." 16:13:40 <dcf1> There's one report saying the custom bridge lines with utls-imitate work: https://github.com/net4people/bbs/issues/131#issuecomment-1284211012 "just tested tor browser on android with ur custom snowflake, and it works, not perfectly or fast, but does connect and brings certain restricted websites." 16:13:49 <meskio> AFAIK you are talking about this discussion: https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/merge_requests/540 16:14:03 <dcf1> yes !540 16:14:18 <meskio> where I see dcf1 advocating to enable uTLS for everybody on snowflake and shelikhoo having some doubts about it 16:15:02 <dcf1> however there's another report that says manually entered utls-imitate bridge lines do not work, but the plain snowflake option with no manual configuration does work: https://github.com/net4people/bbs/issues/125#issuecomment-1285003475 "A friend told me that neither hellochrome_auto nor hellorandomizedalpn was working on his phone. But the snowflake option itself (without custom bridge) was working!" 16:15:14 <dcf1> And we must keep in mind the possibility of user error in any of these reports. 16:15:34 <shelikhoo> Yes. I have stated my point that for a censorship with the ability to customize the censorship tool, utls won't work 16:15:57 <shelikhoo> there are other tools written in golang that have golang fingerprint 16:16:11 <shelikhoo> but only anti-censorship tool uses utls 16:17:49 <shelikhoo> however, I think we can proceed with enable utls by default and see what happens 16:18:06 <shelikhoo> maybe censor don't care to make change to the censor tools 16:18:19 <shelikhoo> (which typically is the case) 16:18:35 <dcf1> I have not heard this concern before; can you elaborate? Are you expecting that utls creates its own identifiable features despite removing identifiable features from client hello? 16:19:29 <dcf1> As far as I know, there's no simple classifier an observer can use to identify the use of utls. I could be wrong; perhaps there are some active attacks or distinctive features later than the handshake; are you thinking of something specific? 16:19:59 <shelikhoo> the client hello sent by utls contain extensions that appear on browser but not on golang's tls implementation 16:20:53 <shelikhoo> for a censor, it may be possible to check whether these extensions are actually implemented 16:21:43 <shelikhoo> but this would require change to the censor's tool, so isn't a immediate concern 16:22:29 <dcf1> Can you give an example? Does the censor have to trick the client into connecting to a server the censor controls, in order to see what happens inside the encrypted record layer? 16:23:05 <dcf1> I can see that it would be easy to test if a *server* were lying about the TLS extensions it supports; it's not so clear it's easy to test a client though. 16:23:06 <meskio> interesting idea, I wonder if that is possible to do without having some TLS certificates to impersonate the domain front 16:23:31 <shelikhoo> sorry, I didn't make specific research about these extensions, I can do that in the future 16:23:41 <shelikhoo> I was just stating an possibility 16:24:06 <meskio> yes, maybe there are extensions that don't can be tested without providing a valid key 16:24:36 <meskio> s/don't// 16:25:05 <meskio> but is a bit more convoluted for the censor, as they need to MitM the connection 16:25:14 <meskio> or as dcf1 says to trick the client to connect to them 16:25:49 <meskio> I'm a bit less worried about this than the block of goTLS fingerprint, as the latter is already happening 16:25:55 <shelikhoo> In the past there are something like https://datatracker.ietf.org/doc/rfc8879/ 16:26:18 <shelikhoo> there was some browser that support certificate compression extension 16:26:52 <shelikhoo> and when it is supported by the server, the client is supposed to compress the certificate 16:27:15 <shelikhoo> (or decompress to be specific) 16:27:52 <meskio> will an invalid certificate produce a different notizable error by the server than a client not being able to decompress it? 16:27:55 <meskio> maybe... 16:28:06 <meskio> but I see the point 16:28:08 <shelikhoo> https://gitlab.com/yawning/utls 16:28:20 <shelikhoo> I see this from this fork 16:28:32 <shelikhoo> that try to fix this by implementing this extension 16:28:49 <shelikhoo> but there could be many more extension like this 16:29:01 <meskio> would be nice if the license of yawning allowed us to upstream his changes 16:29:11 <dcf1> It is in recent versions of utls (v1.1.2+) too: https://github.com/refraction-networking/utls/commit/7344e34650c003f29afed2e5228a5f6e662994f1 https://github.com/refraction-networking/utls/pull/95 16:29:24 <meskio> ohh, cool 16:29:28 <shelikhoo> yes, so I can't make a valid example 16:29:34 <shelikhoo> but it is a possibility 16:30:00 <dcf1> (nota bene the version we ship is currently an old version, v1.0.0 https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40217) 16:31:01 <meskio> shelikhoo: do you have a strong opinion against enabling uTLS? I see your concern, but I would personally enable it now and disable it in the future if it becomes a problem 16:31:21 <shelikhoo> no, I don't object it personally 16:31:32 <shelikhoo> but make sure my concern is heard 16:31:50 <meskio> nice, thanks for bringing it, we should pay attention to it 16:32:01 <meskio> so I guess we agree on merging !540 16:32:05 <meskio> anything else on this topic? 16:32:09 <dcf1> thank you shelikhoo 16:32:32 <shelikhoo> no problem 16:32:40 <meskio> snowflake load after rever broker change 16:32:44 <meskio> shelikhoo? 16:32:51 <shelikhoo> yes 16:33:13 <shelikhoo> earlier this week, I have revert the change to the broker 16:33:30 <shelikhoo> do we observe an increased traffic because of that? 16:34:12 <dcf1> I can make a graph of bandwidth at the bridge real quick; I am curious though about polling rates (client and proxy) 16:34:26 * meskio goes to graphana... 16:35:41 <dcf1> as I understand it, reverting the broker change to permit older proxies was a long-shot guess to remedy the problem that I now believe was actually caused by focused blocking in Iran 16:36:06 <meskio> I don't see any clear difference in the failed client requests in the graphs since the change 16:36:22 <shelikhoo> yes, if that is the case, revert is unnecessary 16:36:34 <shelikhoo> let's enable the secondary bridge 16:36:43 <shelikhoo> and distributed snowflake support 16:36:43 <meskio> there is a raise of idle proxy polls since 2 days, so more proxies around, but I don't see the change in the graphs 16:37:05 <shelikhoo> we should be able to find more proxies to replace them..... 16:37:21 <meskio> nice, I'm happy we can move on the distributed snowflake support \o/ 16:37:34 <shelikhoo> without auto update, these kind of legacy proxy will always exist 16:37:54 <dcf1> haha no, no change in bridge bandwidth either https://share.riseup.net/#t_CKQ2cMaypFIyv2OfxsCw 16:38:35 <meskio> let's move to the next topic then 16:38:46 <meskio> UDP traffic between Iran and the rest of the inet 16:39:15 <shelikhoo> yes. I will revert the revert, and add the definition of secondary bridge into broker next monday 16:39:35 <meskio> +1 16:39:53 <shelikhoo> yes. i am currently conducting a research on selective block of UDP traffic in Iran 16:40:29 <shelikhoo> It seems it is banning unknown traffic in someway 16:40:49 <shelikhoo> which is another attempt to block looks like nothing proxy 16:41:39 <shelikhoo> I will keep working on this research(and see if it impacts snowflake) 16:41:45 <shelikhoo> over 16:42:30 <meskio> the good thing is that snowflake doesn't seem to be affected by that block 16:42:34 <meskio> isn't it? 16:42:55 <shelikhoo> I have yet to test that... 16:43:03 <meskio> ok 16:43:38 <meskio> anything else on this? 16:43:55 <shelikhoo> nothing from me... 16:44:14 <meskio> obfs4proxy meek utls patches 16:44:41 <meskio> I brought back uTLS into obfs4proxy, as Tor Browser uses it for moat and meek 16:45:07 <meskio> I sent the patch upstream, but as I expect that might not be fast to be merged I'm also getting it reviewed by dcf1 16:45:15 <meskio> https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/obfs4/-/merge_requests/1 16:45:47 <meskio> if everybody is ok with it I'll merge that into our copy of the repo and try to include it in the next TB release 16:45:53 <meskio> due next tuesday 16:46:11 <meskio> so we get the security updats of 0.0.14 included in TB 16:46:25 <meskio> what do you think? am I rushing it too much? 16:46:28 <shelikhoo> no objection from me~ 16:47:03 <dcf1> I think it's okay. 16:47:15 <meskio> cool, I'll move that forward 16:47:22 <dcf1> For peace of mind you may want to generate some pcaps and inspect them to see that the fingerprint is really changing. 16:47:47 <meskio> yes, I've being doing that 16:48:06 <dcf1> cool 16:48:13 <meskio> the fp is changing as we are including a newer version of utls with newer firefox fingerprints 16:48:20 <meskio> but that should be good, I hope 16:48:41 <dcf1> There can be compatibility problems with certain fingerprints and certain servers 16:48:44 <meskio> BTW, I'm keeping for now the firefox fingerprint to don't change too many things at once 16:49:01 <dcf1> This is due to what shelikhoo was talking about, the client not actually supporting all the things it claims to support 16:49:15 <dcf1> Example from when I was adding utls to dnstt: https://ntc.party/t/testing-branch-for-utls-support/1593/2 16:49:20 <meskio> dcf1: I've tested it against our meek server and moat, wich are the only two places I expect it to be used to 16:49:33 <dcf1> awesome, I think that is sufficient 16:49:45 <cohosh> in that case, could we alter the configuration of the moat/meek server to fix compatibility issues? 16:49:52 <meskio> we can always change the fingerprint in the bridgeline if we want 16:50:13 <dcf1> cohosh: probably not, since it would depend on the CDN's TLS terminator not us as the origin server 16:50:25 <cohosh> ah right 16:51:44 <meskio> we'll deal with it if it happen, hopefully the CDN's will not change that 16:51:57 <meskio> should we move to 'Testing new PTs'? 16:52:35 <ggus3> yes, i added that 16:53:40 <ggus3> gaba asked me about testing conjure 16:53:59 <cohosh> ah, that needs a bit more work before it's ready 16:54:14 <ggus3> i wonder if we need to point testers to that email that cohosh sent, or should we have a better instructions somewhere. plus where should i report findings 16:54:34 <gaba> it will be ready to test after november in alpha, right? 16:54:37 <cohosh> i have working tor browser integrations, but the conjure station is somewhat unreliable 16:54:54 <cohosh> so i want to make some more client-side improvements to recover from failed connections 16:55:02 <cohosh> ggus3: yes early november sounds right 16:56:06 <ggus3> cohosh: ok, and what kind of testers do we want? folks from a specific region where vanilla tor is blocked? 16:56:28 <ggus3> i thought about russia 16:56:40 <cohosh> honestly at this point any testers, but yeah somewhere with censorship would be good too 16:57:05 <cohosh> it's still in the "just get it working" stage and not quite in the "make it really censorship resistant" phase 16:57:35 <meskio> :) 16:58:01 <shelikhoo> (I was also trying to get webtunnel to build with Tor Browser, but currently postponed by research on iran's censorship) 16:58:35 <meskio> maybe we can get two PTs to test next month :) 16:58:47 <meskio> anything else on this? 16:58:59 <shelikhoo> nothing from me 16:59:00 <ggus3> cohosh: what kind of feedback is needed now? 16:59:02 <cohosh> ggus3: i can follow up with you on this, i got distracted by snowflake development the last few weeks but i really need to get back to getting this working 16:59:23 <ggus3> ok! 16:59:32 <cohosh> ggus3: i'm aware of some pretty annoying issues right now, after i fix those i'll ping you :) 16:59:39 <meskio> too many things happening at once 16:59:47 <ggus3> cohosh: perfect! :) 17:00:09 <meskio> next poing: snowflake in RACE takes too long to transfer messages 17:00:12 <meskio> itchyonion? 17:00:14 <itchyonion> This one is by me. I just discovered it yesterday so I haven't spent too much time on it. The amount of time it takes snowflake to transfer a message in RACE has a huge variance. Sometimes it takes 2 seconds, sometimes it takes longer than 45 seconds. I haven't done any profiling of snowflake, but I'm wondering if it heavily depends on the quality/avialibility of snowflake proxies and is something we have little control of. 17:00:54 <dcf1> Does the time to send a message include initial bootstrapping time, or is it after a connection is already established? 17:01:00 <meskio> for others to understand RACE has their own snowflake setup, they are not using our broker/proxies... 17:01:11 <shelikhoo> I think if it need ICE then it will take a while 17:01:11 <itchyonion> before connection is established 17:01:44 <shelikhoo> the candidate gathering will wait for a time out 17:01:48 <dcf1> In a controlled environment 45 seconds to bootstrap does seem long 17:01:57 <shelikhoo> if there is a STUN server didn't send the reply 17:02:32 <itchyonion> Do we have exsiting metrics or a rough idea of the worst case? 17:02:36 <dcf1> itchyonion: short answer is yes, it can greatly depend on the number and quality of available snowflakes 17:03:02 <cohosh> itchyonion: i don't think metrics from real world snowflake will help here 17:03:05 <dcf1> that's the only source of delay I can think of, several iterations at the beginning to find a proxy that works. There are some hardcoded timeouts. 17:03:20 <itchyonion> hmm OK. There is a 30 seconds timeout set by the RACE test, which is a problem. 17:03:21 <cohosh> since it's so dependent on how many proxies are available 17:04:06 <meskio> does RACE have its own STUN server? could reaching outside STUN servers be problem here? 17:04:38 <dcf1> Some suggestions: make a histogram plot of total time to send a message, see if the times fall into "buckets" or if they are smoothly distributed. Try tuning some of the hardcoded timeouts in the source code. 17:04:43 <cohosh> it's all in its own local environment 17:04:50 <meskio> ack 17:05:05 <cohosh> itchyonion: can you check the logs to see if the broker is saying whether there are proxies available? 17:05:23 <cohosh> if there aren't enough, the broker should say so when the client polls 17:05:24 <itchyonion> ah ok. That's good point. Will do it later today 17:05:48 <dcf1> like if you see spikes at 15s, 30s, 45s, it could mean that something with a timeout of 15s is probabilistically failing 17:06:07 <itchyonion> dcf1 also good point 17:07:52 <meskio> anything else on this? 17:08:00 <itchyonion> that;s all for now 17:08:01 <shelikhoo> nothing from me 17:08:18 <meskio> I leave the last point for next week, as is already late, you can look at the issue if you want 17:08:44 <meskio> #endmeeting