15:58:34 #startmeeting tor anti-censorship meeting 15:58:34 Meeting started Thu Oct 20 15:58:34 2022 UTC. The chair is meskio. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:58:34 Useful Commands: #action #agreed #help #info #idea #link #topic. 15:58:36 hello all 15:58:40 here is our meeting pad: https://pad.riseup.net/p/tor-anti-censorship-keep 15:58:42 Hi~ 15:58:43 here is our meeting pad: https://pad.riseup.net/p/tor-anti-censorship-keep 15:58:50 feel free to add what you've been working on and put items on the agenda 15:59:28 hello 15:59:54 our first point in the agenda is 'blocking by TLS fingerprint in Iran' 16:00:10 mmm, I see someone still writting there, let's wait for a bit 16:00:15 I'm writing a summary quick 16:00:21 thanks :) 16:01:45 hello 16:01:57 I did a lot of investigation into blocking in Iran in the past week 16:02:21 hello 16:02:37 What started a minor breakthrough was looking at a graph of directory requests across multiple countries, with a few days of context before and after Oct 04 16:02:41 https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40207#note_2844116 16:03:24 my conclusion is that the loss of traffic on Oct 04 *is* due to a block in Iran, that apparent effects in other countries are geoip errors, and that the mechanism of blocking is TLS fingerprinting using at least two identified features: ciphersuite order and minimum supported TLS version 16:04:07 I looked at the available blocking reports, and identified two TLS fingerprint features that have to do with blocking 16:04:35 namely, whether the platform has AES acceleration (affects ciphersuite order) and whether it was compiled with go1.18 or later (affects minimum TLS version) 16:04:58 Of the 4 quadrants in the matrix, only 1 is blocked, which happens to be the fingerprint of released versions of Orbot 16:05:24 As far as I can tell, Tor Browser on desktop and Tor Browser for Android are working in Iran and have been 16:05:59 (btw Orbot is preparing a new release with utls enabled that may fix this: https://github.com/guardianproject/orbot/releases/tag/16.6.3-BETA-2-tor.0.4.7.10) 16:06:27 That's my summary of the TLS fingerprint blocking situation, which leads to the next discussion point 16:06:54 nice that orbot is reacting to that 16:06:57 wow nice work dcf1 16:07:06 would be nice to have orbot using circumvention settings 16:07:20 but that needs some work 16:07:35 it's a bit tricky because orbot (at least currently) does not use bridge lines the same way 16:08:02 custom bridges can only be obfs4; the snowflake configurations are hardcoded in executable code and function parameters 16:08:32 ohh, wow 16:09:05 https://github.com/net4people/bbs/issues/131#issuecomment-1272120924 16:09:59 Oh! thanks for the workaround in the link 16:12:09 I should say, re the uTLS enabling discussion, that as far as I can tell, the TLS fingerprints of Tor Browser (android and desktop) are not being blocked 16:12:41 They are close to the fp that is being blocked, so it's not something we can depend on long-term, but appears to be true for now 16:12:55 yes, let's move to the enxt topic as they are linked... 16:13:03 e.g. https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/96#note_2845302 "Tor Browser for Android + Snowflake works, but Orbot (iOS and Android) with snowflake is not working ... Tor Browser for Android 11.5.4 (without any bridge line changes)." 16:13:40 There's one report saying the custom bridge lines with utls-imitate work: https://github.com/net4people/bbs/issues/131#issuecomment-1284211012 "just tested tor browser on android with ur custom snowflake, and it works, not perfectly or fast, but does connect and brings certain restricted websites." 16:13:49 AFAIK you are talking about this discussion: https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/merge_requests/540 16:14:03 yes !540 16:14:18 where I see dcf1 advocating to enable uTLS for everybody on snowflake and shelikhoo having some doubts about it 16:15:02 however there's another report that says manually entered utls-imitate bridge lines do not work, but the plain snowflake option with no manual configuration does work: https://github.com/net4people/bbs/issues/125#issuecomment-1285003475 "A friend told me that neither hellochrome_auto nor hellorandomizedalpn was working on his phone. But the snowflake option itself (without custom bridge) was working!" 16:15:14 And we must keep in mind the possibility of user error in any of these reports. 16:15:34 Yes. I have stated my point that for a censorship with the ability to customize the censorship tool, utls won't work 16:15:57 there are other tools written in golang that have golang fingerprint 16:16:11 but only anti-censorship tool uses utls 16:17:49 however, I think we can proceed with enable utls by default and see what happens 16:18:06 maybe censor don't care to make change to the censor tools 16:18:19 (which typically is the case) 16:18:35 I have not heard this concern before; can you elaborate? Are you expecting that utls creates its own identifiable features despite removing identifiable features from client hello? 16:19:29 As far as I know, there's no simple classifier an observer can use to identify the use of utls. I could be wrong; perhaps there are some active attacks or distinctive features later than the handshake; are you thinking of something specific? 16:19:59 the client hello sent by utls contain extensions that appear on browser but not on golang's tls implementation 16:20:53 for a censor, it may be possible to check whether these extensions are actually implemented 16:21:43 but this would require change to the censor's tool, so isn't a immediate concern 16:22:29 Can you give an example? Does the censor have to trick the client into connecting to a server the censor controls, in order to see what happens inside the encrypted record layer? 16:23:05 I can see that it would be easy to test if a *server* were lying about the TLS extensions it supports; it's not so clear it's easy to test a client though. 16:23:06 interesting idea, I wonder if that is possible to do without having some TLS certificates to impersonate the domain front 16:23:31 sorry, I didn't make specific research about these extensions, I can do that in the future 16:23:41 I was just stating an possibility 16:24:06 yes, maybe there are extensions that don't can be tested without providing a valid key 16:24:36 s/don't// 16:25:05 but is a bit more convoluted for the censor, as they need to MitM the connection 16:25:14 or as dcf1 says to trick the client to connect to them 16:25:49 I'm a bit less worried about this than the block of goTLS fingerprint, as the latter is already happening 16:25:55 In the past there are something like https://datatracker.ietf.org/doc/rfc8879/ 16:26:18 there was some browser that support certificate compression extension 16:26:52 and when it is supported by the server, the client is supposed to compress the certificate 16:27:15 (or decompress to be specific) 16:27:52 will an invalid certificate produce a different notizable error by the server than a client not being able to decompress it? 16:27:55 maybe... 16:28:06 but I see the point 16:28:08 https://gitlab.com/yawning/utls 16:28:20 I see this from this fork 16:28:32 that try to fix this by implementing this extension 16:28:49 but there could be many more extension like this 16:29:01 would be nice if the license of yawning allowed us to upstream his changes 16:29:11 It is in recent versions of utls (v1.1.2+) too: https://github.com/refraction-networking/utls/commit/7344e34650c003f29afed2e5228a5f6e662994f1 https://github.com/refraction-networking/utls/pull/95 16:29:24 ohh, cool 16:29:28 yes, so I can't make a valid example 16:29:34 but it is a possibility 16:30:00 (nota bene the version we ship is currently an old version, v1.0.0 https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40217) 16:31:01 shelikhoo: do you have a strong opinion against enabling uTLS? I see your concern, but I would personally enable it now and disable it in the future if it becomes a problem 16:31:21 no, I don't object it personally 16:31:32 but make sure my concern is heard 16:31:50 nice, thanks for bringing it, we should pay attention to it 16:32:01 so I guess we agree on merging !540 16:32:05 anything else on this topic? 16:32:09 thank you shelikhoo 16:32:32 no problem 16:32:40 snowflake load after rever broker change 16:32:44 shelikhoo? 16:32:51 yes 16:33:13 earlier this week, I have revert the change to the broker 16:33:30 do we observe an increased traffic because of that? 16:34:12 I can make a graph of bandwidth at the bridge real quick; I am curious though about polling rates (client and proxy) 16:34:26 * meskio goes to graphana... 16:35:41 as I understand it, reverting the broker change to permit older proxies was a long-shot guess to remedy the problem that I now believe was actually caused by focused blocking in Iran 16:36:06 I don't see any clear difference in the failed client requests in the graphs since the change 16:36:22 yes, if that is the case, revert is unnecessary 16:36:34 let's enable the secondary bridge 16:36:43 and distributed snowflake support 16:36:43 there is a raise of idle proxy polls since 2 days, so more proxies around, but I don't see the change in the graphs 16:37:05 we should be able to find more proxies to replace them..... 16:37:21 nice, I'm happy we can move on the distributed snowflake support \o/ 16:37:34 without auto update, these kind of legacy proxy will always exist 16:37:54 haha no, no change in bridge bandwidth either https://share.riseup.net/#t_CKQ2cMaypFIyv2OfxsCw 16:38:35 let's move to the next topic then 16:38:46 UDP traffic between Iran and the rest of the inet 16:39:15 yes. I will revert the revert, and add the definition of secondary bridge into broker next monday 16:39:35 +1 16:39:53 yes. i am currently conducting a research on selective block of UDP traffic in Iran 16:40:29 It seems it is banning unknown traffic in someway 16:40:49 which is another attempt to block looks like nothing proxy 16:41:39 I will keep working on this research(and see if it impacts snowflake) 16:41:45 over 16:42:30 the good thing is that snowflake doesn't seem to be affected by that block 16:42:34 isn't it? 16:42:55 I have yet to test that... 16:43:03 ok 16:43:38 anything else on this? 16:43:55 nothing from me... 16:44:14 obfs4proxy meek utls patches 16:44:41 I brought back uTLS into obfs4proxy, as Tor Browser uses it for moat and meek 16:45:07 I sent the patch upstream, but as I expect that might not be fast to be merged I'm also getting it reviewed by dcf1 16:45:15 https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/obfs4/-/merge_requests/1 16:45:47 if everybody is ok with it I'll merge that into our copy of the repo and try to include it in the next TB release 16:45:53 due next tuesday 16:46:11 so we get the security updats of 0.0.14 included in TB 16:46:25 what do you think? am I rushing it too much? 16:46:28 no objection from me~ 16:47:03 I think it's okay. 16:47:15 cool, I'll move that forward 16:47:22 For peace of mind you may want to generate some pcaps and inspect them to see that the fingerprint is really changing. 16:47:47 yes, I've being doing that 16:48:06 cool 16:48:13 the fp is changing as we are including a newer version of utls with newer firefox fingerprints 16:48:20 but that should be good, I hope 16:48:41 There can be compatibility problems with certain fingerprints and certain servers 16:48:44 BTW, I'm keeping for now the firefox fingerprint to don't change too many things at once 16:49:01 This is due to what shelikhoo was talking about, the client not actually supporting all the things it claims to support 16:49:15 Example from when I was adding utls to dnstt: https://ntc.party/t/testing-branch-for-utls-support/1593/2 16:49:20 dcf1: I've tested it against our meek server and moat, wich are the only two places I expect it to be used to 16:49:33 awesome, I think that is sufficient 16:49:45 in that case, could we alter the configuration of the moat/meek server to fix compatibility issues? 16:49:52 we can always change the fingerprint in the bridgeline if we want 16:50:13 cohosh: probably not, since it would depend on the CDN's TLS terminator not us as the origin server 16:50:25 ah right 16:51:44 we'll deal with it if it happen, hopefully the CDN's will not change that 16:51:57 should we move to 'Testing new PTs'? 16:52:35 yes, i added that 16:53:40 gaba asked me about testing conjure 16:53:59 ah, that needs a bit more work before it's ready 16:54:14 i wonder if we need to point testers to that email that cohosh sent, or should we have a better instructions somewhere. plus where should i report findings 16:54:34 it will be ready to test after november in alpha, right? 16:54:37 i have working tor browser integrations, but the conjure station is somewhat unreliable 16:54:54 so i want to make some more client-side improvements to recover from failed connections 16:55:02 ggus3: yes early november sounds right 16:56:06 cohosh: ok, and what kind of testers do we want? folks from a specific region where vanilla tor is blocked? 16:56:28 i thought about russia 16:56:40 honestly at this point any testers, but yeah somewhere with censorship would be good too 16:57:05 it's still in the "just get it working" stage and not quite in the "make it really censorship resistant" phase 16:57:35 :) 16:58:01 (I was also trying to get webtunnel to build with Tor Browser, but currently postponed by research on iran's censorship) 16:58:35 maybe we can get two PTs to test next month :) 16:58:47 anything else on this? 16:58:59 nothing from me 16:59:00 cohosh: what kind of feedback is needed now? 16:59:02 ggus3: i can follow up with you on this, i got distracted by snowflake development the last few weeks but i really need to get back to getting this working 16:59:23 ok! 16:59:32 ggus3: i'm aware of some pretty annoying issues right now, after i fix those i'll ping you :) 16:59:39 too many things happening at once 16:59:47 cohosh: perfect! :) 17:00:09 next poing: snowflake in RACE takes too long to transfer messages 17:00:12 itchyonion? 17:00:14 This one is by me. I just discovered it yesterday so I haven't spent too much time on it. The amount of time it takes snowflake to transfer a message in RACE has a huge variance. Sometimes it takes 2 seconds, sometimes it takes longer than 45 seconds. I haven't done any profiling of snowflake, but I'm wondering if it heavily depends on the quality/avialibility of snowflake proxies and is something we have little control of. 17:00:54 Does the time to send a message include initial bootstrapping time, or is it after a connection is already established? 17:01:00 for others to understand RACE has their own snowflake setup, they are not using our broker/proxies... 17:01:11 I think if it need ICE then it will take a while 17:01:11 before connection is established 17:01:44 the candidate gathering will wait for a time out 17:01:48 In a controlled environment 45 seconds to bootstrap does seem long 17:01:57 if there is a STUN server didn't send the reply 17:02:32 Do we have exsiting metrics or a rough idea of the worst case? 17:02:36 itchyonion: short answer is yes, it can greatly depend on the number and quality of available snowflakes 17:03:02 itchyonion: i don't think metrics from real world snowflake will help here 17:03:05 that's the only source of delay I can think of, several iterations at the beginning to find a proxy that works. There are some hardcoded timeouts. 17:03:20 hmm OK. There is a 30 seconds timeout set by the RACE test, which is a problem. 17:03:21 since it's so dependent on how many proxies are available 17:04:06 does RACE have its own STUN server? could reaching outside STUN servers be problem here? 17:04:38 Some suggestions: make a histogram plot of total time to send a message, see if the times fall into "buckets" or if they are smoothly distributed. Try tuning some of the hardcoded timeouts in the source code. 17:04:43 it's all in its own local environment 17:04:50 ack 17:05:05 itchyonion: can you check the logs to see if the broker is saying whether there are proxies available? 17:05:23 if there aren't enough, the broker should say so when the client polls 17:05:24 ah ok. That's good point. Will do it later today 17:05:48 like if you see spikes at 15s, 30s, 45s, it could mean that something with a timeout of 15s is probabilistically failing 17:06:07 dcf1 also good point 17:07:52 anything else on this? 17:08:00 that;s all for now 17:08:01 nothing from me 17:08:18 I leave the last point for next week, as is already late, you can look at the issue if you want 17:08:44 #endmeeting