16:01:22 #startmeeting anti-censorship team meeting 16:01:22 Meeting started Thu Oct 29 16:01:22 2026 UTC. The chair is cohosh. Information about MeetBot at https://wiki.debian.org/MeetBot. 16:01:22 Useful Commands: #action #agreed #help #info #idea #link #topic. 16:02:04 here is our meeting pad: https://pad.riseup.net/p/r.9574e996bb9c0266213d38b91b56c469 16:02:12 please feel free to add to the agenda 16:02:25 let us know if you need a link to edit it 16:03:56 the first discussion item is about cdn77 blocking in russia 16:04:44 is the reporter around? 16:04:56 hello 16:04:58 yes 16:05:11 I wrote a note on the meeting pad 16:05:47 thanks! 16:06:36 so when you bootstrap snowflake, you can get a blocked cdn77 node, which won't let you bootstrap 16:06:58 if the cdn77 domain fronting channel is blocked, one thing we can do is recommend using ampcache to users in russia 16:07:08 is it a DNS round robin thing? You resolve the same DNS name, and it's chance what IP address it resolves to? 16:07:20 hi 16:07:45 hi 16:08:06 dcf1: do you mean the cdn, or the dpi? 16:08:25 The cdn 16:08:50 If I understand you, 1 cdn77 edge server is blocked, but others are not blocked 16:08:57 yes, default snowflake bridges contain a cdn77 url, it redirects you to some cdn for further connection 16:09:17 no, one specific one, in my experience 16:09:20 redirects? do you mean an HTTP redirect (e.g. status 301)? 16:09:22 is blocked 16:09:27 is rob tday btw 16:09:28 hi, sorry I was in another meeting and forgot we had this one right after >.< 16:10:11 "one specific one, in my experience" that sounds like it matches my understanding ("1 cdn77 edge server is blocked, but others are not blocked"), but you also said "no", so I am confused. 16:10:18 so you start bootstrapping to snowflake, and then cdn77 sends you to some cdn edge node, as I understand it, for further connection 16:10:42 When you say "one specific node", are you talking about one specific IP address? Or is there something else that distinguishes different nodes? 16:11:04 I noticed one node/IP address that is blocked 16:11:31 "then cdn77 sends you to some cdn edge node" that is not how it works, unless cdn77 is very different from other CDNs 16:11:57 the cdn77 edge server probably routes the traffic internally, but that's *after* you successfully communicate with the edge server 16:12:24 pryty26[mds]: FYI: Just stopping by to quickly note that I am unable to attend today - I am running off literally now to meet with a potential sponsor. But I am planning to join next week. 16:12:37 I think the most plausible explanation is that the cdn77 domain name in the URL may resolve to several IP addresses, and it's random chance which one you get assigned on any given attempt 16:13:05 I'm just checking whether that plausible explanation fits with what you have observed, or if the cause may be something less expected 16:13:29 that's weird, because the cdn77 url in tor's default snowflake bridge doesn't contain the specific edge node IP 16:13:30 unic3rn: okay that is interesting 16:13:50 *as one of it's IP addresses in it's hostname 16:14:24 it is possible that there is some kind of HTTP-level redirect or something, I don't necessarily doubt you, it's just not the most likely thing to happen 16:14:52 It's also possible that the edge server is located inside Russia, and the connection gets blocked between the edge server and the Russian network border 16:15:37 that, specifically, is not the case... 16:15:40 url=https://1098762253.rsc.cdn77.org/ fronts=www.cdn77.com,www.phpmyadmin.net 16:16:16 it's also possible that it happens only when one of the possible front domains is randomly chosen (this random selection happens on the client side) 16:16:35 robgjansen: Okay thx 16:16:42 for clarification 16:16:57 Would you be free for talk in DM at tmr or latter? 16:17:01 *later 16:17:24 so you mean: when a domain fronting situation is happening? (fake SNI, same cdn ip) 16:17:51 it's always using domain fronting, but the specific SNI that's chosen is random on each attempt 16:18:17 It could be that it's one of the IP addresses of www.phpmyadmin.net, not of www.cdn77.com. 16:18:56 "the cdn77 url in tor's default snowflake bridge doesn't contain the specific edge node IP" the IP address depends on the front domain, since that is what is looked up in DNS 16:19:12 well, the SNI is not the problem, I think... the problem is that the given edge node IP is getting throttled. 16:19:29 cohosh: May I ask if you could look into the dm 16:19:31 OuO 16:20:08 pryty26[mds]: there is a meeting in progress, we have an agenda, please be considerate 16:20:35 unic3rn: thank you for this information. maybe you can run a test for us. 16:20:44 Keke, Like I'm asking the meeting pad 16:20:51 in my understanding, you can be given any cdn77 node, but you previously mentioned it otherwise? 16:21:02 Try editing the bridge line so that only one of the two possible front domains is used 16:21:08 so try connecting with 16:21:12 fronts=www.cdn77.com 16:21:17 and then try connecting with 16:21:23 fronts=www.phpmyadmin.net 16:21:32 unic3rn: your note says that this blocking prevents snowflake from bootstrapping, does the client not attempt to bootstrap again? maybe we need to address the recovery 16:21:36 It is possible that only one of those 2 bridge lines will work consistently 16:21:41 you mean changing the fronts, not isolating the 2 default bridge lines, right? 16:22:06 Yes, changing the fronts, and to avoid confusion, you should use just 1 bridge line 16:22:18 doesn't matter which bridge fingerprint you use 16:22:23 pryty26[mds]: ok i sent you the link 16:22:33 because previously, I tried connecting through a front, but it didn't change anything. is there a specific list of fronts that can connect? i.e. www.phpadmin.net? 16:22:47 *through a different front 16:23:02 There are 2 front domains in the bridge line, though I believe somewhere we have a list of other ones that may work 16:23:34 cohosh: the problem is that snowflake continues to use that given blocked edge node, even if it reattempts, so... 16:23:48 ah we might cache DNS lookups 16:24:30 cohosh: yes that's possible, I wonder what the DNS TTL is, because that would be the time to wait before getting another chance 16:24:44 thx 16:25:41 * dcf1 searches but gets 429'ed by gitlab 16:26:14 while we look into this, i can update our circumvention settings to recommend ampcache in russia or at least prioritize it: https://bridges.torproject.org/moat/circumvention/map 16:26:37 and open an issue for it 16:27:17 dcf1: if you proxy that blocked edge node through a proxy, then snowflake works again 16:27:30 past potentially related issues tpo/anti-censorship/censorship-analysis#40054 tpo/anti-censorship/team#168 16:28:16 unic3rn: ok that is consistent with DNS round robin and one IP address being blocked 16:29:09 so you mean that it could make some sense, the proxy part 16:30:06 well, it depends, I guess, on whether the proxy server is subject to TSPU on its outgoing connection. Because the proxy server, too, has to resolve a DNS name and may get a blocked IP address or an unblocked IP address. 16:31:31 I cannot find where we had a list of other possible cdn77 front domains, but unic3rn maybe you can do the test with at least the 2 preconfigured ones. Try one at a time. 16:32:00 dcf1: well, yes, but it worked then 16:32:28 there are some listed in https://gitlab.torproject.org/tpo/anti-censorship/team/-/work_items/168 16:32:47 thx cohosh 16:36:11 ok i'll open an issue for this and we can continue discussion and mitigation strategies there 16:36:22 ok 16:36:39 anything more on this topic we should discuss now? 16:36:48 thank you for reporting this unic3rn 16:37:04 for now, that's probably it 16:37:55 okay the next topic is about snowflake metrics and the BEGIN_DIR DoS attack 16:38:09 This is just FYI 16:38:34 cool, thank you for following up on it 16:38:35 The tor 0.4.9.15 release has a mitigation for some kind of ongoing attack that involves a flood of BEGIN_DIR messages 16:38:54 I was not previously aware of the attack, but there are some scattered threads on the tor-relays mailing list 16:39:20 it's consistent with inflated user counts, since directory requests is how user counts are estimated 16:39:47 I also made a graph (linked) that shows elevated dirreq-write bandwidth on snowflake-01 during the event in September, so I'm pretty confident it's the same cause 16:40:05 I posted a message on tor-relays to ask if any other operators had seen their user counts increase while under attack 16:40:34 The snowflake bridges were just upgraded to 0.4.9.15, so maybe we will not have those mysterious graph artifacts in the future. 16:41:24 that's all 16:41:37 nice 16:42:04 anything else for today? pryty26[mds] i saw you added an item and then removed it? 16:45:56 ok i'll end the meeting here 16:46:00 thanks everyone! 16:46:08 #endmeeting