16:01:22 <cohosh> #startmeeting anti-censorship team meeting
16:01:22 <MeetBot> Meeting started Thu Oct 29 16:01:22 2026 UTC.  The chair is cohosh. Information about MeetBot at https://wiki.debian.org/MeetBot.
16:01:22 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:02:04 <cohosh> here is our meeting pad: https://pad.riseup.net/p/r.9574e996bb9c0266213d38b91b56c469
16:02:12 <cohosh> please feel free to add to the agenda
16:02:25 <cohosh> let us know if you need a link to edit it
16:03:56 <cohosh> the first discussion item is about cdn77 blocking in russia
16:04:44 <cohosh> is the reporter around?
16:04:56 <unic3rn> hello
16:04:58 <unic3rn> yes
16:05:11 <unic3rn> I wrote a note on the meeting pad
16:05:47 <cohosh> thanks!
16:06:36 <unic3rn> so when you bootstrap snowflake, you can get a blocked cdn77 node, which won't let you bootstrap
16:06:58 <cohosh> if the cdn77 domain fronting channel is blocked, one thing we can do is recommend using ampcache to users in russia
16:07:08 <dcf1> is it a DNS round robin thing? You resolve the same DNS name, and it's chance what IP address it resolves to?
16:07:20 <gabaaa[mds]> hi
16:07:45 <pryty26[mds]> hi
16:08:06 <unic3rn> dcf1: do you mean the cdn, or the dpi?
16:08:25 <dcf1> The cdn
16:08:50 <dcf1> If I understand you, 1 cdn77 edge server is blocked, but others are not blocked
16:08:57 <unic3rn> yes, default snowflake bridges contain a cdn77 url, it redirects you to some cdn for further connection
16:09:17 <unic3rn> no, one specific one, in my experience
16:09:20 <dcf1> redirects? do you mean an HTTP redirect (e.g. status 301)?
16:09:22 <unic3rn> is blocked
16:09:27 <pryty26[mds]> is rob tday btw
16:09:28 <onyinyang[mds]> hi, sorry I was in another meeting and forgot we had this one right after >.<
16:10:11 <dcf1> "one specific one, in my experience" that sounds like it matches my understanding ("1 cdn77 edge server is blocked, but others are not blocked"), but you also said "no", so I am confused.
16:10:18 <unic3rn> so you start bootstrapping to snowflake, and then cdn77 sends you to some cdn edge node, as I understand it, for further connection
16:10:42 <dcf1> When you say "one specific node", are you talking about one specific IP address? Or is there something else that distinguishes different nodes?
16:11:04 <unic3rn> I noticed one node/IP address that is blocked
16:11:31 <dcf1> "then cdn77 sends you to some cdn edge node" that is not how it works, unless cdn77 is very different from other CDNs
16:11:57 <dcf1> the cdn77 edge server probably routes the traffic internally, but that's *after* you successfully communicate with the edge server
16:12:24 <robgjansen[mds]> pryty26[mds]: FYI: Just stopping by to quickly note that I am unable to attend today - I am running off literally now to meet with a potential sponsor. But I am planning to join next week.
16:12:37 <dcf1> I think the most plausible explanation is that the cdn77 domain name in the URL may resolve to several IP addresses, and it's random chance which one you get assigned on any given attempt
16:13:05 <dcf1> I'm just checking whether that plausible explanation fits with what you have observed, or if the cause may be something less expected
16:13:29 <unic3rn> that's weird, because the cdn77 url in tor's default snowflake bridge doesn't contain the specific edge node IP
16:13:30 <dcf1> unic3rn: okay that is interesting
16:13:50 <unic3rn> *as one of it's IP addresses in it's hostname
16:14:24 <dcf1> it is possible that there is some kind of HTTP-level redirect or something, I don't necessarily doubt you, it's just not the most likely thing to happen
16:14:52 <dcf1> It's also possible that the edge server is located inside Russia, and the connection gets blocked between the edge server and the Russian network border
16:15:37 <unic3rn> that, specifically, is not the case...
16:15:40 <dcf1> url=https://1098762253.rsc.cdn77.org/ fronts=www.cdn77.com,www.phpmyadmin.net
16:16:16 <dcf1> it's also possible that it happens only when one of the possible front domains is randomly chosen (this random selection happens on the client side)
16:16:35 <pryty26[mds]> robgjansen:  Okay thx
16:16:42 <pryty26[mds]> for clarification
16:16:57 <pryty26[mds]> Would you be free for talk in DM at tmr or latter?
16:17:01 <pryty26[mds]> *later
16:17:24 <unic3rn> so you mean: when a domain fronting situation is happening? (fake SNI, same cdn ip)
16:17:51 <dcf1> it's always using domain fronting, but the specific SNI that's chosen is random on each attempt
16:18:17 <dcf1> It could be that it's one of the IP addresses of www.phpmyadmin.net, not of www.cdn77.com.
16:18:56 <dcf1> "the cdn77 url in tor's default snowflake bridge doesn't contain the specific edge node IP" the IP address depends on the front domain, since that is what is looked up in DNS
16:19:12 <unic3rn> well, the SNI is not the problem, I think... the problem is that the given edge node IP is getting throttled.
16:19:29 <pryty26[mds]> cohosh:  May I ask if you could look into the dm
16:19:31 <pryty26[mds]> OuO
16:20:08 <cohosh> pryty26[mds]: there is a meeting in progress, we have an agenda, please be considerate
16:20:35 <dcf1> unic3rn: thank you for this information. maybe you can run a test for us.
16:20:44 <pryty26[mds]> Keke, Like I'm asking the meeting pad
16:20:51 <unic3rn> in my understanding, you can be given any cdn77 node, but you previously mentioned it otherwise?
16:21:02 <dcf1> Try editing the bridge line so that only one of the two possible front domains is used
16:21:08 <dcf1> so try connecting with
16:21:12 <dcf1> fronts=www.cdn77.com
16:21:17 <dcf1> and then try connecting with
16:21:23 <dcf1> fronts=www.phpmyadmin.net
16:21:32 <cohosh> unic3rn: your note says that this blocking prevents snowflake from bootstrapping, does the client not attempt to bootstrap again? maybe we need to address the recovery
16:21:36 <dcf1> It is possible that only one of those 2 bridge lines will work consistently
16:21:41 <unic3rn> you mean changing the fronts, not isolating the 2 default bridge lines, right?
16:22:06 <dcf1> Yes, changing the fronts, and to avoid confusion, you should use just 1 bridge line
16:22:18 <dcf1> doesn't matter which bridge fingerprint you use
16:22:23 <cohosh> pryty26[mds]: ok i sent you the link
16:22:33 <unic3rn> because previously, I tried connecting through a front, but it didn't change anything. is there a specific list of fronts that can connect? i.e. www.phpadmin.net?
16:22:47 <unic3rn> *through a different front
16:23:02 <dcf1> There are 2 front domains in the bridge line, though I believe somewhere we have a list of other ones that may work
16:23:34 <unic3rn> cohosh: the problem is that snowflake continues to use that given blocked edge node, even if it reattempts, so...
16:23:48 <cohosh> ah we might cache DNS lookups
16:24:30 <dcf1> cohosh: yes that's possible, I wonder what the DNS TTL is, because that would be the time to wait before getting another chance
16:24:44 <pryty26[mds]> thx
16:25:41 * dcf1 searches but gets 429'ed by gitlab
16:26:14 <cohosh> while we look into this, i can update our circumvention settings to recommend ampcache in russia or at least prioritize it: https://bridges.torproject.org/moat/circumvention/map
16:26:37 <cohosh> and open an issue for it
16:27:17 <unic3rn> dcf1: if you proxy that blocked edge node through a proxy, then snowflake works again
16:27:30 <dcf1> past potentially related issues tpo/anti-censorship/censorship-analysis#40054 tpo/anti-censorship/team#168
16:28:16 <dcf1> unic3rn: ok that is consistent with DNS round robin and one IP address being blocked
16:29:09 <unic3rn> so you mean that it could make some sense, the proxy part
16:30:06 <dcf1> well, it depends, I guess, on whether the proxy server is subject to TSPU on its outgoing connection. Because the proxy server, too, has to resolve a DNS name and may get a blocked IP address or an unblocked IP address.
16:31:31 <dcf1> I cannot find where we had a list of other possible cdn77 front domains, but unic3rn maybe you can do the test with at least the 2 preconfigured ones. Try one at a time.
16:32:00 <unic3rn> dcf1: well, yes, but it worked then
16:32:28 <cohosh> there are some listed in https://gitlab.torproject.org/tpo/anti-censorship/team/-/work_items/168
16:32:47 <dcf1> thx cohosh
16:36:11 <cohosh> ok i'll open an issue for this and we can continue discussion and mitigation strategies there
16:36:22 <unic3rn> ok
16:36:39 <cohosh> anything more on this topic we should discuss now?
16:36:48 <cohosh> thank you for reporting this unic3rn
16:37:04 <unic3rn> for now, that's probably it
16:37:55 <cohosh> okay the next topic is about snowflake metrics and the BEGIN_DIR DoS attack
16:38:09 <dcf1> This is just FYI
16:38:34 <cohosh> cool, thank you for following up on it
16:38:35 <dcf1> The tor 0.4.9.15 release has a mitigation for some kind of ongoing attack that involves a flood of BEGIN_DIR messages
16:38:54 <dcf1> I was not previously aware of the attack, but there are some scattered threads on the tor-relays mailing list
16:39:20 <dcf1> it's consistent with inflated user counts, since directory requests is how user counts are estimated
16:39:47 <dcf1> I also made a graph (linked) that shows elevated dirreq-write bandwidth on snowflake-01 during the event in September, so I'm pretty confident it's the same cause
16:40:05 <dcf1> I posted a message on tor-relays to ask if any other operators had seen their user counts increase while under attack
16:40:34 <dcf1> The snowflake bridges were just upgraded to 0.4.9.15, so maybe we will not have those mysterious graph artifacts in the future.
16:41:24 <dcf1> that's all
16:41:37 <cohosh> nice
16:42:04 <cohosh> anything else for today? pryty26[mds] i saw you added an item and then removed it?
16:45:56 <cohosh> ok i'll end the meeting here
16:46:00 <cohosh> thanks everyone!
16:46:08 <cohosh> #endmeeting