16:00:26 <onyinyang> #startmeeting tor anti-censorship meeting 16:00:26 <MeetBot> Meeting started Thu Aug 14 16:00:26 2025 UTC. The chair is onyinyang. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:26 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic. 16:00:26 <onyinyang> hello everyone! 16:00:26 <onyinyang> here is our meeting pad: [https://pad.riseup.net/p/r.9574e996bb9c0266213d38b91b56c469](https://pad.riseup.net/p/r.9574e996bb9c0266213d38b91b56c469) 16:00:30 <shelikhoo> hi~hi~ 16:00:53 <meskio> hello 16:01:22 <cohosh> hi 16:03:05 <onyinyang> ok, we have a couple of things to discuss and then the reading group discussion (which I forgot about XD but am currently skimming) 16:03:25 <onyinyang> The first discussion topic is: SQS snowflake rendezvous price check? 16:04:21 <dcf1> this is from me 16:04:27 <cohosh> i just updated the wiki for last month: https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Snowflake-costs 16:05:04 <dcf1> oh thank you 16:05:10 <dcf1> this is what I was curious 16:05:12 <dcf1> about 16:05:46 <dcf1> imo the costs are starting to get out of hand for cohosh to continue paying out of pocket 16:05:53 <meskio> is this in the process to be moved under TPO umbrella? 16:05:58 <cohosh> this is partially my bad for not following up with micah 16:06:00 <cohosh> https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43842#note_3210215 16:06:25 <meskio> ahh, ok, so is in your plate 16:06:32 <meskio> let me know if you need any help with it 16:07:20 <dcf1> I would support cohosh going on strike to force an action 16:07:44 <meskio> I love strikes, I'm in 16:07:48 <cohosh> hehe 16:07:52 <onyinyang> XD 16:07:53 <shelikhoo> hahahaha 16:08:05 <dcf1> I was not aware of the tpo/applications/tor-browser#43842 discussion, thank you 16:08:13 <cohosh> i will send another email, i've lost track of where the conversation was at 16:08:48 <morganava> ^is this an apps problem or higher up the chain? 16:08:56 <morganava> (sorry i've a tor-browser ping) 16:09:16 <cohosh> morganava: no not at all, heh, it's just a coincidence this is the public comment i have about this payment discussion 16:09:48 <morganava> ack 16:09:56 * meskio wonders if tor-browser keyword summons morganava 16:10:06 <morganava> indeed 16:10:07 <cohosh> XD 16:10:13 <dcf1> morganava: but feel free to spread awareness, anti-censorship team members are the type to carry on without complaint 16:10:27 <morganava> boo, complain early complain often 16:10:38 <morganava> squeeky wheel gets the grease and all that 16:10:39 <morganava> anywa 16:10:41 * morganava awaaaaay 16:10:47 <onyinyang> thanks dcf1 for bringing this to our attention and advocating for a change on this :) 16:10:52 <cohosh> regardless of whether tor pays for this, i don't feel comfortable continuing to spend over 30 USD / month on this, i think the next steps are: 16:10:57 <cohosh> - see if tor will pay for it 16:11:08 <cohosh> - if not, try and find other outside funding 16:11:33 <cohosh> - otherwise, it isn't a very cost efficient signalling channel so shut it down until we can figure out how the make it more sustainable 16:11:55 <cohosh> are there any places that are solely relying on it? 16:12:34 <cohosh> maybe china, according to https://bridges.torproject.org/moat/circumvention/map 16:12:58 <cohosh> and https://snowflake-broker.torproject.net/metrics 16:13:10 <cohosh> client-sqs-ips CN=61904,PL=31576, 16:13:21 <cohosh> client-http-ips RU=2606272,IR=1180624,US=575440,DE=400744,CN=291296, 16:13:22 <meskio> from what I got from micah tor does have credits in AWS and it should be fine, but let's see 16:13:27 <dcf1> hej polska 16:13:27 <cohosh> ok 16:13:36 <cohosh> :) 16:13:41 <shelikhoo> I think the domain fronting is also working in China 16:13:49 <shelikhoo> so it is not like a hard dependency 16:13:56 <cohosh> good to know 16:14:43 <shelikhoo> https://gitlab.torproject.org/tpo/anti-censorship/connectivity-measurement/bridgestatus/-/blob/main/recentResult_cnnext?ref_type=heads 16:14:51 <cohosh> that's it from me, thanks for the reminder on that :) 16:14:52 <shelikhoo> or at least equally malfunctioning 16:15:04 <shelikhoo> I think I should have a look at the situation next week.... 16:15:26 <cohosh> oh thanks shelikhoo 16:15:35 <onyinyang> ok let's move on to the next topic 16:15:42 <onyinyang> Why was the snowflake broker restarted 2025-08-02? 16:15:52 <cohosh> this is a curiosity question from me 16:16:06 <shelikhoo> i don't recall restarting it... 16:16:39 <cohosh> i saw a gap in our snowflake metrics for august 2nd 16:17:07 <shelikhoo> do we know who restarted it or is that a automatic restart? 16:18:40 <cohosh> was it restarted when the bridge went down (or maybe it crashed)? 16:20:32 <cohosh> it looks like there was an overload that happened at that time: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40475 16:20:58 <meskio> being the same date looks related 16:21:00 <cohosh> maybe i should check the broker logs 16:21:12 <meskio> maybe it overloaded, is it configured to automatically restart? 16:21:22 <cohosh> not that i know of but maybe there was a crash 16:21:27 <cohosh> which would be good to know about 16:21:33 <shelikhoo> I think it is an automatic restart 16:21:44 <shelikhoo> we should keep a copy of the log 16:21:48 <shelikhoo> if it is still there 16:21:55 <cohosh> shelikhoo: by auto restart, what do you mean? 16:22:10 <cohosh> that it will restart if it crashed or that there is some mechanism in place to periodically restart the broker machine? 16:22:17 <cohosh> or the broker process rather 16:22:29 <cohosh> or maybe it was a hosting provider outage 16:22:35 <shelikhoo> last 16:22:35 <shelikhoo> shelikho pts/0 *** Thu Aug 14 *** still logged in 16:22:35 <shelikhoo> cohosh pts/2 *** Thu Jul 24 *** 16:22:35 <shelikhoo> cohosh pts/0 *** Thu Jul 24 *** 16:22:36 <shelikhoo> cohosh pts/0 *** Thu Jul 24 *** 16:22:39 <shelikhoo> reboot system boot 6.1.131 Tue Jul 1 16:57 still running 16:23:07 <shelikhoo> if the broker process crashed, systemd should restart the process automatically 16:23:10 <dcf1> so it was only the broker process that restarted, not the OS 16:23:14 <shelikhoo> yes 16:23:22 <cohosh> yeah, so there was potentially a crash 16:23:24 <dcf1> and therefore there might be some info in the snowflake-broker log 16:23:38 <meskio> if the logs are kept for 12 days 16:23:49 <cohosh> i'll look into this, thanks for confirming it wasn't intentional 16:25:56 <shelikhoo> (shell is dumping logs....) 16:25:56 <onyinyang> next on the agenda, we have 2 interesting links 16:26:29 <meskio> the IRBlock paper might be interesting for a future reading group 16:27:01 <onyinyang> Yes, now that the pdf is available :D 16:27:30 <meskio> :) 16:28:11 <shelikhoo> nice! 16:28:17 <onyinyang> Should we go ahead and set it as the next reading group reading at some future date? 16:29:02 <meskio> sure 16:29:08 <meskio> I would propose september 11 16:29:17 <meskio> because I'll be AFK the weeks before 16:29:30 <onyinyang> works for me afaik 16:29:44 <shelikhoo> it should work for me 16:29:50 <cohosh> same here 16:31:04 <meskio> then I guess is set 16:31:12 <onyinyang> ok cool :) I will put it in my calendar so I don't forget this time /o\ 16:31:35 <shelikhoo> cohosh: the restart happened at 1990804:Aug 02 12:03:41 snowflake-broker-40349 broker[488594]: 2025/08/02 12:03:41 Loading geoip databases 16:32:42 <cohosh> shelikhoo: thanks! i had the time figured out, is there anything in the logs before that to indicate a crash? 16:33:07 <cohosh> i guess it could've been the OOM killer, which wouldn't show up there 16:33:51 <shelikhoo> yes... my command is running, I will randomly insert more progress report... 16:34:13 <cohosh> ok we can continue in #tor-anticensorship :) 16:34:26 <onyinyang> ok, let's move on to today' 16:34:29 <onyinyang> s reading group 16:34:32 <shelikhoo> 2248897:Aug 02 12:03:34 snowflake-broker-40349 kernel: Out of memory: Killed process 629 (broker) total-vm:7873300kB, anon-rss:5600452kB, file-rss:0kB, shmem-rss:0kB, UID:1003 pgtables:11224kB oom_score_adj:200 16:34:37 <shelikhoo> it is oom killer 16:35:08 <onyinyang> Today we are discussing: Encrypted Client Hello (ECH) in Censorship Circumvention" https://www.petsymposium.org/foci/2025/foci-2025-0016.pdf 16:35:25 <cohosh> shelikhoo: well that's good to know, i'll make an issue for it, thanks! 16:35:39 <onyinyang> Does anyone have a brief summary prepared or want to summarize on the fly? 16:35:39 <shelikhoo> yes, thanks cohosh!!! 16:37:01 <meskio> I can try to do a sumary 16:37:16 <meskio> the paper analyzes the status of ECH 16:37:25 <meskio> both from the deployment side, as in how many websites do support it 16:37:44 <meskio> and on the censorship side, as in if censored and how in different locations 16:37:54 <meskio> the deployment outside cloudflare is minimal 16:38:11 <meskio> but it looks like only one mayor https server (caddy) supports it, and is very recent 16:38:40 <meskio> they did try multiple techniques to test if the top 1M websites support ECH with very few postive responses outside cloudflare 16:39:05 <meskio> on the censor side, they see Russia blocking cloudlfare's ECH specifically 16:39:33 <meskio> and they believe the rest of the censors rely on DNS to block ECH websited, and don't bother to do anything directly with ECH 16:39:42 <meskio> do I miss something important? 16:41:35 <meskio> I guess we can go to the discussion 16:41:46 <onyinyang> thanks meskio 16:42:03 <meskio> for me is unsurprising that only cloudflare supports it, I assume most websites are waiting for nginx or apache to support it 16:42:15 <dcf1> this paper can be compared to the 2019 paper on ESNI https://www.usenix.org/system/files/foci19-paper_chai_update.pdf https://github.com/net4people/bbs/issues/10 16:42:18 <meskio> and nginx and apache are waiting for openssl to support it... 16:43:01 <shelikhoo> yes, thanks meskio. discussion wise, I think ECH is a nice protocol, especially now that many browsers might send ech extension in client hello by default 16:43:53 <meskio> yes, AFAIK this is why Russia is so careful on their block of cloudflare's ECH, doing it only on the right domain name, the right IP range and if the encrypted hello is included 16:44:24 <meskio> also, interesting that the researchers found that the block only affects the cloudflare IP block 16:45:50 <shelikhoo> yes... I wouldn't be surprised if they block by cloudflare SNI and extension 16:46:04 <shelikhoo> but they actually also added ip range constraint 16:46:10 <meskio> I found very useful the paper to understand better ECH, their description of the protocol is clearer than other pieces I have read (or maybe it is for me after reading this other pieces :) 16:47:20 <shelikhoo> in the same time, other censors blocks ECH by blocking it at DNS step 16:47:26 <onyinyang> meskio, yeah, I think it also helps to have the context of how it is/isn't working in the current censorship landscape 16:48:30 <meskio> shelikhoo: yes, but this is only useful as long as users don't find a unkown by the censor DoH server or some other way to scape DNS blocks 16:49:04 <meskio> but sure, as stated in many places before, censors don't tend to care about perfection, but to make a statement 16:49:37 <shelikhoo> meskio: yes, and in cloudflare's case, you can query a non-censored domain to get the ech key 16:49:50 <shelikhoo> and then use that key to connect to anther censored domain 16:50:26 <shelikhoo> ("ech_query_domain" option in v2ray) 16:51:02 <shelikhoo> and then use that key to connect to another censored domain 16:51:53 <meskio> true :D 16:53:09 <meskio> would be interesting to see this research done again in a couple of years, when I hope ECH will have more adoption 16:54:48 <shelikhoo> yes... and adoption outside cloudflare 16:55:20 <meskio> yes 16:56:15 <meskio> but the interesting adoption for anti-censorship is that mayor browsers do send GREASE ECH since a while, so maybe ECH is something that could be used some time soonish 16:57:45 <shelikhoo> yes, and censor is already reacting to this grease with more restraints 16:58:18 <meskio> is there any reaction to GREASE ECH besides russia's block of cloudflare? 16:59:41 <shelikhoo> I think there was a reason why censors in the rest of world block ech by dns instead of tls extension 17:00:16 <meskio> ahh, I see what you mean, yes, which is good news for us 17:00:35 <shelikhoo> yes! 17:00:58 <meskio> pitty that cloudlfare only accept ECH connections to the couldflare-ech.com domain 17:02:06 <cohosh> shelikhoo: meskio: so you're saying the potential blocking resistance could come from a domain fronting style interaction with ech 17:02:07 <shelikhoo> yes, I think this is by kind of by their design 17:02:42 <shelikhoo> cohosh: yes, and this is used in practice already 17:02:43 <cohosh> where if a provider excepts ECH connections to a domain that also serves other content or would force a censor to make a tradeoff to block it 17:02:49 <meskio> yes, we might be able to use ech as signaling channel, for most places cloudflare will work 17:03:29 <cohosh> shelikhoo: where is it used in practice now? 17:03:42 <cohosh> i guess not with cloudflare? 17:04:14 <meskio> v2ray websocket is being used like that in cloudflare, isn't it? 17:04:39 <cohosh> ah, i think i misunderstood meskio's comment above about cloudlflare-ech.com 17:04:49 <cohosh> *cloudflare-ech.com 17:05:46 <shelikhoo> yes.... i think some user are using it with v2ray already 17:06:01 <shelikhoo> but I do not have exact numbers...\ 17:06:08 <meskio> :) 17:06:23 <onyinyang> very cool 17:06:47 <onyinyang> we are a bit over time but is there anything else anyone wants to add on this before we wrap up? 17:06:56 <meskio> I think I'm done 17:07:05 <shelikhoo> EOF from me 17:07:35 <cohosh> sorry one more question 17:07:44 <cohosh> so is this statement in the paper false then: 17:07:45 <cohosh> This makes be- 17:07:45 <cohosh> nign ECH connections to Cloudflare trivially distinguishable from 17:07:45 <cohosh> GREASE ECH connections to their servers and allows censors to 17:07:45 <cohosh> block every ECH connection to Cloudflare by censoring the domain 17:07:47 <cohosh> cloudflare-ech.com in the SNI extension 17:08:26 <cohosh> if there's a way to use Cloudflare ECH by using a domain other than cloudflare-ech.com and relying on a domain fronting type thing, it's not trivially distinguishable? 17:08:34 <dcf1> that statement is true to my knowledge 17:08:45 <meskio> no, there is no other way, this statement is true 17:08:51 <shelikhoo> no, so there is no hiding that one is using ECH 17:09:01 <meskio> what I say is censors need to choose between blocking all cloudflare, or leting us pass 17:09:06 <shelikhoo> but to bypass censor by DNS 17:09:15 <meskio> russia choosed to block all, but others didn't (yet?) 17:09:27 <cohosh> ah thanks, i see my misunderstanding now :) 17:09:27 <shelikhoo> this "domain name fronting" works 17:09:46 <shelikhoo> hehe! nice! 17:09:48 <shelikhoo> EOF 17:10:04 <cohosh> that's it from me, thanks for the discussion 17:10:19 <onyinyang> thanks everyone! 17:10:24 <meskio> yes, nice discussion, thanks 17:10:31 <onyinyang> #endmeeting