16:00:26 <onyinyang> #startmeeting tor anti-censorship meeting
16:00:26 <MeetBot> Meeting started Thu Aug 14 16:00:26 2025 UTC.  The chair is onyinyang. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:26 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:00:26 <onyinyang> hello everyone!
16:00:26 <onyinyang> here is our meeting pad: [https://pad.riseup.net/p/r.9574e996bb9c0266213d38b91b56c469](https://pad.riseup.net/p/r.9574e996bb9c0266213d38b91b56c469)
16:00:30 <shelikhoo> hi~hi~
16:00:53 <meskio> hello
16:01:22 <cohosh> hi
16:03:05 <onyinyang> ok, we have a couple of things to discuss and then the reading group discussion (which I forgot about XD but am currently skimming)
16:03:25 <onyinyang> The first discussion topic is: SQS snowflake rendezvous price check?
16:04:21 <dcf1> this is from me
16:04:27 <cohosh> i just updated the wiki for last month: https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Snowflake-costs
16:05:04 <dcf1> oh thank you
16:05:10 <dcf1> this is what I was curious
16:05:12 <dcf1> about
16:05:46 <dcf1> imo the costs are starting to get out of hand for cohosh to continue paying out of pocket
16:05:53 <meskio> is this in the process to be moved under TPO umbrella?
16:05:58 <cohosh> this is partially my bad for not following up with micah
16:06:00 <cohosh> https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43842#note_3210215
16:06:25 <meskio> ahh, ok, so is in your plate
16:06:32 <meskio> let me know if you need any help with it
16:07:20 <dcf1> I would support cohosh going on strike to force an action
16:07:44 <meskio> I love strikes, I'm in
16:07:48 <cohosh> hehe
16:07:52 <onyinyang> XD
16:07:53 <shelikhoo> hahahaha
16:08:05 <dcf1> I was not aware of the tpo/applications/tor-browser#43842 discussion, thank you
16:08:13 <cohosh> i will send another email, i've lost track of where the conversation was at
16:08:48 <morganava> ^is this an apps problem or higher up the chain?
16:08:56 <morganava> (sorry i've a tor-browser ping)
16:09:16 <cohosh> morganava: no not at all, heh, it's just a coincidence this is the public comment i have about this payment discussion
16:09:48 <morganava> ack
16:09:56 * meskio wonders if tor-browser keyword summons morganava
16:10:06 <morganava> indeed
16:10:07 <cohosh> XD
16:10:13 <dcf1> morganava: but feel free to spread awareness, anti-censorship team members are the type to carry on without complaint
16:10:27 <morganava> boo, complain early complain often
16:10:38 <morganava> squeeky wheel gets the grease and all that
16:10:39 <morganava> anywa
16:10:41 * morganava awaaaaay
16:10:47 <onyinyang> thanks dcf1 for bringing this to our attention and advocating for a change on this :)
16:10:52 <cohosh> regardless of whether tor pays for this, i don't feel comfortable continuing to spend over 30 USD / month on this, i think the next steps are:
16:10:57 <cohosh> - see if tor will pay for it
16:11:08 <cohosh> - if not, try and find other outside funding
16:11:33 <cohosh> - otherwise, it isn't a very cost efficient signalling channel so shut it down until we can figure out how the make it more sustainable
16:11:55 <cohosh> are there any places that are solely relying on it?
16:12:34 <cohosh> maybe china, according to https://bridges.torproject.org/moat/circumvention/map
16:12:58 <cohosh> and https://snowflake-broker.torproject.net/metrics
16:13:10 <cohosh> client-sqs-ips CN=61904,PL=31576,
16:13:21 <cohosh> client-http-ips RU=2606272,IR=1180624,US=575440,DE=400744,CN=291296,
16:13:22 <meskio> from what I got from micah tor does have credits in AWS and it should be fine, but let's see
16:13:27 <dcf1> hej polska
16:13:27 <cohosh> ok
16:13:36 <cohosh> :)
16:13:41 <shelikhoo> I think the domain fronting is also working in China
16:13:49 <shelikhoo> so it is not like a hard dependency
16:13:56 <cohosh> good to know
16:14:43 <shelikhoo> https://gitlab.torproject.org/tpo/anti-censorship/connectivity-measurement/bridgestatus/-/blob/main/recentResult_cnnext?ref_type=heads
16:14:51 <cohosh> that's it from me, thanks for the reminder on that :)
16:14:52 <shelikhoo> or at least equally malfunctioning
16:15:04 <shelikhoo> I think I should have a look at the situation next week....
16:15:26 <cohosh> oh thanks shelikhoo
16:15:35 <onyinyang> ok let's move on to the next topic
16:15:42 <onyinyang> Why was the snowflake broker restarted 2025-08-02?
16:15:52 <cohosh> this is a curiosity question from me
16:16:06 <shelikhoo> i don't recall restarting it...
16:16:39 <cohosh> i saw a gap in our snowflake metrics for august 2nd
16:17:07 <shelikhoo> do we know who restarted it or is that a automatic restart?
16:18:40 <cohosh> was it restarted when the bridge went down (or maybe it crashed)?
16:20:32 <cohosh> it looks like there was an overload that happened at that time: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40475
16:20:58 <meskio> being the same date looks related
16:21:00 <cohosh> maybe i should check the broker logs
16:21:12 <meskio> maybe it overloaded, is it configured to automatically restart?
16:21:22 <cohosh> not that i know of but maybe there was a crash
16:21:27 <cohosh> which would be good to know about
16:21:33 <shelikhoo> I think it is an automatic restart
16:21:44 <shelikhoo> we should keep a copy of the log
16:21:48 <shelikhoo> if it is still there
16:21:55 <cohosh> shelikhoo: by auto restart, what do you mean?
16:22:10 <cohosh> that it will restart if it crashed or that there is some mechanism in place to periodically restart the broker machine?
16:22:17 <cohosh> or the broker process rather
16:22:29 <cohosh> or maybe it was a hosting provider outage
16:22:35 <shelikhoo> last
16:22:35 <shelikhoo> shelikho pts/0        ***    Thu Aug 14 ***   still logged in
16:22:35 <shelikhoo> cohosh   pts/2        ***     Thu Jul 24 ***
16:22:35 <shelikhoo> cohosh   pts/0        ***     Thu Jul 24 ***
16:22:36 <shelikhoo> cohosh   pts/0        ***     Thu Jul 24 ***
16:22:39 <shelikhoo> reboot   system boot  6.1.131          Tue Jul  1 16:57   still running
16:23:07 <shelikhoo> if the broker process crashed, systemd should restart the process automatically
16:23:10 <dcf1> so it was only the broker process that restarted, not the OS
16:23:14 <shelikhoo> yes
16:23:22 <cohosh> yeah, so there was potentially a crash
16:23:24 <dcf1> and therefore there might be some info in the snowflake-broker log
16:23:38 <meskio> if the logs are kept for 12 days
16:23:49 <cohosh> i'll look into this, thanks for confirming it wasn't intentional
16:25:56 <shelikhoo> (shell is dumping logs....)
16:25:56 <onyinyang> next on the agenda, we have 2 interesting links
16:26:29 <meskio> the IRBlock paper might be interesting for a future reading group
16:27:01 <onyinyang> Yes, now that the pdf is available :D
16:27:30 <meskio> :)
16:28:11 <shelikhoo> nice!
16:28:17 <onyinyang> Should we go ahead and set it as the next reading group reading at some future date?
16:29:02 <meskio> sure
16:29:08 <meskio> I would propose september 11
16:29:17 <meskio> because I'll be AFK the weeks before
16:29:30 <onyinyang> works for me afaik
16:29:44 <shelikhoo> it should work for me
16:29:50 <cohosh> same here
16:31:04 <meskio> then I guess is set
16:31:12 <onyinyang> ok cool :) I will put it in my calendar so I don't forget this time /o\
16:31:35 <shelikhoo> cohosh: the restart happened at 1990804:Aug 02 12:03:41 snowflake-broker-40349 broker[488594]: 2025/08/02 12:03:41 Loading geoip databases
16:32:42 <cohosh> shelikhoo: thanks! i had the time figured out, is there anything in the logs before that to indicate a crash?
16:33:07 <cohosh> i guess it could've been the OOM killer, which wouldn't show up there
16:33:51 <shelikhoo> yes... my command is running, I will randomly insert more progress report...
16:34:13 <cohosh> ok we can continue in #tor-anticensorship :)
16:34:26 <onyinyang> ok, let's move on to today'
16:34:29 <onyinyang> s reading group
16:34:32 <shelikhoo> 2248897:Aug 02 12:03:34 snowflake-broker-40349 kernel: Out of memory: Killed process 629 (broker) total-vm:7873300kB, anon-rss:5600452kB, file-rss:0kB, shmem-rss:0kB, UID:1003 pgtables:11224kB oom_score_adj:200
16:34:37 <shelikhoo> it is oom killer
16:35:08 <onyinyang> Today we are discussing: Encrypted Client Hello (ECH) in Censorship Circumvention" https://www.petsymposium.org/foci/2025/foci-2025-0016.pdf
16:35:25 <cohosh> shelikhoo: well that's good to know, i'll make an issue for it, thanks!
16:35:39 <onyinyang> Does anyone have a brief summary prepared or want to summarize on the fly?
16:35:39 <shelikhoo> yes, thanks cohosh!!!
16:37:01 <meskio> I can try to do a sumary
16:37:16 <meskio> the paper analyzes the status of ECH
16:37:25 <meskio> both from the deployment side, as in how many websites do support it
16:37:44 <meskio> and on the censorship side, as in if censored and how in different locations
16:37:54 <meskio> the deployment outside cloudflare is minimal
16:38:11 <meskio> but it looks like only one mayor https server (caddy) supports it, and is very recent
16:38:40 <meskio> they did try multiple techniques to test if the top 1M websites support ECH with very few postive responses outside cloudflare
16:39:05 <meskio> on the censor side, they see Russia blocking cloudlfare's ECH specifically
16:39:33 <meskio> and they believe the rest of the censors rely on DNS to block ECH websited, and don't bother to do anything directly with ECH
16:39:42 <meskio> do I miss something important?
16:41:35 <meskio> I guess we can go to the discussion
16:41:46 <onyinyang> thanks meskio
16:42:03 <meskio> for me is unsurprising that only cloudflare supports it, I assume most websites are waiting for nginx or apache to support it
16:42:15 <dcf1> this paper can be compared to the 2019 paper on ESNI https://www.usenix.org/system/files/foci19-paper_chai_update.pdf https://github.com/net4people/bbs/issues/10
16:42:18 <meskio> and nginx and apache are waiting for openssl to support it...
16:43:01 <shelikhoo> yes, thanks meskio. discussion wise, I think ECH is a nice protocol, especially now that many browsers might send ech extension in client hello by default
16:43:53 <meskio> yes, AFAIK this is why Russia is so careful on their block of cloudflare's ECH, doing it only on the right domain name, the right IP range and if the encrypted hello is included
16:44:24 <meskio> also, interesting that the researchers found that the block only affects the cloudflare IP block
16:45:50 <shelikhoo> yes... I wouldn't be surprised if they block by cloudflare SNI and extension
16:46:04 <shelikhoo> but they actually also added ip range constraint
16:46:10 <meskio> I found very useful the paper to understand better ECH, their description of the protocol is clearer than other pieces I have read (or maybe it is for me after reading this other pieces :)
16:47:20 <shelikhoo> in the same time, other censors blocks ECH by blocking it at DNS step
16:47:26 <onyinyang> meskio, yeah, I think it also helps to have the context of how it is/isn't working in the current censorship landscape
16:48:30 <meskio> shelikhoo: yes, but this is only useful as long as users don't find a unkown by the censor DoH server or some other way to scape DNS blocks
16:49:04 <meskio> but sure, as stated in many places before, censors don't tend to care about perfection, but to make a statement
16:49:37 <shelikhoo> meskio: yes, and in cloudflare's case, you can query a non-censored domain to get the ech key
16:49:50 <shelikhoo> and then use that key to connect to anther censored domain
16:50:26 <shelikhoo> ("ech_query_domain" option in v2ray)
16:51:02 <shelikhoo> and then use that key to connect to another censored domain
16:51:53 <meskio> true :D
16:53:09 <meskio> would be interesting to see this research done again in a couple of years, when I hope ECH will have more adoption
16:54:48 <shelikhoo> yes... and adoption outside cloudflare
16:55:20 <meskio> yes
16:56:15 <meskio> but the interesting adoption for anti-censorship is that mayor browsers do send GREASE ECH since a while, so maybe ECH is something that could be used some time soonish
16:57:45 <shelikhoo> yes, and censor is already reacting to this grease with more restraints
16:58:18 <meskio> is there any reaction to GREASE ECH besides russia's block of cloudflare?
16:59:41 <shelikhoo> I think there was a reason why censors in the rest of world block ech by dns instead of tls extension
17:00:16 <meskio> ahh, I see what you mean, yes, which is good news for us
17:00:35 <shelikhoo> yes!
17:00:58 <meskio> pitty that cloudlfare only accept ECH connections to the couldflare-ech.com domain
17:02:06 <cohosh> shelikhoo: meskio: so you're saying the potential blocking resistance could come from a domain fronting style interaction with ech
17:02:07 <shelikhoo> yes, I think this is by kind of by their design
17:02:42 <shelikhoo> cohosh: yes, and this is used in practice already
17:02:43 <cohosh> where if a provider excepts ECH connections to a domain that also serves other content or would force a censor to make a tradeoff to block it
17:02:49 <meskio> yes, we might be able to use ech as signaling channel, for most places cloudflare will work
17:03:29 <cohosh> shelikhoo: where is it used in practice now?
17:03:42 <cohosh> i guess not with cloudflare?
17:04:14 <meskio> v2ray websocket is being used like that in cloudflare, isn't it?
17:04:39 <cohosh> ah, i think i misunderstood meskio's comment above about cloudlflare-ech.com
17:04:49 <cohosh> *cloudflare-ech.com
17:05:46 <shelikhoo> yes.... i think some user are using it with v2ray already
17:06:01 <shelikhoo> but I do not have exact numbers...\
17:06:08 <meskio> :)
17:06:23 <onyinyang> very cool
17:06:47 <onyinyang> we are a bit over time but is there anything else anyone wants to add on this before we wrap up?
17:06:56 <meskio> I think I'm done
17:07:05 <shelikhoo> EOF from me
17:07:35 <cohosh> sorry one more question
17:07:44 <cohosh> so is this statement in the paper false then:
17:07:45 <cohosh> This makes be-
17:07:45 <cohosh> nign ECH connections to Cloudflare trivially distinguishable from
17:07:45 <cohosh> GREASE ECH connections to their servers and allows censors to
17:07:45 <cohosh> block every ECH connection to Cloudflare by censoring the domain
17:07:47 <cohosh> cloudflare-ech.com in the SNI extension
17:08:26 <cohosh> if there's a way to use Cloudflare ECH by using a domain other than cloudflare-ech.com and relying on a domain fronting type thing, it's not trivially distinguishable?
17:08:34 <dcf1> that statement is true to my knowledge
17:08:45 <meskio> no, there is no other way, this statement is true
17:08:51 <shelikhoo> no, so there is no hiding that one is using ECH
17:09:01 <meskio> what I say is censors need to choose between blocking all cloudflare, or leting us pass
17:09:06 <shelikhoo> but to bypass censor by DNS
17:09:15 <meskio> russia choosed to block all, but others didn't (yet?)
17:09:27 <cohosh> ah thanks, i see my misunderstanding now :)
17:09:27 <shelikhoo> this "domain name fronting" works
17:09:46 <shelikhoo> hehe! nice!
17:09:48 <shelikhoo> EOF
17:10:04 <cohosh> that's it from me, thanks for the discussion
17:10:19 <onyinyang> thanks everyone!
17:10:24 <meskio> yes, nice discussion, thanks
17:10:31 <onyinyang> #endmeeting