16:00:44 <meskio> #startmeeting tor anti-censorship meeting
16:00:44 <MeetBot> Meeting started Thu Dec 19 16:00:44 2024 UTC.  The chair is meskio. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:44 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:00:48 <meskio> hello everybody!!!
16:00:50 <ggus> o/
16:00:52 <theodorsm> hi!
16:00:54 <meskio> here is our meeting pad: https://pad.riseup.net/p/r.9574e996bb9c0266213d38b91b56c469
16:00:56 <meskio> ask me in private to give you the link of the pad to be able to edit it if you don't have it
16:00:58 <meskio> I'll wait few minutes for everybody to add you've been working on and put items on the agenda
16:01:04 <onyinyang> hihi
16:01:34 <cohosh> hi
16:01:42 <shelikhoo> hi~hi~
16:02:26 <gaba> hi
16:02:36 <WofWca[m]> 👋
16:02:48 <meskio> while we wait for everybody to fill up their work and the agenda
16:03:08 <meskio> for the next two weeks we have a break
16:03:17 <meskio> there will not be meetings until January 9th
16:03:23 <meskio> and most of us will be AFK
16:03:57 <meskio> I might look into irc once in a while just in case something is burning
16:04:12 <meskio> but we'll have a well deserved end of the year break
16:04:25 <meskio> it was an eventful year and everybody did an amazing work, thank you
16:04:49 <shelikhoo> hehe! thanks everyone!!!
16:05:32 <meskio> should we move to the discussion?
16:05:41 <meskio> I kept from last week the point about azure
16:06:35 <meskio> we've updated circumvention settings to don't point into azure for snowflake and instead use ampcache
16:06:59 <meskio> and I sent a merge request to tor-browser to use CDN77 meek bridge as builtin bridge
16:07:05 <meskio> but with the wrong bridge, sorry
16:07:11 <meskio> cohosh will fix it
16:07:44 <cohosh> no worries :)
16:07:49 <meskio> I kept the azure meek bridge there as it might be useful as long as it works, some places might dare more to block azure than CDN77 I think
16:08:16 <meskio> but happy to be convinced otherwise
16:08:29 <dcf1> have you had any difficulty changing the origin for Azure Edgio CDN profiles?
16:08:40 <dcf1> Or have you not actually had to change the origin?
16:08:44 <cohosh> we don't have access to our azure portal for some time now :)
16:08:48 <cohosh> we're locked out afaik
16:08:56 <shelikhoo> amp does not work in China, so if it is the default setting, we might need to have some fallback
16:08:58 <meskio> :(
16:09:14 <cohosh> we also can't necessarily afford to make a new azure account
16:09:16 <dcf1> cohosh: ok, so then you haven't made any origin changes
16:09:18 <cohosh> cdn77 is much cheaper
16:09:25 <cohosh> i think we will just abandon azure for now
16:10:03 <meskio> shelikhoo: yes my change was adding ampcache for china together with CDN77 domainfront, should we use sqs in china?
16:10:13 <dcf1> in the later discussion point, we are encountering problems after changing the origin for an edgio cdn profile: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40427#note_3142268
16:10:23 <shelikhoo> I think sqs would work in China, I can add it to test
16:10:37 <shelikhoo> I think sqs would work in China, I can add it to testing target
16:10:44 <cohosh> oh good idea
16:10:56 <cohosh> we haven't gotten anywhere close to our aws limits yet
16:11:12 <meskio> shelikhoo: good, I'll make the change
16:11:22 <shelikhoo> yes!
16:11:25 <meskio> for now just for china, I'll leave the others as they are
16:11:29 <ggus> shelikhoo: as a snowflake domain fallback, maybe add cdn77 - tiktok domains?
16:12:03 <shelikhoo> I don't know for sure but tiktok app is not working for Chinese users
16:12:27 <shelikhoo> There is a domestic version douyin to work with censorship better
16:12:36 <shelikhoo> I am not sure if it would work
16:12:41 <ggus> hm, i thought it was censored, but not blocked.
16:12:56 <shelikhoo> we could try...
16:14:06 <meskio> we are kind of jumping into the next topic
16:14:25 <meskio> we are using phpmyadmin.net for everything in azure
16:14:43 <meskio> and Russia is already using that name to block hetzner IP range if you visit it
16:14:48 <meskio> maybe we should diversify
16:15:07 <meskio> there is a post in the forum with proposals of alternatives: https://forum.torproject.org/t/tor-and-hetzner-block-in-russia/16134/3
16:15:17 <meskio> and tiktok has a bunch of things there
16:15:57 <meskio> AFAIK we are using phpmyadmin.net in: snowflake, moat and meek
16:18:15 <cohosh> there is also a list in this confidential issue that was about vantage point tests: https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/132#note_3031622
16:19:43 <meskio> so maybe something that needs some work to test in our vantage points and decide in different names
16:19:56 <cohosh> one of them, cdn.zk.mk
16:20:00 <cohosh> we've been using for snowflake
16:20:01 <shelikhoo> I think we have got a lot of domains, and we could either handpick one, or just let client try them all and remember the last working one
16:20:26 <shelikhoo> I could try to add domain fronting testing into vantage points
16:21:29 <cohosh> snowflake's multiple front support has been very useful when faced with these decisions
16:21:49 <cohosh> but as we've seen with phpmyadmin, it's noisy and might cause other problems
16:22:25 <cohosh> vantage point testing sounds like a good idea
16:22:46 <meskio> I agree
16:22:48 <ggus> https://www.wappalyzer.com/technologies/cdn/cdn77/ < phpmyadmin.net is cdn77 top #4 website based on traffic (according to wappalyzer)
16:23:50 <meskio> we can keep the status quo for during the break and work on the vantage point testing after to decide in other names
16:24:27 <meskio> shelikhoo: can you open an issue for the vantage point testing so we don't forget about it?
16:24:42 <shelikhoo> Yes! I will do that after meeting
16:24:47 <meskio> thanks
16:25:37 <meskio> should we move on to the next topic?
16:25:51 <cohosh> dcf1: did you want to brainstorm some solutions to the azure origin issues?
16:26:00 <dcf1> no, I have a specific question
16:26:00 <cohosh> or discuss it in the meeting more?
16:26:23 <dcf1> one possible cause of the problem could be the azure cdn sending the wrong sni to the origin
16:26:54 <dcf1> so what I propose is to sample the SNIs that are reaching the new broker, and see if snowflake-broker.bamsoftware.com is among them
16:27:16 <dcf1> I found a tshark command that prints out the SNI and nothing else
16:27:19 <dcf1> tshark -i eth0 -f 'dst port 443' -Y 'ssl.handshake.extension.type == "server_name"' -T fields -e tls.handshake.extensions_server_name
16:27:35 <dcf1> https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40427#note_3142268
16:27:53 <dcf1> so I'm looking for a sanity check or risk assessment that this is an okay thing to do
16:28:26 <dcf1> if snowflake-broker.bamsoftware.com SNI is reaching the broker, we can work around it by changing the DNS record for snowflake-broker.bamsoftware.com to point to the new broker, and start getting TLS certs for that name
16:28:47 <cohosh> the tshark command sounds okay to me
16:28:48 <dcf1> if snowflake-broker.bamsoftware.com SNI is *not* hitting the new broker, then I don't have any other ideas
16:30:42 <shelikhoo> I think running the command is fine.
16:30:43 <dcf1> if it is, then we have 2 action items. I have to change the DNS record, and shelikhoo will need to make a change on the broker to let it generate certs for that name too
16:31:23 <shelikhoo> Yes. I can do that after dns name is adjusted, or we could just copy the certificate for testing
16:31:41 <shelikhoo> So that the dns change can be done independently of acme setup
16:31:57 <dcf1> ok, I will run the tshark command and report on #40427
16:32:29 <meskio> nice
16:32:29 <shelikhoo> Instead of dcf1 and me trying to reduce down time by waiting for each other
16:33:04 <shelikhoo> And find a common time to do these two actions in a short interval
16:34:02 <dcf1> well the way nginx server_name is set up on the snowflake broker is slightly weird (tpo/anti-censorship/pluggable-transports/snowflake#40430) so shelikhoo I might need to you make the config change on te server since you know how it works
16:34:52 <dcf1> I assume it is as easy as making a third copy of the config file
16:35:38 <shelikhoo> yes, just duplicate the config file is fine. I think it would be easier to just copy the certificate to the new broker, and I can compete the rest of setup
16:35:48 <dcf1> sounds good
16:36:14 <shelikhoo> oh, I have access to the old broker as well
16:36:20 <dcf1> yes
16:36:24 <shelikhoo> I can compete the entire thing!
16:36:32 <meskio> :)
16:36:44 <shelikhoo> and then request dns change to see the result
16:36:48 <shelikhoo> ^~^
16:37:33 <shelikhoo> over
16:37:34 <meskio> nice, we have a plan
16:38:22 <meskio> the last discussion topic is: covert-dtls and a test plan?
16:38:38 <theodorsm> I have fixed most of the comments
16:38:52 <dcf1> yea I see discussion and code review in https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/merge_requests/448,
16:38:59 <theodorsm> Need to rebase off main and fix some merge conflicts
16:39:15 <dcf1> but it seems like we might be getting ahead of ourselves,
16:39:27 <dcf1> it might be better to make a test release and ask some people on NTC or similar to try it
16:39:54 <dcf1> I think it would be a mistake to merge before doing that
16:40:16 <ggus> ^ nina13[m] could help w/ a post in russian on NTC
16:40:24 <theodorsm> Sounds good, testing is def. needed
16:40:27 <dcf1> a past example that cohosh did was https://archive.org/details/snowflake-ru_snowflake_fix-20211208-ae7cc478fd34
16:40:40 <dcf1> https://archive.org/details/tor-browser-11.5a13-snowflake-dtlslib-20220712-9d73998bca39
16:40:48 <meskio> I guess we could produce a snowflake client binary and give instructions on how to replace it in TB
16:41:05 <meskio> TB is not using lyrebird for snowflake yet, isn't it?
16:41:17 <dcf1> https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40014#note_2765254
16:41:19 <theodorsm> Would be valuable to test the proxy also
16:41:35 <shelikhoo> the snowflake censorship in russia, if I recall correctly, has stopped already
16:41:57 <shelikhoo> if that is true, then I don't think we could ask someone to test it anymore
16:42:00 <dcf1> shelikhoo: ah, so you're saying that any test might be inconclusive?
16:42:51 <cohosh> you mean the most recent partial blocking, right?
16:43:11 <shelikhoo> yes...
16:43:13 <dcf1> the situation I hope to avoid is that we commit to merging and then something goes wrong with applying it in practice, something that could have been revealed by earlier user testing
16:43:37 <theodorsm> I think testing would be mostly valuable from a stability point of view. Will an unsupported feature be chosen etc
16:43:51 <cohosh> we probably don't need to test it against an active dtls fingerprinting attack for this purpose
16:44:08 <shelikhoo> yes, then it is probably fine
16:44:18 <shelikhoo> we just need to make sure there is no regression
16:44:32 <meskio> theodorsm: for the proxy I run a couple of proxies I could replace them with your patches and see if traffic goes down or something, but I will not have visibility if there is a specific country affected
16:45:33 <meskio> woud be nice to include country stadistics in the prometheus exporter of the proxy
16:45:51 <meskio> (or are there already? never used them)
16:45:53 <theodorsm> meskio: Perfect! Thank you ^^ Testing should be done with both covertdtls-config set to mimic and randomize
16:46:25 <meskio> theodorsm: cool, I'll launch it in the comming days and see what happens
16:46:49 <theodorsm> That is, either mimic or randomize at a time
16:46:53 <theodorsm> :)
16:47:06 <meskio> I can do one of each :D
16:49:08 <meskio> on the client testing who can produce binaries and instructions on how to run it?
16:49:58 <dcf1> or even source code, depends on how many users you want
16:50:22 <theodorsm> I don't know how to distribute the binaries to users, but I can assist writing instructions
16:50:29 <dcf1> if there's a reliable git branch and instructions how to clone it
16:51:41 <meskio> yes maybe just instructions on how to compile it and replace it in TB will be enough, ntcparty people are fairly tech savy
16:51:48 <cohosh> theodorsm: if you write up instructions on the MR, we can give feedback and then take ggus and nina13[m] up on their offer to help translate it and post it
16:52:07 <meskio> cool, we have a plan here
16:52:12 <nina13[m]> will be happy to help
16:52:15 <theodorsm> Nice, will do
16:52:30 <dcf1> ok, thanks, that's what I was looking for, keep the big picture in sight while looking at the details of code review
16:53:00 <dcf1> https://ntc.party/t/testing-invitation-for-tor-browser-with-supportedgroups-patch-countermeasure-in-snowflake-to-evade-censorship-observed-in-russia/2837 from 2022
16:53:54 <meskio> anything more on this topic or other discussion topics?
16:54:03 <shelikhoo> eof from me
16:54:10 <cohosh> theodorsm: i can help produce a tor browser build patch and tor browser binary too
16:54:38 <theodorsm> cohosh: cool! We'll discuss it in the MR
16:54:45 <cohosh> sounds good :)
16:55:31 <meskio> nice that one of the interesting links is a paper on discovering domain fronts, very on topic today
16:55:45 <meskio> maybe something for a reading group to see if we can learn something from it?
16:56:07 <dcf1> I don't know, I found it randomly, I forget what I was looking for
16:56:21 <meskio> :D
16:56:23 <cohosh> heh good find
16:56:44 <meskio> it might not bring anything new, but still produce an interesting discussion
16:56:57 <dcf1> the second one is on the same topic I guess, because the first one cites it and says "in concurrent work" etc.
16:58:12 <meskio> nice
16:58:53 <meskio> how do you feel about discussing it january 16? the second week after the break
16:59:21 <shelikhoo> I think it is a good time
16:59:41 <shelikhoo> I think it is a good time for me
16:59:57 <onyinyang> sure
17:00:23 <meskio> dcf1 will not take it into you if is a bad paper :P
17:00:42 <meskio> I guess we are done for today
17:01:01 <meskio> #endmeeting