16:00:01 <meskio> #startmeeting tor anti-censorship meeting
16:00:01 <MeetBot> Meeting started Thu Nov 14 16:00:01 2024 UTC.  The chair is meskio. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:01 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:00:04 <meskio> hi everybody!!!
16:00:05 <shelikhoo> hi~
16:00:07 <meskio> here is our meeting pad: https://pad.riseup.net/p/r.9574e996bb9c0266213d38b91b56c469
16:00:08 <nipaton> hi
16:00:09 <meskio> ask me in private to give you the link of the pad to be able to edit it if you don't have it
16:00:11 <meskio> I'll wait few minutes for everybody to add you've been working on and put items on the agenda
16:00:30 <cohosh> hi
16:00:59 <meskio> I see everybody has updated their updates, so prepared :)
16:01:25 <meskio> should we go then with the agenda?
16:01:52 <meskio> I kept from last week the point about obfs4 blocked in russia
16:01:57 <meskio> I assume we don't have much updates there
16:02:14 <meskio> but we might have other info about how russian cersorship is happening
16:03:02 <meskio> cohosh: any news about snowflake that are worth discussing?
16:03:06 <meskio> or should we just skip this point?
16:03:34 <nipaton> does snowflake 2.5.1 add any new features?
16:04:04 <meskio> no, AFAIK is only fixing the version string, as the previous release didn't update it
16:04:44 <cohosh> i don't have any insights
16:04:44 <shelikhoo> context: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40407
16:05:13 <cohosh> i don't see any different in the dtls handshakes between proxies that appear to work and proxies that suddenly stop working
16:05:14 <meskio> nipaton: BTW is 2.10.1 the release, but what I said
16:05:34 <nipaton> ah thanks
16:05:36 <shelikhoo> yes, I think the next step could be replaying the traffic
16:05:44 <cohosh> we got some feedback that others are seeing this same behaviour where proxies stop working after transfering traffic
16:05:55 <shelikhoo> and see if the censor can be triggered reliably...
16:06:05 <cohosh> shelikhoo: yeah that's a good idea
16:06:45 <shelikhoo> we currently don't know if the censor is based on ip address or the protocol
16:06:57 <cohosh> it could be both too
16:07:03 <shelikhoo> yes...
16:07:31 <cohosh> it's harder to diagnose this with snowflake than obfs4 because we can't easily control the proxy, we could try a small test network deployment
16:07:45 <cohosh> if we notice the same blocking with that, it seems more likely it's protocol-based
16:08:20 <cohosh> and it would allow us to see the proxy side of the connection too
16:08:24 <shelikhoo> yes...
16:08:33 <meskio> sounds like a good plan
16:09:31 <cohosh> ok i'll work on setting that up
16:10:08 <meskio> thanks for the relesearch work there cohosh, I guess we can move to the next topic
16:10:18 <shelikhoo> thanks! let me know if there is anything I could help with
16:10:18 <dcf1> We have an old patch to pad the traffic fingerprint at the beginning of the data channel, https://github.com/net4people/bbs/issues/255#issuecomment-1566227484
16:10:33 <dcf1> Related to https://www.bamsoftware.com/papers/snowflake/#p104
16:11:02 <dcf1> Traffic analysis fingerprinting would be consistent with a proxy that works at first and then stops.
16:11:41 <shelikhoo> yes, kcp like to send small packets of certain size
16:11:42 <cohosh> dcf1: good call, let's try that from the vantage point
16:12:07 <cohosh> (also shoutout to shell for the really nice log collector setup, that's been so useful)
16:12:22 <meskio> <3
16:12:24 <shelikhoo> hahaha! no problem ^~^
16:13:21 <shelikhoo> eof
16:14:21 <meskio> the next topic is about rdsys version
16:14:27 <meskio> rdsys is still on a 0.x version
16:14:43 <meskio> but we do relay on it for all our bridge distribution and it has already replaced BridgeDB
16:14:48 <meskio> I think is time to call it 1.0
16:15:00 <meskio> even if whatever we tag with 1.0 will not bring anything big new
16:15:05 <dcf1> approve
16:15:12 <meskio> I don't like projects being for ever in a 0.x
16:15:19 * meskio looks at c-tor...
16:15:25 <dcf1> https://0ver.org/#notable-zerover-projects
16:15:31 <shelikhoo> looks good to me
16:16:01 <meskio> cool, I'll tag next release as 1.0 then
16:16:03 <shelikhoo> although I do wish to comment that snowflake 1.x have added a lot of overhead when adding new features
16:16:04 <cohosh> nice, congrats \o/
16:16:08 <shelikhoo> congrats!
16:16:12 <dcf1> Tor Stars: 4,375 First released: 2004 Releases: 515 Current version: 0.4.8.1-alpha (2023) 0ver years: 20.4
16:16:37 <meskio> shelikhoo: the good thing with rdsys is that we are the only consumers, so breaking changes don't require much coordination :)
16:16:49 <cohosh> yeah i regretted tagging snowflake 1.0 before thinking more about the API
16:17:19 <cohosh> not so much that we called it, but more that i didn't think it through well
16:17:44 <shelikhoo> meskio: yes ... so we can upgrade its version without too much issue...
16:17:45 <dcf1> it's fine, the snowflake api has so few consumers, I don't think we should be shy about incrementing the version number
16:17:52 <dcf1> snowflake 23.4 would not bother me
16:18:04 <meskio> I think we should not be that affraid to call things 1.0 once we are relaying on them golang makes things akward with version >1, but still let's increase versions
16:18:34 <shelikhoo> cohosh: there is never a perfect 1.0 in my opinion...
16:18:34 <meskio> look at firefox or chromium, they are in the verison 100somthing
16:18:37 <cohosh> that's fair
16:18:38 <dcf1> yeah once you're past v2, I admit that is awkward with golang
16:19:14 <cohosh> v100: goals
16:19:20 <meskio> :D
16:19:34 <shelikhoo> ooooo!
16:19:39 <dcf1> 💯
16:20:23 <shelikhoo> eof
16:20:26 <meskio> let's move to the next topic then: "future of the snowflake broker"
16:20:27 <WofWca[m]> Yes, some things are really hard to implement 100% backwards-compatibl-y, especially given how many functions are exposed.
16:21:03 <meskio> we might start being charged by elps.is for the broker VM in January
16:21:22 <meskio> so maybe we should start looking into options
16:21:34 <meskio> either to pay for that VM or to move it somewhere
16:22:04 <shelikhoo> yes... either there isn't enough API into internals. or there API will need to break to get new things
16:22:08 <meskio> an option could be to move it into TPA (the sysadmin team at Tor) infraestructure, but I'm not sure what will be our policy about that
16:22:22 <shelikhoo> I think the broker is not so hard to move around
16:22:39 <shelikhoo> but the nat type testing (probetest) is quite heavy
16:22:56 <shelikhoo> when it comes to setup
16:23:19 <dcf1> because of all the special nftables setup?
16:23:48 <shelikhoo> yes...
16:23:58 <onyinyang> o/ sorry, got confused by the time change >.<
16:24:18 <shelikhoo> the firewall and network namespace setup is rather complex
16:24:23 <shelikhoo> and is hard to be automated
16:24:34 <meskio> onyinyang: that happens, is this fun time of the year
16:25:30 <cohosh> we could also put the probetest separate from the broker
16:26:23 <meskio> shelikhoo: I see, probetest in TPA infra will require extra work from the TPA team to setup the firewall and network as we don't have root access
16:26:36 <meskio> should be doable but we'll need to check with them
16:26:44 <shelikhoo> yes.., in theory the nat type can also be implemented in user mode if absolutely necessary
16:28:13 <meskio> I haven't being around long enough, is the broker outside TPA because we wanted it not to be run by TPI? or just because it was easier for dcf1 or whoever set it up?
16:29:15 <meskio> TPO doesn't run bridges or relays, but we do run rdsys...
16:29:43 * meskio not sure if call it TPI or TPO here :P
16:30:45 <dcf1> yes, it's probably because I needed to run it and trying to work through layers of management was difficult. Don't forget the snowflake broker predates Tor's heavy involvement in snowflake.
16:31:06 <dcf1> I probably had it running on a VPS somewhere, until I heard about Eclipsis from someone at a conference.
16:31:43 <meskio> makes sense
16:32:45 <dcf1> separating out probetest makes some sense, in that only proxies and not clients contact the probetest server (though there is nothing permanent about that necessarily, it's a feature of the current design)
16:33:45 <dcf1> so a hostile or compromised probetest server could not trivially get a list of client IP addresses
16:33:57 <dcf1> it's a little different than running the full broker
16:34:04 <shelikhoo_> at least it would allow us to move the broker to TPA managed machine, and thus only run probetest to be run a separate machine with less spec
16:34:13 <shelikhoo_> at least it would allow us to move the broker to TPA managed machine, and thus only run probetest to be run a separate machine with less spec
16:34:51 <dcf1> no, I'm saying the opposite: run probetest on a TPA machine, run the broker outside, if the concern is with TPA having too much direct control over the network
16:35:55 <meskio> what requires more CPU/memory? probetest or the broker?
16:36:40 <shelikhoow> okay, I was thinking broker will not need root to run
16:36:50 <shelikhoow> but what dcf1 said makes sense
16:37:47 <cohosh> yeah from a "we don't run the network" perspective, it makes sense for TPA to run a volunteer proxy service rather than a client-facing service that will leak when a client is connecting to the network
16:38:25 <shelikhoow> the broker would require more resource
16:38:28 <dcf1> probetest tends to get stuck at 100% CPU sometimes, though I suspect that is a bug. I'm not sure if it still occurs. tpo/anti-censorship/pluggable-transports/snowflake#40039
16:38:42 <shelikhoow> according to top command
16:39:10 <dcf1> We have had to upgrade the broker VM because of memory limitations a few times, e.g. tpo/anti-censorship/pluggable-transports/snowflake#40202 tpo/anti-censorship/pluggable-transports/snowflake#40195
16:40:11 <dcf1> #40202 is when we last upgraded to the current configuration, 32 GB RAM and 8 CPU cores. I'm not sure if we need as much as that, back then we decided to just go with the highest tier available and avoid more incremental upgrades.
16:40:49 <meskio> so removing probetest doesn't aliviate much the resources we need for the broker
16:41:01 <meskio> but anyway might make sense to separate it
16:41:48 <shelikhoow> be aware it takes a proxy side upgrade to separate them...
16:42:01 <shelikhoow> which takes a lots of times...
16:42:29 <meskio> ohhh, they are both tight to the same domain name?
16:42:32 <dcf1> oh really? we should decouple the domain names for probetest and the broker right away, then, to make a future separation easier.
16:42:43 <meskio> +1
16:42:48 <cohosh> good call
16:43:39 <meskio> ok, so we are not separating the services soon
16:43:40 <shelikhoow> https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/main/proxy/lib/snowflake.go?ref_type=heads#L60
16:44:00 <shelikhoow> they share the snowflake-broker.torproject.net domain name
16:45:10 <meskio> shelikhoow: as you are doing the current deployment, do you want to take care of separating them?
16:47:26 <shelikhoo> yes, I can get another domain name as an alias of current one
16:47:40 <meskio> sounds good, thank you
16:47:54 <cohosh> shelikhoo: or a another subdomain from TPA
16:48:12 <shelikhoo> https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40349#note_3129780
16:48:17 <meskio> I guess it should be snowflake-probetest.torproject.net
16:48:21 <shelikhoo> yes, it will be another sub domain from TPA
16:48:22 <meskio> or something like that
16:48:28 <shelikhoo> yes...
16:48:47 <meskio> nice
16:49:01 <shelikhoo> I will copy the certificate and start transfer broker next Monday...
16:49:26 <shelikhoo> any blocker we known of ?
16:50:07 <meskio> it looks ready
16:51:51 <shelikhoo> yes, I will run the transfer next monday
16:52:15 <shelikhoo> eof
16:52:33 <meskio> going back to the original question, is still unclear if we'll need to pay, but seeing that we might prefer not to have the broker in TPA infra I'll start looking for funding options for it
16:52:57 <meskio> we could find cheaper options than elips.is
16:54:55 <meskio> I guess we can move to the last discussion topic: "WebTunnel Server utility command interface"
16:54:59 <meskio> shelikhoo: ?
16:55:05 <ggus> when do you need to move from elip.is?
16:55:23 <shelikhoo> yes
16:55:44 <shelikhoo> so the webtunnel will need to add a command to generate certificate hash
16:55:50 <meskio> ggus: January, maybe
16:56:18 <shelikhoo> so that the bridge operator can use it to generate bridge line
16:56:25 <shelikhoo> in V2Ray the command is like
16:56:26 <shelikhoo> ./v2ray tls certChainHash --cert '****.pem'
16:56:34 <meskio> shelikhoo: there is no standard tooling for that in openssl?
16:56:37 <ggus> meskio: we could ask tor-relays@ if someone is interesting on providing this service. i heard that relay ops are very generous :)
16:56:45 <ggus> *interested
16:57:16 <meskio> ggus: we need a trusted entity as this service sees all snowflake clients, but that could be a nice idea
16:57:20 <shelikhoo> meskio: I don't think there is such an standard tooling in openssl, as this hash is a combined hash of the entire chain not just leaf certificate
16:57:56 <meskio> why do we need the whole chain? we only need to validate the leaf, isn't it?
16:58:47 <shelikhoo> the entire chain is verified because the censor can MITM the connection and send an modified chain and observe client behaviour
16:59:17 <shelikhoo> the behavior of a real client is that it would need the chain back to the root certificate to verify the leaf certificate
16:59:30 <shelikhoo> (some browser can cache some certificate)
16:59:44 <meskio> I see, if webtunnel skips verifying the rest of the chain the censor will notice is a webtunnel client using only the leaf to verify the connection
16:59:50 <shelikhoo> yes..
17:00:23 <shelikhoo> and the encrypted alert sometime send out why it aborted the connection
17:01:06 <shelikhoo> so the purposed cli is to just use arg[0] as command name
17:01:26 <shelikhoo> in real usage, it would be the link name of binary
17:01:52 <shelikhoo> if it is a known command name. then the specific command procedure will be run
17:02:14 <shelikhoo> like ln webtunnel tlschainhash
17:02:25 <shelikhoo> ./tlschainhash < xxx.pem
17:02:53 <shelikhoo> if that works we can go ahead with this pattern
17:04:14 <meskio> I find this hacky, I think is better to have a -gen-cert-fp or something, but I don't have a strong opinion here
17:04:40 <shelikhoo> yes... -gen-cert-fp would also works for sure...
17:05:06 <shelikhoo> we can go with -gen-cert-fp
17:05:12 <shelikhoo> eof
17:06:07 <meskio> I like more that option, as it will be less confusing and error prone for people that compiles the binary themselves
17:06:38 <meskio> but you are the one putting more brain into webtunnel, if you prefer the other option I will not opose it
17:07:27 <shelikhoo> I think we can go with the option that more people would understood... let's go with  -gen-cert-fp
17:07:32 <meskio> :)
17:07:55 <meskio> I think we are done with the discussions, and a bit over time
17:08:09 <meskio> there is an interesting link about tor families
17:08:40 <meskio> but let's close this meeting, happy to discuss it if you want in #tor-anticensorship
17:08:43 <meskio> #endmeeting