16:00:07 <onyinyang> #startmeeting tor anti-censorship meeting
16:00:07 <MeetBot> Meeting started Thu Mar 13 16:00:07 2025 UTC.  The chair is onyinyang. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:07 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:00:07 <onyinyang> hello everyone!
16:00:07 <onyinyang> here is our meeting pad: [https://pad.riseup.net/p/r.9574e996bb9c0266213d38b91b56c469](https://pad.riseup.net/p/r.9574e996bb9c0266213d38b91b56c469)
16:00:28 <cohosh> hi
16:00:35 <meskio> hello
16:01:26 <meskio> BTW, the agenda items in the pad are not from last week, they are new
16:01:43 <shelikhoo> hi~
16:04:09 <WofWca[mds]> Hi!
16:04:13 <onyinyang> ah, apologies
16:04:21 <meskio> no prob :)
16:04:33 <onyinyang> it didn't seem like anyone had updated the pad yet so I didn't check the notes from last week
16:04:43 <onyinyang> or the time stamp >.<
16:05:40 <meskio> I did add the snowflake dependencies a couple o days ago
16:05:56 <meskio> when I have a topic for the next weeks meeting I add it so I don't forget
16:06:30 <onyinyang> makes sense
16:06:52 <onyinyang> I think it's been long enough for people to update the pad so it looks like we only have those 2 items to discuss
16:07:02 <onyinyang> The first: snowflake dependencies need go 1.23
16:07:09 <meskio> that one is me
16:07:28 <meskio> some snowflake dependencies require go 1.23
16:07:37 <meskio> do we want to bump snowflake go version to 1.23?
16:08:02 <meskio> we used to look into what is in debian to get a feeling on what is easy for users to use
16:08:07 <shelikhoo__> Personally I think this isn't a big issue...
16:08:09 <shelikhoo__> https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/41387
16:08:35 <shelikhoo__> the tor browser seems already build with version
16:08:47 <meskio> yes, I think they have that in tor-browser
16:08:54 <shelikhoo__> and have the command to fix the build issue
16:08:57 <shelikhoo__> yes
16:08:58 <meskio> and is also in debian backports
16:09:07 <meskio> cool, I'll update it then
16:09:08 <cohosh> should we check with orbot/iptproxy?
16:09:26 <meskio> cohosh: good question, I can do that
16:09:48 <meskio> I'll ask them and if they are ok with it I'll update it
16:10:08 <shelikhoow> sound nice to me
16:10:36 <morganava> meskio: we're staying on (iirc) 1.21 on  macOS until ESR 140 iirc
16:10:39 <morganava> PieroV^
16:11:52 <onyinyang> anything more on this topic?
16:11:58 <meskio> morganava: ohh, I see
16:12:08 <shelikhoo> yes, I think in that case we can wait until we are ready to move
16:12:11 <meskio> so you do need snowflake to keep compiling with 1.21?
16:12:25 <shelikhoo> I don't think there is anything important updates from dependencies
16:12:31 <meskio> we can hold the dependencies then
16:12:46 <meskio> morganava: is there an ETA for ESR 140?
16:12:57 <shelikhoo> https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/merge_requests/536
16:13:45 <shelikhoo> I don't think we are impacted by this
16:13:51 <meskio> :)
16:15:02 <shelikhoo> https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/merge_requests/527
16:15:06 <shelikhoo> or this one
16:15:24 <shelikhoo> both of them are security patch requires a go version update
16:16:43 <shelikhoo> but these security update are not related to what we are currently using, even if we should update them when we can
16:17:41 <meskio> I see firefox ESR 140 is due on June this year, we can hold those changes until then
16:19:31 <meskio> I think we are done with this topic
16:19:56 <shelikhoo> yes let's wait until then unless there is an important update
16:22:50 <onyinyang> ok, sorry for jumping the gun earlier >.<
16:22:55 <meskio> :)
16:23:01 <onyinyang> the next topic is: should we change default relay pattern policy on snowflake broker
16:23:06 <cohosh> this can be quick, i mostly want to check that my understanding is correct
16:23:32 <cohosh> when we implemented support for multiple snowflake bridges, we added a default relay pattern that would be applied to proxies who had not updated
16:24:06 <cohosh> once we added the second bridge, this default pattern because out of date with what would be accepted
16:24:12 <morganava> meskio: September
16:24:23 <cohosh> so we're effectively rejecting proxies that don't advertise a relay pattern
16:24:37 <dcf1> wow great investigation
16:24:56 <cohosh> the question is whether to 1) keep things the way they are but remove the unecessary log messages and change the metrics to be more clear about what's happening
16:24:57 <dcf1> that has been broken for quite a long time then
16:25:08 <cohosh> 2) change the default pattern to something that works
16:25:35 <cohosh> i wasn't sure if this was intentional to avoid surprising relay operators with a new destination
16:26:19 <dcf1> it was always the intention that when a proxy does not send a relay pattern, it in effect is declaring its ability to only connect to the snowflake-01 relay
16:27:09 <cohosh> ok, so that's working as intended then
16:27:46 <shelikhoo> as I have previously, there was 2 conflicting design goal of default value: no regression = we wants default value to initiate the behaviour before this setting was introduced
16:27:49 <cohosh> was it our intention to do some smarter matching? or should we keep rejecting proxies that don't support a superset of the broker supported relays?
16:27:54 <dcf1> proxies that do not send a relay pattern also do not know how to interpret the broker's instruction to connect to a specific relay. they have snowflake.torproject.net hardcoded.
16:28:03 <cohosh> ah that's right
16:28:11 <dcf1> it's not about not surprising them with a different destination, it's that they are incapable of any other destination
16:28:19 <shelikhoo> "best value" =  the developer set a best default value in their eye, ending with a default value just works
16:28:32 <dcf1> shelikhoo: I don't understand what that means.
16:29:32 <dcf1> is there other code in the broker that ensures that proxies that do not send a relay pattern will not be assigned a client that requests a relay other than snowflake-01? I believe that was how it was supposed to work.
16:29:51 <cohosh> no we do a simpler thing of rejecting proxies that don't support all relays
16:30:01 <dcf1> if so, then fixing the default relay pattern to pass CheckProxyRelayPattern should work
16:30:05 <shelikhoo> no regression means if a software is have certain behaviour before the update, it should stay the same behaviour after the update if the input was same
16:30:57 <WofWca[mds]> I'm pretty sure that if the broker accepts a proxy poll, then it simply adds it to the pool, which is then just popped on client requests.
16:31:07 <cohosh> that's right
16:31:17 <dcf1> whatever the intention may have been, the outcome is clearly not desirable, if it means there are thousands of old proxies out there that are silently failing, and a never ending stream of useless log messages from those proxies at the broker
16:31:18 <shelikhoo> so the user can feel comfortable in automating their upgrade without the need to worry about update will break things
16:31:24 <cohosh> the only check we do is to make sure proxies support all broker-supported relays
16:31:38 <dcf1> shelikhoo: who's the "user" here? is it the proxy operator?
16:31:40 <shelikhoo> I think it is long enough for us to just update the default value
16:32:07 <cohosh> so to move forward we can add some more logic to matchSnowflake, and/or we can update the metrics and log messages to be more useful
16:32:10 <shelikhoo> yes, the proxy operator
16:32:51 <cohosh> shelikhoo: see dcf1's point above, updating the default pattern won't work
16:33:07 <dcf1> i wish we had a feature where the broker could send a response that would cause a proxy to log an error message and disable itself. it's not too late to add one, to make things like this less of a problem in the future.
16:33:10 <cohosh> because then we will be assigning clients bound for the second bridge to proxies that can only connect to the first
16:33:30 <cohosh> dcf1: yeah that's a good idea
16:33:52 <dcf1> yeah, to relfect what is really happening, the broker should be logging a "rejecting old proxy" message, not some oblique thing about a relay pattern not matching.
16:34:19 <shelikhoo> I think updating the default value on snowflake-proxy can fix the case where the client is only updating the binary but not the command line
16:34:22 <cohosh> i will start with modifying our metrics and logging first and then we can discuss how complicated we want to make matchSnowflake later
16:34:35 <WofWca[mds]> Just changing the presumed pattern would make the old proxies work 50% of the time IMU
16:34:50 <shelikhoo> and there is a difference between client that didn't update their binary
16:35:02 <shelikhoo> and client that update the binary but not their command line
16:35:20 <cohosh> WofWca[mds]: the problem is if a client included a fingerprint in their bridge line, they will get scary tor errors rather than simply a snowflake connection error
16:35:28 <dcf1> it's fine if we're rejecting old proxies as a matter of simplifying the implementation, it is just less than ideal that the way the rejection happens is through this oblique and hard-to-debug edge case and not a straightforward code check, and it's bad that the proxies that are being rejecting keep on merrily and uselessly polling
16:35:33 <shelikhoo> and there is a difference between proxy that didn't update their binary
16:35:45 <shelikhoo> and proxy that update the binary but not their command line
16:35:52 <WofWca[mds]> cohosh: Good point
16:36:20 <dcf1> yeah the error with a fingerprint mismatch is a hard failure, goes into the circumvention settings failure branch, needs button clicks iirc
16:36:24 <shelikhoo> if a proxy didn't update their binary, then it the proxy will not send the allow relay pattern
16:36:27 <cohosh> shelikhoo: ah i see your point
16:37:09 <shelikhoo> if a proxy's binary is updated but its command line remain the same, then it will send a non-working allow relay pattern
16:37:23 <cohosh> ok we could do something like: add a version check to reject old proxies, then update the default value
16:37:26 <shelikhoo> we can fix the second case by updating the default allowed relay pattern
16:37:49 <shelikhoo> and the first problem we can silent the error message on broker side
16:38:09 <cohosh> so proxies are being rejected by version, not based on whether they advertize a relay pattern
16:38:11 <shelikhoo> cohosh: I have such a thing written for packet transport mode
16:38:41 <cohosh> ok this is my current plan:
16:38:49 <cohosh> - update the broker metrics/logging
16:38:57 <shelikhoo> https://gitlab.torproject.org/shelikhoo/snowflake/-/commit/6ec0025e93c45c59862f7644c48fa9d719ff8c57
16:39:20 <cohosh> - write something that could reject old proxies on version + add dcf1's suggestion to disable proxies if they get rejected
16:40:08 <cohosh> and then we can see how things go
16:40:13 <dcf1> I don't understand the "reject old proxies on version". I think rejecting proxies that don't send a relay pattern is fine. It's just that the code path should be:
16:40:20 <dcf1> proxy doesn't send relay pattern -> broker rejects
16:40:21 <dcf1> not
16:40:32 <cohosh> yeah i also think it's ok to do that
16:40:38 <dcf1> proxy doesn't send relay pattern -> broker assigns default pattern that is guaranteed to fail -> broker rejects
16:40:41 <cohosh> it's not a large number of proxies that we'd lose in any case
16:40:45 <cohosh> we have plenty of capacity
16:41:29 <cohosh> i think the one advantage of allowing proxies to communicate that they want the default is they don't need to update their command line if we change the snowflake bridges such that their existing pattern won't work
16:42:08 <cohosh> and there's no "just sign me up for the default list" option now
16:42:11 <dcf1> I don't really udnerstand the point about command lines either
16:42:28 <shelikhoo> the broker assigns default pattern is only used for the transition period
16:42:31 <cohosh> standalone proxies have to indicate their relay pattern as an argument
16:42:39 <dcf1> For one thing, I would expect that approximatly 0% of proxy operators use the -allowed-relay-hostname-pattern option
16:42:54 <cohosh> oh i'm mistaken
16:42:56 <shelikhoo> so that the old proxy and new updated proxy will all works for sending traffics to the snowflake01 bridge
16:43:01 <dcf1> and if they did, then they presumably know what they're doing and if it stops working it's not our problem
16:43:20 <cohosh> okay i understand now
16:43:47 <dcf1> There's a default for -allowed-relay-hostname-pattern, but it doesn't have the "^", so it should be fine "snowflake.torproject.net$". It's true there's no way to express "just give me the default" but that's already pretty close to the same meaning.
16:43:49 <shelikhoo> but right now this is unnecessarily and become a legacy logic
16:44:03 <cohosh> so my modified plan is to just update metrics/logging and then we can work on this disabling feature so we have it for the future
16:44:57 <dcf1> shelikhoo: "the old proxy and new updated proxy will all works for sending traffics to the snowflake01 bridge" But if I understand correctly, that never worked, not even for a short time. Am I correct?
16:45:28 <dcf1> old proxies are currently being completely disused, and have been since the switch to multi-bridge. or do I understand incorrectly?
16:45:30 <shelikhoo> dcf1: it worked when we have updated the proxy code, but didn't added new server yet
16:45:43 <shelikhoo> and right now it is no longer useful at all
16:45:49 <shelikhoo> we can remove it
16:46:01 <dcf1> ok
16:46:26 <dcf1> that's starting to make sense, I am perhaps starting to see how this situation could have arisen
16:46:51 <cohosh> i'd like to remove the default pattern option at the broker while i'm in there
16:46:59 <cohosh> to simplify the code and prevent misunderstandings in the future
16:47:10 <dcf1> yes, in addition to the metrics/logging, things like the "if nonSupported { pattern = ctx.presumedPatternForLegacyClient }" can be deleted.
16:47:35 <cohosh> cool
16:47:46 <shelikhoo> yes.., just reject the proxy that don't send allowed relay pattern
16:47:51 <dcf1> maybe there's a code path in the legacy proxy that the broker can invoke by sending some kind of response that will cause the proxy to stop working and display a message
16:47:54 <shelikhoo> we no longer need it anymore
16:48:00 <dcf1> even if we don't have such a feature implemented explicitly
16:48:25 <cohosh> dcf1: yeah that's a good idea. i'll take a look, but i'm a little doubtful
16:48:43 <cohosh> i think the proxy poll loop is too resilient to broker responses
16:48:59 <cohosh> maybe we can make the proxy crash lol
16:49:02 <shelikhoo> the core issue is that we don't have a way to auto update snowflake standalone proxy
16:49:06 <dcf1> yes that's what I mean
16:49:15 <shelikhoo> there was some watchtower example
16:49:21 <dcf1> maybe sending a very long response body, or something, I know there's an error check for that at least
16:49:45 <cohosh> yep or there could be a marshalling panic in the response by not sending valid json
16:49:45 <shelikhoo> but in most case human interaction is needed to deal with it
16:50:05 <dcf1> shelikhoo: I disagree about that, I think it's more important that there should be a signal that (1) stops old versions from uselessly polling, and (2) informs operators to update
16:50:39 <dcf1> human interaction is fine, let the proxy operators have agency and control, just let them be informed and not be not helping when they think they are helping
16:51:07 <shelikhoow> dcf1: I am not against these points, I just feel the browser extension's auto update is working really well
16:51:09 <cohosh> and for the future we can modify the proxy code to handle this better
16:51:22 <shelikhoow> we should have a way to let proxy stop polling
16:51:35 <shelikhoow> or be able to say: "poll again in 24 hours"
16:51:46 <shelikhoow> instead of "poll again in 5 second"
16:52:07 <dcf1> it's a conversation for another day, but a homebrew auto-update mechanism esposes a lot of new attack surface, it's a potential way someone could take control over snowflake proxies and use them as an attack vector
16:52:20 <dcf1> shelikhoow: that too, there is an issue for that in fact
16:52:39 <shelikhoow> as for sending a message to the operator, I really don't think there is a way for us to do this in a universal way to send the proxy operator a message
16:52:39 <cohosh> all the way back from trac :)
16:52:56 <dcf1> tpo/anti-censorship/pluggable-transports/snowflake#25598
16:53:51 <cohosh> shelikhoow: what i'm thinking is have the broker send a 400 bad request and in this case let the standalone proxy exit the poll loop
16:53:57 <cohosh> which ofc is only good for the future
16:54:19 <dcf1> shelikhoow: there is not now a way for us to send a message to the proxy operator, that is my point. we could implement a feature where, if the broker response contains some key, then the proxy logs a message and disables itself for 24h or something. it won't work for existing legacy proxies, but it could help avoid this kind of situation in the future.
16:54:33 <shelikhoow> cohosh: yes, I think we should not let the proxy exit the poll loop, as when running inside a container with restart=always
16:54:41 <cohosh> oh
16:54:42 <shelikhoow> it will just restart if the process quit
16:54:59 <cohosh> i was missing that bit
16:55:09 <shelikhoow> so we should maybe just exit the poll loop; print a message; and don't quit
16:55:26 <cohosh> ok so crashing proxies might not work
16:55:42 <shelikhoow> sometime it will work when a restart policy is set
16:55:48 <dcf1> hm, good point, a container restart could even result in *faster* effective polling, because the poll interval is not enforced on the first poll I think
16:55:55 <shelikhoow> but it is not really universal...
16:56:24 <shelikhoow> dcf1: I think your idea would work, just print a message and wait 24 hours
16:56:26 <cohosh> ok we're getting into implementation details, i think this feature is a good idea and worth implementing
16:56:29 <dcf1> we could always just try it: the proxies that restart will restart themselves, the proxies that don't restart will be disabled
16:56:31 <shelikhoow> then print anothe message
16:57:16 <shelikhoow> the proxy couldn't know if it is just restarted or being run the first time
16:57:23 <shelikhoow> because the snowflake is stateless
16:57:47 <shelikhoow> snowflake proxy currently does not have a place to store its state
16:58:28 <dcf1> yeah what I mean is that, suppose we do find a way to crash old proxies, we could do that for 1 hour, and flush out all the old proxies, except for those that auto-restart
16:58:44 <dcf1> ok I'll try to summarize this discussion on the meeting pad
16:58:47 <cohosh> i'm not opposed to that
16:58:48 <shelikhoo> oh! yes!
16:58:54 <shelikhoo> I think that is a nice idea
16:58:55 <cohosh> thanks dcf1
16:59:01 <shelikhoo> thanks dcf1
16:59:10 <cohosh> ok that's it on this discussion point
16:59:20 <onyinyang> cool
16:59:39 <onyinyang> it seems that is it for this week
16:59:58 <onyinyang> and we're at time anyway so I will end the meeting
17:00:06 <onyinyang> thanks for the discussion today!
17:00:07 <onyinyang> #endmeeting