15:58:08 <shelikhoo> #startmeeting tor anti-censorship meeting
15:58:08 <shelikhoo> here is our meeting pad: https://pad.riseup.net/p/tor-anti-censorship-keep
15:58:08 <shelikhoo> feel free to add what you've been working on and put items on the agenda
15:58:08 <MeetBot> Meeting started Thu Sep 22 15:58:08 2022 UTC.  The chair is shelikhoo. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:58:08 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
15:58:11 <shelikhoo> Hi everyone~
15:59:04 <ggus> hello
16:04:38 <dcf1> I'll start by saying something about the snowflake-01 bridge
16:04:43 <shelikhoo> Yes
16:04:44 <shelikhoo> snowflake-01 bridge running into resource limitations
16:04:44 <shelikhoo> need to increase number of tor instances at least
16:04:44 <shelikhoo> https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40173
16:04:56 <dcf1> It's experienced a large increase in users since roughly yesterday
16:05:09 <dcf1> The 4 load-balanced tor instances are at about 100% CPU
16:05:29 <dcf1> The usual remedy to this situation is to increase the number of instances, which is what I plan to do in #40173
16:06:02 <dcf1> However, be aware that the bridge is also running into other resource limitations
16:06:33 <dcf1> even in its current state, the bridge is now at about 70% CPU across all cores -- in other words it cannot handle twice the traffic it is handling now
16:07:07 <dcf1> the snowflake-server process also appears to be restarting due to out-of-memory every few hours (there's 48 GB of RAM on the server)
16:07:41 <dcf1> so I will do what I can today, but we are also likely to need to expand horizontally in the near future
16:08:13 <dcf1> even the extor-static-cookie processes are using 50% of a CPU core each, and those are only highly optimized no-op TCP pipes, basically
16:08:38 <dcf1> the snowflake-01 bridge is certainly one of the largest relays in the tor network at this point
16:09:17 <dcf1> I cannot find the page at tor metrics that ranks relays on consensus weight, and snowflake-01 would not place there both because it's a bridge and because its load balancing messes up the stats,
16:09:28 <dcf1> but it would be among the largest relays
16:09:35 <ggus> what do you think about this proposal? https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/119
16:10:09 <dcf1> thanks ggus, hadn't seen that
16:10:37 <shelikhoo> we also don't have enough spare proxy to enable distributed snowflake...
16:10:39 <dcf1> I don't know, I'm worried about load
16:11:58 <dcf1> maybe don't make it the default yet
16:12:06 <shelikhoo> One way we can solve this is to enable distributed snowflake, this will actually put distributed snowflake into action
16:12:16 <shelikhoo> This step need to happen at the broker
16:12:27 <shelikhoo> and rejected proxies that have not updated
16:12:49 <shelikhoo> so that all proxies in the pool understand distributed snowflake
16:12:54 <dcf1> shelikhoo: and it requires new client software deployment, correct?
16:13:00 <shelikhoo> yes...
16:13:43 <dcf1> that's my plan for now. I'll increase the number of instances, monitor, and update #40173 with the results
16:13:55 <dcf1> I don't know if there's much else we can do in the short term
16:14:08 <dcf1> perhaps the situation will look different after a couple of days
16:15:05 <shelikhoo> I think this is more or less related to russia's conscription....
16:15:23 <shelikhoo> and iran's protest
16:15:40 <dcf1> yeah good point, both happened around the same time
16:16:41 <dcf1> I started a thread on the Iran shutdown, please share any network results or info if you can https://github.com/net4people/bbs/issues/125
16:17:10 <ggus> we're getting too many users requests in telegram. i believe because bridgedb 'settings' bridges are mostly blocked in Iran
16:17:21 <ggus> but we don't have a vantage point there to test
16:17:43 <ggus> so that's why i was suggesting moving snowflake as the first option for IR users
16:18:40 <dcf1> ggus: we may not have good options
16:19:40 <dcf1> after a few hours it will be closer to nighttime in the eastern hemisphere, typically usage of the bridge decreases then
16:20:03 <dcf1> maybe we will get into a manageable state, or maybe it will continue to increase, we'll see
16:20:47 <dcf1> I will try rebuilding snowflake-server with the most recent golang to try and get whatever compiler optimizations are available
16:21:37 <dcf1> I did spend some time in the past trying to reduce the CPU and RAM usage of snowflake-server (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40086) but did not find anything obvious to fix
16:21:38 <ggus> so what's stopping us to have two snowflake broker is because we don't have enough snowflake proxies running an up to date version?
16:21:48 <dcf1> ggus: not 2 brokers, 2 bridges
16:21:56 <dcf1> 1 broker is still okay
16:21:57 <ggus> ah!
16:22:23 <shelikhoo> we already have written the code to support more than one bridges
16:22:35 <shelikhoo> however, it depends on proxy support it
16:22:37 <dcf1> 2 bridges is https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/28651 and https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40129
16:23:00 <shelikhoo> If we have enough spare proxy(so we can reject old one)
16:23:02 <dcf1> we also have a snowflake-02 bridge already set up and ready to start handling traffic, when proxies and client begin to use it
16:23:32 <shelikhoo> we can enable the support on the broker, and let client use it
16:28:04 <shelikhoo> this would reject about 60% of the proxy
16:28:57 <shelikhoo> 70% to be exact
16:29:12 <ggus> that's a lot :/
16:29:39 <dcf1> the absolute number may be more important than the percentage
16:29:54 <dcf1> is 30% of current proxy capacity enough for client demand?
16:29:55 <ggus> shelikhoo: and the majority are snowflake browser add-ons?
16:30:45 <shelikhoo> https://share.riseup.net/#ulsTgJbus-wwSCiBstx6WA
16:30:55 <shelikhoo> yes
16:31:00 <dcf1> I don't know what to suggest. But the snowflake-01 bridge cannot be stretched much farther in the current condition. So it may be a good idea to find ways to accelerate multi-bridge deployment.
16:31:05 <shelikhoo> most of them are browser add-on
16:31:31 <dcf1> the browser add-on ones are good, because they will auto-upgrade, correct?
16:33:01 <dcf1> I found the page with fastest relays: https://metrics.torproject.org/rs.html#toprelays
16:33:32 <dcf1> To give a basis of comparison, snowflake-01 had an average bandwidth of 72 MB/s throughout all of August 2022 (I computed this recently)
16:33:47 <ggus> wow!
16:34:10 <shelikhoo> yes, in theory. I will have a investigation about version rollout
16:34:27 <dcf1> 202 TB of user traffic in August, not counting Tor, WebSocket, etc. overhead
16:34:39 <ggus> shelikhoo: what about promoting snowflake this week and starting announcing that old clients will get disconnected on date XYZ?
16:35:48 <shelikhoo> it is old proxy will be rejected
16:35:52 <shelikhoo> not old client
16:36:10 <ggus> ops! yes
16:36:15 <ggus> old proxies
16:36:22 <shelikhoo> I will need to investigate the root cause of why there is so much old proxies
16:36:28 <dcf1> correct, the change should be fully compatible with old clients, they will just continue to be assigned to the snowflake-01 bridge, which has the fingerprint they expect
16:36:58 <shelikhoo> currently the broker is not collecting info about proxy type when it comes to distributed support
16:37:03 <shelikhoo> I can add this to it
16:37:08 <shelikhoo> and then deploy it
16:37:20 <ggus> and for the new client, we will need a new tb release or connection assist would solve this?
16:37:35 <shelikhoo> to see which kind of proxy is not updating
16:38:01 <shelikhoo> once we enable support, we can release a new tb
16:38:43 <dcf1> ggus: good point, *would* connection assist help with this? the client gets two snowflake bridge lines, chooses them randomly?
16:39:06 <dcf1> I guess it would not work for old clients, because they would not be able to inform the broker of what bridge fingerprint they want, so the connection would fail
16:39:13 <dcf1> maybe connection assist does not help
16:39:26 <ggus> aha interesting
16:39:48 <shelikhoo> connection assist can help us in the future where we can distribute clients to new snowflake bridges
16:39:53 <shelikhoo> but right not it won't work
16:40:04 <shelikhoo> but right now it won't work
16:41:00 <ggus> ok
16:43:00 <shelikhoo> anything more this topic?
16:43:45 <dcf1> i'm done on the bridge topic
16:43:49 <shelikhoo> Gettor Telegram Bot down. Fix deployed
16:44:09 <shelikhoo> We see an increase of user reaching to Telegram Gettor bot
16:44:29 <shelikhoo> and the bot hit Telegram's rate limit
16:44:56 <shelikhoo> a fix was deployed to to reduce api call and restored it to working state
16:47:07 <ggus> cool! :)
16:48:09 <shelikhoo> that is everything from me on this topic
16:48:19 <shelikhoo> anything more we wants to discuss in this meeting?
16:48:40 <ggus> a quick one
16:48:57 <ggus> community team wrote this guide: https://forum.torproject.net/t/iran-circumventing-censorship-with-tor/4590
16:49:06 <ggus> to help IR users to bypass censorship
16:49:39 <ggus> emmapeel is working on the localization
16:50:33 <emmapeel> (the localization is almost done, im working on the translation :D )
16:51:12 <dcf1> that's quick work, good job community team
16:51:13 <emmapeel> the translation is the missing part of the localization we did with the community team
16:53:16 <dcf1> are you looking for feedback, signal boost once the translation is done?
16:54:38 <ggus> yes :)
16:56:11 <shelikhoo> okay... anything more?
16:56:33 <ggus> i'm good
16:56:37 <shelikhoo> yes...
16:56:39 <shelikhoo> #endmeeting