15:58:08 #startmeeting tor anti-censorship meeting 15:58:08 here is our meeting pad: https://pad.riseup.net/p/tor-anti-censorship-keep 15:58:08 feel free to add what you've been working on and put items on the agenda 15:58:08 Meeting started Thu Sep 22 15:58:08 2022 UTC. The chair is shelikhoo. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:58:08 Useful Commands: #action #agreed #help #info #idea #link #topic. 15:58:11 Hi everyone~ 15:59:04 hello 16:04:38 I'll start by saying something about the snowflake-01 bridge 16:04:43 Yes 16:04:44 snowflake-01 bridge running into resource limitations 16:04:44 need to increase number of tor instances at least 16:04:44 https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40173 16:04:56 It's experienced a large increase in users since roughly yesterday 16:05:09 The 4 load-balanced tor instances are at about 100% CPU 16:05:29 The usual remedy to this situation is to increase the number of instances, which is what I plan to do in #40173 16:06:02 However, be aware that the bridge is also running into other resource limitations 16:06:33 even in its current state, the bridge is now at about 70% CPU across all cores -- in other words it cannot handle twice the traffic it is handling now 16:07:07 the snowflake-server process also appears to be restarting due to out-of-memory every few hours (there's 48 GB of RAM on the server) 16:07:41 so I will do what I can today, but we are also likely to need to expand horizontally in the near future 16:08:13 even the extor-static-cookie processes are using 50% of a CPU core each, and those are only highly optimized no-op TCP pipes, basically 16:08:38 the snowflake-01 bridge is certainly one of the largest relays in the tor network at this point 16:09:17 I cannot find the page at tor metrics that ranks relays on consensus weight, and snowflake-01 would not place there both because it's a bridge and because its load balancing messes up the stats, 16:09:28 but it would be among the largest relays 16:09:35 what do you think about this proposal? https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/119 16:10:09 thanks ggus, hadn't seen that 16:10:37 we also don't have enough spare proxy to enable distributed snowflake... 16:10:39 I don't know, I'm worried about load 16:11:58 maybe don't make it the default yet 16:12:06 One way we can solve this is to enable distributed snowflake, this will actually put distributed snowflake into action 16:12:16 This step need to happen at the broker 16:12:27 and rejected proxies that have not updated 16:12:49 so that all proxies in the pool understand distributed snowflake 16:12:54 shelikhoo: and it requires new client software deployment, correct? 16:13:00 yes... 16:13:43 that's my plan for now. I'll increase the number of instances, monitor, and update #40173 with the results 16:13:55 I don't know if there's much else we can do in the short term 16:14:08 perhaps the situation will look different after a couple of days 16:15:05 I think this is more or less related to russia's conscription.... 16:15:23 and iran's protest 16:15:40 yeah good point, both happened around the same time 16:16:41 I started a thread on the Iran shutdown, please share any network results or info if you can https://github.com/net4people/bbs/issues/125 16:17:10 we're getting too many users requests in telegram. i believe because bridgedb 'settings' bridges are mostly blocked in Iran 16:17:21 but we don't have a vantage point there to test 16:17:43 so that's why i was suggesting moving snowflake as the first option for IR users 16:18:40 ggus: we may not have good options 16:19:40 after a few hours it will be closer to nighttime in the eastern hemisphere, typically usage of the bridge decreases then 16:20:03 maybe we will get into a manageable state, or maybe it will continue to increase, we'll see 16:20:47 I will try rebuilding snowflake-server with the most recent golang to try and get whatever compiler optimizations are available 16:21:37 I did spend some time in the past trying to reduce the CPU and RAM usage of snowflake-server (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40086) but did not find anything obvious to fix 16:21:38 so what's stopping us to have two snowflake broker is because we don't have enough snowflake proxies running an up to date version? 16:21:48 ggus: not 2 brokers, 2 bridges 16:21:56 1 broker is still okay 16:21:57 ah! 16:22:23 we already have written the code to support more than one bridges 16:22:35 however, it depends on proxy support it 16:22:37 2 bridges is https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/28651 and https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40129 16:23:00 If we have enough spare proxy(so we can reject old one) 16:23:02 we also have a snowflake-02 bridge already set up and ready to start handling traffic, when proxies and client begin to use it 16:23:32 we can enable the support on the broker, and let client use it 16:28:04 this would reject about 60% of the proxy 16:28:57 70% to be exact 16:29:12 that's a lot :/ 16:29:39 the absolute number may be more important than the percentage 16:29:54 is 30% of current proxy capacity enough for client demand? 16:29:55 shelikhoo: and the majority are snowflake browser add-ons? 16:30:45 https://share.riseup.net/#ulsTgJbus-wwSCiBstx6WA 16:30:55 yes 16:31:00 I don't know what to suggest. But the snowflake-01 bridge cannot be stretched much farther in the current condition. So it may be a good idea to find ways to accelerate multi-bridge deployment. 16:31:05 most of them are browser add-on 16:31:31 the browser add-on ones are good, because they will auto-upgrade, correct? 16:33:01 I found the page with fastest relays: https://metrics.torproject.org/rs.html#toprelays 16:33:32 To give a basis of comparison, snowflake-01 had an average bandwidth of 72 MB/s throughout all of August 2022 (I computed this recently) 16:33:47 wow! 16:34:10 yes, in theory. I will have a investigation about version rollout 16:34:27 202 TB of user traffic in August, not counting Tor, WebSocket, etc. overhead 16:34:39 shelikhoo: what about promoting snowflake this week and starting announcing that old clients will get disconnected on date XYZ? 16:35:48 it is old proxy will be rejected 16:35:52 not old client 16:36:10 ops! yes 16:36:15 old proxies 16:36:22 I will need to investigate the root cause of why there is so much old proxies 16:36:28 correct, the change should be fully compatible with old clients, they will just continue to be assigned to the snowflake-01 bridge, which has the fingerprint they expect 16:36:58 currently the broker is not collecting info about proxy type when it comes to distributed support 16:37:03 I can add this to it 16:37:08 and then deploy it 16:37:20 and for the new client, we will need a new tb release or connection assist would solve this? 16:37:35 to see which kind of proxy is not updating 16:38:01 once we enable support, we can release a new tb 16:38:43 ggus: good point, *would* connection assist help with this? the client gets two snowflake bridge lines, chooses them randomly? 16:39:06 I guess it would not work for old clients, because they would not be able to inform the broker of what bridge fingerprint they want, so the connection would fail 16:39:13 maybe connection assist does not help 16:39:26 aha interesting 16:39:48 connection assist can help us in the future where we can distribute clients to new snowflake bridges 16:39:53 but right not it won't work 16:40:04 but right now it won't work 16:41:00 ok 16:43:00 anything more this topic? 16:43:45 i'm done on the bridge topic 16:43:49 Gettor Telegram Bot down. Fix deployed 16:44:09 We see an increase of user reaching to Telegram Gettor bot 16:44:29 and the bot hit Telegram's rate limit 16:44:56 a fix was deployed to to reduce api call and restored it to working state 16:47:07 cool! :) 16:48:09 that is everything from me on this topic 16:48:19 anything more we wants to discuss in this meeting? 16:48:40 a quick one 16:48:57 community team wrote this guide: https://forum.torproject.net/t/iran-circumventing-censorship-with-tor/4590 16:49:06 to help IR users to bypass censorship 16:49:39 emmapeel is working on the localization 16:50:33 (the localization is almost done, im working on the translation :D ) 16:51:12 that's quick work, good job community team 16:51:13 the translation is the missing part of the localization we did with the community team 16:53:16 are you looking for feedback, signal boost once the translation is done? 16:54:38 yes :) 16:56:11 okay... anything more? 16:56:33 i'm good 16:56:37 yes... 16:56:39 #endmeeting