15:58:00 <shelikhoo> #startmeeting tor anti-censorship meeting
15:58:00 <shelikhoo> here is our meeting pad: https://pad.riseup.net/p/tor-anti-censorship-keep
15:58:00 <shelikhoo> feel free to add what you've been working on and put items on the agenda
15:58:00 <MeetBot> Meeting started Thu Jan 12 15:58:00 2023 UTC.  The chair is shelikhoo. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:58:00 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
15:58:07 <meskio> hello everybody
15:58:11 <shelikhoo> hi~
15:59:06 <hackerncoder> hi
16:00:23 <cece[m]> hi
16:00:55 <meskio> back from the long holidays, it was nice to have some days AFK :)
16:01:20 <itchyonion> hello
16:02:26 <ggus> hi
16:02:39 <shelikhoo> okay, I think we can begin today's announcement part
16:02:40 <shelikhoo> Open Collective funding for Snowflake bridge operations is open
16:02:40 <shelikhoo> https://opencollective.com/censorship-circumvention/projects/snowflake-daily-operations
16:02:40 <shelikhoo> First update post: https://opencollective.com/censorship-circumvention/projects/snowflake-daily-operations/updates/2022-year-in-review
16:03:06 <dcf1> This is a newly set up fundraising platform for keeping the snowflake-01 bridge operational
16:03:37 <dcf1> You may recall that one idea that was considered around April 2022 was crowdfunding (e.g. gofundme) to help pay for the bridge
16:04:08 <dcf1> that was when ln5 found us a server and hosting and we got a 6-month OTF rapid response grant to pay for it
16:04:20 <meskio> nice
16:04:38 <shelikhoo> great!
16:04:40 <dcf1> in the meantime, we were working on setting up a donation system to keep it sustainable in the longer term, and this Open Collective is what we arrived at after investigation
16:05:19 <itchyonion> 👍
16:05:35 <meskio> almost $7k already, pretty cool
16:05:40 <dcf1> The "2022 year in review" post has the history so far, and a graph of users
16:06:19 <dcf1> with a rather sharp escalation in the past 2 months: we're now at the level we were at on September 22, 2022
16:06:39 <dcf1> but now we're more able to handle that level of usage, after performance work
16:07:23 <dcf1> It is a shocking fact but true, that around 2-3% of all Tor users are using snowflake. Now 2-3% of pluggable transport users, 2-3% of *all* users.
16:07:31 <dcf1> *Not
16:08:12 <meskio> wow, that's a lot for a single server
16:08:47 <cece[m]> interesting
16:08:49 <dcf1> So if you encounter someone who wants to donate to support operations, this is the place
16:09:16 <cece[m]> definitely
16:09:43 <dcf1> The "Project" is "Daily Snowflake Operations"; above that is a "Collective" called "Providing for Censorship Circumvention" that is meant to be more general but is not scoped out yet https://opencollective.com/censorship-circumvention
16:09:44 <meskio> will the same site will be used for snowflake-02? or they will be funded separatelly?
16:10:31 <dcf1> in case we get more money than we can reasonably spend on snowflake servers, and want to spend in other useful ways
16:10:37 <dcf1> snowflake-02 is currently separate
16:11:17 <meskio> ok
16:11:18 <ggus> a follow up question: the goal of 11k/y is to cover all the costs of the current snowflake bridge or it's also cover some bridge performance improvement (hardware/bandwidth/etc)?
16:12:25 <dcf1> ln5 would be able to answer more precisely, but as I understand it the main costs are bandwidth, manual or emergency maintenance (i.e. physical visit to the data center), and hardware depreciation/replacement
16:13:29 <ggus> ok
16:13:33 <shelikhoo> have we ever encountered the need to visit the data center?
16:13:57 <shelikhoo> just curious...
16:13:59 <dcf1> yes, for example when the uplink was changed from 1 Gbps to 10 Gbps
16:14:27 <dcf1> also for a RAM upgrade that ln5 did
16:14:28 <ggus> dcf1: when can we promote the open collective link?
16:14:33 <shelikhoo> yes...
16:15:29 <dcf1> ggus: I think it is okay to share; ln5 sent it to comms@ a couple weeks ago
16:16:38 <ggus> i will add to the comms team weekly agenda.
16:16:50 <shelikhoo> anything more we would like to discuss on this topic?
16:16:58 <dcf1> that's all from me
16:17:19 <shelikhoo> okay, now is the discussion part
16:17:20 <shelikhoo> Enable snowflake-02 in Orbot
16:17:20 <shelikhoo> snowflake-02 (enabled in Tor Browser only) currently gets only about 5% the traffic of snowflake-01
16:17:20 <shelikhoo> snowflake-01 reaching its CPU limit
16:18:07 <shelikhoo> so snowflake-02 is already enabled in Tor Browser's stable channel?
16:18:16 <dcf1> snowflake-01 is currently hitting over 500 MB/s outgoing at its peak each day
16:18:27 <dcf1> https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40246
16:18:53 <dcf1> and at those times it's at 80-90% CPU
16:19:28 <dcf1> I'll give you a spot sample of the current bandwidth on -01 and -02...
16:19:49 <dcf1> snowflake-01: 445 MiB/s
16:19:59 <dcf1> snowflake-02: 21 MiB/2
16:20:05 <dcf1> *MiB/s
16:20:18 <dcf1> So yeah, snowflake-02 is currently about 5%
16:20:31 <dcf1> snowflake-02 is enabled for stable, yes, since TB 12.0
16:20:58 <dcf1> Kind of surprising, but believable, that 90%+ of snowflake users would be on Orbot and not Tor Browser
16:21:22 <ggus> probably from iran?
16:21:24 <shelikhoo> I think it is fine for us to ask orbot to enable snowflake-02
16:21:43 <shelikhoo> since it is already working in tor browser
16:21:48 <dcf1> but there was a similar phenomenon with the client TLS fingerprint. when we enabled uTLS in Tor Browser, it made hardly a difference in the graph; when uTLS was enabled in Orbot, it began the sharp rise of the past 2 months
16:22:09 <dcf1> Yes this was a topic that we tabled before the holiday break last year
16:22:59 <dcf1> I don't know exeactly who gets that process started, but snowflake-02 is ready for it
16:23:58 <dcf1> and it should ease the load on snowflake-01 a bit and postpone the need for further hardware upgrades
16:25:19 <meskio> we have a meeting with some guardian project next tuesday (as part of sponsor 96), I can bring it up there
16:25:30 <meskio> or we could just write them directly
16:26:17 <dcf1> either is fine, it can wit until tuesday I think
16:26:32 <meskio> cool, I'll add it to the agenda of the meeting
16:27:18 <shelikhoo> okay, anything more on this topic?
16:27:28 <dcf1> all done
16:27:54 <shelikhoo> deofuscation obfs4 issues are public
16:27:54 <shelikhoo> https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/obfs4/-/issues/40007
16:27:54 <shelikhoo> https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/91
16:28:11 <dcf1> nice, thanks for taking care of that meskio
16:28:35 <meskio> not much to discuss on that, as we discussed before the holidays we made the deobfuscation issues public
16:28:44 <meskio> I sent an email about it to tor-relays
16:29:01 <meskio> 54% of the bridges are up to date
16:29:23 <meskio> so I expect most people will get working bridges if the outdated get blocked by that
16:29:47 <meskio> thank you dcf1 for finding the issues and providing tools to test them :)
16:30:06 <shelikhoo> yes! thanks dcf1!
16:30:29 <shelikhoo> yes, if we have nothing more to discuss we can move to the next topic
16:30:33 <shelikhoo> is the stun.stunprotocol.org situation resolved to everyone's satisfaction, or is more attention required?
16:30:33 <shelikhoo> https://lists.torproject.org/pipermail/anti-censorship-team/2022-December/000271.html
16:30:56 <shelikhoo> the issue here was there is a default stun server complain about the amount of traffic received
16:31:06 <dcf1> my impression is that this is taken care of sufficiently, just wanted to check and make sure
16:31:19 <shelikhoo> and we removed it from the list of default stun servers
16:31:21 <dcf1> so that John doesn't feel we have forgotten the issue
16:31:35 <dcf1> https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40241
16:32:01 <meskio> yes, it sounds like John was happy with the solution
16:32:14 <dcf1> ok, that's all I wanted to check
16:32:57 <shelikhoo> okay let's move to the next topic
16:32:58 <shelikhoo> (From last week) arti-based obfs4 quick reachability monitor https://gitlab.torproject.org/tpo/core/arti/-/issues/717#note_2866528
16:33:39 <shelikhoo> last week, while we did not have a meeting, arma dropped a link for discussion
16:33:55 <shelikhoo> it is about a bridge test system based on arti
16:34:36 <shelikhoo> feel free to comment on it if necessary
16:35:32 <meskio> I haven't had the time to read the issue yet
16:35:53 <shelikhoo> (from my impression, Roger think it should be the network team that work on this, but it have a for anti-censorship tag)
16:35:53 <meskio> for bridgestrap arti might solve some issues and let us have more control on what we do
16:36:36 <meskio> I'm happy if the network team works on it
16:37:01 <meskio> we should help there to move it into a useful direction for us
16:37:20 <meskio> this might be also useful for probetest, I guess
16:37:53 <shelikhoo> I think it is possible to make it a kind of drop in replacement for C-Tor in probetest
16:38:47 <meskio> nice
16:39:57 <shelikhoo> let's keep monitoring this ticket and see how it goes
16:40:22 <shelikhoo> okay the final part
16:40:23 <shelikhoo> snowflake blocking in Russia (maybe TSPU only) by Hello Verify Request (since about 2022-07-20)
16:40:23 <shelikhoo> https://ntc.party/t/second-snowflake-bridge-available-for-testing/3445/7
16:40:23 <shelikhoo> https://ntc.party/t/in-case-snowflake-rendezvous-gets-blocked/1857/9
16:40:23 <shelikhoo> https://gitlab.torproject.org/tpo/anti-censorship/censorship-analysis/-/issues/40030#note_2823140
16:40:23 <shelikhoo> https://explorer.ooni.org/chart/circumvention?since=2022-07-08&until=2022-08-14&probe_cc=CA%2CCN%2CIR%2CRU
16:40:43 <dcf1> I want to make sure this is still on the radar
16:41:17 <shelikhoo> this have been seen at our russia vantage point
16:41:26 <dcf1> I think that Snowflake is functionally blocked on a nontrivial fraction of ISPs in Russia, since last July, and testers have reported a specific feature they think is responsible
16:41:58 <dcf1> I think that what is needed here is another patch to alter the DTLS fingerprint, like what we did in December 2021.
16:42:18 <dcf1> https://explorer.ooni.org/chart/circumvention?since=2022-07-08&until=2022-08-14&probe_cc=CA%2CCN%2CIR%2CRU
16:42:42 <dcf1> OONI MAT still shows it being >50% successful, but there is perhaps an increase in anomalies around 2022-07-20.
16:43:38 <meskio> shelikhoo: is this something you could look into?
16:43:53 <meskio> (is gitlab down?)
16:44:08 <shelikhoo> meskio: yes, I will look into this
16:44:30 <itchyonion> gitlab is down for me as well
16:44:32 <meskio> thank you, good luck with it
16:44:43 <dcf1> (side note, see how correlated the number of tests per day is between RU and CA. I think we need to be suspicious of geolocation errors (RU IPs being mistakenly labeled CA), similar to what we encountered with IR->US with snowflake in https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/96)
16:44:46 <shelikhoo> <gaba> ohh, no anarcat
16:44:46 <shelikhoo> <gaba> gitlab is timing out
16:44:46 <shelikhoo> <lavamind> hello
16:44:46 <shelikhoo> <lavamind> I think a security update is being installed
16:45:20 <meskio> :)
16:46:25 <dcf1> https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40207#note_2844116 rather
16:46:51 <shelikhoo> I think russia is now again giving us a dose of the trouble...
16:47:14 <shelikhoo> like last time, this kind of rollout of censorship seems to happens a lot at winter
16:48:05 <shelikhoo> (the log collector in russia is now down and I am working on fixing it)
16:49:07 <shelikhoo> anything more we wish to discuss in this meeting?
16:49:32 <dcf1> cece[m]: you wrote "Help with: resources", anything specific we can help with?
16:51:45 <meskio> ups, maybe cece[m] lost internet
16:52:21 <dcf1> that's okay, perhaps it is clear with context outside the meeting
16:52:44 <meskio> I'll check with her
16:53:24 <shelikhoo> yes, anything more we would like to discuss in this meeting?
16:53:28 <meskio> nothing else from me for today
16:54:16 <shelikhoo> #endmeeting