15:58:59 <meskio> #startmeeting tor anti-censorship meeting
15:58:59 <MeetBot> Meeting started Thu Sep 29 15:58:59 2022 UTC.  The chair is meskio. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:58:59 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
15:59:03 <meskio> hello everybody
15:59:05 <meskio> here is our meeting pad: https://pad.riseup.net/p/tor-anti-censorship-keep
15:59:07 <itchyonion> hello
15:59:14 <meskio> please add what you've being working on
15:59:15 <shelikhoo> hi~
15:59:18 <meskio> and points for the agenda
15:59:43 <meskio> I haven't touch the list in the discussion section, not sure how many are from last week and doesn't need discussion
16:00:42 <meskio> but let's start from the first one 'snowflake-01 bridge resources'
16:00:49 <meskio> dcf1, ln5?
16:01:08 <dcf1> the snowflake-01 bridge continues to see a lot of traffic, mostly from Iran
16:01:22 <dcf1> it's up around 2.5 Gbps constantly
16:01:38 <meskio> wow, that is impresive
16:01:54 <dcf1> but from the performance graphs it is clear we have hit a bottleneck, after removing some other bottlenecks that are summarized in the pad
16:02:53 <dcf1> current thinking is that we are running into CPU limits with kernel resources
16:03:41 <dcf1> the immediate plan is to disable TCP connection tracking as evidence indicates it may be one of the limits we are facing; and anyway the conntrack table is getting close to filling up
16:03:45 <dcf1> https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40189
16:04:31 <dcf1> and we also plan to do some experimentation with tuning the network interface driver
16:05:01 <dcf1> those change necessitate some preparation (e.g. tpo/anti-censorship/pluggable-transports/snowflake#40186) and a reboot (tpo/anti-censorship/pluggable-transports/snowflake#40188), which are expected to happen in a few hours
16:05:29 <dcf1> that's the news about the bridge
16:05:32 <meskio> I guess we should consider enabling the multi-bridge support soon, was the last point in the agenda, but let me push it to be the next
16:05:36 <cohosh> whew
16:05:46 <cohosh> that is a lot of tuning
16:06:11 <meskio> yeah, amazing work there, is impresive how much you are managing to get out of that machine
16:06:22 <meskio> and they talk about 'web scale'...
16:06:24 <shelikhoo> yeah, thanks dcf...
16:06:25 <meskio> snow-scale
16:06:28 <dcf1> it is quite a good machine too, don't get me wrong
16:07:34 <shelikhoo> gbps is a really cool word....
16:07:58 <shelikhoo> especially on a single device
16:08:04 <dcf1> ln5's post about conntrack and ethtool: https://lists.torproject.org/pipermail/anti-censorship-team/2022-September/000262.html
16:08:49 <ln5> o/
16:09:24 <meskio> should we move to discuss how we enable multi-bridge support? anything else on tunning the server/bridge?
16:09:40 <dcf1> i'm done on the snowflake-01 topic
16:09:47 <anarcat> o/
16:10:10 <shelikhoo> context: https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/95
16:10:32 <shelikhoo> the things that are currently blocking us from enabling this distributed snowflake support is that
16:10:56 <shelikhoo> once we enable this, then all proxy that have not updated to the version support this will be rejected
16:11:46 <shelikhoo> currently, there are some standalone proxy that is not updated
16:12:23 <shelikhoo> and webextension proxy have mostly finished updating
16:12:41 <dcf1> thanks for this investigation shelikhoo, good work
16:13:26 <shelikhoo> 28% of the unrestricted proxy poll don't support distributed snowflake server
16:13:26 <shelikhoo> we should enable this before next year since we will receive a drop of webext proxy as chrome will disable our extension soon
16:13:33 <dcf1> my interpretation is that it's fine to start rejecting these proxies
16:14:01 <cohosh> to be clear, i'm hoping we don't see a drop in the web-based proxies
16:14:02 <shelikhoo> yes, we are considering to set a deadline, and announce it
16:14:03 <dcf1> cohosh has a good observation in https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/95#note_2837409
16:14:13 <meskio> +1
16:14:37 <dcf1> the ones that don't update are standalong ones that people operate themselves and haven't upgraded recently
16:14:53 <dcf1> we care more about absolute numbers than relative numbers, I think
16:15:10 <dcf1> if you want to smooth the transition, you can do an intermediate step before fully deploying the new feature
16:15:41 <dcf1> e.g., when a proxy without relay URL support does a poll, reject it with 10% probability and return a response that will appear in the log
16:16:02 <dcf1> operators who are watching their logs will see it and upgrade, the ones who are not watching will not
16:17:31 <meskio> it looks like we are hitting the limits of the bridge, I'm wondering how long we can wait to do the final switch
16:17:59 <meskio> dcf1: I like youre proposal, but depends how soon we want to do the change
16:18:00 <dcf1> yes, I'm thinking of a timeline for multi-bridge rollout on the order of within 1 month, if that's possible
16:19:00 <dcf1> if the concern is what will happen if we lose a large chunk of proxies all at once, you can dial that rejection probability parameter from 0% to 100% over a week, slowly, and see if things remain stable. it's doesn't have to be a sudden loss of 100% of old proxies.
16:19:26 <dcf1> then, if things prove to keep working well with 100% of old proxies excluded, turn on the new feature
16:19:42 <arma2> technically, we could do the roll-out where if the proxy supports the new url format, we assign it to other snowflakes, and if it doesn't, we only match it with people who wanted to use the snowflake it knows how to use?
16:20:02 <arma2> i'm not saying we necessarily want to add that complexity, but, is it an option technically or am i misunderstanding some piece
16:20:03 <shelikhoo> no, the issue is that we are just rejecting x % of the poll, not x% of the proxy
16:20:30 <shelikhoo> unless we determine whether to reject based on hash of ip address rather than random
16:20:45 <shelikhoo> otherwise the same proxy will just try again
16:20:48 <dcf1> arma2: I think that's correct. clients that want to use the snowflake-01 bridge (implicitly or explicitly) could use an old proxy
16:21:03 <dcf1> shelikhoo: I am not making myself clear. There are 2 different things:
16:21:15 <dcf1> 1. there is a relay URL feature that is incompatible with old proxies
16:21:29 <dcf1> 2. there is concern about what will happen if all the old proxies are suddenly lost
16:21:57 <dcf1> my point is that there is no reason for 1 and 2 to be coupled. you don't have to suddenly lose all old proxies on the same day you activate the relay URL feature
16:22:02 <shelikhoo> arma2: we could do that, but the current code don't support this, and it would take a while to implement multi-pool support
16:22:26 <dcf1> you can remove all the old proxies (suddenly or gradually) first; then, if things remain working well, activate the relay URL feature
16:23:00 <dcf1> it doesn't matter if old proxies continue to poll after getting probabilistically rejected, because you haven't activated the new feature that they are incompatible with yet
16:23:24 <shelikhoo> yes, dcf1. We can remove them first and see what happens. Right now I don't think tor browser is shipped with distributed snowflake proxy support yet
16:23:55 <meskio> ohhh, that is something we need to change...
16:24:48 <cohosh> it's backward compatible with old clients, right?
16:25:03 <shelikhoo> yes. client is backward compatible
16:25:06 <arma2> if i were doing this, which i am not, i would be thinking of (a) change the broker to only pair only style proxies with clients whose requests are compatible with them, then (b) make tor browser clients do the new thing
16:25:08 <shelikhoo> proxy, broker is not
16:25:20 <arma2> s/only style proxies/old style proxies/ oops
16:25:43 <dcf1> cohosh: yes, it's supposed to be. the client upgrade can happen at any time, before or after the broker change, it's just the second bridge won't be used at all until clients have upgraded
16:25:47 <anadahz> o/
16:26:11 <meskio> arma2: I hear shelikhoo saying that your proposal will require some work
16:27:00 <shelikhoo> the first proposal will require additional work, which is the reason we are not doing this in the first place
16:27:34 <arma2> right. is it five lines of code, if you know the right place and the right five lines? :)
16:28:20 <arma2> the benefit would be that we can make dcf's two topics even more unrelated
16:28:57 <shelikhoo> no... I estimate it would be about 40-80 hours on this task....which convert to 1-2 month wall clock time..
16:29:07 <shelikhoo> but I could be wrong
16:29:40 <shelikhoo> we can just do the switch and see what happens
16:29:47 <shelikhoo> since the client is not currently using this
16:29:53 <shelikhoo> we can revert it if we wish
16:30:17 <meskio> that is probably not worth it, how much effort you think it will take to do what dcf proposes to have the option to configure a percentage of old proxies to be rejected
16:30:20 <meskio> ?
16:30:30 <dcf1> yeah I also feel, even a sudden loss of that proxy capability likely will not be a problem
16:31:17 <meskio> we'll see, as we are hitting a limitation on unrestricted NAT proxies...
16:31:19 <dcf1> in other words it does not make me feel uncomfortable if you decide to yolo it. as you say, it's easy to revert if there's a problem.
16:31:41 <shelikhoo> dcf's reject should take 16 hours or 1 - 2 weeks wall clock time
16:31:48 <cohosh> yeah i think we'll be ok
16:31:50 <shelikhoo> maybe we can reject old proxy
16:31:57 <shelikhoo> and see what happens
16:32:13 <shelikhoo> if something goes wrong we can revert it
16:32:40 <cohosh> on the compatibility problem, it may look bad to see a lot of rejections, but we don't have metrics on how long clients are actually waiting, and there are other fixes in the works to alleviate this stress
16:33:34 <meskio> sounds good to me, the original proposal AFAIK is to decide on a date we do the switch and make a call for proxies to update before that date as a last oportunities
16:33:58 <meskio> should we do something like Oct 19th (in 3 weeks)?
16:34:05 <shelikhoo> yes.. but if we wants to revert it, then we can't set a deadline
16:34:23 <shelikhoo> we can say we are going to reject it in 3 weeks
16:34:39 <dcf1> really appreciate what you are doing shelikhoo, it is good that most of the work is already in place at the time it is needed
16:34:50 <shelikhoo> and on next monday, try to start rejecting old proxies for a while
16:34:57 <shelikhoo> and collect some user feedback
16:35:08 <shelikhoo> I will write a guide on how to revert the rejection
16:35:13 <meskio> ahh, I see
16:35:21 <meskio> that is a good idea
16:35:23 <shelikhoo> so that everyone in the team can revert it
16:35:27 <cohosh> shelikhoo: awesome
16:35:41 <shelikhoo> and I do the rejection setting on the monday
16:35:45 <shelikhoo> and see what happens
16:35:46 <meskio> +1
16:36:14 <shelikhoo> if things goes wrong, everyone can revert it, even if I am out of hour
16:36:21 <meskio> I think we should mention that we are rejecting them in tor-relays and the forum, maybe we get some people to update
16:36:28 <shelikhoo> yes
16:36:32 <meskio> +1
16:36:58 <meskio> and if it goes well next week we should start looking into how to get it into TB
16:37:21 <shelikhoo> yes!
16:40:03 <meskio> great, we have a plan
16:40:07 <meskio> anything else on this topic?
16:40:14 <shelikhoo> nothing from me
16:40:22 <meskio> let's move to 'snowflake proxy resources'
16:40:49 <dcf1> I added the discussion item though I have not been working on it
16:40:58 <dcf1> I posted some links with the information I know about
16:41:18 <dcf1> the discussion subthread starts at https://lists.torproject.org/pipermail/anti-censorship-team/2022-September/000249.html
16:41:35 <cohosh> heh you worked on it a bit, your suggestions are good
16:41:39 <dcf1> merge request to have existing unrestricted NAT proxies do more work: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake-webext/-/merge_requests/47
16:42:40 <dcf1> I don't have any specific points to discuss, it looks like meskio is already reviewing !47
16:42:58 <meskio> yes, it looks trivial, but my knowledge of the webextension is not so great
16:43:20 <cohosh> the only thing i'll add is we had some people in universities reach out and offer to run some standalone proxies on un-NAT'd servers
16:43:50 <meskio> nice
16:43:57 <cohosh> there were also a few concerning instances on the forum where people were talking about turning off the firewalls of their home networks to improve their proxy compatability
16:44:26 <arma2> i learned how to haxxor my verizon router to make the standalone snowflake on this laptop be unrestricted, and it makes a huge difference in how many clients i see
16:44:40 <meskio> I wonder if we should work on a callout and/or documentation around how to configure unrestricted proxies, but we can wait to see if next week the problem is still there...
16:44:42 <arma2> (haxxor = log in to router, find section named 'static nat', click on some stuff)
16:45:12 <arma2> meskio: there is a ticket on exactly that,
16:45:14 <arma2> https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40128
16:45:19 <cohosh> i'm hesitant to encourage this in general
16:45:32 <arma2> but the question cohosh raises is a good one, around whether we should be instructing eager naive people to shoot their network in the foot
16:45:37 <meskio> arma2: ohhh, static nat, I didn't know about that, I just redirect a big chunk of UDP ports into my snowflake
16:45:54 <arma2> meskio: i imagine that the crappy web interface varies widely from one router to the next
16:46:26 <meskio> I have an openwrt, and I failed to modify the nat to allow unrestricted proxy...
16:46:37 <arma2> meskio: i also had to tell it to map my laptop to the external ip address, manually, so if the external ip address changes, my laptop won't be able to send any packets until i remember i did that and go fix it on the router. not best for normal users.
16:46:44 <meskio> anyway, I see cohosh concerns, makes sense, let's see if our other approaches work
16:47:34 <cohosh> i think that's all from me on this
16:47:48 <meskio> cool
16:47:50 <shelikhoo> meskio: linux by default don't support full cone masquerading
16:47:51 <shelikhoo> https://github.com/Chion82/netfilter-full-cone-nat
16:47:58 <shelikhoo> you might wish to look into this
16:48:15 <meskio> thanks
16:48:24 <shelikhoo> this is everything from me
16:48:28 <meskio> wow, kernel modules...
16:48:38 <meskio> anyway, 'gettor telegram bot down'
16:48:51 <meskio> I see a fix was deployed last week
16:49:07 <meskio> anything to talk about here? or is a leftover from last week?
16:49:16 <shelikhoo> leftover from last week
16:49:28 <meskio> cool, maybe the same with moat being down?
16:49:40 <cohosh> no that's new
16:49:47 <meskio> ahh, cool
16:49:52 <shelikhoo> but meskio might wish to chat with the developer about this
16:50:04 <cohosh> but i don't have much to say, i just brought it up because we don't know how long it was down for
16:50:26 <meskio> shelikhoo: you mean about gettor telegram? yes, I've being talking with them :)
16:50:27 <shelikhoo> I only deployed the change to get it working, but the developer have then pushed additional commits
16:50:32 <cohosh> some people have suggested it's the reason snowflake grew so popular, but idk how to verify that
16:50:38 <shelikhoo> yes, gettor telegram
16:50:55 <meskio> cool, I'll check on the changes
16:51:00 <meskio> sorry, cohosh continue
16:51:08 <cohosh> that's it, heh
16:51:22 <meskio> so having moat back reduce the load on snowflake?
16:51:37 <meskio> we'll see, as we have a next topic on how iran is blocking obfs4...
16:51:50 <cohosh> oh i don't think that was the case
16:52:14 <arma2> there is an enormous spike in moat requests, on the bridgedb server recently, if i remember that graph correctly
16:52:24 <arma2> enormous to the point that it seems unlikely they are mostly humans
16:52:45 <meskio> ufff
16:52:51 <cohosh> yeah, moat shouldn't get that many requests because you can only really use it once
16:53:07 <meskio> I wonder if they go over the domain front...
16:53:08 <cohosh> within a like 3 hour time span
16:53:28 <dcf1> might be a good idea to regenerate the moat captchas as a precaution, I think meskio did it the last time
16:53:58 <meskio> dcf1: yes, we don't relay on those so much anymore, as most people get bridges from circumvention settings that doesn't use captchas
16:54:07 <meskio> but I can regenerate them just in case
16:55:42 <cohosh> thanks meskio!
16:55:55 <shelikhoo> we have detected some blocking of obfs4 bridges in Iran, as we connect the bridge inside iran from outside
16:55:58 <meskio> ok, let's keep an eye on that, I'll try to investigate the stats to see what they are looking into
16:56:34 <arma2> shelikhoo: re the obfs4 bridge blocking from outside, do we know if the tcp connection worked?
16:56:39 <meskio> yes, the obfs4 bridges was a fresh private bridge, so they block it by protocol or something else
16:56:55 <shelikhoo> yes, I tested before with netcat
16:57:03 <shelikhoo> with content like "11" "wwww"
16:57:15 <arma2> so we could connect, and then we did an obfs4 handshake, and then we couldn't connect?
16:57:20 <cohosh> was the bridge given out?
16:57:38 <arma2> "well that's not good" :)
16:57:39 <meskio> cohosh: no, hasn't being given to anybody, was just configured
16:57:43 <shelikhoo> TCP's SYN SYN_ACK ACK is working
16:57:46 <cohosh> oof
16:57:58 <shelikhoo> and then the payload packet get dropped
16:58:06 <shelikhoo> after the first one
16:58:08 <meskio> they might be blocking random looking traffic
16:58:17 <arma2> in the past, iran did http whitelisting, i.e. they did dpi to guess the protocol and then they squeezed everything that didn't classify as a protocol they liked
16:58:28 <meskio> BTW, the bridge was not running the latest obfs4proxy
16:59:02 <arma2> i would be surprised if they are basing their decision on entropy. but, i have been surprised before. but, rarely by that censoring approach.
16:59:24 <arma2> and, connecting via obfs4 to bridges in iran, from inside iran, still works?
16:59:34 <meskio> yes
16:59:40 <shelikhoo> (for a while)
17:00:04 <arma2> sounds like the next steps are either (a) keep listening for more info in case somebody learns more, or (b) try to do some more concrete and more controlled experiment
17:00:07 <shelikhoo> then we received the report that it is blocked after we start distributing it
17:00:08 <meskio> but maybe our bridge got too public, or our traffic too 1:1 upload/download and they block it
17:01:12 <cohosh> or they are blocking traffic for services created recently by accounts outside the country
17:01:18 <dcf1> also if it doesn't get blocked with obfs4proxy-0.0.14 on both ends that would be awesome, I think that's unlikely to be the issue though
17:02:19 <meskio> yes, we should test that
17:02:39 <shelikhoo> eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
17:02:39 <shelikhoo> RX: bytes  packets  errors  dropped overrun mcast
17:02:39 <shelikhoo> 25579021098 70409889 0       1555    0       0
17:02:39 <shelikhoo> TX: bytes  packets  errors  dropped carrier collsns
17:02:39 <shelikhoo> 16187808560 17206989 0       0       0       0
17:02:50 <shelikhoo> I don't think it is blocked by traffic ratio
17:03:34 <meskio> mmm, good point
17:03:45 <cohosh> while we have the machine, we could try webtunnel to see if it works
17:03:58 <cohosh> even if it's not rolled out in tor browser yet
17:04:08 <meskio> so, or the bridgeline was leaked, or they just block unkown/random/obfs4 traffic
17:04:13 <meskio> also inside the country
17:04:21 * arma2 runs to next meeting, will read backlog after, thanks!
17:04:38 <shelikhoo> yes, we can try it
17:04:54 <shelikhoo> I think iran is blocking TLS based on Go's fingerprint
17:05:10 <shelikhoo> so we need to get uTLS in webtunnel
17:05:14 <meskio> cohosh, dcf1: if you want access to the machine feel free to ask
17:05:18 <cohosh> doesn't snowflake use Go's fingerprint for broker connections though?
17:05:19 <shelikhoo> before sure to work
17:05:37 <cohosh> utls support in snowflake also hasn't shipped in tor browser yet from what i can tell
17:05:41 <shelikhoo> cohosh: depend on platform, the fingerprint is different
17:06:04 <shelikhoo> the one blocked arm based system's fingerprint
17:06:08 <shelikhoo> I think...
17:06:23 <shelikhoo> the one blocked is arm based system's fingerprint
17:06:28 <shelikhoo> like android phone
17:07:00 <meskio> mmmm
17:07:17 <cohosh> ah i see, and we wouldn't get snowflake traffic from mobile networks anyway
17:07:30 <shelikhoo> there is only one way to find out... so we should try it
17:07:46 <meskio> :)
17:08:19 <shelikhoo> I think in iran people usually just use LTE, not a lot of people have fiber connection...
17:08:46 <shelikhoo> based on user feedback
17:09:27 <shelikhoo> maybe they are tethering the LTE connection to their desktop device?
17:10:35 <meskio> I will test if obsf4proxy-0.0.14 does make any difference on the connection
17:10:45 <shelikhoo> yes. let's test it
17:11:55 <meskio> anything else on this topic?
17:12:05 <meskio> or maybe we can close the meeting
17:12:16 <shelikhoo> nothing from me
17:13:09 <meskio> #endmeeting