16:00:32 <meskio> #startmeeting tor anti-censorship meeting
16:00:32 <MeetBot> Meeting started Thu Feb  6 16:00:32 2025 UTC.  The chair is meskio. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:32 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:00:34 <meskio> hi everyone!
16:00:36 <meskio> here is our meeting pad: https://pad.riseup.net/p/r.9574e996bb9c0266213d38b91b56c469
16:00:38 <meskio> ask me in private to give you the link of the pad to be able to edit it if you don't have it
16:00:40 <meskio> I'll wait few minutes for everybody to add you've been working on and put items on the agenda
16:00:53 <onyinyang> hihi
16:01:45 <shelikhoo> hi~
16:01:57 <meskio> mmm, pasting all together I'm not sure if the first lines got into the log
16:03:06 <meskio> I guess I can start with the announcements
16:03:15 <meskio> looks like the azure domain fronts have stopped working
16:03:22 <meskio> thank you dcf1 for monitoring it
16:03:33 <meskio> we have removed the builtin azure meek bridge
16:03:44 <dcf1> yeah some time between 1 January and 4 February.
16:03:53 <meskio> I think the snowflake broker azure instance still needs to be decomissioned, isn't it?
16:04:13 <dcf1> yes, that's something I have to do in the azure portal
16:04:23 <dcf1> just click the delete button
16:05:08 <meskio> :)
16:05:38 <dcf1> I am also going to delete the snowflake-broker.bamsoftware.com DNS record, since it only existed for backward compatibility with the azure CDN
16:05:55 <meskio> sounds good
16:06:09 <shelikhoo> I could also remove that certificate renewal of that on the broker
16:06:16 <shelikhoo> I will do that next Monday
16:06:31 <dcf1> yes, I will record that in an issue as well
16:07:08 <meskio> lets move to the discussions:
16:07:11 <meskio> snowflake with covertdtls builds - ask for testing (on Tor Forum)?
16:07:21 <shelikhoo> (imagine the feeling of entering "terraform destroy" into pty)
16:08:14 <dcf1> yes, as I understand it, the covert-dtls builds are ready, right cohosh?
16:08:38 <shelikhoo> I have tested it as well and it seems to work locally on my machine
16:08:38 <dcf1> we originally decided to get these ready in response to acute blocking in Russia, but that has since gone away
16:08:49 <shelikhoo> dcf1: cohosh is not attending this meeting today
16:09:08 <dcf1> so we are in a chill position where we can invite users to test it then decide whether and how to integrate it
16:09:34 <dcf1> writing an invitation to test with instructions might be the responsibility of theodorsm
16:10:08 <theodorsm> I am still testing and it is not acting as I expected. The Snowflake client rarely becomes DTLS client and I it seems like messages are not manipulated as expected. I have been debuggig it this week
16:10:40 <shelikhoo> maybe we could wait a while?
16:10:44 <meskio> ohhh, then I guess we'll wait for your debugging
16:10:46 <shelikhoo> we are ready when you are
16:10:46 <dcf1> theodorsm: ah, interesting
16:10:57 <theodorsm> Broker is working as expected
16:11:53 <theodorsm> As I mentioned in the repo, I have built a version with s.SetAnsweringDTLSRole(webrtc.DTLSRoleClient) set, but still having unexpected behaviour
16:12:04 <theodorsm> *issue, not repo
16:12:13 <theodorsm> *MR, not issue haha
16:12:36 <dcf1> https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/merge_requests/448#note_3156637
16:12:44 <dcf1> I see, i hadn't noticed that update
16:12:56 <dcf1> I think this topic is done then.
16:13:23 <meskio> yes, let's move to the next one
16:13:25 <shelikhoo> yes.
16:13:40 <meskio> dcf1 turned the old debian 10 broker off, any problems with it?
16:14:17 * meskio goes to look into grafana to see if there is any usage change
16:14:20 <meskio> when was the change?
16:15:20 <dcf1> oops wrong mailing list link in the notes probably
16:15:41 <dcf1> 2025-02-01 19:13:10
16:15:46 <dcf1> https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40412#note_3154987
16:16:17 <meskio> thethere is no notizable change at that time
16:16:26 <dcf1> the old broker was still receiving traffic. it may have been some old never updated proxies because the logs were full of
16:16:37 <dcf1> rejected relay pattern from proxy = bad request
16:16:57 <dcf1> no change, that's great
16:17:15 <meskio> there are some spikes of higher usage around 17UTC
16:17:18 <meskio> maybe related
16:17:19 <dcf1> so I have not deleted the VPS instance yet. Is there anything from it I should save before I delete it?
16:17:50 <meskio> but the grafs look pretty stable, so I guess there is not big number of proxies affected
16:18:10 <shelikhoo> personally I don't think there is anything we need to save from it
16:18:21 <shelikhoo> at least not anything from my home folder
16:18:57 <meskio> same here, I think it can be deleted
16:19:05 <dcf1> the snowflake-broker log files? if we save those, we would resolve to audit them for publishability and then publish them
16:19:44 <dcf1> as we archived the logs of a previous instance of the broker https://archive.org/details/snowflake-broker-logs-20190416
16:20:02 <meskio> nice
16:20:04 <shelikhoo> I have a copy of the some logs as well
16:20:07 <dcf1> https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/30731
16:20:10 <shelikhoo> we might need to combine them
16:20:32 <dcf1> shelikhoo: why combine them? wouldn't the logs be complete on the server?
16:20:33 <shelikhoo> once there is disk full event at broker
16:20:44 <dcf1> oh ok
16:20:52 <shelikhoo> and I have to copy some of the logs to my disk and delete the one on server
16:21:23 <dcf1> I will suggest that it is not a good practice to keep private copies of things like server logs. If we don't have an immediate plan for them, we should default to deleting them.
16:22:19 <dcf1> I am also okay with deleting the logs with the broker VPS.
16:22:30 <shelikhoo> yes... I in general try to avoid non-reversible action
16:22:43 <shelikhoo> unless received approval to do so
16:23:13 <shelikhoo> Yes, so do we wants to publish such logs?
16:23:18 <dcf1> In that case my view is you should do as I am doing right now, ask for a decision from the team in the short term and then put it into action
16:24:01 <meskio> I don't know about publishing the logs
16:24:17 <meskio> I wonder if the old ones have being useful in those ~6 years since
16:24:26 <shelikhoo> yes.. I will do so in the future if possible
16:24:31 <meskio> :)
16:25:03 <shelikhoo> sadly in that case it was to fix a broken snowflake broker
16:25:25 <shelikhoo> it is difficult to request opinion from team in that case
16:26:04 <dcf1> in that case, save them as you did, but then ask for a decision by the team
16:26:35 <dcf1> don't let it default to keeping something forever, as a matter of security hygiene
16:26:39 <shelikhoo> in the more recent deployment, the log is managed by systemd and old log will be automatically deleted
16:27:21 <dcf1> ah yes, that aspect wasn't working for me ;) https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40428
16:27:41 <shelikhoo> (yes... shell looking shell's at 12 TB of disk backups)
16:28:13 <dcf1> ok then, it sounds like the plan is: dcf will delete the old broker VPS without retaining anything from the disk image, and shelikhoo will delete the local copy of snowflake-broker logs from before the disk-full event
16:28:15 <shelikhoo> dcf1: one need to be root to read these logs
16:28:17 <dcf1> ?
16:28:45 <shelikhoo> the systemd write these log to a temporary location that only systemd and root have access
16:28:57 <meskio> dcf1: +1 that plan
16:28:58 <shelikhoo> even if that log is produced by non-root user service
16:29:02 <dcf1> shelikhoo: post that on the issue, please
16:29:05 <shelikhoo> dcf1: copy
16:29:08 <shelikhoo> okay, I will do
16:29:20 <dcf1> judging by the '#' in the desciription, probably I was root at the time
16:29:36 <dcf1> ok thank you team
16:30:50 <meskio> now that the VM is reduced, do we have anything in place to monitor the load and see how is behaving?
16:31:35 <meskio> do eclips.is give some basic history of the VM load in their dashboard?
16:32:03 <shelikhoo> dcf1: okay --user means one's own user unit
16:32:08 <shelikhoo> not any user's user unit
16:32:25 <dcf1> meskio: only 24h I think
16:32:45 <dcf1> the available graphs are like these ones: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40412#note_3154987
16:32:57 <dcf1> I can log in real quick and check
16:33:11 <shelikhoo> I have checked the top on new broker, and the ram usage is about 3258.7 of 8K
16:33:29 <shelikhoo> so I think the new memory limit is not a huge concern for now
16:33:50 <meskio> I see, I wonder how much effort to put on that, but we should evaluate at some point if this VM is beefy enough for the broker or we need to find another solution
16:34:28 <meskio> good to know the memory is not the blocker, I expect CPU is less important here, but maybe I'm understimating the broker
16:34:38 <dcf1> The next step up would be 12 GiB / 6 CPU cores, which would be EUR 54 per month.
16:35:18 <dcf1> CPU graph https://share.riseup.net/#YTsyHhinfnNA4HMtuFNgvw
16:35:32 <shelikhoo> I am happy with deploying the broker somewhere else as well
16:35:45 <dcf1> There's no RAM graph
16:36:01 <dcf1> Network bandwidth graph https://share.riseup.net/#OmAvZ67ciA7jta3vqhE_Bw
16:36:19 <dcf1> Notice how there's spikes aligned to the start of every hour. I wonder if it is some malicious thing.
16:37:27 <dcf1> But yeah, 8 GiB / 4 CPU cores seems like it's okay for immediate needs
16:37:54 <meskio> this CPU usage is %, isn't it? so 200% means 2 cores full usage, so there are still 2 cores iddle (oversimplifying)
16:37:56 <meskio> ???
16:38:02 <dcf1> yeah
16:38:15 <meskio> ok, is not that bad
16:38:23 <meskio> it looks like it can live with that for now
16:39:53 <meskio> so we have are 50% of usage on CPU and RAM, we have space to grow
16:41:54 <meskio> on the hourly spikes, I guess the proxies are not doing any thing specil hourly, this might be clients comming back to life
16:42:06 <meskio> like a botnet sending a keep alive message
16:42:17 <meskio> or people downloading their email horly
16:42:56 <dcf1> could be
16:44:39 <meskio> I don't see any similar thing in grafana, like client polls are not happening hourly
16:45:06 <dcf1> oh interesting
16:45:18 <meskio> the broker is not doing any hourly clean up or whatever, isn't it?
16:45:48 <shelikhoo> the nginx did restart every hour to read changes in certificates
16:45:50 <dcf1> not anything that should affect the network, I don't think
16:46:01 <shelikhoo> but it will not affect network
16:46:07 <shelikhoo> only ram
16:46:26 <dcf1> and that restart wouldn't be aligned to absolute timestamps, would it?
16:46:43 <shelikhoo> the restart script runs every hour
16:46:49 <dcf1> like it might happen at 05:34, 06:34, 07:34, depending on when it was first started.
16:46:53 <meskio> ahh, wait, yes there is some client spikes, per hour
16:46:59 <shelikhoo> it is a timer
16:47:02 <meskio> not that big as the graf in traffic
16:47:06 <dcf1> Oh, it's a restart script, so indeed it might be 05:00, 06:00, 07:00
16:47:11 <shelikhoo> yes
16:47:46 <meskio> https://share.riseup.net/#6eBGqpsg73v9DVj4b21blg
16:48:11 <meskio> ahh, that makes sense
16:49:31 <dcf1> hmm, yes, there is something of an hourly pattern, not very pronounced
16:49:49 <meskio> I don't see this pattern with proxies
16:49:52 <meskio> is a client thing
16:50:44 <dcf1> if it were a proxy address harvesting operation, you would expect it to use client polls
16:51:39 <meskio> could be
16:52:01 <meskio> we could collect IP addresses on those spikes and see if there are a lot of requests from the same subnet
16:52:17 <shelikhoo> I don't think it proxy address harvesting would need to happen on a timer
16:52:24 <meskio> but a sofisticated attacker would use variaty of nets...
16:53:02 <shelikhoo> it is more likely someone that continuously harvest data
16:53:49 <dcf1> I don't know if there's any "likely"... people do a lot of weird stuff
16:54:06 <shelikhoo> yes...
16:54:08 <dcf1> I'm not too concerned about it, just noticed it in the graph
16:54:24 <meskio> :)
16:54:56 <meskio> anthing more on this topic?
16:55:00 <meskio> something else for this meeting
16:55:18 <meskio> ?
16:55:26 <onyinyang> nothing from me
16:55:39 <shelikhoo> "Identifying VPN Servers through Graph-Represented Behaviors"
16:55:39 <shelikhoo> https://dl.acm.org/doi/10.1145/3589334.3645552
16:55:39 <shelikhoo> https://dl.acm.org/doi/pdf/10.1145/3589334.3645552
16:55:39 <shelikhoo> https://github.com/chenxuStep/VPNChecker
16:55:50 <shelikhoo> last week there was a paper in the interesting link
16:56:11 <shelikhoo> at that time we decided to think about whether to make a reading group
16:56:22 <meskio> ohh, true, I've removed from the pad, sorry
16:56:40 <dcf1> I haven't read it yet but it seems like an important one that I am planning to read
16:56:54 <shelikhoo> I have read the paper a little and think it does give us a different perspective in proxy's anti-probing measurement
16:56:56 <dcf1> "graph-represented behaviors", I believe, means it is using access relations to discover proxies
16:57:06 <dcf1> I.e., N:1 access patterns or something like that
16:57:40 <meskio> we could do a reading group about it
16:57:52 <shelikhoo> yes, I recommend it for a reading group
16:58:24 <meskio> in two weeks if FOCI online, maybe no the right time for the reading group
16:58:46 <meskio> or this is what my calendar says, I haven't checked FOCI website
16:59:06 <meskio> yes, FOCI online is Feb 20
16:59:11 * meskio should remember to register
16:59:24 <meskio> what about Feb 27?
16:59:55 <shelikhoo> It works for me
17:00:04 <dcf1> i.e. fat thursday
17:01:11 <meskio> :)
17:01:25 <meskio> ok, I guess we can finish the meeting here
17:01:54 <meskio> #endmeeting