13:01:19 <hiro> #startmeeting network-health 2025-11-17
13:01:19 <MeetBot> Meeting started Mon Nov 17 13:01:19 2025 UTC.  The chair is hiro. Information about MeetBot at https://wiki.debian.org/MeetBot.
13:01:19 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
13:01:35 <hiro> here is the pad: https://pad.riseup.net/p/tor-nethealthteam-2025-keep
13:02:53 <hiro> my updates for this week are: items on the pad + troubleshotting a few clickhouse items that have come up and get the MR for the migration merged
13:03:05 <hiro> I also have to make the psql db there avaible to sarthikg
13:06:02 <GeKo> hiro: are we still ingesting data on metricsdb01?
13:06:27 <hiro> GeKo (IRC): now we had to pause that a few months ago and we might retire that... is there anything we should migrate?
13:06:35 <GeKo> might be interesting to see some performance comparisons for queries between postgresql and clickhouse
13:06:45 <hiro> I see
13:06:51 <GeKo> e.g. i have been doing something simple like
13:07:14 <GeKo> count all the relays in votes for maatuska for one day
13:07:23 <GeKo> and count all the measured relays in those votes
13:07:37 <GeKo> and it takes me 10 minutes to get an answer back from clickhouse
13:07:52 <GeKo> dunno if that's expected but it feels a bit long :)
13:07:53 <hiro> uhm interesting... let me see if I can find why
13:08:04 <hiro> it is not expected
13:08:33 <GeKo> there are some joins involved, yes
13:08:35 <GeKo> see: https://paste.debian.net/hidden/08df9687/
13:08:40 <GeKo> but still
13:08:58 <GeKo> the dates i used are 2025-11-01 2025-11-02
13:09:16 <GeKo> (it's for https://gitlab.torproject.org/tpo/network-health/sbws/-/issues/40246)
13:09:57 <hiro> Just created: https://gitlab.torproject.org/tpo/network-health/metrics/datastore/-/issues/3
13:11:00 <GeKo> yeah.
13:11:10 <GeKo> thanks
13:11:28 <GeKo> i am happy to learn proper sql queries for clickhouse in case that's the issue
13:11:32 <sarthikg[mds]> GeKo: i'm interested in checking this one too. lemme know if you encounter more queries which have a significant performance impact when using clickhouse.
13:11:54 <GeKo> yeah. i am slowly testing different queries and will do
13:12:04 <GeKo> hiro: so, how is the data import going actually?
13:12:14 <GeKo> it seems we stopped having data after 2025-11-10?
13:12:24 <hiro> I am having issues importing consensuses
13:12:40 <hiro> so I will have go back to ingest in days instead of months or speed that up
13:12:46 <hiro> I am talking about historical data
13:13:17 <hiro> I am trying the speedup approach but if that doesn't work, I'll change the script to ingest historical data by days or week and go back to import recent data every few hours
13:13:25 <GeKo> okay
13:13:51 <GeKo> i am more interested in current data right now, which isn't available anymore after 2025-11-10
13:13:54 <GeKo> :)
13:14:02 <GeKo> i have not checked the historical stuff
13:14:10 <GeKo> but maybe the issues are related...
13:14:47 <hiro> yes if the historical doesn't finish in a timely manner the recent is not imported so if I do not manage to speed that up now I'll jsut run the two things independently
13:14:57 <GeKo> ah, okay
13:15:33 <GeKo> should i file an issue for the bandwidth_file_document table being empty?
13:15:49 <hiro> I'll do right away
13:15:58 <GeKo> or is that somehow else on the radar?
13:16:01 <GeKo> okay, thanks
13:16:04 <GeKo> alright
13:16:26 <GeKo> as you might have guessed i've wrestled with clickhouse and tested a bunch of queries last week
13:16:41 <GeKo> getting my scripts adapted and investigated some small things
13:16:44 <hiro> yep thanks for that
13:16:59 <GeKo> i plan to do that further this week and then go back to my main business this week
13:17:10 <GeKo> p181 anomaly analysis write-up
13:17:14 <GeKo> that's it for me
13:19:21 <sarthikg[mds]> not a lot from my side as well. i am now starting back on nsa, and trying to migrate a few api's to the new tables. once i have a validation of them working, and once metricsdb03 psql is accessible, i'll run the aggregator for about a month on it, and will push it for deployment
13:20:46 <hiro> thanks sarthikg
13:21:18 <hiro> anyone else? juga ?
13:22:21 <hiro> Maybe juga is jsut getting adjusted with the wintertime change
13:22:33 <hiro> If everyone is groot we can end the meeting
13:22:35 * hiro is groot
13:23:27 <GeKo> me too
13:24:12 <hiro> #endmeeting