13:01:19 #startmeeting network-health 2025-11-17 13:01:19 Meeting started Mon Nov 17 13:01:19 2025 UTC. The chair is hiro. Information about MeetBot at https://wiki.debian.org/MeetBot. 13:01:19 Useful Commands: #action #agreed #help #info #idea #link #topic. 13:01:35 here is the pad: https://pad.riseup.net/p/tor-nethealthteam-2025-keep 13:02:53 my updates for this week are: items on the pad + troubleshotting a few clickhouse items that have come up and get the MR for the migration merged 13:03:05 I also have to make the psql db there avaible to sarthikg 13:06:02 hiro: are we still ingesting data on metricsdb01? 13:06:27 GeKo (IRC): now we had to pause that a few months ago and we might retire that... is there anything we should migrate? 13:06:35 might be interesting to see some performance comparisons for queries between postgresql and clickhouse 13:06:45 I see 13:06:51 e.g. i have been doing something simple like 13:07:14 count all the relays in votes for maatuska for one day 13:07:23 and count all the measured relays in those votes 13:07:37 and it takes me 10 minutes to get an answer back from clickhouse 13:07:52 dunno if that's expected but it feels a bit long :) 13:07:53 uhm interesting... let me see if I can find why 13:08:04 it is not expected 13:08:33 there are some joins involved, yes 13:08:35 see: https://paste.debian.net/hidden/08df9687/ 13:08:40 but still 13:08:58 the dates i used are 2025-11-01 2025-11-02 13:09:16 (it's for https://gitlab.torproject.org/tpo/network-health/sbws/-/issues/40246) 13:09:57 Just created: https://gitlab.torproject.org/tpo/network-health/metrics/datastore/-/issues/3 13:11:00 yeah. 13:11:10 thanks 13:11:28 i am happy to learn proper sql queries for clickhouse in case that's the issue 13:11:32 GeKo: i'm interested in checking this one too. lemme know if you encounter more queries which have a significant performance impact when using clickhouse. 13:11:54 yeah. i am slowly testing different queries and will do 13:12:04 hiro: so, how is the data import going actually? 13:12:14 it seems we stopped having data after 2025-11-10? 13:12:24 I am having issues importing consensuses 13:12:40 so I will have go back to ingest in days instead of months or speed that up 13:12:46 I am talking about historical data 13:13:17 I am trying the speedup approach but if that doesn't work, I'll change the script to ingest historical data by days or week and go back to import recent data every few hours 13:13:25 okay 13:13:51 i am more interested in current data right now, which isn't available anymore after 2025-11-10 13:13:54 :) 13:14:02 i have not checked the historical stuff 13:14:10 but maybe the issues are related... 13:14:47 yes if the historical doesn't finish in a timely manner the recent is not imported so if I do not manage to speed that up now I'll jsut run the two things independently 13:14:57 ah, okay 13:15:33 should i file an issue for the bandwidth_file_document table being empty? 13:15:49 I'll do right away 13:15:58 or is that somehow else on the radar? 13:16:01 okay, thanks 13:16:04 alright 13:16:26 as you might have guessed i've wrestled with clickhouse and tested a bunch of queries last week 13:16:41 getting my scripts adapted and investigated some small things 13:16:44 yep thanks for that 13:16:59 i plan to do that further this week and then go back to my main business this week 13:17:10 p181 anomaly analysis write-up 13:17:14 that's it for me 13:19:21 not a lot from my side as well. i am now starting back on nsa, and trying to migrate a few api's to the new tables. once i have a validation of them working, and once metricsdb03 psql is accessible, i'll run the aggregator for about a month on it, and will push it for deployment 13:20:46 thanks sarthikg 13:21:18 anyone else? juga ? 13:22:21 Maybe juga is jsut getting adjusted with the wintertime change 13:22:33 If everyone is groot we can end the meeting 13:22:35 * hiro is groot 13:23:27 me too 13:24:12 #endmeeting