13:01:44 #startmeeting network-health 2025-03-10 13:01:44 Meeting started Mon Mar 10 13:01:44 2025 UTC. The chair is hiro. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:01:44 Useful Commands: #action #agreed #help #info #idea #link #topic. 13:01:56 and the pad: https://pad.riseup.net/p/tor-nethealthteam-2025-keep 13:03:34 my update for this week is that I have started testing object storage and loading and offloading of data to and from the db 13:04:01 \o/ 13:04:12 I am also thinking I could move the statuses and family creations from object storage as well and just use the parser for actual documents 13:04:39 interesting 13:05:04 yeah I thought maybe we can save of the queries that we run 13:05:46 and the ingestion could be a lot faster this way 13:06:05 anyways that's all for me 13:06:29 nothing from my side besides what is already on the pad 13:06:39 thanks @GeKo 13:07:07 i managed to implement some query parameters for the /details endpoint and will continue with that this week 13:07:22 \o/ thanks @juga 13:07:44 yw 13:07:54 sarthikg[m]: want to go next? 13:08:27 i got the /router/details route working with most of the information. just finishing that one up.. will continue working on vm-based routes after this 13:08:40 @GeKo btw I see you will be starting to test the imported data. I have processed till October 21 2024 13:08:51 from june 2024 13:09:15 and also from Jan till feb 20 something... I have some gaps that I need to fill in the first part of the year 13:09:21 thanks @sarthikg[m] 13:10:16 hiro: yeah, i'll test as i am going. so, no rush 13:10:24 @GeKo @juga another thing that I think we might think about is if we want to keep all the data in the db (given that the object storage mechanism works fine) 13:10:36 so far, nothing to complain about :) 13:10:39 what I mean is: for example there is one table that we do not need that often (like votes) 13:11:05 hrm 13:11:23 so, you mean we could keep other stuff longer instead? 13:11:27 yeah 13:12:02 i'll think about that 13:12:02 i'm not sure which data to leave, maybe we can think about it together next days 13:12:07 yeah 13:12:11 s/data/tables/ 13:12:23 i am a bit worried about the added complexity to that scheme 13:12:58 which we then would need to document given that we want to provide db dumps at some point for external folks 13:13:04 assuming that's still the plan 13:13:27 yep 13:13:37 which would work more easily by just saying: you are welcome to use that db with X years/months of past data 13:13:52 but also if we can open up object storage we could serve all the data we have there (that we want ofc) 13:14:06 indeed 13:15:20 so you could do COPY table FROM 's3://mybucket/2024/10/13/relay-descriptors/server-descriptors.parquet'; 13:15:26 something like that 13:16:39 anyways let's see how that goes first 13:17:08 wfm 13:17:19 @ggus do we have a meetup scheduled or to schedule for mar/apr? 13:18:22 hiro: nope 13:18:50 ok 13:18:55 anything else for today's sync? 13:18:59 (there are some internal comms to be discussed first) 13:19:21 ok sounds good 13:20:32 nothing from me 13:20:42 * juga is good 13:21:01 oh, we have p183 sync at 14utc, right? 13:21:18 yes 13:21:21 kk 13:21:30 nothing from my side! 13:21:38 ok thank you everyone let's end the meeting 13:21:45 #endmeeting