14:28:52 <karsten> #startmeeting metrics team
14:28:52 <MeetBot> Meeting started Thu Aug  9 14:28:52 2018 UTC.  The chair is karsten. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:28:52 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:29:21 <karsten> I wonder what to put on the agenda today..
14:29:50 <karsten> okay, let's go with your topic:
14:29:55 <karsten> Onionoo questions (irl)
14:30:06 <karsten> what questions are that?
14:30:21 <irl-abdn> so basically it's some questions about the architecture
14:30:30 <karsten> ok.
14:30:42 <irl-abdn> i noticed that i wasn't able to sync the out/state directories locally in the time between runs
14:30:53 <karsten> :(
14:30:57 <irl-abdn> as i'd already done a copy, i was expecting to just get diffs for currently running relays
14:31:09 <irl-abdn> but it looks like onionoo writes out every file every time
14:31:19 <irl-abdn> i wondered why it did this, if we expected this to be happening?
14:31:30 <karsten> I don't think it writes every file.
14:31:36 <karsten> it probably writes a lot of the details files.
14:31:49 <irl-abdn> the details files did take the longest to copy
14:32:02 <karsten> yes, that might be the case.
14:32:05 <irl-abdn> but for each of the document types it was a lot more than just the number of running relays+bridges
14:32:42 <irl-abdn> i guess the question is, for each run how many files do we expect to be touched?
14:32:59 <karsten> we do have statistics how many are being touched.
14:33:31 <irl-abdn> ah cool, i didn't think of that (:
14:33:36 <karsten> so, we can probably optimize this.
14:33:47 <karsten> what's the goal? easier backups? or also faster update runs?
14:33:55 <karsten> or something else?
14:34:06 <irl-abdn> both and also easier spin up of testing environments
14:34:25 <irl-abdn> essentially what i'm doing is a backup and a restore to another machine
14:34:55 <irl-abdn> i think it's likely that we don't currently have any consistent backups, that is they are all changed during the backup
14:35:28 <karsten> that might be the case, yes.
14:35:46 <karsten> what we could do: read a file before writing it and only write it if it's different.
14:36:15 <irl-abdn> yes, this would be great
14:36:32 <hellais> irl-abdn: can you please look at the email and ticket on the removal of the ooniprobe debian packages on deb.tpo (sorry to bug you here, but this is pretty urgent)?
14:36:33 <irl-abdn> to speed things up i was only comparing timestamps and filesizes, not performing diffs and checksums over ssh
14:36:33 <karsten> maybe modified to: check the file size, and if it's different, write. if it's the same, check more thoroughly.
14:36:48 <karsten> irl-abdn: feel free to do hellais' thing first.
14:37:55 <irl-abdn> hellais: aaah! i didn't realise it was still an advertised installation method
14:38:23 <irl-abdn> hellais: i am not currently with any keys so won't be able to fix until later today
14:38:33 <irl-abdn> hellais: i will make it priority when i am back with keys though
14:38:49 <irl-abdn> karsten: yes, this seems sensible
14:39:06 <irl-abdn> anything we can do in onionoo when it's close to the data is better for backup tools that are maybe working over the network and would be slower
14:39:08 <karsten> irl-abdn: want to create a ticket and look into this? or should I start looking?
14:39:36 <irl-abdn> karsten: it is up to you, if you'd like me to do it then i'll have to create the ticket later
14:39:40 <irl-abdn> as i can't log in to trac right now
14:39:56 <irl-abdn> we can always make todo list on the pad
14:40:12 <karsten> added to the pad for now.
14:40:38 <karsten> do you have more questions regarding onionoo?
14:40:57 <irl-abdn> one other
14:41:02 <karsten> sure!
14:41:12 <irl-abdn> is the out/ directory able to be entirely reconstructed from the status/ directory?
14:41:18 <karsten> heh.
14:41:27 <karsten> hopefully.
14:41:27 <irl-abdn> i remember a ticket about the out/ directory writing being deterministic
14:41:33 <irl-abdn> i don't remember if it was completed
14:41:38 <karsten> yes, but I'm not 100% certain that it's the case.
14:41:43 <karsten> I'd say, give it a try.
14:41:48 <irl-abdn> maybe this is a shortcut that we can use to speed up backups
14:41:59 <irl-abdn> it won't get us all the way there but it does halve the time
14:42:10 <karsten> so, it should be.
14:42:40 <irl-abdn> maybe we should add some test cases for this?
14:42:56 <karsten> we don't really have tests on that level.
14:43:15 <hellais> irl-abdn: ok thank you very much.
14:43:39 <karsten> so, for now, I'd say let's try it out. and if it's not the case, let's fix that.
14:43:49 <irl-abdn> ok cool
14:44:02 <irl-abdn> i did eventually get a copy of the status and out directory this morning
14:44:06 <irl-abdn> so i have something to play with
14:44:17 <karsten> that's good!
14:44:33 <karsten> by the way, I also broke an onionoo instance when making backups the other day..
14:44:52 <karsten> I needed a copy of the out/ directory to test the local tool that fixes broken json files.
14:45:15 <irl-abdn> right
14:45:29 <irl-abdn> i wanted to test the rewriting and the upgrade all together
14:45:48 <irl-abdn> which i can now do, it was just a bit of a pain to get to being ready for it
14:45:52 <karsten> what do you think, when can we deploy the fixes?
14:45:58 <irl-abdn> i didn't get to testing it yet
14:46:02 <irl-abdn> only being ready to test it
14:46:09 <karsten> sure. just to have an estimate.
14:46:17 <irl-abdn> oh, probably monday?
14:46:22 <karsten> ok.
14:46:54 <karsten> cool!
14:46:59 <irl-abdn> this is all my questions
14:47:27 <irl-abdn> did you have other topics?
14:47:30 <karsten> okay. I don't really have more topics. most things are in trac tickets already.
14:47:58 <irl-abdn> ok cool
14:47:58 <karsten> in which case let's go async again, if that's okay for you?
14:48:09 <irl-abdn> yep, sounds good (:
14:48:20 <karsten> great. have a nice day! bye. :)
14:48:32 <karsten> #endmeeting