14:29:45 <karsten> #startmeeting metrics team
14:29:45 <MeetBot> Meeting started Thu Aug 16 14:29:45 2018 UTC.  The chair is karsten. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:29:45 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:30:11 <karsten> we already have a few topics. should we add more before we start?
14:30:30 <irl> i did not have any other topics
14:30:34 <karsten> okay.
14:30:39 <karsten> * Onionoo 6.2-1.17.0 deployment vs. 6.2-1.17.1 release (#27163) and deployment (karsten)
14:30:50 <karsten> I saw your last comment after adding this topic.
14:31:04 <karsten> so, should we postpone deployment until tomorrow?
14:31:20 <irl> yes, i don't want to rush considering which other tests may be missing
14:31:27 <karsten> makes sense!
14:31:40 <karsten> but it also doesn't make as much sense to deploy today and once again tomorrow.
14:31:45 <karsten> so, postponed, I'd say.
14:32:02 <irl> ok
14:32:08 <karsten> * Onionoo tickets from last week's questions (irl)
14:32:17 <karsten> I found those questions on the pad.
14:32:23 <karsten> did you want to create tickets for them?
14:32:42 <karsten> otherwise we can carry them over to today's notes.
14:32:58 <irl> i was trying to work out what those tickets were before i created them
14:33:07 <karsten> sounds good, too.
14:33:15 <irl> essentially we want to be in a place where if there are no new files then nothing should be written to disk
14:33:39 <irl> i've been looking already at making sure nodestatus is deterministic, but i've not yet looked at detailsstatus
14:34:02 <irl> we can carry those notes over here and i will think about it more
14:34:09 <karsten> okay, great.
14:34:44 <karsten> reordered the next two.
14:34:45 <karsten> * Reconfigure collector2.tp.o (#27076) (karsten)
14:34:59 <karsten> just a friendly reminder.
14:35:28 <karsten> it's a lot of text, but I think it's not a hard decision.
14:35:35 <karsten> unless I overlooked something.
14:36:12 <irl> i think we can make this change
14:36:35 <karsten> okay. can you comment on the ticket?
14:36:51 <irl> i plan to be setting up a collector at the university, which would sync, so we would still notice if that functionality breaks
14:37:01 <karsten> sounds good!
14:37:09 <nevro> hi y'all, just wanted to mention that I'm here and it's my first time at a Tor meeting. I'm interested specifically in the metrics team.
14:37:30 <karsten> hi nevro!
14:37:39 <karsten> welcome. :)
14:37:47 <irl> hi nevro (:
14:37:56 <nevro> thank you, karsten! I've been reading several of your papers over the past week or so, so it's great to meet you!
14:38:02 <nevro> hi irl :)
14:39:12 <karsten> cool! flexlibris mentioned that you might join us today. she also mentioned that you might be interested in volunteering.
14:39:37 <karsten> do you want to watch the remaining few topics on the agenda, and then we talk about the volunteering part?
14:39:44 <nevro> sounds like a plan!
14:39:50 <karsten> btw, I did not paste the agenda link here...
14:40:09 <karsten> https://storm.torproject.org/shared/5h1Goax5eNusxjXJ_Ty5Wl7hFR1uqCReUiN8xdlBG8T <- agenda pad
14:40:13 <karsten> okay, great!
14:40:26 <flexlibris> glad you're here nevro :)
14:40:44 <karsten> hi flexlibris! thanks for making the introduction. :)
14:40:55 <flexlibris> happy to :)
14:41:10 <karsten> okay, I think that's all for #27076.
14:41:13 <karsten> next is:
14:41:15 <karsten> * Remote database dump (karsten)
14:41:28 <karsten> so, I'm just curious if there's a better way to do this:
14:41:50 <karsten> I'm currently dumping the exonerator database via ssh.
14:42:01 <karsten> the reason is that I'd want to try to optimize it.
14:42:25 <karsten> the thing is, it's 260G large on the server, and there's no way to dump it to the disk there.
14:42:47 <karsten> that's why I'm using ssh with a command and piping the output into a local file.
14:43:06 <karsten> it's running for hours now. is there a smarter way to do this?
14:43:19 <karsten> ssh materculae "pg_dump -U exonerator -h localhost -Fc" > exonerator-2018-08-16.sql
14:43:23 <karsten> is what I'm doing now.
14:43:33 <irl> have postgres data directory in a zfs dataset
14:43:44 <irl> periodically make snapshots and do zfs send recv over ssh
14:44:05 <irl> probably this won't work though
14:44:11 <irl> because we don't have zfs
14:44:15 <karsten> right.
14:44:26 <irl> do you do this often?
14:44:35 <karsten> every couple of years.
14:44:43 <irl> oh, then just run it over ssh
14:44:49 <irl> this is probably not worth optimising
14:44:52 <karsten> but, if it's easier, I'd probably do it more often.
14:45:13 <irl> i think the only optimisations i can think of involve having things updated every hour at least
14:45:33 <irl> either live replication, or something with filesystem snapshots
14:45:48 <irl> but longer periods of time maybe the diffs turn into transferring everything anyway
14:46:00 <karsten> hmm, ok.
14:46:03 <irl> this is very similar to the problem with onionoo
14:46:08 <karsten> true.
14:46:11 <irl> there's not enough local disk space to make a copy of the status directory
14:46:16 <karsten> :(
14:46:19 <irl> so getting a consistent copy of it is impossible
14:46:22 <karsten> not even compressed?
14:46:47 <irl> i didn't try that, but that's a lot of cpu time still
14:47:22 <karsten> so, okay, not something we can solve easily today.
14:47:27 <irl> nope
14:47:29 <irl> but
14:47:32 <karsten> I'll keep the ssh running then.
14:47:36 <irl> when i get these test instances up at the university
14:47:39 <irl> they would have zfs storage
14:47:44 <irl> and we can experiment with this
14:47:50 <karsten> sounds good.
14:48:07 <karsten> okay, last topic on the agenda:
14:48:09 <karsten> * Roadmap change: complete 13, defer 18, resume subset of 1, 3, 9, 12, 15 (karsten)
14:48:17 <karsten> let me explain.
14:48:34 <karsten> I don't think we can make much progress on 18 before mexico city.
14:48:59 <karsten> on the other hand, I could imagine that we finish 13.
14:49:13 <karsten> and then we could resume some of the tasks we deferred earlier.
14:49:26 <karsten> just to burn down the burndown chart.
14:49:46 <karsten> and to not to have to add these partial tasks to the next roadmap.
14:50:02 <karsten> because we still want to do them. they're useful.
14:50:26 <karsten> I was thinking about picking up 9 and 1.
14:50:39 <karsten> and I was wondering whether you'd want to pick up 3 and/or 12.
14:51:04 <irl> i guess 12 is part of #24422
14:51:12 <karsten> right.
14:51:14 <irl> yes, 3 and 12 look like things i can do
14:51:28 <irl> i was actually earlier this week talking with ux team about progressing that
14:51:35 <karsten> neat!
14:52:13 <karsten> okay, sounds like a good new plan then.
14:52:36 <karsten> I'll not update the roadmap just yet but save that for the next meeting, if that's okay.
14:52:47 <irl> i may not be here for the next meeting
14:52:49 <irl> but yes, this is ok
14:52:50 <karsten> just in case you or I figure out that there are obstacles we didn't see now.
14:53:04 <karsten> or the meeting in 2 weeks.
14:53:16 <irl> yep ok
14:53:30 <karsten> so, regarding next week,
14:53:47 <irl> although i may also not be at that one because i'll be on a train
14:53:50 <irl> (in two weeks)
14:53:52 <karsten> do you have branches open that are changing major parts of a codebase?
14:54:04 <irl> i do not have any major changes open, no
14:54:04 <karsten> we can pick a different date for that meeting then.
14:54:42 <karsten> okay. then I'd just go ahead and make larger changes to the codebases if necessary.
14:54:45 <irl> we could do a meeting on monday 27th? i'll be just back from sigcomm then would have tues+wed before going to emf
14:54:55 <karsten> like, change logging or date/time types, etc.
14:55:02 <irl> yes that's ok, no problem
14:55:12 <karsten> 27th sounds fine to me.
14:55:29 <irl> 14:30utc?
14:55:50 <karsten> sure.
14:56:07 <irl> cool
14:56:21 <karsten> great!
14:56:25 <karsten> nevro: still around?
14:56:29 <nevro> yup!
14:56:44 <nevro> very cool that you're going to sigcomm irl
14:57:40 <karsten> so, when flexlibris mentioned your interest in metrics things I came up with two quick ideas where we need help.
14:58:03 <karsten> they are both related to our roadmap item 13 that we just said we'd complete by the end of september.
14:58:09 * karsten finds a link
14:58:28 <karsten> https://trac.torproject.org/projects/tor/wiki/org/teams/MetricsTeam#Research
14:58:59 <karsten> we recently finished two documents that are supposed to help folks understand better what we're doing.
14:59:05 <karsten> https://metrics.torproject.org/reproducible-metrics.html
14:59:10 <karsten> https://metrics.torproject.org/stats.html
15:00:02 <karsten> you're new to most of this stuff, so I wonder if you'd like to take a look at those and tell us whether they make any sense to you.
15:01:12 <karsten> did I just successfully overload you with information? :)
15:01:28 <nevro> I've seen both of these, as flexlibris mentioned them to me. /stats.html is straightforward in the sense that it gives an overview of each of the graphs on the metrics page. That all was rather easy to go through.
15:01:38 <karsten> okay, cool!
15:01:47 <nevro> The other page on the other hand, a tad bit more convoluted
15:02:24 <nevro> I'm not exactly sure who the intended reader is though. Like is it much of a problem if I cannot easily read the reproducible metrics page?
15:02:58 <karsten> well, we should attempt to fix that.
15:03:19 <karsten> I'd be curious which parts are hardest to understand.
15:03:50 <nevro> Sure, so one thing I can do is go through it and make notes?
15:04:05 <karsten> that would be very useful. and add questions.
15:04:14 <nevro> Great, yeah, I can do that!
15:04:39 <karsten> awesome! that would be really cool.
15:05:15 <karsten> is that what you had expected when talking about volunteering? or did you expect to hack on something? maybe we could find something else, just in case this turns out to be less fun than expected.
15:05:45 <nevro> Well, before I do anything, I need to understand how all of this works under the hood. I think this is a great first task.
15:05:53 <karsten> okay, great!
15:05:54 <nevro> I admit that it's all very intimidating.
15:06:06 <nevro> But eventually, I would like to get my hands a bit dirtier. This is a great way to start imo
15:06:08 <karsten> let's fix that, too. :)
15:06:16 <karsten> okay!
15:06:20 <nevro> ty :)
15:06:43 <karsten> so, next meeting is on the 27th, as you might have seen earlier.
15:06:52 <karsten> I'll send out another reminder to the team list.
15:07:07 <karsten> if you have anything until then, just post to the team list.
15:07:25 <karsten> or create a ticket in one of the Metrics/* components.
15:07:50 <karsten> okay. I think that's all for today. unless there's more.
15:08:10 <karsten> by the way, pg_dump just finished. I have a 52G file here now. hope it works.
15:08:40 <irl> (:
15:08:48 <karsten> alright. ending the meeting. thanks, irl and nevro! talk to you in 11 days. bye!
15:08:53 <irl> bye!
15:08:54 <nevro> bye bye!
15:08:59 <karsten> #endmeeting