15:59:15 <karsten> #startmeeting metrics team
15:59:15 <MeetBot> Meeting started Wed Dec 12 15:59:15 2018 UTC.  The chair is karsten. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:59:15 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
15:59:28 <karsten> there, I added two items to the agenda. anything else from you?
15:59:56 <irl> nothing from me
16:00:03 <karsten> ok.
16:00:12 <karsten> * metrics-web issue update (karsten)
16:00:15 <irl> (unless we still need to discuss sponsor13 reporting?)
16:00:39 <karsten> added.
16:00:48 <karsten> so, ~2 weeks ago, metrics-web broke.
16:00:54 <irl> right
16:01:09 <karsten> as of today I fixed most things.
16:01:21 <karsten> the last remaining piece is the bwhist module.
16:01:44 <karsten> apparently, re-importing just the 2018-11 data is prohibitively expensive on the live system.
16:01:51 <karsten> it would take days.
16:02:09 <karsten> I'm not re-importing it on my local server and will then export and import aggregate stats on the live system.
16:02:21 <irl> ok
16:02:25 <karsten> I also looked at the module today, but I'm afraid that rewriting that would take quite long.
16:02:40 <karsten> my current plan is to hope that it doesn't break again until there are no bwhists anymore.
16:03:25 <karsten> that's all on this topic, mostly an update.
16:03:35 <irl> maybe we need a trac keyword for "this isn't needed if privcount happens before we get to it"
16:03:44 <irl> so we can still keep track of issues in case privcount is delayed
16:04:11 <karsten> well, in this case it's not an issue, it's just incredibly slow for re-imports.
16:04:28 <karsten> well, maybe that's an issue. :)
16:04:50 <karsten> not sure if opening a ticket would help, though.
16:04:53 <irl> ok
16:05:13 <karsten> okay, next topic:
16:05:17 <karsten> * minor webstats issue (karsten)
16:05:31 <karsten> I noticed today that we're sanitizing the same old log files over and over.
16:05:47 <karsten> this is the case when a webserver rsyncs logs to us and then suddenly stops.
16:06:06 <karsten> I wonder if you have an idea for that.
16:06:25 <irl> i'm not sure i understand the issue
16:06:26 <karsten> we have a directory on colchicifolium that webservers rsync to.
16:06:40 <karsten> each of them rsyncs into a different subdirectory.
16:06:49 <karsten> and they delete files that don't exist locally to them anymore.
16:07:16 <karsten> but when they stop rsyncing, possibly because they're taken offline, they don't delete what's there in the subdirectory on colchicifolium.
16:07:39 <irl> oh, like permanently offline?
16:07:43 <karsten> yes.
16:08:00 <irl> i think having the webservers in charge of the deleting is the problem there
16:08:05 <irl> that is what collector should be in charge of
16:08:48 <karsten> that would mean we need a policy when do delete something.
16:09:00 <irl> yes
16:09:06 <karsten> to*
16:09:07 <irl> probably this would be similar to bridge descriptors
16:09:12 <karsten> oh, yes.
16:09:46 <irl> i think for now the answer is we clean things up when webservers are taken offline
16:09:49 <karsten> I wonder if there's any way to check when a webserver last rsync'ed anything to us to a specific directory.
16:10:00 <irl> the sysadmin team should open a ticket on collector to let us know and that should just be a step in the process
16:10:19 <karsten> right, that's a possible way.
16:10:33 <irl> in the future, we should have a trace file that gets rsynced too with the timestamp of the start of the last rsync process
16:10:41 <karsten> somewhat similarly, we could check periodically which webservers stopped existing.
16:10:43 <irl> so "date > trace" and then rsync in the cronjob
16:11:00 <karsten> yes, something like that.
16:11:13 <irl> this could also be part of our monitoring setup
16:11:29 <gaba> hey
16:11:29 <karsten> hi gaba!
16:11:36 <irl> hi gaba!
16:12:01 <irl> gaba: http://meetbot.debian.net/tor-meeting2/2018/tor-meeting2.2018-12-12-15.59.log.txt
16:12:09 <irl> (meeting so far)
16:12:23 <gaba> ok, thanks
16:12:46 <karsten> okay, I think I'll clean up things myself for now.
16:12:57 <karsten> it's not a good week to start yet another thing that involves many people.
16:13:05 <irl> heh
16:13:05 <karsten> this could be something to discuss in some more detail in brussels.
16:13:12 <irl> definitely
16:13:17 <karsten> cool!
16:13:23 <gaba> I sent the november report for sponsor 13 to otf today.
16:13:29 <karsten> * Sponsor 13 reporting (irl)
16:13:32 <karsten> speaking of :)
16:13:42 <irl> ok awesome
16:13:43 <karsten> so, you had everything you needed for that?
16:13:49 <gaba> one more thing is confirming that you both have a room booked in hte hotel in brussels and we are meeting at the same hotel
16:13:56 <gaba> yes karsten, thanks!
16:13:56 <irl> so maybe we don't need to do anything else here for sponsor 13
16:14:28 <karsten> gaba: you mean you can confirm this now, or someone still needs to confirm that?
16:14:40 <irl> gaba: is the report public or is it secret?
16:14:56 <gaba> I am confirming this now
16:15:00 <karsten> okay.
16:15:08 <karsten> irl: past reports were not made public, AFAIK.
16:15:16 <gaba> the report is public and is a short one. It is the report on that grant for november
16:15:20 <gaba> ohh, i see
16:15:24 <gaba> yes, not public really
16:15:33 <gaba> public to everybody on the otf talk mailing list :)
16:15:39 <karsten> oh!
16:15:43 <gaba> that is where the otf reports go to
16:15:56 <karsten> okay, I didn't follow where it went.
16:16:12 <karsten> but yes, past reports were all based on our monthly reports plus some additional information.
16:16:36 <gaba> yes
16:16:36 <irl> "This is a private moderate traffic list including past and current OTF supported projects, fellows, AC members, and other key at-large folks from the internet freedom community who’ve opt’d in to join this list."
16:16:53 <irl> unless we have been asked to not distribute the reports, it would be nice to make them available
16:17:34 <karsten> fine by me. we'd have to ask isabela who wrote them.
16:17:45 <gaba> We could have progressions in the wiki about a specific project/sponsor. I need to check with Bekeela on that.
16:17:47 <karsten> except this last one.
16:18:17 <gaba> Sometimes that information (about making reports public) is in the contract. I will check with Bekeela.
16:18:17 <irl> doesn't have to be right now, we could also discuss this at the next tor meeting what to do with all these reports we write
16:18:23 <gaba> yes
16:18:36 <karsten> sure, sounds good.
16:18:52 <karsten> it's a different type of report than the one for sponsor 19, IIUC.
16:19:03 <gaba> yes, it is only a report on progress
16:19:07 <gaba> a very short text this time
16:19:49 <karsten> okay.
16:19:54 <karsten> anything else on this topic?
16:19:58 <karsten> or on other topics?
16:19:58 <irl> not from me
16:20:02 <gaba> not from me
16:20:12 <karsten> great!
16:20:18 <karsten> next meeting next thursday?
16:20:20 <irl> yep
16:20:42 <karsten> cool! have a good evening/day! bye. :)
16:20:46 <irl> bye!
16:20:48 <karsten> #endmeeting