14:00:32 <karsten> #startmeeting metrics team
14:00:32 <MeetBot> Meeting started Thu Oct 13 14:00:32 2016 UTC.  The chair is karsten. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:32 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:00:47 <karsten> agenda pad: https://pad.riseup.net/p/3M7VyrTVgjlF
14:03:03 <karsten> alright, I don't have more topics to add to the pad. should we start?
14:03:07 <iwakeh> yes.
14:03:12 <karsten> * tiny new paragraph in team-page: https://trac.torproject.org/projects/tor/wiki/org/teams/MetricsTeam#CensorshipWatch (iwakeh)
14:03:16 <karsten> looks good!
14:03:20 <iwakeh> ok
14:03:29 <iwakeh> that's what I wanted to hear :-)
14:03:40 <karsten> I really hope we can include such data in the graphs at some point.
14:03:49 <iwakeh> yeah
14:03:52 <karsten> see the thoughts in the UI doc.
14:03:59 <karsten> oh, speaking of, should we publish that?
14:04:02 <karsten> as first draft?
14:04:09 <iwakeh> sure.
14:04:34 <karsten> tor-dev@?
14:04:47 <iwakeh> hmm?
14:04:54 <karsten> thinking about which list.
14:05:04 <iwakeh> It's our target group, isn't it?
14:05:17 <karsten> yes. should be good.
14:05:24 <iwakeh> is there an analysis group ml?
14:05:50 <karsten> no, that would be tor-dev@ I think.
14:05:57 <iwakeh> fine.
14:06:13 <karsten> ok.
14:06:14 <karsten> * Globe redirect (karsten)
14:06:36 <karsten> I didn't hear back from the admins, so I'd say we should just go ahead and change the redirect there.
14:06:49 <karsten> and have it deployed for another two weeks before globe.tp.o goes offline.
14:06:57 <karsten> would you want to write that patch?
14:07:06 <iwakeh> yes, after sync.
14:07:21 <iwakeh> was on last weeks list
14:07:31 <iwakeh> of action items.
14:07:35 <karsten> yep, here's what we wrote down:
14:07:38 <karsten> - iwakeh: if sysadmins agree, make globe redirect manual and add expiration date
14:07:42 <karsten> let's pretend they agree. ;)
14:07:58 <karsten> okay, I'll move that to this week's action items list?... or you already did. great!
14:08:19 <karsten> moving on:
14:08:20 <karsten> * Ask another Tor person to help write CollecTor manual (karsten)
14:08:35 <iwakeh> why?
14:08:37 <karsten> I just thought that we might find another tor person to help with this.
14:08:44 <iwakeh> ah, ok.
14:08:48 <karsten> a native speaker who is less biased by knowing the code.
14:08:53 <iwakeh> hehe
14:09:07 <iwakeh> or have such a person test?
14:09:29 <karsten> so, I have two folks in mind here.
14:09:32 <iwakeh> they would have to know technical cmd-line stuff.
14:09:39 <iwakeh> whom?
14:10:16 <karsten> python devs with some java experience, I think.
14:10:25 <iwakeh> sounds fine.
14:10:26 <iwakeh> and
14:10:38 <iwakeh> there is already the table draft.
14:10:47 <karsten> ah yes, I was thinking to give that to them.
14:11:01 <karsten> or, ask one of them to write a manual and the other to test it.
14:11:10 <iwakeh> neat :-)
14:11:17 <iwakeh> after 1.1.0
14:11:23 <karsten> sounds good!
14:11:58 <karsten> alright!
14:12:01 <karsten> * CollecTor sync (iwakeh)
14:12:02 <iwakeh> should we device a template? or, will they do that?
14:12:09 <karsten> well, fine question!
14:12:24 <iwakeh> the column headers in the
14:12:36 <iwakeh> table would be good section headers.
14:12:51 <linda> I was looking at the privacy and security settings (onion button > privacy and security settings).
14:13:03 <karsten> yes. I think we can suggest those, but let them decide to pick something different.
14:13:09 <iwakeh> sure.
14:13:17 <linda> the text in the security slider tells me to  "mouseover for more details," but nothing happens when I mouse over the text.
14:13:55 <iwakeh> Sync?
14:13:56 <karsten> hi linda. mind asking in #tor-project or #tor, or after the meeting in 45 minutes? :)
14:14:02 <karsten> sync!
14:14:15 <iwakeh> I'm working my way through comment 25.
14:14:33 <iwakeh> Did you read about the path-blindness?
14:14:38 <karsten> yes.
14:14:44 <karsten> sure, I'll give more details.
14:14:50 <karsten> very quickly,
14:14:57 <iwakeh> related to tests.
14:15:05 <karsten> I think there were some changes where you're using downloaded time in recent/,
14:15:10 <iwakeh> so, only what should be there.
14:15:19 <linda> oh crap, sorry.
14:15:21 <linda> :/
14:15:22 <karsten> and in some cases there were singular/plural changes.
14:15:27 <karsten> linda: no worries. thanks! :)
14:15:28 <iwakeh> ah, we decided ealier
14:15:40 <iwakeh> about the exit-lists ..
14:15:52 <karsten> yes, so I wasn't sure if these changes were supposed to be there.
14:15:59 <iwakeh> and  the download time for merrged files in recent.
14:16:00 <karsten> like, did we want to switch to new paths in this commit?
14:16:30 <iwakeh> well, we can stay with old.
14:16:43 <iwakeh> the download time is useful though.
14:16:46 <karsten> might be easier.
14:16:54 <karsten> oh, I think the changes make sense.
14:17:06 <karsten> it's just a question of when to make them.
14:17:10 <iwakeh> so, what to put in 1.1.0?
14:17:24 <karsten> like, we should also make these changes to the other methods that write descriptors to disk.
14:17:25 <iwakeh> is it a problem for main collector?
14:17:41 <karsten> nope. we can change. but maybe in a separate commit?
14:17:51 <iwakeh> then, just leave the exit-list singular.
14:17:59 <iwakeh> and
14:18:06 <karsten> I think microdesc vs. microdescs was another.
14:18:11 <karsten> well, I'll go through them all.
14:18:13 <iwakeh> do another commit on top with that change?
14:18:20 <iwakeh> fine!
14:18:28 <karsten> wait, what's the commit plan?
14:18:42 <iwakeh> on top of the second branch after
14:18:44 <karsten> change this commit to reflect what's in the other code, and then make another commit for the changes?
14:18:51 <iwakeh> I worked through the comments.
14:19:35 <iwakeh> the singular/plural changes, yes.
14:19:49 <iwakeh> the download time for merges from synced
14:20:04 <iwakeh> data I would prefer to put in immediately.
14:20:05 <karsten> oh, I have another question about that.
14:20:14 <iwakeh> sure
14:20:42 <karsten> right now, the three existing data sources in the relaydescs module are run first, and then the sync happens.
14:20:53 <karsten> in particular, the reference checker comes after the first three sources.
14:21:02 <iwakeh> yes.
14:21:10 <karsten> can we change that so that the sync happens as fourth data source, and then the reference checker?
14:21:21 <iwakeh> Well,
14:21:48 <iwakeh> the current three sources are not modularized yet.
14:21:51 <iwakeh> I can
14:21:58 <iwakeh> take a look and if
14:22:15 <iwakeh> the change is minor it could be done. I basically
14:22:22 <iwakeh> didn't want to touch the
14:22:32 <iwakeh> current modules before sync is in place.
14:22:45 <karsten> ok.
14:22:53 <iwakeh> after the first sync
14:23:02 <iwakeh> the difference is not to big anymore.
14:23:03 <karsten> I'd say it's fine to put this on the list for now.
14:23:10 <iwakeh> ticket?
14:23:12 <karsten> and finish the sync.
14:23:13 <karsten> yes.
14:23:23 <iwakeh> yes.
14:23:32 <karsten> oh, another thought (sorry for jumping around):
14:23:38 <iwakeh> no problem.
14:23:58 <karsten> can we add another property for data sinks, that is, recent/ and/or out/?
14:24:12 <karsten> so that we can enable/disable those.
14:24:21 <iwakeh> oh why?
14:24:23 <karsten> longer term, but that would be useful for bridgedescs.
14:24:32 <iwakeh> of course.
14:24:49 <karsten> ah, now that I write it, we depend on out/ for deciding what to write to recent/.
14:25:07 <iwakeh> yes but recent could be left out ;-)
14:25:12 <karsten> hrrrrmmmm, let me put that on my list of things to think about first.
14:25:18 <karsten> yes, recent could be disabled.
14:25:33 <iwakeh> just make the tickets.
14:25:44 <karsten> ok!
14:26:51 <karsten> okay, what else should we talk about with respect to sync?
14:27:12 <iwakeh> I think all is said what fits into irc.
14:27:26 <iwakeh> ah
14:27:34 <iwakeh> what about the stats db?
14:27:48 <karsten> what would go in there?
14:27:55 <iwakeh> yes.
14:28:25 <iwakeh> I can't type to the pad anymore.
14:28:29 <iwakeh> :-(
14:28:33 <karsten> and is the goal to replace the reference checker?
14:28:42 <karsten> and/or the other stats for missing descriptors?
14:28:52 <karsten> (bad pad.)
14:28:56 <iwakeh> well, it would make microdesc very easy and elegant.
14:29:01 <iwakeh> I think.
14:29:06 <karsten> oh, I was wondering about that.
14:29:14 <karsten> we'd still have to make tarballs.
14:29:21 <iwakeh> from out
14:29:39 <iwakeh> and the correct date will be written in the out path.
14:29:40 <karsten> yes, and we need to put microdescs in the right out/ dir.
14:29:53 <karsten> how do we find out?
14:29:55 <iwakeh> wich can be queried from the db.
14:29:57 <karsten> ah!
14:30:11 <karsten> but right now we have our own internal data structure for that.
14:30:22 <karsten> which is not pretty, but which works.
14:30:26 <iwakeh> but not in sync.
14:30:37 <karsten> right, but sync could use that.
14:30:46 <iwakeh> reparse everithing?
14:31:03 <karsten> no, we're keeping a file where we store which descriptors we have and which we're missing.
14:31:16 <iwakeh> well, parse that file?
14:31:34 <iwakeh> this will be part of 1.2.0 so
14:31:52 <karsten> micro,2016-10-12 15:00:00,000149e6ef7102aaca9690d6e8dd2932124b94ab,QmnPC/AIp0xqHEhe5Zii1lOwq5zbJeXzBfW92uGoeBg,2016-10-09 00:05:00
14:31:55 <karsten> lines like that.
14:32:00 <iwakeh> we can compare and decide.
14:32:09 <karsten> 209538 lines in the file, 27M large.
14:32:14 <iwakeh> yes, I've seen these lines.
14:32:44 <karsten> so, I can see us use a database for that, but it seems like a bigger change.
14:32:56 <karsten> maybe out of scope for MOSS.
14:33:02 <iwakeh> it's just an hsqldb.
14:33:08 <iwakeh> part of sync.
14:33:21 <iwakeh> but, first I can
14:33:30 <iwakeh> look and
14:33:45 <iwakeh> structure the upcoming tasks
14:33:58 <iwakeh> so we get an idea how long things will take.
14:34:02 <karsten> ideally, we'd use the same code for both downloading and syncing.
14:34:03 <iwakeh> and then decide.
14:34:11 <karsten> same code for finding out where to store a microdesc.
14:34:22 <karsten> same code and same stats/ file or database.
14:34:38 <iwakeh> agreed.
14:34:39 <karsten> if you could reuse the code that's already there, that might be really quick.
14:34:53 <karsten> and then we could replace that by a database thingy in the future.
14:35:03 <iwakeh> I didn't rule that out. Just have the oom in mind.
14:35:04 <karsten> in theory, all stats/ can go into a database.
14:35:15 <karsten> I want to hunt down and kill that oom.
14:35:16 <iwakeh> yes, that's the goal.
14:35:31 <iwakeh> We get it.
14:35:34 <karsten> :)
14:36:07 <iwakeh> other sync topics?
14:36:35 <karsten> what should I prioritize tomorrow, re sync?
14:36:46 <iwakeh> the paths!
14:36:52 <karsten> ok.
14:36:58 <karsten> after that?
14:37:01 <iwakeh> great!
14:37:13 <iwakeh> the reviews of upcoming commits.
14:37:27 <iwakeh> but I need the paths for that.
14:37:31 <iwakeh> maybe
14:37:38 <iwakeh> look at the protocol again
14:37:43 <karsten> ok.
14:37:53 <iwakeh> I relied on that.
14:38:00 <iwakeh> for bridges at least.
14:38:12 <iwakeh> as I only have a pure download mirror at hand.
14:38:18 <karsten> right.
14:39:03 <karsten> okay, looks like we have a few action items there.
14:39:08 <iwakeh> yes :-)
14:39:22 <karsten> but I think we're making some really good progress! :)
14:39:31 <iwakeh> yep.
14:39:46 <karsten> alright, back to work?
14:39:56 <iwakeh> yes, back to work :-)
14:40:06 <karsten> ! thanks, talk to you next week. bye!
14:40:06 <iwakeh> bye, bye.
14:40:11 <karsten> #endmeeting