14:00:01 <karsten> #startmeeting metrics team
14:00:01 <MeetBot> Meeting started Thu Oct  6 14:00:01 2016 UTC.  The chair is karsten. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:01 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:00:07 <karsten> shall we start?
14:00:14 <karsten> looks like we already have a few agenda items.
14:00:14 <iwakeh> ok.
14:00:20 <karsten> or do you need more time?
14:00:26 <iwakeh> al fine.
14:00:29 <iwakeh> all.
14:00:32 <karsten> ok.
14:00:37 <karsten> * CollecTor sync (iwakeh)
14:01:16 <iwakeh> currently I'm polishing the code and implementing tests, one
14:01:28 <iwakeh> question for the merging:
14:01:44 <iwakeh> with #20228
14:02:08 <iwakeh> we decided to store by acquisition time
14:02:16 <karsten> yep.
14:02:32 <iwakeh> So, which is the acquisition time for descs from other instances?
14:02:49 <iwakeh> The sync-time.
14:03:06 <iwakeh> Does this still make sense for
14:03:10 <iwakeh> torperf?
14:03:37 * karsten looks at code
14:04:21 <karsten> rsyncCatString in ArchiveWriter is relevant here, I think.
14:04:28 <karsten> we're setting that string once per execution.
14:04:41 <karsten> and using it for server descriptors and extra-info descriptors.
14:04:46 <karsten> we could use it for all other descriptors, too.
14:05:06 <karsten> so, the specifics of torperf are a fine question though.
14:05:16 <iwakeh> That's not the question. I think.
14:05:35 <karsten> okay, just torperf is the question?
14:05:37 <iwakeh> CollecTor one downloads torperf-1
14:05:59 <iwakeh> CollecTor two syncs torperf-1 from C1.
14:06:07 <karsten> can we ignore torperf?
14:06:18 <iwakeh> should c2 store torperf using the date
14:06:22 <iwakeh> hu?
14:06:22 <karsten> in the sync, I mean.
14:06:35 <iwakeh> oh, you mean leave it out?
14:06:38 <karsten> yep.
14:06:49 <iwakeh> well, that reduces the effort.
14:06:50 <karsten> the most important descriptors for sync are relaydesc.
14:06:57 <karsten> bridgedesc are useful, too.
14:07:00 <karsten> exitlist maybe, too.
14:07:02 <iwakeh> micro?
14:07:02 <karsten> but torperf, meh.
14:07:08 <karsten> micro are part of relaydesc.
14:07:13 <iwakeh> yes.
14:07:21 <iwakeh> exit-lists
14:07:31 <karsten> yes, somewhat useful.
14:07:38 <karsten> we should include those.
14:07:48 <karsten> unless it's hard.
14:07:51 <iwakeh> fine, then no sync for torperf.
14:07:56 <karsten> sounds good.
14:08:04 <karsten> we should think about sync for onionperf though.
14:08:07 <karsten> when we get to that.
14:08:26 <iwakeh> well, the files will be different ?
14:08:29 <karsten> sure, sure.
14:08:40 <iwakeh> the architecture is open for all that.
14:08:46 <karsten> ok.
14:08:48 <iwakeh> the new one.
14:08:52 <karsten> heh yes.
14:08:53 <iwakeh> ;-)
14:09:24 <iwakeh> that's it.
14:09:29 <karsten> what's the time frame here?
14:09:35 <karsten> when should I reserve time to review patches?
14:09:39 <anathema_db> ops
14:09:43 <anathema_db> I'm here as well!
14:09:44 <iwakeh> unless there are unexpected thing
14:09:48 <karsten> hello anathema_db.
14:09:48 <anathema_db> (sorry I missed the start)
14:09:50 <iwakeh> hi!
14:10:01 <anathema_db> sorry for being absent
14:10:05 <iwakeh> continuing:
14:10:22 <iwakeh> next week.
14:10:29 <karsten> great!
14:10:51 <karsten> alright. moving on?
14:10:55 <iwakeh> sure.
14:10:59 <karsten> * October tasks on the roadmap (karsten)
14:11:21 <karsten> 2016-10: Provide user-friendly documentation that empowers users to independently operate CollecTor instances. (Sponsor X 1.2. CollecTor)
14:11:24 <karsten> 2016-10: Enable CollecTor to synchronize Tor network data from other CollecTor instances. (Sponsor X 1.3. CollecTor)
14:11:27 <karsten> 2016-10: Set up a second CollecTor instance and enable synchronization with the first CollecTor instance. (Sponsor X 1.4. CollecTor)
14:11:42 <karsten> 1.3 and 1.4 are yours right now.
14:11:44 <iwakeh> the last two are in progress.
14:11:47 <iwakeh> right.
14:11:53 <karsten> I started looking into 1.2 yesterday.
14:12:04 <iwakeh> fine.
14:12:10 <karsten> which is the next topic on the agenda.
14:12:17 <karsten> so, I guess we have october under control for now.
14:12:23 <iwakeh> definitly.
14:12:30 <karsten> great!
14:12:37 <karsten> * Operating manuals (karsten)
14:12:45 <iwakeh> actually, just one.
14:12:51 <karsten> yes?
14:13:20 <iwakeh> unless you split for different
14:13:28 <karsten> ah, you mean just one manual?
14:13:39 <iwakeh> tasks like sync, and regular operation.
14:13:43 <iwakeh> yes.
14:13:51 <karsten> nah, one manual for collector is enough.
14:14:05 <iwakeh> huge volume :-)
14:14:09 <karsten> what I did was go one step back and look at how we're operating services more generally.
14:14:23 <karsten> which includes metrics, onionoo, and exonerator, too.
14:14:29 <karsten> painful experience.
14:14:37 <karsten> anyway, I started a latex document here:
14:14:41 <iwakeh> collector is easier.
14:14:58 <karsten> https://storm.torproject.org/shared/A5qfhTJQmArPRewCZAhpqpqsYSUKJ9K3AYc9czrtroy
14:14:59 <iwakeh> will be ...
14:15:09 <karsten> collector was comparatively easy, metrics is harder.
14:15:41 <karsten> I can send you a link that can edit, I just didn't want everyone here to help editing. ;)
14:15:42 * iwakeh waiting for the doc ..
14:16:03 <iwakeh> true :-)
14:16:12 <karsten> basically, I asked myself what questions future operators might have.
14:16:23 <iwakeh> it takes long for the LaTeX
14:16:34 <karsten> I can also upload a pdf somewhere.
14:16:43 <iwakeh> now it's there.
14:16:46 <karsten> ok!
14:17:15 * iwakeh reading ...
14:17:16 <karsten> so, the idea was to use this format as a working format, to compare operating the four services.
14:17:28 <karsten> and then produce or update manuals based on this doc.
14:17:34 <karsten> okay, waiting..
14:18:25 <iwakeh> I think it goes too far for CollecTor ...
14:18:32 <iwakeh> for example,
14:18:47 <iwakeh> we provide a signed verified download
14:19:03 <iwakeh> no need for ant and all the java packages for operation.
14:19:09 <karsten> I thought about that.
14:19:20 <iwakeh> no need to specify a unix user
14:19:23 <karsten> the issue is that the release is outdated.
14:19:29 <iwakeh> ?
14:19:40 <iwakeh> the current?
14:19:40 <karsten> for bridgedescs we need latest git.
14:19:49 <karsten> we should release more often. :)
14:19:52 <iwakeh> well, release
14:19:55 <iwakeh> right
14:19:56 <iwakeh> :-)
14:20:03 <karsten> and then simplify the instructions.
14:20:06 <iwakeh> no git for operation.
14:20:07 <karsten> but I see your point.
14:20:15 <karsten> yeah, that would be neat.
14:20:22 <karsten> not there yet.
14:20:28 <iwakeh> no ant and just plain jvm.
14:20:31 <karsten> I tried to capture the current state.
14:20:51 <iwakeh> collector is there.
14:21:02 <iwakeh> unless, you hide releases ;-)
14:21:09 <karsten> should I release?
14:21:16 <iwakeh> yes.
14:21:24 <iwakeh> it's merged and running.
14:21:37 <iwakeh> projects release worse things ...
14:21:38 <karsten> ah, I thought we were waiting for sync.
14:21:41 <karsten> hehehe
14:21:52 <karsten> okay, cool!
14:21:53 <iwakeh> minor version releases can
14:21:56 <iwakeh> fit in.
14:22:14 <iwakeh> operation would only need to replace
14:22:22 <iwakeh> the jar.
14:22:28 <karsten> yep!
14:22:34 <iwakeh> so all fine.
14:22:42 <karsten> okay, will release tomorrow.
14:22:46 <karsten> 1.1.0, right?
14:22:55 <iwakeh> 1.0.1
14:23:04 <karsten> # Changes in version 1.0.1 - 2016-08-22
14:23:09 <karsten> we already have an 1.0.1.
14:23:15 <iwakeh> 1.0.2
14:23:22 <karsten> okay.
14:23:27 <karsten> # Changes in version 1.1.0 - 2016-09-xx
14:23:31 <karsten> -> 1.0.2
14:23:38 <karsten> -> 1.0.2 - 2016-10-07
14:24:11 <karsten> alright, and then I can simplify instructions.
14:24:24 <iwakeh> that should be simplified.
14:24:35 <iwakeh> basically, we
14:24:54 <iwakeh> should stay away from all operating system related.
14:25:01 <iwakeh> after all it's a java
14:25:13 <iwakeh> application; platform independent.
14:25:25 <iwakeh> (well ...)
14:25:27 <karsten> uhhh..
14:25:34 <karsten> not there yet, I think.
14:25:46 <karsten> well, do you mean windows? :)
14:25:47 <iwakeh> apache is not part of
14:25:52 <iwakeh> the manual.
14:26:05 <iwakeh> I do not want to support m$ systems
14:26:12 <karsten> phew.
14:26:22 <iwakeh> but, the operating manual should abstract
14:26:26 <iwakeh> from the OS.
14:26:38 <iwakeh> as far as possible.
14:26:42 <karsten> as far as possible.
14:26:52 <iwakeh> there is only room to actually support debian.
14:27:05 <iwakeh> but, the abstraction helps reducing
14:27:15 <iwakeh> it such manuals to the minimum.
14:27:17 <karsten> regarding apache, I think we need to include that in the manual.
14:27:31 <karsten> we're somewhat apache-specific with .htaccess and header.html etc.
14:27:36 <iwakeh> Or, demand that operators know how to run that.
14:28:02 <iwakeh> there are only docs served
14:28:13 <iwakeh> no matter which http-server is used.
14:28:17 <karsten> that's a tough question: what can we demand operators to know vs. how many potentially great operators will be shy away?
14:28:22 <iwakeh> the apache related stuff could
14:28:28 <iwakeh> classify as example.
14:28:44 <iwakeh> i'd say look
14:29:01 <iwakeh> at the operators for my mirror and the main collector.
14:29:21 <iwakeh> what they know is minimum for such a service.
14:29:41 <iwakeh> not minimum, but in that  ..
14:29:46 <iwakeh> direction.
14:30:02 <iwakeh> otherwise the http server will be easy pray.
14:30:35 <karsten> oh, I agree that folks should know how to set up apache..
14:30:46 <iwakeh> ups, prey
14:30:54 <karsten> what we need to tell them, though, is what specifically we need from their apache setup.
14:30:59 <karsten> which files to serve.
14:31:10 <iwakeh> yes.
14:31:18 <karsten> which would ideally be in a single directory. but, not there yet.
14:31:18 <iwakeh> but not how.
14:31:29 <karsten> true, but the current doc doesn't do that.
14:31:35 <karsten> except suggest the package.
14:31:59 <karsten> I think that most operators would use that list to get an overview what we're going to do to their system.
14:32:03 <iwakeh> fine, I didn't complete reading yet.
14:32:10 <karsten> you want me to install an openjdk-7-jdk, good bye!
14:32:18 <karsten> things like that.
14:32:19 <iwakeh> or oracle
14:32:35 <iwakeh> oracle java that is.
14:32:54 <karsten> okay, I'll continue with metrics and then onionoo and exonerator.
14:32:59 <karsten> it's still a work in progress as you can see.
14:33:02 <iwakeh> for the task
14:33:15 <iwakeh> 1.2 for sponsorX
14:33:29 <iwakeh> we need a doc just about collector.
14:33:37 <iwakeh> but, this
14:33:43 <karsten> yes, but we need a very similar doc 1 month later.
14:33:49 <iwakeh> can be derived from the overview.
14:33:54 <iwakeh> true.
14:34:04 <iwakeh> have a LateX template.
14:34:24 <karsten> oh, I just picked latex because of the table support.
14:34:31 <karsten> we can easily use .txt or .md here.
14:34:35 <iwakeh> rather txt for
14:34:37 <karsten> for the actual manual.
14:34:41 <iwakeh> yes
14:34:53 <iwakeh> the overview is good.
14:35:06 <iwakeh> especially for us to device a
14:35:09 <karsten> I'll send you a link that can edit.
14:35:17 <iwakeh> generc setup as far as possible.
14:35:19 <iwakeh> fine.
14:35:31 <karsten> yes, I think the questions will be the same for all services.
14:35:52 <iwakeh> yes.
14:36:34 <karsten> okay, moving on?
14:36:39 <iwakeh> ok.
14:36:44 <karsten> * Shutting down globe.torproject.org (karsten)
14:36:56 <karsten> our sysadmins want to shut down globe.torproject.org.
14:37:06 <iwakeh> yes?
14:37:10 <karsten> right now it's serving javascript that redirects to atlas.
14:37:20 <iwakeh> what implications?
14:37:27 <karsten> which is in place for some months.
14:37:38 <karsten> implications of shutting it down? no redirects anymore.
14:37:47 <karsten> and one system less to take care of.
14:37:53 <karsten> well, system or part of a system.
14:38:03 <iwakeh> ok
14:38:17 <karsten> I'm inclined to say yes. any objections or concerns I didn't see?
14:38:20 <karsten> sad users?
14:38:26 <iwakeh> uh,
14:38:38 <iwakeh> what page will be served after shutdown?
14:38:47 <iwakeh> error?
14:39:10 <karsten> server not found I think.
14:39:17 <karsten> http://glboe.torproject.org/
14:39:22 <karsten> like that.
14:39:51 <karsten> maybe this is a more general question: how long do we keep redirects in place?
14:40:11 <iwakeh> well, the redirect should indicate
14:40:14 <karsten> another example: https://metrics.torproject.org/users.html
14:40:20 <karsten> that's in place for years now.
14:40:30 <karsten> which adds a tiny bit of complexity.
14:41:00 <iwakeh> rather have less silent redirects in future.
14:41:08 <iwakeh> telling people
14:41:17 <iwakeh> they should change their bookmarks.
14:41:40 <karsten> as static html page or specific status code or what?
14:41:46 <iwakeh> even with an expiry date.
14:41:55 <karsten> we could still do that with globe.torproject.org, I think.
14:41:59 <karsten> for a few weeks.
14:42:00 <iwakeh> static page with a link to the new.
14:42:13 <iwakeh> people have to notice in order to learn.
14:42:17 <iwakeh> :-)
14:42:32 <iwakeh> actually a usability question.
14:42:59 <karsten> okay, should we rewrite the globe redirect to make it more manual?
14:43:13 <karsten> and more informing? :)
14:43:19 <iwakeh> are there access numbers?
14:43:30 <karsten> let me check..
14:44:44 <karsten> 72 requests on oct 3.
14:44:59 <iwakeh> well, new static page would be good.
14:45:17 <iwakeh> the way it is now doesn't tell users that
14:45:23 <iwakeh> it is expired.
14:45:35 <karsten> would you want to rewrite what we have?
14:45:43 <karsten> warning: javascript.
14:45:46 <iwakeh> what is there?
14:46:04 <karsten> where is it in source control.........
14:46:11 <iwakeh> yes, I can do that.
14:46:35 <karsten> wget https://globe.torproject.org/
14:46:39 <iwakeh> it'll be plain ugly, so people notice ;-)
14:46:43 <karsten> not exactly source control, but heh.
14:47:03 <iwakeh> heh
14:47:29 <karsten> I think it's fine to keep that text and just suggest the link that users need to click themselves.
14:47:38 <karsten> maybe add an expiration date.
14:47:41 <karsten> oct 31.
14:47:45 <iwakeh> which?
14:47:48 <iwakeh> fine.
14:48:01 <karsten> okay, let me first check with the admins that they're fine keeping it for another four weeks.
14:48:09 <karsten> will get back to you as soon as I know.
14:48:16 <iwakeh> ok.
14:49:01 <karsten> how about I do the same with the metrics redirects?
14:49:20 <iwakeh> yes, good idea.
14:49:37 <karsten> https://collector.torproject.org/formats.html
14:49:43 <karsten> there we have a manual redirect.
14:50:01 * iwakeh waiting
14:50:07 <iwakeh> this looks fine.
14:50:42 <karsten> okay!
14:50:46 <karsten> out of topics.
14:50:52 <iwakeh> maybe, a hint about bookmarks.
14:50:53 <karsten> anathema_db: anything you'd like to add or ask?
14:51:07 <karsten> iwakeh: oh, sure, feel free to make the text even better.
14:51:12 <anathema_db> no folks, I was just reading and catching up
14:51:20 <anathema_db> hope to be on track soon
14:51:21 <karsten> I can then adapt the collector page. or you can fix that.
14:51:44 <iwakeh> for release 1.0.2
14:51:55 <iwakeh> you could add an
14:52:04 <iwakeh> expiration date there too.
14:52:10 <karsten> anathema_db: by the way, we recently wrote down some better hints for new contributors to metrics code bases.
14:52:24 <anathema_db> oh cool
14:52:29 * karsten finds them..
14:52:41 <anathema_db> :D
14:52:52 <karsten> unless, iwakeh, do you have a link faster than I?
14:53:22 <iwakeh> https://trac.torproject.org/projects/tor/wiki/org/teams/MetricsTeam/Volunteers
14:54:13 <anathema_db> great thanks
14:54:22 <karsten> anathema_db: if anything there doesn't make sense, or you have different questions, please let us know.
14:54:32 <iwakeh> and also the faq:
14:54:33 <iwakeh> https://trac.torproject.org/projects/tor/wiki/org/teams/MetricsTeam/Documentation
14:55:05 <anathema_db> karsten: sure
14:55:06 <karsten> iwakeh: would you want to send an updated collector redirect page that I put into the 1.0.2 release?
14:55:09 <iwakeh> there is a ticket number for questions and suggestions.
14:55:18 <iwakeh> can do.
14:55:22 <karsten> great!
14:55:59 <karsten> okay. anything else?
14:56:10 <iwakeh> no, all fine.
14:56:25 <karsten> okay, cool!
14:56:48 <karsten> thanks for a great meeting! talk to you next week, or earlier on trac.
14:56:53 <karsten> bye!
14:56:59 <iwakeh> bye, bye.
14:57:01 <karsten> #endmeeting