14:33:09 <karsten> #startmeeting metrics team
14:33:09 <MeetBot> Meeting started Thu Nov 16 14:33:09 2017 UTC.  The chair is karsten. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:33:09 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:33:16 <karsten> https://storm.torproject.org/shared/Ou-1QRctynWbF4yedi-MfDsjImFMFSIEP20fbVGCPRa <- agenda pad
14:33:21 <karsten> lots of topics.
14:33:27 <karsten> jump to #1?
14:33:39 <karsten> * Report on Atlas being rebranded and next steps (irl)
14:34:13 <irl_work> ok, so Atlas still has the old atlas.tpo URLs for now, but the theme is now the Tor Metrics theme, and it feels like it's integrated into the Tor Metrics website
14:34:22 <karsten> yay!
14:34:23 <iwakeh> I read some negative remars about the design.
14:34:32 <iwakeh> remarks,
14:34:35 <irl_work> it's also now renamed to Relay Search, which some people have started to refer to as RS, which I'm ok with
14:35:04 <irl_work> the response has been mostly good, on tor-relays@
14:35:12 <karsten> agreed.
14:35:22 <irl_work> there have been some comments about the large header that is part of the Tor Metrics design
14:35:27 <irl_work> there is a ticket for this
14:35:38 <karsten> yep. we could discuss that quickly here.
14:35:53 <iwakeh> I think this is the usual
14:35:55 <irl_work> ok, shall we do that first or shall i just list everything first
14:36:01 <karsten> #24277
14:36:08 <karsten> oh, please list everything first.
14:36:20 <iwakeh> terse technology overiew vs. web-site for everyone discussion.
14:36:29 <irl_work> ok, some problems occured where old versions of some files were cached
14:36:46 <irl_work> this has generally been resolved now, and i don't think will be an issue again
14:36:57 <irl_work> i mean, unless we do big changes again
14:37:08 <irl_work> the only other thing is some weirdness with iOS and content blocking
14:37:25 <irl_work> i don't have an iOS device and cannot actually debug this myself, and apparently the content block lists are secret
14:37:25 <iwakeh> oh?
14:37:54 <karsten> I do have an iOS device. are there instructions for reproducing the issue?
14:38:10 <irl_work> https://lists.torproject.org/pipermail/tor-relays/2017-November/013589.html
14:38:33 <irl_work> the report is from teor
14:38:38 <hiro> I have one too if needed I cna test
14:38:58 <karsten> can you put it into a ticket?
14:39:09 <irl_work> all resources are either from metrics.tpo or atlas.tpo so if there is an issue, then it's a bug in adblock more than anything else
14:39:11 <karsten> or is your hope that it will just go away?
14:39:21 <irl_work> so far teor is the only person to complain
14:39:30 <irl_work> doesn't mean he's the only one affected though
14:39:32 <karsten> maybe cc him on the ticket.
14:39:35 <karsten> right.
14:39:51 <irl_work> i'll make a ticket after the meeting
14:39:55 <karsten> cool.
14:40:09 <irl_work> ok, that's all the things, we can see which we want to discuss
14:40:23 <karsten> I think overall it went really well!
14:40:32 <karsten> there's always somebody to complain about something.
14:40:32 <irl_work> indeed (:
14:40:40 <irl_work> it went better than expected
14:40:41 <iwakeh> true
14:41:08 <karsten> I'd say let's work on the issues that now surfaced.
14:41:23 <karsten> if I can help with that, let me know. (like, testing something on iOS or high sierra.)
14:41:48 <irl_work> i think all the issues have tickets, except the iOS one
14:41:56 <karsten> ok.
14:41:59 <irl_work> i can CC you if testing is needed
14:42:05 <karsten> yes, please do.
14:42:10 <irl_work> (:
14:42:20 <karsten> so, regarding #24277, I made a suggestion on the ticket.
14:42:35 <irl_work> the smaller font size?
14:42:41 <karsten> " Agreed, the header occupies more space than necessary. You'll notice that that changes when you reduce the window height. Maybe we should use that header height for all window heights?"
14:42:50 <karsten> not even smaller font size, just smaller header.
14:42:59 <karsten> try to resize your window vertically.
14:43:06 <karsten> at some point the header gets smaller, too.
14:43:21 <karsten> but: maybe we should discuss this a little later.
14:43:27 <karsten> ( * Web design needs for Q4/Q1? (karsten))
14:43:30 <irl_work> ok
14:43:35 <iwakeh> yep.
14:43:50 <karsten> okay!
14:43:58 <karsten> next steps after fixing bugs?
14:44:09 <irl_work> for relay search?
14:44:13 <karsten> yes.
14:44:24 <iwakeh> integrate fully?
14:44:29 <irl_work> now that this is merged, it's easier to make other changes in the codebase
14:44:41 <irl_work> i plan to wait for embedded jetty in metrics web before the full integration
14:44:42 <karsten> okay. what's the schedule for fully integrating it?
14:44:46 <karsten> alright.
14:44:57 <karsten> which is another topic for today. :)
14:45:01 <karsten> later.
14:45:09 <iwakeh> :-)
14:45:12 <irl_work> before then, i plan to make sure all useful onionoo fields are in relay search details views
14:45:22 <irl_work> and close the rest of the metrics-2017 bugs on atlas
14:45:26 <karsten> whee!
14:45:38 <karsten> we're going to run out of tickets before we run out of 2017!!
14:45:44 <irl_work> i added exit addresses at the same time as merging the theme, for example
14:46:04 <irl_work> and the unmeasured "property"
14:46:19 <karsten> ok.
14:46:25 <karsten> soon we'll add a few other fields.
14:46:40 <karsten> but, let's see that when we're there.
14:46:50 <irl_work> yep
14:46:59 <karsten> okay, shall we move to the second topic then?
14:47:09 <irl_work> sounds good to me
14:47:18 <karsten> * caching issues when javascript changes, also applies to metrics.tpo (irl)
14:47:26 <karsten> anything we need to keep in mind here for metrics.tpo?
14:47:44 <karsten> or is that mostly an FYI?
14:47:45 <irl_work> so the problem here was that javascript files had changed and depended on new things in other javascript files
14:47:57 <irl_work> if you had one file cached but the new version of the other, everything broke
14:48:01 <irl_work> this is suboptimal
14:48:25 <irl_work> Sebastian suggested a cache breaker on the end of URLs so that when we update a file, we update the URL to indicate to browsers they should change
14:48:44 <irl_work> I merged a modified version of his patch that only adds this to relay search specific resources
14:48:56 <irl_work> I wanted to avoid having two cached versions of things that were loaded from metrics.tpo
14:49:19 <irl_work> Once we merge these together, I think it's sensible to have metrics.tpo also add the cache breaker when we update any JS there
14:49:34 <irl_work> as it would only be being loaded from a JSP and we only change in one place
14:49:35 <karsten> sounds reasonable.
14:49:41 <iwakeh> yep, makes sense.
14:49:53 <irl_work> in theory, apache is doing the right things for caching
14:49:56 <karsten> luckily we don't change the JS very often.
14:50:00 <irl_work> in practice, browsers often do the wrong thing
14:50:16 <irl_work> those with recent firefox/chrome/tor browser did not see issues
14:50:18 <irl_work> safari did
14:50:23 <karsten> ah ok.
14:50:59 <karsten> alright, let's try to keep this in mind for whenever we change the JS next time.
14:51:02 <irl_work> but yes, just something to keep in mind, and i think for now it is addressed
14:51:07 <karsten> ok.
14:51:15 <karsten> * onionperf to be managed by Salt Stack (hiro)
14:51:21 <hiro> yep
14:51:46 <hiro> so I thought I might have to restart the onionperf process to do this, but actually it is not needed
14:52:02 <hiro> so we  shouldn't  have any problems w the measurements
14:52:15 <karsten> is salt stack a new thing we're using on tor hosts?
14:52:32 <hiro> I do not think we are using it on otheer hosts
14:52:42 <karsten> why not?
14:52:45 <hiro> on  tpo we use puppet
14:53:04 <hiro> but onionperf deployments are not tpo
14:53:07 <karsten> and that is insufficient here?
14:53:15 <karsten> (note how little I know about all this...)
14:53:44 <hiro> I figured it would be easier to manage them like this instead  of integrating them into the broader puppet that weasel maintains for all the hosts
14:54:23 <irl_work> i think this does make sense
14:54:24 <hiro> I wanted to have a way to update all the op deployments at the samee time
14:54:27 <iwakeh> how do these differ salt vs. puppet?
14:54:33 <iwakeh> roughly
14:54:47 <irl_work> salt is masterless or agentless?
14:54:56 <hiro> iwakeh puppet is a bit more complex to setup. but they do not differe that mcuh
14:55:25 <iwakeh> ah
14:55:55 <karsten> which of the two lets us achieve our goal of having 2 operators per service in the near future?
14:56:34 <iwakeh> is salt stack open/free sw?
14:56:42 <irl_work> i think we would have problems managing non-tpo hosts from the puppet, and we would need more infrastructure around a puppet based deployment
14:56:44 <hiro> uhm the services and how they are setup will be available to everyone that wants to manage op
14:56:51 <irl_work> salt stack is foss, and is used in qubes also
14:57:09 <iwakeh> ok
14:57:13 <hiro> yes it is opensource and in a way more "lightweight" to setup than puppet
14:57:26 <irl_work> i also had a masters student who ran pathspider (internet measurement tool) using salt stack in the past
14:57:55 <karsten> sounds like we should get irl_work to be the second operator then. :)
14:57:59 <irl_work> oops
14:58:06 <hiro> salt need a master and then you can configur the minions to b managed by the master
14:58:14 <hiro> this is pretty similar to puppet
14:58:27 <iwakeh> ok
14:58:34 <hiro> sure I have no problems w/ that
14:58:39 <karsten> okay. makes sense so far.
14:58:53 <hiro> in fact I discussd the epossibility to have op as a salt deployment w irl in amsterdam
14:59:12 <irl_work> i was going to say, i remember a discussion
14:59:17 <hiro> yes
14:59:47 <irl_work> one big benefit is consistency across the hosts
14:59:57 <irl_work> you know you have the same set up on all of them
15:00:03 <karsten> sounds good.
15:00:08 <hiro> yes but also if we want to have more hosts we do not want to update/modify them 1-by-1
15:00:10 <karsten> speaking of onionperf, there's this other ticket...
15:00:55 <karsten> #24018
15:01:00 <karsten> (finally found it)
15:01:17 <irl_work> this seems like an onionperf development question
15:01:22 <irl_work> as opposed to deployment
15:01:30 <hiro> yes that is something that I am working on
15:01:45 <karsten> development and then deployment, yes.
15:02:03 <karsten> though I think that rob would do the dev part as soon as he knows it's the right thing.
15:02:17 <irl_work> ah ok
15:02:23 <hiro> so there are two possibilities and one is looking at the log  and find out when is there a connection error
15:02:54 <hiro> and send an email to alert that there are  connection errors
15:03:39 <karsten> to be honest, I didn't look enough at the issue to have a useful discussion here.
15:04:02 <karsten> hiro and/or irl_work: would you like to help move that ticket forward?
15:04:27 <hiro> sure
15:04:39 <karsten> great!
15:04:43 <irl_work> i can take a look at the ticket and maybe comment, if that helps
15:04:49 <karsten> it might!
15:05:15 <irl_work> ok
15:05:31 <irl_work> hiro: what needs to happen to add op-ab into the salt setup?
15:05:50 <hiro> connect it to the master
15:06:16 <irl_work> ok, and then there is also a second box that has some kernel config changed that shouldn't ever go to collector but is useful for my research
15:06:21 <irl_work> i guess that just gets connected too
15:06:29 <irl_work> onionperf isn't modified, just the kernel stack
15:06:30 <hiro> and also see if my s etup of op is similar to yours or we should have a separated set of rules
15:06:45 <irl_work> i mean, currently they are blank debian 9 boxes
15:06:51 <hiro> I am not managing w salt much more than the op installation
15:06:51 <irl_work> i lost my zpool
15:07:13 <irl_work> all the storage backing all the virtualisation vanished in the space of 20 mins
15:07:37 <irl_work> but this sounds good
15:07:47 <karsten> great!
15:07:55 <karsten> shall we move on?
15:08:09 <hiro> karsten regarding monitoring op I wanted to try out something like https://prometheus.io/docs/visualization/grafana/ which in the end will be similar to have nagios checks
15:08:23 <irl_work> yay
15:08:32 <irl_work> karsten: i told you prometheus was cool
15:08:37 <hiro> my plan was to do that after salt
15:08:42 <karsten> sounds good! we still need to move forward with the notifications topic.
15:08:53 <hiro> sure
15:08:54 <karsten> but would be neat to see it deployed for this case.
15:09:04 <iwakeh> +1
15:09:23 <karsten> hiro: want to add an action item or two to the pad?
15:09:28 <hiro> sure
15:09:40 <karsten> cool. next topic?
15:10:00 <iwakeh> ok
15:10:21 <karsten> * ExoneraTor integration into Tor Metrics next steps (karsten)
15:10:29 <karsten> so, exonerator runs with jetty!
15:10:34 <karsten> thanks to iwakeh!
15:10:35 <iwakeh> it's running.
15:10:40 <iwakeh> hehe :-)
15:10:42 <irl_work> woo
15:10:45 <karsten> yes, it's deployed.
15:10:50 <iwakeh> isn't it easier now?
15:10:57 <karsten> it is. :)
15:11:00 <iwakeh> :-)
15:11:08 <karsten> we knew that. but we also knew it wouldn't be easy to get there.
15:11:21 <iwakeh> oh, well ...
15:11:32 <karsten> next step: integration into Tor Metrics.
15:11:48 <karsten> I was wondering: should we take a similar approach like atlas and do the integration in two steps?
15:12:06 <iwakeh> Here it could just be done in one step.
15:12:28 <irl_work> if the integration happened now in one step, wouldn't we go back to not jetty?
15:12:32 <iwakeh> There using the same basis , i.e., jetty
15:12:42 <karsten> well, after metrics-web runs with jetty.
15:12:45 <irl_work> ah ok
15:12:58 <iwakeh> metrics-web will run on jetty.
15:13:14 <iwakeh> (different topic later)
15:13:15 <irl_work> i did make a number of changes to get the theme to work by putting in a bunch of absolute urls
15:13:18 <karsten> then we only need to resolve the issue with exonerator's queryresponse being part of metrics-lib.
15:13:31 <karsten> I see.
15:13:56 <iwakeh> or stay in metrcis-web?
15:13:57 <karsten> okay. one step then.
15:14:05 <karsten> hmm?
15:14:17 <iwakeh> if exonerator is metrics-web
15:14:31 <karsten> ugh.
15:14:36 <iwakeh> there is only one module using the QueryResult.
15:14:39 <karsten> that means putting the database on the same host.
15:14:49 <karsten> I'm thinking of the deployment side here.
15:14:53 <iwakeh> db can sit anywhere in the infrastructure.
15:15:34 <iwakeh> maybe, that is for the integration ticket discussion?
15:15:44 <karsten> ok.
15:16:21 <iwakeh> actually,
15:16:36 <iwakeh> are we now going to have ExoneraTor releases, too?
15:16:57 <karsten> we should, probably.
15:17:13 <karsten> so, I heard that there might be value in having exonerator as standalone tool.
15:17:33 <karsten> for people who'd rather install it locally to keep queries local.
15:17:46 <iwakeh> now it's possible.
15:17:46 <karsten> not the primary use case.
15:17:52 <irl_work> can the backend be a standalone tool and you just query it from Tor Metrics
15:17:53 <karsten> yes.
15:17:59 <irl_work> like relay search/onionoo
15:18:04 <iwakeh> yep.
15:18:26 <karsten> yes, we even added a json interface lately.
15:18:37 <iwakeh> undocumented.
15:18:40 <karsten> hehe
15:18:43 <karsten> okay, move to ticket?
15:18:48 <iwakeh> yep.
15:19:08 <karsten> * Onionoo release this week? (karsten)
15:19:18 <iwakeh> which milestone & tickets?
15:19:28 <iwakeh> https://trac.torproject.org/projects/tor/query?status=accepted&status=assigned&status=closed&status=merge_ready&status=needs_information&status=needs_review&status=needs_revision&status=new&status=reopened&component=Metrics%2FOnionoo&milestone=%5EOnionoo-1.7.0&order=priority
15:19:30 <karsten> so, we have an upcoming major protocol update.
15:19:50 <karsten> but we need to put out a minor version to start announcing it via the next_scheduled_major_version thing.
15:20:09 <karsten> I was thinking to just put out what we have now, plus trivial things maybe.
15:20:21 <iwakeh> ok, so just what is ready to be released and move the rest?
15:20:23 <karsten> but just to have the next_scheduled_who_picked_this_long_name thing.
15:20:30 <iwakeh> makes sense.
15:20:39 <karsten> yes, like that.
15:20:42 <karsten> move the rest.
15:20:53 <karsten> except for things that take < 30 minutes.
15:21:10 <karsten> maybe we can take a look at the open tickets after the meeting?
15:21:11 <iwakeh> yes, all that takes less time than moving the milestone ;-)
15:21:15 <karsten> hehe
15:21:18 <karsten> indeed!
15:21:20 <iwakeh> yes
15:21:48 <karsten> okay!
15:21:56 <karsten> next topic?
15:22:19 <karsten> * Blog post plans for Atlas integration, metrics timeline, ExoneraTor integration, etc.? (karsten)
15:22:29 <karsten> I was asked by stephw whether we want to put out a blog post.
15:22:42 <iwakeh> quite a bit to tell
15:22:44 <karsten> not this week, not next week.
15:22:59 <karsten> but, do we want to plan something like that?
15:23:13 <irl_work> i nearly wrote a blog post about relay search
15:23:19 <irl_work> then i didn't
15:23:24 <karsten> plus maybe metrics-bot.
15:23:50 <karsten> should I tell her we're interested?
15:24:13 <irl_work> yes, but more in a before 2018 way than in a this week way
15:24:23 <karsten> agreed.
15:24:45 <irl_work> i think i'd like to do a blog post once relay search has 100% coverage of onionoo fields
15:25:03 <karsten> sounds good.
15:25:32 <karsten> alright.
15:25:35 <karsten> next topic?
15:25:43 <irl_work> ok
15:25:47 <karsten> * metrics-web restructuring and jetty (iwakeh)
15:26:03 <iwakeh> yep
15:26:12 <iwakeh> metrics-web runs fine on/with jetty;
15:26:21 <karsten> yay!
15:26:22 <iwakeh> I'm currently testing the restructuring (move to metrics-base).
15:26:32 <iwakeh> that is a lot, too.
15:26:40 <karsten> how can I help?
15:26:48 <iwakeh> I think I post the patch branch tomorrow for testing.
15:26:53 <karsten> ok.
15:26:57 <iwakeh> Fixed some minor issues that came up running metrics-web from scratch w/o data.
15:27:28 <iwakeh> There are very minor things left concerning R scripting/paths/dependencies.
15:27:36 <karsten> ok. :)
15:27:38 <iwakeh> I'll write these up on the ticket.
15:27:59 <karsten> great! will prioritize the review/testing.
15:28:04 <irl_work> iwakeh: please let me know if you need any testing for running it from scratch
15:28:09 <irl_work> as i need to do that anyway
15:28:21 <iwakeh> I'll post on the other ticket
15:28:32 <iwakeh> that you opened earlier.
15:28:37 <irl_work> ok (:
15:28:44 <iwakeh> for testing web-stuff
15:28:56 <iwakeh> integration the patch will work fine.
15:29:22 <karsten> last topic for today?
15:29:32 <karsten> (sorry for rushing!)
15:29:34 <iwakeh> so, integrating RS will be a good test.
15:29:38 <iwakeh> fine.
15:29:52 <karsten> * Web design needs for Q4/Q1 (#24277 and others)? (karsten)
15:30:09 <karsten> I was wondering whether we should get help from our/a web designer.
15:30:24 <irl_work> i have some opinions on information architecture
15:30:25 <karsten> we're changing the website quite a bit.
15:30:31 <irl_work> but i can put these in the ticket
15:30:35 <iwakeh> not that much accumulated, yet?
15:30:46 <karsten> information architecture?
15:30:59 <karsten> ah, you can put them in a ticket. sure!
15:31:00 <irl_work> hierachical organisation of information to let people discover the things they need
15:31:12 <karsten> yes, that would be useful in a ticket.
15:31:13 <irl_work> the two navs don't make much sense to me
15:31:32 <irl_work> i can see what the original idea was, but as we get more things in Tor Metrics it needs rethinking
15:31:35 <irl_work> but yes, ticket
15:31:38 <karsten> agreed!
15:31:49 <karsten> regarding accumulating things,
15:31:57 <karsten> we have the atlas integration and exonerator integration,
15:32:04 <karsten> we have metrics timeline stuff on graph pages,
15:32:08 <iwakeh> web stuff parent ticket?
15:32:11 <karsten> we have too much purple on the banner ;).
15:32:16 <iwakeh> hehe
15:32:34 <karsten> so, maybe half as much work as last year.
15:32:56 <karsten> I'm just thinking that the alternative is that we're doing it ourselves, which takes more time and produces half as good results.
15:33:18 <karsten> but maybe we can ask the UX team for help.
15:33:32 <iwakeh> maybe also a minor modernization.
15:33:39 <iwakeh> reduction of js.
15:33:42 <karsten> so, should we collect some ideas on such a web stuff parent ticket?
15:33:47 <karsten> yes and yes.
15:33:50 <irl_work> this sounds good
15:33:57 <iwakeh> and, collect the relevant tickets there, too.
15:34:00 <karsten> and then next week I bring it up at the vegas team meeting.
15:34:03 <karsten> yes.
15:34:47 <karsten> great!
15:34:54 <karsten> uff.
15:34:59 <karsten> I think we're done.
15:35:03 <iwakeh> through.
15:35:15 <irl_work> (:
15:35:29 <karsten> alright! :) so much to do. let's talk more next week!
15:35:47 <karsten> thanks! bye, bye.
15:35:50 <irl_work> bye!
15:35:52 <iwakeh> fine, bye bye
15:36:00 <karsten> #endmeeting