14:59:37 <karsten> #startmeeting metrics team
14:59:37 <MeetBot> Meeting started Thu Oct 17 14:59:37 2019 UTC.  The chair is karsten. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:59:37 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
15:01:18 * karsten added topics to the agenda
15:01:44 <karsten> anything else for the agenda?
15:02:29 <karsten> gaba: are you around?
15:02:30 <gaba> hi!
15:02:32 <karsten> heh
15:02:35 <karsten> hi!
15:02:37 * gaba arriving :)
15:02:38 <irl> no more topics from me
15:03:23 * karsten waits a minute for gaba to add topics and then starts.
15:03:31 <gaba> agenda in https://storm.torproject.org/shared/5h1Goax5eNusxjXJ_Ty5Wl7hFR1uqCReUiN8xdlBG8T
15:03:42 <gaba> nothing from me
15:03:50 <karsten> okay, great!
15:03:51 <karsten> Tor Browser measurements (karsten)
15:03:54 <karsten> sysrqb: are you around?
15:04:36 <karsten> this topic makes more sense with sysrqb around. let's continue with the next and put this one to the end.
15:04:48 <karsten> OONI metadata database (karsten)
15:04:57 <karsten> I created #32126 with my results.
15:05:21 <karsten> I'd like to put this on hold for now and resume in a couple weeks or so.
15:05:42 <karsten> the EC2 instance is still running, though, and if there's something I can do quickly, I'd do it tomorrow.
15:06:04 <karsten> irl: maybe take a brief look at that ticket, if you can, and tell me your thoughts.
15:06:13 <irl> yeah, we should also ask anti-censorship to look at it
15:06:37 <karsten> yes, absolutely, but that might not happen as quickly.
15:06:47 <karsten> I just don't want to keep the instance running for weeks.
15:06:48 <sysrqb> karsten: ah, sorry, yes! (i'll lurk for now)
15:07:07 <karsten> sysrqb: hi!
15:07:22 <sysrqb> hi!
15:07:37 <karsten> so, regarding #32126, I could also imagine leaving it where it is for now and doing something else.
15:08:10 <karsten> I'll schedule termination of that ec2 instance for tomorrow afternoon.
15:08:25 <irl> yes that sounds fine, there is enough to discuss what the next steps are
15:08:52 <karsten> btw, who else should we cc on that ticket?
15:09:10 <karsten> and are there team handles, or do we just cc people one by one?
15:09:27 <irl> i think phw is probably best
15:09:39 <irl> and he can add others if there are others
15:09:48 <karsten> yes, okay.
15:09:57 <karsten> alright!
15:10:05 <karsten> moving on?
15:10:17 <irl> ok
15:10:20 <karsten> Tor Browser measurements (karsten)
15:10:49 <karsten> sysrqb: I saw emails being exchanged.
15:10:54 <sysrqb> Yes
15:10:59 <sysrqb> we made some progress
15:11:44 <sysrqb> I suspect no one will make more progress on this until November
15:11:58 <karsten> okay.
15:12:01 <sysrqb> Dennis is working on his thesis and the browser team is overloaded right now
15:12:21 <karsten> should we put this on the agenda for some time mid-november?
15:12:33 <gaba> yes
15:12:34 <sysrqb> but Dennis is interested in helping us setup our measurement environment
15:12:38 <sysrqb> yes, that sounds good
15:13:11 <karsten> okay, great. made a note.
15:13:18 <sysrqb> thanks
15:13:31 <karsten> thanks for starting this!
15:13:56 <sysrqb> you're welcome, glad we can make some small progress
15:14:19 <karsten> cool! moving on.
15:14:25 <karsten> Onionoo release, new deployments, automation and documentation (irl)
15:14:50 <karsten> thanks for reviewing #31941.
15:14:59 <karsten> I can put out a new release tomorrow morning.
15:15:02 <irl> this is not quite done, because i did one day of exit scanner this week, but that's helpful because we need the release before i can finish this
15:15:43 <irl> i have an ansible playbook that picks up where tpa puppet leaves off for the backend hosts, and installs onionoo with the two users, running the processes as systemd services
15:16:14 <irl> we can also upgrade by changing the version number in the playbook, and running it again (although currently you have to stop onionoo first, i haven't got it to detect yet when the updater is done)
15:16:32 <irl> is it reasonable to assume the updater is done when it prints the done log message and that's the last thing in the log?
15:17:07 <karsten> I don't remember exactly which one the done log message is, but yes, when it says it's done, it's done.
15:17:22 <irl> ok, so i'll have it so it refuses to run if that's not the last log message
15:17:35 <karsten> that should work.
15:17:56 <irl> i've done a little more on https://help.torproject.org/metrics/ops/onionoo-ops/ but will still need a chunk of time next week to finish that off
15:18:06 <irl> which can happen while i'm copying over onionoo out directory to the new backend
15:18:28 <irl> oh, there is one other question
15:18:29 <karsten> how do we do this deployment?
15:18:34 <karsten> yes, go ahead.
15:18:41 <irl> how are you managing sudo passwords?
15:18:58 <irl> the ansible script i have assumes that one of two things is happening
15:19:11 <irl> either your sudo password is the same for everything, or you use pass in exactly the same way that I do
15:19:44 <karsten> there's just one sudo password for everything, right?
15:19:53 <irl> there can be
15:19:58 <irl> you can't use your LDAP password anymore
15:20:07 <irl> you have to set a sudo password at db.torproject.org
15:20:15 <karsten> I think I did, long ago.
15:20:28 <irl> but if you set a password for all hosts, then you can use option 1 and it will work
15:21:04 <irl> i think for the deployment of the release you do tomorrow, the existing hosts can be manually upgraded
15:21:13 <karsten> okay.
15:21:30 <irl> i have fixed some security issues in the deployments related to the writing of logs, which means that you can't use this script on the existing hosts
15:21:46 <karsten> when do we switch to the new deployment method?
15:21:47 <irl> i think we are replacing the old hosts with all new hosts though
15:21:53 <karsten> ah!
15:22:11 <irl> so we switch them as the new hosts appear
15:22:13 * phw sees his nickname mentioned and reads backlog
15:22:16 <karsten> so, create new host, install, make available, turn off old host.
15:22:20 <irl> yep
15:22:38 <karsten> phw: hi! this was about #32126. maybe take a quick look at that, if you can.
15:22:44 <irl> for the second host i'll give you the instructions and then we see how far it goes
15:22:48 <karsten> hehe
15:23:11 <karsten> when are we going to create new hosts? should we just do it together?
15:23:33 <irl> i'll make a ticket with tpa next week to ask for the hosts to be created
15:23:46 <irl> once it is created we can pick a time
15:23:47 <karsten> please ask for more disk space while you're at it.
15:23:58 <irl> ok
15:24:05 <karsten> okay, sounds good!
15:24:25 <irl> that's all for this topic
15:24:27 <karsten> so, I'm going to put out an onionoo release and deploy it on both instances tomorrow?
15:24:32 <irl> yep
15:25:24 <karsten> great, moving on:
15:25:27 <karsten> Exit scanner (gaba)
15:26:19 <gaba> ok, mostly I want to check that we are on track
15:26:30 <gaba> irl: you said you worked this week on this?
15:26:31 <irl> i'd say we are 2 weeks behind
15:26:32 <gaba> which ticket?
15:26:52 <irl> i made pathspider run on debian 10
15:27:19 <irl> there's not really a ticket, it's more dusting stuff off to get it ready to start on tickets
15:27:26 <gaba> ok
15:28:11 <karsten> should we make a project plan at some point?
15:28:30 <irl> yes
15:28:58 <irl> i think i will do this in a trello while i'm still thinking about it
15:29:05 <irl> there's a lot of state to restore from swap
15:29:25 <karsten> sounds good. want to invite gaba and me to that board?
15:29:56 <irl> done
15:30:03 <gaba> thanks
15:30:10 <karsten> great!
15:31:23 <irl> anything else for this topic?
15:31:26 <phw> karsten: the graphs are neat! just yesterday, i was wondering what timeout to pick when automating a tor bootstrap and the graphs are very helpful in deciding
15:31:43 <karsten> phw: glad to hear. :)
15:31:59 <karsten> irl: is there anything we can promise (to ourselves) to do until next thursday?
15:32:20 <irl> add onionoo-backend-01.torproject.org to the onionoo.tpo rotation
15:33:07 <karsten> okay.
15:33:07 <irl> have a plan we can discuss, and a list of reasons why the new exit scanner doesn't work yet, as opposed to not existing
15:33:25 <irl> i think those two we can do
15:33:34 <gaba> irl: the scope for the exit scanner is:
15:33:34 <gaba> . New version of exit list format
15:33:34 <gaba> . Get an MVP with
15:33:35 <gaba> - written in python - support por IPv6 - support for new exit list format
15:33:38 <gaba> right?
15:34:06 <irl> initial scope is: MVP in python that produces the same exit list format as before
15:34:18 <gaba> ok
15:34:51 <gaba> ticket #29650
15:34:59 <gaba> oops, not that one
15:35:00 <gaba> exactly
15:35:14 <irl> i think some of the tickets may not be what we really want to do
15:35:20 <gaba> ok
15:35:32 <irl> i'll work on this in the trello to make sure we are doing what we want, and making the best use of the time
15:35:45 <irl> as i said, we were meant to start on this 2 weeks ago
15:35:56 <irl> but debian 10 release means other things started falling behind too
15:36:35 <gaba> sounds good
15:36:42 <karsten> okay, shall we move on?
15:36:45 <irl> ok
15:37:00 <karsten> Tor metrics data portal next steps (karsten)
15:37:31 <karsten> is there anything we need to do here in the next few weeks?
15:37:46 <karsten> the project was approved, as I hear.
15:38:11 <gaba> nothing on our side. Simple secure is going to be working on it
15:38:26 <karsten> okay, good.
15:38:28 <gaba> I will see them next week at mozfest and can check on timeline
15:39:03 <karsten> okay.
15:39:06 <irl> cool
15:39:28 <karsten> Review of #19332 (karsten)
15:39:57 <karsten> do you have an ETA for this?
15:40:00 <irl> i started this, but did not finish
15:40:15 <irl> it's probably ok
15:40:23 <karsten> if this takes longer, I'd check that we're not losing data at the moment.
15:40:34 <karsten> because of rsync deleting old data, or something.
15:40:56 <irl> i didn't fully go over all of the test cases yet
15:41:03 <irl> but the code itself looked good
15:41:20 <irl> i think it's ok to merge it and i can review the test cases retroactively
15:41:27 <karsten> okay.
15:41:40 <karsten> I'll merge and release and deploy that tomorrow then.
15:41:47 <irl> cool, sorry for the delay
15:42:13 <karsten> no worries. thanks!
15:42:30 <karsten> next:
15:42:34 <karsten> Roadmap - how are we doing? (gaba)
15:43:02 <gaba> it seems that we are fine here
15:43:04 <karsten> the OONI metadata database is now #32126.
15:43:10 <gaba> we need plan for exit scanner and that is the next thing
15:43:10 <karsten> sort of.
15:43:29 <karsten> which I'd want to move to the backlog for now.
15:43:45 <irl> ok
15:43:59 <gaba> ok
15:44:28 <irl> for me, onionoo hosts are still in progress, but i'm going to also be working on the exit scanner project bootstrap next week which will result in a plan
15:44:37 <irl> and maybe even some code
15:45:12 <karsten> I just put #19332 back, for merging and releasing and deploying.
15:46:20 <karsten> I also moved a card from the icebox to backlog. this is related to #32126. well, it's that ticket now.
15:46:29 <karsten> let me consolidate that.
15:46:50 <irl> sounds good
15:47:11 <gaba> sounds good
15:48:24 <karsten> I wonder if I should resume working on #31204 now.
15:48:44 <karsten> as there's not much I can do for the exit scanner project, yet.
15:49:05 <karsten> or is there anything else that needs my attention?
15:49:19 <irl> i think #31204 looks good
15:49:51 <irl> one thing i have noticed is that we're using a whole load of really deprecated stuff in the jetty we are using
15:49:52 <gaba> yes
15:50:24 <karsten> ah, I can check that. can you create a ticket for that, with examples?
15:50:47 <irl> ok
15:50:57 * karsten creates a placeholder card.
15:52:05 <karsten> alright. anything else for the roadmap topic?
15:52:26 <irl> not from me
15:52:29 <gaba> nop
15:52:35 <karsten> next meeting next week as usual?
15:52:39 <irl> yep
15:53:19 <karsten> great! have a good day/evening. bye! o/
15:53:24 <karsten> #endmeeting