14:59:37 <karsten> #startmeeting metrics team 14:59:37 <MeetBot> Meeting started Thu Oct 17 14:59:37 2019 UTC. The chair is karsten. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:59:37 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic. 15:01:18 * karsten added topics to the agenda 15:01:44 <karsten> anything else for the agenda? 15:02:29 <karsten> gaba: are you around? 15:02:30 <gaba> hi! 15:02:32 <karsten> heh 15:02:35 <karsten> hi! 15:02:37 * gaba arriving :) 15:02:38 <irl> no more topics from me 15:03:23 * karsten waits a minute for gaba to add topics and then starts. 15:03:31 <gaba> agenda in https://storm.torproject.org/shared/5h1Goax5eNusxjXJ_Ty5Wl7hFR1uqCReUiN8xdlBG8T 15:03:42 <gaba> nothing from me 15:03:50 <karsten> okay, great! 15:03:51 <karsten> Tor Browser measurements (karsten) 15:03:54 <karsten> sysrqb: are you around? 15:04:36 <karsten> this topic makes more sense with sysrqb around. let's continue with the next and put this one to the end. 15:04:48 <karsten> OONI metadata database (karsten) 15:04:57 <karsten> I created #32126 with my results. 15:05:21 <karsten> I'd like to put this on hold for now and resume in a couple weeks or so. 15:05:42 <karsten> the EC2 instance is still running, though, and if there's something I can do quickly, I'd do it tomorrow. 15:06:04 <karsten> irl: maybe take a brief look at that ticket, if you can, and tell me your thoughts. 15:06:13 <irl> yeah, we should also ask anti-censorship to look at it 15:06:37 <karsten> yes, absolutely, but that might not happen as quickly. 15:06:47 <karsten> I just don't want to keep the instance running for weeks. 15:06:48 <sysrqb> karsten: ah, sorry, yes! (i'll lurk for now) 15:07:07 <karsten> sysrqb: hi! 15:07:22 <sysrqb> hi! 15:07:37 <karsten> so, regarding #32126, I could also imagine leaving it where it is for now and doing something else. 15:08:10 <karsten> I'll schedule termination of that ec2 instance for tomorrow afternoon. 15:08:25 <irl> yes that sounds fine, there is enough to discuss what the next steps are 15:08:52 <karsten> btw, who else should we cc on that ticket? 15:09:10 <karsten> and are there team handles, or do we just cc people one by one? 15:09:27 <irl> i think phw is probably best 15:09:39 <irl> and he can add others if there are others 15:09:48 <karsten> yes, okay. 15:09:57 <karsten> alright! 15:10:05 <karsten> moving on? 15:10:17 <irl> ok 15:10:20 <karsten> Tor Browser measurements (karsten) 15:10:49 <karsten> sysrqb: I saw emails being exchanged. 15:10:54 <sysrqb> Yes 15:10:59 <sysrqb> we made some progress 15:11:44 <sysrqb> I suspect no one will make more progress on this until November 15:11:58 <karsten> okay. 15:12:01 <sysrqb> Dennis is working on his thesis and the browser team is overloaded right now 15:12:21 <karsten> should we put this on the agenda for some time mid-november? 15:12:33 <gaba> yes 15:12:34 <sysrqb> but Dennis is interested in helping us setup our measurement environment 15:12:38 <sysrqb> yes, that sounds good 15:13:11 <karsten> okay, great. made a note. 15:13:18 <sysrqb> thanks 15:13:31 <karsten> thanks for starting this! 15:13:56 <sysrqb> you're welcome, glad we can make some small progress 15:14:19 <karsten> cool! moving on. 15:14:25 <karsten> Onionoo release, new deployments, automation and documentation (irl) 15:14:50 <karsten> thanks for reviewing #31941. 15:14:59 <karsten> I can put out a new release tomorrow morning. 15:15:02 <irl> this is not quite done, because i did one day of exit scanner this week, but that's helpful because we need the release before i can finish this 15:15:43 <irl> i have an ansible playbook that picks up where tpa puppet leaves off for the backend hosts, and installs onionoo with the two users, running the processes as systemd services 15:16:14 <irl> we can also upgrade by changing the version number in the playbook, and running it again (although currently you have to stop onionoo first, i haven't got it to detect yet when the updater is done) 15:16:32 <irl> is it reasonable to assume the updater is done when it prints the done log message and that's the last thing in the log? 15:17:07 <karsten> I don't remember exactly which one the done log message is, but yes, when it says it's done, it's done. 15:17:22 <irl> ok, so i'll have it so it refuses to run if that's not the last log message 15:17:35 <karsten> that should work. 15:17:56 <irl> i've done a little more on https://help.torproject.org/metrics/ops/onionoo-ops/ but will still need a chunk of time next week to finish that off 15:18:06 <irl> which can happen while i'm copying over onionoo out directory to the new backend 15:18:28 <irl> oh, there is one other question 15:18:29 <karsten> how do we do this deployment? 15:18:34 <karsten> yes, go ahead. 15:18:41 <irl> how are you managing sudo passwords? 15:18:58 <irl> the ansible script i have assumes that one of two things is happening 15:19:11 <irl> either your sudo password is the same for everything, or you use pass in exactly the same way that I do 15:19:44 <karsten> there's just one sudo password for everything, right? 15:19:53 <irl> there can be 15:19:58 <irl> you can't use your LDAP password anymore 15:20:07 <irl> you have to set a sudo password at db.torproject.org 15:20:15 <karsten> I think I did, long ago. 15:20:28 <irl> but if you set a password for all hosts, then you can use option 1 and it will work 15:21:04 <irl> i think for the deployment of the release you do tomorrow, the existing hosts can be manually upgraded 15:21:13 <karsten> okay. 15:21:30 <irl> i have fixed some security issues in the deployments related to the writing of logs, which means that you can't use this script on the existing hosts 15:21:46 <karsten> when do we switch to the new deployment method? 15:21:47 <irl> i think we are replacing the old hosts with all new hosts though 15:21:53 <karsten> ah! 15:22:11 <irl> so we switch them as the new hosts appear 15:22:13 * phw sees his nickname mentioned and reads backlog 15:22:16 <karsten> so, create new host, install, make available, turn off old host. 15:22:20 <irl> yep 15:22:38 <karsten> phw: hi! this was about #32126. maybe take a quick look at that, if you can. 15:22:44 <irl> for the second host i'll give you the instructions and then we see how far it goes 15:22:48 <karsten> hehe 15:23:11 <karsten> when are we going to create new hosts? should we just do it together? 15:23:33 <irl> i'll make a ticket with tpa next week to ask for the hosts to be created 15:23:46 <irl> once it is created we can pick a time 15:23:47 <karsten> please ask for more disk space while you're at it. 15:23:58 <irl> ok 15:24:05 <karsten> okay, sounds good! 15:24:25 <irl> that's all for this topic 15:24:27 <karsten> so, I'm going to put out an onionoo release and deploy it on both instances tomorrow? 15:24:32 <irl> yep 15:25:24 <karsten> great, moving on: 15:25:27 <karsten> Exit scanner (gaba) 15:26:19 <gaba> ok, mostly I want to check that we are on track 15:26:30 <gaba> irl: you said you worked this week on this? 15:26:31 <irl> i'd say we are 2 weeks behind 15:26:32 <gaba> which ticket? 15:26:52 <irl> i made pathspider run on debian 10 15:27:19 <irl> there's not really a ticket, it's more dusting stuff off to get it ready to start on tickets 15:27:26 <gaba> ok 15:28:11 <karsten> should we make a project plan at some point? 15:28:30 <irl> yes 15:28:58 <irl> i think i will do this in a trello while i'm still thinking about it 15:29:05 <irl> there's a lot of state to restore from swap 15:29:25 <karsten> sounds good. want to invite gaba and me to that board? 15:29:56 <irl> done 15:30:03 <gaba> thanks 15:30:10 <karsten> great! 15:31:23 <irl> anything else for this topic? 15:31:26 <phw> karsten: the graphs are neat! just yesterday, i was wondering what timeout to pick when automating a tor bootstrap and the graphs are very helpful in deciding 15:31:43 <karsten> phw: glad to hear. :) 15:31:59 <karsten> irl: is there anything we can promise (to ourselves) to do until next thursday? 15:32:20 <irl> add onionoo-backend-01.torproject.org to the onionoo.tpo rotation 15:33:07 <karsten> okay. 15:33:07 <irl> have a plan we can discuss, and a list of reasons why the new exit scanner doesn't work yet, as opposed to not existing 15:33:25 <irl> i think those two we can do 15:33:34 <gaba> irl: the scope for the exit scanner is: 15:33:34 <gaba> . New version of exit list format 15:33:34 <gaba> . Get an MVP with 15:33:35 <gaba> - written in python - support por IPv6 - support for new exit list format 15:33:38 <gaba> right? 15:34:06 <irl> initial scope is: MVP in python that produces the same exit list format as before 15:34:18 <gaba> ok 15:34:51 <gaba> ticket #29650 15:34:59 <gaba> oops, not that one 15:35:00 <gaba> exactly 15:35:14 <irl> i think some of the tickets may not be what we really want to do 15:35:20 <gaba> ok 15:35:32 <irl> i'll work on this in the trello to make sure we are doing what we want, and making the best use of the time 15:35:45 <irl> as i said, we were meant to start on this 2 weeks ago 15:35:56 <irl> but debian 10 release means other things started falling behind too 15:36:35 <gaba> sounds good 15:36:42 <karsten> okay, shall we move on? 15:36:45 <irl> ok 15:37:00 <karsten> Tor metrics data portal next steps (karsten) 15:37:31 <karsten> is there anything we need to do here in the next few weeks? 15:37:46 <karsten> the project was approved, as I hear. 15:38:11 <gaba> nothing on our side. Simple secure is going to be working on it 15:38:26 <karsten> okay, good. 15:38:28 <gaba> I will see them next week at mozfest and can check on timeline 15:39:03 <karsten> okay. 15:39:06 <irl> cool 15:39:28 <karsten> Review of #19332 (karsten) 15:39:57 <karsten> do you have an ETA for this? 15:40:00 <irl> i started this, but did not finish 15:40:15 <irl> it's probably ok 15:40:23 <karsten> if this takes longer, I'd check that we're not losing data at the moment. 15:40:34 <karsten> because of rsync deleting old data, or something. 15:40:56 <irl> i didn't fully go over all of the test cases yet 15:41:03 <irl> but the code itself looked good 15:41:20 <irl> i think it's ok to merge it and i can review the test cases retroactively 15:41:27 <karsten> okay. 15:41:40 <karsten> I'll merge and release and deploy that tomorrow then. 15:41:47 <irl> cool, sorry for the delay 15:42:13 <karsten> no worries. thanks! 15:42:30 <karsten> next: 15:42:34 <karsten> Roadmap - how are we doing? (gaba) 15:43:02 <gaba> it seems that we are fine here 15:43:04 <karsten> the OONI metadata database is now #32126. 15:43:10 <gaba> we need plan for exit scanner and that is the next thing 15:43:10 <karsten> sort of. 15:43:29 <karsten> which I'd want to move to the backlog for now. 15:43:45 <irl> ok 15:43:59 <gaba> ok 15:44:28 <irl> for me, onionoo hosts are still in progress, but i'm going to also be working on the exit scanner project bootstrap next week which will result in a plan 15:44:37 <irl> and maybe even some code 15:45:12 <karsten> I just put #19332 back, for merging and releasing and deploying. 15:46:20 <karsten> I also moved a card from the icebox to backlog. this is related to #32126. well, it's that ticket now. 15:46:29 <karsten> let me consolidate that. 15:46:50 <irl> sounds good 15:47:11 <gaba> sounds good 15:48:24 <karsten> I wonder if I should resume working on #31204 now. 15:48:44 <karsten> as there's not much I can do for the exit scanner project, yet. 15:49:05 <karsten> or is there anything else that needs my attention? 15:49:19 <irl> i think #31204 looks good 15:49:51 <irl> one thing i have noticed is that we're using a whole load of really deprecated stuff in the jetty we are using 15:49:52 <gaba> yes 15:50:24 <karsten> ah, I can check that. can you create a ticket for that, with examples? 15:50:47 <irl> ok 15:50:57 * karsten creates a placeholder card. 15:52:05 <karsten> alright. anything else for the roadmap topic? 15:52:26 <irl> not from me 15:52:29 <gaba> nop 15:52:35 <karsten> next meeting next week as usual? 15:52:39 <irl> yep 15:53:19 <karsten> great! have a good day/evening. bye! o/ 15:53:24 <karsten> #endmeeting