15:04:15 <karsten> #startmeeting metrics team meeting
15:04:15 <MeetBot> Meeting started Thu Oct 24 15:04:15 2019 UTC.  The chair is karsten. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:04:15 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
15:04:33 <karsten> https://storm.torproject.org/shared/5h1Goax5eNusxjXJ_Ty5Wl7hFR1uqCReUiN8xdlBG8T <- agenda pad as usual
15:05:52 <karsten> okay, let's start.
15:05:55 <karsten> Announcement: onionoo-backend-01 is live (irl)
15:05:59 <karsten> yay!
15:06:07 <karsten> how is it doing?
15:06:18 <irl> this is deployed using the ansible scripts in metrics-cloud, so far no one has complained
15:06:33 <irl> we did need to use xfs instead of ext4 for the filesystem due to the vast number of inodes
15:07:01 <irl> Filesystem      Size  Used Avail Use% Mounted on
15:07:02 <irl> /dev/sdb         60G   37G   24G  62% /srv
15:07:06 <karsten> is it because of many files or because of many files in certain directories?
15:07:23 <irl> too many files
15:07:39 <karsten> hmm, okay.
15:07:44 <karsten> nothing we can solve easily.
15:07:51 <irl> it's going fine with xfs
15:08:10 <karsten> how many backends do we have now?
15:08:26 <irl> 3 backends, to be reduced to 2 at some point we decide we're happy with the new one
15:08:43 <irl> there will then be another 3rd backend to replace another old one, and then we end up with 2 new ones
15:09:04 <irl> #32268
15:09:32 <karsten> can I log into those?
15:09:48 <irl> yes
15:09:55 <irl> they use the same onionoo and onionoo-unpriv user accounts
15:10:13 <karsten> did you end up writing that ops document?
15:10:41 <irl> it's in progress
15:11:17 <irl> upgrading to a new version consists of changing a variable in https://gitweb.torproject.org/user/irl/metrics-cloud.git/tree/ansible/onionoo-backends.yml
15:11:22 <irl> and then running the playbook
15:11:48 <karsten> maybe we can do such things together the first time. in a non-emergency situation.
15:11:56 <irl> yes, definitely
15:12:16 <karsten> do we have anything in the merge/release queue?
15:12:42 <irl> i couldn't say
15:12:55 <karsten> maybe #32065?
15:13:07 <phw> #19332 also seems merge ready?
15:13:07 <karsten> not exactly ready for merge, but looks like relatively easy to fix.
15:13:15 <irl> could be an easy fix yeah
15:13:28 <karsten> phw: this was about onionoo.
15:13:39 <phw> oh, sorry
15:13:48 <karsten> phw: #19332 is already merged, AFAIK.
15:14:14 <karsten> sorry, I'm still recovering from many days of focusing on a single coding task..
15:15:22 <karsten> phw: already deployed, might require some tweaking:
15:15:23 <karsten> https://metrics.torproject.org/collector/recent/bridgedb-metrics/2019/10/22/
15:15:34 <phw> karsten: oh, neat!
15:16:18 <karsten> we might have to change the directory structure a bit there, but overall it looks fine.
15:16:49 <karsten> irl: so, let's look into making some easy onionoo fixes, put out a release, and deploy that on the 3 hosts.
15:17:28 <irl> sounds good
15:18:09 <irl> i'm not sure then what my next priorities are, would it be to work on the exit scanner or to finish the onionoo service documentation?
15:18:43 <irl> i think finishing the documentation is more important
15:18:56 <karsten> I'd say finish the thing you were working on before starting something new.
15:19:09 <irl> it's a good policy
15:20:11 <karsten> alright, let's quickly talk about:
15:20:16 <karsten> CollecTor index.json update (karsten)
15:20:27 <karsten> this turned out to be bigger than I had thought.
15:20:31 <irl> #31024
15:20:42 <irl> #31204 maybe
15:20:48 <karsten> yes, that.
15:21:23 <karsten> it's almost ready for review, but I feel like I shouldn't waste your time with almost-ready code and instead finish this first.
15:21:35 <karsten> maybe one thing though:
15:22:03 <karsten> do you see any issue with having a separate directory managed by the indexer where we store (hard) links to files we're serving?
15:22:14 <karsten> right now we have archive/ and recent/.
15:22:36 <karsten> the new directory, htdocs/, would contain directories htdocs/archive/relay-descriptors/...
15:22:49 <karsten> which in turn contain (hard) links to the existing two directories.
15:23:05 <karsten> this has some nice properties, but do you see any potential issues here?
15:23:18 <irl> i don't know enough about file systems to know if there would be problems here
15:23:43 <karsten> okay.
15:23:45 <irl> if they are hard links, maybe this doubles the size of backups
15:24:05 <irl> i don't know what else might be recursing through this tree
15:24:28 <karsten> I think the alternative is to keep a directory of copies.
15:24:46 <karsten> which doubles the size of backups for certain.
15:25:01 <karsten> worth checking with admins first?
15:25:26 <karsten> probably.
15:25:41 <irl> yes i think checking is a good idea
15:25:48 <irl> i can't think of anything else that might be an issue though
15:26:15 <karsten> okay.
15:26:41 <karsten> if you have thoughts on the sample output of #31204, please leave a comment there.
15:26:49 <karsten> also hoping for a comment from atagar.
15:27:19 <karsten> anyway. what remains from our agenda that doesn't require a gaba?
15:27:20 <irl> ok
15:27:42 <karsten> roadmap maybe?
15:27:47 <irl> we can look at the roadmap
15:28:38 <karsten> what about those jetty parts?
15:28:43 <karsten> did you create a ticket?
15:29:06 <irl> ah no, i forgot
15:29:20 <karsten> you could also mail me those warnings, and I create a ticket.
15:29:39 <irl> i think the problem was not that it caused a stacktrace, i think it's about the ASM version
15:29:54 <karsten> a dependencies issue?
15:30:48 <irl> https://www.eclipse.org/lists/jetty-announce/msg00125.html
15:31:20 <irl> we need to update to a newer minor version because there are bytecode compatibility issues with the jre we are using
15:31:23 <irl> that was it
15:32:10 <irl> we should just update to the latest release on the branch we are on ideally
15:32:53 <karsten> well, right now we're using versions found in debian (old?)stable.
15:33:19 <irl> ah we're running on two debian versions right now
15:33:26 <irl> debian stable has java 11
15:33:32 <irl> debian oldstable has java 9
15:33:47 <irl> maybe java 9 doesn't support ASM 7
15:33:54 <karsten> and we're developing for java 8.
15:34:16 <irl> java 8 language but it needs to run in java 11 jvm
15:34:20 <karsten> yep.
15:34:49 <irl> i think we can block upgrading jetty on moving everything to the new setup with java 11
15:35:00 <irl> we don't want to break the java 9 stuff just before we retire it
15:35:51 <karsten> okay, I'll look a bit more into this.
15:36:20 <karsten> this is fine as a start, no need to create a ticket. unless you want to do that anyway.
15:36:45 <karsten> any other changes to the roadmap?
15:36:52 <irl> so far it's not been a real issue, we can make a ticket if we find it is
15:37:01 <karsten> ok.
15:37:09 <irl> i moved #29653 and #29624 to icebox, they are out of scope for the MVP
15:37:17 <karsten> sounds good.
15:37:24 <irl> just adding the tickets me and acute came up with to the backlog for onionperf
15:37:35 <karsten> I guess I can move #31071 to backlog.
15:38:04 <karsten> next step there is to merge translations and release+deploy.
15:39:10 <irl> then i guess i add the exit scanner milestones to the backlog
15:39:17 <karsten> yep!
15:40:32 <irl> the bootstrapping task goes to done
15:40:40 <karsten> makes sense.
15:40:49 <irl> i'll leave #31659 in progress for the documentation
15:41:12 <irl> we can move the jetty card to the icebox to remind us to look again later?
15:41:19 <irl> or just archive it
15:42:12 <karsten> so, it's really just warnings now?
15:42:32 <karsten> if so, archiving sounds fine.
15:42:43 <irl> there are no warnings, but i read that changelog entry that strongly recommended we upgrade
15:42:54 <irl> so i thought we should at least look at it
15:43:03 <irl> ok let's archive it
15:43:08 <irl> we gave it thought
15:43:12 <karsten> heh. yep.
15:43:26 <karsten> speaking of bugs, I'll have to look into #32194.
15:43:46 <karsten> who would have thought that thread-safety is hard.
15:44:01 * irl has been rewriting code to use kqueue instead of threads
15:44:06 <irl> kqueue is soooo nice
15:44:14 <irl> won't work here though because we actually are cpu intensive
15:44:35 <irl> that was for network i/o bound code
15:44:48 <karsten> the issue is that we were trying to reduce memory overhead.
15:45:00 <karsten> which came at the price of lack of thread safety.
15:45:45 <irl> right
15:46:19 <karsten> anyway, not throwing out threads this week. ;)
15:46:30 <irl> i'm going to add #32262 to in progress next week once i start it
15:46:49 <karsten> sounds good!
15:46:57 <karsten> or, maybe just do that now?
15:47:06 <karsten> so that gaba knows what's going on when she looks after the meeting?
15:47:10 <irl> yeah ok
15:47:52 <irl> i think that's all for me for the roadmap
15:48:01 <karsten> yes, same here.
15:48:14 <karsten> I'll move the other topics to the next meeting agenda.
15:48:23 <irl> ok cool
15:48:24 <karsten> which means we're out of topics (and almost out of time).
15:48:43 <karsten> great stuff! next meeting next week?
15:48:46 <irl> yep
15:48:52 <karsten> any daylight savings changes until then?
15:48:57 <karsten> or is it the week after?
15:48:57 <irl> who knows
15:49:00 <karsten> hehe
15:49:08 <karsten> same UTC time.
15:49:16 <irl> i have utc time on my clocks anyway
15:49:17 <karsten> alright. have a great UTC evening!
15:49:22 <irl> heh
15:49:25 <irl> bye!
15:49:26 <karsten> bye!
15:49:29 <karsten> #endmeeting