14:58:55 <karsten> #startmeeting metrics team meeting
14:58:55 <MeetBot> Meeting started Thu Sep 17 14:58:55 2020 UTC.  The chair is karsten. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:58:55 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:58:58 <karsten> https://pad.riseup.net/p/tor-metricsteam-2020.1-keep <- agenda pad
14:59:07 <acute> hi!
14:59:11 <karsten> hi!
14:59:13 <asn> karsten: o/
14:59:21 <karsten> hey asn!
14:59:29 <asn> karsten: (super quick question as i intercept the meeting: how can i run the unittests in metricsweb with junit?)
14:59:42 <karsten> `ant test`
14:59:42 <mikeperry> hi
14:59:54 <karsten> hi mikeperry!
15:00:08 <gaba> please add anything to the agenda. This may be the last meeting in a while (or not)
15:00:10 <asn> karsten: thank youuuu
15:00:16 <asn> karsten: that worked so easy!
15:00:33 <karsten> asn: interesting issue you're looking at there! I haven't yet looked in detail.
15:01:08 <karsten> shall we start, and folks add topics as we go?
15:01:17 <gaba> ok
15:01:26 <asn> karsten: no worries! i'm making some unittests on the meanwhile
15:01:37 <karsten> asn: great! thank you. :)
15:01:40 <karsten> OnionPerf development wrapping (freezing for the next 2 months)
15:01:55 <karsten> good news, the filter mode has landed!
15:02:01 <gaba> \o/
15:02:08 <acute> :D
15:02:19 <mikeperry> yay
15:02:25 <karsten> planned change log for 0.8: https://gitlab.torproject.org/tpo/metrics/onionperf/-/blob/develop/CHANGELOG.md
15:02:59 <karsten> it's not tagged yet, because I wasn't sure if we had to fix another bug.
15:03:20 <karsten> turns out we had a setup issue with op-hk5, but that was unrelated to the onionperf code.
15:03:50 <karsten> so, if there are no remaining issues we need to include, we can call 0.8 done.
15:04:17 <karsten> (thanks, acute, for finding and fixing that op-hk5 setup issue!)
15:04:35 <acute> :)
15:05:11 <karsten> how about I tag 0.8 this evening, unless somebody tells me I shouldn't in the next hour or two?
15:05:34 <gaba> ok :)
15:06:06 <acute> sounds good
15:06:27 <karsten> this also means we won't get to those other two issues in Next.
15:06:53 <karsten> tpo/metrics/onionperf#33420 and tpo/metrics/onionperf#33421
15:07:23 <karsten> however, I did confirm that we can still apply those features to measurements done with 0.7, if we reprocess those logs at a later time.
15:07:36 <mikeperry> is the filter mode documented? don't see it in the README
15:07:45 <karsten> it is. in the develop branch.
15:07:48 <karsten> let me find a link.
15:08:08 <karsten> https://gitlab.torproject.org/tpo/metrics/onionperf/-/tree/develop#filtering-measurement-results
15:08:40 <karsten> do you want to give this a try before I tag 0.8?
15:08:51 <karsten> I could as well do that tomorrow rather than this evening then.
15:09:51 <mikeperry> I have some partial onionperf measurements. it ran for a while and then died when the ssh connection broke. I forgot to tput it in a screen :/
15:10:19 <karsten> ah. yes, running in screen or tmux makes a lot of sense.
15:11:30 <karsten> the other remaining part before freezing onionperf development is documentation.
15:11:38 <karsten> tpo/metrics/onionperf#40001
15:12:00 <karsten> I'm uploading tarballs as we speak. and I have been doing that since yesterday. it's going to take days.
15:12:18 <karsten> we also need to put the documentation that we have somewhere.
15:12:26 <karsten> acute: what do you think about putting it on the wiki?
15:12:41 <acute> sounds ok
15:13:05 <karsten> "it" being the start page, the list of long-running instances, and the instructions for setting up long-running instances.
15:13:28 <karsten> and then we'll have to change onionperf.torproject.org to something new.
15:13:32 <karsten> and the link from metrics-web.
15:13:54 <acute> yes that's what I thought
15:14:24 <karsten> alright, I'll update the wiki then, and leave a comment on the issue.
15:14:32 <karsten> how do we update onionperf.torproject.org?
15:14:49 <acute> Do you prefer a redirect or a small static page?
15:14:56 <karsten> up to you!
15:15:12 <karsten> I don't feel strongly. either works.
15:15:28 <acute> ok, I'll have a think about it and update the ticket :)
15:15:36 <karsten> awesome!
15:16:18 <karsten> acute: do you have permissions to update onionperf.tp.o?
15:16:31 <karsten> and could you do that, once there's a plan?
15:17:19 <acute> karsten: I don't think so, yet
15:17:27 <acute> but yes, happy to do it!
15:17:28 <gaba> where is hosted?
15:17:31 <karsten> how was it created the last time?
15:17:35 <karsten> automatically?
15:18:08 <acute> irl set it up using the automatically generated html documentation
15:18:52 <karsten> can you two figure that out, possibly even next week?
15:19:13 <karsten> sounds like this needs the sysadmins.
15:19:17 <gaba> acute: it could be moved to the gitlab repo and be hosted there
15:19:28 <acute> I will ask him for info and update the ticket
15:19:55 <gaba> thanks
15:20:03 <karsten> okay, great.
15:20:17 <karsten> anything else on this first topic?
15:21:17 <karsten> alright, moving on then!
15:21:22 <karsten> Funding proposal for next OnionPerf phase (link in the mail that was sent).
15:21:41 <karsten> gaba or alsmith, do you want to go through this?
15:22:01 <alsmith> yep! hello all.
15:22:07 <gaba> I added the objectives that you all talked about in the pad.
15:22:19 <mikeperry> do we have jnewsome around?
15:22:32 <alsmith> first i am curious if anybody has feedback on karsten’s onion perf deployment architecture doc, linked in the pad
15:22:58 <alsmith> required attachment for the proposal and we should make any changes we’d like to see reflected there
15:23:02 <jnewsome> o/
15:24:07 <mikeperry> jnewsome: do you have this proposal link?
15:24:42 <alsmith> if nobody has feedback on the architecture doc, we can move on to discussing the objectives
15:24:43 <jnewsome> I don't; and I think I have an outdated link to the meeting pad :)
15:24:54 <karsten> https://pad.riseup.net/p/tor-metricsteam-2020.1-keep
15:24:57 <alsmith> jnewsome i will send you the proposal link
15:25:15 <jnewsome> alsmith: I think gaba just sent it
15:25:20 <alsmith> ha!
15:25:40 <jnewsome> yup, doubly got it now :). thanks!
15:26:29 <alsmith> ok. i think the next step with the objectives is to estimate our time in months per objective so we can build a timeline + the budget
15:26:35 <gaba> if we could add activities to those objectives that woudl be great :)
15:26:37 <alsmith> gaba does that sound right?
15:26:43 <gaba> yes
15:27:40 <karsten> alsmith: I left a few comments on the proposal this afternoon.
15:27:49 <alsmith> thanks karsten
15:27:49 <karsten> not long before the meeting, though.
15:27:58 <alsmith> i haven’t had a chance to look, but i will
15:28:14 <karsten> maybe some of those comments can be helpful to derive activities.
15:28:27 <alsmith> ok, checking see
15:28:40 <karsten> shall we quickly go through the objectives to see what input we can get from folks here?
15:28:53 <alsmith> that sounds great
15:29:00 <karsten> Objective 1: Automate setup of OnionPerf instances.
15:29:16 <karsten> acute: this is closely related to the metrics-cloud work that you and irl worked on.
15:29:37 <gaba> O1.1 Improve monitoring
15:29:38 <gaba> O1.2 Improve ease of deployment and maintenance
15:29:38 <karsten> hiro also mentioned terraform as a possible tool here.
15:29:41 <gaba> ?
15:30:14 <karsten> those two sound related, yes.
15:30:27 <karsten> maybe in changed order, but yes.
15:30:38 <karsten> (first we need to deploy something, then we can monitor it.)
15:31:00 <gaba> ok
15:31:27 <alsmith> i switched the order in the pad
15:31:59 <karsten> I guess we're going to start this objective by setting up all 10 (?) instances manually, just to make sure we get measurements starting on day 1.
15:32:23 <karsten> and in parallel to those measurements running we'll work on automating deployment.
15:32:49 <gaba> Then our obj2 maybe should be inside obj 1
15:32:51 <gaba> Objective 2: Diversify the demographics of our data
15:32:59 <karsten> or maybe that should be objective 1.
15:33:08 <karsten> and objective 2 is automating what we did in objective 1.
15:33:10 <gaba> right
15:33:13 <alsmith> I think it should be 1
15:33:20 <gaba> and the activity is to evaluate where to setup this instances
15:33:45 <karsten> also to keep them running throughout the year.
15:33:51 <karsten> or, updated versions of them.
15:34:48 <karsten> okay, I think that's a good start for 1 and 2.
15:35:02 <karsten> objective 3 is not so much about automation, but about visualization.
15:35:11 <karsten> we do have visualizations that are updated automatically.
15:35:21 <karsten> we just need to adapt them to our grown number of measurements.
15:35:34 <karsten> as I wrote in the proposal in a comment.
15:35:44 <alsmith> i accepted that comment, also changed it here in the pad
15:36:14 <karsten> the activity here would be to go through our existing visualizations and update them to a larger number of measurements.
15:36:28 <karsten> this can be as simple as reducing details of each measurement, for all of them to fit in,
15:36:49 <karsten> or it could mean that we need to come up with new data visualizations.
15:36:50 <gaba> we are not going to include "Develop developer-facing tooling to quickly graph baseline performance metrics." as an activity?
15:37:03 <gaba> this was in the onionperf proposal before
15:37:25 <karsten> gaba: I think we meant the visualizations in `onionperf visualize` with that.
15:37:26 <gaba> and do we need to include any additional onionperf filter?
15:37:33 <gaba> ahhhhhh
15:37:33 <gaba> ok
15:37:38 <karsten> this is about updating the graphs on Tor Metrics.
15:37:51 <karsten> additional filters are also not directly related.
15:37:54 <gaba> ok
15:38:17 <karsten> objective 4 is a rather small activity.
15:38:41 <gaba> can we include it in other objective?
15:38:42 <karsten> we'll have to set up storage, upload tarballs, and keep them there (and add new ones) throughout the year.
15:38:45 <gaba> as an activity
15:38:55 <mikeperry> well for objective 3 we could add shadow work to make it easier to get consistent visualization out of shadow to match onionperf
15:39:24 <mikeperry> and objective 4 could also include shadow result storage
15:39:39 <karsten> mikeperry: both make sense!
15:39:44 <mikeperry> we may want to archive simulation runs, or at least results
15:40:40 <jnewsome> mikeperry: karsten: yup, agreed
15:40:42 <mikeperry> all of the shadow work questions are probably better answered by jnewsome than me, thats why I pinged him
15:40:42 <karsten> regarding objective 4, we'll also want to write some verification scripts, so that the simulation run tarballs that we get from other folks contain the right data that we would expect.
15:41:23 <karsten> the verification scripts can be part of O4.1 and O4.2 though.
15:42:43 <karsten> ideally, I'll want to run `onionperf archive`, and onionperf goes through the files in onionperf-data/, tells me the relevant data for the wiki table (start date, tor version, etc.), tells me about missing dates or file types, and creates a tarball for me to upload.
15:43:02 <karsten> and on the receiving side I want to run the same script to verify this metadata.
15:43:35 <karsten> objective 5 is all about shadow.
15:43:58 <jnewsome> is there another doc I should be looking at too? I only see objectes 1-3
15:44:01 <karsten> jnewsome: is it still the case that a large shadow simulation requires a host with crazy amounts of memory?
15:44:02 <jnewsome> *objectives
15:44:14 <gaba> jnewsome: you can look at the pad from the meeting
15:44:16 <karsten> I'm looking at the objectives in the pad: https://pad.riseup.net/p/tor-metricsteam-2020.1-keep
15:44:17 <alsmith> jnewsome we’re working from the metrics team meeting pad now
15:44:18 <gaba> we are changing things there
15:44:20 <mikeperry> for objective 5 we should keep in mind that it may be weeks (months?) before this proposal would be accepted and work would commense
15:44:27 <jnewsome> ah thanks
15:44:35 <alsmith> yes mikeperry
15:44:47 <alsmith> jan 1 assumed start
15:45:06 <mikeperry> so at that point, we may be able to use jnewsom's work to use more smaller aws instances instead of one huge one with lots of ram
15:45:15 <karsten> ah, is that in the making?
15:45:20 <mikeperry> I am not sure if that changes cost or simulation speed
15:45:36 <jnewsome> it currently doesn't support distributing across physical machines, if that's what you mean. scaling across machines is in our NSF proposal but still a ways off
15:45:42 <mikeperry> and how we should estimate that or decide which one we need
15:46:13 <karsten> okay, sounds like we should assume that we'll need machines with lots of memory for this proposal.
15:46:24 <mikeperry> jnewsome: do you think we should still prefer larger instances with lots of ram?
15:46:32 <gaba> we should include that in the budget of the proposal
15:46:36 <mikeperry> and if so, how much?
15:46:46 <jnewsome> if you want to run a realistic tor network, yes lots of ram and ideally lots of cpu cores
15:46:51 <gaba> can we add to the pad on how much we will need
15:47:33 <karsten> https://aws.amazon.com/ec2/instance-types/
15:47:40 <karsten> go to memory optimized.
15:48:55 <karsten> which of those would work?
15:49:20 <karsten> (9 minutes until meeting end.)
15:49:27 <jnewsome> I'm not super confident here, but I think you'd want the biggest of these you could get
15:49:49 <jnewsome> i.e. I'm fairly confident you can fully utilizae 64 cores and 512 GB of RAM
15:50:39 <karsten> $3.6864 per Hour
15:51:03 <jnewsome> I can ask Rob - I know he has a recent paper trying to see how big a simulated network needs to be to be accurate
15:51:11 <gaba> alsmith: do we need anything else for this proposal? i will follow up with karsten tomorrwo to estimate activities.
15:51:16 <jnewsome> and probably has a better feel if not hard data for what it'd take to run such networks
15:51:31 <mikeperry> that aws page is not showing me pricing. is it because I am not logged in?
15:51:32 <gaba> and we can wait for your discussion with rob (jnewsome) to include what we need for obj 5
15:51:34 <karsten> jnewsome: I think that would be very helpful.
15:51:36 <alsmith> gaba this is good. we’ll need estimated staff time and aws costs for o4
15:51:38 <alsmith> o5*
15:51:43 <karsten> mikeperry: https://aws.amazon.com/ec2/pricing/on-demand/
15:52:02 <karsten> (there might be other prices for other pricing models than on demand. I have no idea.)
15:52:10 <gaba> karsten: estimation is something we can do tomorrow. right? do we need anything else
15:52:17 <gaba> ?
15:52:31 <karsten> did we miss anything obvious in this proposal?
15:52:36 <karsten> would be my question.
15:52:53 <karsten> anything we could do if only we had lots of AWS resources?
15:53:40 <karsten> ah, the postgresql database is not contained in any of the objectives.
15:53:51 <karsten> maybe that should be part of objective 1.
15:54:06 <gaba> it is not part of ojb 2 ?
15:54:08 <karsten> using that database to do an analysis of all the fine data we get from 10 different measurement spots.
15:54:28 <gaba> because it is part of deployment
15:54:29 <acute> is this a db which contains all measurements from our instances?
15:54:30 <karsten> we won't need the huge database for normal operation.
15:54:49 <karsten> it's a database we still need to envision where we can add lots of measurement details for custom analysis.
15:55:01 <karsten> it's a database large enough that it doesn't fit on our laptops.
15:55:08 <acute> ok
15:55:30 <karsten> we don't have to ask for it. it was just an idea for putting AWS resources at good use.
15:55:37 <karsten> it could be fun.
15:55:59 <karsten> I refrained from putting in fancy big data stuff.
15:56:24 <karsten> 2 minutes left.
15:56:25 <alsmith> karsten if you think it makes sense, lets add it. maybe when estimating activities with gaba?
15:56:29 <gaba> One more thing we have in the agenda is about this meetings.
15:56:32 <karsten> okay.
15:56:51 <gaba> Do we keep running them? Anybody want to run them?
15:57:58 <karsten> sounds like we'll resume them in november then.
15:58:19 <gaba> ok
15:58:21 <alsmith> i have to switch gears. thanks everyone for your work on the proposal! o/
15:58:22 <karsten> okay, gotta end this meeting.
15:58:26 <karsten> #endmeeting