15:01:21 <karsten> #startmeeting metrics team meeting 15:01:21 <MeetBot> Meeting started Thu May 14 15:01:21 2020 UTC. The chair is karsten. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:21 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic. 15:01:33 <karsten> https://pad.riseup.net/p/tor-metricsteam-2020.1-keep <- agenda pad 15:02:29 <gaba> o/ 15:02:32 <karsten> hi! 15:03:20 <karsten> I added three onionperf tickets to the agenda. anything else? 15:03:42 <karsten> if not, we can start. 15:03:55 <acute> nothing from me 15:04:23 <dennis_jackson> nothing from me 15:04:31 <karsten> alright. 15:04:34 <gaba> only a reminder where we are tracking onionperf work 15:05:00 <karsten> maybe before we start, let me say that I'm really excited how we're moving forward with onionperf work these days! 15:05:16 <gaba> \o/ 15:05:16 <karsten> it's really great to have reviews and patches and feedback from people. 15:05:20 <acute> :D 15:05:26 <gaba> yes! 15:05:34 * phw nods 15:05:48 <karsten> so, 15:05:55 <karsten> first ticket I have on the list is #34141. 15:06:11 <karsten> the question for today is: anything that we're overlooking here? 15:06:23 <karsten> or can we move forward and stop generating that format on onionperf instances? 15:06:48 <karsten> that would simplify some things going forward. 15:07:12 <karsten> if nothing comes to mind in the next hour or three, I'd say let's do it. 15:07:21 <dennis_jackson> +1 15:07:36 <karsten> and if something comes up, please put it on the ticket! 15:07:53 <karsten> okay. next ticket is #33258. 15:08:02 <karsten> this is now in need of review. 15:08:19 <karsten> it touches on a lot of the visualization code. 15:08:27 <karsten> and now uses pandas and seaborn. 15:09:08 <phw> i can review it if nobody else wants it 15:09:21 <karsten> I think that would be great! 15:09:24 <acute> I've used pandas quite a bit, but not seaborn 15:09:40 <phw> i haven't used either ;) 15:09:59 <karsten> fwiw, it's my first time using seaborn, too. ;) 15:09:59 <acute> I'm happy to help review it also 15:10:26 <acute> it's quite a change! 15:10:33 <karsten> it's a rewrite, yes. 15:10:48 <phw> acute: i'll leave it for you then, if you don't mind 15:10:50 <karsten> it's also not making the exact same plots anymore. 15:11:02 <karsten> there's a change history on the ticket on what has changed. 15:11:24 <acute> this is ok 15:11:36 <karsten> acute: if you want to pick it up, you could do a very quick review round, ask questions, and then do another round. if that helps. 15:11:49 <acute> exactly what I was thinking! 15:11:52 <karsten> happy to explain why things have changed. 15:12:04 <karsten> are you maybe around after the meeting? 15:12:11 <karsten> I can be here until 16:00 utc. 15:12:11 <acute> yes :) 15:12:28 <karsten> okay, let's talk after the meeting then. 15:12:46 <acute> great! 15:12:51 <karsten> cool! 15:12:58 <karsten> next ticket is #26673. 15:13:08 <karsten> it's already closed, I just had a thought after closing it: 15:13:16 <karsten> should we update the document version to 1.1? 15:13:26 <karsten> this change adds a new field to the json format. 15:13:37 <karsten> which means it's a backward-compatible change. 15:13:44 <karsten> but a 1.0 parser won't understand this field. 15:13:57 <karsten> not too late to do it. 15:14:18 <karsten> related question: should we use a string for the version, as in "1.1", rather than a double? 15:14:32 <karsten> or float or whatever json calls them. 15:14:36 <dennis_jackson> It seems nice to demarcate when the extra info became universally available, but I guess with JSON it won't make too much difference either way 15:15:14 <karsten> agreed. what's the preference? 15:15:33 <karsten> what would you prefer when working with these files? 15:15:49 <dennis_jackson> I would prefer the explicit string 15:16:06 <karsten> okay. 15:16:08 <dennis_jackson> but as I said, no strong feelings here 15:16:14 <karsten> right, same here. 15:16:14 <phw> are we using semantic versioning? is there any meaning hiding behind 1.1? 15:16:33 <karsten> fine question. so far there's just a version 1.0 and nothing else. 15:16:58 <karsten> but we're using semantic versioning in other metrics code bases, so why not do the same here. 15:17:30 <karsten> as in: a 1.0 parser can parse everything with 1.x, but it might break with a 2.x format. 15:17:42 <phw> 1.1 sounds good to me if it can mean "there's a new feature but it doesn't break existing parsers" 15:17:52 <karsten> yes. 15:18:10 <karsten> and I'd change the type to a string. "1.1". 15:18:21 <karsten> which _might_ break 1.0 parsers... hmm. 15:18:38 <karsten> maybe that means this is 2.0? 15:18:43 <karsten> err, "2.0". 15:18:49 <phw> i was about to say the same :) 15:18:54 <karsten> heh 15:19:06 <karsten> works for me. 15:19:22 <karsten> will open a ticket for this. 15:19:24 <dennis_jackson> meta-semantic versioning 15:19:48 <dennis_jackson> :P 15:19:48 <karsten> alright! 15:19:51 <karsten> :) 15:20:24 <karsten> how do we proceed this week? 15:20:48 <karsten> phw: do you plan to work on tickets this week? 15:21:16 <phw> karsten: yes, friday will be my onionperf day -- unless you need me to pay attention to anything outside of friday 15:22:04 <karsten> sounds good. do you already have something to work on tomorrow? want me to help pick something? 15:22:17 <phw> i have yet to find a ticket though. any suggestions? you suggested #34142 but sneaky acute reviewed it before i had a chance 15:22:26 <karsten> heh 15:22:35 <acute> oh no! 15:22:41 <phw> acute: ...which i appreciated ;) 15:23:15 <acute> sorry! I had quite a bit of time on my hands this week :) 15:23:25 <karsten> phw: is #30586 something you might enjoy doing? 15:24:00 <phw> karsten: yes, i'll pick that one. (we have to clean up bridgedb's setup.py at some point, so i might as well get started now) 15:24:10 <karsten> yay! 15:24:14 <phw> do i reassign it to me instead of metrics-team? 15:24:14 <karsten> by the way, 15:24:18 <karsten> yes, please. 15:24:41 <karsten> this is a python-noob question: is there a way to run and use onionperf without sudo powers? 15:25:11 <karsten> right now, installing onionperf is the only (?) step that requires sudo powers during normal operation. 15:25:33 <phw> hmm, what's the part that requires sudo powers? i can at least start it without arguments as non-root user 15:26:20 <karsten> current instructions are to install it using `sudo python3 setup.py install` and then run it using just `onionperf`. 15:27:22 <karsten> I'm just thinking that it would be easier if the user could build tgen and onionperf and just start onionperf (with some paths as args) than use sudo. 15:27:32 <karsten> but maybe this isn't the best use of this meeting time. 15:27:43 <phw> i didn't use sudo in these commands 15:27:56 <karsten> ah, interesting. 15:28:02 <phw> and the onionperf binary ended up in ~/.local/bin 15:28:21 <karsten> yes, that's what I'd have hoped. great! 15:28:23 <karsten> problem solved. 15:28:33 <karsten> :) 15:28:36 <karsten> sorry for the noise. 15:28:44 <karsten> anything else for today? 15:29:27 <karsten> otherwise let's end this meeting now and meet again next week! 15:29:30 <arma2> hey, speaking of onionperf, speaking of tor control logs, 15:29:37 <dennis_jackson> nothing from me 15:29:41 <arma2> an ooni person is working on making it easy for ooni to torify its queries 15:29:42 <dennis_jackson> a wild Arma appears! 15:29:57 <arma2> and i was going to suggest that they check out, uh, whatever we recommend for learning the path that their query took 15:30:09 <arma2> since the path through tor should really be part of the ooni test output if it's using tor 15:30:29 <arma2> is there a best practice there? i'm guessing they haven't thought about this much and don't have a control port at all currently 15:30:44 <dennis_jackson> Ah, I am curious about best practice about this as well. It would be very useful for the Tor Browser measurements. 15:31:01 <karsten> so, what onionperf does is that it first logs everything to two log files. 15:31:11 <arma2> i thought of asking here because i saw karsten messing with a trac ticket that mentioned control port logs 15:31:24 <karsten> then, once per day it goes through them and matches tgen transfers to streams by source port. 15:31:37 <karsten> that produces some errors, though. 15:31:54 <karsten> which is why we should also compare transfer and stream start and end times. 15:32:06 <karsten> and then from streams we can find the circuit by ID. 15:32:24 <karsten> I wonder where we would document this. 15:32:41 <karsten> does it make sense? 15:32:54 <arma2> are the control port logs only to match up paths? or are they also where you learn things like tor's bw events 15:33:17 <karsten> we do have bw events, too. 15:33:19 <arma2> and, i wonder if doing it in real time is too unworkable. since ooni isn't going to want to do a bulk daily thing. and, dennis, what best matches your use case? 15:33:55 <karsten> the real time approach would likely be similar. 15:34:16 <karsten> error codes are another reason for doing this. 15:34:35 <karsten> when tgen sees an error it's useful to look at control events to see why things failed. 15:35:07 <karsten> let me document this in some form. where would I send it? tor-dev@? 15:35:41 <dennis_jackson> So I have a similar but slightly different use case. To associate particular HTTP requests with particular circuits. Not necessarily, but ideally in real time. 15:35:43 <arma2> hm. i think if we like the current plan, yes tor-dev makes sense. if we don't like it and want to do one that's better or works for more cases, then not-tor-dev-yet 15:36:09 <arma2> like, does dennis know his source port for the socks request. maybe yes maybe no 15:36:13 <arma2> i don't know about the ooni situation 15:36:22 <dennis_jackson> I wondered about a SOCK proxy recording username/password combinations and logging the request, then forwarding it to Tor, then recovering the circuit from the control port 15:36:43 <arma2> our gsoc student, woswos, will be wanting to solve this too: his whole project is basically to do requests over tor to cloudflare places and record how it went for various exits 15:36:44 <dennis_jackson> But its been a while since I was thinking about this, sorry if the above is incoherent. 15:37:24 <karsten> the choice to use the source port was made by rob. I only added the timing part. 15:37:34 <karsten> I'm not sure how that works for others. 15:37:41 <arma2> dennis_jackson: i hope that adding a bonus socks proxy in the middle isn't needed. socks to socks can't be the best solution. i hope. :) 15:38:04 <arma2> maybe the next step is a trac ticket, where karsten writes up what onionperf does, 15:38:08 <dennis_jackson> Indeed. I felt I like there ought to be a clean solution, but couldn't see it. 15:38:22 <karsten> sounds good. 15:38:22 <arma2> and a tor-dev@ mail, where we try to accumulate the use cases that people have, on that ticket 15:38:31 <arma2> and then i can point woswos to the ticket, and also ooni-person 15:38:38 <karsten> yep! 15:38:44 <karsten> will do tomorrow. 15:39:12 <arma2> woswos: ^ consider yourself preemptively pointed 15:39:12 <karsten> anything else for today? 15:39:41 * arma2 places shari's crickets very carefully on the floor 15:39:52 <karsten> thanks. :) 15:40:01 <karsten> thanks, everyone, talk to you next week! 15:40:07 <karsten> #endmeeting