13:59:46 <karsten> #startmeeting metrics team meeting
13:59:46 <MeetBot> Meeting started Thu Dec 17 13:59:46 2015 UTC.  The chair is karsten. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:59:46 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
13:59:56 <karsten> hi. who's here for the meeting?
14:00:17 <thms> hi, me
14:00:24 <karsten> hi thms!
14:00:38 <karsten> thms: please add topics to the pad.
14:00:58 <thms> right, the pad…
14:02:31 <SeanSaito> hi!
14:02:38 <karsten> hi SeanSaito!
14:02:55 <SeanSaito> can I have the link the pad?
14:03:00 <karsten> sure: https://pad.riseup.net/p/zUNzEIFRq5S4
14:03:17 <SeanSaito> thanks!
14:04:24 <virgil> I exist.
14:04:31 <karsten> SeanSaito: feel free to add topics, too.
14:04:33 <karsten> virgil: hi!
14:04:41 <karsten> (same)
14:05:27 <karsten> okay, let's start at the beginning of the agenda, and maybe when we're at the bottom there will be more topics.
14:05:33 <karsten> * Onionoo performance update (karsten)
14:05:52 <virgil> Karsten: did Vikram did in touch with you?
14:05:53 <karsten> did people notice that onionoo has become faster?
14:06:02 <SeanSaito> Yes!
14:06:16 <karsten> virgil: no, I only saw in your report that he was planning to send patches.
14:06:21 <virgil> mehhhhh
14:06:26 <virgil> let me go harass him.
14:06:27 <karsten> SeanSaito: great! :)
14:06:33 <virgil> he has 4 patches for Onionoo improvements
14:06:34 <karsten> here's a graph:
14:06:39 <karsten> https://people.torproject.org/~karsten/volatile/onionoo-requests-2015-12-17.png
14:06:42 <karsten> virgil: sounds great!
14:06:58 <karsten> here's what happened: we turned on apache caching.
14:07:32 <karsten> now onionoo doesn't have to process all incoming requests anymore, but most of them can be answered by apache.
14:07:41 <thms> roll eyes
14:08:07 <karsten> we even had apache caching turned on before migrating from sewerzowi to the new host, but we didn't enable it when moving.
14:08:32 <karsten> okay, that's all on that topic. :)
14:08:53 <karsten> moving on
14:08:54 <karsten> * Server-side JavaScript on Metrics (karsten)
14:09:01 <Sebastian> nickm: you haven't specified how someone is supposed to opt in
14:09:22 <karsten> so, we had a discussion on tor-dev@ to switch to d3.js.
14:09:54 <karsten> and the question came up whether we can avoid executing javascript on the client.
14:10:24 <karsten> for example, by running node.js on the server and producing all visualizations there.
14:10:39 <karsten> dear javascript experts, is that possible?
14:10:50 <karsten> and can we still have interactive visualizations?
14:11:44 <karsten> what if the user wants to pick a different time frame for a visualization? and can we build that in a way that it doesn't take seconds to draw a new graph?
14:12:08 <thms> it’s a little complex and hard to answer on irc i fear
14:12:37 <thms> you can generate everything on teh server and control it with form elements like in the old days
14:13:04 <thms> you can detect if the browser has javascript enabled and serve him accordingly
14:13:07 <karsten> ah, that sounds like what we're currently doing on metrics.
14:13:16 <karsten> just that we're using R/ggplot2 for generating graphs.
14:13:17 <thms> you can also make the app downoadable for local execution
14:13:50 <thms> yes, exactly. no magic there
14:14:22 <karsten> how would we build this without building it twice?
14:15:01 <thms> the question is just how much effort you want to put into it and how complex/maintainable it get’s
14:15:15 <thms> oh, not beuilding it twice should be easy
14:15:38 <thms> (relatively easy perhaps…)
14:16:22 <karsten> okay. do you think it's realistic to try it out for one visualization and then decide whether that's feasible or not?
14:17:03 <virgil> karsten: sent you email on incoming Onionoo performance patches.
14:17:17 <thms> yes, but you’ll want to know how long that would take
14:17:18 <karsten> it's too bad that Letty is not here today (she said she couldn't make it), but what we could do is take her visualization and try to make it usable without client-side javascript.
14:17:59 <karsten> virgil: great, will take a look after the meeting. thanks!
14:18:10 <karsten> thms: sure, having a rough idea would be cool.
14:18:46 <karsten> but, it's something for 2016 anyway.
14:18:59 <karsten> we could talk about it again in two weeks.
14:19:01 <thms> karsten: there is no magic bullet. you know forms. you know downloadable apps. you know pre-generation (and caching!). and roundtripping. all the stuff you have and need and hated without AJAX
14:20:10 <thms> but it’s possible and not too hard too. but, to generate forms and javascript interactive controls automagically from the same codebase - hmmm - that shouldn’t be totally easy if the case is not quite general.
14:20:30 <thms> like: the regular graphs on metrics are general, lettys graphic isn’t
14:20:58 <thms> maybe we can talk about it at 32c3
14:21:07 <karsten> thms: yes, the regular graphs on metrics are of course written for a model where they are generated on the server.
14:21:20 <karsten> sure, let's do that!
14:21:35 <karsten> okay, thanks!
14:21:44 <thms> karsten: i mean the controls (forms or ajax). teh graphics themself are easy
14:22:14 <karsten> well, letty's visualization is also animated.
14:22:30 <karsten> but yeah, I figured that we wouldn't be able to do that without javascript. ;)
14:22:54 <karsten> okay, needs more thought and talking, but let's move on to the next topic. I see there's more on the agenda now.
14:22:55 <thms> yeah, animation is a thing that you will only get with serious efforts in CSS. that not feasable in the general case
14:23:12 <karsten> :)
14:23:14 <karsten> * metrics-lib release (karsten)
14:23:20 <karsten> https://lists.torproject.org/pipermail/tor-dev/2015-December/010016.html
14:23:29 <karsten> we finally released version 1.0.0!
14:23:46 <thms> applause! :)
14:23:52 <karsten> and have plans to release version 1.1.0 this year, which will hopefully catch up with all descriptor contents available on collector.
14:24:13 <karsten> thanks! and thanks for testing it.
14:24:27 <Sebastian> oh, I'm dumb.
14:24:42 <karsten> okay, nothing more on that topic from me, just wanted to announce it here.
14:24:49 <karsten> moving on
14:24:50 <karsten> * analytics server status (thms (formerly known as tomlurge))
14:25:20 <thms> just wanted to say taht i’m working on it again after a brief pause
14:25:36 <ln5> GeKo: started
14:25:41 <thms> updating the converter with the awesome new metrics-lib features
14:25:42 <Sebastian> nickm: nvm.
14:26:00 <karsten> thms: great!
14:26:03 <thms> and then switching to Avro, hopefully before christmas
14:26:21 <thms> and the maybe having something runnable for 32c3
14:26:27 <karsten> whee!
14:26:29 <thms> _maybe_…
14:26:44 <thms> there’s still hope ;-)
14:26:51 <karsten> anything you want feedback on here?
14:27:18 <thms> if somebody want’s to review ConvertToJson, you’re welcome!
14:27:41 <karsten> link?
14:27:43 <thms> but it doesn’t throw errors, so … probably not worth it right now
14:27:46 <thms> mom
14:28:05 <thms> https://github.com/tomlurge/mteam/blob/master/src/mteam/ConvertToJson.java
14:29:19 <thms> i have the problem that all results are generated into one file. i could use help to change that, make it more modular
14:29:31 <thms> that’s it from me
14:29:47 <karsten> okay, I'll try to find time to take a look.
14:29:53 <karsten> thanks for the update!
14:30:05 <karsten> * Roster update (virgil, SeanSaito)
14:30:20 <virgil> Sean, go ahead.
14:31:03 <virgil> okay I'll do it.
14:31:25 <virgil> so you can see the surface improvements on www.tor-roster.org
14:31:36 <virgil> most notably, pagination, can search by fingerprint/nickname, etc.
14:31:52 <virgil> nothing unsurprising
14:32:07 <SeanSaito> also migrated to Digital Ocean
14:32:10 <SeanSaito> from Heroku
14:32:23 <virgil> and there's still some badges
14:32:33 <virgil> I have a first-approximation for the diversity metric.
14:32:41 <virgil> see pretty badges: http://www.tor-roster.org/family_detail/DC41244B158D1420C98C66F7B5E569C09DCE98FE
14:32:46 <virgil> SO SHINY
14:33:27 <SeanSaito> more cosmetic updates to come in the next few days
14:33:37 * karsten still looks
14:33:38 <SeanSaito> including switching to openmap instead of google map
14:33:40 <virgil> In short the first-approximation of the diversity metric using the BGP-routing tables from routeviews.org
14:34:07 <virgil> and then calculates between-ness centrality between each pairs of ASs that has a Tor node on it.
14:34:26 <virgil> currently working on making pretty pictures, but you can see some ugly ones here:
14:35:07 <virgil> http://dl.dropbox.com/u/3308162/twopi.pdf
14:35:38 <virgil> green nodes are ASs with a Tor node on them, orange nodes are without a Tor node.
14:36:00 <virgil> in short, the more edges a node has, the more useful it is to tap that AS-number and do traffic correlation attacks.
14:36:40 <virgil> I am currently spending time figuring out why this Java-graphing program doesn't want to display all my edges.
14:37:04 <thms> but it looks neat :)
14:37:13 <virgil> as a preliminary result though, with only 3 ASs you can sit between 90% of BGP-routes between Tor nodes.
14:37:23 <virgil> which is... I'm not exactly sure how concerning that is.
14:38:04 <virgil> but I am checking it and once I believe the results (and have pretty pictures), I'll take it to the Network team and see exactly how bad this is.
14:38:46 <karsten> where would people file bugs for the current roster?
14:39:05 <virgil> send us an email?
14:39:10 <karsten> or would you rather want to keep working on the things you're working on anyway and ask for feedback later?
14:39:20 <karsten> is there a bug tracker of some sort?
14:39:26 <virgil> github?
14:39:28 <SeanSaito> there is a github!
14:39:41 <karsten> yep. do you want issues to be opened there?
14:39:50 <SeanSaito> sure, that would be helpful!
14:39:52 <SeanSaito> https://github.com/seansaito/Roster
14:39:58 <karsten> okay.
14:40:10 <thms> why do use a java graphing lib? sounds like this could beneft from some interactivity like choosing all AS in the USA, or selecting some wellknown AS, etc. which would be easier to implement with D3 probably?
14:43:42 <karsten> virgil: see thms's suggestion above if you like ^
14:44:01 <virgil> thms: because I'm old.
14:44:11 <karsten> SeanSaito: I'll take a closer look and send some feedback/create some issues on github.
14:44:31 <karsten> great stuff!
14:44:31 <virgil> thms: Highly in favor of D3, I just don't know it.
14:44:37 <thms> virgil: probably younger than me ;-)
14:44:46 <virgil> thms: right now using gephi
14:44:49 <virgil> and graphviz
14:46:14 <virgil> everything here is open-source, and a D3 interactivitiy would be lovely.
14:46:39 <thms> ok. well … i see :)
14:46:57 <virgil> right now main goal is simply to quantify susceptibility to traffic correlation attacks at which the result will be an Real number, not an interactive fun thing ;)
14:47:40 <thms> yep. often a simple list does just as well
14:48:03 <karsten> hopefully the result will be easy to understand for users.
14:48:14 <virgil> okay I'm done.
14:48:15 <karsten> like, why do I get 15 points there, and that other relay gets 1500?
14:48:29 <virgil> karsten: yes, a sort of diagonistics for where the points came from
14:48:55 <karsten> right.
14:48:59 <SeanSaito> karsten: yes, I hope to clean that part up after the cosmetic treatments
14:49:22 <karsten> cool! looking forward to it.
14:49:23 <virgil> total_points = \sum_{i=1}^n w_i p_i
14:49:38 <virgil> where w_i is the weight of the category, and p_i is the number of points
14:49:42 <virgil> in short something like that
14:49:51 <virgil> linear functions FTW
14:50:11 <karsten> sounds good.
14:50:25 <karsten> though by points I meant points for BGP diversity in particular.
14:50:52 <karsten> I guess you'll come up with some algorithm to assign a diversity number to a relay or family.
14:51:09 <karsten> and the user will want to understand that number, or at least how it compares to others.
14:52:05 <karsten> weighting and summing up all points for an overall point number is fine, too, but I think explanations for the single inputs to that function need to be given, too.
14:53:04 <karsten> but, let me not distract you from doing that research first. ;)
14:53:14 <karsten> moving on
14:53:18 <karsten> * Anyone good at pretty pictures of networks? (virgil)
14:53:21 <karsten> I guess we covered that.
14:53:35 <karsten> * Next meeting on Dec 31?
14:53:50 <karsten> so, the next meeting is in 2 weeks, which happens to be on dec 31.
14:54:04 <karsten> how do people think about that?
14:54:09 <karsten> will everyone be busy on dec 31?
14:54:20 <SeanSaito> I will be! sorry
14:54:24 <thms> I’ll not be around on 31.12
14:54:46 <virgil> I can do 3 weeks
14:54:53 <virgil> spend the extra week with family yo.
14:55:08 <karsten> that would be jan 7.
14:55:14 <karsten> how does that sound?
14:55:20 <thms> that would be okay
14:56:05 <SeanSaito> ok!
14:56:08 <karsten> ok. I'll suggest that on the mailing list.
14:56:40 <karsten> in 1.5 weeks or so, or nobody will remember in 2 weeks.
14:56:54 <karsten> great.
14:56:57 <karsten> anything else?
14:58:13 <karsten> okay. thanks for coming and thanks for a good meeting!
14:58:20 <karsten> #endmeeting