13:59:33 <karsten> #startmeeting metrics team
13:59:33 <MeetBot> Meeting started Thu Nov  5 13:59:33 2015 UTC.  The chair is karsten. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:59:33 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
13:59:34 <tomlurge> nickm: ticket # or mail-adress, please
13:59:47 <karsten> hi. who's here for the metrics team meeting?
14:00:00 <tomlurge> tomlurge is
14:00:06 <karsten> hello!
14:00:23 <nickm> tomlurge: looks like I forgot to make a ticket; I'm nickm@torproject.org .
14:00:30 <karsten> agenda draft is here: https://pad.riseup.net/p/zUNzEIFRq5S4
14:01:24 <karsten> please feel to suggest more topics.
14:03:22 <karsten> starting with the first topic in 1 minute.
14:04:48 <karsten> 1) participation in 1-1-1 task exchange (karsten)
14:05:14 <karsten> so, we'll have a 1-1-1 task exchange session right after this one.
14:05:31 <karsten> last week's was not that well attended, and I wonder how to fix that.
14:06:00 <karsten> I suggest that people from this meeting stay around another 30 minutes or so and participate in that 1-1-1 session.
14:06:31 <karsten> guess this was more of an announcement than a discussion topic. unless people have thoughts on this?
14:07:08 <karsten> okay, 2) team landing page and mailing list (karsten)
14:07:24 <karsten> look here: https://trac.torproject.org/projects/tor/wiki/org/teams/MetricsTeam
14:07:50 <karsten> that page also lists the team mailing list.
14:08:17 <karsten> oh, and it contains a loooong list of open tickets. just in case people get bored.
14:09:44 <karsten> and moving on to
14:09:53 <karsten> 3) analytics server and descriptor-to-JSON converter (tomlurge, karsten)
14:10:02 <karsten> tomlurge: want to give an update on the analytics server?
14:10:56 <tomlurge> that crawled along last week. should pick up speed again now that we decided to go with the simple Java converters
14:11:14 <karsten> right. to provide more context,
14:11:33 <karsten> we tried to use a grammar and parser generator to build a translator from tor-dir-spec to json.
14:11:45 <karsten> that turns out to be a neat idea but harder to do in practice,
14:12:05 <karsten> so we should just build the converter using metrics-lib and continue working on the grammar independent of this.
14:12:22 <isis> nickm: thanks!
14:13:00 <karsten> what's the current state of the analytics server? it can read gz-compressed json and provide an SQL-like interface?
14:13:29 <tomlurge> yes, it can do that. it’s just the data that’s missing
14:13:41 <karsten> ok.
14:14:08 <tomlurge> do you have an idea of how work on grammer and parser wil go on?
14:14:25 <tomlurge> the grammer looked very useful on itself
14:14:46 <karsten> I hope to find somebody to pick this topic up as a 1-1-1 task.
14:14:59 <karsten> well, if somebody else here wants to pick it up, that would work, too.
14:15:12 <karsten> basically, I cannot put more hours into this right now.
14:15:26 <karsten> but I think I left it in a state where somebody else can continue.
14:15:33 <karsten> with lots of comments and open issues and questions.
14:15:38 <karsten> let me paste the url..
14:15:47 <tomlurge> ok. pity…
14:16:02 <karsten> https://people.torproject.org/~karsten/volatile/BridgeNetworkStatus.g4
14:16:26 <karsten> indeed. the issue is that stuff keeps piling up while doing fun stuff, and that makes it less fun.
14:18:08 <tomlurge> so we’re stuck with the java-converter(s) for the near future? that’s at least not the worst kind of motivation.
14:18:43 <karsten> well, even if we have a working thing based on antlr, the converters will keep working.
14:19:10 <karsten> hard to say when the antlr (or whatever tool we'll pick) will be working.
14:19:24 <karsten> it could be that somebody shows up who has done 10 similar things and does this in an afternoon.
14:19:55 <karsten> well, not quite, but in fewer afternoons than I would have to spend on this.
14:20:22 <karsten> okay, should we move on?
14:20:28 <karsten> to the next topic, that is.
14:20:39 <tomlurge> right now I’m thinking of them as a special case solution because Drill (the SQL like interface to JSON data) needs very verbose data. I hope that the really tedious parts (checking for attributes that are not present in the data but specified in the spec) could be reused in a more elegant solution later
14:20:52 <tomlurge> yep. let’s move on! :)
14:21:07 <karsten> sorry, I didn't know you were typing.
14:21:11 <karsten> didn't want to rush.
14:21:11 <tomlurge> np
14:21:30 <tomlurge> nobno, i’m okay
14:21:36 <karsten> I don't fully understand what drill needs.
14:22:10 <karsten> but I could imagine that we'll be able to satisfy it with either converter.
14:22:42 <karsten> in the antlr case we'd be able to build our parse tree walker in a way that it outputs what we need.
14:22:51 <tomlurge> drill provides an SQL interface on JSON data. that is astonishing in itself. to pull it of it needs the data to be very regular. which means that we have to add a lot of null values bevause the collector data is very irregular
14:22:55 <karsten> and in the metrics-lib case we can define what we want to output, too.
14:23:35 <karsten> does drill really not handle elements not being available?
14:24:03 <karsten> but,
14:24:08 <tomlurge> yep, if you ask for them.
14:24:15 <karsten> we should probably take this offline.
14:24:34 <tomlurge> if you only query the regular parts, it won’t bother apart irregular parts
14:24:54 <karsten> err,
14:24:54 <tomlurge> yes, let’s do this by mail
14:24:58 <karsten> ok.
14:25:05 <karsten> let's move on then.
14:26:00 <karsten> 4) ideas for data visualizations (karsten)
14:26:10 <karsten> I just mailed ideas to metrics-team@, link is https://lists.torproject.org/pipermail/metrics-team/2015-November/000002.html
14:26:58 <karsten> people here should take a look, and if there's anything that they'd want to work on, let me know.
14:27:51 <karsten> happy to help with providing data.
14:28:09 <virgil> this one is my personal favorite
14:28:10 <virgil> https://torflow.uncharted.software/
14:29:15 <karsten> oh, that is new.
14:29:49 <virgil> I personally would like to see OPI's OnionMap in better shape.  I would do it myself but I'm sure something more JavaScript versatile could do it much better/faster.
14:30:26 <virgil> OnionMap: https://opi.github.io/onionmap/
14:31:12 <virgil> that's all I got.  I currently have no new visualizations I wish to undertake.
14:31:34 <karsten> I should try to get links to those to onto metrics.
14:32:02 <karsten> though onionmap doesn't load here. but, heh, that could be because onionoo is rather broken at the moment.
14:32:42 <dgoulet> asn: ack, agreed with you there, edge case an issue but manageable
14:32:55 <asn> dgoulet: how manageable?
14:33:04 <asn> it just seems quite unlikely, not manageable.
14:33:21 <karsten> okay, let's see if somebody else picks an idea from the metrics-team mailing list.
14:33:27 <karsten> no need to find somebody right now.
14:33:31 <asn> dgoulet: let's move #tor-project :)
14:33:40 <virgil> accept my request to metrics-team!
14:33:57 <karsten> virgil: hmm?
14:34:23 <karsten> virgil: did you have trouble subscribing?
14:34:41 <virgil> nevermind found it
14:35:11 <virgil> karsten: ever considered putting Onionoo on a CDN?
14:35:16 <virgil> say something like Cloudflare/Fastly?
14:35:33 <virgil> that way when it goes down it'll at least spit out cached versions
14:35:50 <karsten> well, it's not just content. it contains a query module, too.
14:37:17 <virgil> I don't quite understand how Onionoo is setup, but this seems like something where we want more uptime than we currently have.
14:37:27 <virgil> and CDNs are fastly for ensuring things like this.  That's all I have.
14:37:33 <virgil> CDNs are made for*
14:38:10 <karsten> let's add it to the agenda.
14:38:27 <karsten> but let's get to Roster updates (virgil, Sean) first.
14:38:33 <karsten> how are things going?
14:39:11 <karsten> should we paste your bullet points from the pad here?
14:39:15 <virgil> Roster is still going.  Been a bit slower as I've tried to get the ATLAS probes and tor2web metrics moving forward.
14:39:22 <karsten> (the pad is mostly for agenda-making.)
14:39:25 <virgil> most exciting thing is the list of badges.
14:39:37 <virgil> http://www.tor-roster.org/badges
14:39:44 <virgil> You can see it, it works.
14:39:57 <virgil> Sean is busy with exams
14:40:04 <virgil> and I'm doing graph-theory on the BGP-peering database.
14:40:29 <virgil> oh
14:40:32 <virgil> for frontend people
14:40:36 <virgil> I did discover something nice
14:40:46 <virgil> https://icomoon.io/
14:40:58 <virgil> you specify a list of SVGs to become ligatures for a custom font
14:41:17 <virgil> and then you can do like <font face="MyIcons">gear</font>
14:41:37 <virgil> and it will display a gear.  The benefit of doing this is that this way you can apply all of your CSS styling to your icons.
14:41:48 <virgil> pretty pimp. IMHO.
14:41:54 <virgil> done.
14:41:55 <karsten> nice work on the badges.
14:42:17 <virgil> these badges will obviously eventually be changed.
14:42:19 <karsten> I wonder if the colors work if you're just seeing one of them.
14:42:35 <virgil> no, the colors are just stopgap
14:42:51 <asn> virgil: fun badges :)
14:42:59 <virgil> it'll probably be something like
14:43:11 <tomlurge> virgil: thanks for the link to icommon.io! that looks good
14:43:15 <virgil> "Bronze Silver Gold ELECTRIC."
14:43:25 <virgil> where electric does some sort of CSS pulsating.
14:43:34 <karsten> :)
14:43:58 <tomlurge> “blink”?
14:44:12 <virgil> blink too much
14:44:59 <virgil> probably sometihing like this http://codepen.io/iamvdo/pen/maJhu
14:45:49 <virgil> that's it.  We'll be adding more badges into the points calculation and they will appear in the list when they do.
14:45:53 <tomlurge> oha! pretty nifty. the “blink” comment was ironic, of course.
14:46:08 <karsten> nice. okay, I still have "read roster code" on my list and hope to find time for that soon.
14:46:25 <karsten> of course, if others here feel like reviewing that code, that would be neat.
14:46:49 <karsten> (which wouldn't mean that I'd not read it in that case.)
14:46:58 <karsten> ((too many nots?))
14:47:38 <karsten> virgil: I'm not sure what to do with atlas credits.
14:47:49 <karsten> but I also didn't read anything. can you give a short summary?
14:48:23 <virgil> karsten: In short you can run programs on 9000 internet-connected raspberry pi's scattered throughout the world.
14:48:38 <virgil> trace-routes and pings are the most obvious thing to run
14:48:59 <virgil> or DNS lookups
14:49:29 <karsten> who decides which programs are run?
14:49:45 <virgil> karsten: whoever has the credits.  You spend credits to run programs on them.
14:50:16 <virgil> karsten: and Tor Project is currently generating a large number of credits, and will be generating an increasing amount the more ATLAS probes we deploy.
14:50:18 <karsten> what if one such program gets the owner of one of those 9000 raspberries into trouble?
14:50:45 <virgil> karsten: I presume if you ran naughty programs your acct would be canceled.
14:51:01 <karsten> naughty may depend on jurisdiction.
14:51:17 <karsten> did you talk to ooni people about this?
14:51:40 <karsten> they might be interested, but they're also thinking a lot about ethical implications for ooni-probe operators.
14:51:58 <Yawning> wait, if there's 9000 rpis around the world that we can run code on
14:52:04 <Yawning> why not add them to the bridge pool
14:52:07 <Yawning> >.>
14:52:21 <Yawning> or is it "only short lived processes"
14:52:27 <virgil> Yawning: that's too much.  These are like single-request processes.
14:52:30 <virgil> very short-lived.
14:52:44 <Yawning> ah
14:53:13 <karsten> ok.
14:53:13 <Letty|Katja> hello
14:53:19 <karsten> hi Letty|Katja!
14:53:30 <karsten> we have 6 minutes left in this meeting. want to use them?
14:53:51 <karsten> (we can still talk about the only remaining topic in 2 weeks.)
14:54:39 <Letty|Katja> i need to upload the new prototype, so you can see the new version with the data for october
14:54:51 <karsten> (in 2 weeks or on the team mailing list, which was made for discussions taking place between two irc meetings.)
14:54:59 <karsten> Letty|Katja: curious!
14:55:42 <Letty|Katja> http://letty.io/tor/onion_v2.html (use the exit button and then play)
14:56:20 <karsten> okay, I think some context is needed for this. :)
14:56:29 <karsten> let me explain the idea, if you don't mind?
14:56:40 <Letty|Katja> that would be great :)
14:56:59 <karsten> every circle represents how many votes contain a give flag for a relay.
14:57:11 <karsten> the big green one is where all votes/authorities agree on something.
14:57:20 <karsten> in this case the number of relays having the Exit flag.
14:57:36 <karsten> the second ring is where 1 authority doesn't give that relay the Exit flag.
14:57:59 <karsten> the non-green circles are when a relay received fewer Exit flags than are required for that flag to make it into the consensus.
14:58:08 <karsten> overall, Exit looks quite healthy.
14:58:25 <karsten> oh, look, Fast is different.
14:58:46 <karsten> quite some relays that don't have their Fast flag in the consensus.
14:58:59 <karsten> you should have graphed BadExit, too.
14:59:08 <karsten> there was so much disagreement..
14:59:21 <karsten> this looks great!
14:59:24 <phw> fwiw, the atlas network is not flexible because it only allows ping, traceroute, dns and tls measurements, iirc.  it would be useful, however, to measure the reachability of services such as the dirauths over time.
14:59:41 <tomlurge> katja|letty: one question: do the diameters or the area of the rings represent the number of votes? or a combination thereof? it’s a difficult decision i think because the visual effect is quite different.
15:00:10 <karsten> phw: I wonder if dgoulet would be interested in that.
15:00:27 <tomlurge> ^Letty|Katja
15:00:32 <Yawning> could also check for people doing sketch stuff with dns/tls?
15:00:40 <karsten> tomlurge: areas.
15:00:59 <karsten> but, I think we ran out of time.
15:01:18 <phw> Yawning: yes.  many probes are sitting in people's living rooms, giving us access to personal internet connections.
15:01:27 <Yawning> ooh
15:01:31 <dgoulet> phw: yes! reachability I want that here ygzf7uqcusp4ayjs.onion :)
15:01:34 <karsten> Letty|Katja: very quickly, I think this is great. it needs some more text, more data (which I'm happy to provide), and then it should go to tor-dev@ for more feedback.
15:01:47 <karsten> Letty|Katja: happy to help with the text, if you want.
15:01:59 <phw> the atlas network has a great API that allows for convenient automation.  you can schedule periodic tasks and get the results over their API.
15:01:59 <tomlurge> areas make the inner rings seem bigger in comparison
15:02:26 <Letty|Katja> karsten: i have all data inside (also badexit) i just need to add the buttons. sure, we can talk about that via mail :)
15:02:34 <karsten> tomlurge: actually, Letty|Katja started with diameter, and I suggested area. ;)
15:02:53 <karsten> Letty|Katja: or we could chat more tomorrow if you're around?
15:03:02 <tomlurge> visually giving the impression of more consensus than there really is
15:03:29 <Letty|Katja> karsten: sure, i check my calender later
15:03:35 <karsten> tomlurge: hmmmm. maybe.
15:04:06 <karsten> Letty|Katja: sounds great! maybe email me and we talk about next steps.
15:04:15 <tomlurge> but he other way roud it’s not better, just the otehrw ay round. but i don’t know a solution. there should be an algorithm that middles between the two
15:04:16 <Letty|Katja> tomlurge: we can talk about that after? feedback is great
15:04:17 <karsten> (like providing you with tons of data.)
15:04:23 <tomlurge> ok
15:04:43 <karsten> okay, 5 minutes over time. we'll have to end the meeeting now, because the other meeting started 5 minutes ago!
15:04:56 <karsten> thanks, everyone! talk to you in two weeks. or on the mailing list.
15:04:59 <karsten> #endmeeting