17:00:21 <sbs> #startmeeting
17:00:21 <MeetBot> Meeting started Mon Oct  5 17:00:21 2015 UTC.  The chair is sbs. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:21 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
17:00:28 <sbs> well, let's start the meeting
17:00:31 <sbs> so, who is here?
17:00:40 * poly waves
17:00:46 <vtduncan> hi i'm here :)
17:01:16 <sbs> vtduncan: hi!
17:01:30 <juga> hi
17:01:53 <hodgepodge> hi!
17:02:01 <tomlurge> hi, I’m tomlurge from the measurement team
17:02:10 <sbs> tomlurge: hi!
17:02:34 * dalla_ here
17:03:48 <sbs> well, let's start with some report backs - I'll go first
17:06:04 <sbs> As you probably already know from ooni-dev@ the ADINA hackathon was postponed, but we managed to organize a minor hackfest in Rome. I will soon send a report onto this hackfest in list. In addition to me, attended the hackfest also poly, vtduncan, dalla_, Nuke and others. We did great progress with respect to measurement-kit and network-meter. Also we did good stuff with respect to data visualization and present
17:06:10 <sbs> ation. Guess others will tell more on this later.
17:06:17 <sbs> EOF
17:06:40 <antonio__> Hi
17:06:48 <sbs> antonio__: hi!
17:08:55 <juga> ok, i go next
17:09:07 <juga> waiting for psiphon PR to be review and OpenVPN request to be merged
17:09:10 <juga> EOF
17:09:18 <poly> my turn!
17:09:43 <poly> I worked with sbs on network-meter at the mini-hackathon in Rome
17:10:10 <poly> we have been able to work on many features and plan even more during the event
17:10:47 <poly> this included: removing redudant caching from NM
17:11:00 <poly> implementing proper OONI integration
17:11:15 <poly> enabling running multiple tests in parallel
17:11:30 <poly> storing the results of completed tests
17:11:40 <poly> and presenting all of this to the user via the GUI
17:12:06 <poly> we planned the next step which is visualizing the results, and we'll be working on that in the few weeks to come
17:12:08 <poly> EOF
17:12:11 <hodgepodge> I'll go next - first of all, I posted commentary on an [ooni-dev] thread with regards to adoption of Hadoop by the measurement team for performing ad-hoc analytics on data aggregated from ooni-probes. If anyone here would like to touch base with me on that, I'd love to hear from you.
17:12:14 <hodgepodge> In addition, over the course of the past couple of weeks, I have also been working on an API for ooni-probe metrics - currently, I am developing the back-end for the service in Java, and writing a Python API wrapper to allow devs. to query the data that OONI has collected in a familiar, and flexible way. The back-end is currently in PostgreSQL.
17:12:21 <hodgepodge> Hopefully this will reduce the turnaround time associated with the development of tools that use ooni-probe metrics under the covers. Currently, I am generating choropleth maps for different countries for which network tampering has been detected, and summarizing which pluggable transports are blocked in a given country.
17:12:35 <hodgepodge> Also, over the next couple of weeks, I will determine how to be more terse in my reply-back blocks.
17:12:39 <hodgepodge> EOF
17:13:04 <vtduncan> I'll go next
17:13:21 <vtduncan> made this vis at the mini hackfest:
17:13:26 <vtduncan> https://vtduncan.github.io/ooni-globe
17:13:37 <vtduncan> it's quite basic, here's the repo:
17:13:43 <vtduncan> https://github.com/vtduncan/ooni-globe
17:14:00 <vtduncan> also made this tool:
17:14:01 <vtduncan> https://github.com/vtduncan/asn-geo
17:14:18 <vtduncan> for mapping OONI reports to lat/lon coordinates (roughly)
17:14:36 <vtduncan> pulled it out because it might be useful for others' maps
17:14:45 <vtduncan> that's all from me
17:14:56 <hodgepodge> Very nice! I'm a little surprised by how many points you have - the dataset that I am working with only has a handful of ASNs.
17:15:26 <vtduncan> it was however much copied to the USB drive before it had to be pulled to run to the hackfest!
17:15:49 <vtduncan> so about 2 years of data, missing most of 2015 I think
17:16:06 <sbs> hodgepodge vtduncan: yes, that dataset was the mirror of the data currently available on ooni.tpo/reports/
17:16:31 <sbs> hodgepodge vtduncan: minus a part that was not copied because at a given point I had to stop mirroring and to rush to take a train to Rome :P
17:16:48 <vtduncan> oh and one ASN will show up as many points on the map
17:17:05 <vtduncan> there is jitter added so you can see how many reports for each location
17:18:24 <vtduncan> (p.s. there is a slider hidden on the left of the globe in case you hadn't found it)
17:18:32 <vtduncan> that's all - for real this time
17:19:47 <sbs> excellent! anyone else?
17:22:17 <tomlurge> should I …?
17:23:16 <sbs> tomlurge: yes, please
17:24:15 <tomlurge> i’m here because i hope for more feedback on my mail about hadoop. thanks hodepodge for your answer! i guess though that PostgreSQL is not the right tool for our job. we have a lot of data and we’d like to provide a toolset with which people can work on it in different ways: mapReduce, SQL, R, Tableau etc…
17:25:15 <hodgepodge> Gotcha. Would it be worthwhile to have this discussion now, or post-meeting?
17:25:37 <tomlurge> as you wish!
17:27:30 <hodgepodge> I think that it would be best to have this discussion after the meeting, since we're mainly doing status reporting right now. I'd love to also hear from others who would like to participate in the discussion after the meeting.
17:27:39 <hodgepodge> Does that work for you?
17:27:41 <sbs> tomlurge: maybe discussion post meeting? I personally know little about the backend and cannot help in this respect, but I'll make sure that other devs who can reply to your question on the mailing list look at it and reply
17:28:21 <sbs> any other reports backs? or shall we move to the todo-next phase?
17:28:24 <tomlurge> post meeting it is. thanks!
17:28:52 <antonio__> At the mini hackfest I worked to the measurement-kit app for iOS with sbs and nuke. I worked to set up a cocoapod to easily integrate the library into an xcode project.
17:28:57 <antonio__> EOF
17:31:03 <dalla_> The team I was part started reorganizing data to generate written reports. We managed to create reports for two tests. https://github.com/alemela/ooni-report. EOF
17:31:15 <sbs> excellent!
17:32:51 <sbs> ok, guess it's time to say what we'll do next, I'll start:
17:33:06 <sbs> - review, test, and merge juga's pull request
17:33:26 <sbs> - open a pad on riseup.net so that everybody at the hackathon can contribute to write a report back for the ooni mailing list
17:34:21 <sbs> - do more measurement-kit development (need to integrate geoip and json to implement the ooni collector client)
17:34:45 <sbs> EOF
17:35:19 <juga> recode psiphon PR if needed
17:35:20 <juga> EOF
17:36:14 <poly> sbs: any updates on the issue I opened with ooni-probe?
17:36:35 <poly> (regarding writing reports to stdout)
17:36:48 <sbs> you mean this one? https://github.com/TheTorProject/ooni-probe/issues/415
17:37:41 <poly> yeah
17:38:15 <sbs> poly: no, I don't think so - is this something that blocks network-meter development?
17:38:43 <poly> no, it can be worked around AFAIK
17:39:10 <sbs> sbs: ok, then I'll work around it, for now
17:39:54 <sbs> poly: (but I will ping others to see if we can have feedback on that)
17:40:40 <hodgepodge> @all: by the way, it sounds like we are all developing tooling for accessing, and normalizing ooni-probe reports - it would be worthwhile to have a discussion after the meeting with regards to how people are normalizing metrics, and what they are looking for within the measurements sampled by ooni-probes. IMO will help myself, and the measurement team determine how to model our databases to suit devs.
17:41:03 <hodgepodge> (again, I'll learn how to be more terse)
17:41:44 <sbs> hodgepodge: I totally agree on this!
17:43:46 <sbs> any other todos?
17:44:55 <vtduncan> well, there's lots to do on the visualization but I won't be working on it for a little while
17:45:44 <vtduncan> one of the main goals is to give probe operators acknowledgement/recgonition that their data is being contributed
17:46:06 <vtduncan> and identify areas where more probes are needed
17:46:54 <hodgepodge> I'll be adding more visualizations as well. Currently I have a breakdown of which pluggable transports are being blocked, and where, by correlating ASN to the administrative region associated with the netblock. I'm also correlating reports to censorship reports from diff. journalists.
17:47:50 <sbs> hodgepodge: awesome!
17:51:33 <sbs> well, if there are no more todos, I guess we can end the meeting
17:53:50 <sbs> thanks everyone for attending!
17:53:54 <sbs> #endmeeting