17:00:21 #startmeeting 17:00:21 Meeting started Mon Oct 5 17:00:21 2015 UTC. The chair is sbs. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:00:21 Useful Commands: #action #agreed #help #info #idea #link #topic. 17:00:28 well, let's start the meeting 17:00:31 so, who is here? 17:00:40 * poly waves 17:00:46 hi i'm here :) 17:01:16 vtduncan: hi! 17:01:30 hi 17:01:53 hi! 17:02:01 hi, I’m tomlurge from the measurement team 17:02:10 tomlurge: hi! 17:02:34 * dalla_ here 17:03:48 well, let's start with some report backs - I'll go first 17:06:04 As you probably already know from ooni-dev@ the ADINA hackathon was postponed, but we managed to organize a minor hackfest in Rome. I will soon send a report onto this hackfest in list. In addition to me, attended the hackfest also poly, vtduncan, dalla_, Nuke and others. We did great progress with respect to measurement-kit and network-meter. Also we did good stuff with respect to data visualization and present 17:06:10 ation. Guess others will tell more on this later. 17:06:17 EOF 17:06:40 Hi 17:06:48 antonio__: hi! 17:08:55 ok, i go next 17:09:07 waiting for psiphon PR to be review and OpenVPN request to be merged 17:09:10 EOF 17:09:18 my turn! 17:09:43 I worked with sbs on network-meter at the mini-hackathon in Rome 17:10:10 we have been able to work on many features and plan even more during the event 17:10:47 this included: removing redudant caching from NM 17:11:00 implementing proper OONI integration 17:11:15 enabling running multiple tests in parallel 17:11:30 storing the results of completed tests 17:11:40 and presenting all of this to the user via the GUI 17:12:06 we planned the next step which is visualizing the results, and we'll be working on that in the few weeks to come 17:12:08 EOF 17:12:11 I'll go next - first of all, I posted commentary on an [ooni-dev] thread with regards to adoption of Hadoop by the measurement team for performing ad-hoc analytics on data aggregated from ooni-probes. If anyone here would like to touch base with me on that, I'd love to hear from you. 17:12:14 In addition, over the course of the past couple of weeks, I have also been working on an API for ooni-probe metrics - currently, I am developing the back-end for the service in Java, and writing a Python API wrapper to allow devs. to query the data that OONI has collected in a familiar, and flexible way. The back-end is currently in PostgreSQL. 17:12:21 Hopefully this will reduce the turnaround time associated with the development of tools that use ooni-probe metrics under the covers. Currently, I am generating choropleth maps for different countries for which network tampering has been detected, and summarizing which pluggable transports are blocked in a given country. 17:12:35 Also, over the next couple of weeks, I will determine how to be more terse in my reply-back blocks. 17:12:39 EOF 17:13:04 I'll go next 17:13:21 made this vis at the mini hackfest: 17:13:26 https://vtduncan.github.io/ooni-globe 17:13:37 it's quite basic, here's the repo: 17:13:43 https://github.com/vtduncan/ooni-globe 17:14:00 also made this tool: 17:14:01 https://github.com/vtduncan/asn-geo 17:14:18 for mapping OONI reports to lat/lon coordinates (roughly) 17:14:36 pulled it out because it might be useful for others' maps 17:14:45 that's all from me 17:14:56 Very nice! I'm a little surprised by how many points you have - the dataset that I am working with only has a handful of ASNs. 17:15:26 it was however much copied to the USB drive before it had to be pulled to run to the hackfest! 17:15:49 so about 2 years of data, missing most of 2015 I think 17:16:06 hodgepodge vtduncan: yes, that dataset was the mirror of the data currently available on ooni.tpo/reports/ 17:16:31 hodgepodge vtduncan: minus a part that was not copied because at a given point I had to stop mirroring and to rush to take a train to Rome :P 17:16:48 oh and one ASN will show up as many points on the map 17:17:05 there is jitter added so you can see how many reports for each location 17:18:24 (p.s. there is a slider hidden on the left of the globe in case you hadn't found it) 17:18:32 that's all - for real this time 17:19:47 excellent! anyone else? 17:22:17 should I …? 17:23:16 tomlurge: yes, please 17:24:15 i’m here because i hope for more feedback on my mail about hadoop. thanks hodepodge for your answer! i guess though that PostgreSQL is not the right tool for our job. we have a lot of data and we’d like to provide a toolset with which people can work on it in different ways: mapReduce, SQL, R, Tableau etc… 17:25:15 Gotcha. Would it be worthwhile to have this discussion now, or post-meeting? 17:25:37 as you wish! 17:27:30 I think that it would be best to have this discussion after the meeting, since we're mainly doing status reporting right now. I'd love to also hear from others who would like to participate in the discussion after the meeting. 17:27:39 Does that work for you? 17:27:41 tomlurge: maybe discussion post meeting? I personally know little about the backend and cannot help in this respect, but I'll make sure that other devs who can reply to your question on the mailing list look at it and reply 17:28:21 any other reports backs? or shall we move to the todo-next phase? 17:28:24 post meeting it is. thanks! 17:28:52 At the mini hackfest I worked to the measurement-kit app for iOS with sbs and nuke. I worked to set up a cocoapod to easily integrate the library into an xcode project. 17:28:57 EOF 17:31:03 The team I was part started reorganizing data to generate written reports. We managed to create reports for two tests. https://github.com/alemela/ooni-report. EOF 17:31:15 excellent! 17:32:51 ok, guess it's time to say what we'll do next, I'll start: 17:33:06 - review, test, and merge juga's pull request 17:33:26 - open a pad on riseup.net so that everybody at the hackathon can contribute to write a report back for the ooni mailing list 17:34:21 - do more measurement-kit development (need to integrate geoip and json to implement the ooni collector client) 17:34:45 EOF 17:35:19 recode psiphon PR if needed 17:35:20 EOF 17:36:14 sbs: any updates on the issue I opened with ooni-probe? 17:36:35 (regarding writing reports to stdout) 17:36:48 you mean this one? https://github.com/TheTorProject/ooni-probe/issues/415 17:37:41 yeah 17:38:15 poly: no, I don't think so - is this something that blocks network-meter development? 17:38:43 no, it can be worked around AFAIK 17:39:10 sbs: ok, then I'll work around it, for now 17:39:54 poly: (but I will ping others to see if we can have feedback on that) 17:40:40 @all: by the way, it sounds like we are all developing tooling for accessing, and normalizing ooni-probe reports - it would be worthwhile to have a discussion after the meeting with regards to how people are normalizing metrics, and what they are looking for within the measurements sampled by ooni-probes. IMO will help myself, and the measurement team determine how to model our databases to suit devs. 17:41:03 (again, I'll learn how to be more terse) 17:41:44 hodgepodge: I totally agree on this! 17:43:46 any other todos? 17:44:55 well, there's lots to do on the visualization but I won't be working on it for a little while 17:45:44 one of the main goals is to give probe operators acknowledgement/recgonition that their data is being contributed 17:46:06 and identify areas where more probes are needed 17:46:54 I'll be adding more visualizations as well. Currently I have a breakdown of which pluggable transports are being blocked, and where, by correlating ASN to the administrative region associated with the netblock. I'm also correlating reports to censorship reports from diff. journalists. 17:47:50 hodgepodge: awesome! 17:51:33 well, if there are no more todos, I guess we can end the meeting 17:53:50 thanks everyone for attending! 17:53:54 #endmeeting