16:59:56 <anadahz> #startmeeting
16:59:56 <MeetBot> Meeting started Mon Jan  4 16:59:56 2016 UTC.  The chair is anadahz. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:59:56 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
17:00:04 <anadahz> hi everyone
17:00:41 <anadahz> so who's attending today's meeting?
17:00:58 * irl 
17:01:05 <simonv3> me
17:01:12 * anadahz as well
17:02:36 <anadahz> ok last week there were some OONI folks attending 32c3
17:02:50 * sbs is here
17:03:00 <anadahz> hi sbs
17:04:09 <anadahz> we managed to brainstorm and talk with response to OONI roadmap 2016
17:04:10 <simonv3> did you all get a chance to meet in person?
17:04:14 <simonv3> cool
17:04:43 <anadahz> as willscott already mentioned https://pad.puscii.nl/p/ooni32c3
17:05:08 <sbs> anadahz: hi!
17:05:43 <anadahz> hellais: has a version with priorities so i guess he 'll update it soon
17:06:52 <anadahz> together with willscott we gave a presenation at 32c3: https://media.ccc.de/v/32c3-7143-the_state_of_internet_censorship
17:07:30 <anadahz> so let's start with some reports back
17:08:21 <anadahz> what did you do last week or last 2 weeks since we had no IRC meeting last week
17:09:32 <sbs> I'm studying how to integrate Tor inside measurement-kit EOF
17:11:29 <anadahz> from my side i partly reviewed some pluggable transports tests and fixed some lepidopter build script and crontab bugs
17:11:50 <hellais> hey
17:11:52 <hellais> sorry for the delay
17:12:12 <anadahz> i have added some test servers (bare metal) to reduce the AWS costs for the pipeline
17:12:17 <anadahz> hi hellais
17:12:43 <anadahz> EOF
17:13:27 <irl> i've been looking at the lepidopter dependencies and packaging those which are not yet in debian, there are only two that are missing, meek and fteproxy
17:13:43 <hellais> I've been working on making 1-1 mapping between legacy raw YAML reports and JSON based ones. This will mean that from now on the pipeline will operate only on JSON (this means sanitisation, stream generation, heuristics, etc. will be done from the JSON reports)
17:14:02 <irl> meek is blocking on a bug in meek, the debian package that infinity0 has is heavily patched and not suitable for inclusion, fteproxy packages exist and just need sponsorship which i can do, so that should be simple
17:14:09 <hellais> it's running on the server anadahz setup for me and so far 205/315 GB have been converted
17:17:22 <anadahz> hellais: how fast is the server compared to AWS?
17:17:50 <hellais> anadahz: this problem is not IO bound so it's much slower since it only has 8 cores
17:20:48 <simonv3> I did a whole bunch of cleaning up and styling for the ooni-api, as well as adding really basic country information for each country. I opened some requests for API endpoints on filtering content. EOF
17:25:41 <simonv3> It sounds like from the meeting at 32c3 we’ve decided to limit the amount of tests run - is that something that will impact the ooni-api front-end?
17:26:04 <simonv3> Ex - does it make sense for me to style and build the pages for tests that aren’t going to be used anymore?
17:32:26 <anadahz> simonv3: i think joe and hellais know more about this
17:41:05 <hellais> simonv3: I don't think this will impact the ooni-api frontend. Since that will happen in quite a bit of time and still we want to have the views ready for the historical data.
17:41:26 <hellais> We did however come up with some visualisations that we would like to have implemented inside of the ooni-api and some strategies to implement it
17:41:31 <hellais> at least for some set of countries
17:42:06 <hellais> we would like to have in the country views a bar chart that shows number of sites detected as blocked/number of sites tested
17:42:20 <hellais> potentially also tokenising on categories
17:43:08 <simonv3> awesome, yeah an outline of those things would be great
17:43:15 <simonv3> I think some visualization is definitely missing
17:44:29 <hellais> we also think there needs to be some sort of timeline with the number of measurements per country over time. But I think this is lesser of a priority than the former
17:46:27 <anadahz> hellais: could you upload somewhere the prioritized OONI roadmap notes you did at 32c3?
17:48:24 <hodgepodge> @hellais,@landers has there been any further discussion lately regarding merging the joe/redo branch of ooni-pipeline into master?
17:48:36 <hellais> anadahz: you mean the pieces of papers with notes?
17:49:28 <anadahz> hellais: yes the day on the elevators
17:50:01 <hellais> hodgepodge: hey. Yeah we came to the conclusion that it's best to first make the step of converting all the raw reports into JSON and start from now on to operate only on JSON files (sanitisation, heuristics, database insertion, etc.)
17:50:29 <hellais> hodgepodge: I think the code in that branch still need a bit of work before it can be merged into master as it's still very messy and not well documented
17:51:09 <hellais> anadahz: ok will do, but the tasks there are not prioritised.
17:56:29 <anadahz> it seems that there are no more reports back let's move on to the things that we are planning to work next week
17:57:55 <anadahz> i was planning to release the img file of lepidopter and start learning about prometheus monitoring (http://prometheus.io)
17:58:27 <anadahz> hellais: do you have something that needs to be prioritized from my side?
18:00:24 <hellais> anadahz: it would be good to have that postgres database setup on one of the two servers
18:00:50 <hellais> I will be working mainly on data pipeline processing, implementing some heuristics for some of the current tests and extracting data
18:02:08 * gxg waves
18:03:35 <anadahz> hellais: ACK priority set!
18:04:45 <hodgepodge> I'll be working predominantly on service-discovery, and seeing if there are better ways by which we can identify hosts to use within the pipeline (i.e. via service discovery). The goal is to find a way to be run ooni-pipeline against a single host, or a group of hosts, either locally, or remotely.
18:07:22 <simonv3> hellias do you have documented what you would like to see in the countries view - could you send me pictures or an outline? Maybe as a github issue?
18:07:39 <sbs> I will continue to understand how to integrate Tor
18:07:42 <hellais> simonv3: yes I shall file tomorrow some github issues about it
18:10:01 <simonv3> thanks :)
18:13:48 <anadahz> ok i think it's time to end this meeting
18:14:11 <anadahz> feel free to continue with any comments/suggestions/discussions
18:14:22 <anadahz> thanks everyone for attending
18:14:27 <anadahz> #endmeeting