16:59:56 #startmeeting 16:59:56 Meeting started Mon Jan 4 16:59:56 2016 UTC. The chair is anadahz. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:59:56 Useful Commands: #action #agreed #help #info #idea #link #topic. 17:00:04 hi everyone 17:00:41 so who's attending today's meeting? 17:00:58 * irl 17:01:05 me 17:01:12 * anadahz as well 17:02:36 ok last week there were some OONI folks attending 32c3 17:02:50 * sbs is here 17:03:00 hi sbs 17:04:09 we managed to brainstorm and talk with response to OONI roadmap 2016 17:04:10 did you all get a chance to meet in person? 17:04:14 cool 17:04:43 as willscott already mentioned https://pad.puscii.nl/p/ooni32c3 17:05:08 anadahz: hi! 17:05:43 hellais: has a version with priorities so i guess he 'll update it soon 17:06:52 together with willscott we gave a presenation at 32c3: https://media.ccc.de/v/32c3-7143-the_state_of_internet_censorship 17:07:30 so let's start with some reports back 17:08:21 what did you do last week or last 2 weeks since we had no IRC meeting last week 17:09:32 I'm studying how to integrate Tor inside measurement-kit EOF 17:11:29 from my side i partly reviewed some pluggable transports tests and fixed some lepidopter build script and crontab bugs 17:11:50 hey 17:11:52 sorry for the delay 17:12:12 i have added some test servers (bare metal) to reduce the AWS costs for the pipeline 17:12:17 hi hellais 17:12:43 EOF 17:13:27 i've been looking at the lepidopter dependencies and packaging those which are not yet in debian, there are only two that are missing, meek and fteproxy 17:13:43 I've been working on making 1-1 mapping between legacy raw YAML reports and JSON based ones. This will mean that from now on the pipeline will operate only on JSON (this means sanitisation, stream generation, heuristics, etc. will be done from the JSON reports) 17:14:02 meek is blocking on a bug in meek, the debian package that infinity0 has is heavily patched and not suitable for inclusion, fteproxy packages exist and just need sponsorship which i can do, so that should be simple 17:14:09 it's running on the server anadahz setup for me and so far 205/315 GB have been converted 17:17:22 hellais: how fast is the server compared to AWS? 17:17:50 anadahz: this problem is not IO bound so it's much slower since it only has 8 cores 17:20:48 I did a whole bunch of cleaning up and styling for the ooni-api, as well as adding really basic country information for each country. I opened some requests for API endpoints on filtering content. EOF 17:25:41 It sounds like from the meeting at 32c3 we’ve decided to limit the amount of tests run - is that something that will impact the ooni-api front-end? 17:26:04 Ex - does it make sense for me to style and build the pages for tests that aren’t going to be used anymore? 17:32:26 simonv3: i think joe and hellais know more about this 17:41:05 simonv3: I don't think this will impact the ooni-api frontend. Since that will happen in quite a bit of time and still we want to have the views ready for the historical data. 17:41:26 We did however come up with some visualisations that we would like to have implemented inside of the ooni-api and some strategies to implement it 17:41:31 at least for some set of countries 17:42:06 we would like to have in the country views a bar chart that shows number of sites detected as blocked/number of sites tested 17:42:20 potentially also tokenising on categories 17:43:08 awesome, yeah an outline of those things would be great 17:43:15 I think some visualization is definitely missing 17:44:29 we also think there needs to be some sort of timeline with the number of measurements per country over time. But I think this is lesser of a priority than the former 17:46:27 hellais: could you upload somewhere the prioritized OONI roadmap notes you did at 32c3? 17:48:24 @hellais,@landers has there been any further discussion lately regarding merging the joe/redo branch of ooni-pipeline into master? 17:48:36 anadahz: you mean the pieces of papers with notes? 17:49:28 hellais: yes the day on the elevators 17:50:01 hodgepodge: hey. Yeah we came to the conclusion that it's best to first make the step of converting all the raw reports into JSON and start from now on to operate only on JSON files (sanitisation, heuristics, database insertion, etc.) 17:50:29 hodgepodge: I think the code in that branch still need a bit of work before it can be merged into master as it's still very messy and not well documented 17:51:09 anadahz: ok will do, but the tasks there are not prioritised. 17:56:29 it seems that there are no more reports back let's move on to the things that we are planning to work next week 17:57:55 i was planning to release the img file of lepidopter and start learning about prometheus monitoring (http://prometheus.io) 17:58:27 hellais: do you have something that needs to be prioritized from my side? 18:00:24 anadahz: it would be good to have that postgres database setup on one of the two servers 18:00:50 I will be working mainly on data pipeline processing, implementing some heuristics for some of the current tests and extracting data 18:02:08 * gxg waves 18:03:35 hellais: ACK priority set! 18:04:45 I'll be working predominantly on service-discovery, and seeing if there are better ways by which we can identify hosts to use within the pipeline (i.e. via service discovery). The goal is to find a way to be run ooni-pipeline against a single host, or a group of hosts, either locally, or remotely. 18:07:22 hellias do you have documented what you would like to see in the countries view - could you send me pictures or an outline? Maybe as a github issue? 18:07:39 I will continue to understand how to integrate Tor 18:07:42 simonv3: yes I shall file tomorrow some github issues about it 18:10:01 thanks :) 18:13:48 ok i think it's time to end this meeting 18:14:11 feel free to continue with any comments/suggestions/discussions 18:14:22 thanks everyone for attending 18:14:27 #endmeeting