16:59:21 #startmeeting 16:59:21 Meeting started Mon Nov 30 16:59:21 2015 UTC. The chair is hellais. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:59:21 Useful Commands: #action #agreed #help #info #idea #link #topic. 16:59:31 so folks, let's get started. Who is here? 16:59:34 here 16:59:41 hello 17:00:03 * sbs here 17:02:02 I guess we can start with some report backs 17:02:23 what occupied your time last week from the ooni multiverse 17:02:24 ? 17:03:00 in the network meter universe I worked with sbs on merging improved regex parsing into NM 17:03:35 sbs provided me with feedback on a couple of points and I've discussed and implemented the changes 17:03:52 they should be pushed to master anytime now 17:04:05 EOU 17:04:10 i've been writing ooni-sysadmin shell scripts. got done with the "deploy backend" one and nearly with the "deploy a scheduled ooni-probe bridge reachability test" one. 17:04:19 eof 17:05:37 I've been working on making changes to the ooni-pipeline to work with the latest version of AWS and re-ran it against all historical data up until date. I also worked on moving the OTF funding proposal forward and talking to joe and brennan to coordinate the OONI data explorer/API work to be done with them. 17:05:58 EOF 17:06:35 Making progresses towards measurement-kit 0.1.0 and reviewed poly's pull request 17:06:37 EOF 17:07:02 [13measurement-kit] 15AntonioLangiu pushed 1 new commit to 06fix/coveralls: 02http://git.io/vB70A 17:07:02 13measurement-kit/06fix/coveralls 14f725418 15Antonio Langiu: Fix 17:08:13 if we are done, are there some topics you would like to bring to the table for discussion? 17:08:55 the next step for Network Meter I think is deciding how to parse the data for visualization 17:09:14 poly: what sort of visualisations are you thinking of? 17:09:17 at the hackathon, we agreed to use D3/C3 for visualization 17:09:37 visualization for the results for the plugin outputs within NM 17:09:42 poly: is there a ticket about it? If not it would be good to open it and invite feedback and discuss it there. 17:09:56 [13measurement-kit] 15bassosimone pushed 1 new commit to 06fix/coveralls: 02http://git.io/vB7uq 17:09:56 13measurement-kit/06fix/coveralls 14e072ec6 15Simone Basso: build matrix: try #1 17:09:57 mockups of what the visualisations would look like are also very useful to describe it 17:10:38 there is indeed a ticket: https://github.com/measurement-kit/network-meter/issues/31 17:11:01 poly: I would suggest starting with some concrete examples of what specific questions you plan to answer visually and then move towards generalising it for all 17:11:14 * poly takes notes down 17:11:29 poly: excellent! It would be good to add to the ticket some notes about what is the produced output that is planned to be visualised 17:11:59 I imagine that in the future most of the heavy lifting would be done in the backend API and in fact the collectors already return a report ID that is added to the report (at least for OONI results) 17:12:27 hellais: what do you mean by heavy lifting? 17:13:28 sbs: I mean the analysis of the result. For example: the http_requests test is currently purely detected on the client side, but in the future shall use the baseline control for comparison since Tor is not an ideal control channel 17:13:37 same thing goes for the DNS consistency test 17:14:21 for OONI tests at least, you are interested in the end in a boolean flag that is an indicator for censorship and I think we have learned that doing this entirely in the client is not feasable especially given the areas where we plan to deploy 17:14:40 s/censorship/anomalous network activity/ 17:14:59 hellais: how is progress on pipeline-ng for generating those baseline controls? 17:15:05 [13measurement-kit] 15bassosimone pushed 1 new commit to 06fix/coveralls: 02http://git.io/vB72i 17:15:05 13measurement-kit/06fix/coveralls 14b605ac4 15Simone Basso: check value of variables 17:15:47 willscott: we had some issues with the pipeline and some of the machines running the baselines. We expect to have a working advanced prototype of the api and pipeline by end of year 17:16:14 are you leading up work on that if I have more questions / want to help? 17:16:22 hellais: understood -- I guess that the OONI apps for mobile and network meter share similar design goals in terms of what is to be presented to the user and in terms of interaction with the backend, then 17:16:35 moving into making them a useful baseline for other test developers will require a bit more time and I am thinking that there should be a way to apply authorship to reports in order to have good baselines 17:16:36 Hi #ooni 17:16:50 willscott: yes, please direct questions to me :) 17:16:54 sorry bit late 17:17:06 landers and another developer are also working on this 17:17:17 sbs: yes I believe so 17:17:45 willscott: what do you think of the plan of adding some sort of signature to the report headers to attribute authorship of the measurements. 17:18:29 willscott: because we also don't want to end up considering as a baseline results that come from the same ASN as a user, but that are not run under our supervision 17:19:28 [13measurement-kit] 15bassosimone pushed 1 new commit to 06fix/coveralls: 02http://git.io/vB7VK 17:19:28 13measurement-kit/06fix/coveralls 14ab419fb 15Simone Basso: try with sudo equal to false 17:19:35 i'm worried that signature validation on every report multiple times would be slow. The thing should be validated once on input, and shouldn't be touched within the analysis code, presumably 17:21:01 willscott: yeah that is a fair point, that is why I think it should go hand in hand with the changing of the reporting mechanism. If we reduce the overhead of having to de-serialise YAML and convert to JSON in the pipeline we probably have enough CPU to do the validation 17:21:27 having a way to know that two reports come from the same client sounds good though 17:21:34 I am also thinking that probably a full signature is not needed for the actual content of the reports, but perhaps something weak like a checksum 17:21:58 the alternative would be to make authenticity be applied to the reports by specific dedicated collectors, that are end to end authenticated with trusted probes 17:22:34 also it wouldn't necessarily have to be that much CPU intensive as most reports would not include the signature, but only the ones that are run for control 17:23:14 If there isn't a ticket already open about this I should probably open one to discuss this furthere there 17:23:22 sounds good 17:25:53 [13measurement-kit] 15bassosimone pushed 1 new commit to 06fix/coveralls: 02http://git.io/vB7rQ 17:25:53 13measurement-kit/06fix/coveralls 14c1e6742 15Simone Basso: comment out code for testing -- hello ooni meeting :) 17:26:05 bnvk: HI! 17:26:27 bnvk: (sorry for the capital) is simon around? 17:27:35 [13measurement-kit] 15bassosimone pushed 1 new commit to 06fix/coveralls: 02http://git.io/vB7oB 17:27:35 13measurement-kit/06fix/coveralls 147520ce9 15Simone Basso: more commenting of .travis.yml 17:27:39 anadahz: simonv3 is MIA 17:27:52 I suspect some timezone related issues 17:29:05 https://github.com/TheTorProject/ooni-probe/issues/438 17:29:10 Hi, I'm just going to casually interject here with regards to checksums, we could use a fast hash function like BLAKE2. If we're using YAML within the pipeline, we should probably switch to JSON until the final reporting stage to reduce the intensity of the YAML type conversion. 17:29:13 willscott: ^^ I made a quick braindump of what we said 17:30:46 hodgepodge: yeah that could be a good idea, though we probably would have to do some benchmarks of this and figure out some best scheme. It's probably a matter of finding some ideal compromise 17:31:07 hodgepodge: can you add that as a comment to the ticket? 17:34:48 on my side: reviewed PRs and help on the OpenVPN and psiphon OONI tests, add some features and bug fixes on lepi# 17:35:07 sbs: shall we contact OTF ? 17:37:33 anadahz: I sent emails to everyone and updated the shared document on the current status, some persons haven't replied yet and I'd wait some 3-5 more days 17:37:50 what is this about? 17:38:09 hellais: reimburrsment for the mini hackthon 17:38:25 ah, shouldn't it go through Aspire? 17:39:14 anyways we can talk about this at a later time 17:39:52 hellais: it was just a reminder for sbs/me 17:40:28 anadahz: sounds good :) 17:40:40 so is there more to be discussed? 17:40:53 otherwise we shall go into next steps 17:42:05 I plan to be submitting the OTF proposal end of day tomorrow and ping them about it. Then I will be working on the ooni-pipeline code and following joe to get him up to speed with what is needed. I will also be following simonv3 that will be working on the GUI related stuff and writing some code here and there. Summary: main focus: * ooni-pipeline code * ooni-api code 17:42:10 EOF 17:43:05 I will continue the process of releasing measurement-kit 0.1.0, help with proposal 17:43:19 EOF 17:43:51 so we've got the add_headers_to_db task running, next is the spark_apps task 17:44:02 after those, we can get the API running 17:44:14 that's roughly it, right hellais ? 17:44:39 landers: correct 17:44:45 then EOF 17:47:06 will finish polishing OONI documentation, bring up some ooni probes, finish work on some reports, fi# 17:47:42 nalize lepidopter user documentation 17:47:45 EOF 17:48:09 I'll brainstorm requirements and plans for visualization within NM 17:48:12 EOF 17:51:53 oh i 'll prepare the OONI report for last month 17:52:16 in case you have anything to share... 17:53:16 you could add it on a pad 17:53:58 anadahz: yeah please add it to a pad 17:53:59 ^ just for Oct. 2015 17:55:08 hellais: ok 17:55:15 ah ok, I shall do the one for November in that case 17:56:52 ok need to go.. 17:57:01 thanks everyone for attending 17:57:03 afk 18:00:11 indeed thanks for being in the meeting 18:00:12 oh 18:00:25 simonv3: it seems like there was a timezone management issue 18:00:35 we just ended the meeting 18:00:51 #endmeeting