16:00:27 <anadahz> #startmeeting
16:00:27 <MeetBot> Meeting started Mon Apr 24 16:00:27 2017 UTC.  The chair is anadahz. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:27 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:00:58 <anadahz> Hello everyone, just started the weekly irc meeting..
16:01:03 <anadahz> Who's around?
16:02:44 * darkk is
16:02:52 <slacktopus> <agrabeli> hello
16:05:31 <slacktopus> <nuke> hi
16:06:04 <anadahz> so it seems that we are gathering :)
16:06:18 <SuperQ> Hi!
16:07:02 <slacktopus> <agrabeli> perhaps we can start with the first agenda item on updating the french test list?
16:07:27 <slacktopus> <agrabeli> for reference, the agenda is here: https://pad.riseup.net/p/ooni-irc-pad
16:07:39 * landers here
16:09:20 <anadahz> @agrabeli thanks yes let's proceed with the first agenda (I was waiting a bit for the people to gather).
16:10:58 <slacktopus> <agrabeli> @anadahz great
16:11:10 <anadahz> #topic Updating French test lists
16:11:14 <slacktopus> <agrabeli> information about how to contribute to test lists in general can be found here: https://ooni.torproject.org/get-involved/contribute-test-lists/
16:12:36 <hellais> o/
16:12:37 <darkk> seems, xeniax failed to join...
16:12:38 <slacktopus> <agrabeli> @anadahz did you have any specific questions you wanted to discuss? :slightly_smiling_face:
16:12:57 <hellais> I think they may have also leaked they nickserv password
16:13:21 <anadahz> My point of starting this was sort of monitoring the French elections, and given the run-off election that will happen in some days it may be very interesting to find out if any network interference is occuring/occured
16:14:11 <anadahz> IIRC there were a number of people living in France and many oonitarians as well (even hanging around in this channel).
16:15:04 <liken> hey, xenia here. changed identity. i've been struggling to find different sources for blocked sites. i found some information that will need some translation from my side.
16:15:28 <liken> also set up a lepidopter in paris and it runs since 22/04.
16:15:48 <hellais> liken: one thing to keep in mind is that the test-lists are not lists of blocked websites, but are rather a list of sites that are relevant to the context of the country that are worthwhile to monitor
16:16:50 <liken> hellais, thanks for the precisions. the docs I found speak about "terrorist threat", and I think that this kind of ressources is worth and relevatn, but correct me. i am new
16:17:19 <slacktopus> <agrabeli> hi @liken!
16:17:21 <hellais> ideally they would include content that is as diverse as possible covering things that are popular in the country and are likely to be missed out when using the global list
16:17:32 <slacktopus> <agrabeli> yes, those types of docs would be relevant
16:17:40 <liken> https://www.cnil.fr/sites/default/files/atoms/files/cnil_rapport_blocage_sites_internet_2016_0.pdf this pdf is a report from 2016 about censorship in France produced by CNIL
16:18:04 <slacktopus> <agrabeli> it would be useful to ensure that blocked sites are included in the test list
16:18:12 <slacktopus> <agrabeli> to examine whether and to what extent they are still blocked
16:18:29 <liken> yes!
16:18:31 <slacktopus> <agrabeli> given ofc that they are not like "2000 websites" :P
16:18:48 <hellais> "it would be useful to ensure that the test list includes some of the blocked sites"
16:19:36 <slacktopus> <agrabeli> generally we try to ensure that test lists have as many different websites that fall under these 31 categories as possible: https://github.com/citizenlab/test-lists/blob/master/lists/00-LEGEND-new_category_codes.csv
16:19:44 <anadahz> Right now the list has 13 entries: https://github.com/citizenlab/test-lists/blob/master/lists/fr.csv
16:19:57 <slacktopus> <agrabeli> Because that means that the user is testing a diverse set of URLs, and the research bias is limited
16:20:11 <slacktopus> <agrabeli> @anadahz yes, it needs some serious updating...
16:20:27 <slacktopus> <agrabeli> generally we try to keep test lists up to 1,000 URLs maximum
16:21:00 <slacktopus> <agrabeli> because larger test lists would consume way too much bandwitdh
16:21:08 <anadahz> so my point was also to call for attention from other people in France that have further info or could contribute to the list
16:21:08 <liken> ok! now the reports speak about 300< blocked urls
16:21:11 <slacktopus> <agrabeli> we try to optimize for quality in test lists over quantityt
16:21:53 <liken> "we try to optimize for quality in test lists over quantityt" - sure! but by now it clearly demands refreshing
16:22:25 <slacktopus> <sbs> Yo
16:22:31 <slacktopus> <agrabeli> @liken great, then it seems reasonable to add all 300< blocked urls, and to supplement those with additional urls that fall under these categories: https://github.com/citizenlab/test-lists/blob/master/lists/00-LEGEND-new_category_codes.csv
16:22:34 <anadahz> relevant/irrelevant French Guiana uses the same blocking infrastructure as France, though geographically very far!
16:22:44 <slacktopus> Action: sbs had issue with my wifi
16:24:00 <hellais> "Sites à caractère pédopornographique" - these MUST not be included in the test list
16:24:18 <slacktopus> <agrabeli> @liken: a sort of "guide" on how to contribute to test lists can be found here: https://ooni.torproject.org/get-involved/contribute-test-lists/
16:24:35 <liken> agrabeli: unfortunately, I've only found these reports that do not contain URLs. I need to do more research to get the URLs themselves...
16:25:01 <liken> they are not accessible for citizens. as soon as I get the URLs, will put them in the lists
16:25:22 <hellais> also it's much better if the list is not only URLs from the government issued blocklists, but also includes content based on research
16:25:33 <slacktopus> <agrabeli> yep
16:25:56 <slacktopus> <agrabeli> contributing to test lists is more about research than about dumping urls from blocklists
16:26:21 <liken> agrabeli, hellais: ok, understood
16:26:24 <slacktopus> <agrabeli> esp since if sites are already known to be blocked, that is somewhat less interesting and probably more "justifiable" by the local government
16:26:34 <anadahz> agrabeli: Given the fact that you have also some experience with election monitoring what else could be useful in the case of France?
16:26:34 <liken> sure
16:27:47 <slacktopus> <agrabeli> @anadahz I think some research would be required on what are the major political topics discussed in that country at the moment, and on who are the opposing political voices. Then it would be useful to find their "voices" through their blogs or other sites and add them to the test list.
16:28:20 <slacktopus> <agrabeli> Basically, it's useful to think from the perspective of the government, what would they potentially have incentive to block...and to add it for testing.
16:28:47 <slacktopus> <agrabeli> Apart from that, it's useful to ensure that such sites are tested consistently across time, before the elections, during the elections, and after
16:29:15 <slacktopus> <agrabeli> This enables for more comprehensive and accurate data analysis later on, which can help determine with more accuracy whether and to what extent censorship events have occurred.
16:30:08 <slacktopus> <agrabeli> it seems that @liken is already doing that, having set up lepidopter :slightly_smiling_face:
16:30:23 <darkk> I think that adding URLs from gov. blocklists is still quite valuable as it allows to learn more regarding blocking infrastructure.
16:30:51 <yulax> !pertinentquestions
16:30:57 <anadahz> Indeed monitoring before/after the election period is very useful. We do have already a number of probes and even some regular/stable ones.
16:31:12 <yulax> hmm
16:31:23 <slacktopus> <agrabeli> @darkk indeed. Just saying that test lists shouldn't be limited to gov blocklists imho, but that they should also include other sites (which might be unofficially, and thus less justifiably, blocked).
16:31:32 <liken> agrabeli: i am doing that but I need to add new urls, will do it in the following days. need to check if some farright websites are blocked [given that the 2nd tour will be LePen vs Macron]
16:31:58 <slacktopus> <agrabeli> @anadahz awesome!
16:32:35 <slacktopus> <agrabeli> @liken yeah it would be useful to include such sites as well
16:33:04 <anadahz> darkk: yep that's usually the best way to find out the censorship infrastructure of an ISP/country/region
16:33:20 <slacktopus> <agrabeli> in general, I would say that it could be useful to add sites even if they are currently not blocked right now in France, because they might get blocked later on. And lepidopter would allow us to detect that, when/if they get blocked.
16:34:17 <anadahz> Though I'm afraid that some URLs will be blocked in a non clean way (like with a block page as in: https://explorer.ooni.torproject.org/measurement/AycuR1ToXlUD5a2JsLa8IrzrgL8pOq6GwJ9Cs7TvxZMx64KQqaXfPlGZQxIIwAck?input=http:%2F%2Fkhilafah.net)
16:34:20 <slacktopus> <agrabeli> So basically, test lists include a mix of URLs that are blocked and that are not blocked currently in a given place.
16:34:55 <liken> agrabeli: I also want to test some popular websites related to islam. And few very popular anarchist resources [such as paris-luttes or zad.nadir.org]
16:35:11 <liken> agrabeli: OK now it is very clear for me. thanxxx
16:35:19 <slacktopus> <agrabeli> @liken: Would be great if all such sites are added ;)
16:35:44 <slacktopus> <agrabeli> @liken: Thank *you* a million for contributing to the test list! Much appreciated :slightly_smiling_face:
16:35:54 <slacktopus> <agrabeli> @liken: And thanks for running ooniprobe in general :slightly_smiling_face:
16:36:15 <slacktopus> <agrabeli> @liken: And feel free to ping us anytime (here or via email) with any other questions you may have.
16:39:15 <hellais> do we have other items to discuss?
16:39:34 <SuperQ> Any questions for me?
16:41:23 <hellais> I haven't had a chance to look at the monitoring stuff that carefully. But from what I can tell you suggest we use the blackbox exporter that basically polls the HTTP endpoints for some expected payload?
16:41:40 <SuperQ> Yes
16:41:48 <hellais> Is there something we can do something more similar to a process monitor?
16:41:49 <liken> agrabeli: thank you for help and inspiration! need to run now.
16:41:49 <SuperQ> That's one use case.
16:42:06 <hellais> s/something we can/something we can use to/
16:42:12 <darkk> btw, does blackbox exporter poll endpoint periodically or does it poll when collector comes?
16:42:30 <SuperQ> hellais: Sorta, typically we recommend process monitoring via supervisors.
16:42:36 <SuperQ> runit, systemd, docker.
16:42:37 <SuperQ> etc.
16:42:46 <darkk> hellais: why?
16:42:50 <hellais> hum, is there a docker exporter?
16:43:04 <SuperQ> Docker has some built-in metrics
16:43:14 <SuperQ> Ahh, crap, brb, dinner is ready.
16:43:20 <SuperQ> I'll be back in a bit
16:44:17 <hellais> darkk: because there are some we may want to monitor via something other than HTTP (I am thinking of the tcp-echo, though I guess that can also be done via the blackbox_exporter)
16:44:18 <darkk> hellais: process being launched says nothing (SIGSTOP!), open TCP port says almost nothing (okay, kernel is alive, but userspace may be dead and/or SIGSTOPped)
16:44:46 <darkk> hellais: yep, we should do a chat with echo server and BBexp support that)
16:45:28 <hellais> yeah, now that I think of it I guess we can actually get a lot of mileage just out of the blackbox_exporter and some carefully crafter rules
16:45:31 <hellais> *crafted
16:45:51 <darkk> yep, and cron jobs should report their liveness to pushgateway
16:46:22 <hellais> I think I will have to spend more time looking into this, before I have smart questions to ask
16:48:26 <hellais> do we have more to discuss in this meeting?
16:52:20 <anadahz> Not from my side.
16:53:38 <hellais> I guess we can end this gathering 6mXs early?
16:53:38 <anadahz> let's end this meeting some minutes before the official meeting duration (1h+), noted in history ;)
16:54:01 <anadahz> Thanks everyone for attending!
16:54:06 <anadahz> #endmeeting