16:59:53 <hellais> #startmeeting
16:59:53 <MeetBot> Meeting started Mon Apr 25 16:59:53 2016 UTC.  The chair is hellais. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:59:53 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
17:00:23 <hellais> so let's start this
17:00:24 <hellais> who is here?
17:00:29 <landers> here
17:00:32 * sbs is here
17:02:41 * anadahz hello
17:04:17 <hellais> ok so what have you been up to last week?
17:04:48 <landers> i kicked my "https endpoint for collectors and bouncers" branch forward a bit
17:05:25 <landers> they work on top of HS and TCP endpoints now, but something's wrong with the TLS endpoint + ooni combo
17:05:37 <hellais> I ported the HTTP Invalid Request Line test to use the TCP Test template instead of the HTTP one (i.e. it now fully supports what ooniprobe tests for): https://github.com/measurement-kit/measurement-kit/pull/507
17:05:38 <landers> /rage EOF
17:05:57 <hellais> Added support for SSL cert validation in MK: https://github.com/measurement-kit/measurement-kit/pull/512
17:06:17 <agrabeli> hi
17:06:21 <hellais> Made a bit of progress on the web-connectivity test
17:06:57 <hellais> Created a branch for release 1.4.0 and backported patches into it and did a bit of cleaning up in preparation for the release: https://github.com/TheTorProject/ooni-probe/pull/481
17:07:16 <hellais> I am going to say we should aim for releasing on Wednesday
17:07:24 <hellais> /EOF
17:08:12 <willscott> awesome!
17:11:06 <sbs> I was mainly working on a paper but still managed to review and merge sone MK enhancements, plus I did more work - not sure if fully committed on github - to sketch out the NDT test -- EOF
17:11:27 <anadahz> Continued work on sys admin tasks, found a long-term solution on where to host the increasing number of OONI reports gathered; we now officially a petabyte scale VM! Tested ooni-probe web connectivity and started testing the current version of ooni-backend --EOF
17:15:52 <sbs> anadahz: wow! what is the solution to scale up?
17:17:20 <anadahz> sbs: Greenhost offered OONI a Petabyte scale VM
17:18:02 <anadahz> starting with 2T and increase on demand
17:19:58 <hellais> to be clear there will still need to be quite a bit of changes to the pipeline to actually manage to scale up to that size.
17:21:33 <anadahz> hellais: indeed but we have sorted out a quite big issue already; disk space!
17:21:45 <hellais> I wanted to bring up the question of perhaps changing a bit the structure of this meeting as I think that just doing report backs and next steps is an activity that can probably be better done asynchronously via email.
17:23:05 <hellais> More specifically I would suggest that instead we use this space as an opportunity to discuss specific problems we have encountered that we want help with, decisions that have to be made or simply to discuss some relevant topic with who is interested.
17:24:10 <hellais> To do so we should agree on some agenda on the mailing list before the meeting and solicit feedback for items to put on the agenda.
17:25:02 <sbs> anadahz: awesome!
17:25:49 <hellais> For report backs and next steps: I would say every Monday we send them to ooni-dev prefixing them with [Status-Updates]. They don't have to be something too long or involved and can be just the 1 liners we post here.
17:25:52 <hellais> Thoughts?
17:26:28 <sbs> hellais: uhm, not sure about the proposal change: I think there is some value in being all together in this channel at the same place, even if, depending on what is discussed, the amount of individual involvement may of course vary
17:28:25 <agrabeli> sbs & hellais: perhaps a middle ground would be to post updates and next steps on ooni-dev, and to further discuss specific "issues" on the monday meetings when/if necessary?
17:29:45 <agrabeli> specifically, we can do report backs via mailing list, and if someone would like to discuss something pertaining to their work in more depth during Monday meetings, they can do so - while generally maintaining that Monday meetings are more "topic-specific"
17:32:58 <hellais> yeah, to be clear I wasn't suggesting we remove Monday meetings entirely, but that we do the report backs outside of meeting time.
17:33:09 <anadahz> By having this weekly gathering people are always allowed to pop up and bring their on discussions something that is/has hapeened quite some times in the past. I don't think that we need to change the form of the weekly gathering but rather lax on the form, people can report back if they like and in between have also related/unrelated discussions
17:34:49 <sbs> so, to better understand, the thing that you guys find disfunctional is doing the report back during the meeting because it's something that could also be done the async way, correct?
17:35:16 <hellais> sbs: exactly.
17:35:39 <agrabeli> sbs: yep
17:36:08 <sbs> but, do you also think that doing that during meetings is also reducing the overall value of the meeting and the interaction therein in some way, or is efficency and async the main reason?
17:36:32 <agrabeli> sbs: possibly both
17:37:46 <agrabeli> sbs: I guess it depends. There can be a lot of value from discussing specific tasks and deliverables from the previous and upcoming week, but I think we would be optimizing for time if those were stated in the mailing list and then specific concerns were addressed in the monday meeting, along with other topics
17:38:49 <sbs> agrabeli: speaking of efficiency, I'm concerned that by optimizing the report back time we're also reduce presence here in meetings as a byproduct
17:39:18 <agrabeli> sbs: yes, you're right, that is probably a risk
17:39:26 <sbs> to better qualify what I mean: I'm more likely to listen to what happens here in the meeting and ask questions than if I'm receiving emails from the mailing list
17:40:29 <agrabeli> sbs: right. though what I was thinking was that we could utilize this time here to discuss, for example, how to solve problems pertaining to the pipeline, or other topics which could be equally - if not more - interesting to the ooni team and others in general
17:42:26 <anadahz> Changing the form of the IRC gathering will make many people to not be around..
17:45:05 <sbs> agrabeli: I understand and share the aim of making discussion more useful, and sometimes I had the feeling the meeting could be improved and too much time was spent reporting back and stating next goals
17:45:44 <sbs> how about doing a quick phase at the beginning of report backs and next steps and then more time afterwards to discuss?
17:47:37 <hellais> Yes, I see the point towards not removing the report backs. I guess the main issue is that I feel like we are not spending enough time actually in the core of the meeting to talk about next steps and about things are not just what we have been doing, but that could be of interest also to people that are joining ooni for the first time
17:48:20 <hellais> to improve that I would suggest we try to at least suggest an agenda for next weeks meeting and see how that goes
17:49:12 <hellais> sbs: yes I think already doing report backs + next steps at the same time could increase the quality of this to some degree
17:51:52 <sbs> speaking of things that we could probably discuss today - and trying to go a little beyond the mechanical just-report-back - I would be interested to know more about https deployment and what needs to be done to secure 1.4.0
17:52:53 <sbs> (another thing I'm wondering: how do other projects do this thing of meetings and reports back?)
17:54:00 <hellais> sbs: so regarding HTTPS I guess landers can tell you more about what is being done there. From what I have seen a bit of progress has been done and we shouldn't be too far away from having it.
17:54:33 <landers> ah didn't realize 1.4.0 was waiting on it. i can give it more attention
17:54:52 <hellais> landers: no, 1.4.0 is not dependent on this being integrated.
17:54:58 <sbs> landers: is it? I did not meant it actually
17:55:16 <landers> never mind; misunderstood a few lines up
17:55:40 <sbs> my question on 1.4.0 was more on what kind of testing do we need to perform, what kind of help is needed, etc to declare the release ready
17:57:11 <sbs> I see that release/1.4.0 is a branch based onto master, so basically it's a set of final fixes to be pulled in before master is feature ready for 1.4.0?
17:57:45 <sbs> landers: ah, sorry: "secure 1.4.0" where secure was to mean "finalize" rather than "http-ize" :-P
17:57:58 <sbs> s/http/https/
17:58:02 <hellais> sbs: ah ok, in that case I think the most useful things to test for are ensuring that it works properly in terms of runtime error bugs, but mainly that the generated reports are correct and conformant to the new data format
17:59:26 <hellais> I have done quite a bit of end to end testing to make sure that the various combinations work properly (ooniprobe 1.4.0 - old oonibackend with yaml), (ooniprobe 1.4.0 - new oonibackend with json), oonireport upload, oonideckgen, ooniresources
17:59:52 <sbs> at this stage the backend deployed is synced with oonib master?
17:59:57 <hellais> though there is probably no such thing as too much (or even enough for that matter) testing
18:00:07 <hellais> sbs: no the deployed backend is still the old one
18:00:19 <hellais> anadahz: when do you think you will manage to get the latest backend deployed?
18:00:41 <sbs> alright, so this means we need to deploy the new backend first, right?
18:01:30 <hellais> sbs: there is a testing one running here: httpo://eeevbta2hvnfwkxk.onion
18:01:33 <anadahz> hellais: have you used the test backend that I setup?
18:01:43 <hellais> anadahz: yes I have, but the configuration is a bit ghetto
18:02:07 <anadahz> hellais: what cahnges do you need?
18:02:08 <hellais> anadahz: like it running under screen inside a docker container and the submitted reports are not mapped to the host machine
18:02:26 <hellais> anadahz: also it only runs the collector and not the bouncer
18:02:55 <anadahz> hellais:hm.. so we need to fix before deploying it
18:03:53 <anadahz> hellais: you can add a volume to map the docker with the host filesystem
18:05:23 <anadahz> hellais: I can create a VM that could be used to test ooni-backend, though it will be slower to update/deploy
18:06:04 <hellais> anadahz: yes the volume mapping is what is done with the currently running docker container.
18:06:35 <anadahz> I think that we should fix the tools so that ooni* applications should be tested automagically
18:07:07 <anadahz> I was under the impression that developers have such tools while they code :P
18:07:12 <sbs> anadahz: what do you mean by fixing the tools?
18:07:22 <hellais> anadahz: what tools?
18:07:47 <anadahz> So please OONI developers share your test/deploy tools, recipes, ugly scripts, paper notes :)
18:08:21 <anadahz> hellais: like when you build the ooni-probe connectivity test what do you use to test it?
18:08:32 <hellais> anadahz: if you are referring to unittests, yes we do have also end to end unittests for oonibackend and they are run by travis on commit. Though this is not a replacement for manual end to end testing.
18:09:24 <anadahz> I was not referring only to testing but how to run deploy the ooni-*
18:10:26 <hellais> anadahz: The first testing I do is by running it inside of a vagrant virtual machine. The code for that is commited inside of ooniprobe and oonibackend
18:10:31 <sbs> anadahz: when testing, I personally rely a lot on Vagrant boxes
18:11:20 <anadahz> hellais: sbs nice problem sovled then if I made a vagrant server we could be able to use these file to test everythong?
18:12:06 <anadahz> I'm asking because I spent many hours figuring how to better deploy ooni-prone and ooni-backend
18:12:07 <hellais> anadahz: I personally don't see the point of this since the benefit of using vagrant is that it's running locally on your computer and you don't have to ssh into something remote
18:12:32 <anadahz> hellais: yes but then for testers this is a PITA
18:12:36 <hellais> anadahz: tbh since docker has been creating so many issues I think we should consider just deploying it without it.
18:13:03 <hellais> anadahz: also let's not conflate the issue of local testing, with the issue of deployment lifecycle
18:13:05 <anadahz> hellais: ok but everything as a cost, which in my case is time
18:13:30 <sbs> anadahz: no, of course that's not problem solved, I agree with you that testing is not so simple, perhaps we can add some more script to automate some tasks once one is inside the vagrant box
18:14:14 <sbs> anadahz: for MK I've started adding a directory of such stuff useful for when you're inside Vagrant (https://github.com/measurement-kit/measurement-kit/tree/master/build/vagrant) and perhaps we could do the same for ooni-probe...
18:14:34 <sbs> anadahz: I am thinking, for example, to a script that creates for you a deck and runs the deck
18:15:32 <anadahz> sbs: yes that is helpful
18:16:02 <hellais> for reference this is the procedure I follow when doing a release to ensure everything is working: https://github.com/TheTorProject/ooni-spec/blob/master/Release-Procedure.md
18:16:23 <sbs> hellais anadahz: re docker I don't fully understand the terms of the discussion... so the problem is that the real oonib runs on docker?
18:16:24 <anadahz> The Vagrantfile for ooni-probe hasn't been updated since Nov 3, 2015
18:16:41 <sbs> anadahz: I think I have a diff (or an open PR) to update it, wait
18:17:26 <hellais> sbs: would this script be different than oonideckgen && ooniprobe -i deck-it/0.1.0-it-user.deck?
18:17:50 <hellais> anadahz: what issues do you have with the Vagrantfile for ooniprobe?
18:18:09 <hellais> I have been using it extensively in the past week for testing ooniprobe and it works well
18:18:16 <sbs> anadahz: uhm, no actually I worked with oonib's Vagrant file, disregard
18:18:35 <hellais> sbs: yeah your changes to the one of oonibackend have been merged.
18:18:56 <anadahz> hellais: I don't use Vagrant, but I'm sure that many things changed since 10/2015
18:19:07 <sbs> hellais: yeah yeah, sorry, I probably had some dangling pointer to deleted branches inside my branin :-)
18:19:25 <anadahz> hellais: which you 've probably solved in a test repo of yours already
18:19:39 <hellais> anadahz: nope, it's been working fine since then.
18:20:22 <anadahz> hellais: for instance it's running Ubuntu precise
18:20:53 <hellais> anadahz: so?
18:21:18 <sbs> hellais: regarding the release procedure, thanks for sharing it! will follow it when testing and eventually PR if needed
18:22:22 <hellais> anadahz: precise is the oldest still not EOL ubuntu release. If it works on that then it should work on newer versions.
18:22:48 <sbs> anadahz hellais: in any case it's possible to create Vagrant files targeting multiple distros
18:24:41 <anadahz> hellais: It's hard to find a VM that currently support this Ubuntu version and running Vagrant instances in production is not the best case
18:25:22 <anadahz> hellais: so we should perhaps do the testing in relevant OS target that we will most probably host OONI infrastructure
18:26:10 <hellais> anadahz: yeah sure we can do that, though I don't think there is going to be any significant difference between distros.
18:26:21 <hellais> anadahz: my point though is that we are mixing two topics:
18:26:42 <hellais> 1. The topic of how do I do testing to make a release of my software
18:27:06 <hellais> 2. Once a new release of the software I am interested in deploying is out, how do I roll it out
18:27:17 <hellais> I don't think these are necessarily inter-connected
18:27:45 <hellais> 1. is about doing unittesting, automated end to end testing and manual testing
18:28:23 <sbs> anadahz: what would be a relevant target distribution?
18:28:28 <anadahz> hellais: there are always changes between distros even between different version of distros, package names is one of them.
18:28:28 <hellais> 2. Is about putting the released software into a staging environment, doing a bit of manual testing on the staging deployed instance and then have a process for making the staging become production.
18:29:13 <hellais> anadahz: true, but that is not something that you are testing when doing 1.
18:29:45 <hellais> 1. is only about testing the software, to ensure that the code is working properly. It's not about testing the deployment method for it.
18:30:30 <hellais> anadahz: that said if you tell me a distro version you would like to use as the default of the Vagrant machines even when testing the code I can do that and test them
18:30:42 <hellais> though vagrant is not a system that is meant to be used for production systems
18:30:50 <hellais> it's just for local testing
18:32:47 <anadahz> sbs: multiple distros as you mentioned will be good!
18:33:29 <sbs> anadahz: ok, cool! What would be the most must-have distro in the Vagrant file?
18:34:09 <anadahz> hellais: obviously you need to test if a software is running in multiple OSes
18:34:41 <anadahz> hellais: the initial discussion was to avoid duplication of work
18:35:13 <anadahz> so since the people that create the software have a recipe of testing we could use this one to test it
18:36:38 <anadahz> if it's Vagrnat, docker, or a custom hacked script that run in an unpopular software it's up to you
18:36:57 <anadahz> sbs: Debian stable
18:37:48 <anadahz> sometimes even finding out what is the correct order of installing pip packages is a pain
18:38:15 <anadahz> ^ and this is a ridicilous small portion of the testing/deployment
18:39:59 <sbs> anadahz: okay
18:40:19 <hellais> anadahz: ok I will update the vagrant file to use debian jessie.
18:41:09 <anadahz> So from what I understand If I deploy somewhere Vagrant I can test ooni-* MK without issues?
18:42:28 <hellais> anadahz: yeah you should, but vagrant makes sense only if you run it on your local machine. If you are going to run it on a dedicated VPS you should probably just copy the deploy script from vagrant and use that
18:43:52 <sbs> and the machines in production instead use docker?
18:44:03 <anadahz> hellais: OK would this: https://github.com/TheTorProject/ooni-backend/blob/master/Vagrantfile be sufficient to test ooni-backend?
18:44:13 <hellais> sbs: yeah the machines in production are currently docker based.
18:44:38 <hellais> though we actually are currently running only 1 docker container on each machine, which sort of defies the purpose of having docker
18:45:09 <sbs> anadahz: yeah, tested this myself when merging the JSON pull request
18:45:31 <sbs> anadahz: just follow the instructions printed at the end and everything should be alright
18:46:00 <sbs> hellais: there is in any case probably value in using docker meaning that you can move things around and that you can setup your own oonib more easily
18:46:41 <sbs> I was wondering what are the issues caused by docker and whether it would make sense to have a Dockerfile in tree
18:48:04 <anadahz> sbs: with 'follow the instructions printed at the end' you meant run the $setup_script (line 113)?
18:48:11 <hellais> sbs: we do have dockerfiles for oonib and ooniprobe in the ooni-sysadmin repository, but we haven't really streamlined the deployment process.
18:49:05 <hellais> sbs: testing and debugging why a deployed docker image doesn't work is a PITA as you can't interact directly with the OS running the app
18:49:26 <anadahz> sbs: the oonib.conf is missing, it uses the sample oonib configuration instead
18:49:31 <sbs> anadahz: https://github.com/TheTorProject/ooni-backend/blob/master/Vagrantfile#L156 <-- here
18:49:50 <hellais> it may also be that I am doing it wrong and don't understand the docker paradigm very well
18:50:09 <anadahz> sbs: so I need to manually set the oonib config to test it?
18:50:45 <anadahz> sbs: do you provide these values eveytime, or have a repo where you copy them when you are testing it?
18:50:48 <hellais> anadahz: if you run it with the example oonib configuration file it will be working
18:50:58 <sbs> hellais: ah, yes, I understand... actually I have little to zero expertise on docker debugging so I don't know if there is a better way of doing that
18:51:04 <hellais> anadahz: the purpose of the example is to have sane defaults
18:51:52 <sbs> anadahz: I used the default file and only edited the policy entry to be null for a reason that currently is not totally clear to me
18:52:16 <sbs> hellais: do you remember (or could guess) why I needed to edit the policy entry?
18:53:16 <hellais> sbs: ah yes, that is true. By default oonib will only accept URLs that are in the alexa top 1k list. By setting the policy file to null you are basically saying allow everything
18:54:24 <MightyOctopus> [13ooni-backend] 15hellais created 06release/1.2.0 (+2 new commits): 02https://git.io/vwztM
18:54:24 <MightyOctopus> 13ooni-backend/06release/1.2.0 144c279e8 15Arturo Filastò: Convert msg to debug message to avoid generating too much noise
18:54:24 <MightyOctopus> 13ooni-backend/06release/1.2.0 14c1f2125 15Arturo Filastò: Bump to version 1.2.0
18:54:59 <MightyOctopus> [13ooni-backend] 15hellais opened pull request #70: Release/1.2.0 (06master...06release/1.2.0) 02https://git.io/vwztb
18:55:09 <sbs> hellais: regarding debugging docker, have you heard of `nsenter`?
18:55:41 <sbs> https://github.com/jpetazzo/nsenter
18:55:53 <sbs> or, as the readme of nsenter suggests, using docker exec to have a shell?
18:57:04 <hellais> sbs: yeah I remember looking at that a while ago, but the kernel I was running docker on had issues with it.
18:57:28 <hellais> sbs: it seems like though now docker has native support for this sort of behavior.
18:57:56 <sbs> hellais: yes, even though I don't fully understand the whole thing :-)
18:58:36 <hellais> anyways I guess this meeting has now gone like 1 hour overtime so I guess we should cut this short
18:59:20 <sbs> no objections: I need dinner!
18:59:25 <hellais> some solution for deploying the new oonibackend needs to be found. I personally don't have a strong preference for either a docker based thing or a VPS based solution. The important thing is that we have some documented process of doing these upgrades
18:59:51 <hellais> and possibly that there is some sort of staging->production rollover happening
19:00:20 <hellais> anadahz: do you think we can get this done this week?
19:01:14 <sbs> as for the next steps (very quickly): I shall review and merge open PRs and continue with NDT
19:01:26 <anadahz> hellais: yes
19:01:59 <anadahz> Running the default oonib config results to: Invalid report directory: data/reports/!
19:02:38 <hellais> anadahz: are you running it from inside the directory where the config file is in?
19:02:48 <anadahz> hellais: but please let me know what configuration do you need to test it
19:03:20 <anadahz> hellais: https://github.com/TheTorProject/ooni-backend/blob/master/Vagrantfile#L154
19:03:21 <hellais> anadahz: the same configuration that is on the production machine, except with a different HS address
19:03:58 <hellais> anadahz: if you are inside of the /data/oonib directory then you should not have any issues running it
19:04:10 <hellais> do you have the data/ directory in there?
19:04:41 <hellais> anyways let's talk about this outside of meeting time
19:04:44 <hellais> #endmeeting