16:59:53 #startmeeting 16:59:53 Meeting started Mon Apr 25 16:59:53 2016 UTC. The chair is hellais. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:59:53 Useful Commands: #action #agreed #help #info #idea #link #topic. 17:00:23 so let's start this 17:00:24 who is here? 17:00:29 here 17:00:32 * sbs is here 17:02:41 * anadahz hello 17:04:17 ok so what have you been up to last week? 17:04:48 i kicked my "https endpoint for collectors and bouncers" branch forward a bit 17:05:25 they work on top of HS and TCP endpoints now, but something's wrong with the TLS endpoint + ooni combo 17:05:37 I ported the HTTP Invalid Request Line test to use the TCP Test template instead of the HTTP one (i.e. it now fully supports what ooniprobe tests for): https://github.com/measurement-kit/measurement-kit/pull/507 17:05:38 /rage EOF 17:05:57 Added support for SSL cert validation in MK: https://github.com/measurement-kit/measurement-kit/pull/512 17:06:17 hi 17:06:21 Made a bit of progress on the web-connectivity test 17:06:57 Created a branch for release 1.4.0 and backported patches into it and did a bit of cleaning up in preparation for the release: https://github.com/TheTorProject/ooni-probe/pull/481 17:07:16 I am going to say we should aim for releasing on Wednesday 17:07:24 /EOF 17:08:12 awesome! 17:11:06 I was mainly working on a paper but still managed to review and merge sone MK enhancements, plus I did more work - not sure if fully committed on github - to sketch out the NDT test -- EOF 17:11:27 Continued work on sys admin tasks, found a long-term solution on where to host the increasing number of OONI reports gathered; we now officially a petabyte scale VM! Tested ooni-probe web connectivity and started testing the current version of ooni-backend --EOF 17:15:52 anadahz: wow! what is the solution to scale up? 17:17:20 sbs: Greenhost offered OONI a Petabyte scale VM 17:18:02 starting with 2T and increase on demand 17:19:58 to be clear there will still need to be quite a bit of changes to the pipeline to actually manage to scale up to that size. 17:21:33 hellais: indeed but we have sorted out a quite big issue already; disk space! 17:21:45 I wanted to bring up the question of perhaps changing a bit the structure of this meeting as I think that just doing report backs and next steps is an activity that can probably be better done asynchronously via email. 17:23:05 More specifically I would suggest that instead we use this space as an opportunity to discuss specific problems we have encountered that we want help with, decisions that have to be made or simply to discuss some relevant topic with who is interested. 17:24:10 To do so we should agree on some agenda on the mailing list before the meeting and solicit feedback for items to put on the agenda. 17:25:02 anadahz: awesome! 17:25:49 For report backs and next steps: I would say every Monday we send them to ooni-dev prefixing them with [Status-Updates]. They don't have to be something too long or involved and can be just the 1 liners we post here. 17:25:52 Thoughts? 17:26:28 hellais: uhm, not sure about the proposal change: I think there is some value in being all together in this channel at the same place, even if, depending on what is discussed, the amount of individual involvement may of course vary 17:28:25 sbs & hellais: perhaps a middle ground would be to post updates and next steps on ooni-dev, and to further discuss specific "issues" on the monday meetings when/if necessary? 17:29:45 specifically, we can do report backs via mailing list, and if someone would like to discuss something pertaining to their work in more depth during Monday meetings, they can do so - while generally maintaining that Monday meetings are more "topic-specific" 17:32:58 yeah, to be clear I wasn't suggesting we remove Monday meetings entirely, but that we do the report backs outside of meeting time. 17:33:09 By having this weekly gathering people are always allowed to pop up and bring their on discussions something that is/has hapeened quite some times in the past. I don't think that we need to change the form of the weekly gathering but rather lax on the form, people can report back if they like and in between have also related/unrelated discussions 17:34:49 so, to better understand, the thing that you guys find disfunctional is doing the report back during the meeting because it's something that could also be done the async way, correct? 17:35:16 sbs: exactly. 17:35:39 sbs: yep 17:36:08 but, do you also think that doing that during meetings is also reducing the overall value of the meeting and the interaction therein in some way, or is efficency and async the main reason? 17:36:32 sbs: possibly both 17:37:46 sbs: I guess it depends. There can be a lot of value from discussing specific tasks and deliverables from the previous and upcoming week, but I think we would be optimizing for time if those were stated in the mailing list and then specific concerns were addressed in the monday meeting, along with other topics 17:38:49 agrabeli: speaking of efficiency, I'm concerned that by optimizing the report back time we're also reduce presence here in meetings as a byproduct 17:39:18 sbs: yes, you're right, that is probably a risk 17:39:26 to better qualify what I mean: I'm more likely to listen to what happens here in the meeting and ask questions than if I'm receiving emails from the mailing list 17:40:29 sbs: right. though what I was thinking was that we could utilize this time here to discuss, for example, how to solve problems pertaining to the pipeline, or other topics which could be equally - if not more - interesting to the ooni team and others in general 17:42:26 Changing the form of the IRC gathering will make many people to not be around.. 17:45:05 agrabeli: I understand and share the aim of making discussion more useful, and sometimes I had the feeling the meeting could be improved and too much time was spent reporting back and stating next goals 17:45:44 how about doing a quick phase at the beginning of report backs and next steps and then more time afterwards to discuss? 17:47:37 Yes, I see the point towards not removing the report backs. I guess the main issue is that I feel like we are not spending enough time actually in the core of the meeting to talk about next steps and about things are not just what we have been doing, but that could be of interest also to people that are joining ooni for the first time 17:48:20 to improve that I would suggest we try to at least suggest an agenda for next weeks meeting and see how that goes 17:49:12 sbs: yes I think already doing report backs + next steps at the same time could increase the quality of this to some degree 17:51:52 speaking of things that we could probably discuss today - and trying to go a little beyond the mechanical just-report-back - I would be interested to know more about https deployment and what needs to be done to secure 1.4.0 17:52:53 (another thing I'm wondering: how do other projects do this thing of meetings and reports back?) 17:54:00 sbs: so regarding HTTPS I guess landers can tell you more about what is being done there. From what I have seen a bit of progress has been done and we shouldn't be too far away from having it. 17:54:33 ah didn't realize 1.4.0 was waiting on it. i can give it more attention 17:54:52 landers: no, 1.4.0 is not dependent on this being integrated. 17:54:58 landers: is it? I did not meant it actually 17:55:16 never mind; misunderstood a few lines up 17:55:40 my question on 1.4.0 was more on what kind of testing do we need to perform, what kind of help is needed, etc to declare the release ready 17:57:11 I see that release/1.4.0 is a branch based onto master, so basically it's a set of final fixes to be pulled in before master is feature ready for 1.4.0? 17:57:45 landers: ah, sorry: "secure 1.4.0" where secure was to mean "finalize" rather than "http-ize" :-P 17:57:58 s/http/https/ 17:58:02 sbs: ah ok, in that case I think the most useful things to test for are ensuring that it works properly in terms of runtime error bugs, but mainly that the generated reports are correct and conformant to the new data format 17:59:26 I have done quite a bit of end to end testing to make sure that the various combinations work properly (ooniprobe 1.4.0 - old oonibackend with yaml), (ooniprobe 1.4.0 - new oonibackend with json), oonireport upload, oonideckgen, ooniresources 17:59:52 at this stage the backend deployed is synced with oonib master? 17:59:57 though there is probably no such thing as too much (or even enough for that matter) testing 18:00:07 sbs: no the deployed backend is still the old one 18:00:19 anadahz: when do you think you will manage to get the latest backend deployed? 18:00:41 alright, so this means we need to deploy the new backend first, right? 18:01:30 sbs: there is a testing one running here: httpo://eeevbta2hvnfwkxk.onion 18:01:33 hellais: have you used the test backend that I setup? 18:01:43 anadahz: yes I have, but the configuration is a bit ghetto 18:02:07 hellais: what cahnges do you need? 18:02:08 anadahz: like it running under screen inside a docker container and the submitted reports are not mapped to the host machine 18:02:26 anadahz: also it only runs the collector and not the bouncer 18:02:55 hellais:hm.. so we need to fix before deploying it 18:03:53 hellais: you can add a volume to map the docker with the host filesystem 18:05:23 hellais: I can create a VM that could be used to test ooni-backend, though it will be slower to update/deploy 18:06:04 anadahz: yes the volume mapping is what is done with the currently running docker container. 18:06:35 I think that we should fix the tools so that ooni* applications should be tested automagically 18:07:07 I was under the impression that developers have such tools while they code :P 18:07:12 anadahz: what do you mean by fixing the tools? 18:07:22 anadahz: what tools? 18:07:47 So please OONI developers share your test/deploy tools, recipes, ugly scripts, paper notes :) 18:08:21 hellais: like when you build the ooni-probe connectivity test what do you use to test it? 18:08:32 anadahz: if you are referring to unittests, yes we do have also end to end unittests for oonibackend and they are run by travis on commit. Though this is not a replacement for manual end to end testing. 18:09:24 I was not referring only to testing but how to run deploy the ooni-* 18:10:26 anadahz: The first testing I do is by running it inside of a vagrant virtual machine. The code for that is commited inside of ooniprobe and oonibackend 18:10:31 anadahz: when testing, I personally rely a lot on Vagrant boxes 18:11:20 hellais: sbs nice problem sovled then if I made a vagrant server we could be able to use these file to test everythong? 18:12:06 I'm asking because I spent many hours figuring how to better deploy ooni-prone and ooni-backend 18:12:07 anadahz: I personally don't see the point of this since the benefit of using vagrant is that it's running locally on your computer and you don't have to ssh into something remote 18:12:32 hellais: yes but then for testers this is a PITA 18:12:36 anadahz: tbh since docker has been creating so many issues I think we should consider just deploying it without it. 18:13:03 anadahz: also let's not conflate the issue of local testing, with the issue of deployment lifecycle 18:13:05 hellais: ok but everything as a cost, which in my case is time 18:13:30 anadahz: no, of course that's not problem solved, I agree with you that testing is not so simple, perhaps we can add some more script to automate some tasks once one is inside the vagrant box 18:14:14 anadahz: for MK I've started adding a directory of such stuff useful for when you're inside Vagrant (https://github.com/measurement-kit/measurement-kit/tree/master/build/vagrant) and perhaps we could do the same for ooni-probe... 18:14:34 anadahz: I am thinking, for example, to a script that creates for you a deck and runs the deck 18:15:32 sbs: yes that is helpful 18:16:02 for reference this is the procedure I follow when doing a release to ensure everything is working: https://github.com/TheTorProject/ooni-spec/blob/master/Release-Procedure.md 18:16:23 hellais anadahz: re docker I don't fully understand the terms of the discussion... so the problem is that the real oonib runs on docker? 18:16:24 The Vagrantfile for ooni-probe hasn't been updated since Nov 3, 2015 18:16:41 anadahz: I think I have a diff (or an open PR) to update it, wait 18:17:26 sbs: would this script be different than oonideckgen && ooniprobe -i deck-it/0.1.0-it-user.deck? 18:17:50 anadahz: what issues do you have with the Vagrantfile for ooniprobe? 18:18:09 I have been using it extensively in the past week for testing ooniprobe and it works well 18:18:16 anadahz: uhm, no actually I worked with oonib's Vagrant file, disregard 18:18:35 sbs: yeah your changes to the one of oonibackend have been merged. 18:18:56 hellais: I don't use Vagrant, but I'm sure that many things changed since 10/2015 18:19:07 hellais: yeah yeah, sorry, I probably had some dangling pointer to deleted branches inside my branin :-) 18:19:25 hellais: which you 've probably solved in a test repo of yours already 18:19:39 anadahz: nope, it's been working fine since then. 18:20:22 hellais: for instance it's running Ubuntu precise 18:20:53 anadahz: so? 18:21:18 hellais: regarding the release procedure, thanks for sharing it! will follow it when testing and eventually PR if needed 18:22:22 anadahz: precise is the oldest still not EOL ubuntu release. If it works on that then it should work on newer versions. 18:22:48 anadahz hellais: in any case it's possible to create Vagrant files targeting multiple distros 18:24:41 hellais: It's hard to find a VM that currently support this Ubuntu version and running Vagrant instances in production is not the best case 18:25:22 hellais: so we should perhaps do the testing in relevant OS target that we will most probably host OONI infrastructure 18:26:10 anadahz: yeah sure we can do that, though I don't think there is going to be any significant difference between distros. 18:26:21 anadahz: my point though is that we are mixing two topics: 18:26:42 1. The topic of how do I do testing to make a release of my software 18:27:06 2. Once a new release of the software I am interested in deploying is out, how do I roll it out 18:27:17 I don't think these are necessarily inter-connected 18:27:45 1. is about doing unittesting, automated end to end testing and manual testing 18:28:23 anadahz: what would be a relevant target distribution? 18:28:28 hellais: there are always changes between distros even between different version of distros, package names is one of them. 18:28:28 2. Is about putting the released software into a staging environment, doing a bit of manual testing on the staging deployed instance and then have a process for making the staging become production. 18:29:13 anadahz: true, but that is not something that you are testing when doing 1. 18:29:45 1. is only about testing the software, to ensure that the code is working properly. It's not about testing the deployment method for it. 18:30:30 anadahz: that said if you tell me a distro version you would like to use as the default of the Vagrant machines even when testing the code I can do that and test them 18:30:42 though vagrant is not a system that is meant to be used for production systems 18:30:50 it's just for local testing 18:32:47 sbs: multiple distros as you mentioned will be good! 18:33:29 anadahz: ok, cool! What would be the most must-have distro in the Vagrant file? 18:34:09 hellais: obviously you need to test if a software is running in multiple OSes 18:34:41 hellais: the initial discussion was to avoid duplication of work 18:35:13 so since the people that create the software have a recipe of testing we could use this one to test it 18:36:38 if it's Vagrnat, docker, or a custom hacked script that run in an unpopular software it's up to you 18:36:57 sbs: Debian stable 18:37:48 sometimes even finding out what is the correct order of installing pip packages is a pain 18:38:15 ^ and this is a ridicilous small portion of the testing/deployment 18:39:59 anadahz: okay 18:40:19 anadahz: ok I will update the vagrant file to use debian jessie. 18:41:09 So from what I understand If I deploy somewhere Vagrant I can test ooni-* MK without issues? 18:42:28 anadahz: yeah you should, but vagrant makes sense only if you run it on your local machine. If you are going to run it on a dedicated VPS you should probably just copy the deploy script from vagrant and use that 18:43:52 and the machines in production instead use docker? 18:44:03 hellais: OK would this: https://github.com/TheTorProject/ooni-backend/blob/master/Vagrantfile be sufficient to test ooni-backend? 18:44:13 sbs: yeah the machines in production are currently docker based. 18:44:38 though we actually are currently running only 1 docker container on each machine, which sort of defies the purpose of having docker 18:45:09 anadahz: yeah, tested this myself when merging the JSON pull request 18:45:31 anadahz: just follow the instructions printed at the end and everything should be alright 18:46:00 hellais: there is in any case probably value in using docker meaning that you can move things around and that you can setup your own oonib more easily 18:46:41 I was wondering what are the issues caused by docker and whether it would make sense to have a Dockerfile in tree 18:48:04 sbs: with 'follow the instructions printed at the end' you meant run the $setup_script (line 113)? 18:48:11 sbs: we do have dockerfiles for oonib and ooniprobe in the ooni-sysadmin repository, but we haven't really streamlined the deployment process. 18:49:05 sbs: testing and debugging why a deployed docker image doesn't work is a PITA as you can't interact directly with the OS running the app 18:49:26 sbs: the oonib.conf is missing, it uses the sample oonib configuration instead 18:49:31 anadahz: https://github.com/TheTorProject/ooni-backend/blob/master/Vagrantfile#L156 <-- here 18:49:50 it may also be that I am doing it wrong and don't understand the docker paradigm very well 18:50:09 sbs: so I need to manually set the oonib config to test it? 18:50:45 sbs: do you provide these values eveytime, or have a repo where you copy them when you are testing it? 18:50:48 anadahz: if you run it with the example oonib configuration file it will be working 18:50:58 hellais: ah, yes, I understand... actually I have little to zero expertise on docker debugging so I don't know if there is a better way of doing that 18:51:04 anadahz: the purpose of the example is to have sane defaults 18:51:52 anadahz: I used the default file and only edited the policy entry to be null for a reason that currently is not totally clear to me 18:52:16 hellais: do you remember (or could guess) why I needed to edit the policy entry? 18:53:16 sbs: ah yes, that is true. By default oonib will only accept URLs that are in the alexa top 1k list. By setting the policy file to null you are basically saying allow everything 18:54:24 [13ooni-backend] 15hellais created 06release/1.2.0 (+2 new commits): 02https://git.io/vwztM 18:54:24 13ooni-backend/06release/1.2.0 144c279e8 15Arturo Filastò: Convert msg to debug message to avoid generating too much noise 18:54:24 13ooni-backend/06release/1.2.0 14c1f2125 15Arturo Filastò: Bump to version 1.2.0 18:54:59 [13ooni-backend] 15hellais opened pull request #70: Release/1.2.0 (06master...06release/1.2.0) 02https://git.io/vwztb 18:55:09 hellais: regarding debugging docker, have you heard of `nsenter`? 18:55:41 https://github.com/jpetazzo/nsenter 18:55:53 or, as the readme of nsenter suggests, using docker exec to have a shell? 18:57:04 sbs: yeah I remember looking at that a while ago, but the kernel I was running docker on had issues with it. 18:57:28 sbs: it seems like though now docker has native support for this sort of behavior. 18:57:56 hellais: yes, even though I don't fully understand the whole thing :-) 18:58:36 anyways I guess this meeting has now gone like 1 hour overtime so I guess we should cut this short 18:59:20 no objections: I need dinner! 18:59:25 some solution for deploying the new oonibackend needs to be found. I personally don't have a strong preference for either a docker based thing or a VPS based solution. The important thing is that we have some documented process of doing these upgrades 18:59:51 and possibly that there is some sort of staging->production rollover happening 19:00:20 anadahz: do you think we can get this done this week? 19:01:14 as for the next steps (very quickly): I shall review and merge open PRs and continue with NDT 19:01:26 hellais: yes 19:01:59 Running the default oonib config results to: Invalid report directory: data/reports/! 19:02:38 anadahz: are you running it from inside the directory where the config file is in? 19:02:48 hellais: but please let me know what configuration do you need to test it 19:03:20 hellais: https://github.com/TheTorProject/ooni-backend/blob/master/Vagrantfile#L154 19:03:21 anadahz: the same configuration that is on the production machine, except with a different HS address 19:03:58 anadahz: if you are inside of the /data/oonib directory then you should not have any issues running it 19:04:10 do you have the data/ directory in there? 19:04:41 anyways let's talk about this outside of meeting time 19:04:44 #endmeeting