17:00:05 <hellais> #startmeeting OONI gathering 2016-08-15
17:00:05 <MeetBot> Meeting started Mon Aug 15 17:00:05 2016 UTC.  The chair is hellais. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:05 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
17:00:27 <landers> here
17:00:39 <anadahz> Hello everyone
17:00:59 <willscott> hi hi
17:01:20 <darkk> +1
17:01:44 <hellais> excellent
17:03:14 <anadahz> darkk: is going to be officially working for OONI :)
17:03:36 * darkk has not updated linkedin yet :)
17:04:58 <hellais> heheheh
17:05:07 <hellais> ok, so what do we have for the agenda this week?
17:06:12 <anadahz> it seems that we don't have new items but we can pick up one of the untouched agenda topics
17:07:10 <anadahz> (non processed due to time constraints)
17:08:02 <hellais> anadahz: sure sounds good, what untouched items do we have?
17:08:57 <anadahz> #topic Lepidopter Raspberry Pi image: Discuss if we should maintain managed/non managed releases for different set of users; technical savvy that are able to manage lepidopter by themselves, or users that trust us and will prefer automatic updates with remote management capabilities (if needed).
17:12:01 <darkk> I see no reason to maintain separate RPi _image_ for techies. I'd rather install/update Raspbian deb package from separate deb package repository (probably, with dependencies in the same repo). What's the idea behind image separation?
17:12:30 <hellais> ah right, so I think that lepidopter is in the end something that is mainly targetted towards non-technically savvy users and people that will just put this box in their house and more or less forget about it once they have first set it up. I think the overhead of having to maintain two separate images isn't really justified by the effort required to do so. Moreover I think that if you are a techie you will
17:12:37 <hellais> probably just install ooniprobe from the deb and not even need lepidopter.
17:12:50 <hellais> darkk: yes I agree
17:12:58 <anadahz> Currently the process of making an image require a number of resources and most important testing we have reached a point that specific groups (such as OONI partners) need different customized and often pre-configured releases.
17:14:02 <hellais> anadahz: yeah, but if you are a techie you will just install rasbian and apt-get install ooniprobe (the fact that ooniprobe isn't in rasbian or may not work from the debian repositories is separate to this). For those that require a specific configuration we are moving these configurations inside of the application logic itself so there is no need to have separate releases.
17:14:29 <anadahz> The idea of lepidopter (at least that how it's started) was to be able to stand as a distribution that users with minimal physical attendace can run and provide useful network (ooni-probe) measurements.
17:14:40 <darkk> What is pre-configured besides network settings and encryption keys?
17:14:50 <hellais> what we have actually seen is that people that are technical and want to do their own deployments end up rolling out their own images anyways, like venezuela inteligente
17:15:43 <anadahz> darkk: Maintaining a separate deb package repository requires also quite some maintainance
17:16:21 <hellais> darkk: the only "customizations" that we have done so far are related to adding a hidden service to the image and configuring it to exclude from running certain tests.
17:16:46 <hellais> anadahz: true, but it's something that we are doing anyways for debian and I bet there is way to in some way re-use the debian packages for rasbian as well
17:17:23 <hellais> also making packages is something that is going to be much simpler and less error prone than a full distribution I think
17:18:14 <anadahz> hellais: Currently there is no easy to install ooniprobe in a way that can act as a 24/7 probe with minimal attendace. One should need to figure things as log rotation, old reports cleanup and updates that AFAIK are not being by any package in any OS.
17:18:42 <darkk> logrotate script is usual part of conffiles
17:19:33 <hellais> anadahz: yeah, but that is a bug in the package and should be fixed in the package itself. It doesn't seem like something that should warrant us making a whole distro just to configure some scripts that should be part of the package itself anyways
17:20:08 <hellais> also most of these things will be handled natively by ooniprobe itself starting in version 2.0.0 so many of this will no longer be needed
17:20:30 <hellais> the only thing that would be missing is updates of the system and the software itself
17:20:54 <anadahz> darkk: This is the latest customization (annulet release): https://github.com/TheTorProject/lepidopter/tree/release/annulet and this is the ticket for another: https://github.com/TheTorProject/lepidopter/issues/64
17:20:55 <hellais> that however from what you stated above is the main distinguisher you are making between lepidopter for techies vs lepidopter for general users.
17:21:38 <anadahz> darkk: unfortunately the way ooniprobe writes logs cannot be handled by logrotate properly
17:22:38 <hellais> anadahz: I think the issues around logging are resolved inside of 2.0.0 and if they are not that should be implemented as a fix in ooniprobe itself.
17:23:05 <anadahz> hellais: yes everything is a bug/feature in the software that needs to be fixed/implemented. But specific cannot just happen in OS packages.
17:23:17 <darkk> anadahz: correct, I remember there was some pain in twisted+logrotate setup I had.
17:24:46 <anadahz> hellais: ooniprobe 2.0.0 could hopefully resolves some of the issues. but I guess people will still have different needs that we 'll not have implemented in ooniprobe-* itself.
17:25:52 <darkk> anadahz: is there anything stopping us from building the image as a set of packages and shipping configuration files as separate packages like ooni-annulet, ooni-lepidopter, ooni-whatever?
17:26:34 <darkk> is it harder than current way?
17:26:44 <anadahz> darkk: yes time :) Any new release produces a crazy amount of testing.
17:27:35 <anadahz> and if we continue to publicly offer these releases we need to maintain them along the time
17:28:46 <darkk> how is release published & maintained? to we update SD card image like any other debian system or is it full reflash for every update?
17:29:14 <darkk> (side question) is it ok to disclose bridges @ github? :)
17:30:50 <landers> i think no
17:31:08 <anadahz> darkk: Yes when a release is being made we build an image (https://github.com/TheTorProject/lepidopter/blob/master/lepidopter-vmdebootstrap_build.sh) and publish it online: https://get.ooni.torproject.org/lepidopter/
17:32:01 <hellais> darkk: yeah, it's not ok to disclose bridges on github if you are referring to the bridges in the release/annulet branch I believe those are all public bridges though (the ones included inside of Tor Browser).
17:35:32 <anadahz> hellais: what would be the preferred way of adding public bridges in a repository?
17:36:30 <darkk> anadahz: and how does user update unattended & 24/7-running RPi? Does the user reflash the lepidopter on every release? Is it done semi-manually (ansible?) via hidden service? Is it some cron-job pulling the latest updates?
17:36:50 <hellais> anadahz: sorry I missed an adjective in my above statement. What I meant to say way "it's not ok to disclose _private_ bridges on github" (for example the ones obtained from bridges.tpo)
17:37:24 <hellais> if they public it's fine, but if they are private then of course they should not be committed publicly otherwise the censor finds them and blocks them
17:39:33 <anadahz> darkk: currently only ooniprobe and decks are being updated, I'm in the process of figuring out a good solution for implementings over the air unattended updates in lepidopter: https://github.com/TheTorProject/lepidopter/pull/69
17:43:50 <hellais> note: if and when we implement OTA updates if we go for this route of having two separate images for techies and non techies we will have to maintain two update channels.
17:45:20 <darkk> IMHO, it it a design goal to have OTA update that is _more_ robust than `apt-get update && apt-get upgrade` or not? Should OTA survive power failure happening during the upgrade process?
17:45:27 <anadahz> I'm currently going through OSTree and SWUpdate as well as Yoctoproject that seems to have some support from the open embedded world and usually they implement OTA updates to their releases
17:45:32 <darkk> s/IMHO, it/Is/
17:48:08 <anadahz> hellais: If you change our build procedure to Yoctoproject and choose an OTA software with roles support this will ease a lot testing a distribution.
17:51:22 <anadahz> darkk: There are different OTA approaches; ideally we will use/customize/implement one that has fallback suport. Meaning that every time an update is perfomed the newly update "system" will do a check and if it's successfully boots it will keep this image as the default or fallback to the previous known working system image.
17:51:37 <anadahz> s/you/we
17:53:44 <anadahz> apt-get operations are good up to a certain point, and maintaining a deb repo requires some love.
17:56:38 <darkk> yep, I see. I thought of some trivial like having three partitions: root-1, root-1, /data and bootloader but there is something more advanced, :) The only open question is if it's hard to drop Raspbian base or not? E.g. do we have to reproduce some blob layout or not. But read-only root (say, yocto-style) will probably not satisfy techies wanting to have their own patched tcpdump with custom
17:56:40 <darkk> dissectors, so we may have to distribution channels, but I doubt that `image`-way is goo for techie channel.
17:56:51 <darkk> *two distribution channels...
17:57:20 * darkk is to used to `edit-after-send` behavior :(
17:57:31 <darkk> *too :-D
17:58:15 <anadahz> darkk: we don't use Raspbian, lepidopter is actually running Debian jessie
17:58:56 <darkk> cool! I thought it's harder to run raw debian @ RPi then simple debootstap.
17:59:17 <anadahz> Raspbian has different packages that tend to conflict with other Debian packages.
18:04:05 <anadahz> darkk: I'm learning how Yocto works but it seems that their principle is: provide a limited OS image with only the required packages and _nothing_ else. Even if you select deb_packages as the default you still have no apt-get/aptitude to the final image. :)
18:04:28 <darkk> yep, like openwrt without opkg :)
18:05:10 <darkk> it was a usual way to reduce image size as opkg-installed packages were not compressed to read/only squashfs
18:06:33 <anadahz> I will be happy of any suggestions in: https://github.com/TheTorProject/lepidopter/pull/69
18:09:06 <anadahz> It seems that we have exhausted all the available meeting time!
18:09:26 <hellais> ok since we are approaching the end (we actually have overcome it) do we think we have some resolution on this topic of having two distributions or not?
18:10:39 <hellais> looking at the stats it seems like most of the users of lepidopter are actually partners as such I think we should for sure give priority to the use case 1
18:11:00 <anadahz> From the techie-side I don't think many people will be happy knowing that their RasPi do filesystem-updates automagically.
18:11:27 <anadahz> hellais: Were do you derive your results from?
18:11:45 <hellais> (that is in the last 5 days measurements coming from lepidopter based installations where from 12 ASNs and of these 8 are partners)
18:11:46 <willscott> as an end user, i would prefer that it auto-patched/updated rather than that I’m expected to do maintanence on the thing
18:12:29 <hellais> anadahz: looking at measurements that are annoated with the platform: lepidopter key collected in the past 5 days.
18:12:50 <anadahz> hellais: yes it seems that we lost some devices..
18:13:29 <anadahz> hellais: look before the annotations recursion bug (and before the partnership program)
18:15:52 <darkk> I'd package ooni-probe without yocto-magic for techies (probably, with some default configuration that rotates logs & cleans stale files). IMHO, they do not need all the unattended upgrades complexity and users running gentoo on their PIs can write ebuilds basing on debian packages (we have to maintain for debian users, if any).
18:17:00 <darkk> 10k people are OK with their RIPE Atlas probe being updated over the air and having no shell at the box :)
18:17:33 <hellais> darkk: +1
18:19:08 <hodgepodge> I agree as well, as someone who just sort of casually runs a VPS, I'd rather have ooni-probe run as a daemon, and self-update
18:19:39 <hodgepodge> Since honestly, I'm going to forget about it. It'd also open up the possibility for things like remote instrumentation, and automating the use of various test decks to further investigate problem areas
18:20:26 <anadahz> darkk: the problem is that we do have SSH at the box
18:21:01 <darkk> anadahz: and plan to do probes orchestration to be able to submit tasks to probes in real-time
18:21:19 <darkk> anadahz: I don't get the problem :-(
18:22:08 <anadahz> darkk: https://github.com/TheTorProject/lepidopter/issues/57
18:22:49 <anadahz> ^ This cannot be implemented nicely with a debian repo
18:23:44 <anadahz> and ensuring that the OS will always work
18:25:15 <anadahz> given that the users can possibly (and will) change configuration in lepidopter
18:26:12 <darkk> anadahz: Agreed. I'd have these discussions split 1) unattended RPi-based system with almost(?) no user control 2) unattended VDS somewhere in the well-connected rack (x86/x86_64 box that has Debian deployed by hosting provider) 3) techie who wants to have all the ebuilds & watchdogs written himself. Do we have good user story behind (3)?
18:28:24 <anadahz> The solution discussed is not for the third group but for the second
18:28:37 <anadahz> There are different level of techie people..
18:29:52 <anadahz> Maybe I wasn't clear enough but I was perhaps considering to use Yocto to ease us with the OTA updates not to ease the techies
18:31:18 <anadahz> if case 2) is going to be solved by ooniprobe v2 then perhaps there is no need to talk further
18:32:26 <anadahz> But actually ooniprobe v2 will not be designed for headless servers (VDS, VM, VPS,).
18:33:26 <anadahz> So I'm not sure how a person can run ooniprobe in a headless server setup.
18:34:13 <darkk> I get your concern now. But I have nothing to add at this moment, maybe, next morning :)
18:34:28 <darkk> I really though that you were worried about case #3
18:35:47 <anadahz> so hodgepodge will not able to run ooniprobe v2 officially (without hacks)
18:36:34 <hodgepodge> I came in a little late to the meeting, but isn't the ability to run ooni-probe headless one of the big drivers of lepidopter since you could just plug in the device, and forget about it?
18:37:58 <anadahz> hodgepodge: this behaviour will change though in ooniprobe v2 as a user will need to read and agree with the informed consent. And this will happen in a web browser
18:38:20 <hellais> anadahz: there is also a cli version of the informed consent procedure
18:38:30 <hellais> a browser is not needed, but we do want to force users to go through this
18:38:56 <anadahz> hellais: check last's weeks log for a related discussion: http://meetbot.debian.net/ooni/2016/ooni.2016-08-08-17.00.log.html
18:39:01 <hodgepodge> Yeah, that's definitely important, especially if we're going to be incorporating the ability to perform remote test instrumentation
18:39:08 <hodgepodge> Since you know, that's like super botnetty
18:40:35 <anadahz> hellais: i'm confused!
18:42:10 <hellais> anadahz: why?
18:42:30 <hellais> if you run ooniprobe (not the agent) you get the same interactive informed consent procedure, but from a CLI
18:42:35 <hellais> it's not skippable though
18:42:47 <hellais> like we want to ensure people actualy read it and type yes in the terminal
18:42:47 <anadahz> hellais: last week you mentioned that will not be the indended way of running ooniprobe
18:44:35 <hellais> anadahz: well ooniprobe is for manually running tests ooniprobe-agent for running them periodically automatically
18:45:05 <hellais> ooniprobe stays as the CLI to the software so if you need to do things interactively with it that is the entry point
18:45:41 <hodgepodge> Will ooniprobe and ooniprobe-agent be bundled within the same package, hellais, anadahz?
18:45:57 <hellais> hodgepodge: yes.
18:46:04 <hodgepodge> Awesome, thanks!
18:46:05 <anadahz> so headless servers will still cronjobs for regularly running ooniprobe tests and updating the test decks?
18:46:21 <anadahz> so headless servers will still need cronjobs for regularly running ooniprobe tests and updating the test decks?
18:46:22 <hellais> anadahz: no. headless servers run ooniprobe-agent!
18:46:43 <hellais> but before running that the user will need to agree to the informed consent otherwise it will not do anything
18:47:08 <darkk> and it may have cronjobs from ancient package :)
18:47:09 <hellais> to agree to the informed consent you run ooniprobe and it gives you the text and has a first start wizard
18:47:56 <hellais> darkk: yeah, we need to come up with some solid strategy for migrating over to 2.0.0, because I expect there to be quite a bit of cleanup necessary for systems updating from 1.x to 2.x
18:47:57 <anadahz> hellais: so the wizard is going to be implemeted for terminals as well?
18:48:08 <hellais> anadahz: the wizard is implemented in the terminal as well
18:50:25 <hellais> it's basically the same as the wizard in the GUI except that it doesn't have the quiz questions. The reason why I omitted that is that if they get the answer wrong it's a bit more of a pain than in a web gui to go through it again.
18:51:19 <anadahz> anyway I was under the impression (from the last meeting http://meetbot.debian.net/ooni/2016/ooni.2016-08-08-17.00.log.html#l-39) that running ooniprobe in shell will not be officially supported
18:53:08 <hellais> anadahz: yeah, that is not the recommended way of running tests. If you are just a casual user and just want to contribute measurements you should start the system daemon. If you are a power user and want to develop your own tests or trigger some test manually and know what you are doing then you can do it with the CLI.
18:58:04 <darkk> so... do we have anything left for today besides wishing a good vacation to everyone going on vacation? :)
18:58:22 * darkk learns how to respect privacy online
18:59:36 <hellais> from my side I am satisfied, I think we can finish it here only 1 hour late :P
18:59:51 * hodgepodge feels like he has a lot to catch up on, you've all been talking for like, two hours
19:00:33 <hellais> hehehehe, well then
19:01:18 <hellais> I guess we can end this here, thank you all for attending the meeting and I suspect I will not manage to attend next weeks meeting and will be back online starting from the 24th evening
19:01:32 <hellais> see you around
19:01:36 <hellais> #endmeeting