16:59:32 <hellais> #startmeeting
16:59:32 <MeetBot> Meeting started Mon Dec 14 16:59:32 2015 UTC.  The chair is hellais. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:59:32 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:59:43 * gxg waves
16:59:49 <hellais> ok beautiful humans, you know the drill, who is here?
17:00:01 <anadahz> im here
17:00:05 <simonv3_> i’m here
17:00:10 <bnvk> hallo
17:01:07 <landers> here
17:01:55 <sbs> ehlo
17:02:20 <hellais> perfect, so what have you been up to last week?
17:03:25 <sbs> I've been working to prepare measurement-kit release
17:03:39 <sbs> not ready yet, but only few tasks missing:
17:04:01 <sbs> https://github.com/measurement-kit/measurement-kit/issues?q=is%3Aopen+is%3Aissue+milestone%3Arelease-0.1.0
17:04:10 <sbs> plus helped a bit with OONI proposal
17:04:11 <sbs> EOF
17:05:57 <hellais> I've worked on the OONI proposal, debugging and refactoring the data pipeline with joe to insert all the reports inside of the test_keys field and started adapting the ooni-api to the data model changes - EOF
17:06:30 <landers> been working on some sysadmin and pipeline tasks,
17:06:33 <landers> have a branch on each of those repos that i'm pushing lots to.
17:06:35 <landers> most relevant to other people would be the pipeline changes;
17:06:38 <landers> i'm taking the luigi stuff out of the
17:06:39 <landers> raw-reports -> sanitised -> postgres pipeline
17:06:41 <landers> EOF
17:07:52 <anadahz> from my side: finished the ansible roles for ooniprobe, third party tools and tor pluggagle transports
17:08:04 <anadahz> EOF
17:10:04 <simonv3> I’ve finished getting the front-end for ooni-api to match the paperframes, it’s waiting on a PR to be merged, using the data fixtures.
17:10:13 <simonv3> I also spent a little bit of time trying to connect to the staging server, but when I do so the client is stuck on a loading screen, so I have to chat to joe and hellias this week about that. EOF
17:11:55 <hellais> simonv3: yeah the datamodel has changed a bit and I think we should opt for implementing first a very basic MVP that doesn't take into account anomalies, but just allows browsing the reports
17:12:28 <hellais> I pushed to a fork of your branch the changes that should allow you to get started with the new data model: https://github.com/TheTorProject/ooni-api/pull/10
17:12:35 <bnvk> I just followed along with simonv3 progress and checked up on things. Glad to see it's all progressing along :)
17:12:38 <bnvk> EOF
17:14:48 <hellais> I am thinking that the order in which things should be implemented are: 1) Implement external pagination for measurement listing 2) Implement external sorting for the measurement listing 3) Make the viewer of the measurement/report something that can include a view depending on the type of test in question
17:15:29 <hellais> I am thinking that there could be something like an ng-switch or a chain of ng-if that include some templates that implement the views specific to every test and then on top there is the standard boilerplate view common to every report/measurement
17:15:39 <hellais> simonv3: does this make sense?
17:16:11 <simonv3> yeah that makes a ton of sense
17:16:24 <simonv3> it’d definitely be good to lay out / style the page based on the test
17:16:28 <hodgepodge> Do we need test-specific views, though?
17:16:58 <hellais> hodgepodge: yeah I think we do, since the sorts of information needed to be presented varies greatly from test to test.
17:17:09 <hodgepodge> That's true - good point. The PR looks great, by the way!
17:17:17 <simonv3> looking at it now
17:17:18 <hellais> Like the HTTP requests test has an array of request/responses, while the DNS consistency test has a lot queries
17:18:02 <hellais> I think that moreover every type of nettest in question should have it's own controller so the logic required to formulate the view can all be implemented in a test specific way
17:19:40 <hellais> also previously the assumption was made that the report_id would be unique, but that is actaully not the case since there can be many measurements with the same report_id, what is unique is the tuple (report_id, input)
17:20:03 <simonv3> yeah that makes sense
17:20:10 <hodgepodge> Do we need the report_id to be unique?
17:20:44 <hellais> hodgepodge: not really, I don't think it creates issues in the data model
17:21:22 <hellais> simonv3: https://github.com/TheTorProject/ooni-api/commit/cdfb988e4e9fc5ff7162c8bc9e244a86e7dcb6ad this is what I am talking about in terms of splitting it up into sub-views
17:21:27 <hodgepodge> I don't think so either - usually you'd use an auto-generated PK for all of your queries in lieu of a generated report id.
17:22:27 <hellais> I am bit torn over having another model for just the reports that doesn't keep into consideration the inputs
17:22:44 <hellais> because if we don't have that it can be quite tedious to scroll through all the reports looking for the ones you are interested in
17:23:03 <hellais> that is there is some benefit in keeping the report based abstraction
17:23:48 <hodgepodge> I'm not really sure what you mean - do you mean that the report-based abstraction was only really valuable when dealing with YAML reports that consisted of multiple metrics?
17:24:18 <hellais> hodgepodge: that is what I am wondering, I have not yet made up my mind over it's usefulness or not.
17:24:45 <hodgepodge> I think it would be nice to be able to attribute a list of metrics to a single ooni-probe run.
17:25:44 <sbs> hodgepodge: I agree
17:26:02 <simonv3> I’ll have to check whether angular.routes lets you dynamically choose controllers, though I think ui-router does
17:26:06 <hodgepodge> I think the model we have now is sufficient, especially since we're able to associate multiple metrics with a discrete ooni-probe run.
17:26:17 <hodgepodge> (i.e. be grouping on report_id)
17:26:28 <hodgepodge> ls
17:26:30 <hodgepodge> oops
17:27:10 <hellais> yeah I don't think we necessarily need another table for this to work. We could have a specific API endpoint that does something like 'SELECT report_id, probe_cc, probe_asn FROM reports GROUP BY report_id, probe_cc, probe_asn;'
17:28:40 <hodgepodge> Right. Hopefully Strongloop allows for us to share materialized views fairly loosely in between individual API models.
17:29:04 <hodgepodge> I'm not too familiar with Strongloop yet, but, we'll see.
17:31:08 <hellais> I am actually starting to realise it has a LOT of limitation in terms of what can be done with it. Running SQL queries in this way is somewhat a hack, but in the end I considered it to be to time consuming to switch over to some other framework in mid course, so I think we are going to have to work with it and it's limitation.
17:32:44 <hodgepodge> I agree. We still have time to switch, since the PoC is still being developed.
17:32:59 <hodgepodge> The front-end/back-end code should be able to change independently, right?
17:33:26 <bnvk> on the plus side it makes really nice documentation
17:33:35 <hodgepodge> That's true. We'll give it a plus one there.
17:33:51 <gxg> The API explorer ♥
17:34:00 * hodgepodge might be super biased about the back-end stack
17:34:33 <hellais> hodgepodge: in theory yes, but it would require a bit of work that I don't think we can afford with such a tight deadline.
17:34:43 <hellais> hodgepodge: what are your preferences for back-end stack?
17:35:10 <hodgepodge> That's true, and we are running out of time for the PoC.
17:37:39 <hodgepodge> What we have is good, but I'd ideally want to have more control over the models. Then again, we might just need to look into Strongloop further?
17:41:11 <hellais> perhaps, I would say we stick with this, do the ghetto hax that are needed to get this PoC up and running. Focus on making the GUI look nice and usable. There will always be time to improve the backend in the future
17:42:19 <hellais> should we move into some next steps?
17:42:20 <hodgepodge> Well, maybe. I don't think we should have any ghetto hax in the controller/templating areas because you tend to have low-bandwidth given your other commitments.
17:42:35 <hodgepodge> It'd be hard to unfuck the ghetto hax, so to speak.
17:43:01 <hellais> hodgepodge: by controller/templating you mean the frontend aspect?
17:48:26 <hellais> so I plan to implement a method for accessing the list of reports by ID (suggestions for how this API endpoint should be called) and continue following joe and simonv3 on the API and pipeline respectively
17:49:36 <landers> i'll wrap up my pipeline work, document it, and wrap up the sysadmin ansible-frontend-to-the-shell-scripts PR. EOF
17:50:52 <sbs> alangiu and I will finish measurement-kit 0.1.0 -- EOF
17:53:23 <anadahz> work on the deployment of ooniprobes, revise ooniprobe tests EOF
17:53:41 <simonv3> I’ll be iterating on the changes hellias made on the front-end, I was also thinking of tweaking the grunt tasks so that we don’t have to add each controller to the index page individually - EOF
17:55:11 <hellais> excellent
17:57:08 <hellais> in that case unless there is anything else to add I thank you all very much for attending and talk to you all in the next days.
17:57:11 <hellais> #endmeeting