13:11:16 <zack> #startmeeting
13:11:16 <MeetBot> Meeting started Fri Jun 12 13:11:16 2015 UTC.  The chair is zack. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:11:16 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
13:11:23 <zack> start time slipped my mind :)
13:11:32 <zack> matthieucan: thanks for the SMS!
13:11:36 <matthieucan> sure
13:11:39 <zack> #topic roll call
13:11:44 * orestis 
13:11:45 <zack> looks like everyone is here
13:11:54 <zack> (except me, that is :-P)
13:11:57 <clemux> ack
13:12:13 <zack> #topic next meeting
13:12:23 <matthieucan> shoudl be fine for me
13:12:27 <zack> does anyone have issue with next week, same time?
13:12:34 <orestis> fine fo rme as well
13:12:42 <clemux> no problem
13:12:57 <jpleau> fine here
13:13:00 <zack> #agreed next meeting as usual
13:13:13 <zack> so, who was first in the weekly review, again? :)
13:13:33 <orestis> last time me :p
13:13:40 <zack> ok
13:13:47 <zack> #topic weekly review - orestis
13:14:05 <zack> orestis: please go through your items one by one
13:14:10 <orestis> alright
13:14:28 <orestis> well all items with PR#6 are done
13:14:32 <orestis> i mean the PR is closed
13:14:37 <matthieucan> nice job for that!
13:14:46 <zack> so they've been reviewed as well, right?
13:14:51 <matthieucan> yes
13:14:56 <orestis> that covers parse and render d/copyright
13:14:59 <orestis> yes
13:15:00 <zack> ok, please move them to archived
13:15:16 <zack> sorry, to "reviewed"
13:15:34 <orestis> next is the file by file api doc
13:15:53 <orestis> which is ok i think
13:15:59 <zack> in which list is that?
13:16:08 <orestis> Done
13:16:15 <zack> "license api file-by-file" ?
13:16:38 <orestis> Submit proposal for file by file api
13:16:47 <zack> k
13:16:52 <zack> yes, we've consensus on that I think
13:16:56 <zack> and I've merged your code
13:16:59 <zack> so it's definitely reviewed
13:17:15 <zack> there is the issue raised by jpleau about where the api will be rooted
13:17:18 <orestis> so there are two items for PR#10 which is under review
13:17:21 <zack> by I think it's kinda independent from the API design itself
13:17:43 <matthieucan> yeah, we can easily change route roots
13:17:48 <zack> orestis: those are the 2nd and 3rd item under "done", right?
13:17:52 <orestis> yes
13:17:55 <zack> matthieucan: ack, so let's avoid blocking on that
13:18:02 <matthieucan> yep
13:18:10 <zack> matthieucan: jpleau: are you still on the review of those two?
13:18:19 <jpleau> I think the api doc needs to be updated with the /suites|version|all/ URI
13:19:00 <zack> orestis: ^^^ do you confirm? if so, can you please add an item for that?
13:19:01 <matthieucan> yes, reviewing PR 10. It looks ok, I should be able to test it more deeply tonight
13:19:08 <orestis> yes that's true. i am planning to do that once the PR is confirmed
13:19:09 <jpleau> Unless it was decided not to that of course :) (/api/file/package/suite|version|all/path)
13:19:16 <zack> sounds great
13:19:28 <jpleau> For checksums did we want to do similar or keep the package and suites in query string parameters?
13:19:31 <matthieucan> perfect if it matches the sources api
13:19:39 <matthieucan> hm
13:19:48 <jpleau> I think orestis suggested it was better to keep ?package and ?suite
13:20:01 * zack has no strong opinion on this
13:20:01 <orestis> for the checksum i proposed not to change them since it would defer from the s.d.n api
13:20:34 <jpleau> orestis: oh nevermind, yes. I forgot we had checksum on sources api. :)
13:20:45 <jpleau> I want to do more tests tonight though bnefore confirming!
13:20:46 <zack> jpleau: which doesn't mean it cannot be improved of course :)
13:20:53 <zack> but the day we do that, we can do it more broadly
13:21:00 <zack> s/can/should/
13:21:14 <zack> so, next item is "show license of individual..."
13:21:31 <zack> although I don't understand why that is specific of sha1
13:21:38 <zack> searching by package/version/path should be the same, no?
13:21:46 <orestis> yes that i have everything ready with the new template etc.. i didn't pull request coz it depends on PR#10
13:22:21 <zack> orestis: but is it specific of sha256 searches? or more general?
13:22:27 <matthieucan> orestis: great news, I'll hurry up with PR#10
13:22:31 <orestis> well i was thinking it wasn't going to be exactly the same  since searching by checksum gives much more results
13:22:56 <zack> right, but ideally it would be using the same code, except it will show multiple results in one page
13:23:00 <zack> or something like that, right?
13:23:03 <orestis> and i tried to include some 'stats' for the checksum i.e which is the most frequent license of the checksum etc
13:23:18 <jpleau> zack: What I was proposing for sha256 was : /api/sha256/(package_name|all)/(suite|version|all)/checksum
13:23:39 <jpleau> If we can adapt the sources api, it'd mean all the APIs follow the same route scheme
13:23:50 <orestis> zack: well yes the code will be almost the same with file search.. i just haven't had the time to do that as well
13:24:01 <zack> orestis: that's OK, it was just to understand
13:24:23 <orestis> yes i know :)
13:24:27 <zack> cool
13:24:31 <zack> so, that's pending
13:24:42 <zack> orestis: maybe you can add "PR: waiting for PR#10"
13:24:42 <orestis> so as soon as we have PR#10 merged i ll pull request that
13:24:48 <zack> (in the title)
13:24:53 <orestis> yes sure..
13:25:05 <zack> jpleau: given it depends on a change on the already deployed API, can we make that a separate feature/discussion?
13:25:19 <jpleau> zack: yep !
13:25:29 <zack> so, next topic
13:25:31 <orestis> and for the batch api doc i am working on it but i am waiting to have the final routes for the file-by-file
13:25:38 <zack> #topic orestis - next week
13:25:44 <matthieucan> orestis: ack
13:25:50 <zack> orestis: sounds like a great link to next week planning ;)
13:26:13 <zack> what would you like to work on?
13:26:29 <orestis> so next week i guess i ll do the batch api but besides that i don't know what you think its best to push for
13:26:54 <zack> uhm
13:27:10 <zack> that would be good already
13:27:23 <zack> then, I guess there are 2 independent topics that can go in parallel, and according to your pref
13:27:29 <matthieucan> what about license statistics? you said you began doing that?
13:27:32 <zack> one would be the license scanner, client-side
13:27:41 <zack> another is looking into SPDX and export
13:27:52 <zack> matthieucan: right, I'm working on it for a research project, but i don't have the resulting data yet
13:27:59 <zack> it's taking longer than expected (no wonder)
13:28:17 <zack> a task related to do that would be deciding the DB schema/data model for storing license statements in the DB
13:28:27 <orestis> matthieucan: no it is just a simple thing i did..
13:28:36 <zack> (it's already on trello: DB schema for persistent copyright files)
13:28:51 <orestis> yes i was thinking that its better to have the db before the stats.. but then wouldn't that interfere with clemux work?
13:29:13 <zack> hopefully, clemux work will be done without affecting the DB structure
13:29:24 <zack> so as long as your interface is the DB, it should be fine
13:29:33 <clemux> ack
13:29:35 <zack> orestis: but really, it's about what you'd like to prefer to work on
13:29:42 <zack> above are macro areas to prioritize
13:29:52 <zack> feel free to pick from there, and mix and match other minor items
13:30:06 <zack> (in the future, it would be useful if these debates happen _before_ the meeting
13:30:15 <zack> so that we arrive at the meeting with already a proposal from you on what you'd like to work one)
13:30:22 <orestis> ack
13:30:44 <zack> so, your preferences?
13:30:54 <zack> (or we can decide after the meeting, if you need to think about this)
13:31:39 <orestis> well ok i think i ll do the batch api, fix the bugs that we discovered (the heuristics for machine readable, fail to dump when render fails) and yes i d like some more time to think about it and ping you about something else
13:31:48 <zack> ok, great
13:32:01 <zack> please assign to yourself those items, and ping us afterwords for the rest
13:32:16 <zack> #topic weekly review - clemux
13:32:21 <zack> clemux: you're on stage!
13:33:29 <clemux> so, I submitted a first draft of the workflow graph for celery tasks
13:33:43 <zack> and we've reached consensus on it, right?
13:34:30 <clemux> the only problem left is update_metadata
13:34:42 <zack> right, but it doesn't seem to be a blocker
13:34:50 <zack> I'm fine with leaving it as you designed for now
13:34:57 <zack> we can improve/split it later
13:35:06 <clemux> ok
13:35:13 <zack> IIRC i'm waiting for your PR to merge
13:36:09 <clemux> I have a commit to push, I'll do that immediately after the meeting
13:36:14 <zack> k
13:36:18 <jpleau> (quick question) Is it possible with the tools we have to have an html version of the graph with tooltips describing each task? It could be great addition in doc/, and it's visual
13:36:33 <zack> not sure if dot handles that
13:36:47 <zack> it can create html maps, IIRC, but of the old-style kind
13:37:11 <jpleau> oh, ok. I'll see if I can come up with something for that :)
13:37:18 <zack> awesome
13:37:26 <zack> clemux: what else from this week?
13:37:45 <clemux> the next task was to determine how to use the sqlalchemy session in celery tasks
13:37:49 <clemux> I'm still stuck on that :(
13:37:58 <zack> is that the nested transaction question?
13:38:02 <zack> I've thought about that a bit
13:38:10 <clemux> yes that's the main problem
13:38:31 <zack> so, we can dig more deeply post meeting, but in a sense I think we can be way more "adventorous" than the old updater was
13:38:44 <zack> I think it'd be totally fine to have one transaction per package, for instance
13:38:53 <zack> without nesting transactions
13:39:06 <zack> clemux: that would solve most of the problem, right?
13:39:19 <clemux> it'd be simpler, yes
13:39:26 <matthieucan> clemux: yep sorry, couldn't help you on that :/
13:39:41 <clemux> zack: however, having all hooks in a single transaction is still a problem for me
13:39:49 <clemux> for a given package, I mean
13:39:57 <zack> right
13:40:03 <zack> but we probably don't need that either
13:40:13 <zack> for instance, sloccount can run in its own transaction
13:40:25 <zack> as long as it knows the package has been unpacked already
13:40:33 <zack> would that help?
13:40:43 <clemux> yes
13:40:58 <clemux> that should unblock the task
13:41:08 <zack> and, if we reach a point where we cannot have smaller transactions any more, well too bad, we will group together tasks that you've currently split
13:41:13 <zack> it will just reduce parallelism
13:41:20 <zack> (but not by much)
13:42:04 <clemux> indeed
13:42:19 <zack> anything else to report about, before moving to next week planning?
13:42:41 <clemux> zack: so you're saying a package could be updated with the results from sloccount but not e.g. ctags?
13:42:48 <clemux> (if ctags failed)
13:43:03 <zack> clemux: for instance, yes
13:43:11 <zack> as long as we can keep track of the fact the task has failed, that is
13:43:19 <clemux> this should not be a problem
13:43:57 <clemux> ok, I feel better, I should be able to move on faster now
13:44:07 <zack> more importantly, I guess I'm saying that even if we need --- for now --- to group all plugins together in a single tasks,
13:44:12 <zack> it wouldn't be such a big deal
13:44:28 <zack> because the increase in parallelism in treating one package independently from the other would be huuuge anyhow
13:44:53 <clemux> zack: I thought one of the point of the new architecture was to be able to rerun e.g. only ctags on all packages
13:45:04 <zack> that's correct, yes
13:45:05 <clemux> with new release, other options, ...
13:45:15 <zack> (and a good point)
13:45:21 <clemux> that's why I hadn't considered going that way
13:45:34 <zack> ok, so let's settle with what I mentioned before :)
13:45:39 <zack> if a single plugin fails, then too bad
13:45:48 <zack> the others would have succeeded anyhow
13:46:01 <zack> deal?
13:46:09 <clemux> it'll be retried later and the failures will be monitorable
13:46:12 <clemux> yup
13:46:15 <zack> awesome
13:46:29 <zack> anything else to discuss before planning next week?
13:46:40 <clemux> nothing else
13:46:48 <zack> #topic next week - clemux
13:47:05 <zack> there's plenty of stuff on "this week" for you already
13:47:09 <zack> should we just confirm all of it?
13:47:12 <clemux> yes
13:47:29 <zack> ok, I guess the transaction issue has been an unexpected blocker this week, right?
13:47:32 <clemux> yes
13:47:42 <zack> great it's solved now ;)
13:48:03 <zack> ok, so last topic, a great classic
13:48:05 <zack> #topic misc
13:48:14 <zack> anything else people would like to discuss?
13:48:31 <orestis> nope
13:48:39 <matthieucan> I'm fine
13:48:54 <jpleau> nothing here
13:49:12 <zack> let's adjourn then, and sorry again for my delay!
13:49:35 <clemux> nothing else from me
13:49:41 <zack> #endmeeting