11:49:24 <buxy> #startmeeting
11:49:24 <MeetBot> Meeting started Wed Aug  7 11:49:24 2013 UTC.  The chair is buxy. Information about MeetBot at http://wiki.debian.org/MeetBot.
11:49:24 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
11:49:57 <buxy> #link https://trello.com/b/faDgzjwO/pts-rewrite the coordination board we use
11:51:13 <buxy> mlalic: before going further, can you please estimate the stories https://trello.com/c/1kjNz0kL/104-create-a-framework-for-the-action-needed-panel and https://trello.com/c/qfB4evF7/105-convert-the-existing-todo-problems-entries-in-actionitem
11:52:06 <buxy> I want to plan them in this week obviously since they impact most of the future task
11:52:25 <mlalic> Yes, of course. Just give me a moment to make read through some of your additions to the specs
11:55:54 <mlalic> okay, done
11:56:14 <buxy> mlalic: you did complete all stories for this week, for an estimation of 25 points, can we increase the limit for the coming week or did it feel about just right?
11:56:39 <buxy> (and this despite your travel)
11:57:02 <mlalic> Yes, everything was completed despite the travel. I'd say it would be safe to have a slightly larger load, but not too much, though
11:57:18 <buxy> mlalic: how much do you suggest?
11:58:05 <mlalic> If this was 25, then I'd say at most 30.
11:58:24 <buxy> Ok.
11:58:48 <buxy> #topic planning of next iteration
11:59:58 <buxy> Looking at https://trello.com/c/kjJrHDdl/95-wnpp-derived-information-8 I have the feeling that "8" is a bit too much for an estimation. Do you want to revise it or do you believe it's still correct?
12:02:02 <buxy> (or maybe it's correct, I missed the part about the fact that we display stuff on dependent packages)
12:02:03 <mlalic> Hmm, to be honest, I wouldn't change it. The reason is that I would need to see what sort of alternatives there are to the the way the dependencies are found in the current PTS
12:03:01 <mlalic> Yes, exactly. If it were just a link to the wnpp bug, it'd definitely be less
12:04:53 <buxy> We must decide when is a convenient time to meet next week because wednesday isn't that day.
12:05:11 <buxy> Do we want to meet before that date or after it?
12:06:35 <mlalic> I'm thinking before could be better to make sure there is definitely something in the works on Wednesday
12:06:59 <buxy> pabs, zack and me will be at debconf and there will always be talks, so we must find a time that suits us (starting with me :))
12:07:12 <mlalic> Well, that is the priority, of course
12:07:30 <mlalic> If it suits you better to have it after Wednesday, then so be it :)
12:07:54 <buxy> mlalic: so the morning is not ok for you, we must find a time in the afternoon or the evening, is that correct?
12:08:01 <mlalic> As for me, any time after 1 PM (to account for lunch time) is okay.
12:09:58 <buxy> mlalic: what about tuesday between 13:30 and 15:30 ?
12:10:12 <mlalic> Yes, this is perfect.
12:10:55 <buxy> #agreed next meeting will be on tuesday (Aug 13th) between 13:30-15:30 (UTC+2)
12:11:11 <buxy> so I have to drop some task to account for the missing day
12:12:08 <buxy> ok, scheduled 23 points of tasks
12:12:17 <buxy> mlalic: does it look to you?
12:12:20 <buxy> +ok
12:13:03 <mlalic> Yeah, that seems good
12:13:21 <buxy> #topic problems with the email interface
12:14:02 <buxy> mlalic: did you investigate a bit the OOM we had on pts.debian.net?
12:16:05 <mlalic> Yeah, I looked at it a bit. As mentioned, went through the various logs and ended up fixing the other bug which was uncovered thanks to this crash. Didn't do any profiling or tests to compare the memory usage of the old dispatch yet, though
12:16:47 <buxy> The other bug being the problems with "From" containing unicode?
12:16:52 <mlalic> yes
12:17:08 <mlalic> Well, not From, but X-PTS-Package, etc.
12:17:12 <mlalic> (The injected headers)
12:17:39 <buxy> mlalic: that's the part I don't get, those do not contain any non-ascii characters, how could they trigger the problem?
12:17:53 <mlalic> ah, so... BytesIO is a weird class
12:18:03 <mlalic> it lets users pass it either unicode or plain str objects
12:18:05 <mlalic> but
12:18:08 <mlalic> if you end up mixing the two
12:18:21 <mlalic> then it will fail if any of the str objects contain >128 bytes
12:18:40 <mlalic> So, having the header unicode and the body str -> problem when the body contains utf-8 encoded data
12:19:22 <mlalic> (This is documented as expected behavior in python2)
12:19:27 <buxy> ok
12:20:28 <buxy> and can't you trigger this failure by forcing the email generation via BytesIO without actually sending it?
12:21:29 <mlalic> I could do the same call as the underlying django SMTP backend does, but other than testing if it fails, there's no meaningful test of its output, really.
12:23:38 <buxy> not even through an email.generator.Generator?
12:24:51 <buxy> anyway, ensuring it doesn't fail is already a first step, even if we can't check the output for correctness
12:25:10 <mlalic> What I mean is, I can get the bytes which would be passed to the smtp connection, but other than parsing them back into a email.message.Message, there's not much to do with them.
12:25:24 <mlalic> The content and headers are already tested for correctness
12:26:06 <mlalic> So simply one more test which tries to do this message -> bytes -> message conversion should be enough to be the regression test for this bug. The others wouldn't have to change.
12:26:23 <buxy> ok, then let's do this
12:27:40 <mlalic> Right. As for the memory a single management command uses, there are definitely no unused imports in the command itself and we cannot control what Django imports anyway.
12:27:58 <buxy> #action mlalic adds a non-regression test to ensure we don't mix unicode and str in headers and always have mails that can be converted to bytes by BytesIO
12:28:12 <mlalic> So for now it seems the direction should be to make sure that there aren't hundreds of forked processes...
12:28:58 <buxy> Yes that seems the best solution to me too. But exim doesn't seem to make it easy for us.
12:29:13 <buxy> Have you looked into exim's configuration for this aspect?
12:30:59 <mlalic> Not in great detail no. For now all I saw were options to limit the size of a message
12:31:55 <buxy> #action mlalic to look into how to configure exim to avoid too many parallel deliveries over pipes to memory-expensive commands, including asking a question on the upstream mailing lists if needed
12:33:19 <buxy> I'm wondering whether the exim configuration that you're using is more sensible to this problem than the setup with .forward that we have on packages.qa.debian.org
12:33:22 <mlalic> Yep, will do...
12:33:40 <buxy> Do you feel like experimenting this too?
12:34:11 <mlalic> To tell you the truth, I'm not sure how it is set up there. If you point me to something I could read on that, then yeah, I'd try and see what the difference is.
12:34:20 <buxy> Something like setup a .forward with a script that sleeps 120, send 150 mails to it, and look at the number of processes over time.
12:36:32 <buxy> #link http://anonscm.debian.org/gitweb/?p=mirror/dsa-exim.git;a=summary exim configuration of the PTS
12:37:06 <buxy> and associated to that there's a set of files in /srv/packages.qa.debian.org/mail/ but you have those in the SVN repositiory of the PTS
12:37:17 <mlalic> Yep, I know about those files
12:37:27 <buxy> there are multiple .forward-* files to deliver to each command
12:37:34 <mlalic> Okay, then yeah, this'll actually be the first thing I try then...
12:38:39 <buxy> you can see that the official config have some limit
12:38:45 <buxy> queue_only_load = 15
12:39:08 <buxy> and smtp_accept_queue_per_connection = 50
12:39:11 <buxy> etc.
12:39:26 <buxy> those are the few parameters that I have found that we can use to limit the damage
12:40:34 <buxy> #action mlalic to experiment if the current exim setup of packages.qa.debian.org copes better with too many mails
12:41:14 <mlalic> This seems good. I've never really administered a server nor configured exim for any sort of high load so it makes sense that the official config is that much better :)
12:42:10 <mlalic> But yes... I'll try this out and see what happens. :)
12:43:56 <buxy> good
12:44:36 <buxy> #action mlalic to inform buxy when the deployment is done so that we can reenable forward from packages.qa.debian.org to pts.debian.net
12:45:26 <buxy> after this incident I have disabled the forward but never enabled it back due to the other bug with email headers, now that this bug is fixed, we can enable it back even if we haven't found the solution for too-high load yet
12:46:41 <buxy> #topic status of pts.debian.net
12:46:42 <mlalic> Yeah, with keeping track of the status of the server for another similar high-load spike which brings it down
12:47:57 <buxy> mlalic: I expect that queue_only_load = 5 should be enough for now until we have a better solution
12:48:11 <buxy> (or maybe a bit higher)
12:48:58 <buxy> mlalic: when you do the next deployment, can you also schedule the extraction of all the files?
12:49:11 <buxy> I believe this hasn't been done yet.
12:49:42 <buxy> #action mlalic to schedule extraction of all changelog/copyrigh/control/rules files after next deployment
12:50:01 <mlalic> Yeah, sure, the task itself is smart enough now :)
12:51:21 <buxy> #topic collaboration with Paul Wise (pabs)
12:51:53 <buxy> mlalic: IIRC you haven't said anything yet about this? do you have an opinion? do the suggested rules make sense to you?
12:53:39 <mlalic> Ah... Well, yeah, I mean, to be honest, I didn't think my opinion mattered... I'm really happy someone wants to join, of course, no doubt about that. The rules are mostly to keep in line with GSoC's rules which is completely fine with me.
12:54:22 <mlalic> So, no objection from me, on the contrary, quite happy to see more contributors already :)
12:54:26 <buxy> I don't know if Paul has plans to work on the PTS next week during debconf but it's possible.
12:55:37 <buxy> mlalic: do you feel like reviewing and merging yourself or would you rather that Paul goes through me for that?
12:55:48 <buxy> or maybe something in between (I review, you merge)
12:56:15 <buxy> ah pabs arrived, welcome
12:56:18 <mlalic> This sounds good... I mean, since you review my code as well, it seems only logical, doesn't it? :)
12:56:31 <buxy> ok, makes sense.
12:56:40 <mlalic> But I can do the merges since that could be more "raw" work...
12:57:35 <buxy> pabs: we were just discussing the practical details so that you can work on the new PTS
12:57:59 <pabs> cool
12:58:02 <pabs> sorry for my lateness
12:58:39 <buxy> #agreed buxy will review external contributions first and mlalic will merge them when they are ready
12:59:04 <mlalic> Btw, if possible I wouldn't mind receiving the commitdiff emails for those commits, as well...
13:00:10 <buxy> pabs: have a look at https://trello.com/b/faDgzjwO/pts-rewrite to see what we planned for next week
13:00:37 <buxy> pabs: we agreed to meet tuesday between 13:30 and 15:30, could you make it or did you have talk in conflict?
13:00:54 <buxy> s/did/do/
13:01:41 <buxy> mlalic: for this pabs would have to push his work in the the repository, that's possible, we can decide to use people/pabs/* branches for example
13:02:00 <buxy> I think it's a good idea.
13:02:54 <mlalic> If it's too impractical for anyone, then no need to go out of your way on my account...
13:03:36 <buxy> no, no, I have no problem with that, the alternative would require using multiple personal repositories which isn't really warranted here
13:04:16 <buxy> and d-i already uses a similar people/ structure for topic branches
13:04:25 <mlalic> Okay, great.
13:05:34 <buxy> #topic other questions
13:05:46 <buxy> mlalic: do you have any questions for me?
13:06:11 * buxy goes away for a few minutes
13:06:51 <mlalic> For this deployment, I am thinking of deploying the todos/problems features which are already implemented (even though a new framework will be in the works for next week), correct?
13:07:47 <mlalic> Also, do you think I should merge the alternative-layout branch (now that the general panel has been on the left for a while)
13:08:59 <mlalic> The app which would consolidate all email features cannot be called "email" (since that conflicts with python's email), so something else is needed. email_utils? email_processing?
13:12:10 <buxy> hum, pts.email is in conflict with "email" ?
13:12:42 <buxy> for the deployement, yes you deploy what you have, even if further changes are expected next week
13:14:41 <mlalic> django-admin does not want to create that app
13:15:36 <buxy> for the layout-alternative branch, I don't really know, I fear the answer is most complicated because we probably want different behaviour depending on the fields/panels involved.
13:15:43 <buxy> Leave it as is.
13:16:14 <buxy> That's the kind of tweaking that pabs might be interested to try out as well.
13:17:03 <mlalic> Ok...
13:18:57 <buxy> for the app name, I don't like the too long names. is "mail" any better ? :-)
13:19:36 <mlalic> okay, that seems to work :)
13:19:36 <buxy> otherwise "ptsmail" or "ptsemail"
13:21:57 <buxy> pabs: any questions on your side? have you seen my former questions to you?
13:22:25 <KGB-1> 03zack 05feature/db-ng c03503a 06debsources 10(10 files in 3 dirs) * module structure refactoring: move hooks to plugins.* namespace
13:23:02 <buxy> mlalic: do you have other stuff on your side or are we mostly done?
13:23:38 <mlalic> As far as I'm concerned nothing else for now...
13:24:38 <buxy> Ok, we'll wait a few minutes to see if pabs will reappear, otherwise I'll finish the meeting.
13:24:48 <pabs> buxy: sorry, got distracted. will get up to speed before the next meeting
13:25:23 <buxy> pabs: is next tuesday between 13:30-15:30 good for you?
13:25:38 * pabs looks at schedule
13:25:44 <pabs> UTC or Swiss time?
13:25:54 <buxy> swiss time (UTC+2)
13:26:31 <buxy> maybe you want to attend the secure boot talk but I think those a 14:30 can be skipped
13:26:40 <pabs> that looks fine to me
13:26:58 <buxy> great
13:27:09 <mlalic> Great! I'm really happy you'll be joining us, pabs!
13:27:44 <buxy> thank you all
13:27:48 <buxy> #endmeeting