15:59:37 #startmeeting 15:59:37 Meeting started Thu Jun 1 15:59:37 2017 UTC. The chair is lamby. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:59:37 Useful Commands: #action #agreed #help #info #idea #link #topic. 15:59:44 Good evening all 15:59:50 #topic Introductions 15:59:54 Who is around? :) 15:59:55 Good evening! 16:00:00 hi 16:00:00 o/ 16:00:01 hey siamezzze 16:00:05 hey h01ger, Faux o/ 16:00:08 lamby: can you please #chair me? 16:00:10 * danielsh` (multitasking) 16:00:20 h01ger: Tell me what to paste :) 16:00:22 hi 16:00:26 good evening 16:00:27 * emaste waves 16:00:29 lamby: #chair h01ger 16:00:29 hello 16:00:29 hello! 16:00:35 hello from FreeBSDLandia 16:00:36 #chair h01ger 16:00:36 Current chairs: h01ger lamby 16:00:46 fsckd: hey, what are you working on? 16:00:55 hey fsckd! 16:00:57 woo new faces 16:01:02 * h01ger works on debian mostly but also maintains this tests.r-b.o thing 16:01:33 * h01ger singled out fsckd because the other people here have been at various irc and RL meetings before… (and the topic is as it is :) 16:01:41 i am an ARch Linux user and i've recently taken an interest in reproducible builds 16:01:52 fsckd: cool cool! 16:02:05 hello 16:02:13 welcome aboard 16:02:14 #info Apologies from Mattia (mapreri) -- it's Festa della Repubblica tomorrow 16:02:19 hey sangy 16:02:44 * vagrantc works on debian, and maintains armhf hardware for the debian tests.reproducible-builds.org 16:02:46 * h01ger is happy about more involvement from the Arch Linux community 16:02:56 btw: when you make patches, be sure to send them upstream so that everyone profits from your work in the long term 16:03:10 #info there is #archlinux-reproducible on freenode 16:03:18 * lamby works on Debian, diffoscope, etc. etc. 16:03:23 #save 16:04:05 the agenda is at https://pad.riseup.net/p/reproducible-irc-meeting-9 16:04:09 So, this will probably be quite a quick meeting. Do say if something or a topic is not clear; sometimes us old timers can assume too much. 16:04:36 * sangy works (volunteers?) on Arch security. 16:04:39 Okay, let's get going. 16:04:41 #topic Feedback for the reproducible.json spec format 16:04:47 bmwiedemann: You're up ^ 16:04:51 sangy: hehe :) (works/volunteers :) 16:05:30 the idea of the reproducible.json format is for people to be able to test rb externally and publish results 16:05:46 #info proposed format: http://rb.zq1.de/spec/json-format.txt 16:05:47 and then we can build tools that make nice graphs, stats etc out of it 16:05:59 bmwiedemann: did you look at the existing debian json when writing that format? 16:06:04 similar to the ones that are already on tests.r-b.org 16:06:14 ( https://tests.reproducible-builds.org/reproducible.json.bz2 that is ) 16:06:25 yes, heavily inspired by that one. 16:06:31 #info https://tests.reproducible-builds.org/reproducible.json.bz2 is the format debian is using 16:06:40 (moin) 16:06:47 hi Eric[m] 16:06:49 off the top of my head: need to add a format number somewhere 16:07:07 i.e., "'reproducible-json-data': 1" or so 16:07:11 bmwiedemann: i think there are more stati (waiting for deps, blacklisted) 16:07:25 or "format-version": 1 16:07:41 * h01ger nods danielsh`'s idea 16:07:42 bmwiedemann, no, BTDT. The field name should be unique for the file format 16:07:51 where should the format definition live? 16:08:36 human-readable definition or some .schema file for auto-validation? 16:08:38 In the filename, to keep the format beautifully simple. 16:08:47 bmwiedemann, for background: https://issues.apache.org/jira/browse/SVN-3744 16:08:49 ^ the version, not the schema. 16:09:08 bmwiedemann: is http://rb.zq1.de/compare.factory/reproducible.json already updated daily or is that a one time data dump? 16:09:58 h01ger: updated whenever a build round finished, usually every 1-2 weeks. you can see one dir level up. compare.factory is just a symlink to the latest one 16:10:05 ic 16:10:14 still very fine for now 16:10:52 lynxis: ^^ can you please look at that json format and comment whether its suitable for lede + coreboot or tell us whats needed? 16:11:12 _hc: uniq[m] : ^^ can you please look at that json format and comment whether its suitable for f-droid or tell us whats needed? 16:11:16 I wonder if this somewhat overlaps with the whole publishing-logs-to-binary-logs. Not saying we should do that instead, but we could think ahead and ensure that each entry could serve double duty in both. 16:11:36 (ie. "source" key?) 16:12:07 lamby i just had that same thought but i'm not sure if this json doc would actually communicate much if immutable-logged? 16:12:08 i'd rather keep the publishing-logs-to-binary-logs out of this for now… 16:12:09 would it be better if build_date were build_time? 16:12:26 while i'm guessing it's just the unix clock... it 16:12:37 seconds since epoch ... 16:12:40 it is 16:12:46 btw, the example on http://rb.zq1.de/spec/json-format.txt doesn't match the spec does it? (suite -> release) 16:12:54 that doesn't look like a date to my human eyes :) 16:12:57 bmwiedemann: i think the next step is to write a parser for this json and feed the data into postgresql 16:12:58 ah, we renamed that later 16:13:13 h01ger: any language preference? 16:13:15 bmwiedemann: then the next step is to modify our debian python scripts, running as jenkins jobs, to enable them to build debian pages and suse pages 16:13:18 bmwiedemann: python 16:13:29 Eric[m]: (My muse was that entry dict entry would be what would be posted, but perhaps too much would change for binary logs anyway that this is highly premature) 16:13:57 bmwiedemann: and once we have this json parser, the format defintion should be part of that code, possible as comments 16:14:10 cause that will be the de facto reference implementation 16:14:13 sangy: fixed the suite=> release in examples 16:14:23 istm that json blob would be a nice simple standard (doesn't do too much, thus hard to argue about..) for drawing pictures of progress and such 16:15:14 but i'm sitting here with the hash-all-the-things hat on again and so of course the metadata i'd want to immutable-log has a very different set of additional stuff in it (which no one else will likely agree with...) 16:15:34 16:15:48 nodnod 16:15:53 #info: the next step is to write a parser for this json and feed the data into postgresql on jenkins.d.n - then the next step is to modify our debian python scripts, running as jenkins jobs, to enable them to build debian pages and suse pages 16:15:55 Eric[m]: I have a checksums.txt in the same dir keeping track of binary results 16:15:57 it seems odd that the build_date isn't more specifically defined 16:16:10 e.g. any time between when the build start or finish 16:16:14 #info once we have this json parser, the format defintion should be part of that code, possible as comments - because that will be the de facto reference implementation 16:16:26 why not record build_date_start, build_date_finish ? 16:16:55 vagrantc: because it does not matter so much - and it makes implementation easier for producers (like me) 16:17:13 bmwiedemann: i'm happy to write this code or help writing it… mapreri and spectranaut also know that part of the code base really well, partly better than myself (cause they wrote the latest iterations of it) 16:17:57 though i dont know what else to discuss here in the meeting. someone needs to code something, but i think thats it… 16:18:37 h01ger: I can draft a simple stand-alone snippet, but that will maybe only be 20 lines 16:18:57 Following vagrantc, couldn't each node on this list map to a file with the complete build log, start_date and other relevant info? 16:19:08 bmwiedemann: cool cool 16:19:21 bmwiedemann: i'd be very glad to guide you into the code base 16:19:26 sangy: Some sort of external URI reference? 16:19:57 I'd imagine so, maybe it could be implicit if we define a directory structure, but that may be overcomplicating thigns? 16:20:04 sangy: should be optional. 16:20:15 Yeah, that sounds a little overcomplicated. 16:20:33 are we still discussing the format? 16:20:35 * vagrantc wonders what the real goal is 16:20:35 like, first we want stats and graphs, and then we want to click on package names and get details 16:21:14 should be able to find build logs by the package and build time, i guess 16:21:26 makes sense 16:21:29 we want t.r-b.o/suse/$srcpkg to work and have nice graphs and stuff for suse as we do for debian 16:21:41 That would be awesome 16:21:43 and then we want the same for guix, archlinux, lede, foo 16:21:43 bmwiedemann: Could use https://github.com/bdrung/image-release-spec as some kind of template. 16:22:03 yeah 16:22:05 Anyway, sounds like we have enough here to get going on…? Do we have some specific action points 16:22:08 * ? 16:22:27 i #info'ed some points but i didnt want to #action them 16:22:29 #save 16:23:01 i don't know if this will be interesting but for an example of how far out i might be going in my checksums.txt 16:23:02 https://github.com/polydawn/hitch/blob/7c30b81209d7495eb09a6f079b3458beeebb6b74/api/records.go 16:23:13 still waiting for the feedback from lede/fdroid 16:23:15 lamby: can you please say #save? seem it didnt work for me 16:23:17 #save 16:23:20 merci 16:23:29 #info https://github.com/polydawn/hitch/blob/7c30b81209d7495eb09a6f079b3458beeebb6b74/api/records.go 16:23:31 sigh, works now indeed. 16:23:36 #info https://github.com/bdrung/image-release-spec 16:23:48 whats that polydawn url about? 16:24:03 5 lines up ;) 16:24:12 i link that not to say it's a good idea that i've got there, but because it's an example of how much someone could possibly dissent what the spec should look like when they get increasingly aggressive about linking build records and dependency graphs 16:24:22 I think we have enough here. Next topic! 16:24:25 #topic tests.r-b.o/Debian once Stretch has been released 16:24:34 h01ger: I think this is yours, judging by the colour…? 16:24:50 so, stretch will be released on 2017-06-17 16:24:58 at which point testing will become buster 16:25:01 \o/ 16:25:08 and i think we should keep logs+stats for stretch 16:25:20 so i think we ought to rename testing to stretch now 16:25:27 and add buster on 2017-06-18 16:25:31 not sure if this is on-topic, but given that almost all the armhf boards require newer kernels, i'd love to upgrade armhf to stretch, at least... 16:25:36 and stop testing "testing" ;) 16:25:40 vagrantc: thats a different topic 16:25:55 vagrantc: lets discuss that in a minute, it is related :) 16:26:21 * h01ger is not sure what to discuss about this topic here and now though 16:26:22 i generally always prefer to reference things by codename, when possible 16:26:31 h01ger: why stop testing 'testing' ? wont that be updated again soon? 16:26:43 a few commits are needed to make this happen, so i guess this was just a heads-up topic 16:26:53 bmwiedemann, 'testing' is a floating label... the suggestion is to test what the label points to, directly. 16:26:53 bmwiedemann: because we'll test buster instead, which is/will be "testing" 16:27:06 ah. good. 16:27:16 Can we avoid this for buster+1 ? ie. always use codenames from now on? 16:27:17 h01ger: will we keep testing stretch once it's released? 16:27:34 We'll potentially hit this with other distros anyway. 16:27:53 lamby++ 16:28:08 vagrantc: yes, but way slower and eventually not really anymore 16:28:20 h01ger: right. 16:28:37 #topic upgrading build nodes once stretch is out 16:28:40 could test security updates and proposed-updates... 16:28:46 vagrantc: your topic :) 16:28:51 frankly, i think we should do it 16:28:57 should we do it now? 16:28:58 "Just" add some stuff to .htaccess, right… ;) 16:29:03 upgrade all build nodes to stretch on the 17th or 18th or so 16:29:31 vagrantc: i'd rather do more urgent stuff first, like preparing buster, and maybe even the json parser, to make bmwiedemann happy and keep momentum 16:29:31 if there's some RC bug we haven't yet discovered, would be a shame to not get it fixed in release 16:29:42 vagrantc: are you really suffering to run those boards on jessie for the next 2.5 weeks? 16:30:11 h01ger: no, it's fine ... but this late in the release cycle... 16:30:21 i like to try and make sure there are no hidden bugs awaiting us 16:30:27 why not just upgrade one of them to discover bugs? 16:30:28 vagrantc: oh, i would also like to add your 3 new nodes first… ;) 16:31:05 (we often do that in our SUSE-internal CI hosts) 16:31:06 bmwiedemann: that's feasible 16:31:07 and if mapreri wants to go crazy on this and just do it, i woudlnt object either 16:31:26 but i dont think it would be helpful to update just some 16:31:38 that will increase maintenance costs for that period 16:31:45 true enough... 16:31:57 we're talking about 38 hosts here… 16:31:58 * vagrantc will talk to mapreri 16:32:29 also, one of those 38 is jenkins, and i do expect some more issues there, once we upgrade that to stretch 16:32:43 jenkins runs 1400 jobs, only 200 of them are related to reproducible… 16:32:52 in general, i've been testing new machines with stretch first, and then re-installing with jessie 16:32:56 newer jenkins needs java-1.8 - and sometimes slaves then too 16:33:07 bmwiedemann: we dont have java on any slave 16:33:43 well, it's been at the back of my mind weather to go for it 16:33:55 and we already use java-1.8 on jenkins because upstream thought it would be nice to change this requirement when going from 2.46.2 to 2.46.3 16:34:01 will check in what mapreri thinks, and check in before doing anything drastic 16:34:26 vagrantc: i'd prefer we wait til after the release and then upgrade all nodes in one go 16:35:05 i won't push it 16:35:11 thanks 16:35:25 figured it was worth mentioning, though 16:35:31 sure thing! 16:35:37 :) 16:35:46 #topic Reproducible Builds Summit #3 2017 16:35:48 Next? :) 16:36:15 so, i've spoken/mailed with gunner and would be happy to have another summit with us 16:36:46 \o/ \o/ \o/ 16:36:48 gunner is our moderator/facilitator who together with beatrix helps to run the meetings smoothly, nicely & effectivly, and documented 16:36:49 nice to be thinking about it with plenty of lead time 16:37:00 h01ger: most excellent 16:37:08 so, while we do have two dates to choose now 16:37:09 nice. any proposal for the place? 16:37:18 we dont have any funding yet 16:37:28 * vagrantc hopes for something central, like iceland 16:37:36 so the summit might as well not happen, if we dont get funding… or might happen very differently 16:37:47 no proposals for a place yet… 16:38:11 We haven't had pushback *against* funding yet though? We've just not asked? 16:38:17 lamby: yes 16:38:23 h01ger: what are the suggested dates? 16:38:43 possible dates are either 30 Oct-3 Nov or 6-10 Nov 16:38:51 2017 16:39:10 so Mon-Fri each 16:39:13 * h01ger will set up a doodle poll so that you all can state your preferences on these dates 16:39:28 bmwiedemann: thats ment to be Tue-Thu each 16:39:32 3 day summit 16:39:39 #action h01ger to set up a doodle poll re Summit #3 dates 16:39:46 somewhere in Europe presumably? 16:39:49 are you aware of any other free software events on these dates? 16:40:09 emaste: yes, something like that, though we thought about Tunesia too 16:40:13 LWN event calendar is good this 16:40:24 lamby: can you check that calendar please? 16:40:51 emaste: also gwolf's offer for mexico city still stands, i assume, but i also assume flying us there will be too expensive. dunno 16:40:59 openstack summit in sydney 2017-11-06 to 08 16:41:16 piggybacking on another event can give some people extra reason to go 16:41:32 last news about the meeting: /me has found some help with organizing it. so you should see a more relaxed and less exhausted h01ger at the meeting 16:42:00 vagrantc: piggypacking on another event sounds like a good idea but in practice has often turned out to be a not so good idea in the end 16:42:00 good 16:42:08 O'Reilly Security Conference clashes (NY) 16:42:29 for the first week 16:42:30 But… even if there were more clashes, probably do the doodle anyway as that will capture the same details. 16:42:35 yeah 16:42:39 yeah 16:43:04 h01ger: curious why those two dates came up? 16:43:15 vagrantc: available in gunner's calendar 16:43:27 got it 16:43:28 not in december, not in the next 3 months, this year 16:43:51 #info Tentative dates are a) Oct 31 → Nov 2, b) Nov 7 → Nov 9. 16:44:02 there's a 3rd fallback date too, but thats in early mid december… 16:44:23 will you put it in doodle poll too? 16:44:31 #info no work has been done on funding the event yet… talk to us if you want to sponsor it 16:44:42 emaste: not sure, probably rather not 16:45:01 Feel like we've got everything captured here. Next topic? :) 16:45:13 why not? if people dont like it, it will not get the most 'yes'es 16:45:14 almost 16:45:23 one thing: you should all attend the summit! :-D 16:45:50 bmwiedemann: because i would rather do it in january than mid december 16:46:11 * h01ger has a date in mid december in leipzig… ;) 16:46:18 hehe 16:46:21 Yes, definitely; if you are remotely interested in Reproducible Builds (ie. you are in this channel) then you _should_ be at this meeting. 16:46:38 totally worth it 16:46:38 Don't think "oh, no, I am not doing / thinking enough…" 16:46:48 #topic Next meeting when? 16:46:49 what lamby says! 16:46:55 when is the next irc meeting? 16:47:07 (this is basically a standard topic for each meeting…) 16:47:18 Thu 2017-07-06 ? 16:47:22 (even if we have a very fixed schedule…) 16:47:28 1st Thursday of July, ie. 6th :) 16:47:38 lamby: didn't you come up with a rough plan for rotating the time? 16:47:40 which time? 16:47:52 I will send out a confirmation of it all after this meeting :) 16:48:20 #info next meeting on the first thursday of the month, that will be july 6th 2017 at a to be decided time 16:48:32 there might be people not subscribed to the rb-general ML 16:48:32 #action lamby will announce the next meeting 16:48:33 Thanks 16:48:43 #topic Any other business 16:48:46 #save 16:48:46 bmwiedemann: you're point being? 16:48:56 they will not know the time then ;-) 16:49:06 we'll also blog it 16:49:08 and tweet it 16:49:11 and /topic it 16:49:12 :-) 16:49:14 fair enough 16:49:31 thanks all 16:49:37 thanks emaste 16:49:38 * h01ger is happy about 13 people being present at this meeting 16:49:53 well, 12, lets not count meetbot ;) 16:50:12 hey, even robots have feelings. 16:50:22 citation needed 16:50:32 marvin 16:50:40 * sangy hastily edits wikipedia's robot entry 16:50:56 any other other business? 16:50:56 * vagrantc tries to think of any outstanding issues that warrant irc discussion... 16:51:09 I got one 16:51:23 I recently stumbled over gcc's profile guided optimisations 16:51:34 * h01ger has submitted a BoF about jenkins.debian.org & its team maintenance for DebConf17 16:51:39 causing non-determinism (e.g. in gcc6 and python for openSUSE) 16:52:15 has anyone else seen such? 16:52:57 I have a quick query 16:52:59 the effect is typically some diffs in assembler, where code gets moved around 16:53:24 are there any up for grabs kinda tasks that I can start looking at to get myself familiar with any codebase, etc.? 16:54:09 sangy: in what direction? working on tools to track rb work or fixing rb issues? or other? 16:54:15 bmwiedemann: once we have shared json and thus suse in our db, we can make the notes cross-distro (see the branch in git for a notes.yaml format discussion there) and then you can documents your findings there and then we can hopefully share problems and solutions better 16:54:31 but first the packages.json, then the notes.yaml… :) 16:54:49 Profile guided optimizations is a losing game 16:54:50 sangy: we have several codebases… 16:55:10 bmwiedemann: mostly the former 16:55:13 (eg diffoscope, the webpages, build_path_prefix fixes in 30 projects or so) 16:55:21 We actually had a sticky note for that very thing on one of our pages about "defining reproducible" at last summit iirc 16:55:27 h01ger: yep, that's right. 16:56:03 Eric[m]: I found that it is possible to do PGO and RB, but it is not so easy. 16:56:06 for Arch Linux, you might want to look at anthraxx's pacman patch and try that? 16:56:26 the trick is to make sure to only have reproducible input to the profiling-run 16:56:32 Profiling to select optimizations: great. Doing it in the middle of the build, thus making the result host and lunar phase dependent: not great. Answer: save your profile results, bless them forever... 16:56:40 Oh can that really be managed? 16:56:44 h01ger: yeah, that's something I'm already looking at 16:56:57 sangy: cool. i was about to give you the urls… :) 16:57:20 Yes ISTM that the profiling data must just become a build input 16:57:36 I'd have thought universe-1.0 insanity like temperatures and stuff would be able to derail it (low probability, but nonetheless probabilistically) 16:57:40 shall we close the meeting and keep the discussion happening as they do? 16:57:45 Eric[m]: I did a fix in https://build.opensuse.org/package/view_file/Base:System/gzip/gzip.spec?expand=1 line 70 16:57:51 +1 to the emaste phrasing there 16:58:35 h01ger: I think so. 16:58:40 sangy: re, the tools to track rb… thats in jenkins.debian.net.git 16:58:51 OK 16:59:00 h01ger: awesome, thanks 16:59:05 h01ger: Yes :) 16:59:05 sangy: look for bin/reproducible_db_maintenance.py and bin/reproducible*html*.py 16:59:09 #endmeeting