16:01:00 #startmeeting 16:01:00 Meeting started Tue Jun 19 16:01:00 2018 UTC. The chair is vagrantc. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:01:00 Useful Commands: #action #agreed #help #info #idea #link #topic. 16:01:12 #topic hello 16:01:30 if folks could say a little hello so we know who's here 16:01:38 * yashsriv is here 16:01:42 * sangy waves 16:02:33 let's wait a minute or two, see if others can show up 16:02:41 o/ 16:05:40 alright, i'm guessing that's it! 16:05:51 * sangy shrugs 16:05:59 lamby marked themselves in apologies, so i'm guessing won't make it 16:06:30 a meeting reminder via email 2-3 days in advance is kind of key, sorry i didn't think of it 16:06:46 I kinda rushed it, because I wanted to get back to them sooner rather than later. 16:06:55 #topic progress on debian rebuilder 16:07:07 yashsriv: this is you :) 16:07:08 hi 16:07:10 yashsriv: you have the mic! 16:07:48 so the script I was working on is nearing completion. It manages to take a buildinfo file as input and then perform the complete build 16:08:11 any links to it? 16:08:24 https://salsa.debian.org/yashsriv-guest/debian-rebuilder-setup/blob/integrate-srebuild/builder/srebuild 16:08:51 #link https://salsa.debian.org/yashsriv-guest/debian-rebuilder-setup/blob/integrate-srebuild/builder/srebuild 16:09:13 o/ 16:09:18 I think the idea with this is that any interested orgs can use this ansible playbooks to deploy a re-builder and attest for its reproducibility. 16:09:19 (sorry for the late arrival) 16:09:20 mapreri: o/ 16:09:40 on our side, NYU will deploy a rebuilder and sign any artifacts it reproduces :) 16:09:47 very cool 16:10:21 So the builder was the very first step. The next step is setting up a scheduler to schedule these builds from time to time and an interface for exposing the end results 16:11:02 no meetbot? 16:11:04 yashsriv: do you think it would be trivial enough to have it run pbuilder instead? 16:11:07 jelle: ye there is 16:11:39 oh lol, I'm blind sorry. 16:11:56 If pbuilder supports prebuild hooks, then should be trivial enough. I had looked into both but then theses scripts already existed so I got biased. 16:12:34 it might be good to have the infrastructure reproducing builds in the archive using sbuild, since this is more similar to the official buildd machines 16:12:49 yashsriv: pbuilder supports many more hooks than sbuild, afaik :) https://manpages.debian.org/stretch/pbuilder/pbuilder.8.en.html grep for --hookdir 16:12:58 as well as getting the other main tool used to build packages working with sbuild 16:12:58 but yes, also that 16:13:31 can somebody remind me where's the agenda? 16:13:39 https://pad.riseup.net/p/reproducible-irc-meeting-12 16:13:41 mapreri: ^ 16:13:45 ta 16:13:54 regarding the scheduler, I had a doubt. The buildd servers can be used to schedule for every new package entering if I'm not wrong. 16:14:17 *information from buildd 16:14:17 define "build servers" there 16:14:52 yashsriv: you might need to wait till things land in the archive 16:15:26 yashsriv: as well as we still don't have the actual .buildinfo content exposed anywhere 16:15:33 anywhere public 16:15:48 http://buildinfo.debian.net ?? 16:15:57 the official builds aren't published there 16:16:19 (yet, there is an open bug somewhere) 16:16:30 sure, we'd like to fix that, of course 16:17:01 and yeah, keyword "yet". I'm assuming using buildinfo.debian.net for testing/development is correct? 16:17:01 yashsriv: i wouldn't block your work on that part just yet 16:17:21 sangy: I'd say so, yes 16:17:30 sure, for testing the infrastructure, buildinfo.debian.net should work for now 16:17:50 hell, the very goal of buildinfo.d.n was to expirement in a thing sharing proof of builds (i.e. the .buildinfo) :) 16:18:19 does buildinfo.d.n have a machine readable interface as well? 16:18:22 #link https://bugs.debian.org/763822 16:18:25 so what I was thinking about is that these buildinfo files get pushed out, and they get pulled by (e.g., NYU) to schedule a rebuild locally. Finally they'd push their own buildinfo (and signed metadata) 16:18:33 yashsriv: yes 16:18:38 yashsriv: it has a fair api, yes 16:19:00 #link https://bugs.debian.org/862073 16:19:03 I was thinking for arch of rebuilding based on a package (pkg.tar.xz) the .deb equivalent 16:19:13 well 16:19:31 yashsriv: and if the API is missing anything you need, lamby is generally responsive to bug reports 16:19:35 jelle: I think that can happen with arch, since the .BUILDINFO is packaged in the .tar.xz 16:20:03 sangy: but we can also start publishiing the BUILDINFO, since it contains pkgname, pkgbase and pkgver and pkgarch 16:20:15 yashsriv: the /api/ endpoint only has the submission thing, but there are other urls to get .buildinfo quite programatically. agian, expanding it is not a issue. 16:20:30 jelle: well that'd be nice. We don't have a working archive yet though. (do we want to add that as a meeting topic? :P) 16:20:46 perfect. So now what should we aim to expose once we got something built? As someone mentioned yesterday, BUILDINFO files aren't necessarily exactly the same 16:20:58 sangy: the upside of using a .tar.xz is that we can verify it though. So does debian plan to sign these BUILDINFO files? 16:21:04 so we'd probably need to expose the SHA sums of the end products? 16:21:30 yashsriv: I'm biased, but in-toto link metadata would tie things very nicely in this case 16:21:31 jelle: buildinfo files should always be signed 16:21:40 ok :) 16:22:31 yashsriv: I'd just POST your signed buildinfo to buildinfo.d.n and have it deal with the next steps 16:22:47 buildinfo.d.n already has a concept of "reproducible" and "unreproducible", although sometimes not perfect 16:24:04 well, I guess that settles it? action point -> try to integrate with pbuilder for scheduling etc? 16:24:10 the main problem being is it has know way to know which .buildinfo's should reasonably know which other .buildinfo files should be compared for reproducibility 16:24:43 sangy: wait ... where did that come from? :) 16:24:46 sangy: integrate with pbuilder was more so like I can use it without learning another tool :3 but from what I saw of that script it shouldn't be hard to do at all 16:25:01 oh ok ok :P 16:25:05 adding a scheduler, well… that's another point. 16:25:06 vagrantc: from there 16:25:33 the intention of this is something entirely independent from jenkins.debian.net ? 16:25:38 vagrantc: re: how to know which .buildinfo files to correlate. I *do* ferviently believe that in-toto link metadata would make it easier 16:25:42 er, tests.reproducible-builds.org 16:25:55 vagrantc: that's what h0lger intended, yes. As it'd be more lightweight and one-click-deployment for interersted organizations 16:26:10 yeah, i think this came out of discussions at the last summit 16:26:18 tbh, I've been hacking around the t.r-b.o infra in the recent past, I feel like I'm near enough to make it less jenkins-y and more non-debian-friendly, which could easily include integrate it with the rebuilder. Alas, not so easy this last part. 16:27:19 while i see the advantages of everything all being in one place, i think it's best to not have to integrate everything in one place to experiment with a few different methods 16:27:23 but if it lives by itself, it can always be plugged in anywhere, so :) 16:27:38 ack 16:27:45 especially at this point 16:27:58 so i like the idea of something lightweight and "simple" :) 16:28:42 so, do we have any actionable items for this topic? any further things that need discussion? 16:29:11 jelle: did you want to bring up the arch-archive stuff or hosting the buildinfo's somewhere? 16:29:17 jelle: and did you want to add some talk of an archlinux rebuilder as a separate agenda item, or did you get enough from the discussion so far? 16:30:32 vagrantc: we can discuss the arch one, so far the 'my plan' is to make users able to verify packages 16:30:56 * mapreri has to go afk, sorry for leaving unexpectedly! 16:30:58 shall we move on to that as a topic? 16:31:05 sure 16:31:17 #topic verifying archlinux packages 16:31:34 jelle: there was a plan to have .BUILDINFO files be hosted on archweb no? 16:31:48 also, if anyone has any additional items for the agenda, just let me know, we've got very little scheduled on the agenda 16:32:23 might be good to give a little status update, we now can create reproducible packages with the latest released pacman. This also includes an updated BUILDINFO file. We have worked on creating a script to not remove packages which are mentioned in a BUILDINFO file in our repo, since our archive doesn't have infinite space. 16:32:34 This has led to a rebuild of old packages without BUILDINFO files! 16:32:58 sangy: we could do that, but archweb would provide BUILDINFO files unsigned, which then might defeat the point? 16:33:19 jelle: I don't see why can't the developer sign the buildinfo files? 16:34:26 sangy: well, archweb would read updated .pkg.tar.xz's and extract BUILDINFO files 16:34:55 right, which could be signed by develoeprs 16:35:02 the ideal for .buildinfo files is that multiple independent parties sign separate .buildinfo files which can then be used to corroborate each other 16:36:01 hmmmm 16:36:30 (or raise questions about the results, if they don't match) 16:36:38 hehe 16:37:14 jelle: so is the reason they aren't being signed right now is becuase they are embedded in the .tar.xz which is signed? 16:37:39 sangy: yes, well and they aren't separate files only included in our packages 16:38:02 on archlinux.org we could extract them and put them on our website, but then they aren't signed. 16:38:06 jelle: the .pkg.tar.xz includes what, typically? (sorry for my arch ignorance) 16:38:27 vagrantc: the actualy package data, an some metadata 16:38:46 vagrantc: the pkg itself (data), a .SRCINFO file and an .MTREE file, now there's a .BUILDINFO file too 16:39:08 (not so sure about the .SRCINFO actually) 16:39:27 hm. embedding the .buildinfo means you actually have to extract the archive to verify the checksums on the rest of it, then? 16:39:50 yes 16:40:16 the argument for that is that you want to check against it anyway 16:40:54 makes bit-for-bit reproducibility of the .pkg.tar.xz basically impossible... 16:41:36 on the plus side, you can actually distribute the .buildinfo that you're actually using for that particular build trivially 16:42:52 or rather, the .buildinfo used to produce the rest of the .pkg.tar.xz 16:43:58 yup 16:44:03 so, you could distribute third-party .buildinfo files through some other out-of-band mechanism, which are signed 16:44:31 that'd be ideal. And having the original buildinfo signed by the packager would also be good 16:44:33 vagrantc: yup that is still possible 16:44:46 (late 'hello' ;) ) 16:44:53 btw, we could change our upload process, that just requires more work :) 16:45:08 personally I think it might be a bit strange to put it unsigned on our website. 16:45:22 yeah, i wouldn't bother extracting an unsigned .buildinfo 16:45:32 jelle: I still don't see why can't we sign it before putting it in the .tar.xz for the time being 16:45:56 do the uploaders upload source-only? or do they upload binaries? 16:45:57 sangy: how would you reproduce it then ;-) 16:46:02 vagrantc: we upload binaries 16:46:09 oh. 16:46:15 we sign the .pkg.tar.xz locally, sign it and upload it 16:46:20 then they *definitely* should sign them! :) 16:46:56 well yeah it can be done, just requires changes and extracting them from the .pkg.tar.xz :) 16:47:47 jelle: I'm down for working on that, but I have a lot in my plate right now :S. Any updates on the archive? 16:48:04 my focus right now, is getting a script together so users can reproduce based on a .pkg.tar.xz and then later we can start thinking about publishing :) 16:48:33 yeah, I think without archive we can't really reproduce anything :P 16:48:36 sangy: ah, yes, our archive needs fixing, since now we archive once a day and might miss packages depending on a package which was updated in that timeslot. So twice on the same day. 16:48:46 sangy: well we have the archive, just one small issue. 16:49:47 ack, nice :) 16:50:21 anthraxx: created a script to remove old packages which are not referenced in any of our repo packages 16:51:29 if we want to re-produce every package we ever had in our repository we need more space :) 16:51:36 yeah I remember the discussion about those spureous mylinuxkeyring packages 16:51:55 and yeah :[ 16:52:38 yup, we also had packages with a BUILDINFO referencing packages which are not in our repo!!!!! This is still an issue which we need to resolve, basically disallowing such an upload 16:53:08 right. I mean that's the nice thing about this, it catches stuff like this :) 16:54:11 I'm not too up to speed with the state of things, would that archive also be the place for third parties to upload their signed buildinfo files (for cross-checking)? 16:54:15 well, so I don't know. I guess action point -> sign the .BUILDINFO? :D 16:54:31 raboof[m]: no, I think that'd be another platform (similar to buildinfo.d.n) 16:54:54 raboof[m]: no, it's an archive of our old packages required to reproduce the repository packages (since rolling release) 16:55:00 that sounds like a sensible thing to do in any case, looking at the above, yes 16:56:59 due to time, maybe we've had enough on this topic for the meeting? can keep discussing later, of course :) 16:57:38 sure! 16:57:56 #topic next meeting 16:58:01 I might be up for collaborating on that repo for signed buildinfo files (either based on http://buildinfo.debian.net/ or a standalone thing) - though I have limited time as well :) 16:58:06 so, so we have a date for the next meeting? 16:58:33 and also, maybe someone to commit to sending reminder emails 2-3 days in advance? 16:58:45 vagrantc: I can do that 16:59:07 vagrantc: and should we schedule same tuesday a month from now? I'd like lamby and h01ger to be able to make it though... 17:00:11 can't speak to their availability, but otherwise same time and day of week a month from now seems reasonable 17:01:04 ok, should we set that date tentatively then? 17:01:22 i think so, and leave it open for revolt :) 17:01:46 then let's do Tuesday 17th of July on 16:00 UTC :D 17:01:48 looks like the 17th? 16UTC ? 17:02:06 perfect 17:02:44 +1 17:02:52 👍 17:03:09 #agreed tentative date for next meeting 17th of July, 16:00 UTC. 17:03:10 I'll send the reminder on the 26th then. Possibly another one the week after 17:03:26 #commit sangy to send reminders 17:03:54 ok, thanks all! 17:03:56 #endmeeting