Saturday, March 26, 2011

Debian ftpmaster Meeting — Almost over

So since the last progress report I also got round to take a look at the following issues:
  • Ben Hutchings reported that older Lenny CDs do not actually install anymore. As we are still telling people that they don't need to throw away old CDs this was a tad disappointing to me. Got myself a 5.0.0 disc through jigdo and cdimage's fine snapshot service for packages that aren't actually in (old)stable anymore. Using a network mirror during installation is apparently the default (although you can decline it). Doing so will try to go to the security mirror and try to upgrade the base installation it just expanded from the debs on the CD. As the archive signing key changed recently, this leads to apt moaning about untrusted packages, which isn't actually exported to the debian-installer UI. The installation just hangs in "Select and install software" with a prompt being displayed on tty4. Mark Hymers agreed that it makes sense to switch the Release signature for lenny-security back to the old key for the time being and called for objections on debian-devel. Doing so should fix installation for oldstable users with old CDs again. In the future d-i should upgrade debian-archive-keyring first before the security archive's contacted and I've filed a bug report about that (#619751).
  • Closed some old crufty bug reports on the pseudo package buildd.debian.org that were either already solved or easy to solve. (#607619, #602841, #586882, #520479, #507153, #605285)
  • Scripted the generation of keys to get the initial batch of them done. 27 machine keys have been generated so far, with the exception of some machines that are currently down or lack entropy (like the kfreebsd-* ones which aren't fed by the keys). The keys are already activated on the archive side. s390, half of powerpc and some amd64 already run with autosigning turned on. armel will follow tomorrow thanks to Riku Voipio. We'll get more coverage soon, I bet.
  • Fixed up retry as an answer to the build log. There are two possible actions that a buildd admin can do when he received a log: retry and give-back. The former will cause a build retry on the same buildd it was tried before. The latter will give it back to wanna-build to reschedule it. Due to the way retry is implemented you cannot retry binNMUs, though. There's #524547 filed against buildd about that. (There's still no support to schedule a build on a specific set of buildds neither.)
  • Fixed up a typo in the wanna-build trigger for the main archive that caused double builds when we manual re-run it instead of being pushed by the archive.
  • Thought about how to use Packages-arch-specific properly from WB::QD (our Perl-ish quinn-diff replacement). Currently the handling of annotations like "!linux-any" is broken because the logic does not account for architecture wildcards (besides of "any" which is special™). Luckily there's some sort of test suite, which I already adapted locally, but the code isn't really the best to read at the moment. So that's still on my to-do list to solve #603762.
  • There's a post on arch:all autobuilding not happening just yet pending in the queue. I hope to get it out tomorrow or the day after.
It was a productive hacking event for me, that for sure. But now it's almost over and they're actually stealing us an hour tonight. I would've liked to go home with less items on my to-do list, though (i.e. it just grew, it didn't shrink).

Friday, March 25, 2011

Debian ftpmaster Meeting — Autosigning

Proposals for autosigning were floating around for quite some time. The most controversial parts were how we secure the machines that do the building (and in turn: how do we secure the key) and who's going to manage the keyring (because there are multiple teams involved; such discussions can indeed take quite a bit time).

What we've agreed upon now is as follows:
  • All buildds with autosigned must be debian.org machines, which means that they are administrated by the Debian Sysadmins. This involves regular upgrades of the machine, firewalling and monitoring. They are all doing this already (kudos!), so there was no change needed.
  • The machines must be restricted so that only a limited set of people may access it. That's done, it's just the buildd admins for the specific buildd and the global group builddadm (plus the admins and the local admin, of course).
  • Every host has its own GPG key. Key generation locally on the buildds is eased by having the fine Entropy Key hooked up to provide entropy centrally and distributed to the debian.org machines. Private keys don't need to and must not leave the host. The keys are rotated every 120 days, expiry dates will be used to ensure that and to remind us when rotation is due. Sadly there is no HSM involved. Given the geographic dispersiveness of the Debian infrastructure that's not done easily. Some machines (like s390 racks) would also be unable to connect local hardware.
  • The keys are restricted to specific architectures. They are not able to upload any source to the archive. However every binary on the specific architecture can be uploaded using the key, there are no further restrictions on package priorities or the like. They are maintained by the wanna-build admins on ftp-master by signing the armored public keys with their personal GPG keys (instructions available on the list). Thus a trust chain is established: who added or removed which key and when.
  • We plan to suppress networking in the chroot soon. This will possibly be done with unshare(1) in util-linux, as soon as we sort out proper localhost networking. This was not deemed a blocker at this point.
Kudos to Mark Hymers and Joerg Jaspert (both ftpmasters) for implementing the necessary bits on the archive side. It turned out that dak grew support for most bits already in the meantime and it boiled down to sane key management, keyring distribution and setup. sbuild and buildd needed a bit more hackery, but a few patches later it seems to work fine.

So what's the point of this exercise? The main goal is to reduce the build turnaround time. This means cleaning Dependency-Waits and Build-Depends-Uninstallable much more quickly than it used to be. (With a signing run once a day and multiple dependency levels you'd need to wait some days for a leaf package to be buildable again.) This should help speeding up transitions a fair bit. Autosigning also means getting security updates faster, at least if there's a buildd that is not occupied otherwise.

The key generation and configuration deployment will gradually happen in the next days and weeks. It will be used on the regular archive, the security archive and backports (i.e. the archives run by the ftpmasters). As some logs will still need regular signing the deployment cannot happen entirely centralized as the buildd admins need to cope with a new log format. But those steps are tiny given that we can now add keys by ourselves and the archive will even accept them.

Debian ftpmaster Meeting — The wanna-build/buildd part

I've joined the Debian ftpmaster team in the Linuxhotel in Essen-Horst and so far my coding/hacking has been quite productive (it wasn't on dak after all). Linuxhotel has both a nice working and holiday atmosphere. Albeit I'm not taking much time off anyway.
  • Reenabled mipsel d-i autobuilding. (#618989)
  • Added support to filter the buildd overview pages by out-of-date/uncompiled. (#555527)
  • Adapted the wanna-build triggers (i.e. the scripts that import an archive into wanna-build and which are called by dak instances, for instance) to not start processing immediately but flag that a push happened. The real work is then done by a cronjob that loops through the various flags until there's nothing to do anymore. That avoids losing triggers on the way due to locking. (#602841)
  • All buildds (regardless whether they are running lenny or squeeze) are now running sbuild/buildd 0.61.0. Of course there are quite some patches on top of the upstream version. Packages are available in our repository.
  • Autosigning: adjusted buildd to pass a keyid to sbuild and to arrange for the then-signed .changes to be uploaded (configurable per dist in .builddrc); this involved some hackery in sbuild to actually cope correctly with a keyid passed on the CLI and to sign the package at the right time in the build process
  • Updated the unit tests of the build log importer: mocking more objects (especially the PostgreSQL log database; the tests were broken ever since pkg_history was added as a table) and testing that the actual content we write to disk matches up with our expectations
  • Added support for MIME encoded build logs to the build log importer. The log is still transmitted by mail from the buildd to the admins/security team and to the central log host. However it's now gzip-compressed, which shouldn't cause "this mail is too big" bounces anymore and also save some unneeded traffic for our buildd host sponsors. Furthermore .changes files are now attached to the mail instead of placed somewhere within the log, so it's also easier to sign packages without relying on regular expressions identifying the right portion within the log.
  • Added initial support for arch:all autobuilding to the database, wanna-build and buildd. The merging still needs more thinking as the cases in which an arch:all needs to be built still need to be determined. (Also it needs a Packages file for all the arch:all packages in a suite because it's not guaranteed that the newest arch:all is listed in any of the arch-specific Packages files.)
  • Adjusted my own scripts for build processing (which are used by a few others) to at least ignore autosigned logs.  It still needs to grow deMIME abilities, though.
Autosigning will get its own posting later on, unless Joerg gets there first.  There is currently one buildd (zandonai/s390) that has working autosigning for all suites on ftp-master (but not for security, backports or edu).  More will be added in the next days.