Tuesday, December 6, 2011

New Debian buildd at Karlsruhe Institute of Technology

It took quite a lot of effort to persuade all decision makers to make this happen, but here it is: A new Debian buildd is being hosted at Karlsruhe Institute of Technology, to support the s390(x) ports. Its name is zemlinsky. So we've got some redundancy now and despite them being some sort of fringe architectures, they're looking pretty good. s390x is currently bootstrapped in the archive and it's progressing pretty quick. This new fast builder is one of the reasons why the slope is so steep.

Pointing people at the Debian Machine Usage Policies (DMUP) is pretty helpful to get a consent, with relation to network usage and acceptable use of the machines themselves. In this case the hardest part was drafting a user agreement that allows other non-university persons to log into the box, which is crucial to have it maintained by the Debian System Administrators.

Thanks to all the people at IPD Reussner, Steinbuch Centre for Computing and BelWü who helped me getting this done.

Wednesday, November 30, 2011

How to install Debian within z/VM with just x3270

If you want to copy debian-installer for System z onto a z/VM user's CMS disk, you don't need access to FTP (and hence the host's TCP/IP stack). You can just use x3270 and transfer files with it. For odd reasons I forgot about this, so let's document it here:
  • Figure out which CMS disk is actually writeable for you (most likely the A disk) by issuing Q DISK and looking for "R/W" in the STAT column.
  • Select File, File Transfer.
  • Tick "Send to host" and "Host is VM/CMS".
  • Download the four installer files to your local machine: debian.exec (a startup script that punches the kernel, boot parameters and initrd onto cards and executes the card deck), kernel.debian, parmfile.debian (the boot parameters) and initrd.debian.
  • Transfer debian.exec and parmfile.debian with "Transfer ASCII file" ticked and Record Format set to "Variable" ("Add/remove CR at end of line" and "Remap ASCII characters" should both be ticked by default; LRECL and BLKSIZE should both be empty.) The host filename is the CMS filename, so something like "DEBIAN EXEC A" (with the spaces, and A replaces with the letter of the writeable disk of your choice). Both files must be prefectly readable in XEDIT (i.e. properly converted to EBCDIC).
  • Then transfer kernel.debian and initrd.debian with "Transfer binary file" ticked and Record Format set to "Fixed" with LRECL = 80. This may take a while.
  • Boot up the installer by issuing DEBIAN to z/VM, starting the script. If you needed to rename the files above (the drive letter doesn't matter), you need to adjust DEBIAN EXEC first.
  • You should see Linux kernel messages now. Profit.

Wednesday, November 23, 2011

Useful Firefox extensions (followup)

Since my last post about Firefox extensions I've enabled two other addons:

Through the comments I got pointed to Fox to Phone which enables you to send links from your browser directly to your Android phone with Chrome to Phone installed. Thanks for that.

Another useful extension that was recommended to me is LeechBlock. You give it a list of news sites you regularly frequent and it will make sure that you only spend a given time budget on them per day or that you only browse them in the evenings (or even a combination of both).

As I expected I did deactivate RequestPolicy again. That said, Facebook recently switched its certificates, so Certificate Patrol was unhappy. It's impressive and sad how many pages actually do cross-site requests to embed Facebook's buttons. If somebody would invent something less annoying to stop this mess, that would be great.

Thursday, November 3, 2011

PAV on Linux on System z

There are various presentations that state the goodness of PAV on Linux. Most revolve around using multipath-tools to assemble a volume if you don't have HyperPAV. But it turns out that the DASD device driver does multipathing for them internally in current kernels (which includes the squeeze kernel).

So all you need to do is setting those alias devices online. When you do that the kernel will log that it detected a new device, but you'll find that it won't create any dasd* device nodes for them, nor will it list partitions. lsdasd will only show you "alias" without mentioning the base volume, but you can fetch that information easily from the uid sysfs entry.

Tuesday, November 1, 2011

Useful Firefox extensions

Many people around me switched to Chrome or Chromium. I also used it for a bit, but I was a bit disappointed about the extensions available. To show why, here's a list of the extensions I've currently installed:
  • Adblock Plus: I guess everyone knows this. It sits quitely in the background and removes quite a bunch of eye-distracting stuff from web pages. You see the web differently and are always confused when viewing pages on your smartphone.
  • British English Dictionary: Pretty self-explaining. Firefox has an integrated spell checker and that one needs a dictionary.
  • BugMeNot: You can right-click on a login field and let it insert BugMeNot data automatically. I don't use it often enough to remember that it's there.
  • Certificate Patrol: SSL's X.509 trust model is weak, to say the least. This extension implements the "save on first visit" trust model and warns you if the certificate or the CA of a URL changes.
  • Download Statusbar: If you're used from, say, Chromium to see your download progress above the status bar, this extension will give you that. I don't like the separate window, especially with Awesome as my window manager.
  • Firebug: Most people know this, too. Very useful when you do web development. You can see the effects of CSS as you type, for instance.
  • Flashblock: YouTube has its HTML5 trial, so you don't need Flash for it. Sadly I had to give in recently (for live streaming sites like the German Parliament, go figure) and have it installed for "emergency" cases. But really nobody needs Flash advertisements or other silly Flash animations, so Flashblock will conveniently refuse to load the Flash plugin unless you tell it to.
  • German Dictionary: As above.
  • Google Translator for Firefox: Translates marked sections on a web page in place. It lets Google guess the source language, so you really only mark the area, click the button and be done.
  • Greasemonkey: Modifies web pages in place to make them more sane, using little scripts. I use two currently:
    • Better Outlook Web Access (OWA): For odd reasons I'm forced to use OWA 2007 at a company of Windows and Mac users. This makes it a little bit more bearable, by allowing you to save your password and by adding a message preview pane. You really don't want to use OWA Light 2007 without this.
    • Tagesschau.de - video tag: A simple script that lets you watch videos on tagesschau.de without resorting to Flash. They ship Theora videos alongside H.264, which is supported by Firefox out of the box.
  • Lazarus: Form Recovery: This already saved me many, many times. Sometimes I hit the stupid Thinkpad page back/page forward keys, or the browser crashes or I'm torn away from a page in another way. This extension keeps your form content mostly save, so that you don't need to start from scratch. A must have.
  • Live HTTP headers: curl -D usually works. But sometimes you need authentication and cookies and stuff, hence doing it in the browser makes sort of sense.
  • Modify Headers: I don't really use it anymore. There was this legend that YouTube lets you watch videos that are blocked in your country (Hello, GEMA, I'm looking at you!) if you provide a X-Forwarded-For header with an IP from another country. Never worked for me.
  • Mozilla Archive Format: If you need to archive a web page or want to collect a bunch of pages for offline reading, this is the way to go. You can conveniently select either single pages or several tabs to be stored.
  • Perspectives: I'm paranoid, so here's the second SSL certificate check. Perspectives uses a bunch of network notaries hosted on PlanetLab to check if everyone sees the same certificate. Together with Certificate Patrol that means if that you save a self-signed certificate on first visit if the notaries all agree that it's the currently installed one.
  • PwdHash: Password re-use on different web sites is bad. There are sites where I'm not very concerned about the strength of my password, but where I don't want to leak my main ones. PwdHash uses the domain name and the password I give as components to a hash function. So if I'm at a computer without access to my saved passwords, I can easily reconstruct the hash. If need be, I can use the JavaScript on the developer's web site to do that.
  • RequestPolicy: It's currently installed but I don't think it will stick around much longer. Too many sites are using embedded cross-site requests, which this extension will allow you to review. As an example every Blogger site that's on a custom domain (like my blog) will trigger it. Instead half of your web will have red flags everywhere full of blocked requests. Not that helpful. Interestingly enough it also polices Flash's outgoing web requests. Which breaks even more often than normal web pages.
  • Secure Login: This gives you a button (or a keyboard shortcut) that allows you to choose from several logins and then goes on and posts those credentials securely to the page in question, bypassing any JavaScript that might want to see them. You also get auto-login bookmarks for those sites which don't allow you to remain logged-in across browser sessions. (Like OWA.)
If Firefox on Android were quicker to start and faster overall, I might even use it there. But as-is it's not very useful. Sadly this also means that I can't use Firefox Sync on my phone and as I don't use Chrome on my desktop I also can't use Chrome to Phone. So I usually go and build a QR code on my laptop and read that with Android's Barcode Scanner.

Of course I'm actually using Iceweasel and I'm very grateful for Mike Hommey's efforts to track the release channel on mozilla.debian.net.

Tuesday, October 4, 2011

Call for testing: Upcoming Squeeze point release 6.0.3

There's a new call for testing for the next point release of Debian Squeeze. Please test the packages in squeeze-proposed-updates on some stable machines if possible, so that we don't screw up your production machines with bad updates in a week. The point release is scheduled for October 8th, i.e. next Saturday. Don't forget to copy the debian-release mailing list when you encounter regressions. Thanks for your efforts.

If you want to receive these notices by mail, please subscribe to the debian-stable-announce mailing list.

Sunday, September 18, 2011

python-gnucash, historic build stats

Two tiny bits:

Tuesday, August 9, 2011

DebConf11: Gobby documents

If you still want to grab documents that used to be on gobby.debian.net:
  • A Gobby server is back at that address. However its SSL certificate is issued on the name of gobby.0x539.de, which is upstream's public Gobby server. So you can just accept the certificate and then access the documents in the "debconf11" folder.
  • I put up a tarball of all resulting plain-text documents. If you miss some content in one of them, please don't hesitate to contact me. I might be able to restore it from the records that dkg sent me before shredding the server.

Sunday, August 7, 2011

Debian s390: channel numbers and consoles

Three things I learned about Debian s390 today:
  • The kernel expects hexadecimal channel numbers in lower case. Trying uppercase digits is futile.
  • To get a getty in the HMC's Operating System Messages window of an LPAR, just uncomment the dumb console entry in /etc/inittab.
  • For the integrated ASCII console to work on LPAR, you need to put up a getty onto ttysclp0. You can reuse the same parameters as for ttyS0 (the device that just works with z/VM). Unlike the 3270 interface of z/VM the integrated ASCII console is actually pretty nice and usable. You can even run vim in it without getting completely crazy.

Thursday, July 28, 2011

caff harmful unless you know what you're doing

So there are two things I stumbled upon with caff:
  • If you have two keys, you want to set $CONFIG{'local-user'} to the content of $CONFIG{'keyid'}. For some reason unbeknownst to me this option is not even listed in the configuration file template. keyid does something different that you'd expect…
  • More importantly it uses its own gpg.conf for whatever reason (probably because it sets its own GnuPG homedir and does not override the configuration file location). So if you, like me, put the right settings for strong signatures into ~/.gnupg/gpg.conf, you need to replicate them into ~/.caff/gnupghome/gpg.conf.
Thanks to Tom Marble for the hint. I'm still sad that I'd basically need to re-do yesterday's keysigning (which was about 100 e-mails), just to switch from the default SHA1 to SHA256…

Monday, June 27, 2011

YouTube serving its content over IPv6

In the aftermath of the World IPv6 Day YouTube seems to be serving its content over IPv6 now. Interestingly the frontpage is still served via IPv4 (if you're not in a Google IPv6 whitelisted network). But all the Flash and HTML5 video content is served through IPv6 if available, as the cache servers return proper AAAA DNS records. Apparently that's the case unless your network is blacklisted because of bad IPv6 support and even if Google has some caches at your provider's site (which is the case for Alice DSL in Germany, at least).

I think that's quite some motivation for the providers to at least fix IPv6 connectivity if available and to suppress rogue IPv6 router advertisements in their networks. I had to ensure the former today and the latter is a constant source of grief with the bulk of L2 switches and Wi-Fi access points not being IPv6 ready.

Saturday, June 25, 2011

Porting a library to gtk3: change soname

Last week I tried switching a library to Gtk3. The needed changes to the code are available through --with-gtk3. However this is generally not enough. Even if your symbol list doesn't change, the ABI changes implicitly. The library in question had a .symbols file, but that's not enough because the resulting GUI application will bail out at runtime if symbols of both Gtk2 and Gtk3 are found in the same address space. That's mostly because C symbols don't contain any signatures with return types and parameters.

So if your library upstream did not change the soname for the Gtk3 build, please encourage them to do so. Also keep in mind that this most likely means new pkg-config files specific to the Gtk3 build, too. At least if you want your reverse-depends to be able to build against either Gtk2 or Gtk3 in a predictable way.

An example is this change to gtk-vnc, which uses gtk-vnc-2.0 as the new API/pkg-config name for the Gtk3 build, gtk-vnc-1.0 remains the old Gtk2 one. The soname changes from libgtk-vnc-1.0.so.0 to libgtk-vnc-2.0.so.0.

(Thanks to Michael Biebl and Julien Cristau for pointing out the obvious to me.)

Friday, June 24, 2011

I'm going to DebConf 11

I finally got around to book my (train) trip back from Zagreb. The hotline of Deutsche Bahn was… interesting. The booking system crashed and I was called back. But in the end it worked and I can print it the next time I get near a vending machine.

I'm worried how I manage to get there from Banja Luka on my own. And the travel to Zagreb together with Joachim Breitner won't be as comfortable as it should be. But meh, I'm going to DebConf again! \o/

Monday, June 20, 2011

Call for testing: Upcoming Squeeze point release 6.0.2

I just posted a call for testing to the not yet well-known debian-stable-announce mailing list. Please test the packages in squeeze-proposed-updates on some stable machines if possible, so that we don't screw up your production machines with bad updates in a week. The point release is scheduled for June 25th, i.e. next Saturday. Don't forget to copy the debian-release mailing list when you encounter regressions. Thanks for your efforts.

Tuesday, June 14, 2011

About versions

So Christian likes to give out awards for the best bug reporting estimates and also does statistics about developers per capita.  I've got at least one area where he's on top:

The award for the introduction of the highest version into the archive goes to Nicolas Spalinger and Christian Perrier for ttf-sil-gentium. The use of a date as an epoch is amazing. The runner-up is Joey Hess with intercal (soon to be gone from Debian altogether), reusing the version number in the epoch. Somehow that fits with the crazy language the package contains.

The award for the most minimal version goes to Guido Günther with libvirt-glib. It's a number less than zero but still not negative. The runner-up is Raphael Geissert with switchsh which just happens to use 2007 as a checkout date.

Saturday, March 26, 2011

Debian ftpmaster Meeting — Almost over

So since the last progress report I also got round to take a look at the following issues:
  • Ben Hutchings reported that older Lenny CDs do not actually install anymore. As we are still telling people that they don't need to throw away old CDs this was a tad disappointing to me. Got myself a 5.0.0 disc through jigdo and cdimage's fine snapshot service for packages that aren't actually in (old)stable anymore. Using a network mirror during installation is apparently the default (although you can decline it). Doing so will try to go to the security mirror and try to upgrade the base installation it just expanded from the debs on the CD. As the archive signing key changed recently, this leads to apt moaning about untrusted packages, which isn't actually exported to the debian-installer UI. The installation just hangs in "Select and install software" with a prompt being displayed on tty4. Mark Hymers agreed that it makes sense to switch the Release signature for lenny-security back to the old key for the time being and called for objections on debian-devel. Doing so should fix installation for oldstable users with old CDs again. In the future d-i should upgrade debian-archive-keyring first before the security archive's contacted and I've filed a bug report about that (#619751).
  • Closed some old crufty bug reports on the pseudo package buildd.debian.org that were either already solved or easy to solve. (#607619, #602841, #586882, #520479, #507153, #605285)
  • Scripted the generation of keys to get the initial batch of them done. 27 machine keys have been generated so far, with the exception of some machines that are currently down or lack entropy (like the kfreebsd-* ones which aren't fed by the keys). The keys are already activated on the archive side. s390, half of powerpc and some amd64 already run with autosigning turned on. armel will follow tomorrow thanks to Riku Voipio. We'll get more coverage soon, I bet.
  • Fixed up retry as an answer to the build log. There are two possible actions that a buildd admin can do when he received a log: retry and give-back. The former will cause a build retry on the same buildd it was tried before. The latter will give it back to wanna-build to reschedule it. Due to the way retry is implemented you cannot retry binNMUs, though. There's #524547 filed against buildd about that. (There's still no support to schedule a build on a specific set of buildds neither.)
  • Fixed up a typo in the wanna-build trigger for the main archive that caused double builds when we manual re-run it instead of being pushed by the archive.
  • Thought about how to use Packages-arch-specific properly from WB::QD (our Perl-ish quinn-diff replacement). Currently the handling of annotations like "!linux-any" is broken because the logic does not account for architecture wildcards (besides of "any" which is special™). Luckily there's some sort of test suite, which I already adapted locally, but the code isn't really the best to read at the moment. So that's still on my to-do list to solve #603762.
  • There's a post on arch:all autobuilding not happening just yet pending in the queue. I hope to get it out tomorrow or the day after.
It was a productive hacking event for me, that for sure. But now it's almost over and they're actually stealing us an hour tonight. I would've liked to go home with less items on my to-do list, though (i.e. it just grew, it didn't shrink).

Friday, March 25, 2011

Debian ftpmaster Meeting — Autosigning

Proposals for autosigning were floating around for quite some time. The most controversial parts were how we secure the machines that do the building (and in turn: how do we secure the key) and who's going to manage the keyring (because there are multiple teams involved; such discussions can indeed take quite a bit time).

What we've agreed upon now is as follows:
  • All buildds with autosigned must be debian.org machines, which means that they are administrated by the Debian Sysadmins. This involves regular upgrades of the machine, firewalling and monitoring. They are all doing this already (kudos!), so there was no change needed.
  • The machines must be restricted so that only a limited set of people may access it. That's done, it's just the buildd admins for the specific buildd and the global group builddadm (plus the admins and the local admin, of course).
  • Every host has its own GPG key. Key generation locally on the buildds is eased by having the fine Entropy Key hooked up to provide entropy centrally and distributed to the debian.org machines. Private keys don't need to and must not leave the host. The keys are rotated every 120 days, expiry dates will be used to ensure that and to remind us when rotation is due. Sadly there is no HSM involved. Given the geographic dispersiveness of the Debian infrastructure that's not done easily. Some machines (like s390 racks) would also be unable to connect local hardware.
  • The keys are restricted to specific architectures. They are not able to upload any source to the archive. However every binary on the specific architecture can be uploaded using the key, there are no further restrictions on package priorities or the like. They are maintained by the wanna-build admins on ftp-master by signing the armored public keys with their personal GPG keys (instructions available on the list). Thus a trust chain is established: who added or removed which key and when.
  • We plan to suppress networking in the chroot soon. This will possibly be done with unshare(1) in util-linux, as soon as we sort out proper localhost networking. This was not deemed a blocker at this point.
Kudos to Mark Hymers and Joerg Jaspert (both ftpmasters) for implementing the necessary bits on the archive side. It turned out that dak grew support for most bits already in the meantime and it boiled down to sane key management, keyring distribution and setup. sbuild and buildd needed a bit more hackery, but a few patches later it seems to work fine.

So what's the point of this exercise? The main goal is to reduce the build turnaround time. This means cleaning Dependency-Waits and Build-Depends-Uninstallable much more quickly than it used to be. (With a signing run once a day and multiple dependency levels you'd need to wait some days for a leaf package to be buildable again.) This should help speeding up transitions a fair bit. Autosigning also means getting security updates faster, at least if there's a buildd that is not occupied otherwise.

The key generation and configuration deployment will gradually happen in the next days and weeks. It will be used on the regular archive, the security archive and backports (i.e. the archives run by the ftpmasters). As some logs will still need regular signing the deployment cannot happen entirely centralized as the buildd admins need to cope with a new log format. But those steps are tiny given that we can now add keys by ourselves and the archive will even accept them.

Debian ftpmaster Meeting — The wanna-build/buildd part

I've joined the Debian ftpmaster team in the Linuxhotel in Essen-Horst and so far my coding/hacking has been quite productive (it wasn't on dak after all). Linuxhotel has both a nice working and holiday atmosphere. Albeit I'm not taking much time off anyway.
  • Reenabled mipsel d-i autobuilding. (#618989)
  • Added support to filter the buildd overview pages by out-of-date/uncompiled. (#555527)
  • Adapted the wanna-build triggers (i.e. the scripts that import an archive into wanna-build and which are called by dak instances, for instance) to not start processing immediately but flag that a push happened. The real work is then done by a cronjob that loops through the various flags until there's nothing to do anymore. That avoids losing triggers on the way due to locking. (#602841)
  • All buildds (regardless whether they are running lenny or squeeze) are now running sbuild/buildd 0.61.0. Of course there are quite some patches on top of the upstream version. Packages are available in our repository.
  • Autosigning: adjusted buildd to pass a keyid to sbuild and to arrange for the then-signed .changes to be uploaded (configurable per dist in .builddrc); this involved some hackery in sbuild to actually cope correctly with a keyid passed on the CLI and to sign the package at the right time in the build process
  • Updated the unit tests of the build log importer: mocking more objects (especially the PostgreSQL log database; the tests were broken ever since pkg_history was added as a table) and testing that the actual content we write to disk matches up with our expectations
  • Added support for MIME encoded build logs to the build log importer. The log is still transmitted by mail from the buildd to the admins/security team and to the central log host. However it's now gzip-compressed, which shouldn't cause "this mail is too big" bounces anymore and also save some unneeded traffic for our buildd host sponsors. Furthermore .changes files are now attached to the mail instead of placed somewhere within the log, so it's also easier to sign packages without relying on regular expressions identifying the right portion within the log.
  • Added initial support for arch:all autobuilding to the database, wanna-build and buildd. The merging still needs more thinking as the cases in which an arch:all needs to be built still need to be determined. (Also it needs a Packages file for all the arch:all packages in a suite because it's not guaranteed that the newest arch:all is listed in any of the arch-specific Packages files.)
  • Adjusted my own scripts for build processing (which are used by a few others) to at least ignore autosigned logs.  It still needs to grow deMIME abilities, though.
Autosigning will get its own posting later on, unless Joerg gets there first.  There is currently one buildd (zandonai/s390) that has working autosigning for all suites on ftp-master (but not for security, backports or edu).  More will be added in the next days.