Thursday, December 29, 2016

Automating the installation of Debian on z/VM instances

I got tired of manually fetching installation images via either FTP or by manually transferring files to z/VM to test s390x installs. Hence it was about time to automate it. Originally I wanted to instrument an installation via vmcp from another instance on the same host but I figured that I cannot really rely on a secondary instance when I need it and went the s3270/x3270-script way instead.

The resulting script isn't something I'm particularly proud of, especially as it misses error handling that really should be there. But this is not expect  instead you operate on whole screens of data and z/VM is not particularly helpful in telling you that you just completed your logon either. Anyway, it seems to work for me. It downloads the most recent stable or daily image if they are not present yet, uploads them via DFT to CMS and makes sure that the installation does not terminate when the script disconnects. Sadly DFT is pretty slow, so I'm stuck with 70 kB/s and about five minutes of waiting until kernel and initrd are finally uploaded. Given that installations themselves are usually blazingly fast on System z, I'm not too annoyed by that, though.

I previously wrote about a parmfile example that sets enough options to bring debian-installer to the point of a working network console via SSH without further prompting. It's a little unfortunate that s390-netdevice needs to be preseeded with the hardware addresses of the network card in all cases, even if only one is available. I should go and fix that. For now this means that the parmfile will be dependent on the actual VM system definition. With that in mind there is an example script in the same gist that writes out a parmfile and then calls the reinstall script mentioned above. Given that debian-installer now supports HTTPS (so far only in the daily images) you can even do a reasonably secure bootstrapping of the network console credentials and preseeding settings.

If you put this pretty generic preseed configuration file onto a securely accessible webserver and reference it from the parmfile, you can also skip the more tedious questions at the beginning of debian-installer. A secure transport is encouraged as preseed files can do anything to your installation process. Unfortunately it seems that there is no way to preseed SSH keys for the resulting installation yet, neither for the created user nor for root. So I haven't achieved my desired target of a fully automated installation just yet. Debian's Jenkins setup just went with insecure defaults, but given that my sponsored VMs are necessarily connected to the public Internet that seemed like a bad idea to me. I suppose one way out would be to IP/password ACL the preseed file. Another one to somehow get SSH key support into user-setup.

Sunday, October 4, 2015

Root on LVM on Debian s390x, new Hercules

Two s390x changes landed in Debian unstable today:
With this it should be possible to install Debian on s390x with root on LVM. I'd be happy to hear feedback about installations with any configuration, be it root on a single DASD or root on LVM. Unless you set both mirror/udeb/suite and mirror/suite to unstable you'll need to wait until the changes are in testing, though. (The debian-installer build does not matter as zipl-installer is not part of the initrd and sysconfig-hardware is part of the installation.)

Furthermore I uploaded a new version of Hercules - a z/Architecture emulator - to get a few more years of maintenance into Debian. See its upstream changelog for details on the changes (old 3.07 → new 3.11).

At this point qemu at master is also usable for s390x emulation. It is much faster than Hercules, but it uses newfangled I/O subsystems like virtio. Hence we will need to do some more patching to make debian-installer just work. One patch for netcfg is in to support virtio networking correctly, but then it forces the user to configure a DASD. (Which would be as wrong if Fibre Channel were to be used.) In the end qemu and KVM on s390x look so much like a normal x86 VM that we could drop most of the special-casing of s390x (netcfg-static instead of netcfg; network-console instead of using the VM console; DASD configuration instead of simply using virtio-blk devices; I guess we get to keep zIPL for booting).

Saturday, September 19, 2015

Working with z/VM DIRMAINT as a mere user

Two helpful CMS commands to know if the users on the z/VM host you connect to are managed using DIRMAINT:

  • DIRMAINT PW: This will ask you for your current password and then for the new one. This way you can change your own user password instead of having to ask the administrator for it.
  • DIRMAINT REVIEW: If you want to see the directory entry for your user (i.e. the configuration of your VM), you can issue this command. It will put the directory onto the card stack, from which you can retrieve it onto your A disk using RETRIEVE <number> <USERNAME> DIRECT A. The number is listed after RDR FILE in CP's output.
REVIEW is something I always expire from my mind after a few weeks of not using a mainframe. And then I do not usually find it quickly, even knowing where to look. (For some reason GET won't work unless you are a privileged user of the system.)

I guess by now most of the systems use DIRMAINT, but it's an IBM product that requires a separate license in addition to z/VM. Hence on some systems the user directory is still maintained by hand. In this case the password is written verbatim into the file and the administrator needs to change it manually for you.

Sunday, August 30, 2015

Automating the 3270 part of a Debian System z install

If you try to install Debian on System z within z/VM you might be annoyed at the various prompts it shows before it lets you access the network console via SSH. We can do better. From within CMS copy the default EXEC and default PARMFILE:


Now edit DEBAUTO EXEC A and replace the DEBIAN in 'PUNCH PARMFILE DEBIAN * (NOHEADER' with DEBAUTO. This will load the alternate kernel parameters file into the card reader, while still loading the original kernel and initrd files.

Replace PARMFILE DEBAUTO A's content with this (note the 80 character column limit):

ro locale=C                                                              
s390-netdevice/choose_networktype=qeth s390-netdevice/qeth/layer2=true   
netcfg/get_ipaddress=<IPADDR> netcfg/get_netmask=       
netcfg/get_gateway=<GW> netcfg/get_nameservers=<FIRST-DNS>    
netcfg/confirm_static=true netcfg/get_hostname=debian                    

Replace <IPADDR>, <GW>, and <FIRST-DNS> to suit your local network config. You might also need to change the netmask, which I left in for clarity about the format. Adjust the device address of your OSA network card. If it's in layer 3 mode (very likely) you should set layer2=false. Note that mixed case matters, hence you will want to SET CASE MIXED in xedit.

Then there are the two URLs that need to be changed. The authorized_keys_url file contains your SSH public key and is fetched unencrypted and unauthenticated, so be careful what networks you traverse with your request (HTTPS is not supported by debian-installer in Debian).

preseed/url is needed for installation parameters that do not fit the parameters file - there is an upper character limit that's about two lines longer than my example. This is why this example only contains the bare minimum for the network part, everything else goes into this preseeding file. It file can optionally be protected with a MD5 checksum in preseed/url/checksum.

Both URLs need to be very short. I thought that there was a way to specify a line continuation, but in my tests I was unable to produce one. Hence it needs to fit on one line, including the key. You might want to use an IPv4 as the hostname.

To skip the initial boilerplate prompts and to skip straight to the user and disk setup you can use this as preseed.cfg:

d-i debian-installer/locale string en_US
d-i debian-installer/country string US
d-i debian-installer/language string en
d-i time/zone US/Eastern
d-i mirror/country manual
d-i mirror/http/mirror string
d-i mirror/http/directory string /debian
d-i mirror/http/proxy string

I'm relatively certain that the DASD disk setup part cannot be automated yet. But the other bits of the installation should be preseedable just like on non-mainframe hardware.

Wednesday, February 18, 2015

Caveats of the HP MicroServer Gen8

If you intend to buy the HP MicroServer Gen8 as a home server there are a few caveats that I didn't find on the interwebs before I bought the device:
  • Even though the main chassis fan is now fixed in AHCI mode with recent BIOS versions, there is still an annoying PSU fan that's tiny and high frequency. You cannot control it and the PSU seems to be custom-built.
  • The BIOS does not support ACPI S3 (suspend-to-RAM) at all. Apparently it being a server BIOS they chose to not include the code paths in the BIOS needed to properly turn off devices and turn them back on. This means that it's not possible to simply suspend it and have it woken up when your media center boots.
  • In contrast to the older AMD-based MicroServers the Gen8 comes with iLO, which will consume quite a few watts just for being present even if you don't use it. I read figures of about ten watts. It also cannot be turned off, as it does system management like fan control.
  • The HDD cages are not vibration proof or decoupled.
If you try to boot FreeBSD with its zfsloader you will likely need to apply a workaround patch, because the BIOS seems to do something odd. Linux works as expected.

Tuesday, October 14, 2014

pbuilder and pam_tmpdir

It turns out that my recent woes with pbuilder were all due to libpam-tmpdir being installed (at least two old bug reports exist about this issue: #576425 and #725434). I rather like my private temporary directory that cannot be accessed by other (potential) users on the same system. Previously I used a hook to fix this up by ensuring that the directory actually exists in the chroot, but somehow that recently broke.

A rather crude but working solution seems to be "session required user_readenv=1" in /etc/pam.d/sudo and "TMPDIR=/tmp" in /root/.pam_environment. One could probably skip for root, but I did not want to start fighting with pam-auth-update as this is in /etc/pam.d/common-session*.

Update: Roberto C. S├ínchez comments that "export TMPDIR=/tmp" in /root/.pbuilderrc also works.

Friday, November 29, 2013

On PDiffs Usefulness for Stable

Axel just said that PDiffs are really useful for stable because it changes seldom. The truth is that there are no PDiffs for stable. I consider this a bug because you need to download a huge blob on every point release even though only few stanzas changed.

The real problem of PDiffs is the count that needs to be applied. We run "dinstall" much more often now than we used to and we cannot currently skip diffs in the middle when looking at a dak-maintained repository. It would be more useful if you could fetch a single diff from some revision to the current state and apply this instead of those (download, apply, hash check)+ cycles. 56 available index diffs are also not particularly helpful in this regard. They cover 14 days with four dinstalls each, but it takes a very long time to apply them, even on modern machines.

I know that there was ongoing work improving PDiffs that wanted to use git to generate the intermediate diffs. The current algorithm used in dak does not require old versions except one to be kept, but keeping everything is the far other end of the spectrum where you have a git repository that potentially gets huge unless you rewrite it. It should be possible to just implement a rolling ring buffer of Packages files to diff against, reprepro already generates the PDiffs in the right way, the Index format is flexible enough to support this. So someone would just need to make that bit of work in dak.