aii-pxelinux: add support for Grub-based UEFI PXE boot loader #216

jouvin · 2016-10-28T11:19:42Z

UEFI requires a PXE loader different from pxelinux.0. One possible loader is based on Grub. RH provides it as part of the OS. On EL6, it installed in /boot/efi/EFI/redhat/grub.efi and is based on Legacy Grub. In EL7, it is provided by the package grub2-efi and is based on Grub2 (this is a set of files that must be copied on the TFTP server which doesn't need to run EL7). As Legacy Grub is no lo longer maintained, the proposal is to support the Grub2-based UEFI boot loader (below in the description, Grub will refer to Grub2).

The Grub-based PXE loader has two differences with respect to its configuration comparted to pxelinux.0:

PXE configuration files must be located in the same directories as the loader, not in pxelinux.cfg subdirectory. This is easily fixed by putting the loader in the pxelinux.cfg directory and is transparent to Quattor.
PXE configuration file is a standard Grub2 configuration file (the loader is a Grub shell in fact) which differs slightly from the pxelinux.0 (SYSLINUX) configuration file (but the name based on the host address in hexa remains the same).

As a result, aii-pxelinux must be updated to process differently a host using UEFI or BIOS. Rather than adding a property in the host profile to say if it using UEFI or BIOS (with the risk of being unsynchronized with the actual setting on the machine), as suggested by @jrha in #216 (comment), the idea is to have aii-pxelinux generating both the Grub configuration used by UEFI and the PXELINUX configuration for each host in different directories (for exampleefi.cfg and pxelinux.cfg). The DHCP configuration will define which path must be used based on the DHCParch option and the PXE loader used (see http://www.syslinux.org/wiki/index.php?title=PXELINUX#UEFI for details). The actual configuration directory to use for each variant will be the directory where the Grub-based loader is installed and pxelinux.cfg (or whatever is set by DHCP option pxelinux.pathprefix) respectively.

Things to do in aii-pxelinux are:

Add an option nbpdir_grub2_efi to define where the Grub config files must be created.
Generate Grub config file for the installation instead of the SYSLINUX one. A typicial file should be:

set default=0
# Setting a small non zero value may give a chance to get a Grub shell for debugging if needed
set timeout=2
menuentry "Install Scientific Linux 6x (x86_64)" {
    set root=(pxe)
    linuxefi /quattor/centos72_x86_64/vmlinuz ks=http://quattorsrv.lal.in2p3.fr/ks/grid304.lal.in2p3.fr.ks ksdevice=em1
    initrdefi /quattor/centos72_x86_64/initrd.img
}

In the Grub config file, (pxe) refers to the TFTPD root (generally /tftpboot. Every absolute path given in Grub commands are in fact relative to this path.

Link to a localboot.cfg file for booting from local disk (aii-shellfe --boot). This Grub config file must typically contains:

set default=0
# Setting a small non zero value may give a chance to get a Grub shell for debugging if needed
set timeout=2

menuentry "Local Disk" {
    exit
}

Add a configuration option grub2_efi_kernel_root. In the Grub config file used for the installation, note that the kernel/initd path is different from the PXELINUX one, as it has to include every element under /tftpboot. There is no easy and general way to compute it from another configuration option. A reasonable default value could be npbdir with the first (generally /tftpboot) and last element (pxelinux.cfg) removed.
Optionally, add a configuration option grub_efi_linux_cmd to address aii-pxelinux: add support for Grub-based UEFI PXE boot loader #216 (comment). This will allow to tell AII which Grub command to use instead of linuxefi/initrdefi to load the installation kernel and will let the site decide whether it wants to use linux/initrd or linux16/initrd16 (initrd command to use may be decuded from linux command).

Contributes to #97

The text was updated successfully, but these errors were encountered:

jrha · 2016-10-28T11:48:20Z

We started trying to support EFI network boot at RAL, but have put it to one side for now.
Our boot-servers use the following structure:

/osinstall/nbp/
/osinstall/nbp/bios
/osinstall/nbp/bios/pxelinux.0
/osinstall/nbp/payload
/osinstall/nbp/payload/pxelinux.cfg
/osinstall/nbp/efi32
/osinstall/nbp/efi32/syslinux.efi
/osinstall/nbp/efi64/
/osinstall/nbp/efi64/bootx64.efi
/osinstall/nbp/efi64/syslinux.efi

Then in dhcpd.conf:

shared-network "Example Network" {
    if option architecture-type = 00:00 {
        filename "/bios/pxelinux.0";
    } elsif option architecture-type = 00:09 {
        filename "/efi64/syslinux.efi";
    } elsif option architecture-type = 00:07 {
        filename "/efi64/bootx64.efi";
    } elsif option architecture-type = 00:06 {
        filename "/efi32/syslinux.efi";
    } else {
        filename "/bios/pxelinux.0"; 
    }
    include "/etc/dhcp/dhcpd.conf.quattor";
}

jouvin · 2016-10-28T13:00:35Z

@jrha what you mention in the "non Quattor" and rather trivial (as soon as you found the info!). AII impact is a bit more tricky, as explained in the description (about to be updated!).

jrha · 2016-10-28T13:30:35Z

AII could generate both configs for BIOS and UEFI and then let dhcpd select the correct one at boot time, which would avoid having to know anything in advance.

jouvin · 2016-10-28T13:35:23Z

Not really... Read the issue description: the problem is not in the DHCP config (DHCP just decides which PXE loader to use). After that, the same configuration entry is used and so it has to point to the right file. This is managed by aii-shellfe --install/boot.

jrha · 2016-10-28T14:20:33Z

For example...

shared-network "Example Network" {
    if option architecture-type = 00:00 {
        filename "/bios/pxelinux.0";
        option pxelinux.pathprefix "/payload/bios/";
    } elsif option architecture-type = 00:09 {
        filename "/efi64/syslinux.efi";
        option pxelinux.pathprefix "/payload/efi64/";
    } elsif option architecture-type = 00:07 {
        filename "/efi64/bootx64.efi";
        option pxelinux.pathprefix "/payload/efi64/";
    } elsif option architecture-type = 00:06 {
        filename "/efi32/syslinux.efi";
        option pxelinux.pathprefix "/payload/efi/";
    } else {
        filename "/bios/pxelinux.0";
        option pxelinux.pathprefix "/payload/bios/";
    }
    include "/etc/dhcp/dhcpd.conf.quattor";
}

jouvin · 2016-10-28T14:36:37Z

It may be an approach too... but one reason I didn't mention it is that I am not sure at all that pxelinux.pathprefix is taken into account by the RH UEFI PXE. Have you checked?

jrha · 2016-10-28T15:33:26Z

Nope, was just an idea.

jouvin · 2016-10-28T15:43:22Z

Unfortunately, I just checked and as I was afraid pxelinux.pathprefix (in fact dhcp option code 210) is ignored by RH UEFI PXE... Nevertheless, I think your idea is very good :-) and will help to manage the potential duplication of kernel/initrd directories. I'm updating the description reusing this idea.

jouvin · 2016-10-28T16:00:13Z

BTW, with PR quattor/template-library-standard#85 and the manual edition of the Grub configuration file, I have been able to install successfully a system, using the machine boot menu (F12) to force the boot on the hard disk and workaroud the problem of telling UEFI PXE to boot from the local disk.

stdweird · 2016-10-28T16:28:00Z

what pxe boot cfg files was used to tell UEFI to boot from localdisk?
afaik, you can tell PXELINUX to actually from local disk, or (what we typically do) continue the boot sequence (ie leave PXELINUX).

jouvin · 2016-10-28T16:46:27Z

Yes, this is exactly the problem I think. In PXELINUX, you were saying localboot 0 which doesn't mean that PXELINUX boots from local disk (as we tend to say) but that you exit PXELINUX and let the BIOS look for the next device to boot. The problem is that I don't see any way of doing this with Grub2: you can chain another boot loader but this is not what you want. You'd like Grub2 to exit without doing anything and let UEFI look for the next device, as it happened in PXELINUX. The only Grub2 command close to this is quit but according to the test I did, this is not working like this...

This issue is annoying but I think we should start implement the basic things. At the end this is (hoefully) just finding the right contents for one config file and this should not impact AII itself...

stdweird · 2016-10-28T17:54:59Z

from http://www.syslinux.org/archives/2013-August/020655.html

I've got a patch to support LOCALBOOT -1 on EFI, which will boot the
next entry in the EFI Boot Manager (the thing configured with
efibootmgr), and which I'll commit shortly, but it doesn't support
LOCALBOOT 0.

i'm sure what syslinux version RHEL ships, but a recent syslinux.efi might support LOCALBOOT -1?

stdweird · 2016-10-28T17:57:57Z

that discussion goes on Yes, I think so. The integer really means the same as EFI's ExitBootServices. and that the localboot integer has not meaning in EFI, and is ignored (so localboot 1 might work; or maybe just loclaboot). patch should be in 6.02.

stdweird · 2016-10-28T18:14:04Z

More info and tips: http://www.syslinux.org/wiki/index.php?title=PXELINUX#UEFI
the syslinux.efi cannot be used for secure boot, but should work well otherwise. the x64 one is the same for disk boot as for network boot, but the if we use localboot, it will pick up the one from RHEL, so secure boot from disk with RHEL EFI might work. but secure boot from network won't unless the RHEL supports localboot or we find a generic way to tell to boot from correct disk.

jouvin · 2016-10-28T18:55:13Z

I don't think that RHEL UEFI PXE is based on SYSLINUX one. This is why the config file format is not the same and why it doesn't accept the same options. I'll try to look at it in more details but having the RH one working would be great as from Grub you can do a lot of troubleshooting... something that used to be difficult with SYSLINUX.

stdweird · 2016-10-28T18:57:59Z

if the syslinux one support localboot, you will get the grub based one to debug things. it will just not be the one you get during pxe phase. that being said, getting the RHEL one to work with localboot is even better.
maybe RHEL support can shed some light on the matter.

stdweird · 2016-10-28T19:18:14Z

https://access.redhat.com/solutions/629813

RHEL recommends

root (hd0,0)
chainloader /efi/redhat/grub.efi

and that there's no localboot equilvalent.

but this was/is for EL6, not sure the path is wrong/not properly adapted or grub specific so YMMV

for quattor, most nodes will be hd(0,0), so at least a good default.

jouvin · 2016-10-28T19:27:02Z

In fact, this was the idea I had after digging in several places and that I tried unfortunately without success. I guess it should be /EFI/redhat/grub.efi. I'll double check next week.

jouvin · 2016-10-31T09:41:21Z

I spent some time this morning trying to troubleshoot this chainloading problem... I realized that the PXE loader from RH that Is used if in fact based on Legacy Grub. With this version, chainloading doesn't seem to work as expected (may be because it detects that the chainloaded loader is the same as itself and in fact does nothing...).

I tried to use grubx64.efi from EL7, a Grub2-based loader, and succeeded to chainload to an EL6 (legacy Grub) system doing the configuration manually at the Grub prompt. It was as simple as:

# In grub2, partition numbers start at 1
set root=(hd0,1)
chainloader /efi/redhat/gruf.efi

I now need to figure out how to get a host-based configuration name. The grubx64.efi that I used seems to use grub.cfg rather than an host entry based on the IP address example... Not yet clear if I'll success to use this PXE loader or if it's better to revert a SYSLINUX-based one.

jrha · 2016-10-31T11:03:37Z

Ooh I forgot about that, we changed our localboot.cfg to do localboot -1 to support HPE nodes booting via BIOS PXE emulation in UEFI.

jouvin · 2016-10-31T12:46:24Z

Huge step forward! I managed to get a a Grub2-based PXE loader to chainload to an EL6 or EL7 machine and to install (start the installer of) an EL7 machine. Unfortunately, Grub2 doesn't seem to be able to boot an EL6 kernel (error is "kernel too old") because when booting in UEFI mode, the standard 'linux' command to load the kernel is disabled and replaced with a 'linuxefi' command (the same applies to 'initrd') that enforces some checks on the kernel version. I guess that this is related to secure booting but I did not find yet a way to workaround this (not sure if this is even possible).

I used grubx64.efi (and related files) provided by EL7.2 in grub2-efi package (the PXE server doesn't have to be an EL7 machine, just need to copy the files provied by the RPM somewhere under /tftpboot). As with the Legacy Grub PXE loader, the configuration files have to leave into the same directory as grubx64.efi. But the device associated with the network is (pxe) instead of (nd) and it refers to /tftpboot (or whatever is your TFTPD root directory), meaning that there is no need to duplicate/symlink kernel/initrd directories. Just the path may be different than what has to be provided for PXELINUX. For exemple in our case, those directories and pxelinux.0 reside in /tftpboot/quattor. That means that in the Grub2 configuration, the kernel/initrd path must be /quattor/kernel_dir (instead of /kernel_dir with PXELINUX).

Some other differences introduced by using Grub2 are related to the configuration files searched:

The default file is grub.cfg (instead of default) but we don't really use it
The host specific config file, based on the IP address is grub.cfg-HEXIP (where HEXIP is the same as the hexadecimal address used with PXELINUX, with letters in uppercase).
The host specific config file based on the MAC address is grub.cfg-01-xx-xx-xx-xx-xx-xx ('xx-xx-xx-xx-xx-xx being the MAC address with letters in lowercase).

As for me, the remaining questions are:

Is it acceptable to say that we support UEFI only for EL7+ distributions? I'll tend to say yes and we may continue to search in the background if we find a way to start the EL6 installer. After the installatin, there is no problem to boot the EL6 installed system because the kernel is not boot directly but by chainloading to th Grub installed by EL6.
Do we want to have a look at SYSLINUX EFI support? I'll tend to say no in the short term, as various posts tend to say that EFI support in SYSLINUX is not yet mature and also because using Grub is quite handy in term of debugging (you have access to the Grub shell to troubleshoot problems).

jouvin · 2016-10-31T12:59:33Z

For the record, the Grub2 config files I used are:

localboot.cfg (for booting on local disk):

set default=0
set timeout=0

menuentry "Local Disk" {
    set root=(hd0,1)
    chainloader /efi/redhat/grub.efi
}

config file for triggering CentOS 7.2 installation:

set default=0
set timeout=0
menuentry "Install Scientific Linux 6x (x86_64)" {
    set root=(pxe)
    #linuxefi /quattor/sl6x_x86_64/vmlinuz ks=http://quattorsrv.lal.in2p3.fr/ks/grid304.lal.in2p3.fr.ks ksdevice=em1
    #initrdefi /quattor/sl6x_x86_64/initrd.img
    linuxefi /quattor/centos72_x86_64/vmlinuz ks=http://quattorsrv.lal.in2p3.fr/ks/grid304.lal.in2p3.fr.ks ksdevice=em1
    initrdefi /quattor/centos72_x86_64/initrd.img
}

jrha · 2016-10-31T13:13:19Z

Yes, for me UEFI only makes sense with EL7+.

stdweird · 2016-10-31T13:15:26Z

@jouvin thanks for all the testing!
i'm still interested in syslinux support / example config, as it is much more close to what we are currently using for legacy bios (in particular, the fact that we do not need to specify the hd(0,1) somehow for normal boot; there is nothing in the profiles that allows to configure this afaik).
we will stop caring about el6 in a few months, and maybe syslinux will provide a solution out for those demands.

jouvin · 2016-10-31T13:26:06Z

@stdweird (hd0,1) should work for all machines where the system disk is the first one seen by UEFI. It should cover most of the cases, but I agree that it will probably fail for some others (I remember Sun 4500 disk servers where the system disk was the highest numbered disk...). hd0 has nothing to do with hd block devices on Linux. They map everything that is not a CDROM or a floppy, basically. 1 means the first partition and should work as on UEFI-capable file system, the UEFI partition containing Grub to boot the the system should generally be the first one. And it if is not, this is in the profile as the profile contains partitions and their order.

jouvin · 2016-10-31T16:59:29Z

Unfortunately, after more testing, the proposed solutin in #216 (comment) is not so easy... In fact, for localboot.cfg the path to use to chain to the Grub installed on the system disk is varying between major version (EL6 and EL7 are different) and worst between distribution. In EL7, RH uses /efi/redhat/grubx64.efi, CentOS /efi/centos/grubx64.efi and I don't know about SL... And I don't know if there is any plan to change this in EL8... This means some strong coupling between the actual OS version deployed on the machine and the PXE config file and I have the feeling that this is not a good thing... The only "magic command" would have been chainloader +1 (meaning load one block on partition defined in the root variable) but blocklist specification (I tried several variants) seems to be disabled in this version of grubx64.efi.

I spent some time trying to find a workaround but had to give up after trying unsuccessfully to use Grub ls command and testing capabilities (Grub shell is very close to a Bash shell) but I didn't find how to return the command output in a variable. I also tried the (not very much documented) search command but it is clearly not for this too.

May be in the short term, the easiest will be to stay with SYSLINUX if it has a decent enough EFI support. 6.03 is released and in the posts I read today was reported as the first working version.

jouvin · 2016-11-02T13:32:36Z

I gave a try to SYSLINUX. I built 6.03 following the instructions (in particular not to rebuild ldlinux). But after loading it, my test machine hangs and I tend to think that this is the sign of something "wrong" in ldlinux.e64(or its interaction with the machine firmware). I may try to give a try to 6.04 pre-release but this does not look very promising...

jouvin · 2016-11-02T16:01:57Z

It works 😄. I managed to get the Grub2-based UEFI PXE boot loader to do exactly what PXELINUX has been doing: exit PXE and let the machine firmware decide which device to boot next (normally the hard disk). The Grub2 command to use is just exit! No need for chaining to another loader (despite it should work too if the blocklist specification was not disabled in the version I have).

I am updating the description on what has to be done for Grub2 (rather than Legacy Grub) support.

- nbpdir_grub, installation-kernel-root, grub2_efi_linux_cmd - Options added commented-out in the default aii-shellfe.conf - aii-core now has a dependency on aii-pxelinux constants: build order modified in pom.xml - Contributes to quattor#216

- Configuration is built for every variant enabled - Significant code cleanup - Contributes to quattor#216