#3277 PR `open`: WIP: Autodetect EFI shim¶

Labels: enhancement

pcahyna opened issue at 2024-07-12 11:18:¶

fixes #3276

schlomo commented at 2024-07-12 12:40:¶

Ubuntu 24.04 ⚠️¶

root@rear-u2404:/src/rear-pcahyna# REAR_VAR=/tmp/rear ./usr/sbin/rear -v mkrescue
Relax-and-Recover 2.7 / Git
Running rear mkrescue (PID 3728 date 2024-07-12 12:18:13)
Using log file: /tmp/rear/log/rear/rear-rear-u2404.log
Running workflow mkrescue on the normal/original system
Using UEFI Boot Loader for Linux (USING_UEFI_BOOTLOADER=1)
Using autodetected kernel '/boot/vmlinuz-6.8.0-38-generic' as kernel in the recovery system
Creating disk layout
Verifying that the entries in /tmp/rear/lib/rear/layout/disklayout.conf are correct
Created disk layout (check the results in /tmp/rear/lib/rear/layout/disklayout.conf)
Creating recovery system root filesystem skeleton layout
Skipping 'lo': not bound to any physical interface.
Trying to autodetect from EFI variables what to use as UEFI bootloader file...
Detected current EFI bootloader /boot/efi/EFI/ubuntu/shimx64.efi
Detected distribution EFI bootloader shimx64.efi in /boot/efi/EFI/ubuntu/BOOTX64.CSV
Trying to find what to use as UEFI bootloader...
Trying to find grub*.efi in the directory of the Secure Boot bootloader /boot/efi/EFI/ubuntu/shimx64.efi to be used as UEFI bootloader...
Using '/boot/efi/EFI/ubuntu/shimx64.efi' as UEFI Secure Boot bootloader file
Copying logfile /tmp/rear/log/rear/rear-rear-u2404.log into initramfs as '/tmp/rear-rear-u2404-partial-2024-07-12T12:18:18+00:00.log'
Copying files and directories
Copying binaries and libraries
Copying only currently loaded kernel modules (MODULES contains 'loaded_modules') and those in MODULES_LOAD
Omit copying files in /lib*/firmware/ (FIRMWARE_FILES='no')
Symlink '/usr/share/misc/magic' -> '/usr/share/file/magic' refers to a non-existing directory on the recovery system.
It will not be copied by default. You can include '/usr/share/file/magic' via the 'COPY_AS_IS' configuration variable.
Testing that the recovery system in /var/tmp/rear.k8egAkMMqKPF8jd/rootfs contains a usable system
/usr/lib/x86_64-linux-gnu/systemd/libsystemd-core-255.so requires additional libraries
        libsystemd-shared-255.so => not found
ReaR recovery system in '/var/tmp/rear.k8egAkMMqKPF8jd/rootfs' needs additional libraries, check /tmp/rear/log/rear/rear-rear-u2404.log for details
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (81021118 bytes) in 9 seconds
Did not find /boot/grub/locale files (minor issue for UEFI ISO boot)
Making ISO image
Wrote ISO image: /tmp/rear/lib/rear/output/rear-rear-u2404.iso (198M)
Copying resulting files to nfs location
Saving /tmp/rear/log/rear/rear-rear-u2404.log as rear-rear-u2404.log to nfs location
Copying result files '/tmp/rear/lib/rear/output/rear-rear-u2404.iso /var/tmp/rear.k8egAkMMqKPF8jd/tmp/VERSION /var/tmp/rear.k8egAkMMqKPF8jd/tmp/README /var/tmp/rear.k8egAkMMqKPF8jd/tmp/rear-rear-u2404.log' to /var/tmp/rear.k8egAkMMqKPF8jd/outputfs/rear-u2404 at nfs location
Exiting rear mkrescue (PID 3728) and its descendant processes ...
Running exit tasks

But the resulting ISO image is not bootable.

Log file from rear -D mkrescue: rear-rear-u2404.log

The isofs directory:

root@rear-u2404:/var/tmp/rear.XAjwSd2enV3Cv6t/tmp/isofs# tree -f
.
├── ./EFI
│   └── ./EFI/BOOT
│       ├── ./EFI/BOOT/BOOTX64.efi
│       ├── ./EFI/BOOT/fonts
│       │   └── ./EFI/BOOT/fonts/unicode.pf2
│       ├── ./EFI/BOOT/grub.cfg
│       ├── ./EFI/BOOT/grubx64.efi
│       └── ./EFI/BOOT/locale
├── ./boot
│   ├── ./boot/efiboot.img
│   └── ./boot/grub
│       └── ./boot/grub/grub.cfg
└── ./isolinux
    ├── ./isolinux/chain.c32
    ├── ./isolinux/hdt.c32
    ├── ./isolinux/initrd.cgz
    ├── ./isolinux/isolinux.bin
    ├── ./isolinux/isolinux.cfg
    ├── ./isolinux/kernel
    ├── ./isolinux/libcom32.c32
    ├── ./isolinux/libgpl.c32
    ├── ./isolinux/libmenu.c32
    ├── ./isolinux/libutil.c32
    ├── ./isolinux/menu.c32
    ├── ./isolinux/message
    ├── ./isolinux/pci.ids
    ├── ./isolinux/rear.help
    ├── ./isolinux/reboot.c32
    └── ./isolinux/vesamenu.c32

and the comparison between the efi files:

root@rear-u2404:/var/tmp/rear.XAjwSd2enV3Cv6t/tmp/isofs# ll ./EFI/BOOT/BOOTX64.efi ./EFI/BOOT/grubx64.efi /boot/efi/EFI/ubuntu/shimx64.efi /boot/efi/EFI/ubuntu/grubx64.efi 
-rwx------ 1 root root 2656136 Jul 12 12:33 ./EFI/BOOT/BOOTX64.efi*
-rwx------ 1 root root 2656136 Jul 12 12:33 ./EFI/BOOT/grubx64.efi*
-rwxr-xr-x 1 root root 2656136 Jul 12 12:03 /boot/efi/EFI/ubuntu/grubx64.efi*
-rwxr-xr-x 1 root root  966664 Jul 12 12:03 /boot/efi/EFI/ubuntu/shimx64.efi*

so it seems to not have picked the shim even though the console output during mkrescue suggested that it did:

Trying to autodetect from EFI variables what to use as UEFI bootloader file...
Detected current EFI bootloader /boot/efi/EFI/ubuntu/shimx64.efi
Detected distribution EFI bootloader shimx64.efi in /boot/efi/EFI/ubuntu/BOOTX64.CSV
Trying to find what to use as UEFI bootloader...
Trying to find grub*.efi in the directory of the Secure Boot bootloader /boot/efi/EFI/ubuntu/shimx64.efi to be used as UEFI bootloader...
Using '/boot/efi/EFI/ubuntu/shimx64.efi' as UEFI Secure Boot bootloader file

HTH

pcahyna commented at 2024-07-12 13:02:¶

Thanks @schlomo for your test - tests on other distributions is exactly what is needed.

I recall now that the code has had one more huge gap - it is OUTPUT=USB only, support for OUTPUT=ISO has needed to be added IIRC. Can you please retry on Ubuntu, but with OUTPUT=USB ?

schlomo commented at 2024-07-12 13:07:¶

@pcahyna unfortunately I don't know how to test USB mode on VirtualBox, and I don't have any time left.

SLE15.6 also doesn't work, similar to Ubuntu.

I'd like to suggest using #3278 as a quick fix for the moment and this version as an improvement on that. The approaches are fundamentally different so that they can either complement each other or you'll simply remove the 1 file I added.

WDYT?

schlomo commented at 2024-07-12 13:49:¶

@pcahyna do you see any chance to unify the EFI boot between at least ISO and USB, preferably also RAWDISK? Maybe using functions to do the actual work? Or symlinking scripts between the different directories?

Maybe another approach could be to use your improved "getting infos from EFI vars" code with my simpler "enable secure boot if shim is used" idea in #3278? Maybe that would be less effort?

Will you find time to complete this work in the near future or for the planned 3.0 release?

pcahyna commented at 2024-07-12 15:55:¶

Will you find time to complete this work in the near future or for the planned 3.0 release?

I believe I will, but without any attempt at

unify the EFI boot between at least ISO and USB, preferably also RAWDISK? Maybe using functions to do the actual work? Or symlinking scripts between the different directories?

This would be too much work, I am afraid. Unless the ISO part proves to be easier to do this way rather than via an incremental change, which I doubt.

pcahyna commented at 2024-07-12 15:56:¶

Maybe another approach could be to use your improved "getting infos from EFI vars" code with my simpler "enable secure boot if shim is used" idea in https://github.com/rear/rear/pull/3278? Maybe that would be less effort?

I think not, but I need to check again what needs to be done for ISO and why you have not needed any such change in your simpler approach.

pcahyna commented at 2024-07-12 16:59:¶

@pcahyna unfortunately I don't know how to test USB mode on VirtualBox, and I don't have any time left.

SLE15.6 also doesn't work, similar to Ubuntu.

@schlomo I am sorry that I forgot why I had not proposed this PR earlier in the first place (testing on other distros was not the main reason - the need to complete the ISO support was) and had to be reminded via your test failure and thus I wasted your time.

This brings me a larger question though: who will test changes on Ubuntu/Debian? Will you be able to find the time to retest again after this is completed? I don't have any Ubuntu or Debian test environment and this is not the first time that the desire of supporting Ubuntu/Debian and the simultaneous lack of ability of testing changes on it has been an obstacle. I checked the Ubuntu package: it is in the Universe repository, meaning Community-maintained instead of Canonical-supported - a very different situation from Red Hat and SUSE who provide supported versions to customers and employ people that support their distros. In Ubuntu, the package seems to be simply taken from Debian. And in Debian, the maintainer (Frédéric Bonnard) has not touched the package for a long time and even the one CVE bug is still unpatched: https://tracker.debian.org/pkg/rear .

I am afraid that this model for Debian and Ubuntu support is not sustainable.

I'd like to suggest using #3278 as a quick fix for the moment and this version as an improvement on that. The approaches are fundamentally different so that they can either complement each other or you'll simply remove the 1 file I added.

Generally I prefer to not add half-finished stuff, but if it does not make integrating a more complete solution more difficult, I think that perfect is the enemy of good and what you suggest is the best solution. Let me check in more detail what the exact differences between the approaches are, please, and I will give my review on #3278 then.

gdha commented at 2024-07-17 06:28:¶

@pcahyna @schlomo I prefer to have a solid solution and not force a v3.0 release on short notice. WDYT?

schlomo commented at 2024-07-17 06:35:¶

@gdha I'm actually for iterative improvement. #3278 adds support for secure boot auto-configuration without improving anything for systems where secure boot is not active. And it doesn't change anything else, too.

All users with active secure boot will benefit from it and I would like to give that to our users.

In the absence of paid development or a roadmap with people committing to work on it within a given time I think that as an Open Source project we need to take what we can get when it comes.

gdha commented at 2024-07-17 06:40:¶

@schlomo

In the absence of paid development or a roadmap with people committing to work on it within a given time I think that as an Open Source project we need to take what we can get when it comes.

Indeed, that is true. However, remember our words "why rush it if we are not paid for a new release?"

pcahyna commented at 2024-07-17 18:41:¶

I am with @schlomo here. I would prefer to take longer for a solid solution, if the less solid solution either: -- makes things worse in some cases (I don't see this in the case https://github.com/rear/rear/pull/3278 ), or: -- adds poorly designed configuration options or changes the interpretation of current configuration options in future-incompatible way, as it would create more work for the user (basically they would have to adapt again when a more solid solution is implemented and I don't think this would be the case in #3278 either, although I still need to check).

And I don't think it necessarily means to force a v3.0 release on short notice - we can still attempt to get this PR merged before 3.0, it merely means that if it proves too difficult, we have at least something.

Regarding "we are not paid for ...", any thoughts on the "we are not paid for Ubuntu /Debian, yet we support it" problem?

gdha commented at 2024-07-18 06:37:¶

@pcahyna