#3170 Issue `closed`: Alma Linux 9.3 not able to install bootloader.¶

Labels: support / question, fixed / solved / done

rvdkraan opened issue at 2024-02-29 14:23:¶

When recovering using a usb key a warning is shown that the bootlaoder could not be installed.
no code for the os used to auto recover. Advising to manually install the bootloader.

The OS was successfully recovered however without a bootloader the system will not start.

Begin dumping out configuration and system information:
 This is a 'Linux-x86_64' system, compatible with 'Linux-i386'.
 Configuration tree:
  Linux-i386.conf : OK
   GNU/Linux.conf : OK
   Fedora.conf : missing/empty
   Fedora/i386.conf : missing/empty
   Fedora/9.3.conf : missing/empty
   Fedora/9.3/i386.conf : missing/empty
   RedHatEnterpriseServer.conf : missing/empty
   RedHatEnterpriseServer/i386.conf : missing/empty
   RedHatEnterpriseServer/9.3.conf : missing/empty
   RedHatEnterpriseServer/9.3/i386.conf : missing/empty
   site.conf : OK
   local.conf : OK
 System definition:
  ARCH="Linux-i386"
  OS="GNU/Linux"
  OS_MASTER_VENDOR="Fedora"
  OS_MASTER_VERSION="9.3"
  OS_MASTER_VENDOR_ARCH="Fedora/i386"
  OS_MASTER_VENDOR_VERSION="Fedora/9.3"
  OS_MASTER_VENDOR_VERSION_ARCH="Fedora/9.3/i386"
  OS_VENDOR="RedHatEnterpriseServer"
  OS_VERSION="9.3"
  OS_VENDOR_ARCH="RedHatEnterpriseServer/i386"
  OS_VENDOR_VERSION="RedHatEnterpriseServer/9.3"
  OS_VENDOR_VERSION_ARCH="RedHatEnterpriseServer/9.3/i386"
# Backup with NETFS:
  NETFS_KEEP_OLD_BACKUP_COPY=""
  NETFS_PREFIX="hms-alma"
  NETFS_RESTORE_CAPABILITIES=("No")
  BACKUP_DUPLICITY_NAME="rear-backup"
  BACKUP_INTEGRITY_CHECK=""
  BACKUP_MOUNTCMD=""
  BACKUP_ONLY_EXCLUDE="no"
  BACKUP_ONLY_INCLUDE="no"
  BACKUP_OPTIONS=""
  BACKUP_RESTORE_MOVE_AWAY_DIRECTORY="/var/lib/rear/moved_away_after_backup_restore/"
  BACKUP_RESTORE_MOVE_AWAY_FILES=("/boot/grub/grubenv" "/boot/grub2/grubenv")
  BACKUP_RSYNC_OPTIONS=("--sparse" "--archive" "--hard-links" "--numeric-ids" "--stats")
  BACKUP_SELINUX_DISABLE="1"
  BACKUP_TYPE=""
  BACKUP_UMOUNTCMD=""
  BACKUP_URL="usb:///dev/disk/by-label/REAR-000"
# Backup program is 'tar':
  BACKUP_PROG_ARCHIVE="backup"
  BACKUP_PROG_COMPRESS_OPTIONS=("--gzip")
  BACKUP_PROG_COMPRESS_SUFFIX=".gz"
  BACKUP_PROG_CRYPT_ENABLED="false"
  BACKUP_PROG_CRYPT_KEY=""
  BACKUP_PROG_CRYPT_OPTIONS="/usr/bin/openssl des3 -salt -k "
  BACKUP_PROG_DECRYPT_OPTIONS="/usr/bin/openssl des3 -d -k "
  BACKUP_PROG_EXCLUDE=("/tmp/*" "/dev/shm/*" "/var/lib/rear/output/*" "/var/tmp/rear.p8MJE6hNidGJS4u")
  BACKUP_PROG_OPTIONS=("--anchored")
  BACKUP_PROG_OPTIONS_CREATE_ARCHIVE=""
  BACKUP_PROG_OPTIONS_RESTORE_ARCHIVE=""
  BACKUP_PROG_SUFFIX=".tar"
  BACKUP_PROG_WARN_PARTIAL_TRANSFER="1"
# Output to USB:
  USB_BIOS_BOOT_DEFAULT=""
  USB_BOOTLOADER=""
  USB_BOOT_PART_SIZE="0"
  USB_DEVICE=""
  USB_DEVICE_BOOT_LABEL="REARBOOT"
  USB_DEVICE_FILESYSTEM="ext3"
  USB_DEVICE_FILESYSTEM_LABEL="REAR-000"
  USB_DEVICE_FILESYSTEM_PARAMS=""
  USB_DEVICE_FILESYSTEM_PERCENTAGE="100"
  USB_DEVICE_PARTED_LABEL=""
  USB_PARTITION_ALIGN_BLOCK_SIZE="8"
  USB_RETAIN_BACKUP_NR="2"
  USB_SUFFIX=""
  USB_UEFI_PART_SIZE="1024"
  OUTPUT_EFISTUB_SYSTEMD_BOOTLOADER="/usr/lib/systemd/boot/efi/systemd-bootx64.efi"
  OUTPUT_LFTP_PASSWORD=""
  OUTPUT_LFTP_USERNAME=""
  OUTPUT_MOUNTCMD=""
  OUTPUT_OPTIONS=""
  OUTPUT_PREFIX="hms-alma"
  OUTPUT_PREFIX_PXE=""
  OUTPUT_UMOUNTCMD=""
  OUTPUT_URL=""

ReaR version ("/usr/sbin/rear -V"):
Relax-and-Recover 2.7 / Git
OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):

NAME="AlmaLinux"
VERSION="9.3 (Shamrock Pampas Cat)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.3"
PLATFORM_ID="platform:el9"
PRETTY_NAME="AlmaLinux 9.3 (Shamrock Pampas Cat)"
ANSI_COLOR="0;34"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:almalinux:almalinux:9::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"

ALMALINUX_MANTISBT_PROJECT="AlmaLinux-9"
ALMALINUX_MANTISBT_PROJECT_VERSION="9.3"
REDHAT_SUPPORT_PRODUCT="AlmaLinux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.3"

ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

OUTPUT=USB
BACKUP=NETFS

BACKUP_URL="usb:///dev/disk/by-label/REAR-000"

USE_DHCLIENT=no
USE_STATIC_NETWORKING=y

Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR):
Axiomsys
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
X86 64bit
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):

BIOS Information
        Vendor: American Megatrends Inc.
        Version: 5.12
        Release Date: 04/20/2018
        Address: 0xF0000
        Runtime Size: 64 kB
        ROM Size: 16 MB
        Characteristics:
                PCI is supported
                BIOS is upgradeable
                BIOS shadowing is allowed
                Boot from CD is supported
                Selectable boot is supported
                BIOS ROM is socketed
                EDD is supported
                5.25"/1.2 MB floppy services are supported (int 13h)
                3.5"/720 kB floppy services are supported (int 13h)
                3.5"/2.88 MB floppy services are supported (int 13h)
                Print screen service is supported (int 5h)
                Serial services are supported (int 14h)
                Printer services are supported (int 17h)
                ACPI is supported
                USB legacy is supported
                BIOS boot specification is supported
                Targeted content distribution is supported
                UEFI is supported
        BIOS Revision: 5.12

Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
Dual SSD
Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"):

NAME                               KNAME     PKNAME    TRAN   TYPE FSTYPE      LABEL   SIZE MOUNTPOINT
/dev/sda                           /dev/sda            sata   disk                   119.2G
|-/dev/sda1                        /dev/sda1 /dev/sda         part vfat                600M /boot/efi
|-/dev/sda2                        /dev/sda2 /dev/sda         part xfs                   1G /boot
`-/dev/sda3                        /dev/sda3 /dev/sda         part LVM2_member       117.7G
  |-/dev/mapper/almalinux_hms-root /dev/dm-0 /dev/sda3        lvm  xfs                  70G /
  |-/dev/mapper/almalinux_hms-swap /dev/dm-1 /dev/sda3        lvm  swap                3.9G [SWAP]
  `-/dev/mapper/almalinux_hms-home /dev/dm-2 /dev/sda3        lvm  xfs                43.8G /home
/dev/sdb                           /dev/sdb            sata   disk                   111.8G
`-/dev/sdb1                        /dev/sdb1 /dev/sdb         part xfs               111.8G /database

gdha commented at 2024-03-01 14:28:¶

@rvdkraan A debug log of a 'mkrescue' could be helpful.

jsmeix commented at 2024-03-01 14:35:¶

@rvdkraan
plus a debug log of "rear -D recover", see in
https://en.opensuse.org/SDB:Disaster_Recovery
in the section
"Debugging issues with Relax-and-Recover"
both parts
"To analyze and debug a 'rear mkrescue/mkbackup' failure
the following information is mandatory"
and
"To analyze and debug a "rear recover" failure the
following additional information is also mandatory"

In general:
Caution with possible secrets in a full debug log file:
When 'rear' is run via '-D' in debugscript mode
it logs executed commands via the bash command 'set -x'
that print commands and their arguments as they are executed
so in particular when arguments contain secret values
(e.g. something like a password or whatever else)
such secret values may appear in the log file.
Also secrets may be stored in some other files
like /var/lib/rear/layout/disklayout.conf
or /var/lib/rear/layout/diskrestore.sh
cf. [password=<password>] in the section
"Disk layout file syntax" in
doc/user-guide/06-layout-configuration.adoc
online at
https://github.com/rear/rear/blob/rear-2.7/doc/user-guide/06-layout-configuration.adoc
So before you attach your full debug log file and other files
here (GitHub is a public accessible place) inspect your files
and verify that they do not accidentally contain secrets.

pcahyna commented at 2024-03-01 15:01:¶

When recovering using a usb key a warning is shown that the bootlaoder could not be installed.
no code for the os used to auto recover. Advising to manually install the bootloader.

It would be also helpful to provide the actual warning message and information about the firmware used to boot (BIOS or UEFI? If UEFI, is Secure Boot used?)

rvdkraan commented at 2024-03-05 10:30:¶

Hi @gdha @jsmeix @pcahyna

Thank you all for the feedback.
It really helps to have an active community to help each other.

I have attached the log files.
My disk is not encrypted. And did not find any secrets. If you do we need to delete this post.

rear-hms-alma recover.log
rear-log mkrescue.log
EFI
boot

U use the BIOS boot mode. EFI gives an warning.
It is also a bit confusing that mkrescue finishes ok but the log file shows a few errors.

warning in console about the bootloader:

WARNING:
For this system
RedHatEnterpriseServer/9.3 on Linux-i386 (based on Fedora/9.3/i386)
there is no code to install a boot loader on the recovered system
or the code that we have failed to install the boot loader correctly.
Please contribute appropriate code to the Relax-and-Recover project,
see http://relax-and-recover.org/development/
Take a look at the scripts in /usr/share/rear/finalize - for example
for PC architectures like x86 and x86_64 see the script
/usr/share/rear/finalize/Linux-i386/660_install_grub2.sh
and for POWER architectures like ppc64le see the script
/usr/share/rear/finalize/Linux-ppc64le/660_install_grub2.sh
| IF YOU DO NOT INSTALL A BOOT LOADER MANUALLY, |
| THEN YOUR SYSTEM WILL NOT BE ABLE TO BOOT. |
You can use 'chroot /mnt/local bash --login'
to change into the recovered system and
manually install a boot loader therein.

TIA
Ruben

jsmeix commented at 2024-03-05 13:48:¶

@rvdkraan

I am not a bootloader expert and
I am even less an UEFI expert
but I try as good as I can:

Fist some general info:

The primary user config variables
that determine which kind of bootloader
will be (re)installed during "rear recover" are
BOOTLOADER
USING_UEFI_BOOTLOADER
UEFI_BOOTLOADER
see their descriptions in usr/share/rear/conf/default.conf

When there are issues with installing the bootloader
of the recreated system during "rear recover"
it should help to specify the right values as needed
for those config variables.

What does not work automatically or cannot work is
when you boot via UEFI on the original system
where "rear mkrescue/mkbackup" was run
BUT
you want to boot via BIOS on the replacement system
where "rear recover" is run.

In general ReaR is meant to recreate a system
as it was before.
In particular ReaR is meant to recreate a system
on fully compatible replacement hardware
(also fully compatible replacement virtual hardware),
see the section
"Fully compatible replacement hardware is needed" in
https://en.opensuse.org/SDB:Disaster_Recovery

It might work (but I did not test it)
when you had UEFI on the original system
but now use BIOS on the replacement system
(provided you can boot the ReaR recovery system
on the replacement system with BIOS)
that you manually adapt user config variables
in /etc/rear/local.conf and /etc/rear/rescue.conf
and the var/lib/rear/recovery/bootloader value
inside the booted ReaR recovery system
before you run "rear recover".

Excerpts where I had a look at your log files:

Ecxerpts from your
https://github.com/rear/rear/files/14493833/rear-log.mkrescue.log

+ source /usr/share/rear/prep/default/320_include_uefi_env.sh
...
++ DebugPrint 'Found EFI system partition /dev/sda1 on /boot/efi type vfat'
++ USING_UEFI_BOOTLOADER=1

+ source /usr/share/rear/layout/save/default/445_guess_bootloader.sh
...
++ Log 'No known bootloader matches the first bytes on /dev/sda'
...
++ Log 'No known bootloader matches the first bytes on /dev/sdb'
...
++ Log 'No known bootloader matches the first bytes on /dev/sdc'
...
++ Log 'GRUB 2 is installed (grub-probe or grub2-probe exist).'
...
++ echo GRUB2-EFI

so you have GRUB2-EFI in var/lib/rear/recovery/bootloader
which looks right - as far as I see at first glance.

Then excerpts from your
https://github.com/rear/rear/files/14493832/rear-hms-alma.recover.log

+ source /etc/rear/rescue.conf
...
++ USING_UEFI_BOOTLOADER=1
++ UEFI_BOOTLOADER=/boot/efi/EFI/almalinux/grubx64.efi

+ source /usr/share/rear/finalize/Linux-i386/660_install_grub2.sh
...
++ '[' GRUB2 '!=' GRUB2-EFI ']'
++ return 0

so finalize/Linux-i386/660_install_grub2.sh does nothing
and the subsequent script

+ source /usr/share/rear/finalize/Linux-i386/670_run_efibootmgr.sh
...
++ Log efibootmgr --create --gpt --disk /dev/sda --part 1 --write-signature --label '"RedHatEnterpriseServer' '9.3"' --loader '"\EFI\almalinux\grubx64.efi"'
2024-03-05 09:28:27.331724852 efibootmgr --create --gpt --disk /dev/sda --part 1 --write-signature --label "RedHatEnterpriseServer 9.3" --loader "\EFI\almalinux\grubx64.efi"
++ efibootmgr --create --gpt --disk /dev/sda --part 1 --write-signature --label 'RedHatEnterpriseServer 9.3' --loader '\EFI\almalinux\grubx64.efi'
EFI variables are not supported on this system.
++ LogPrintError 'efibootmgr failed to create EFI Boot Manager entry on /dev/sda partition 1 (ESP /dev/sda1 )'

failed because "EFI variables are not supported on this system"
(I think this is because you use BIOS on the replacement system)
so in the end no bootloader (neither for BIOS nor for UEFI)
gets installed.

Because I am not a bootloader expert
I may misunderstand things so my above analysis
how things fail in this specific case could be just wrong.

pcahyna commented at 2024-03-05 14:23:¶

@rvdkraan

U use the BIOS boot mode. EFI gives an warning.

I suppose you mean "I use the BIOS boot mode. EFI gives an warning."

If so, and you are booting in BIOS mode, this is the reason for the bootloader warning that you see. You need to recover to a matching system, as explained by @jsmeix (i.e. backup was created on EFI - recover to a system with EFI). Even then, with BIOS, I think that the recovered system will work OK, if you select the restored disk in the BIOS boot menu as the boot device, and you can thus ignore the message.

The EFI problem in the screenshot is a known bug of GRUB, see https://github.com/rear/rear/issues/2971 . It will be solved by a GRUB update. In the meantime, as a workaround, set SECURE_BOOT_BOOTLOADER=/boot/efi/EFI/almalinux/shimx64.efi in /etc/rear/local.conf and recreate the rescue USB. This should make your USB bootable in EFI mode and the bootloader warning that you see at the end should not appear.

rvdkraan commented at 2024-03-05 14:28:¶

I try to edit the config.
The hardware should be almost identical, the disk in the system I am restoring to is bigger.
Both systems support BIOS and UEFI but the UEFI boot of rear does not work.
I saw that this should be fixed in a new release.

pcahyna commented at 2024-03-05 14:32:¶

@rvdkraan the point is, by setting SECURE_BOOT_BOOTLOADER=/boot/efi/EFI/almalinux/shimx64.efi UEFI should work already in this release

jsmeix commented at 2024-03-06 16:09:¶

Only FYI as a side note:

Out of curiosity I was wondering how far it could work
to call existing ReaR scripts to install a bootloader
from within the ReaR recovery system.

I use a QEMU/KVM system with BIOS and a single /dev/sda.

In the ReaR recovery system
before running "rear recover"
I did

RESCUE localhost:~ # export GRUB2_INSTALL_DEVICES="/dev/sdQQQ"

to make finalize/Linux-i386/660_install_grub2.sh
intentionally fail
so I got

RESCUE localhost:~ # rear -D recover
...
Installing GRUB2 boot loader...
Installing GRUB2 on /dev/sdQQQ (specified in GRUB2_INSTALL_DEVICES)
Failed to install GRUB2 on /dev/sdQQQ
Failed to install GRUB2 on /dev/sdQQQ
WARNING:
For this system
SUSE_LINUX/15.5 on Linux-i386 (based on SUSE/15/i386)
there is no code to install a boot loader on the recovered system
or the code that we have failed to install the boot loader correctly.
Please contribute appropriate code to the Relax-and-Recover project,
see http://relax-and-recover.org/development/
Take a look at the scripts in /usr/share/rear/finalize - for example
for PC architectures like x86 and x86_64 see the script
/usr/share/rear/finalize/Linux-i386/660_install_grub2.sh
and for POWER architectures like ppc64le see the script
/usr/share/rear/finalize/Linux-ppc64le/660_install_grub2.sh
---------------------------------------------------
|  IF YOU DO NOT INSTALL A BOOT LOADER MANUALLY,  |
|  THEN YOUR SYSTEM WILL NOT BE ABLE TO BOOT.     |
---------------------------------------------------
You can use 'chroot /mnt/local bash --login'
to change into the recovered system and
manually install a boot loader therein.
...

Then I did

RESCUE localhost:~ # export GRUB2_INSTALL_DEVICES="/dev/sda"

RESCUE localhost:~ # rear -d shell
...
REAR localhost:/usr/share/rear # export NOBOOTLOADER="yes"

REAR localhost:/usr/share/rear # Source finalize/Linux-i386/660_install_grub2.sh
...
Installing GRUB2 boot loader...
Generating grub configuration file ...
Found theme: /boot/grub2/themes/SLE/theme.txt
Found linux image: /boot/vmlinuz-5.14.21-150500.55.28-default
Found initrd image: /boot/initrd-5.14.21-150500.55.28-default
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
done
Installing GRUB2 on /dev/sda (specified in GRUB2_INSTALL_DEVICES)
Installing for i386-pc platform.
Installation finished. No error reported.

REAR localhost:/usr/share/rear # exit

RESCUE localhost:~ # reboot

The recreated system boots normally for me.

To make rear shell working
within the ReaR recovery system one must adapt
usr/share/rear/init/default/050_check_rear_recover_mode.sh
as follows

--- a/usr/share/rear/init/default/050_check_rear_recover_mode.sh
+++ b/usr/share/rear/init/default/050_check_rear_recover_mode.sh
@@ -15,7 +15,7 @@
 if test -f /etc/rear-release ; then
     # We are in the ReaR rescue/recovery system:
     case "$WORKFLOW" in
-        (recover|layoutonly|restoreonly|finalizeonly|mountonly|opaladmin|help)
+        (recover|layoutonly|restoreonly|finalizeonly|mountonly|opaladmin|shell|help)
             LogPrint "Running workflow $WORKFLOW within the ReaR rescue/recovery system"
             ;;
         (*)

i.e. add shell to the workflows that are allowed
to be run from within the ReaR recovery system.

rvdkraan commented at 2024-03-07 08:20:¶

@pcahyna
The workaround worked and I was able to recover. Thanks.
@jsmeix
I do not fully grasp the shell you mentioned.
However it nice to know that a failed recovery for the bootloader can be fixt manually.

pcahyna commented at 2024-03-08 15:47:¶

@jsmeix rear shell looks nice, TIL! But in this case we don't even need it, if the user boots from the recovered disk, the EFI shim's fallback mechanism will fix up the boot entry automatically. https://github.com/rhboot/shim/blob/main/README.fallback

jsmeix commented at 2024-03-08 16:21:¶

Yes, this issue only triggered me to try out
calling ReaR scripts from within the ReaR recovery system.
I used BIOS because @rvdkraan tried to recover on BIOS.

pcahyna commented at 2024-03-08 16:26:¶

Fore me the interesting thing to note is that restoring a UEFI system in BIOS mode "just works" and the warning can thus be considered excessive. I suppose the opposite is not true (the bootloader would not get installed and the system would not boot).

rvdkraan commented at 2024-03-12 12:39:¶

Hi @pcahyna

Rebooting gave me an screen showing only a white underscore like to have on a prompt but than without blinking.
However maybe there was an other reason the recovered system did not boot.

pcahyna commented at 2024-03-12 14:31:¶

Hi @pcahyna

Rebooting gave me an screen showing only a white underscore like to have on a prompt but than without blinking. However maybe there was an other reason the recovered system did not boot.

hi @rvdkraan , when did that happen? Was it before applying the SECURE_BOOT_BOOTLOADER= workaround, i.e. after the recovery has printed the warning

| IF YOU DO NOT INSTALL A BOOT LOADER MANUALLY, |
| THEN YOUR SYSTEM WILL NOT BE ABLE TO BOOT. |

?

[Export of Github issue for rear/rear.]

#3170 Issue closed: Alma Linux 9.3 not able to install bootloader.¶