#3263 Issue open: OUTPUT=PORTABLE on SLES15/SP5 in rear recover dracut fails to build initramfs

Labels: support / question

saschalucas opened issue at 2024-07-02 11:41:

  • ReaR version ("/usr/sbin/rear -V"): 2.7-git.5431.6389691b.master / 2024-05-03

  • OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):

NAME="SLES"
VERSION="15-SP5"
VERSION_ID="15.5"
  • ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
OUTPUT=PORTABLE
OUTPUT_URL=file:///var/lib/rear/output
POST_RECOVERY_COMMANDS+=( "tar -xf /root/server1005-commvault-certs.tar.gz -C /mnt/local/opt/commvault/Base64/" )

BACKUP=EXTERNAL
EXTERNAL_RESTORE=/root/commvault-restore.sh

FIRMWARE_FILES=( 'no' )
MODULES=( 'no_modules' )

DISKS_TO_BE_WIPED='/dev/vd[a-z] /dev/sd[a-z]'
USER_INPUT_TIMEOUT=30

  • Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR): VM (esx)

  • System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device): x86_64

  • Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot): BIOS/grub

  • Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe): local disk

  • Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"):

NAME                        KNAME     PKNAME    TRAN TYPE FSTYPE      LABEL  SIZE MOUNTPOINT
/dev/sda                    /dev/sda                 disk                     20G 
|-/dev/sda1                 /dev/sda1 /dev/sda       part                      8M 
`-/dev/sda2                 /dev/sda2 /dev/sda       part LVM2_member         20G 
  |-/dev/mapper/system-swap /dev/dm-0 /dev/sda2      lvm  swap                 4G [SWAP]
  `-/dev/mapper/system-root /dev/dm-1 /dev/sda2      lvm  ext4                19G /
/dev/sdb                    /dev/sdb                 disk LVM2_member         10G 
|-/dev/mapper/system-swap   /dev/dm-0 /dev/sdb       lvm  swap                 4G [SWAP]
`-/dev/mapper/system-data   /dev/dm-2 /dev/sdb       lvm  ext4                 7G /data
/dev/sr0                    /dev/sr0            ata  rom                    1024M
  • Description of the issue (ideally so that others can reproduce it):
    On SLES15/SP5 rear recover fails to build initramfs. On screen and log this content is shown:
2024-07-02 13:11:01.185390577 Including finalize/SUSE_LINUX/i386/550_rebuild_initramfs.sh
2024-07-02 13:11:06.369117289 Recreating initrd with /usr/bin/dracut...
2024-07-02 13:11:06.632109467 Warning:
                              Failed to recreate initrd with /usr/bin/dracut.
                              Check '/rear/var/log/rear/rear-server1005-rescue.log' why /usr/bin/dracut failed
                              and decide if the recreated system will boot
                              with the initrd 'as is' from the backup restore.

2024-07-02 13:11:06.639258185 Including finalize/Linux-i386/610_EFISTUB_run_efibootmgr.sh

gdha commented at 2024-07-02 14:12:

@saschalucas Did you check '/rear/var/log/rear/rear-server1005-rescue.log' why /usr/bin/dracut failed?

saschalucas commented at 2024-07-02 18:54:

@saschalucas Did you check '/rear/var/log/rear/rear-server1005-rescue.log' why /usr/bin/dracut failed?

yes. /rear/var/log/rear/rear-server1005-rescue.log is attached as rear-recover-server1005.log. Sorry for the renaming ... but I found rescue didn't match the rear workflow. In the log itself there is nothing else like on screen and like mentioned above in the issue:

2024-07-02 13:11:01.185390577 Including finalize/SUSE_LINUX/i386/550_rebuild_initramfs.sh
2024-07-02 13:11:06.369117289 Recreating initrd with /usr/bin/dracut...
2024-07-02 13:11:06.632109467 Warning:
                              Failed to recreate initrd with /usr/bin/dracut.
                              Check '/rear/var/log/rear/rear-server1005-rescue.log' why /usr/bin/dracut failed
                              and decide if the recreated system will boot
                              with the initrd 'as is' from the backup restore.

2024-07-02 13:11:06.639258185 Including finalize/Linux-i386/610_EFISTUB_run_efibootmgr.sh

So the log contains no further hint what went wrong.

jsmeix commented at 2024-07-03 07:08:

@saschalucas
you need at least debug mode '-d'
preferably full debugscript mode '-D'
to be able to see what went wrong.

In general:
Caution with possible secrets in a debug log file:
When 'rear' is run via '-D' in debugscript mode
it logs executed commands via the bash command 'set -x'
that print commands and their arguments as they are executed.
So in particular when arguments contain secret values
(e.g. something like a password or whatever else)
such secret values may appear in the log file.
Also secrets may be stored in some other files
like /var/lib/rear/layout/disklayout.conf
or /var/lib/rear/layout/diskrestore.sh
cf. [password=<password>] in the section
"Disk layout file syntax" in
doc/user-guide/06-layout-configuration.adoc
online at
https://github.com/rear/rear/blob/rear-2.7/doc/user-guide/06-layout-configuration.adoc
So before you attach your debug log file or other files here
(GitHub is a public accessible place), inspect your files
and verify that they do not accidentally cointain secrets.

jsmeix commented at 2024-07-03 07:18:

@saschalucas
because you use OUTPUT=PORTABLE:
What rescue system do you use?
I ask because

In portable mode ReaR assumes that all required software
is available in the rescue system. There are no further
checks for missing software. If you miss some software,
the recovery process will fail.

cf.
https://github.com/rear/rear/blob/master/doc/user-guide/17-Portable-Mode.adoc

saschalucas commented at 2024-07-03 16:59:

-D reveals the problem:

++ LogPrint 'Recreating initrd with /usr/bin/dracut...'
2024-07-03 18:29:35.377244440 Recreating initrd with /usr/bin/dracut...
++ chroot /mnt/local /bin/bash -c 'PATH=/sbin:/usr/sbin:/usr/bin:/bin /usr/bin/dracut --force'

dracut: Cannot find module directory /lib/modules/6.6.14-1-lts/
dracut: and --no-kernel was not specified
++ LogPrintError 'Warning:
Failed to recreate initrd with /usr/bin/dracut.
Check '\''/rear/var/log/rear/rear-server1005-rescue.log'\'' why /usr/bin/dracut failed
and decide if the recreated system will boot
with the initrd '\''as is'\'' from the backup restore.
'
2024-07-03 18:29:35.760132926 Warning:
                              Failed to recreate initrd with /usr/bin/dracut.
                              Check '/rear/var/log/rear/rear-server1005-rescue.log' why /usr/bin/dracut failed
                              and decide if the recreated system will boot
                              with the initrd 'as is' from the backup restore.

The SLES version of finalize/SUSE_LINUX/i386/550_rebuild_initramfs.sh fails here, because it is looking for a kernel from the rescue system systemrescue-11.0. So I'm bitten by a side effect of the portable mode here.

For curiosity on Oracle Linux 8 dracut seems to be called in an other way. Under the same rescue system systemrescue-11.0 the recreation of initramfs seems to work:

2024-06-26 08:21:28.990404381 Including finalize/Fedora/550_rebuild_initramfs.sh
2024-06-26 08:21:28.994816038 Original OLD_INITRD_MODULES=(  )
2024-06-26 08:21:29.006200942 New INITRD_MODULES=' '
2024-06-26 08:21:29.029542327 Running dracut...
2024-06-26 08:21:44.491985680 Updated initrd with new drivers for kernel 4.18.0-513.18.1.0.1.el8_9.x86_64.
2024-06-26 08:21:44.500304291 Running dracut...
2024-06-26 08:21:59.546876691 Updated initrd with new drivers for kernel 4.18.0-513.9.1.el8_9.x86_64.
2024-06-26 08:21:59.556064290 Running dracut...
2024-06-26 08:22:15.147509223 Updated initrd with new drivers for kernel 5.15.0-201.135.6.el8uek.x86_64.
2024-06-26 08:22:15.156579498 Running dracut...
2024-06-26 08:22:30.757426667 Updated initrd with new drivers for kernel 5.15.0-205.149.5.1.el8uek.x86_64.



-------------------------------------------------------------------------------



[Export of Github issue for [rear/rear](https://github.com/rear/rear).]