#2500 Issue closed: OUTPUT=USB fails to UEFI boot on Lenovo X390 but OUTPUT=RAWDISK works

Labels: enhancement, fixed / solved / done, special hardware or VM

MrManor opened issue at 2020-10-13 19:47:

Relax-and-Recover (ReaR) Issue Template

  • ReaR version ("/usr/sbin/rear -V"):
    Relax-and-Recover 2.6 / 2020-06-17
  • OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):
ituska:~ # cat /etc/os-release
NAME="openSUSE Leap"
VERSION="15.2"
ID="opensuse-leap"
ID_LIKE="suse opensuse"
VERSION_ID="15.2"
PRETTY_NAME="openSUSE Leap 15.2"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:leap:15.2"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"
  • ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
ituska:~ # cat /etc/rear/*.conf | grep -v ^#
OS_VENDOR=SUSE_LINUX
OS_VERSION=15

OUTPUT=USB
USB_DEVICE=/dev/disk/by-label/REAR-000
USING_UEFI_BOOTLOADER=yes
GRUB_RESCUE=n
EXCLUDE_BACKUP+=( fs:/crash )
COPY_AS_IS+=( /usr/src/linux-* )

SSH_FILES='yes'
SSH_UNPROTECTED_PRIVATE_KEYS="yes"

BACKUP=BORG
BORGBACKUP_HOST="raagi.tng.vink-slott.dk"
BORGBACKUP_USERNAME="borg"
BORGBACKUP_REPO="/~/${HOSTNAME}"
BORGBACKUP_PRUNE_KEEP_HOURLY=5
BORGBACKUP_PRUNE_KEEP_WEEKLY=2
BORGBACKUP_COMPRESSION="zlib,8"

BORGBACKUP_ENC_TYPE="keyfile"
export BORG_KEYS_DIR=~root/keys
export BORG_PASSPHRASE="****"
COPY_AS_IS_BORG=( $BORG_KEYS_DIR )

export BORG_RELOCATED_REPO_ACCESS_IS_OK="yes"
export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK="yes"

  • Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR):
    Lenovo X390

  • System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
    x86

  • Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
    UEFI with GRUB2

  • Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
    localSSD

  • Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT" or "lsblk" as makeshift):

NAME                    KNAME          PKNAME         TRAN   TYPE  FSTYPE        SIZE MOUNTPOINT
/dev/sda                /dev/sda                      usb    disk  iso9660       7.2G 
|-/dev/sda1             /dev/sda1      /dev/sda              part  vfat          400M 
`-/dev/sda2             /dev/sda2      /dev/sda              part  ext3          6.8G 
/dev/nvme0n1            /dev/nvme0n1                  nvme   disk                477G 
|-/dev/nvme0n1p1        /dev/nvme0n1p1 /dev/nvme0n1   nvme   part  vfat          500M /boot/efi
|-/dev/nvme0n1p2        /dev/nvme0n1p2 /dev/nvme0n1   nvme   part  crypto_LUKS   147G 
| `-/dev/mapper/cr_root /dev/dm-1      /dev/nvme0n1p2        crypt xfs           147G /
|-/dev/nvme0n1p3        /dev/nvme0n1p3 /dev/nvme0n1   nvme   part  crypto_LUKS 314.1G 
| `-/dev/mapper/cr_home /dev/dm-2      /dev/nvme0n1p3        crypt xfs         314.1G /home
`-/dev/nvme0n1p4        /dev/nvme0n1p4 /dev/nvme0n1   nvme   part  crypto_LUKS  15.4G 
  `-/dev/mapper/cr_swap /dev/dm-0      /dev/nvme0n1p4        crypt swap         15.4G [SWAP]

(/dev/sda is target USB stick for rear)

  • Description of the issue (ideally so that others can reproduce it):
    Backup seems to complete without major issues, and content in backup target. But when I try to select the USB for boot, the Lenovo returns to the boot menu within seconds

  • Workaround, if any:
    No idea

  • Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
    rear -D mkrescue

verbatim content from a prevous run

ituska:~ # rear -v mkbackup                                    
Relax-and-Recover 2.6 / 2020-06-17
Running rear mkbackup (PID 4163)
Using log file: /var/log/rear/rear-ituska.log
Running workflow mkbackup on the normal/original system
Using UEFI Boot Loader for Linux (USING_UEFI_BOOTLOADER=1)
Using autodetected kernel '/boot/vmlinuz-5.3.18-lp152.44-default' as kernel in the recovery system
Creating disk layout
Overwriting existing disk layout file /var/lib/rear/layout/disklayout.conf
Using sysconfig bootloader 'grub2-efi'
Verifying that the entries in /var/lib/rear/layout/disklayout.conf are correct ...
Creating recovery system root filesystem skeleton layout
To log into the recovery system via ssh set up /root/.ssh/authorized_keys or specify SSH_ROOT_PASSWORD
Trying to find what to use as UEFI bootloader...
Trying to find a 'well known file' to be used as UEFI bootloader...
Using '/boot/efi/EFI/opensuse/grubx64.efi' as UEFI bootloader file
Copying logfile /var/log/rear/rear-ituska.log into initramfs as '/tmp/rear-ituska-partial-2020-10-13T21:06:19+02:00.log'
Copying files and directories
Copying binaries and libraries
Copying all kernel modules in /lib/modules/5.3.18-lp152.44-default (MODULES contains 'all_modules')
Copying all files in /lib*/firmware/
Testing that the recovery system in /tmp/rear.vR2n5ifUx2qbJXr/rootfs contains a usable system
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (357287957 bytes) in 33 seconds
GRUB2 modules to load: cryptodisk fat gcry_rijndael gcry_sha256 luks part_gpt xfs
Saved /var/log/rear/rear-ituska.log as rear/ituska/20201013.2106/rear-ituska.log
Creating backup archive 'rear_6' in Borg repository /~/ituska on raagi.tng.vink-slott.dk
Creating archive at "ssh://borg@raagi.tng.vink-slott.dk:22/~/ituska::rear_6"
Pruning old backup archives in Borg repository /~/ituska on raagi.tng.vink-slott.dk
Exiting rear mkbackup (PID 4163) and its descendant processes ...
Running exit tasks

MrManor commented at 2020-10-13 19:51:

rear-ituska.log
Log from rear -D mkrescue

gozora commented at 2020-10-13 21:43:

Hello @MrManor,

What kernel version are you using?

Maybe it is https://github.com/rear/rear/issues/2254 related?

V.

MrManor commented at 2020-10-14 06:38:

What kernel version are you using?

Linux ituska 5.3.18-lp152.44-default #1 SMP Wed Sep 30 18:51:43 UTC 2020 (914f31e) x86_64 x86_64 x86_64 GNU/Linux

Maybe it is #2254 related?
I dont think so - To me seems that the PC never starts to load the kernel. When I select the USB stick from the bios menu with possible devices for boot, the PC switches to a black (textmode?) screen for a less than 1 sec - and then returns to the bios boot select menu. So I was more guessing that I (or ReaR) got something UEFI related wrong.
That was also why I tried inserting the USING_UEFI_BOOTLOADER statement above.

gozora commented at 2020-10-14 07:08:

Can you try with ReaR 2.5 ?
We have introduced new way how EFI images (*.efi) are created in ReaR 2.6 ...

Maybe it will help, and you finally hit the problem from #2254 :-).

V.

MrManor commented at 2020-10-14 07:55:

Can you try with ReaR 2.5 ?

You want me to downgrade? I am running 2.6 / 2020-06-17 at the moment.

We have introduced new way how EFI images (*.efi) are created in ReaR 2.6 ...

It is a long tread - I'll will have to dig into it after work...

Maybe it will help, and you finally hit the problem from #2254 :-).

Well that would be progress, wouldn't it :-)

As I have never worked with ReaR before: I would like to know what to expect when I some day manage to boot from the USB stick? Will it load a GRUB menu, a kernel starting to load, or some ReaR related menu?

MrManor commented at 2020-10-14 16:10:

So I removed ReaR 2.6 and found Rear v. 2.3 in OpenSUSE's official repository. I removed "USING_UEFI_BOOTLOADER=yes" from the config, and did another run. No difference - the Lenovo still refuses to boot.

gozora commented at 2020-10-14 17:59:

That is actually a good sign, means one problem less.
When time permits. I'll install OpenSUSE 15.2 with ReaR 2.6 on USB and Borg back-end and try to reproduce your problem.

V.

jsmeix commented at 2020-10-16 11:38:

@MrManor
I am not at all a booting expert and even less an expert with UEFI booting
so just as a basically blind shot into the dark:

Do you perhaps have other hardware that can boot from USB?
If yes could you try if the USB with the ReaR recovery system boots on different hardware?
Of course you would not run "rear recover" on the other hardware but there should be no harm
when you only boot the ReaR recovery system and when it boots only shut it down.

My reason behind is that I noticed that you have a NVMe disk
which indicates that the Lenovo X390 is rather modern hardware
and perhaps this particular hardware behaves somehow special?

@gozora
for SUSE systems we have
usr/share/rear/finalize/SUSE_LINUX/i386/675_install_shim.sh
cf. https://github.com/rear/rear/issues/2116
therein in particular UEFI (secure boot NOT enabled) + GRUB2-EFI
so it seems on SUSE systems calling "shim-install" is needed in any case for UEFI
so it may be also needed to make a UEFI bootable USB device
but I am no expert here and may confuse things.

jsmeix commented at 2020-10-16 12:30:

@MrManor
regarding how things look like when booting from USB via UEFI and GRUB
you may have a look at
https://github.com/rear/rear/issues/2276
where @gozora posted some screenshots, e.g.
https://github.com/rear/rear/issues/2276#issuecomment-559269612
https://github.com/rear/rear/issues/2276#issuecomment-562843028
how the GRUB menue would look like
provided the UEFI firmware was able to load and run GRUB which seems to fail here.
Then in the GRUB menue you could select what kernel (and initrd) GRUB should load.

MrManor commented at 2020-10-16 15:31:

Do you perhaps have other hardware that can boot from USB?

I found a very old desktop, one capable of both EFI and legacy boot. I am sure no expert on UEFI boot either, so I am not sure if the "U" makes any difference.

Tried the USB keys made using ReaR v. 2.3 and also the one made using ReaR v. 2.6. They both loaded successfully when selecting EFI from the PC boot menu. I came to a grub menu in both cases, although only one menu item from the v2.3 generation. From the v 2.6 I selected "Secure boot" and in both cases I managed to boot up all the way, to a root login prompt.

MrManor commented at 2020-10-16 15:39:

My reason behind is that I noticed that you have a NVMe disk
which indicates that the Lenovo X390 is rather modern hardware
and perhaps this particular hardware behaves somehow special?

Yes it is quite new. But it should be possible to build a USB boot stick for it. After all, it came with windows and I replaced it using a LEAP 15.2 install placed on a USB

OliverO2 commented at 2020-11-09 21:11:

@MrManor
The RAWDISK output method in ReaR has its independent implementation for creating bootable media. It creates an image file which can be copied with dd to a USB stick. You could quickly try if that works for you:

  1. In your ReaR configuration file:
    • Change OUTPUT=USB to OUTPUT=RAWDISK
    • Add OUTPUT_URL="file:///var/lib/rear/output"
    • If secure boot is enabled, set SECURE_BOOT_BOOTLOADER to your system's shim path, e.g. SECURE_BOOT_BOOTLOADER="/boot/efi/EFI/BOOT/shim.efi" as described in default.conf.
  2. Run sudo rear mkrescue
  3. Copy the image to your USB stick:
    • Use lsblk -o +MODEL to find the correct disk device path for your USB stick (typically /dev/sdX).
    • RE-CHECK. The following command will overwrite the entire USB stick.
    • Use the following command, replacing USB-DEVICE with the USB stick’s device name. sudo zcat "/var/lib/rear/output/$(hostname)/rear-$(hostname).raw.gz" | sudo dd bs=1M of=/dev/USB-DEVICE

MrManor commented at 2020-11-10 19:25:

@OliverO2: Thanks for looking into this problem.

I found the shim.efi and modified site.conf like this (rest of site.conf is like shown in my first post):

### Rescue boot ###
OUTPUT=RAWDISK
OUTPUT_URL="file:///var/lib/rear/output"
SECURE_BOOT_BOOTLOADER="/boot/efi/EFI/opensuse/shim.efi"
USB_DEVICE=/dev/disk/by-label/REAR-000
#USING_UEFI_BOOTLOADER=yes
GRUB_RESCUE=n
#ONLY_INCLUDE_VG=( "vg00" )
#EXCLUDE_BACKUP+=( fs:/crash fs:/usr/sap fs:/oracle )
EXCLUDE_BACKUP+=( fs:/crash )
COPY_AS_IS+=( /usr/src/linux-* )

Ran command as root

ituska:~ # rear mkrescue
ERROR: Could not copy result files to /var/lib/rear/output/ituska at file location
Aborting due to an error, check /var/log/rear/rear-ituska.log for details
Terminated

But it failed. I really cant guess why. There is plenty of space on disk, the ../output catalog is there, but not mutch in it.

klaus@ituska:~> sudo tree /var/lib/rear/output/ituska
/var/lib/rear/output/ituska
├── rear-ituska.log
└── VERSION

rear-ituska.log

By the way. This using the downgraded 2.3 version of Rear as suggested by @gozora earlier.

OliverO2 commented at 2020-11-10 20:35:

Thanks for providing detailed information.

As OUTPUT=RAWDISK premiered in ReaR v2.4, it was not used in your last run with v2.3. So you need to use a more recent version of ReaR. Version 2.6 will be fine.

MrManor commented at 2020-11-10 21:17:

Success! I was able to boot into the rescue system using the usb build from rear mkrescue.

OliverO2 commented at 2020-11-10 21:18:

Excellent! Thanks for your feedback!

MrManor commented at 2020-11-10 22:07:

So now I know how to build a usb for rescue boot. Not very user frendly - but it can boot.
Can I modify my settings to use this way of creating a boot stick during normal rear mkbackup? (I did try to just change RAW back to USB and run rear mkbackup, but this was no good)

OliverO2 commented at 2020-11-10 22:19:

RAWDISK was created to have bootable disk images which can be transferred to any kind of disk (USB stick or other). RAWDISK recovery images are expected to be stored somewhere in the regular file system and dd'ed to a USB stick when necessary. In my case, I store those disk images on backup media right next to the backup data.

So no, there is no built-in mechanism to automatically transfer RAWDISK images to a USB stick during mkbackup.

jsmeix commented at 2020-11-11 07:52:

Only as a side note about a similar issue see
https://github.com/rear/rear/issues/2210

Accordingly something like

rear mkbackup && dd bs=1M if=<RAWDISK_image_file> of=<USB_disk_device>

could do "all in one" automatically
but it mercilessly overwrites any <USB_disk_device> that exists at that time
so manually overwriting the right USB device is probably safer.

Alternatively "inside" ReaR one could use POST_BACKUP_SCRIPT
https://github.com/rear/rear/blob/master/usr/share/rear/conf/default.conf#L3145
but POST_BACKUP_SCRIPT only works for the mkbackup/mkbackuponly workflows
(i.e. not for "rear mkrescue").

jsmeix commented at 2020-11-11 07:59:

FYI
the error exit in https://github.com/rear/rear/issues/2500#issuecomment-724915242
that shows up in https://github.com/rear/rear/files/5519698/rear-ituska.log as

2020-11-10 20:15:01.486152863 Copying result files '/tmp/rear.MDScTcSPRhEjgI8/tmp/VERSION /tmp/rear.MDScTcSPRhEjgI8/tmp/README /tmp/rear.MDScTcSPRhEjgI8/tmp/rear-ituska.log' to /var/lib/rear/output/ituska at file location
cp: cannot stat '/tmp/rear.MDScTcSPRhEjgI8/tmp/README': No such file or directory
2020-11-10 20:15:01.489660275 ERROR: Could not copy result files to /var/lib/rear/output/ituska at file location

was fixed via https://github.com/rear/rear/pull/2147 for ReaR 2.6

jsmeix commented at 2020-11-11 08:17:

@OliverO2
thank you for your help here!

FYI:
I improved the RAWDISK dd example command in default.conf via
https://github.com/rear/rear/commit/7c6949fe228eee7aaf869f64f15e77546bbd126e

OliverO2 commented at 2020-11-11 08:23:

@jsmeix
Good point. While you're at it, you might consider compression (enabled by default) and change the line to something similar as posted above:

sudo zcat "$image_file" | sudo dd bs=1M of="$disk_device"

jsmeix commented at 2020-11-11 09:00:

@OliverO2
thank you to point that out. I did
https://github.com/rear/rear/commit/8f09ede7d617290e948dd779539d4da385d454e3

MrManor commented at 2020-11-11 19:16:

Thanks a lot for your attention and pointing me to a workable solution. I guess this issue can be closed by that. Sorry if my limited English made me miss some points during this issue, but I do suspect others will end up in the same situation like me - pussled by not being able to boot after running rear mkbackup.

jsmeix commented at 2020-11-12 07:31:

@MrManor
thank you for your issue report and your cooperative feedback.
Don't worry about your English - it is more than good enough.

I reopen the issue because we need some enhancement in ReaR
to make a UEFI bootable USB medium for the OUTPUT=USB method
that works as good as OUTPUT=RAWDISK already does.

But I am not at all a sufficient UEFI expert to implement that myself.
So someone else would have to compare the OUTPUT=USB code
with the OUTPUT=RAWDISK code to find out what OUTPUT=RAWDISK
does better and then improve the OUTPUT=USB code accordingly.

Because all that would have to happen on a voluntary base
it means it is unknown if that would happen and when.
Accordingly the "Milestone" of this issue is "ReaR future",
cf. https://github.com/rear/rear/milestones

So at least for now using OUTPUT=RAWDISK
plus manual creating the bootable USB medium
is the right way for those cases where OUTPUT=USB
fails to make a UEFI bootable USB medium.

MrManor commented at 2020-11-17 18:50:

Thanks for your feedback - and I must say I was very unsure if I should close the issue. I could not really grasp from your feedback if you considered the problem as solved.
I will stay following this issue and please let me know if I can of any assistance in testing.

github-actions commented at 2021-01-17 02:56:

Stale issue message

gdha commented at 2022-02-28 15:50:

A customer using IoT Lenovo SE30 with Ubuntu 20.04 seems to have the same issue as above issue is describing - by adding
REAR_INITRD_COMPRESSION=lzma

to local.conf file and reformatted the USB device and then made the image.

It fails to boot.
Then I go into and edit the boot strings and add (hd0,1) to both, kernel and initrd lines.
Then I boot, it takes a good 3 min and guess what, I get the “ReaR” login screen now, awesome!!!

gdha commented at 2022-03-03 16:24:

Note: have a closer look at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=755256
When we are using REAR_INITRD_COMPRESSION=lzma
we should include with grub2 configuration the line:
insmod xzio
and perhaps also something like:
set root='hd0,msdos1'

jsmeix commented at 2022-03-09 14:02:

I think that with https://github.com/rear/rear/pull/2763 merged
this issue should be fixed - if not we can reopen it.

Likely not fully automated everywhere but the new
GRUB2_SEARCH_ROOT_COMMAND config variable
should at least provide "final power to the user"
to specify what he needs in his particular case.

[Export of Github issue for rear/rear.]