#1374 Issue `closed`: ReaR UEFI image fails booting if secure boot enabled¶

Labels: fixed / solved / done, minor bug

porana opened issue at 2017-06-05 10:54:¶

Relax-and-Recover (ReaR) Issue Template¶

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

rear version (/usr/sbin/rear -V): rear118a-1.18.a-7.1
OS version (cat /etc/rear/os.conf or lsb_release -a):

 
LSB Version:    n/a
Distributor ID: SUSE
Description:    SUSE Linux Enterprise Server for SAP Applications 12 SP2
Release:        12.2
Codename:       n/a

rear configuration files (cat /etc/rear/site.conf or cat /etc/rear/local.conf): /etc/rear/local.conf

OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://xx.xx.xx.xx/home/rear-nfs/
NETFS_KEEP_OLD_BACKUP_COPY=yes
ISO_MKISOFS_BIN=/usr/bin/ebiso
USING_UEFI_BOOTLOADER=1
UEFI_BOOTLOADER=/boot/efi/EFI/sles_sap/shim.efi
REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" snapper chattr lsattr )
COPY_AS_IS=( "${COPY_AS_IS[@]}" /usr/lib/snapper/installation-helper /etc/snapper/config-templates/default )
BACKUP_PROG_INCLUDE=( '/var/cache/*' '/var/lib/mailman/*' '/var/tmp/*' '/var/lib/pgsql/*' '/usr/local/*' '/opt/*' '/var/lib/libvirt/images/*' '/boot/grub2/i386/*' '/var/opt/*' '/srv/*' '/boot/grub2/x86_64/*' '/var/lib/mariadb/*' '/var/spool/*' '/var/lib/mysql/*' '/tmp/*' '/home/*' '/var/log/*' '/var/lib/named/*' '/var/lib/machines/*' )
SSH_ROOT_PASSWORD="xxxxxx"

Are you using legacy BIOS or UEFI boot?
UEFI with secure boot enabled
Brief description of the issue:
rear rescue image fails to boot when secure boot enabled
Work-around, if any:
disable secure boot. boot with rear rescue image and complete rear recover. enable secure boot again.

i have seen similar issue https://github.com/rear/rear/issues/702 with SLES11 SP3.

we are using SLES 12 SP2, is there any change is required for SLES12 SP2 to make it work?

porana commented at 2017-06-06 10:00:¶

One more update:

when we tried to boot the image from EFI shell its giving Security violation error.

FS0:\EFI\BOOT\>BOOTX64.efi 
Command Error Status: Security Violation
FS0:\EFI\BOOT\>

from the above error message, we can interpret that the BOOTX64.efi is missing signature certificate.
Please correct my interpretation if its not correct.

jsmeix commented at 2017-06-07 15:00:¶

@porana
please provide a complete log file with debugging information
for a "rear -d -D mkrescue/mkbackup" run.
You may also have a look at
"Debugging issues with Relax-and-Recover" at
https://en.opensuse.org/SDB:Disaster_Recovery

porana commented at 2017-06-07 19:19:¶

rear-rear199.zip
i have uploaded the complete log file with debug information.

jsmeix commented at 2017-06-09 12:58:¶

I am not at all an UEFI or even Secure Boot expert
but as far as I see there is not something apparently
going wrong in your "rear -d -D mkrescue" log.

@porana
could you re-run "rear -d -D mkrescue" with

KEEP_BUILD_DIR="yes"

in your etc/rear/local.conf file and
then inspect if the contents of your
$TMPDIR/rear.XXXXXXXXXXXXXXX/rootfs/
look o.k. - in particular whether or not all
needed files for UEFI Secure Boot are there?

porana commented at 2017-06-13 07:30:¶

Hi @jsmeix,

Thanks for your time. as per our understanding, secure boot does not have any extra files other than shim.efi. secure boot validates the efi applications(like BOOTX64.efi. shim.efi, grub.efi, etc,...) using secure boot tools. whether it has valid signature on that file?

we have following observations from the log file:

/usr/share/rear/output/ISO/Linux-i386/25_populate_efibootimg.sh
++ cp -v /boot/efi/EFI/sles_sap/shim.efi /tmp/rear.IfJpNOAQSVgsczd/tmp/mnt/EFI/BOOT/BOOTX64.efi
'/boot/efi/EFI/sles_sap/shim.efi' -> '/tmp/rear.IfJpNOAQSVgsczd/tmp/mnt/EFI/BOOT/BOOTX64.efi'

here its copying shim.efi to BOOTX64.efi, here if we verify the signature using sbverify tool, it will show you some signature table has present.

rear199:/var/log/rear # sbverify /boot/efi/EFI/sles_sap/shim.efi
warning: data remaining[1045472 vs 1170720]: gaps between PE/COFF sections?
Signature verification failed

we tried to reproduce the same manually. we can see that shim.efi has signature table present.

rear199:/var/log/rear # cp -v /boot/efi/EFI/sles_sap/shim.efi /tmp/mnt/EFI/BOOT/BOOTX64.efi
'/boot/efi/EFI/sles_sap/shim.efi' -> '/tmp/mnt/EFI/BOOT/BOOTX64.efi'
rear199:/var/log/rear # sbverify /tmp/mnt/EFI/BOOT/BOOTX64.efi
warning: data remaining[1045472 vs 1170720]: gaps between PE/COFF sections?
Signature verification failed

after copying to BOOTX64.efi also, there is no change in the signature table.

after this rear is trying to create bootable grub2 image using grub2-mkimage command.

grub2-mkimage -v -O x86_64-efi -c /tmp/mnt/EFI/BOOT/embedded_grub.cfg -o /tmp/mnt/EFI/BOOT/BOOTX64.efi -p /EFI/BOOT part_gpt part_msdos fat ext2 normal chain boot configfile linux linuxefi multiboot jfs iso9660 usb usbms usb_keyboard video udf ntfs all_video gzio efi_gop reboot search test echo btrfs

after running this comamnd, we verified the BOOTx64.efi with sbverify comamnd. here its giving the error saying no signature table present

rear199:/tmp/mnt/EFI/BOOT # sbverify /tmp/mnt/EFI/BOOT/BOOTX64.efi
No signature table present
Signature verification failed

looks like grub2-mkimage is wiping out the secure boot signature table on BOOTX64.efi.
we are not sure here, is there a way to create signature on the BOOTX64.efi, after grub2-mkimage comamnd executed. or is there a way to retain the signature table while running grub2-mkimage on BOOTX64.efi.

@gdha , @gozora , @pavoldomin do you have any input on this issue, as you guys worked similar issue https://github.com/rear/rear/issues/702 earlier.

gozora commented at 2017-06-13 08:23:¶

Hi @porana,

I'll certainly take a look on this issue but it will take couple of days. I've returned from vacation yesterday evening and need to solve couple of (for me) priority things first.
It would help me if you could upload your recovery image (*.iso) somewhere. You can provide me login credentials to c@gozora.sk if needed.

V.

porana commented at 2017-06-13 10:25:¶

Hi @gozora,

Thanks for your commitment.
i have uploaded the *.iso file to dropbox and i have given permissions for you (c@gozora.sk). you can download it from there. if you have any problem creating account for dropbox, please let me know i can share you the direct link for the iso download.

Thanks & Regards,
Jagan

gozora commented at 2017-06-13 10:49:¶

@porana I've got the iso, thanks.

gozora commented at 2017-06-13 12:43:¶

@porana just by heaving a quick look on this issue please try to download and install latest ReaR version which contains some commits that might help to resolve your issue (9a31a5fa and his predecessors).

I'd recommend first try ReaR 2.1 without following parameters:

USING_UEFI_BOOTLOADER=1
UEFI_BOOTLOADER=/boot/efi/EFI/sles_sap/shim.efi

If you still have trouble with booting please attach output log from rear -d -D mkrescue (located in /var/log/rear/ directory).

V.

porana commented at 2017-06-14 07:00:¶

Hi @gozora,

Even with Rear 2.1 also, its same issue. you can find the attached log file for the same.

rear-rear199.zip
its still not booting with secure boot enabled.

gozora commented at 2017-06-14 08:47:¶

Thanks @porana,
Is Relax-and-Recover (no Secure Boot)** boot option working for you?

V.

porana commented at 2017-06-14 09:37:¶

@gozora I think there was some miscommunication. Let me explain you the issue once again.

Option 1:
if Secure boot enabled at BIOS level on the server, the rescue image fails to boot. we did not even get the grub menu of Relax-and-Recover.

Option 2:
if Secure boot disabled at BIOS level on the server, the same rescue image boots and we get the grub menu of Relax-and-Recover. i.e. it will show all the 4 options on the grub menu.

Relax-and-Recover (no Secure Boot)
Relax-and-Recover (Secure Boot)
Reboot
Exit to EFI shell

in this case both Relax-and-Recover (Secure Boot) and Relax-and-Recover (no Secure Boot) options works fine and we are able to do recovery.

in our case all our severs are default secure boot enabled at BIOS level. so we are not able to boot with the rescue image. if we disable the secure boot at BIOS level, then only we are able to boot with the rescue image.

let us know, if you still does not understand the issue. we will try to send some screen shots.

ipcctv commented at 2017-06-14 21:19:¶

I've been looking at this on HP DL360 Gen 9 servers recently when using
RHEL. In my rear logs, I saw io errors when accessing the NVRAM variables.
Access to this area being blocked for security reasons by secure boot.

On HP servers at least, if you do have a good backup from server A, and
are trying to restore to a different server, you can during boot, use the
advanced features (F9) I think and there is a 'add' option. You get a file
explorer type menu to navigate down to the boot option.

On 14 June 2017 at 10:37, porana notifications@github.com wrote:

@gozora https://github.com/gozora I think there was some
miscommunication. Let me explain you the issue once again.

Option 1:
if Secure boot enabled at BIOS level on the server, the rescue image
fails to boot. we did not even get the grub menu of Relax-and-Recover.

Option 2:
if Secure boot disabled at BIOS level on the server, the same rescue
image boots and we get the grub menu of Relax-and-Recover. i.e. it will
show all the 4 options on the grub menu.

Relax-and-Recover (no Secure Boot)

Relax-and-Recover (Secure Boot)

Reboot

Exit to EFI shell

in this case both Relax-and-Recover (Secure Boot) and Relax-and-Recover
(no Secure Boot) options works fine and we are able to do recovery.

in our case all our severs are default secure boot enabled at BIOS
level. so we are not able to boot with the rescue image. if we disable the
secure boot at BIOS level, then only we are able to boot with the
rescue image.

let us know, if you still does not understand the issue. we will try to
send some screen shots.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/rear/rear/issues/1374#issuecomment-308379113, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AQbHjR57ld5Sp34vh7zfi48H7LMSn_W6ks5sD6n2gaJpZM4Nv5u5
.

porana commented at 2017-06-15 12:21:¶

Hi @gozora, for better understanding of the issue, we have created a video and uploaded to the dropbox. please have a watch.

gozora commented at 2017-06-16 07:27:¶

@porana I'll try to reproduce your problem next days.

porana commented at 2017-06-16 09:15:¶

@gozora Thanks a lot for your time.
hope you understood the issue now.

gozora commented at 2017-06-16 17:23:¶

@porana looks like I'm on the track!

First time in my life when I'm actually happy that something is not booting :-).

V.

porana commented at 2017-06-17 08:56:¶

@gozora, I'm also very happy after seeing that message.
good that you are able to reproduce the issue looks like.

the below SUSE link can give you some insight into the secure boot, how it implemented in SLES.
https://www.suse.com/documentation/sled11/book_sle_admin/data/sec_uefi_secboot.html

gozora commented at 2017-06-18 11:55:¶

Finally I have SLE12 SP2 booted in Secure boot mode!
@porana can you please send me output from following command:

echo "===PK==="; mokutil --pk; echo "===KEK==="; mokutil --kek; echo "===DB==="; mokutil --db

I'm starting to have a dim feeling that current ReaR code can't work correctly with enabled Secure boot ...
I'll continue with investigation in next days.

V.

gozora commented at 2017-06-18 16:23:¶

Hello @porana,

I've created small patch that should resolve your problem.

Please download and install ReaR from my Github page (git clone https://github.com/gozora/rear.git).
If not done already, add UEFI_BOOTLOADER=/boot/efi/EFI/sles_sap/shim.efi to your /etc/rear/local.conf. Then you just need to recreate your ReaR recovery system (rear mkrescue) and try to boot newly created ISO.
With this patch, ReaR booted with Secure boot without any problems.

V.

porana commented at 2017-06-19 12:33:¶

HI @gozora,

Thanks for the patch. it has solved our problem.
now we are able to boot with rescue image when secure boot enabled.

once again, thanks a lot!!!

gozora commented at 2017-06-21 15:46:¶

With #1385 merged, this issue can be closed.

@porana please be aware that we've changed ReaRs behavior a bit.
If you decide to use upstream version be sure to update your configuration as follows:

- UEFI_BOOTLOADER=/boot/efi/EFI/sles_sap/shim.efi
+ SECURE_BOOT_BOOTLOADER="/boot/efi/EFI/sles_sap/shim.efi"

porana commented at 2017-06-22 06:55:¶

@gozora, Thanks
this make sense, instead of just copying the same Boot loader.

jsmeix commented at 2017-06-22 07:37:¶

@gozora
as always you did great debugging and
fixing/improving UEFI related stuff in ReaR.
Many thanks for it!

gozora commented at 2017-06-22 07:40:¶

@jsmeix it was fun after all, and except that I have test environment with working Secure Boot!!!
So, no problem ;-).

[Export of Github issue for rear/rear.]

#1374 Issue closed: ReaR UEFI image fails booting if secure boot enabled¶