#2705 PR merged: minimal changes needed for hybrid boot (bios+uefi)

Labels: enhancement, cleanup, fixed / solved / done, hacktoberfest-accepted

DEvil0000 opened issue at 2021-10-29 16:35:

  • Type: New Feature / Enhancement
  • Impact: Normal
  • Reference to related issue (URL): #2698
  • How was this pull request tested?
    created with the following config on a BIOS machine:
OUTPUT=USB
BACKUP=BORG
USB_DEVICE=/dev/disk/by-label/REAR-000
USB_DEVICE_FILESYSTEM_LABEL=REAR-000
USB_DEVICE_PARTED_LABEL=gpt
USB_DEVICE_FILESYSTEM=ext4
USB_UEFI_PART_SIZE="512"
CLONE_USERS=(root)
CLONE_GROUPS=(root)
CLONE_ALL_USERS_GROUPS="false"
BORGBACKUP_REPO="/borg"
BORGBACKUP_UMASK="0002"
BORGBACKUP_ENC_TYPE="repokey"
export BORG_RELOCATED_REPO_ACCESS_IS_OK="yes"
export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK="yes"
export TMPDIR="/wsp_var/tmp/"
USE_RESOLV_CONF="no"
USE_DHCLIENT=no
USE_STATIC_NETWORKING=1
USE_SERIAL_CONSOLE=y
COPY_KERNEL_PARAMETERS=( 'net.ifnames' 'biosdevname', 'console' )
REAR_INITRD_COMPRESSION="fast"
USING_UEFI_BOOTLOADER=1
OUTPUT_URL=usb:///dev/disk/by-label/REARBOOT
USB_BOOTLOADER=grub
GRUB2_DEFAULT_BOOT="1"
USB_BOOT_PART_SIZE="2048"
MODULES=( 'no_modules' )
FIRMWARE_FILES=( 'no' )
EXCLUDE_RUNTIME_LOGFILE="yes"
SERIAL_CONSOLE_DEVICES="/dev/ttyS0"

tested boot on PCEngines APU2 (BIOS) and HP gen9 UEFI.

  • Brief description of the changes in this pull request:
    As discussed in #2698 this is a startingpoint for implementing a hybrid boot supporting BIOS and UEFI from the same media.

  • prerequisit:
    For this to work you must have the following packages installed: grub-efi-amd64-bin and grub-pc-bin. They will install the grub binaries needed for efi and bios boot installation. Don't confuse those packages with grub-efi, grub-efi-amd64 and grub-pc which configure the systems primary boot option and they conflict the other one for a reason I don't quite get. (this is on ubuntu 20.04)

  • format command change
    In addition to the --efi format switch I added a --bios switch. If none is given it will do a hybrid installation.

  • more details:
    This is more like a minimal changeset and prrof of concept but not a full featured or super clean PR.
    Since realizing how this is done in RAWDISK (basically a complete rewrite bypassing lib code and features) and that RAWDISK does use syslinux but not grub (with which I am more familiar) I decided to ignore the RAWDISK for the moment.
    I mostly rearanged the code in 300_format_usb_disk.sh without changing much beside checking the --efi/--bios switches. There are however two fixes to mention: 1) setting esp with parted for EFI to have the correct partition type set. 2) using partprobe more offten to make sure data gets written to the correct locations. I think this code change is very clean.
    The code change in 850_save_sysfs_uefi_vars.sh is a bit more hacky and you may want to do this differently.
    I did not change output/USB/Linux-i386/300_create_grub.sh for the moment since it was not strictly needed but it may be better to install the grub efi binary with grub-install --efi-directory=<path> --target=x86_64-efi --recheck <device> instead of the way done in 850_save_sysfs_uefi_vars.sh and such.

  • future work:

  • while working on this I realized once more that the same basic stuff like partitioning, formatting, writing bootloaders and boot config is written multiple times in the code base and basically every workflow and output method has its own rewrites instead of using some common lib code for those things. I think a major cleanup of those things would make perfect sense. Maybe starting with the partitioning code - moving in a library script and creating functions out of it.

  • Also testing is very time consuming and a bit of a pain due to all those options and combinations. Maybe it would be a good idea to use some build/test server. For example by adding travis jobs or such.
  • Since I only did the work for output=USB and GRUB it may make sense to add this feature to all variants or deprecate some of those. Also keep in mind that GRUB could mount a ISO image and boot from there (stored on boot partition / where the kernel and initrd is stored).
  • When using Serial tty and non Serial tty machines the grub config is at the moment not flexible enough to provide both options. One with ttyS0 and one entry with just tty. This issue may occur everywhere a tty is used in some sort. So thats also some feature or more flexibiliy to consider adding.

Of course I would like to see a merge and the hacktoberfest-accepted label ;)
I am aware that this PR is not touching all config variants and is maybe not super clean. A complete integration on all configuration variants would however mean a quite big change.

edit: please note that there is no 32bit grub efi I know of but it may exist

jsmeix commented at 2021-11-02 08:11:

@DEvil0000
thank you for your enhancement and cleanup work.
It is much appreciated!

I will test it soon...

jsmeix commented at 2021-11-02 08:54:

I think the reason why in ReaR same functionality is implemented
several times in separated code instead of using common library functions
is that contributions happen this way.

I think this is because for an individual contributor it is simpler
and more straightforward to implement what he specifically needs
in his own separated code than to adapt and enhance already existing code
because the latter may cause regressions in special cases of the existing code
or in areas of the existing code where the current contributor has no good knowledge
in particular when the existing code is not well documented so the current contributor
cannot fully understand why existing code is as is so he better does not touch it,
cf. https://github.com/rear/rear/wiki/Coding-Style

Also we ReaR uspream maintainers usually don't have sufficient knowledge
in all those various different areas what ReaR is about (we are not better
than any other contributor) to be able to make generic common functions
from various code pieces that do not have regressions or shortcomings.
In particular we can not at all test all the possible cases of existing code
so in general we don't touch existing code unless we must.
Simply put: There is no "master mind" who could clean it all up.

I think in the end we must accept that in practice
ReaR's code as a whole cannot be clean,
cf. "Dirty hacks welcome" in
https://github.com/rear/rear/wiki/Coding-Style

This does not mean that certain parts of ReaR's code cannot be cleaned up.
It is only meant to explain why ReaR's code is not clean by default.
It is a lot of laborious work to make code clean.

Here a quotation from a colleague at SUSE
where I do 100% agree:

If you look at things from an evolutionary point of view
the solution that works good enough while consuming
least amount of resources wins.
It does not have to be perfect, or even good,
just as good or better than alternatives.
And here I think is the biggest strength of Linux, the pragmatism,
we try out things and keep these that work well enough.
In that sense Linux is full of ugly hacks that, in the end,
resemble how biological systems are structured.
Sure it's hell to maintain, it's complex, we are unable to
build an abstract model the system and reason about it,
but in the end it does the job.
Sometimes I wonder if this is unevitable property
of the universe we live in.
Maybe you can't have nice and clean system
that is efficient enough.

DEvil0000 commented at 2021-11-02 10:59:

@jsmeix I was not asking why it is like it is but more pointing out arear where things can or should improve. Maybe it is a good iead to track such things in tickets and assign a milestone. Maybe lable them as "help welcome" or such.

jsmeix commented at 2021-11-02 13:12:

I tested it as far as I currently use a USB disk:

On my homeoffice laptop
with UEFI in legacy BIOS mode
and a USB disk as /dev/sdb
with this etc/rear/local.conf

FIRMWARE_FILES=( 'no' )
MODULES=( 'loaded_modules' )
PROGRESS_MODE="plain"
PROGRESS_WAIT_SECONDS="3"
SSH_ROOT_PASSWORD="rear"
OUTPUT=USB
USB_DEVICE_FILESYSTEM_PERCENTAGE=10
USB_DEVICE_FILESYSTEM_LABEL='MY-DATA'
USB_BOOT_PART_SIZE=1024
USB_BOOTLOADER="grub"
USB_DEVICE_BOOT_LABEL=MY-BOOT
OUTPUT_URL=usb:///dev/disk/by-label/MY-BOOT
USB_DEVICE_PARTED_LABEL=gpt
BACKUP=NETFS
BACKUP_URL=usb:///dev/disk/by-label/MY-DATA

I get

# usr/sbin/rear -D format /dev/sdb
Relax-and-Recover 2.6 / Git
Running rear format (PID 819 date 2021-11-02 13:06:27)
Command line options: usr/sbin/rear -D format /dev/sdb
Using log file: /root/rear.wingcon-feature_hybrid_boot/var/log/rear/rear-linux-h9wr.log
Using build area: /var/tmp/rear.b3dmAzJw333oeKE
Running 'init' stage ======================
Running workflow format on the normal/original system
Running 'format' stage ======================
USB or disk device /dev/sdb is not formatted with ext2/3/4 or btrfs filesystem
Formatting /dev/sdb will remove all currently existing data on that whole device
UserInput -I USB_DEVICE_CONFIRM_FORMAT needed in /root/rear.wingcon-feature_hybrid_boot/usr/share/rear/format/USB/default/200_check_usb_layout.sh line 62
Type exactly 'Yes' to format /dev/sdb with ext3 filesystem
(default 'No' timeout 300 seconds)
Yes
UserInput: No choices - result is 'Yes'
Repartitioning /dev/sdb
Creating partition table of type gpt on /dev/sdb
The --bios toggle was used with format - making an BIOS bootable device /dev/sdb
Creating BIOS boot partition /dev/sdb1
Setting 'bios_grub' flag on BIOS boot partition /dev/sdb1
The --efi toggle was used with format - making an EFI bootable device /dev/sdb
Creating EFI system partition /dev/sdb2 with size 512 MiB aligned at 8 MiB
Setting 'esp' flag on EFI partition /dev/sdb2
Creating boot partition /dev/sdb3 with size 1024 MiB aligned at 8 MiB
Setting 'legacy_boot' flag on boot partition /dev/sdb3
Creating ReaR data partition /dev/sdb4 up to 10% of /dev/sdb
Creating vfat filesystem on EFI system partition on /dev/sdb2
Creating ext2 filesystem with label 'MY-BOOT' on boot partition /dev/sdb3
Creating ext3 filesystem with label 'MY-DATA' on ReaR data partition /dev/sdb4
Adjusting filesystem parameters on ReaR data partition /dev/sdb4
Exiting rear format (PID 819) and its descendant processes ...

which results this /dev/sdb

# parted -s /dev/sdb unit MiB print
Model: TOSHIBA External USB 3.0 (scsi)
Disk /dev/sdb: 476940MiB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 
Number  Start    End       Size      File system  Name     Flags
 1      0.02MiB  8.00MiB   7.98MiB                primary  bios_grub
 2      8.00MiB  520MiB    512MiB    fat32        primary  boot, esp
 3      520MiB   1544MiB   1024MiB   ext2         primary  legacy_boot
 4      1544MiB  47694MiB  46150MiB  ext3         primary

# lsblk -ipo NAME,KNAME,LABEL,UUID,PTUUID,PARTUUID /dev/sdb
NAME        KNAME     LABEL    UUID                                 PTUUID                               PARTUUID
/dev/sdb    /dev/sdb                                                9b0a1287-b816-4a56-958d-8842d9eb9d65 
|-/dev/sdb1 /dev/sdb1                                               9b0a1287-b816-4a56-958d-8842d9eb9d65 06d5168b-3135-4492-8881-9f0ecaa0d56f
|-/dev/sdb2 /dev/sdb2 REAR-EFI 9F73-4885                            9b0a1287-b816-4a56-958d-8842d9eb9d65 cdb9a83a-233f-4cd7-bd88-59f89512ac89
|-/dev/sdb3 /dev/sdb3 MY-BOOT  fb10f97d-d189-439b-910c-669da4d05214 9b0a1287-b816-4a56-958d-8842d9eb9d65 d4edc850-a5e9-4925-8cbf-86677f551eec
`-/dev/sdb4 /dev/sdb4 MY-DATA  8472ff6f-8a30-4d94-b065-66595eb9052f 9b0a1287-b816-4a56-958d-8842d9eb9d65 584151c2-7ada-4ffb-bfa3-38dc4b3e98a4

For comparison what rear -D format /dev/sdb had resulted before

# parted -s /dev/sdb unit MiB print
Model: TOSHIBA External USB 3.0 (scsi)
Disk /dev/sdb: 476940MiB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 
Number  Start    End       Size      File system  Name     Flags
 1      0.02MiB  8.00MiB   7.98MiB                primary  bios_grub
 2      8.00MiB  1032MiB   1024MiB   ext2         primary  legacy_boot
 3      1032MiB  47694MiB  46662MiB  ext3         primary

# lsblk -ipo NAME,KNAME,LABEL,UUID,PTUUID,PARTUUID /dev/sdb
NAME        KNAME     LABEL   UUID                                 PTUUID                               PARTUUID
/dev/sdb    /dev/sdb                                               546e029a-9ee2-4a33-a747-aa14bfface75 
|-/dev/sdb1 /dev/sdb1                                              546e029a-9ee2-4a33-a747-aa14bfface75 2733c621-ac3a-4a98-af0f-d02da15dec36
|-/dev/sdb2 /dev/sdb2 MY-BOOT 16d6cce4-cef4-4d1a-8fe8-e2f6af0f535a 546e029a-9ee2-4a33-a747-aa14bfface75 ff41080b-6262-4cab-b82d-7c9a0a213dce
`-/dev/sdb3 /dev/sdb3 MY-DATA bbc17933-7a79-46bb-9756-d173a7e9d481 546e029a-9ee2-4a33-a747-aa14bfface75 4bdfeea3-1b96-4832-b63f-820b91a3880a

cf. https://github.com/rear/rear/pull/2662#issuecomment-888310850

After rear mkbackup I got that on my /dev/sdb

# find /tmp/sdb3
/tmp/sdb3
/tmp/sdb3/rear
/tmp/sdb3/rear/linux-h9wr
/tmp/sdb3/rear/linux-h9wr/20211102.1312
/tmp/sdb3/rear/linux-h9wr/20211102.1312/rear-linux-h9wr.log
/tmp/sdb3/rear/linux-h9wr/20211102.1312/initrd.cgz
/tmp/sdb3/rear/linux-h9wr/20211102.1312/kernel
/tmp/sdb3/lost+found
/tmp/sdb3/boot
/tmp/sdb3/boot/grub2
/tmp/sdb3/boot/grub2/i386-pc
/tmp/sdb3/boot/grub2/i386-pc/biosdisk.mod
/tmp/sdb3/boot/grub2/i386-pc/cmdline_cat_test.mod
...
/tmp/sdb3/boot/grub2/i386-pc/zfsinfo.mod
/tmp/sdb3/boot/grub2/fonts
/tmp/sdb3/boot/grub2/fonts/unicode.pf2
/tmp/sdb3/boot/grub2/grub.cfg
/tmp/sdb3/boot/grub2/grubenv
/tmp/sdb3/linux-h9wr

# find /tmp/sdb4
/tmp/sdb4
/tmp/sdb4/lost+found
/tmp/sdb4/rear
/tmp/sdb4/rear/linux-h9wr
/tmp/sdb4/rear/linux-h9wr/20211102.1312
/tmp/sdb4/rear/linux-h9wr/20211102.1312/backup.log
/tmp/sdb4/rear/linux-h9wr/20211102.1312/backup.tar.gz

Booting from that USB disk works well for me
on my other older x86_64 laptop with traditional BIOS
both booting the ReaR recovery system and
chainloading its bootloader on its built-in harddisk
to boot its original system.

jsmeix commented at 2021-11-02 13:15:

@rear/contributors
could you have a look here if you find time for it.

I would like to merge it tomorrow afternoon unless there are objections.

jsmeix commented at 2021-11-03 11:23:

Tested "rear format" and "rear mkbackup" plus "rear recover"
on the same two VMs as in
https://github.com/rear/rear/pull/2703#issuecomment-952888484

original system            replacement system
sda 8 GiB system disk      sda the 8 GiB ReaR "USB" disk from the original system
sdb 8 GiB ReaR "USB" disk  sdb 9 GiB replacement system disk

etc/rear/local.conf

DISKS_TO_BE_WIPED=''
FIRMWARE_FILES=( 'no' )
MODULES=( 'loaded_modules' )
PROGRESS_MODE="plain"
PROGRESS_WAIT_SECONDS="3"
SSH_ROOT_PASSWORD="rear"
OUTPUT=USB
USB_DEVICE_FILESYSTEM_PERCENTAGE=90
USB_DEVICE_FILESYSTEM_LABEL="MY-DATA"
USB_BOOT_PART_SIZE=1024
USB_BOOTLOADER="grub"
USB_DEVICE_BOOT_LABEL="MY-BOOT"
OUTPUT_URL=usb:///dev/disk/by-label/MY-BOOT
USB_DEVICE_PARTED_LABEL=gpt
BACKUP=NETFS
BACKUP_URL=usb:///dev/disk/by-label/MY-DATA

Excerpts:

# usr/sbin/rear -D format /dev/sdb
Relax-and-Recover 2.6 / Git
Running rear format (PID 1355 date 2021-11-03 11:20:57)
Command line options: usr/sbin/rear -D format /dev/sdb
Using log file: /root/rear.wingcon-feature_hybrid_boot/var/log/rear/rear-localhost.log
Using build area: /var/tmp/rear.K0IoksPaoVt9sZD
Running 'init' stage ======================
Running workflow format on the normal/original system
Running 'format' stage ======================
USB or disk device /dev/sdb is not formatted with ext2/3/4 or btrfs filesystem
Formatting /dev/sdb will remove all currently existing data on that whole device
UserInput -I USB_DEVICE_CONFIRM_FORMAT needed in /root/rear.wingcon-feature_hybrid_boot/usr/share/rear/format/USB/default/200_check_usb_layout.sh line 62
Type exactly 'Yes' to format /dev/sdb with ext3 filesystem
(default 'No' timeout 300 seconds)
Yes
UserInput: No choices - result is 'Yes'
Repartitioning /dev/sdb
Creating partition table of type gpt on /dev/sdb
Making a BIOS bootable device /dev/sdb
Creating BIOS boot partition /dev/sdb1
Setting 'bios_grub' flag on BIOS boot partition /dev/sdb1
Making an EFI bootable device /dev/sdb
Creating EFI system partition /dev/sdb2 with size 512 MiB aligned at 8 MiB
Setting 'esp' flag on EFI partition /dev/sdb2
Creating boot partition /dev/sdb3 with size 1024 MiB aligned at 8 MiB
Setting 'legacy_boot' flag on boot partition /dev/sdb3
Creating ReaR data partition /dev/sdb4 up to 90% of /dev/sdb
Creating vfat filesystem on EFI system partition on /dev/sdb2
Creating ext2 filesystem with label 'MY-BOOT' on boot partition /dev/sdb3
Creating ext3 filesystem with label 'MY-DATA' on ReaR data partition /dev/sdb4
Adjusting filesystem parameters on ReaR data partition /dev/sdb4
Exiting rear format (PID 1355) and its descendant processes ...

# parted -s /dev/sdb unit MiB print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sdb: 8192MiB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 
Number  Start    End      Size     File system  Name     Flags
 1      0.02MiB  8.00MiB  7.98MiB               primary  bios_grub
 2      8.00MiB  520MiB   512MiB   fat32        primary  boot, esp
 3      520MiB   1544MiB  1024MiB  ext2         primary  legacy_boot
 4      1544MiB  7373MiB  5829MiB  ext3         primary

# lsblk -ipo NAME,KNAME,LABEL,SIZE /dev/sdb
NAME        KNAME     LABEL     SIZE
/dev/sdb    /dev/sdb              8G
|-/dev/sdb1 /dev/sdb1             8M
|-/dev/sdb2 /dev/sdb2 REAR-EFI  512M
|-/dev/sdb3 /dev/sdb3 MY-BOOT     1G
`-/dev/sdb4 /dev/sdb4 MY-DATA   5.7G

"rear mkbackup" on the original system and
then booting the replacement system from that disk
plus "rear recover" worked well for me.

[Export of Github issue for rear/rear.]