#2656 PR `merged`: fixed output usb format script for none efi cases¶

Labels: bug, fixed / solved / done

DEvil0000 opened issue at 2021-07-12 18:03:¶

Type: Bug Fix
Impact: High
Reference to related issue (URL): most parts of #2648
How was this pull request tested?
output usb on apu1 and apu2 boards - with msdos and gpt using extlinux/syslinux boot.
Brief description of the changes in this pull request:
this PR fixes all blocking bugs for output usb on none efi devices.
** fixing boot by adding gpt boot partition (ef02)
** adding boot partition to get it closer to how efi formatting works
** adding option for gpt boot flag on separate boot partition

Additional note:
you may need to set OUTPUT_URL=usb:///dev/disk/by-label/REAR-BOOT

jsmeix commented at 2021-07-14 10:08:¶

@DEvil0000
I am really not a booting expert so I may ask dumb questions
but more likely I am just confused by the too similar naming
of different things.

In case of GPT with BIOS there is now

        if ! parted -s $RAW_USB_DEVICE unit s mkpart primary $bootloader_partition_start_byte $bootloader_partition_end_byte >&2 ; then
            Error "Failed to create BOOTLOADER system partition on '$RAW_USB_DEVICE'"
        fi
...
    if ! parted -s $RAW_USB_DEVICE unit B mkpart primary $boot_partition_start_byte $boot_partition_end_byte >&2 ; then
        Error "Failed to create BOOT system partition on '$RAW_USB_DEVICE'"
    fi

so first a BOOTLOADER system partition gets created
and then a BOOT system partition gets created
which looks (at least on first glance) as if the same thing happened two times.

But I assume that actually
first one is a small BIOS boot partition for GRUB2 to store its core.img
and
the second one a bigger one for what is in the /boot/ directory
in particular kernel, initrd (i.e. the ReaR recovery system initrd),
and the rest of GRUB2 in /boot/grub2/

FYI:
The ReaR recovery system can be up to about 1G
and when third party backup tools are used even more, cf.
https://github.com/rear/rear/discussions/2640#discussioncomment-908335
and the whole recovery system is in an initrd so depending on initrd compression
the ReaR recovery system initrd can have several hundred MB.

DEvil0000 commented at 2021-07-14 10:47:¶

You are correct about the GPT partitions. The first one contains the core.img of grub and the second one contains the /boot (bootloader config, kernel, initrd). Will add a comment. Also sektor 34 to 2047 for this first partition sounds okay to me but I am not a expert on this also.
1G is what I expected as size for /boot partition. My initrd is about 450MB compressed so two of them fit in. I used a optional USB_BOOT_PART_SIZE defaulting to USB_UEFI_PART_SIZE for a config option.

jsmeix commented at 2021-07-14 11:41:¶

In ReaR we use in general not sectors because their size can vary.
Normally one sector is 512 bytes but it can be also 4K = 4096 bytes per sector.
I guess there are USB disks out there with 4K sector size
in particular for disks >= 2TiB which can be expected for backup disks.

Therefore we use in general only byte values for parted calls.
For us it is no drawback that we must calculate the exact byte values on our own.
Actuall it is an advantage because only with byte values parted does exactly
what is requested without automated rounding or automated alignment
or parted fails if it cannot do what is requested which is what we want.

With 4K sectors the current code creates the BIOS boot partition
up to excluding byte 2048 * 4096 = 8388608 = 8 MiB which matches
USB_PARTITION_ALIGN_BLOCK_SIZE="8" in default.conf
that specifies where the second '/boot' partition starts.

But with 4K sectors the current code would fail if a user sets e.g.
USB_PARTITION_ALIGN_BLOCK_SIZE="4"
in his etc/rear/local.conf

But that is a minor issue.
I would merge your pull request with that minor issue and enhance it
later to make it work with any USB_PARTITION_ALIGN_BLOCK_SIZE

If you like you may make it safe against this issue
by using something like

boot_partition_start_byte=$(( USB_PARTITION_ALIGN_BLOCK_SIZE * MiB_bytes ))
bios_boot_partition_end_byte=$(( boot_partition_start_byte - 1 ))

to let the BIOS boot partition end just before the '/boot' partition starts.

jsmeix commented at 2021-07-14 12:10:¶

As far as I understand
https://en.wikipedia.org/wiki/GUID_Partition_Table

Thus, on a disk with 512-byte sectors ...
the first usable block is LBA 34 or higher.
While on a 4,096-byte sectors disk ...
the first usable block is LBA 6 or higher.

it now depends on what sector 34 means for parted.
Does it always mean 512 * 34 = 17408 bytes (if a sector in parted is always 512 bytes)
or could it also mean 4096 * 34 = 139264 bytes < 0.133 MiB for 4K sectors?

Because even 139264 bytes is much less than 1 MiB
starting the BIOS boot partition at parted sector 34
would always work even down to the minumum of
USB_PARTITION_ALIGN_BLOCK_SIZE="1"

But because according to
https://en.wikipedia.org/wiki/GUID_Partition_Table
a default GPT area (i.e. with up to 128 partitions) goes up to (excluding) byte
max{ 512 * 34 = 17408 , 4096 * 6 = 24576 } = 24576
it should be safe to specify

bios_boot_partition_start_byte=24576

and call parted with bios_boot_partition_start_byte and the above bios_boot_partition_end_byte

And explain all this in exhaustive comments in the code
so that others who may look at that code later
(hopefully after more than 10 years)
understand why the code is as is.

DEvil0000 commented at 2021-07-14 16:48:¶

You are correct, my USB sticks all have 512byte sectors and the code may not be safe for bigger sectors.
Since you used the same source (wiki) the first usable sector for gpt with default length/efi minimum length is max{ 512 * 34 = 17408 , 4096 * 6 = 24576 } = 24576. From my understanding this is 512byte bootloader + 512byte gpt header + 32x512byte for 128 partition entry slots and then the next sector is usable.
I am not sure about that but in my understanding the special purpose partitions are expected in a specific order and without additional empty space/gaps. Sometimes I figured a bios refused to boot when the partition was not within the first X sectors or a expected sector.

Also have a look at wiki: bios boot partition and the svg there (lower half) partition layout for gpt with efi and bios grub this would be the layout for a EFI+BIOS hybrid formatting.

jsmeix commented at 2021-07-15 09:05:¶

https://en.wikipedia.org/wiki/GUID_Partition_Table reads (excerpts)

The UEFI specification stipulates that a minimum of 16,384 bytes,
regardless of sector size, are allocated for the Partition Entry Array.
Thus, on a disk with 512-byte sectors, at least 32 sectors
are used for the Partition Entry Array,
and the first usable block is LBA 34 or higher.
While on a 4,096-byte sectors disk, at least 4 sectors are used
for the Partition Entry Array,
and the first usable block is LBA 6 or higher.
...
the Partition Entry Array describes partitions,
using a minimum size of 128 bytes for each entry block

so this vaules are minumum values but the whole GPT partitioning space
can be arbitrarily bigger e.g. one may have 256 GPT partitions.

The German
https://de.wikipedia.org/wiki/GUID_Partition_Table
describes this more clearly

Die EFI-Spezifikationen schreiben ein Minimum von 16 384 Bytes
für die Partitionstabelle vor, so dass es Platz für 128 Einträge gibt.
...
Die maximale Anzahl möglicher Partitionen hängt von der
Größe der Partitionstabelle ab, die laut Spezifikation mindestens
für 128 Partitionen Platz bietet. Nicht alle Systeme bieten diese
Möglichkeit, jedoch kann z. B. das Partitionierungswerkzeug gdisk
eine größere Partitionstabelle erzeugen.
...
Advantages of GPT; 3. Arbitrary number of partitions - depends
on space allocated for the partition table … By default the GPT table
contains space for defining 128 partitions. However if the user wants
to define more partitions, he/she can allocate more space to the
partition table (currently only gdisk is known to support this feature).

So what the first byte on a GPT disk is that can be used by partitions
depends on the space allocated for the partition table which can be arbitrarily big.

This is why I wrote in
https://github.com/rear/rear/pull/2656#issuecomment-879837623

a default GPT area (i.e. with up to 128 partitions)

because format/USB/default/300_format_usb_disk.sh calls

parted ... mklabel gpt

which creates a default GPT area.

But the size of a default GPT area is different depending on the sector size.

I think if a firmware assumes the first parition starts at a fixed byte value
(e.g. the next byte after a default GPT area with 512 bytes sector size)
that firmware has a severe bug.

Partitions start and end as specified in the partition table
and firmware must read and obey the partition table.

jsmeix commented at 2021-07-15 09:39:¶

When some BIOS firmware refuses to boot from USB disk
there could be various reasons that are often related
to restrictions in the firmware how far it can access data on disks
and additional restrictions how far it can access data on USB disks, cf.
https://github.com/rear/rear/issues/2534

Therefore all data that is needed by the firmware to boot
(in particular all booting related partitions)
should be at the beginning of the disk to avoid possible issues.

DEvil0000 commented at 2021-07-16 16:11:¶

I am quite sure our understanding of this is the same. Maybe I was just not clear enough.
Considering the default size of gpt as in my example it is the byte amount mentioned and the next sector start after that is usable for partitions. This is mostly independent of sector size - the difference is due to the next sector start. GPT however might have a bigger partition table as well in theory but I never saw a bigger one in reality.
Partition alignment is some other thing to consider which does afaik not apply to BOOTLOADER partition. It may in addition be irrelevant for BOOT and EFI partitions but that something I am not sure about.

DEvil0000 commented at 2021-07-16 16:37:¶

Not sure what would be the best way of fixing the sector size (bios boot partition start/end) issue. Would appreciate a merge and fix from you.

jsmeix commented at 2021-07-19 07:23:¶

OK.

@rear/contributors
if there are no objections I would like to merge it as is tomorrow morning and
then I will further adapt and enhance things via a subsequent pull request from me.

[Export of Github issue for rear/rear.]

#2656 PR merged: fixed output usb format script for none efi cases¶