#3093 Issue closed: AWS EC2 Xen based Amazon Linux 2 : ReaR won't consider BIOS boot partition (/dev/xvda128 was not created by kernel < 5.14)

Labels: support / question, fixed / solved / done, not ReaR / invalid, special hardware or VM

ramzcode opened issue at 2023-11-27 07:13:

  • ReaR version ("/usr/sbin/rear -V"):
    2.7

  • OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):
    Amazon Linux 2

  • ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

# Default is to create Relax-and-Recover rescue media as ISO image
# set OUTPUT to change that
# set BACKUP to activate an automated (backup and) restore of your data
# Possible configuration values can be found in /usr/share/rear/conf/default.conf
#
# This file (local.conf) is intended for manual configuration. For configuration
# through packages and other automated means we recommend creating a new
# file named site.conf next to this file and to leave the local.conf as it is.
# Our packages will never ship with a site.conf.
RAWDISK_IMAGE_NAME="NEW-TESTRAWV2"
OUTPUT=RAWDISK
KEEP_BUILD_DIR="yes"
BACKUP=NETFS
USE_DHCLIENT=yes
OUTPUT_URL=nfs://172.31.3.205/backup/linux
BACKUP_URL=nfs://172.31.3.205/backup/linux

  • Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR):
    AWS

  • System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
    x86

  • Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
    BIOS

  • Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
    NVMe Local Drives

  • Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"):

NAME         KNAME      PKNAME    TRAN TYPE FSTYPE LABEL SIZE MOUNTPOINT
/dev/xvda    /dev/xvda                 disk                8G
`-/dev/xvda1 /dev/xvda1 /dev/xvda      part xfs    /       8G /
  • Fdisk Output ("fdisk -l /dev/DEVICENAME")
Device       Start      End  Sectors Size Type
/dev/xvda1    4096 16777182 16773087   8G Linux filesystem
/dev/xvda128  2048     4095     2048   1M BIOS boot

Partition table entries are not in disk order.

    Device                                       Start                      End                  Sectors                 Size Type
>>  /dev/xvda1                                    4096                 16777182                 16773087                   8G Linux filesystem
    /dev/xvda128                                  2048                     4095                     2048                   1M BIOS boot
  • Description of the issue (ideally so that others can reproduce it):

ReaR ignores or fails to record the Boot or Partition that is not listed under lsblk but available under fdisk or cfdisk

  • Workaround, if any:
    Still not available

jsmeix commented at 2023-11-27 11:24:

I have neither a AWS EC2 system nor do I use Amazon Linux 2
so I canot tell how ReaR behaves with that.

To have at least a chance to imagine
what goes on on your particular system
we need at least a "rear -D mkrescue" debug log file, cf.
https://github.com/rear/rear/blob/master/.github/ISSUE_TEMPLATE.md

jsmeix commented at 2023-11-27 11:33:

In contrast to this issue here
the similar issue there
https://github.com/rear/rear/issues/3090#issue-2006425127
shows '/dev/nvme0n1p128' in the 'lsblk' output.
A difference seems to be that
here it is on a virtual disk '/dev/xvda128'
while there it is on real hardware '/dev/nvme0n1p128'.

ramzcode commented at 2023-11-27 11:51:

@jsmeix Yes, you are correct, nvme** based VM's are fine but with XVDA it is not.

Debug Log:

shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
pwd: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
Relax-and-Recover 2.7 / Git
Running rear mkrescue (PID 15607 date 2023-11-27 11:48:05)
Command line options: /usr/sbin/rear -D mkrescue
Using log file: /var/log/rear/rear-ip-172-31-5-70.log
Using build area: /var/tmp/rear.LmaHbT3FNaxrYVs
Running 'init' stage ======================
Running workflow mkrescue on the normal/original system
Running 'prep' stage ======================
Using autodetected kernel '/boot/vmlinuz-5.10.199-190.747.amzn2.x86_64' as kernel in the recovery system
Running 'layout/save' stage ======================
Creating disk layout
Overwriting existing disk layout file /var/lib/rear/layout/disklayout.conf
Disabling excluded components in /var/lib/rear/layout/disklayout.conf
Using guessed bootloader 'EFI' for 'rear recover' (found in first bytes on /dev/xvda)
Verifying that the entries in /var/lib/rear/layout/disklayout.conf are correct
Created disk layout (check the results in /var/lib/rear/layout/disklayout.conf)
Running 'rescue' stage ======================
Creating recovery system root filesystem skeleton layout
Adding net.ifnames=0 to KERNEL_CMDLINE
Adding biosdevname=0 to KERNEL_CMDLINE
Handling network interface 'eth0'
eth0 is a physical device
Handled network interface 'eth0'
Skipping 'virbr0': not bound to any physical interface.
Included current keyboard mapping (via 'dumpkeys -f')
Included default US keyboard mapping /lib/kbd/keymaps/legacy/i386/qwerty/defkeymap.map.gz
Included other keyboard mappings in /lib/kbd/keymaps
Copying logfile /var/log/rear/rear-ip-172-31-5-70.log into initramfs as '/tmp/rear-ip-172-31-5-70-partial-2023-11-27T11:48:09+0000.log'
Running 'build' stage ======================
Copying files and directories
Copying binaries and libraries
Copying all kernel modules in /lib/modules/5.10.199-190.747.amzn2.x86_64 (MODULES contains 'all_modules')
Copying all files in /lib*/firmware/
Skip copying broken symlink '/etc/mtab' target '/proc/3968/mounts' on /proc/ /sys/ /dev/ or /run/
Removed SSH key files without passphrase from recovery system (SSH_UNPROTECTED_PRIVATE_KEYS not true): etc/ssh/ssh_host_ecdsa_key etc/ssh/ssh_host_ed25519_key etc/ssh/ssh_host_rsa_key
Testing that the recovery system in /var/tmp/rear.LmaHbT3FNaxrYVs/rootfs contains a usable system
Testing each binary with 'ldd' and look for 'not found' libraries within the recovery system
Testing that the existing programs in the PROGS array can be found as executable command within the recovery system
Testing that each program in the REQUIRED_PROGS array can be found as executable command within the recovery system
Running 'pack' stage ======================
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (185261172 bytes) in 29 seconds
Running 'output' stage ======================
Creating 202 MiB raw disk image "trianz-ip-172-31-5-70.raw"
Using syslinux to install a Legacy BIOS bootloader
Copying resulting files to nfs location
Saving /var/log/rear/rear-ip-172-31-5-70.log as rear-ip-172-31-5-70.log to nfs location
Copying result files '/var/tmp/rear.LmaHbT3FNaxrYVs/tmp/trianz-ip-172-31-5-70.raw /var/tmp/rear.LmaHbT3FNaxrYVs/tmp/VERSION /var/tmp/rear.LmaHbT3FNaxrYVs/tmp/README /var/tmp/rear.LmaHbT3FNaxrYVs/tmp/rear-ip-172-31-5-70.log' to /var/tmp/rear.LmaHbT3FNaxrYVs/outputfs/ip-172-31-5-70 at nfs location
Exiting rear mkrescue (PID 15607) and its descendant processes ...
Running exit tasks
To remove the build area you may use (with caution): rm -Rf --one-file-system /var/tmp/rear.LmaHbT3FNaxrYVs

Conf File:

RAWDISK_IMAGE_COMPRESSION_COMMAND=''
RAWDISK_IMAGE_NAME="trianz-$HOSTNAME"
OUTPUT=RAWDISK
BACKUP=NETFS
USE_DHCLIENT=yes
OUTPUT_URL=nfs://172.31.3.205/backup/linux
BACKUP_URL=nfs://172.31.3.205/backup/linux
FULLBACKUPDAY=(`date +%a`)
BACKUP_TYPE=incremental
CLONE_ALL_USERS_GROUPS="false"
CLONE_USERS=('rearusernew')
PROGS=( "${PROGS[@]}" 'sudo' 'python' 'su' 'lsattr' 'lzop' 'gzip' )
BACKUP_PROG_EXCLUDE=( "${BACKUP_PROG_EXCLUDE[@]}" '/var/tmp/rear.*' '/usr/lib/systemd/system/gce*' '/usr/lib/systemd/system/google*' '/etc/dhcp/dhclient.d/google_hostname.sh' '/etc/wicked/extensions/hostname')
COPY_AS_IS=( "${COPY_AS_IS[@]}" '/etc/security/*' '/var/log/btmp' '/usr/bin/nohup' '/usr/bin/sudo' '/usr/sbin/mount.cifs' '/usr/lib64/libcrypto*' '/usr/libexec/sudo/*' '/usr/libexec/*python*' '/etc/alternatives/*python*' '/usr/lib64/*python*' '/usr/lib64/libyaml-0.so.2*' '/usr/bin/pyvenv*' '/usr/bin/python*' '/usr/libexec/os-probes/*' '/usr/lib/grub/i386-pc/*' '/usr/bin/os-prober' '/usr/sbin/mount*' '/usr/lib*/*python*' '/var/lib/ca-certificates/*' '/usr/lib*/*pam*' '/usr/lib*/security/pam*' '/etc/security/pam_env.conf' '/etc/pam*' '/etc/sudoers' '/rearusernew' '/etc/shadow' '/etc/ssh/*' '/etc/security/limits.conf' '/etc/authselect' '/etc/apparmor.d/abstractions/libpam-systemd' '/usr/lib/x86_64-linux-gnu/libpam*' '/usr/lib/x86_64-linux-gnu/security/pam*' '/lib/x86_64-linux-gnu/security/pam*' '/usr/share/pam*' '/var/lib/pam' '/lib64/security/*pam*' '/lib64/*pam*' '/usr/lib/sudo' '/usr/lib/snapper/installation-helper' '/etc/snapper/config-templates/default' '/etc/login.defs' '/etc/environment' '/etc/securetty' '/etc/default/locale' '/boot/*')
KEEP_BUILD_DIR="yes"

jsmeix commented at 2023-11-27 12:03:

@ramzcode
in your
https://github.com/rear/rear/issues/3093#issuecomment-1827682900
your "Debug Log" is only the screen output
but we need the "rear -D mkrescue" debug log file
i.e. your "log file: /var/log/rear/rear-ip-172-31-5-70.log".

In general:
Caution with possible secrets in a full debug log file:
When 'rear' is run via '-D' in debugscript mode
it logs executed commands via the bash command 'set -x'
that print commands and their arguments as they are executed
so in particular when arguments contain secret values
(e.g. something like a password or whatever else)
such secret values may appear in the log file.
Also secrets may be stored in some other files
like /var/lib/rear/layout/disklayout.conf
or /var/lib/rear/layout/diskrestore.sh
cf. [password=<password>] in the section
"Disk layout file syntax" in
doc/user-guide/06-layout-configuration.adoc
online at
https://github.com/rear/rear/blob/rear-2.7/doc/user-guide/06-layout-configuration.adoc
So before you attach your full debug log file and other files
here (GitHub is a public accessible place) inspect your files
and verify that they do not accidentally cointain secrets.

ramzcode commented at 2023-11-27 13:21:

@jsmeix Sorry for the confusion, my bad mistook the question. Attached the full log file for your reference. I just want to highlight my finding that the /dev/xvda128 don't even exist as well. and i don't see it under the dmesg log as well.

rear-ip-172-31-5-70.log

pcahyna commented at 2023-11-27 13:39:

Hi @ramzcode what problem do you actually encounter? Is it the rescue image not booting? Failing to recover? Recovering but then not being able to boot the recovered system? Please specify. What does it mean that "ReaR ignores or fails to record the Boot or Partition that is not listed under lsblk but available under fdisk or cfdisk" ? How does the problem manifest?

Please also attach the file /var/lib/rear/layout/disklayout.conf if you believe that ReaR failed to record some partition.

ramzcode commented at 2023-11-27 14:19:

@pcahyna Hi, I have included my query in the first block of this thread. Backup and Recovery via RAW based disk image boots, but while setting up the disk layout and installing the boot loader on target VM ReaR complains that no BIOS type partitions found. This is because from the source the Disk Layout builder of ReaR failed to identify the BIOS boot Partition i.e. /dev/xvda128.

Now the prob is recovery works but it won't boot due to improper partition structure on the target.

NOTE: If you notice LSBLK will not list the 128 Partition that is boot type. and ReaR relies on lsblk for building Disk layout as of my understanding.

image

pcahyna commented at 2023-11-27 15:06:

NOTE: If you notice LSBLK will not list the 128 Partition that is boot type. and ReaR relies on lsblk for building Disk layout as of my understanding.

My understanding is that ReaR uses parted, not lsblk. Please provide the output of parted -m -s /dev/xvda print and also the /var/lib/rear/layout/disklayout.conf file.

It is indeed weird that lsblk does not show /dev/xvda128. Does this device node exist? What does ls -l /dev/xvda128 show?

jsmeix commented at 2023-11-27 15:46:

As far as I see in
https://github.com/rear/rear/files/13474729/rear-ip-172-31-5-70.log
the reason is that /dev/xvda128 is not in /sys/block/*
(excerpts)

+ source /usr/share/rear/layout/save/GNU/Linux/200_partition_layout.sh
...
++ for disk in '/sys/block/*'
++ blockd=loop0
...
++ for disk in '/sys/block/*'
++ blockd=loop1
...
++ for disk in '/sys/block/*'
++ blockd=loop2
...
++ for disk in '/sys/block/*'
++ blockd=loop3
...
++ for disk in '/sys/block/*'
++ blockd=loop4
...
++ for disk in '/sys/block/*'
++ blockd=loop5
...
++ for disk in '/sys/block/*'
++ blockd=loop6
...
++ for disk in '/sys/block/*'
++ blockd=loop7
...
++ for disk in '/sys/block/*'
++ blockd=xvda
++ devname=/dev/xvda

so '/sys/block/xvda' is the only one in '/sys/block/*'
that has a matching disk device '/dev/xvda'.

From my current point of view it is
no issue in ReaR when basic system tools (like 'lsblk')
and basic system files and directories (like '/sys/block/')
do not show or contain "usually expected" things.

From my current point of view in such cases it means
the system is somehow not sufficiently normal or
not sufficiently in compliance with common standards.

See in particular the section "Inappropriate expectations" in
https://en.opensuse.org/SDB:Disaster_Recovery

jsmeix commented at 2023-11-27 15:58:

@ramzcode
as far as I currently see
ReaR won't work with that kind of system
(and likely lots of other stuff won't work too).
So I would recommend that you find out how to setup
a system in AWS that behaves sufficiently in compliance
with common standards.

By the way:
I do not understand what the reason for that strange
partition number 128 for a boot partition could be.
Normally boot partitions (e.g. bios_grub, ESP, '/boot')
are the first partitions on a disk and not the very last one
that is possible with standard GPT.

pcahyna commented at 2023-11-27 16:09:

so '/sys/block/xvda' is the only one in '/sys/block/*'
that has a matching disk device '/dev/xvda'.

If that's the reason, ReaR would not find /dev/xvda1 either (the partition with the root filesystem).

ramzcode commented at 2023-11-27 16:28:

Yes /dev/xvda1 is the root XFS filesystem, and it is recorded.
Below is the output of all Parted related Disk layout captured by ReaR.

[root@ip-172-31-5-70 tmp]# cat part*
BYT;
/dev/xvda:8590MB:xvd:512:512:gpt:Xen Virtual Block Device:;
128:1049kB:2097kB:1049kB::BIOS Boot Partition:bios_grub;
1:2097kB:8590MB:8588MB:xfs:Linux:;

1 8587820544 2097152 Linux none

1 8587820544 2097152

1 8587820544 2097152

pcahyna commented at 2023-11-27 17:09:

Please also attach the file /var/lib/rear/layout/disklayout.conf

jsmeix commented at 2023-11-28 08:14:

@pcahyna
regarding '/dev/xvda1' in your
https://github.com/rear/rear/issues/3093#issuecomment-1828145575

Partitions are not like /sys/block/sda1 but /sys/block/sda/sda1
so here it is /sys/block/xvda/xvda1 which exists according to
https://github.com/rear/rear/files/13474729/rear-ip-172-31-5-70.log

jsmeix commented at 2023-11-28 08:31:

I confused myself with disks and partitions in /sys/block/
versus disks and partitions in /dev/

Actually it is here that
/sys/block/xvda/ and /sys/block/xvda/xvda1 exist
but /sys/block/xvda/xvda128 does not exist according to
https://github.com/rear/rear/files/13474729/rear-ip-172-31-5-70.log
(excerpts)

+ source /usr/share/rear/layout/save/GNU/Linux/200_partition_layout.sh
...
++ extract_partitions /dev/xvda
...
++ declare sysfs_name=xvda
++ sysfs_paths_unfiltered=(/sys/block/$sysfs_name/$sysfs_name*)
...
++ for possible_sysfs_partition in '"${sysfs_paths_unfiltered[@]}"'
++ [[ /sys/block/xvda/xvda1 = ...

which is the only found partition in /sys/block/

'xvda128' does not appear in
https://github.com/rear/rear/files/13474729/rear-ip-172-31-5-70.log

ramzcode commented at 2023-11-28 08:41:

@jsmeix Yes, the underlying problem was due to the driver it seems,
Now after upgrading the kernel to 5.14
for Amazon Linux 2 on Xen hypervisor typed instance,
the lsblk is now listing it correctly.
below is the issue for the same with Amazon Linux Team with more clarity.

https://github.com/amazonlinux/amazon-ec2-utils/issues/31

jsmeix commented at 2023-11-28 08:50:

@ramzcode
thank you for your information what the root cause was!
From
https://github.com/amazonlinux/amazon-ec2-utils/issues/31
(excerpts):

underlying was the /dev/xvda128 itself not created.

an issue with the xen block driver on earlier kernels

In particular
https://github.com/amazonlinux/amazon-ec2-utils/issues/31#issuecomment-1828928217
(excerpt)

The partitions being in the wrong order is actually intentional.
For historical reasons, a lot of things out there
assume root to be partition 1 on Amazon Linux,
but since it needs to be resized at boot, it has to be at the end.

is interesting for me because it answers my question in
https://github.com/rear/rear/issues/3093#issuecomment-1828121935

[Export of Github issue for rear/rear.]