#2801 Issue `closed`: Broken 'part' entries in disklayout.conf in case of 'unknown' partition table¶

Labels: enhancement, support / question, fixed / solved / done

tpatel80 opened issue at 2022-05-09 18:51:¶

Relax-and-Recover (ReaR) Issue Template¶

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

ReaR version ("/usr/sbin/rear -V"):
Relax-and-Recover 2.6 / Git
OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):

OS_VENDOR=RedHatEnterpriseServer
OS_VERSION=8.2

ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

KEEP_BUILD_DIR=""
OUTPUT=ISO
BACKUP=NETFS
BACKUP_PROG=tar
BACKUP_TYPE=incremental
BACKUP_PROG_OPTIONS+=( --anchored --xattrs --xattrs-include=security.capability )
FULLBACKUPDAY="Mon"
BACKUP_PROG_EXCLUDE=('/var/log/*'  '/fs/site*' '/opt/IBM/zimon/data*' '/var/log/messages*' '/var/crash/*' '/var/tmp/*')
BACKUP_OPTIONS="nfsvers=4,nolock"
BACKUP_URL=nfs://192.168.1.101/backup/
USE_STATIC_NETWORKING=Y
REAR_INITRD_COMPRESSION="lzma"
REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" parted sfdisk )
PROGS=( "${PROGS[@]}" partprobe fdisk cfdisk mkofboot ofpath ybin yabootconfig bootlist pseries_platform nvram ofpathname bc agetty )

test "${FIRMWARE_FILES[*]}" || FIRMWARE_FILES=( 'no' )

Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR):
IBM
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
PPC64LE
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
Petitboot
Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
local disk
Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"):

NAME                               KNAME     PKNAME    TRAN TYPE FSTYPE      LABEL   SIZE MOUNTPOINT
/dev/sda                           /dev/sda                 disk                   893.8G 
|-/dev/sda1                        /dev/sda1 /dev/sda       part                       4M 
|-/dev/sda2                        /dev/sda2 /dev/sda       part ext4                500M /boot
`-/dev/sda3                        /dev/sda3 /dev/sda       part LVM2_member       893.3G 
  |-/dev/mapper/VolGroup00-lv_root /dev/dm-0 /dev/sda3      lvm  ext4                 50G /
  `-/dev/mapper/VolGroup00-lv_swap /dev/dm-1 /dev/sda3      lvm  swap                1.5G [SWAP]

Description of the issue (ideally so that others can reproduce it):

REAR fails with following error:

ERROR: 
====================
BUG in /usr/local/src/rear/usr/share/rear/layout/save/default/950_verify_disklayout_file.sh line 254:
'Entries in /usr/local/src/rear/var/lib/rear/layout/disklayout.conf are broken ('rear recover' would fail)'
--------------------

Workaround, if any:
Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):

To paste verbatim text like command output or file content,
include it between a leading and a closing line of three backticks like

```
verbatim content
```

tpatel80 commented at 2022-05-09 18:51:¶

rear.log

jsmeix commented at 2022-05-10 07:12:¶

@tpatel80
your https://github.com/rear/rear/files/8654468/rear.log
contains (excerpts):

2022-05-09 18:31:50.466037669 Verifying that the 'part' entries for /dev/sda in /usr/local/src/rear/var/lib/rear/layout/disklayout.conf are correct
++ partitions=()
++ read keyword dummy part_size part_start part_name part_flags part_dev junk
+++ grep '^part /dev/sda ' /usr/local/src/rear/var/lib/rear/layout/disklayout.conf
++ test -b ''
++ broken_part_errors+=("$part_dev is not a block device")
...
++ read keyword dummy part_size part_start part_name part_flags part_dev junk
++ test -b ''
++ broken_part_errors+=("$part_dev is not a block device")
...
++ read keyword dummy part_size part_start part_name part_flags part_dev junk
++ test -b ''
++ broken_part_errors+=("$part_dev is not a block device")

The matching code in
usr/share/rear/layout/save/default/950_verify_disklayout_file.sh
is (excerpts):

    while read keyword dummy part_size part_start part_name part_flags part_dev junk ; do
        test -b "$part_dev" || broken_part_errors+=( "$part_dev is not a block device" )

This shows that

read keyword dummy part_size part_start part_name part_flags part_dev junk

does not get any value for part_dev from the

grep '^part /dev/sda ' /usr/local/src/rear/var/lib/rear/layout/disklayout.conf

output.

So the error is right because according to the 'lsblk' output
in your https://github.com/rear/rear/issues/2801#issue-1230116754
there should be

part /dev/sda <size> <start> <type|name> <flags> /dev/sda1
part /dev/sda <size> <start> <type|name> <flags> /dev/sda2
part /dev/sda <size> <start> <type|name> <flags> /dev/sda3

entries in your
/usr/local/src/rear/var/lib/rear/layout/disklayout.conf

Next step is to find out why read does not get part_dev values:

Your https://github.com/rear/rear/files/8654468/rear.log
contains (excerpts):

+ source /usr/local/src/rear/usr/share/rear/layout/save/GNU/Linux/200_partition_layout.sh
...
+++ parted -s /dev/sda print
...
Error: Can't have a partition outside the disk!
++ disktype=unknown
...
++ parted -m -s /dev/sda print
Error: Can't have a partition outside the disk!
...
++ echo 'part /dev/sda 4194304 1048576 none  /dev/sda1'
...
++ echo 'part /dev/sda 524288000 5242880 none  /dev/sda2'
...
++ echo 'part /dev/sda 959127224320 529530880 none  /dev/sda3'
.
.
.
+ source /usr/local/src/rear/usr/share/rear/layout/save/GNU/Linux/220_lvm_layout.sh
...
++ lvm pvdisplay -c
...
  WARNING: Device /dev/sda3 has size of 1873295360 sectors which is smaller than corresponding PV size of 1874350080 sectors. Was device resized?
  WARNING: One or more devices used as PVs in VG VolGroup00 have changed sizes.

The

++ echo 'part /dev/sda 4194304 1048576 none  /dev/sda1'
++ echo 'part /dev/sda 524288000 5242880 none  /dev/sda2'
++ echo 'part /dev/sda 959127224320 529530880 none  /dev/sda3'

show what is wrong:
A 'part' entry in disklayout.conf has the syntax

part <device> <size> <start> <type|name> <flags> /dev/<partition>

see the section
"Disk layout file syntax" in
https://github.com/rear/rear/blob/master/doc/user-guide/06-layout-configuration.adoc

For example on my homeoffice laptop I have

part /dev/sda 8388608 1048576 rear-noname bios_grub /dev/sda1
part /dev/sda 4294967296 9437184 rear-noname swap /dev/sda2
part /dev/sda 214748364800 4304404480 rear-noname legacy_boot /dev/sda3
part /dev/sda 107374182400 219052769280 rear-noname none /dev/sda4

In contrast you have

part /dev/sda 4194304 1048576 none  /dev/sda1
part /dev/sda 524288000 5242880 none  /dev/sda2
part /dev/sda 959127224320 529530880 none  /dev/sda3

so one of <type|name> or <flags> is missing.

Next step is why <type|name> or <flags> is missing:
Your https://github.com/rear/rear/files/8654468/rear.log
contains (excerpts):

+ source /usr/local/src/rear/usr/share/rear/layout/save/GNU/Linux/200_partition_layout.sh
.
.
.
++ parted -m -s /dev/sda print
Error: Can't have a partition outside the disk!
+++ grep '^/' /var/tmp/rear.3lGTm845SuRAkd2/tmp/parted
+++ cut -d : -f 6
++ disk_label=unknown
++ cp /var/tmp/rear.3lGTm845SuRAkd2/tmp/partitions /var/tmp/rear.3lGTm845SuRAkd2/tmp/partitions-data
++ declare type
++ [[ unknown = \m\s\d\o\s ]]
++ [[ unknown = \g\p\t ]]
++ [[ unknown = \g\p\t\_\s\y\n\c\_\m\b\r ]]
++ [[ unknown = \d\a\s\d ]]
++ declare flags flaglist

The matching code in
usr/share/rear/layout/save/GNU/Linux/200_partition_layout.sh
is (excerpts)

        parted -m -s $device print > $TMP_DIR/parted
        disk_label=$(grep ^/ $TMP_DIR/parted | cut -d ":" -f "6")
...
    if [[ "$disk_label" = "msdos" ]] ; then
...
    if [[ "$disk_label" = "gpt" || "$disk_label" = "gpt_sync_mbr" || "$disk_label" = "dasd" ]] ; then

Your disk_label=unknown which does not match
any of "msdos" "gpt" "gpt_sync_mbr" or "dasd"
so the partition type is not set
which leads to it be empty in disklayout.conf

I guess your disk label is actually one of
"msdos" "gpt" "gpt_sync_mbr" or "dasd"
but the current code in 200_partition_layout.sh
does not work because it does not sufficiently
"Try hard to care about possible errors", cf.
https://github.com/rear/rear/wiki/Coding-Style
because it blindly proceeds in case of errors like

++ parted -m -s /dev/sda print
Error: Can't have a partition outside the disk!

and I assume in your case the

parted -m -s /dev/sda print

output is not what the current code in 200_partition_layout.sh
normally needs and then things arbitrarily fail in ReaR.

@tpatel80
you should fix your partitioning so that parted
does not show errors.

I will have a look at
usr/share/rear/layout/save/GNU/Linux/200_partition_layout.sh
how to make it behave more reliably and failsafe,
in particular error out directly therein when things failed
instead of error out later in 950_verify_disklayout_file.sh
(at least I had added 950_verify_disklayout_file.sh
as generic safety net against our many old code places
that blindly proceed instead of caring about errors).

jsmeix commented at 2022-05-10 08:06:¶

https://github.com/rear/rear/pull/2802
intended to avoid the difficulty to detect
the root cause of an error like in this issue
but my attempt made things even worse.

jsmeix commented at 2022-05-10 08:51:¶

Now its successor https://github.com/rear/rear/pull/2803
tries to avoid the difficulty to detect the
root cause of an error like in this issue here
by ensuring a supported partition table is found
and if not it errors out.

jsmeix commented at 2022-05-10 10:33:¶

https://github.com/rear/rear/pull/2804
tries to ensure syntactically correct 'disk' and 'part' entries
which also intends to avoid the difficulty to detect
the root cause of an error like in this issue here.

pcahyna commented at 2022-05-10 12:59:¶

I believe it is needed to know how to reproduce the issue before merging any fix (like #2804). @tpatel80 how is the disk partitioned? I suppose your machine is PowerNV (BareMetal)?

tpatel80 commented at 2022-05-10 13:10:¶

@pcahyna the machine is a baremetal IBM Power System IC922 (9183-22X) factory installed with Redhat 8.2.

pcahyna commented at 2022-05-10 13:16:¶

@tpatel80 I will try (I have used ReaR on POWER bare metal and I have not seen this kind of issue, but it was the version shipped with RHEL, not the development version from Git)

jsmeix commented at 2022-05-10 13:36:¶

Neither https://github.com/rear/rear/pull/2803
nor https://github.com/rear/rear/pull/2804
intend to fix the initially reported issue here
i.e. none makes correct entries in disklayout.conf.

Both are only meant to detect early i.e. directly
in the code where the entries are genereated in
layout/save/GNU/Linux/200_partition_layout.sh
when invalid entries would be generated
(regardless what the actual reason is).

I.e. https://github.com/rear/rear/pull/2803
and https://github.com/rear/rear/pull/2804
are only meant to better care about possible errors
and exit directly where the error happened
instead of blindly proceeding and let things fail
at some later place (950_verify_disklayout_file.sh).

pcahyna commented at 2022-05-10 14:01:¶

@tpatel80 I am not able to reproduce the issue on RHEL 8.2 / POWER bare metal. What does

parted -m -s /dev/sda print

and

parted -s /dev/sda print

print for you?

The message WARNING: Device /dev/sda3 has size of 1873295360 sectors which is smaller than corresponding PV size of 1874350080 sectors. Was device resized? indicates that there is something wrong with partitioning on your machine.

pcahyna commented at 2022-05-10 14:01:¶

Also please add the output of fdisk -l /dev/sda

jsmeix commented at 2022-05-10 14:08:¶

Only a side note FYI:
When rear is run in debug mode KEEP_BUILD_DIR is true
so that [/var]/tmp/rear.XXXXXX/tmp/parted should be kept
that contains the parted -m -s output.

tpatel80 commented at 2022-05-10 14:24:¶

I've attempted to correct the end value for sda3 to 1874329599. Will report back if this helps in any manner.

**sh-4.4# parted -m -s /dev/sda print**
Error: Can't have a partition outside the disk!
BYT;
/dev/sda:960GB:scsi:512:4096:unknown:AVAGO MR9361-8i:;

**sh-4.4# parted -s /dev/sda print**
Error: Can't have a partition outside the disk!
Model: AVAGO MR9361-8i (scsi)
Disk /dev/sda: 960GB
Sector size (logical/physical): 512B/4096B
Partition Table: unknown
Disk Flags:

**sh-4.4# fdisk -l /dev/sda**
Disk /dev/sda: 893.8 GiB, 959656755200 bytes, **1874329600** sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 262144 bytes / 262144 bytes
Disklabel type: dos
Disk identifier: 0x3db2a74e

Device     Boot   Start        End    Sectors   Size Id Type
/dev/sda1  *       2048      10239       8192     4M 41 PPC PReP Boot
/dev/sda2         10240    1034239    1024000   500M 83 Linux
/dev/sda3       1034240 1875384319 **1874350080** 893.8G 8e Linux LVM


**sh-4.4# fdisk -l -u /dev/sda**
Disk /dev/sda: 893.8 GiB, 959656755200 bytes, 1874329600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 262144 bytes / 262144 bytes
Disklabel type: dos
Disk identifier: 0x3db2a74e

Device     Boot   Start        End    Sectors   Size Id Type
/dev/sda1  *       2048      10239       8192     4M 41 PPC PReP Boot
/dev/sda2         10240    1034239    1024000   500M 83 Linux
/dev/sda3       1034240 1875384319 1874350080 893.8G 8e Linux LVM

tpatel80 commented at 2022-05-10 14:29:¶

Updating END value of sda3 seems to have done the trick. Running mkbackup now.

[Tue May 10 14:26:31 root@hpnls4vfs5:~ ] # parted -m -s /dev/sda print
BYT;
/dev/sda:960GB:scsi:512:4096:msdos:AVAGO MR9361-8i:;
1:1049kB:5243kB:4194kB:::boot, prep;
2:5243kB:530MB:524MB:ext4::;
3:530MB:960GB:959GB:::lvm;

[Tue May 10 14:26:40 root@hpnls4vfs5:~ ] # parted -s /dev/sda print
Model: AVAGO MR9361-8i (scsi)
Disk /dev/sda: 960GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  5243kB  4194kB  primary               boot, prep
 2      5243kB  530MB   524MB   primary  ext4
 3      530MB   960GB   959GB   primary               lvm

pcahyna commented at 2022-05-11 09:31:¶

@tpatel80 I believe you still have a problem with LVM, it thinks that the PV is longer than the partition that it sits on. Not sure how serious it is or how to correct it...

jsmeix commented at 2022-05-11 12:21:¶

@tpatel80
only out of curiosity (I am not a Red Hat user) regarding your
https://github.com/rear/rear/issues/2801#issuecomment-1122369017

the machine is a baremetal IBM Power System IC922 (9183-22X)
factory installed with Redhat 8.2.

Because you have parted errors and LVM warnings
about partion sizes and PV sizes on your original system
(so it cannot be ReaR that recreated something wrong)
I wonder if your original system is really
"factory installed with Redhat 8.2"?

I assume a Red Hat installation does not result
wrong partitioning and/or wrong LVM PV setup
(I think otherwise Red Hat would have already noticed)
so I guess smoeone or something had somehow changed
the pristine partitioning on your original system.

Alternatively a Red Hat installation may do
something wrong only on special hardware like
your baremetal IBM Power System IC922 (9183-22X).

FYI:
This would not be the first case we had here at ReaR
where disaster recovery setup with ReaR reveals
a broken setup on the original system.
In particular we had "fun" with certain special
third-party software that a bit messed up things
in the original system.

tpatel80 commented at 2022-05-11 13:21:¶

@jsmeix I suspect the image used to clone these systems had different disk size. We've contacted IBM for details.

jsmeix commented at 2022-05-13 06:35:¶

@tpatel80
so the installation on this particular machine
was not an actual Red Hat installation
but an imaged clone of a Red Hat installation
and ReaR revealed that the likely 100% clone
(e.g. like a simple 'dd' whole disk clone)
does not 100% match the clone's target machine hardware
(e.g. the disk image is a bit bigger than the target disk).

In general regarding a bit different hardware disk sizes
(in contrast virtual disk sizes can be made 100% same)
you may have a look at the description about
AUTORESIZE_PARTITIONS
AUTORESIZE_EXCLUDE_PARTITIONS
AUTOSHRINK_DISK_SIZE_LIMIT_PERCENTAGE
AUTOINCREASE_DISK_SIZE_THRESHOLD_PERCENTAGE
in usr/share/rear/conf/default.conf
what ReaR would do during "rear recover" to deal with
a bit different disk sizes on replacement hardware.

To keep issues because of a bit different disk sizes
out of the way I would recommend to not use a disk
up to its very end but leave some reasonable amount of
disk space unused at the end so that the a bit smaller
actually used disk space fits within the smallest existing
hardware disk size among one kind of disks that one uses
(e.g. among all 1TB disks that one uses).

This belongs to the general topic
"Prepare for disaster recovery from the very beginning" in
https://en.opensuse.org/SDB:Disaster_Recovery

I think I add a subsection there about recommended disk usage.

jsmeix commented at 2022-05-18 12:27:¶

With https://github.com/rear/rear/pull/2803
and https://github.com/rear/rear/pull/2804
merged I think this issue is sufficiently solved.

[Export of Github issue for rear/rear.]

#2801 Issue closed: Broken 'part' entries in disklayout.conf in case of 'unknown' partition table¶

tpatel80 opened issue at 2022-05-09 18:51:¶

Relax-and-Recover (ReaR) Issue Template¶

tpatel80 commented at 2022-05-09 18:51:¶

jsmeix commented at 2022-05-10 07:12:¶

jsmeix commented at 2022-05-10 08:06:¶

jsmeix commented at 2022-05-10 08:51:¶

jsmeix commented at 2022-05-10 10:33:¶

pcahyna commented at 2022-05-10 12:59:¶

tpatel80 commented at 2022-05-10 13:10:¶

pcahyna commented at 2022-05-10 13:16:¶

jsmeix commented at 2022-05-10 13:36:¶

pcahyna commented at 2022-05-10 14:01:¶

pcahyna commented at 2022-05-10 14:01:¶

jsmeix commented at 2022-05-10 14:08:¶

tpatel80 commented at 2022-05-10 14:24:¶

tpatel80 commented at 2022-05-10 14:29:¶

pcahyna commented at 2022-05-11 09:31:¶

jsmeix commented at 2022-05-11 12:21:¶

tpatel80 commented at 2022-05-11 13:21:¶

jsmeix commented at 2022-05-13 06:35:¶

jsmeix commented at 2022-05-18 12:27:¶

#2801 Issue `closed`: Broken 'part' entries in disklayout.conf in case of 'unknown' partition table¶