#2299 PR merged: Better is_multipath_path function (issue #2298) plus some other stuff¶
Labels: enhancement, bug, cleanup, fixed / solved / done
jsmeix opened issue at 2019-12-11 15:55:¶
-
Type: Bug Fix / Cleanup
-
Impact: High
In my case disklayout.conf became totally useless.
If the '/dev/sda' entries were only commented out
the bad disklayout.conf could still be manually adapted
but without any '/dev/sda' entries it is useless in practice
(I cannot guess during "rear recover" what right values could be). -
Reference to related issue (URL):
https://github.com/rear/rear/issues/2298 -
How was this pull request tested?
Currently untested preliminary initial state.
"rear mkrescue" looks goot to me with that changes
but I need to test "rear recover" (hopefully tomorrow). -
Brief description of the changes in this pull request:
Better (i.e. hopefully more fail safe) is_multipath_path function,
cf.
https://github.com/rear/rear/issues/2298#issuecomment-564590474
Have lsblk output as disklayout.conf header comments
so that it is easier to make sense of the values in the subsequent
entries.
Some TODO comments added in
layout/prepare/default/010_prepare_files.sh
because I wonder about how some things therein are meant to work.
Added xdd (belongs to vi) to the PROGS array because also
a tool to display binary files is needed in the recovery system.
jsmeix commented at 2019-12-11 15:57:¶
With the lsblk output as disklayout.conf header comments
I get this disklayout.conf:
# Disk layout dated 20191211160659 (YYYYmmddHHMMSS)
# NAME KNAME PKNAME TRAN TYPE FSTYPE SIZE MOUNTPOINT
# /dev/sda /dev/sda sata disk 931.5G
# |-/dev/sda1 /dev/sda1 /dev/sda part vfat 500M
# |-/dev/sda2 /dev/sda2 /dev/sda part 100M
# |-/dev/sda3 /dev/sda3 /dev/sda part swap 20G [SWAP]
# |-/dev/sda4 /dev/sda4 /dev/sda part ext4 400G /
# `-/dev/sda5 /dev/sda5 /dev/sda part ext4 400G /other
# /dev/sdb /dev/sdb usb disk 465.8G
# `-/dev/sdb1 /dev/sdb1 /dev/sdb part ext3 46.6G
# /dev/sr0 /dev/sr0 sata rom 1024M
# Disk /dev/sda
# Format: disk <devname> <size(bytes)> <partition label type>
disk /dev/sda 1000204886016 gpt
# Partitions on /dev/sda
# Format: part <device> <partition size(bytes)> <partition start(bytes)> <partition type|name> <flags> /dev/<partition>
part /dev/sda 524288000 1048576 rear-noname boot,legacy_boot,esp /dev/sda1
part /dev/sda 104857600 525336576 rear-noname bios_grub /dev/sda2
part /dev/sda 21474836480 630194176 rear-noname swap /dev/sda3
part /dev/sda 429496729600 22105030656 rear-noname none /dev/sda4
part /dev/sda 429496729600 451601760256 rear-noname none /dev/sda5
# Disk /dev/sdb
# Format: disk <devname> <size(bytes)> <partition label type>
#disk /dev/sdb 500107859968 msdos
# Partitions on /dev/sdb
# Format: part <device> <partition size(bytes)> <partition start(bytes)> <partition type|name> <flags> /dev/<partition>
#part /dev/sdb 50002395136 8388608 primary boot /dev/sdb1
# Filesystems (only ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs are supported).
# Format: fs <device> <mountpoint> <fstype> [uuid=<uuid>] [label=<label>] [<attributes>]
fs /dev/sda4 / ext4 uuid=cff8eaf4-2369-439b-8ef2-620dd515d767 label= blocksize=4096 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=16384 default_mount_options=user_xattr,acl options=rw,relatime,data=ordered
fs /dev/sda5 /other ext4 uuid=2aa61137-5fc3-4fca-8b8e-959bc4c3676d label= blocksize=4096 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=16384 default_mount_options=user_xattr,acl options=rw,relatime,data=ordered
# Swap partitions or swap files
# Format: swap <filename> uuid=<uuid> label=<label>
swap /dev/sda3 uuid=6d4a07be-f9d7-4e56-a141-75758754e822 label=
cf. https://github.com/rear/rear/issues/2298#issue-536410401
As far as I see the initial comment header in disklayout.conf
# Disk layout dated 20191211160659 ...
that changes each time when "rear mkrescue" is run
cannot cause a falsely detected changed layout by "rear checklayout"
because layout/compare/default/500_compare_layout.sh
compares only non-comment lines in disklayout.conf
jsmeix commented at 2019-12-11 16:55:¶
I did a "rear recover" test on my laptop
and from what one can see on the terminal all looks OK
(excerpts):
Welcome to Relax-and-Recover. Run "rear recover" to restore your system !
RESCUE linux-88cr:~ # lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 931.5G 0 disk
|-sda1 8:1 0 500M 0 part
|-sda2 8:2 0 100M 0 part
|-sda3 8:3 0 20G 0 part
|-sda4 8:4 0 400G 0 part
`-sda5 8:5 0 400G 0 part
sdb 8:16 0 465.8G 0 disk
`-sdb1 8:17 0 46.6G 0 part
sr0 11:0 1 1024M 0 rom
RESCUE linux-88cr:~ # cat ./rear.github.master/var/lib/rear/layout/disklayout.conf
# Disk layout dated 20191211170123 (YYYYmmddHHMMSS)
# NAME KNAME PKNAME TRAN TYPE FSTYPE SIZE MOUNTPOINT
# /dev/sda /dev/sda sata disk 931.5G
# |-/dev/sda1 /dev/sda1 /dev/sda part vfat 500M
# |-/dev/sda2 /dev/sda2 /dev/sda part 100M
# |-/dev/sda3 /dev/sda3 /dev/sda part swap 20G [SWAP]
# |-/dev/sda4 /dev/sda4 /dev/sda part ext4 400G /
# `-/dev/sda5 /dev/sda5 /dev/sda part ext4 400G /other
# /dev/sdb /dev/sdb usb disk 465.8G
# `-/dev/sdb1 /dev/sdb1 /dev/sdb part ext3 46.6G
# /dev/sr0 /dev/sr0 sata rom 1024M
# Disk /dev/sda
# Format: disk <devname> <size(bytes)> <partition label type>
disk /dev/sda 1000204886016 gpt
# Partitions on /dev/sda
# Format: part <device> <partition size(bytes)> <partition start(bytes)> <partition type|name> <flags> /dev/<partition>
part /dev/sda 524288000 1048576 rear-noname boot,legacy_boot,esp /dev/sda1
part /dev/sda 104857600 525336576 rear-noname bios_grub /dev/sda2
part /dev/sda 21474836480 630194176 rear-noname swap /dev/sda3
part /dev/sda 429496729600 22105030656 rear-noname none /dev/sda4
part /dev/sda 429496729600 451601760256 rear-noname none /dev/sda5
# Disk /dev/sdb
# Format: disk <devname> <size(bytes)> <partition label type>
#disk /dev/sdb 500107859968 msdos
# Partitions on /dev/sdb
# Format: part <device> <partition size(bytes)> <partition start(bytes)> <partition type|name> <flags> /dev/<partition>
#part /dev/sdb 50002395136 8388608 primary boot /dev/sdb1
# Filesystems (only ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs are supported).
# Format: fs <device> <mountpoint> <fstype> [uuid=<uuid>] [label=<label>] [<attributes>]
fs /dev/sda4 / ext4 uuid=cff8eaf4-2369-439b-8ef2-620dd515d767 label= blocksize=4096 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=16384 default_mount_options=user_xattr,acl options=rw,relatime,data=ordered
fs /dev/sda5 /other ext4 uuid=2aa61137-5fc3-4fca-8b8e-959bc4c3676d label= blocksize=4096 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=16384 default_mount_options=user_xattr,acl options=rw,relatime,data=ordered
# Swap partitions or swap files
# Format: swap <filename> uuid=<uuid> label=<label>
swap /dev/sda3 uuid=6d4a07be-f9d7-4e56-a141-75758754e822 label=
RESCUE linux-88cr:~ # rear -D recover
Relax-and-Recover 2.5 / Git
Running rear recover (PID 839)
Using log file: /var/log/rear/rear-linux-88cr.log
Running workflow recover within the ReaR rescue/recovery system
Using backup archive '/tmp/rear.UktcIFtnmwX1eU0/outputfs/rear/linux-88cr/ReaRbackup/backup.tar.gz'
Will do driver migration (recreating initramfs/initrd)
Calculating backup archive size
Backup archive size is 2.5G /tmp/rear.UktcIFtnmwX1eU0/outputfs/rear/linux-88cr/ReaRbackup/backup.tar.gz (compressed)
Comparing disks
Device sda has expected (same) size 1000204886016 (will be used for recovery)
Disk configuration looks identical
UserInput -I DISK_LAYOUT_PROCEED_RECOVERY needed in /usr/share/rear/layout/prepare/default/250_compare_disks.sh line 148
Proceed with recovery (yes) otherwise manual disk layout configuration is enforced
(default 'yes' timeout 30 seconds)
UserInput: No real user input (empty or only spaces) - using default input
UserInput: No choices - result is 'yes'
User confirmed to proceed with recovery
Start system layout restoration.
Disk '/dev/sda': creating 'gpt' partition table
Disk '/dev/sda': creating partition number 1 with name ''sda1''
Disk '/dev/sda': creating partition number 2 with name ''sda2''
Disk '/dev/sda': creating partition number 3 with name ''sda3''
Disk '/dev/sda': creating partition number 4 with name ''sda4''
Disk '/dev/sda': creating partition number 5 with name ''sda5''
Creating filesystem of type ext4 with mount point / on /dev/sda4.
Mounting filesystem /
Creating filesystem of type ext4 with mount point /other on /dev/sda5.
Mounting filesystem /other
Creating swap on /dev/sda3
Disk layout created.
Restoring from '/tmp/rear.UktcIFtnmwX1eU0/outputfs/rear/linux-88cr/ReaRbackup/backup.tar.gz' (restore log in /var/lib/rear/restore/recover.backup.tar.gz.839.restore.log) ...
Backup restore program 'tar' started in subshell (PID=2760)
Restored 336 MiB [avg. 114846 KiB/sec]
...
Restored 6123 MiB [avg. 87085 KiB/sec]
OK
Restored 6176 MiB in 75 seconds [avg. 84327 KiB/sec]
Restoring finished (verify backup restore log messages in /var/lib/rear/restore/recover.backup.tar.gz.839.restore.log)
Recreating directories (with permissions) from /var/lib/rear/recovery/directories_permissions_owner_group
Migrating disk-by-id mappings in certain restored files in /mnt/local to current disk-by-id mappings ...
Migrating network configuration files according to the mapping files ...
Running mkinitrd...
Recreated initrd (/sbin/mkinitrd).
Installing GRUB2 boot loader...
Determining where to install GRUB2 (no GRUB2_INSTALL_DEVICES specified)
Found possible boot disk /dev/sda - installing GRUB2 there
Finished recovering your system. You can explore it under '/mnt/local'.
Exiting rear recover (PID 839) and its descendant processes ...
Running exit tasks
You should also rm -Rf /tmp/rear.UktcIFtnmwX1eU0
RESCUE linux-88cr:~ # reboot
BUT
the recreated system can no longer boot from its built-in /dev/sda
(the BIOS tells that an operating system needs to be installed on the
disk)
but it can be booted from the USB disk ReaR recovery system bootloader
in its syslinux boot menue by selecting Boot Local disk (hd1)
cf.
https://github.com/rear/rear/issues/2276#issuecomment-553888458
so something on the built-in /dev/sda got lost or damaged by "rear
recover"
that is needed by that laptop's firmware (UEFI capable firmware that I
use
in legacy BIOS mode) to boot.
jsmeix commented at 2019-12-12 09:54:¶
With the essential help of a colleague the
no longer boot from built-in /dev/sda after rear recover
problem was solved:
The missing piece on /dev/sda that got destroyed by "rear recover"
was the enabled boot flag on the GPT’s protective MBR partition, cf.
https://www.gnu.org/software/parted/manual/html_node/disk_005fset.html
that reads
Command: disk_set flag state
Changes a flag on the disk.
A flag can be either “on” or “off”.
Some or all of these flags will be available,
depending on what disk label you are using:
‘pmbr_boot’
(GPT) - this flag enables the boot flag
on the GPT’s protective MBR partition.
The disk’s flags are displayed by the print command
on the "Disk Flags:" line. They are also output as
the last field of the disk information in machine mode.
(parted) disk_set pmbr_boot on
Set the PMBR’s boot flag.
How things looked in the ReaR recovery system
where we fixed that:
Before:
RESCUE linux-88cr:~ # parted -s /dev/sda unit MiB print
Model: ATA HGST HTS541010A9 (scsi)
Disk /dev/sda: 953870MiB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1.00MiB 501MiB 500MiB fat16 sda1 boot, legacy_boot, esp
2 501MiB 601MiB 100MiB sda2 bios_grub
3 601MiB 21081MiB 20480MiB linux-swap(v1) sda3 swap
4 21081MiB 430681MiB 409600MiB ext4 sda4
5 430681MiB 840281MiB 409600MiB ext4 sda5
After:
RESCUE linux-88cr:~ # parted -s /dev/sda unit GiB print
Model: ATA HGST HTS541010A9 (scsi)
Disk /dev/sda: 932GiB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: pmbr_boot
Number Start End Size File system Name Flags
1 0.00GiB 0.49GiB 0.49GiB fat16 sda1 boot, legacy_boot, esp
2 0.49GiB 0.59GiB 0.10GiB sda2 bios_grub
3 0.59GiB 20.6GiB 20.0GiB linux-swap(v1) sda3 swap
4 20.6GiB 421GiB 400GiB ext4 sda4
5 421GiB 821GiB 400GiB ext4 sda5
The difference is Disk Flags: empty which does not work
versus Disk Flags: pmbr_boot which is needed
at least by my UEFI firmware that I use in legacy BIOS mode
to boot from a GPT disk with GRUB2 installed "in MBR"
according to (xdd output shrinked):
RESCUE linux-88cr:~ # dd if=/dev/sda of=/sda.mbr.bin bs=512 count=1
RESCUE linux-88cr:~ # xxd -g 32 -c 32 /sda.mbr.bin
00000000: eb...00 .c..............................
00000020: 00...00 ................................
00000040: 00...00 ................................
00000060: 00...bc ...........t...pt....y|..1......
00000080: 00...72 . ..d|<.t...R..}.....|.A..U..ZRr
000000a0: 3d...5c =..U.u7...t21..D.@.D..D.....f..\
000000c0: 7c...08 |f.\.f..`|f.\..D..p.B..r...p.v..
000000e0: cd..d1 ..s.Z........}...f....d.@f.D....
00000100: c1...5c .......@.D.......f..f.`|f..uNf.\
00000120: 7c...88 |f1.f.4..1.f.t.;D.}7....0.......
00000140: d0...80 .Z....p..1......r...`......1....
00000160: 8e...fe ......a.&Z|..}....}.4...}.......
00000180: 47...72 GRUB .Geom.Hard Disk.Read. Error
000001a0: 0d...00 ...........<.u..................
000001c0: 02...00 ...........mpt..................
000001e0: 00...aa ..............................U.
By default neither xdd nor hexdump is in the ReaR recovery system.
I learned hereby a tool to display binary files is needed.
I prefer xdd over hexdump in the recovery system
because xdd is more convenient to use,
belongs to vi that we have anyway in the recovery system,
/usr/bin/xxd is smaller than /usr/bin/hexdump
(19K versus 51K on my openSUSE Leap 15.0 system) and
/usr/bin/xxd needs less libraries than /usr/bin/hexdump
# ldd /usr/bin/xxd
linux-vdso.so.1 (0x00007fff367ec000)
libc.so.6 => /lib64/libc.so.6 (0x00007f55ccf72000)
/lib64/ld-linux-x86-64.so.2 (0x00007f55cd532000)
# ldd /usr/bin/hexdump
linux-vdso.so.1 (0x00007fffff9fb000)
libtinfo.so.6 => /lib64/libtinfo.so.6 (0x00007f038d077000)
libc.so.6 => /lib64/libc.so.6 (0x00007f038ccbd000)
/lib64/ld-linux-x86-64.so.2 (0x00007f038d4b2000)
so I will add xdd to the PROGS array.
jsmeix commented at 2019-12-12 10:54:¶
The fix for the
no longer boot from built-in /dev/sda after rear recover
problem will be done separatedly via
https://github.com/rear/rear/issues/2300
parted "disk_set pmbr_boot on" needed in "rear recover"
to boot from GPT disk on BIOS system
jsmeix commented at 2019-12-12 10:55:¶
@rear/contributors
I would like to merge this pull request soon (ideally tomorrow)
unless there are objections.
jsmeix commented at 2019-12-13 09:53:¶
If there are no objections I merge it today afternoon.
jsmeix commented at 2019-12-13 10:50:¶
@schabrolles
could you have a look here (provided you find a bit of time)?
In particular at my change of the is_multipath_path function:
https://github.com/rear/rear/pull/2299/files#diff-cc2bbe1c50edffdb8abd2b4ab63944a3
Perhaps you see something that is obviously wrong.
I would like to merge it today afternoon if there are no objections.
jsmeix commented at 2019-12-13 14:26:¶
@schabrolles
thank you for your prompt review!
I appreciate it because I know you have
currently almost no time for ReaR.
[Export of Github issue for rear/rear.]