#2703 PR `merged`: Enhanced disk write-protection¶

Labels: enhancement, fixed / solved / done

jsmeix opened issue at 2021-10-27 11:15:¶

Type: Enhancement
Impact: Normal
Reference to related issue (URL):
https://github.com/rear/rear/pull/2626
How was this pull request tested?
Not yet tested by me - will test soon...
Brief description of the changes in this pull request:

The specific WRITE_PROTECTED_PARTITION_TABLE_UUIDS
is replaced by WRITE_PROTECTED_IDS with generic functionality
cf. https://github.com/rear/rear/pull/2626#issuecomment-950953826
together with the new WRITE_PROTECTED_ID_TYPES which
defaults to UUID PTUUID PARTUUID WWN so that the user can
specify different lsblk columns as needed in his particular environment

WRITE_PROTECTED_FILE_SYSTEM_LABEL_PATTERNS
is shortened to WRITE_PROTECTED_FS_LABEL_PATTERNS

jsmeix commented at 2021-10-27 12:44:¶

I tested this one similar as I did in
https://github.com/rear/rear/pull/2626#issuecomment-950935246
but this pull request is based on latest ReaR upstream code
so things behave a bit different now:

I tested it on the two VMs with those GPT disks

original system            replacement system
sda 8 GiB system disk      sda the 8 GiB ReaR "USB" disk from the original system
sdb 8 GiB ReaR "USB" disk  sdb 9 GiB replacement system disk

as in https://github.com/rear/rear/pull/2626#issuecomment-950935246

I use almost same etc/rear/local.conf
as in https://github.com/rear/rear/pull/2626#issuecomment-950935246
but now only DISKS_TO_BE_WIPED='' is added

DISKS_TO_BE_WIPED=''
FIRMWARE_FILES=( 'no' )
MODULES=( 'loaded_modules' )
PROGRESS_MODE="plain"
PROGRESS_WAIT_SECONDS="3"
SSH_ROOT_PASSWORD="rear"
OUTPUT=USB
USB_DEVICE_FILESYSTEM_PERCENTAGE=90
USB_DEVICE_FILESYSTEM_LABEL="MY-DATA"
USB_BOOT_PART_SIZE=1024
USB_BOOTLOADER="grub"
USB_DEVICE_BOOT_LABEL="MY-BOOT"
OUTPUT_URL=usb:///dev/disk/by-label/MY-BOOT
USB_DEVICE_PARTED_LABEL=gpt
BACKUP=NETFS
BACKUP_URL=usb:///dev/disk/by-label/MY-DATA

After "rear mkbackup" things in
/var/tmp/rear.AtrtkGDGb5CP4qY/rootfs/etc/rear/rescue.conf
look better now (excerpt)

# The following lines were added by 490_store_write_protect_settings.sh
WRITE_PROTECTED_UUIDS=( 8bd9f9e8-9043-4a80-84e1-64d417244529 e38c933c-d18f-4821-92c9-c99673738660 efee5e95-40a0-4a8e-bbe1-1cad7c0d255d )
WRITE_PROTECTED_FS_LABEL_PATTERNS=( 'MY-DATA' )

Those UUIDs and the LABEL match only /dev/sdb3

# lsblk -ipo NAME,LABEL,UUID,PTUUID,PARTUUID /dev/sdb
NAME        LABEL   UUID                                 PTUUID                               PARTUUID
/dev/sdb                                                 efee5e95-40a0-4a8e-bbe1-1cad7c0d255d 
|-/dev/sdb1                                              efee5e95-40a0-4a8e-bbe1-1cad7c0d255d aa3d3f8b-3adb-4ef8-a4c0-0e44cdadfff6
|-/dev/sdb2 MY-BOOT ff15e0f1-b4bb-4a02-bb10-1502ec2fe0be efee5e95-40a0-4a8e-bbe1-1cad7c0d255d 281246c5-d29f-4ed2-8cfb-c7404eca010d
`-/dev/sdb3 MY-DATA 8bd9f9e8-9043-4a80-84e1-64d417244529 efee5e95-40a0-4a8e-bbe1-1cad7c0d255d e38c933c-d18f-4821-92c9-c99673738660

because prep/USB/default/480_initialize_write_protect_settings.sh
is run with USB_DEVICE=/dev/disk/by-label/MY-DATA

# grep -A20 'source /root/rear.github.master/usr/share/rear/prep/USB/default/480_initialize_write_protect_settings.sh' var/log/rear/rear-localhost.log
+ source /root/rear.github.master/usr/share/rear/prep/USB/default/480_initialize_write_protect_settings.sh
++ local uuids
+++ write_protection_uuids /dev/disk/by-label/MY-DATA
++++ write_protected_candidate_device /dev/disk/by-label/MY-DATA
++++ local device=/dev/disk/by-label/MY-DATA
++++ [[ /dev/disk/by-label/MY-DATA == /sys/block/* ]]
++++ test -b /dev/disk/by-label/MY-DATA
++++ echo /dev/disk/by-label/MY-DATA
+++ local 'device=/dev/disk/by-label/MY-DATA '
+++ local column
+++ sort -u
+++ awk NF
+++ for column in UUID PTUUID PARTUUID
+++ lsblk -ino UUID /dev/disk/by-label/MY-DATA
+++ for column in UUID PTUUID PARTUUID
+++ lsblk -ino PTUUID /dev/disk/by-label/MY-DATA
+++ for column in UUID PTUUID PARTUUID
+++ lsblk -ino PARTUUID /dev/disk/by-label/MY-DATA
++ uuids='8bd9f9e8-9043-4a80-84e1-64d417244529
e38c933c-d18f-4821-92c9-c99673738660
efee5e95-40a0-4a8e-bbe1-1cad7c0d255d'

This is good enough for now but I would prefer when
the whole disk /dev/sdb could be used because the goal is
to write-protect the whole disk.

"rear recover"
the complete terminal output (except backup restore progress messages):

# rear -D recover
Relax-and-Recover 2.6 / Git
Running rear recover (PID 666 date 2021-10-27 14:17:30)
Command line options: /bin/rear -D recover
Using log file: /var/log/rear/rear-localhost.log
Using build area: /var/tmp/rear.XDBgxpA2pnwa9Yt
Running 'init' stage
Running workflow recover within the ReaR rescue/recovery system
Running 'setup' stage
Running 'verify' stage
Using backup archive '/var/tmp/rear.XDBgxpA2pnwa9Yt/outputfs/rear/localhost/20211027.1337/backup.tar.gz'
Will do driver migration (recreating initramfs/initrd)
Backup archive /var/tmp/rear.XDBgxpA2pnwa9Yt/outputfs/rear/localhost/20211025.1417/backup.tar.gz detected.
Backup archive /var/tmp/rear.XDBgxpA2pnwa9Yt/outputfs/rear/localhost/20211027.1337/backup.tar.gz detected.
Select a backup archive.
1) 20211025.1417
2) 20211027.1337
#? 2
Using backup archive '/var/tmp/rear.XDBgxpA2pnwa9Yt/outputfs/rear/localhost/20211027.1337/backup.tar.gz'.
Calculating backup archive size
Backup archive size is 1.2G     /var/tmp/rear.XDBgxpA2pnwa9Yt/outputfs/rear/localhost/20211027.1337/backup.tar.gz (compressed)
Running 'layout/prepare' stage
Comparing disks
Device sda is designated as write-protected (needs manual configuration)
Switching to manual disk layout configuration
Cannot use /dev/sda (same name and same size) for recreating /dev/sda (/dev/sda is write-protected)
Cannot use /dev/sda (same size) for recreating /dev/sda (/dev/sda is write-protected)
Could not automap /dev/sda (no disk with same size 8589934592 found)
Original disk /dev/sda does not exist (with same size) in the target system
No device found where to /dev/sda could be mapped so that it will not be recreated
Current disk mapping table (source => target):
  There is no valid disk mapping
Currently unmapped disks and dependant devices will not be recreated
(unless identical disk mapping and proceeding without manual configuration):
  /dev/sda /dev/sda1 /dev/sda2 fs:/
UserInput -I LAYOUT_MIGRATION_CONFIRM_MAPPINGS needed in /usr/share/rear/layout/prepare/default/300_map_disks.sh line 306
Confirm or edit the disk mapping
1) Confirm disk mapping and continue 'rear recover'
2) Confirm identical disk mapping and proceed without manual configuration
3) Edit disk mapping (/var/lib/rear/layout/disk_mappings)
4) Use Relax-and-Recover shell and return back to here
5) Abort 'rear recover'
(default '1' timeout 300 seconds)
4
UserInput: Valid choice number result 'Use Relax-and-Recover shell and return back to here'

Welcome to Relax-and-Recover.

rear> echo "/dev/sda /dev/sdb" >/var/lib/rear/layout/disk_mappings
rear> exit
Are you sure you want to exit the Relax-and-Recover shell ? y
exit
Current disk mapping table (source => target):
  /dev/sda => /dev/sdb

UserInput -I LAYOUT_MIGRATION_CONFIRM_MAPPINGS needed in /usr/share/rear/layout/prepare/default/300_map_disks.sh line 306
Confirm or edit the disk mapping
1) Confirm disk mapping and continue 'rear recover'
2) n/a
3) Edit disk mapping (/var/lib/rear/layout/disk_mappings)
4) Use Relax-and-Recover shell and return back to here
5) Abort 'rear recover'
(default '1' timeout 300 seconds)

UserInput: No real user input (empty or only spaces) - using default input
UserInput: Valid choice number result 'Confirm disk mapping and continue 'rear recover''
User confirmed disk mapping
Disabling excluded components in /var/lib/rear/layout/disklayout.conf
Applied disk layout mappings to /var/lib/rear/layout/disklayout.conf
Applied disk layout mappings to /var/lib/rear/layout/config/df.txt
Applied disk layout mappings to /etc/rear/rescue.conf
Examining gpt disk /dev/sdb to automatically resize its last active partition
Checking /dev/sdb1 if it is the last partition on /dev/sdb
Checking /dev/sdb2 if it is the last partition on /dev/sdb
Found 'rear-noname' partition /dev/sdb2 as last partition on /dev/sdb
Determining if last partition /dev/sdb2 is resizeable
Last partition /dev/sdb2 not resizeable (used during boot)
Determining new size for last partition /dev/sdb2
Determining if last partition /dev/sdb2 actually needs to be increased or shrinked
New /dev/sdb is 1073741824 bytes bigger than old disk
Skip increasing last partition /dev/sdb2 (not resizeable)
UserInput -I LAYOUT_FILE_CONFIRMATION needed in /usr/share/rear/layout/prepare/default/500_confirm_layout_file.sh line 26
Confirm or edit the disk layout file
1) Confirm disk layout and continue 'rear recover'
2) Edit disk layout (/var/lib/rear/layout/disklayout.conf)
3) View disk layout (/var/lib/rear/layout/disklayout.conf)
4) View original disk space usage (/var/lib/rear/layout/config/df.txt)
5) Use Relax-and-Recover shell and return back to here
6) Abort 'rear recover'
(default '1' timeout 300 seconds)

UserInput: No real user input (empty or only spaces) - using default input
UserInput: Valid choice number result 'Confirm disk layout and continue 'rear recover''
User confirmed disk layout file
Marking component '/dev/sdb' as done in /var/lib/rear/layout/disktodo.conf
Marking component '/dev/sdb1' as done in /var/lib/rear/layout/disktodo.conf
Marking component '/dev/sdb2' as done in /var/lib/rear/layout/disktodo.conf
Marking component 'fs:/' as done in /var/lib/rear/layout/disktodo.conf
Running 'layout/recreate' stage
UserInput -I LAYOUT_CODE_CONFIRMATION needed in /usr/share/rear/layout/recreate/default/100_confirm_layout_code.sh line 26
Confirm or edit the disk recreation script
1) Confirm disk recreation script and continue 'rear recover'
2) Edit disk recreation script (/var/lib/rear/layout/diskrestore.sh)
3) View disk recreation script (/var/lib/rear/layout/diskrestore.sh)
4) View original disk space usage (/var/lib/rear/layout/config/df.txt)
5) Use Relax-and-Recover shell and return back to here
6) Abort 'rear recover'
(default '1' timeout 300 seconds)

UserInput: No real user input (empty or only spaces) - using default input
UserInput: Valid choice number result 'Confirm disk recreation script and continue 'rear recover''
User confirmed disk recreation script
UserInput -I WIPE_DISKS_CONFIRMATION needed in /usr/share/rear/layout/recreate/default/120_confirm_wipedisk_disks.sh line 36
Disks to be overwritten: /dev/sdb 
1) Confirm disks to be completely overwritten and continue 'rear recover'
2) Use Relax-and-Recover shell and return back to here
3) Abort 'rear recover'
(default '1' timeout 300 seconds)

UserInput: No real user input (empty or only spaces) - using default input
UserInput: Valid choice number result 'Confirm disks to be completely overwritten and continue 'rear recover''
User confirmed disks to be overwritten
Wiping child devices of /dev/sdb in reverse ordering: /dev/sdb2 /dev/sdb1 /dev/sdb 
Wiped first 16777216 bytes of /dev/sdb2
Wiped last 16777216 bytes of /dev/sdb2
Wiped first 8388608 bytes of /dev/sdb1
Skip wiping at the end of /dev/sdb1 (dvice size 8388608 not greater than the bytes that were wiped)
Wiped first 16777216 bytes of /dev/sdb
Wiped last 16777216 bytes of /dev/sdb
Start system layout restoration.
Disk '/dev/sdb': creating 'gpt' partition table
Disk '/dev/sdb': creating partition number 1 with name ''sdb1''
Disk '/dev/sdb': creating partition number 2 with name ''sdb2''
Creating filesystem of type ext4 with mount point / on /dev/sdb2.
Mounting filesystem /
Disk layout created.
UserInput -I LAYOUT_MIGRATED_CONFIRMATION needed in /usr/share/rear/layout/recreate/default/200_run_layout_code.sh line 98
Confirm the recreated disk layout or go back one step
1) Confirm recreated disk layout and continue 'rear recover'
2) Go back one step to redo disk layout recreation
3) Use Relax-and-Recover shell and return back to here
4) Abort 'rear recover'
(default '1' timeout 300 seconds)

UserInput: No real user input (empty or only spaces) - using default input
UserInput: Valid choice number result 'Confirm recreated disk layout and continue 'rear recover''
User confirmed recreated disk layout
Running 'restore' stage
Restoring from '/var/tmp/rear.XDBgxpA2pnwa9Yt/outputfs/rear/localhost/20211027.1337/backup.tar.gz' (restore log in /var/lib/rear/restore/recover.backup.tar.gz.666.restore.log) ...
Backup restore program 'tar' started in subshell (PID=3791)
Restored 155 MiB [avg. 53170 KiB/sec] 
.
.
.
OK
Restored 2822 MiB in 109 seconds [avg. 26518 KiB/sec]
Restoring finished (verify backup restore log messages in /var/lib/rear/restore/recover.backup.tar.gz.666.restore.log)
Created SELinux /mnt/local/.autorelabel file : after reboot SELinux will relabel all files
Recreating directories (with permissions) from /var/lib/rear/recovery/directories_permissions_owner_group
Running 'finalize' stage
Applying disk layout mappings in /var/lib/rear/layout/disk_mappings to certain restored files...
The original restored files get saved in var/lib/rear/saved_original_files/ (in /mnt/local)
Applied disk layout mappings to restored 'boot/grub2/grub.cfg' (in /mnt/local)
Failed to apply layout mappings to boot/grub2/device.map for /dev/sdb (probably no mapping for /dev/sdb in /var/lib/rear/layout/disk_mappings)
Failed to apply disk layout mappings to restored 'boot/grub2/device.map' (in /mnt/local)
Applied disk layout mappings to restored 'etc/sysconfig/bootloader' (in /mnt/local)
Skip patching symlink etc/mtab target /mnt/local/proc/4429/mounts on /proc/ /sys/ /dev/ or /run/
Applied disk layout mappings to restored 'etc/fstab' (in /mnt/local)
Applied disk layout mappings to restored 'etc/smartd.conf' (in /mnt/local)
Applied disk layout mappings to restored 'etc/sysconfig/smartmontools' (in /mnt/local)
Failed to apply layout mappings to /var/lib/rear/recovery/diskbyid_mappings for /dev/sdb (probably no mapping for /dev/sdb in /var/lib/rear/layout/disk_mappings)
Failed to apply layout mappings to /var/lib/rear/recovery/diskbyid_mappings (may cause failures during 'rename_diskbyid')
Migrating disk-by-id mappings in certain restored files in /mnt/local to current disk-by-id mappings ...
Replaced '/dev/disk/by-id/ata-QEMU_HARDDISK_QM00001' by '/dev/disk/by-id/ata-QEMU_HARDDISK_QM00002' in /mnt/local//etc/default/grub_installdevice
Replacing restored udev rule '/mnt/local//etc/udev/rules.d/70-persistent-net.rules' with the one from the ReaR rescue system
Migrating restored network configuration files according to the mapping files ...
UserInput -I RESTORED_FILES_CONFIRMATION needed in /usr/share/rear/finalize/default/520_confirm_finalize.sh line 41
Confirm restored config files are OK or adapt them as needed
1) Confirm it is OK to recreate initrd and reinstall bootloader and continue 'rear recover'
2) Edit restored etc/fstab (/mnt/local/etc/fstab)
3) View restored etc/fstab (/mnt/local/etc/fstab)
4) Use Relax-and-Recover shell and return back to here
5) Abort 'rear recover'
(default '1' timeout 300 seconds)
UserInput: No real user input (empty or only spaces) - using default input
UserInput: Valid choice number result 'Confirm it is OK to recreate initrd and reinstall bootloader and continue 'rear recover''
Continuing 'rear recover' by default
Running mkinitrd...
Recreated initrd (/sbin/mkinitrd).
Installing GRUB2 boot loader...
Determining where to install GRUB2 (no GRUB2_INSTALL_DEVICES specified)
Found possible boot disk /dev/sdb - installing GRUB2 there
Running 'wrapup' stage
Finished 'recover'. The target system is mounted at '/mnt/local'.
Exiting rear recover (PID 666) and its descendant processes ...
Running exit tasks
To remove the build area you may use (with caution): rm -Rf --one-file-system /var/tmp/rear.XDBgxpA2pnwa9Yt

The recreated system boots well from sdb and seems to work OK.

jsmeix commented at 2021-10-27 12:51:¶

Excerpts how automated disk mapping changed from
https://github.com/rear/rear/pull/2626#issuecomment-950935246

Comparing disks
Device sda is designated as write-protected (needs manual configuration)
Switching to manual disk layout configuration
Cannot use /dev/sda (same name and same size) for recreating /dev/sda (/dev/sda is write-protected)
Cannot use /dev/sda (same size) for recreating /dev/sda (/dev/sda is write-protected)
Could not automap /dev/sda (no disk with same size 8589934592 found)
Original disk /dev/sda does not exist (with same size) in the target system
Using /dev/sdb (the only appropriate) for recreating /dev/sda
Current disk mapping table (source => target):
  /dev/sda => /dev/sdb

to here where the latest Github master code is used

Comparing disks
Device sda is designated as write-protected (needs manual configuration)
Switching to manual disk layout configuration
Cannot use /dev/sda (same name and same size) for recreating /dev/sda (/dev/sda is write-protected)
Cannot use /dev/sda (same size) for recreating /dev/sda (/dev/sda is write-protected)
Could not automap /dev/sda (no disk with same size 8589934592 found)
Original disk /dev/sda does not exist (with same size) in the target system
No device found where to /dev/sda could be mapped so that it will not be recreated
Current disk mapping table (source => target):
  There is no valid disk mapping
Currently unmapped disks and dependant devices will not be recreated
(unless identical disk mapping and proceeding without manual configuration):
  /dev/sda /dev/sda1 /dev/sda2 fs:/

so there is no longer an automated mapping to
"just the last available disk regardless of its size"
which means the issue in the "Note" at the end of
https://github.com/rear/rear/pull/2626#issue-662429889
seems to be solved in latest Github master code.

jsmeix commented at 2021-10-27 13:12:¶

The "rear recover" log shows why /dev/sdb excluded from device mapping choices

... /dev/sdb is designated as write-protected by UUID 8bd9f9e8-9043-4a80-84e1-64d417244529
... 
... /dev/sdb excluded from device mapping choices (is designated as write-protected)

which is wrong because UUID 8bd9f9e8-9043-4a80-84e1-64d417244529
does not belong to sdb but to sda

# lsblk --output NAME,LABEL,UUID,PTUUID,PARTUUID
NAME   LABEL   UUID                                 PTUUID                               PARTUUID
sda                                                 efee5e95-40a0-4a8e-bbe1-1cad7c0d255d 
├─sda1                                              efee5e95-40a0-4a8e-bbe1-1cad7c0d255d aa3d3f8b-3adb-4ef8-a4c0-0e44cdadfff6
├─sda2 MY-BOOT ff15e0f1-b4bb-4a02-bb10-1502ec2fe0be efee5e95-40a0-4a8e-bbe1-1cad7c0d255d 281246c5-d29f-4ed2-8cfb-c7404eca010d
└─sda3 MY-DATA 8bd9f9e8-9043-4a80-84e1-64d417244529 efee5e95-40a0-4a8e-bbe1-1cad7c0d255d e38c933c-d18f-4821-92c9-c99673738660
sdb                                                 fb6e13f1-f9c6-490f-b9dc-c49e10639cd9 
├─sdb1                                              fb6e13f1-f9c6-490f-b9dc-c49e10639cd9 e8c2b900-3064-401b-8ad7-fa2b45d65f73
└─sdb2         83a3f413-6e7d-4591-bd60-c87803613166 fb6e13f1-f9c6-490f-b9dc-c49e10639cd9 6a2c6852-e1d9-4b97-9e64-86f6d17e4b2b

In particular the "rear recover" log shows

... /dev/sda is designated as write-protected by UUID 8bd9f9e8-9043-4a80-84e1-64d417244529
... /dev/sdb is designated as write-protected by UUID 8bd9f9e8-9043-4a80-84e1-64d417244529

so same UUID for both sda and sdb.

Investigating...

jsmeix commented at 2021-10-27 13:55:¶

With latest changes things look better:

"rear recover" excerpts:

Comparing disks
Device sda is designated as write-protected (needs manual configuration)
Switching to manual disk layout configuration
Cannot use /dev/sda (same name and same size) for recreating /dev/sda (/dev/sda is write-protected)
Cannot use /dev/sda (same size) for recreating /dev/sda (/dev/sda is write-protected)
Could not automap /dev/sda (no disk with same size 8589934592 found)
Original disk /dev/sda does not exist (with same size) in the target system
Using /dev/sdb (the only appropriate) for recreating /dev/sda
Current disk mapping table (source => target):
  /dev/sda => /dev/sdb

UserInput -I LAYOUT_MIGRATION_CONFIRM_MAPPINGS needed in /usr/share/rear/layout/prepare/default/300_map_disks.sh line 306
Confirm or edit the disk mapping
1) Confirm disk mapping and continue 'rear recover'
2) n/a
3) Edit disk mapping (/var/lib/rear/layout/disk_mappings)
4) Use Relax-and-Recover shell and return back to here
5) Abort 'rear recover'

All user dialogs were accepted with the default (i.e. just proceed) and
the recreated system boots well from sdb and seems to work OK.

jsmeix commented at 2021-10-27 14:00:¶

@rear/contributors
please have a look here (as time permits)

pcahyna commented at 2021-10-27 14:04:¶

@jsmeix I will have a look next week (this week we will have a holiday).

jsmeix commented at 2021-10-28 06:31:¶

@pcahyna
you got the reviewer job ;-)
Enjoy your holiday - relax and recover !

jsmeix commented at 2021-10-28 10:56:¶

With latest changes now all UUIDs are used.

After "rear mkbackup" I have in /etc/rear/rescue.conf
(I wrapped the UUIDs here for better readability)

WRITE_PROTECTED_UUIDS=( 281246c5-d29f-4ed2-8cfb-c7404eca010d
 8bd9f9e8-9043-4a80-84e1-64d417244529 aa3d3f8b-3adb-4ef8-a4c0-0e44cdadfff6
 e38c933c-d18f-4821-92c9-c99673738660 efee5e95-40a0-4a8e-bbe1-1cad7c0d255d
 ff15e0f1-b4bb-4a02-bb10-1502ec2fe0be )
WRITE_PROTECTED_FS_LABEL_PATTERNS=( 'MY-DATA' )

which are all UUIDs of the ReaR's own disk sda in the recovery system

RESCUE localhost:~ # lsblk -ipo KNAME,LABEL,UUID,PTUUID,PARTUUID /dev/sda
KNAME     LABEL   UUID                                 PTUUID                               PARTUUID
/dev/sda                                               efee5e95-40a0-4a8e-bbe1-1cad7c0d255d 
/dev/sda1                                              efee5e95-40a0-4a8e-bbe1-1cad7c0d255d aa3d3f8b-3adb-4ef8-a4c0-0e44cdadfff6
/dev/sda2 MY-BOOT ff15e0f1-b4bb-4a02-bb10-1502ec2fe0be efee5e95-40a0-4a8e-bbe1-1cad7c0d255d 281246c5-d29f-4ed2-8cfb-c7404eca010d
/dev/sda3 MY-DATA 8bd9f9e8-9043-4a80-84e1-64d417244529 efee5e95-40a0-4a8e-bbe1-1cad7c0d255d e38c933c-d18f-4821-92c9-c99673738660

jsmeix commented at 2021-10-28 11:20:¶

Tested "rear recover" with only
WRITE_PROTECTED_FS_LABEL_PATTERNS=( 'MY-DATA' )
in /etc/rear/rescue.conf
(i.e. without any WRITE_PROTECTED_UUIDS)
which also works OK for me (excerpt)

Comparing disks
Device sda is designated as write-protected (needs manual configuration)
Switching to manual disk layout configuration
Cannot use /dev/sda (same name and same size) for recreating /dev/sda (/dev/sda is write-protected)
Cannot use /dev/sda (same size) for recreating /dev/sda (/dev/sda is write-protected)
Could not automap /dev/sda (no disk with same size 8589934592 found)
Original disk /dev/sda does not exist (with same size) in the target system
Using /dev/sdb (the only appropriate) for recreating /dev/sda
Current disk mapping table (source => target):
  /dev/sda => /dev/sdb

and the "rear -D recover" log contains

... /dev/sda is not write-protected by UUID
... /dev/sda is designated as write-protected, its label 'MY-DATA' matches 'MY-DATA'

jsmeix commented at 2021-10-29 12:51:¶

I would like to optimize things by avoiding needless duplicate tests like

Cannot use /dev/sda (same name and same size) for recreating /dev/sda (/dev/sda is write-protected)
Cannot use /dev/sda (same size) for recreating /dev/sda (/dev/sda is write-protected)

cf. https://github.com/rear/rear/pull/2703#issuecomment-953751199

jsmeix commented at 2021-10-29 12:55:¶

With latest commit needless duplicate tests do no longer happen.

"rear -D recover" excerpts in same test environment as above in
https://github.com/rear/rear/pull/2703#issuecomment-952955861

Comparing disks
Device sda is designated as write-protected (needs manual configuration)
Switching to manual disk layout configuration
Cannot use /dev/sda (same name and same size) for recreating /dev/sda (/dev/sda is write-protected)
Could not automap /dev/sda (no disk with same size 8589934592 found)
Original disk /dev/sda does not exist (with same size) in the target system
Using /dev/sdb (the only appropriate) for recreating /dev/sda
Current disk mapping table (source => target):
  /dev/sda => /dev/sdb

jsmeix commented at 2021-10-29 13:14:¶

The latest commit
https://github.com/rear/rear/pull/2703/commits/4738313cdc011e905b8e63dbb012748e0dcb9f79
should at least a bit address the issue in the "Note" at the end of
https://github.com/rear/rear/pull/2626#issue-662429889
by less misleading wording why that disk is chosen
so it should now look like

Comparing disks
Device sda is designated as write-protected (needs manual configuration)
Switching to manual disk layout configuration
Cannot use /dev/sda (same name and same size) for recreating /dev/sda (/dev/sda is write-protected)
Could not automap /dev/sda (no disk with same size 8589934592 found)
Original disk /dev/sda does not exist (with same size) in the target system
Using /dev/sdb (the only available of the disks) for recreating /dev/sda
Current disk mapping table (source => target):
  /dev/sda => /dev/sdb

jsmeix commented at 2021-11-03 14:40:¶

@pcahyna
do you have time to have a look here?

pcahyna commented at 2021-11-03 15:15:¶

@jsmeix I am going to have a look tomorrow, sorry for the delay

jsmeix commented at 2021-11-05 11:50:¶

I did not yet test my recent commits.

jsmeix commented at 2021-11-05 12:12:¶

@pcahyna
I think we cannot use SERIAL because on my KVM/QEMU system I get

# lsblk -ipo KNAME,LABEL,UUID,PTUUID,PARTUUID,SERIAL,WWN /dev/sda
KNAME     LABEL UUID                                 PTUUID                               PARTUUID                             SERIAL  WWN
/dev/sda                                             fc224eb1-9058-4b88-864a-1fb89018ecf7                                      QM00001 
/dev/sda1                                            fc224eb1-9058-4b88-864a-1fb89018ecf7 735f37c2-d5f1-4874-a546-b25d99b5cdfc         
/dev/sda2       83a3f413-6e7d-4591-bd60-c87803613166 fc224eb1-9058-4b88-864a-1fb89018ecf7 d5166397-8e27-41f4-94e1-428549ca36c4

# lsblk -ipo KNAME,LABEL,UUID,PTUUID,PARTUUID,SERIAL,WWN /dev/sdb
KNAME     LABEL    UUID                                 PTUUID                               PARTUUID                             SERIAL  WWN
/dev/sdb                                                e3c7fe07-f86b-434b-aefe-c5e64eda0e36                                      QM00003 
/dev/sdb1                                               e3c7fe07-f86b-434b-aefe-c5e64eda0e36 fc826b1b-6ed7-46d0-9f47-948cb77b4b42         
/dev/sdb2 REAR-EFI 296B-6A05                            e3c7fe07-f86b-434b-aefe-c5e64eda0e36 da1f0a9c-bb29-43dd-9161-0b72074c91ac         
/dev/sdb3 MY-BOOT  5badfdc3-10eb-4f0c-a4e0-a6a5176781de e3c7fe07-f86b-434b-aefe-c5e64eda0e36 a1fee4e3-fcbe-4d5a-8e6d-2105669d514c         
/dev/sdb4 MY-DATA  c0beb048-c9d8-4e17-a1cd-5c02096a6a38 e3c7fe07-f86b-434b-aefe-c5e64eda0e36 0137fd52-555d-4a16-8dc1-0240571f9c8b

where SERIAL QM00001 and QM00003 are much too less unique
to be useful as reasonably unique disk identifiers.

Another possibly too less unique UUID is

KNAME     LABEL    UUID
...
/dev/sdb2 REAR-EFI 296B-6A05

where 296B-6A05 does not look like a real UUID
but more like some rather static default/fallback value.

With that I get after "rear mkbackup" in /var/tmp/rear.XXX/rootfs/etc/rear/rescue.conf

WRITE_PROTECTED_IDS=( 0137fd52-555d-4a16-8dc1-0240571f9c8b 296B-6A05
 5badfdc3-10eb-4f0c-a4e0-a6a5176781de QM00003 a1fee4e3-fcbe-4d5a-8e6d-2105669d514c
 c0beb048-c9d8-4e17-a1cd-5c02096a6a38 da1f0a9c-bb29-43dd-9161-0b72074c91ac
 e3c7fe07-f86b-434b-aefe-c5e64eda0e36 fc826b1b-6ed7-46d0-9f47-948cb77b4b42 )
WRITE_PROTECTED_FS_LABEL_PATTERNS=( 'MY-DATA' )

jsmeix commented at 2021-11-05 12:59:¶

In the ReaR recovery system on the replacement VM I have now

RESCUE localhost:~ # lsblk -ipo KNAME,LABEL,UUID,PTUUID,PARTUUID,SERIAL,WWN /dev/sda
KNAME     LABEL    UUID                                 PTUUID                               PARTUUID                             SERIAL  WWN
/dev/sda                                                e3c7fe07-f86b-434b-aefe-c5e64eda0e36                                      QM00001 
/dev/sda1                                               e3c7fe07-f86b-434b-aefe-c5e64eda0e36 fc826b1b-6ed7-46d0-9f47-948cb77b4b42         
/dev/sda2 REAR-EFI 296B-6A05                            e3c7fe07-f86b-434b-aefe-c5e64eda0e36 da1f0a9c-bb29-43dd-9161-0b72074c91ac         
/dev/sda3 MY-BOOT  5badfdc3-10eb-4f0c-a4e0-a6a5176781de e3c7fe07-f86b-434b-aefe-c5e64eda0e36 a1fee4e3-fcbe-4d5a-8e6d-2105669d514c         
/dev/sda4 MY-DATA  c0beb048-c9d8-4e17-a1cd-5c02096a6a38 e3c7fe07-f86b-434b-aefe-c5e64eda0e36 0137fd52-555d-4a16-8dc1-0240571f9c8b

RESCUE localhost:~ # lsblk -ipo KNAME,LABEL,UUID,PTUUID,PARTUUID,SERIAL,WWN /dev/sdb
KNAME     LABEL UUID                                 PTUUID                               PARTUUID                             SERIAL  WWN
/dev/sdb                                             b87810ea-83f1-4583-a8d7-e095820e72cb                                      QM00002 
/dev/sdb1                                            b87810ea-83f1-4583-a8d7-e095820e72cb 859897f6-e166-4d07-a2a4-bd9c74fdada4         
/dev/sdb2       83a3f413-6e7d-4591-bd60-c87803613166 b87810ea-83f1-4583-a8d7-e095820e72cb b60fb4b2-0242-40fd-b152-520026504d78

so the SERIAL number of that exact same ReaR disk
changed from QM00003 on the original system
to QM00001 on the replacement system
so for KVM/QEMU disks the SERIAL numbers
are some completely useless fallback values
so we cannot use them.

Nevertheless by luck "rear recover" just worked
because the sdb SERIAL QM00002 is not one that is remembered as write-protected.

jsmeix commented at 2021-11-05 13:26:¶

With latest commits "rear mkbackup" and "rear recover" work well for me.

I could not test WWN because the disks on my test VMs don't have a WWN set.

jsmeix commented at 2021-11-06 08:30:¶

As always sleeping on an issue helps:

From my above experience with different behaviour what values
a certain lsblk column can have in different storage environments
I do not like the current hardcoded list of lsblk columns in
the function write_protection_ids()

for column in UUID PTUUID PARTUUID WWN

So I will add one more config variable in default.conf like

WRITE_PROTECTED_ID_TYPES="UUID PTUUID PARTUUID WWN"

and then use it in write_protection_ids() like

for column in $WRITE_PROTECTED_ID_TYPES

This provides final power to the user
so if needed he can specify the lsblk columns he wants
and it avoids bugs for us when UUID PTUUID PARTUUID WWN
cause problems in this or that unforeseen special cases.

For example I have no idea how UUID PTUUID PARTUUID WWN
behave for multipath (i.e. when same disks appear multiple times).
Even if it is rather unlikely that someone uses a multipath disk
as ReaR recovery system disk it is not impossible.
I would expect that when the ReaR recovery system disk
appears multiple times things "just work" because each appearance
of the ReaR recovery system disk must be write-protected.
But I don't know how things actually behave in practice.
E.g. each appearance of the ReaR recovery system disk is write-protected
but its higher level multipath device is not write-protected and
its higher level multipath device is what could get overwritten.

jsmeix commented at 2021-11-08 16:19:¶

Implemented https://github.com/rear/rear/pull/2703#issuecomment-962418441
via the recent commits.

The same test as above works well for me with that commits.

jsmeix commented at 2021-11-09 13:42:¶

Tested again with external real USB disk as ReaR disk
on the above two VMs.

I have now on the original VM

# lsblk -ipo NAME,KNAME,MODEL,LABEL,SERIAL
NAME        KNAME     MODEL            LABEL    SERIAL
/dev/sda    /dev/sda  QEMU_HARDDISK             QM00001
|-/dev/sda1 /dev/sda1                           
`-/dev/sda2 /dev/sda2                           
/dev/sdb    /dev/sdb  QEMU_HARDDISK             QM00003
|-/dev/sdb1 /dev/sdb1                           
|-/dev/sdb2 /dev/sdb2                  REAR-EFI 
|-/dev/sdb3 /dev/sdb3                  MY-BOOT  
`-/dev/sdb4 /dev/sdb4                  MY-DATA  
/dev/sdc    /dev/sdc  External_USB_3.0          20170521000273F
|-/dev/sdc1 /dev/sdc1                           
|-/dev/sdc2 /dev/sdc2                  REAR-EFI 
|-/dev/sdc3 /dev/sdc3                  MY-BOOT  
`-/dev/sdc4 /dev/sdc4                  MY-DATA

I use this etc/rear/local.conf

DISKS_TO_BE_WIPED=''
FIRMWARE_FILES=( 'no' )
MODULES=( 'loaded_modules' )
PROGRESS_MODE="plain"
PROGRESS_WAIT_SECONDS="3"
SSH_ROOT_PASSWORD="rear"
OUTPUT=USB
USB_DEVICE_FILESYSTEM_PERCENTAGE=90
USB_DEVICE_FILESYSTEM_LABEL="MY-DATA"
USB_BOOT_PART_SIZE=1024
USB_BOOTLOADER="grub"
USB_DEVICE_BOOT_LABEL="MY-BOOT"
OUTPUT_URL=usb:///dev/disk/by-label/MY-BOOT
USB_DEVICE_PARTED_LABEL=gpt
BACKUP=NETFS
BACKUP_URL=usb:///dev/disk/by-label/MY-DATA
WRITE_PROTECTED_ID_TYPES="MODEL"
WRITE_PROTECTED_IDS=( "External_USB_3.0" )

i.e. same as above with two added WRITE_PROTECTED_ entries
to test how things work with one single different ID type.

"rear mkbackup" (excerpts)

# usr/sbin/rear -D mkbackup
...
USB disk IDs of '/dev/disk/by-label/MY-DATA' added to WRITE_PROTECTED_IDS
File system label of '/dev/disk/by-label/MY-DATA' added to WRITE_PROTECTED_FS_LABEL_PATTERNS

After "rear -D mkbackup" I got now (excerpt)

# cat /var/tmp/rear.7iI69pq3OvRqLyc/rootfs/etc/rear/rescue.conf
...
WRITE_PROTECTED_IDS=( External_USB_3.0 External_USB_3.0 )
WRITE_PROTECTED_FS_LABEL_PATTERNS=( 'MY-DATA' )
...

The duplicate External_USB_3.0 comes from my setting in local.conf
plus the automated addition by prep/USB/default/480_initialize_write_protect_settings.sh
but duplicated IDs in WRITE_PROTECTED_IDS don't matter.

The WRITE_PROTECTED_FS_LABEL_PATTERNS=( 'MY-DATA' )
which comes from /dev/sdc4 will also write-protect /dev/sda
on the replacement VM by luck because there 'MY-DATA'
is also used on /dev/sda4 - see the following:

On the replacement VM I have

RESCUE localhost:~ # lsblk -ipo NAME,KNAME,MODEL,LABEL,SERIAL
NAME        KNAME     MODEL            LABEL    SERIAL
/dev/sda    /dev/sda  QEMU_HARDDISK             QM00001
|-/dev/sda1 /dev/sda1                           
|-/dev/sda2 /dev/sda2                  REAR-EFI 
|-/dev/sda3 /dev/sda3                  MY-BOOT  
`-/dev/sda4 /dev/sda4                  MY-DATA  
/dev/sdb    /dev/sdb  QEMU_HARDDISK             QM00002
|-/dev/sdb1 /dev/sdb1                           
`-/dev/sdb2 /dev/sdb2                           
/dev/sdc    /dev/sdc  External_USB_3.0          20170521000273F
|-/dev/sdc1 /dev/sdc1                           
|-/dev/sdc2 /dev/sdc2                  REAR-EFI 
|-/dev/sdc3 /dev/sdc3                  MY-BOOT  
`-/dev/sdc4 /dev/sdc4                  MY-DATA

RESCUE localhost:~ # rear -D recover
...
Comparing disks
Device sda is designated as write-protected (needs manual configuration)
Switching to manual disk layout configuration
Cannot use /dev/sda (same name and same size) for recreating /dev/sda (/dev/sda is write-protected)
Could not automap /dev/sda (no disk with same size 8589934592 found)
Original disk /dev/sda does not exist (with same size) in the target system
Using /dev/sdb (the only available of the disks) for recreating /dev/sda
Current disk mapping table (source => target):
  /dev/sda => /dev/sdb

The "rear recover" log shows the write-protection test results

RESCUE localhost:~ # grep '^2021.* write-protected' /var/log/rear/rear-localhost.log
2021-11-09 14:14:17.231764721 /dev/sda is not write-protected by ID
2021-11-09 14:14:17.240623355 /dev/sda is designated as write-protected, its label 'MY-DATA' matches 'MY-DATA'
2021-11-09 14:14:17.261690422 /dev/sdb is not write-protected by ID
2021-11-09 14:14:17.269836792 /dev/sdb is not write-protected by file system label
2021-11-09 14:14:17.293289865 /dev/sdc is designated as write-protected by ID External_USB_3.0
2021-11-09 14:14:17.328838709 /dev/sda is not write-protected by ID
2021-11-09 14:14:17.338716114 /dev/sda is designated as write-protected, its label 'MY-DATA' matches 'MY-DATA'
2021-11-09 14:14:17.342630182 Device sda is designated as write-protected (needs manual configuration)
2021-11-09 14:14:17.448098582 /dev/sda is not write-protected by ID
2021-11-09 14:14:17.456840183 /dev/sda is designated as write-protected, its label 'MY-DATA' matches 'MY-DATA'
2021-11-09 14:14:17.460422624 Cannot use /dev/sda (same name and same size) for recreating /dev/sda (/dev/sda is write-protected)
2021-11-09 14:14:17.518655279 /dev/sda is not write-protected by ID
2021-11-09 14:14:17.527663776 /dev/sda is designated as write-protected, its label 'MY-DATA' matches 'MY-DATA'
2021-11-09 14:14:17.531242534 /dev/sda excluded from device mapping choices (is designated as write-protected)
2021-11-09 14:14:17.555084512 /dev/sdb is not write-protected by ID
2021-11-09 14:14:17.563664684 /dev/sdb is not write-protected by file system label
2021-11-09 14:14:17.587401855 /dev/sdc is designated as write-protected by ID External_USB_3.0
2021-11-09 14:14:17.591366414 /dev/sdc excluded from device mapping choices (is designated as write-protected)

So a different ID type also works well for me.

jsmeix commented at 2021-11-09 13:42:¶

If there are no objections I would like to merge it tomorrow afternoon.

jsmeix commented at 2021-11-10 12:32:¶

I did one more test on the above two VMs
but without external real USB disk as ReaR disk.

Now I have in local.conf only

WRITE_PROTECTED_ID_TYPES="PTUUID"

but no WRITE_PROTECTED_IDS.

This results basically the same behaviour as before this pull request
where only PTUUID was used as ID.

After "rear mkbackup" I have

# cat /var/tmp/rear.n6Yn1A9hsPcm7E2/rootfs/etc/rear/rescue.conf
...
WRITE_PROTECTED_IDS=( e3c7fe07-f86b-434b-aefe-c5e64eda0e36 )
WRITE_PROTECTED_FS_LABEL_PATTERNS=( 'MY-DATA' )
...

# lsblk -ipo NAME,LABEL,UUID,PTUUID,PARTUUID /dev/sdb
NAME        LABEL    UUID                                 PTUUID                               PARTUUID
/dev/sdb                                                  e3c7fe07-f86b-434b-aefe-c5e64eda0e36 
|-/dev/sdb1                                               e3c7fe07-f86b-434b-aefe-c5e64eda0e36 fc826b1b-6ed7-46d0-9f47-948cb77b4b42
|-/dev/sdb2 REAR-EFI 296B-6A05                            e3c7fe07-f86b-434b-aefe-c5e64eda0e36 da1f0a9c-bb29-43dd-9161-0b72074c91ac
|-/dev/sdb3 MY-BOOT  5badfdc3-10eb-4f0c-a4e0-a6a5176781de e3c7fe07-f86b-434b-aefe-c5e64eda0e36 a1fee4e3-fcbe-4d5a-8e6d-2105669d514c
`-/dev/sdb4 MY-DATA  c0beb048-c9d8-4e17-a1cd-5c02096a6a38 e3c7fe07-f86b-434b-aefe-c5e64eda0e36 0137fd52-555d-4a16-8dc1-0240571f9c8b

During "rear recover" I get on the terminal

Comparing disks
Device sda is designated as write-protected (needs manual configuration)
Switching to manual disk layout configuration
Cannot use /dev/sda (same name and same size) for recreating /dev/sda (/dev/sda is write-protected)
Could not automap /dev/sda (no disk with same size 8589934592 found)
Original disk /dev/sda does not exist (with same size) in the target system
Using /dev/sdb (the only available of the disks) for recreating /dev/sda
Current disk mapping table (source => target):
  /dev/sda => /dev/sdb

and the recovery log shows

RESCUE localhost:~ # grep '^2021.* write-protected' /var/log/rear/rear-localhost.log
2021-11-10 13:19:06.123714730 /dev/sda is designated as write-protected by ID e3c7fe07-f86b-434b-aefe-c5e64eda0e36
2021-11-10 13:19:06.153061505 /dev/sdb is not write-protected by ID
2021-11-10 13:19:06.163243663 /dev/sdb is not write-protected by file system label
2021-11-10 13:19:06.211360274 /dev/sda is designated as write-protected by ID e3c7fe07-f86b-434b-aefe-c5e64eda0e36
2021-11-10 13:19:06.215985676 Device sda is designated as write-protected (needs manual configuration)
2021-11-10 13:19:06.356062734 /dev/sda is designated as write-protected by ID e3c7fe07-f86b-434b-aefe-c5e64eda0e36
2021-11-10 13:19:06.360857672 Cannot use /dev/sda (same name and same size) for recreating /dev/sda (/dev/sda is write-protected)
2021-11-10 13:19:06.439972367 /dev/sda is designated as write-protected by ID e3c7fe07-f86b-434b-aefe-c5e64eda0e36
2021-11-10 13:19:06.444778534 /dev/sda excluded from device mapping choices (is designated as write-protected)
2021-11-10 13:19:06.475940197 /dev/sdb is not write-protected by ID
2021-11-10 13:19:06.485878743 /dev/sdb is not write-protected by file system label

jsmeix commented at 2021-11-26 12:02:¶

I think we need further enhanced disk write-protection
for the disks in DISKS_TO_BE_WIPED
see how that currently works
https://github.com/rear/rear/pull/2714/commits/0cb21299efea81900285a8f224e0a0c6a20160e5

[Export of Github issue for rear/rear.]

#2703 PR merged: Enhanced disk write-protection¶

jsmeix opened issue at 2021-10-27 11:15:¶

jsmeix commented at 2021-10-27 12:44:¶

jsmeix commented at 2021-10-27 12:51:¶

jsmeix commented at 2021-10-27 13:12:¶

jsmeix commented at 2021-10-27 13:55:¶

jsmeix commented at 2021-10-27 14:00:¶

pcahyna commented at 2021-10-27 14:04:¶

jsmeix commented at 2021-10-28 06:31:¶

jsmeix commented at 2021-10-28 10:56:¶

jsmeix commented at 2021-10-28 11:20:¶

jsmeix commented at 2021-10-29 12:51:¶

jsmeix commented at 2021-10-29 12:55:¶

jsmeix commented at 2021-10-29 13:14:¶

jsmeix commented at 2021-11-03 14:40:¶

pcahyna commented at 2021-11-03 15:15:¶

jsmeix commented at 2021-11-05 11:50:¶

jsmeix commented at 2021-11-05 12:12:¶

jsmeix commented at 2021-11-05 12:59:¶

jsmeix commented at 2021-11-05 13:26:¶

jsmeix commented at 2021-11-06 08:30:¶

jsmeix commented at 2021-11-08 16:19:¶

jsmeix commented at 2021-11-09 13:42:¶

jsmeix commented at 2021-11-09 13:42:¶

jsmeix commented at 2021-11-10 12:32:¶

jsmeix commented at 2021-11-26 12:02:¶

#2703 PR `merged`: Enhanced disk write-protection¶