#2626 PR merged: Stop ReaR from overwriting its own disk and backup drives

Labels: enhancement, fixed / solved / done, critical / security / legal, severe improvement

OliverO2 opened issue at 2021-06-05 12:54:

Use Cases

  1. I start rear recover on a virtual machine. The original system has three drives (sda, sdb, sdc), the recovery VM has two drives (vda, vdb), plus the recovery partition on another drive (vdc). The intention is to leave sdc unmapped. ReaR asks remapping questions for sda and sdb. Then it decides to map sdc to vdc without even asking. Continuing would overwrite the recovery system. I have to edit /var/lib/rear/layout/disk_mappings manually.

  2. I start rear recover on a virtual machine, with a USB backup drive connected as /dev/sda. The original system's boot drive was also /dev/sda. ReaR asks remapping questions, but I do not notice that the /dev/sdaX partitions on the VM are actually those of the backup drive, while the VM boot drive would have been /dev/vdaX. ReaR happily overwrites the backup drive. This should never be possible.

Background

  • Overwriting a ReaR rescue system partition is currently possible if it resides on a writable disk device (physical or virtual). This applies to the RAWDISK and USB output methods.
  • Accidentally overwriting backup devices is currently possible in any ReaR configuration.
  • Related issue: #1271.

Solution

  • This PR protects against accidentally overwriting a rescue system partition by keeping track of its partition ID. The capability is automatically enabled for the RAWDISK and USB output methods. No configuration is necessary.

  • To allow for additional user-configurable protection of devices, this PR introduces the following configuration section: ## # Designate target disk devices or partitions as write-protected to avoid being accidentally overwritten during # "rear recover" # # List of partition table UUIDs, which designate write-protected disk devices. ReaR's own disk device will be # automatically added to this list if necessary. # Example: WRITE_PROTECTED_PARTITION_TABLE_UUIDS+=("ecacbce4-e05e-4eb9-835c-ade0c3ed0fea") WRITE_PROTECTED_PARTITION_TABLE_UUIDS=() # List of (shell glob) patterns, which designate matching file system labels as write-protected partitions. # Entries may be quoted and contain blanks, but they may not contain single quotes themselves. # Example: WRITE_PROTECTED_FILE_SYSTEM_LABEL_PATTERNS+=("Backup *") WRITE_PROTECTED_FILE_SYSTEM_LABEL_PATTERNS=()

  • Disks designated as write protected or containing a partition designated as write protected will not be used as recovery targets by ReaR, whether in MIGRATION_MODE or not.

Testing

  • I have tested complete rear mkrescue and rear recover cycles for RAWDISK output in various configurations on Ubuntu 20.04.
  • I have not tested USB output, as I find it too baroque to use. However, the added code for USB is simple and should just work.

Compatibility

This PR uses command invocations and options which are present in existing ReaR code with the following exception:

Note

In migration mode, ReaR asks to remap disks until there is only one left. Then it maps the last drive without asking. The comment states:

Automatically map when only one appropriate current block device is found where to it could be mapped. At the end the mapping file is shown and the user can edit it if he does not like an automated mapping

Here, ReaR takes away control from the user (or at least makes it more complicated that it needs to be). I think this should change (but separately from this PR) as it does not conform to ReaR's principle.

jsmeix commented at 2021-06-08 11:57:

E.g. lsblk --output PTUUID --noheadings --nodeps /dev/sda
works with lsblk in util-linux 2.33.2 in SLES12
but not with lsblk in util-linux 2.19.1 in SLES11
The available columns for lsblk in util-linux 2.19.1 in SLES11:

# lsblk --help

Usage:
 lsblk [options] [<device> ...]

Options:
 -a, --all            print all devices
 -b, --bytes          print SIZE in bytes rather than in human readable format
 -d, --nodeps         don't print slaves or holders
 -e, --exclude <list> exclude devices by major number (default: RAM disks)
 -f, --fs             output info about filesystems
 -h, --help           usage information (this)
 -i, --ascii          use ascii characters only
 -m, --perms          output info about permissions
 -l, --list           use list format ouput
 -n, --noheadings     don't print headings
 -o, --output <list>  output columns
 -r, --raw            use raw format output
 -t, --topology       output info about topology

Available columns:
       NAME  device name
      KNAME  internel kernel device name
    MAJ:MIN  major:minor device number
     FSTYPE  filesystem type
 MOUNTPOINT  where the device is mounted
      LABEL  filesystem LABEL
       UUID  filesystem UUID
         RO  read-only device
         RM  removable device
      MODEL  device identifier
       SIZE  size of the device
      OWNER  user name
      GROUP  group name
       MODE  device node permissions
  ALIGNMENT  alignment offset
     MIN-IO  minimum I/O size
     OPT-IO  optimal I/O size
    PHY-SEC  physical sector size
    LOG-SEC  logical sector size
       ROTA  rotational device
      SCHED  I/O scheduler name

SLES10 seems to have no lsblk at all.

jsmeix commented at 2021-06-08 12:18:

@OliverO2
you wrote

ReaR takes away control from the user ...
I think this should change ... as it does not conform to ReaR's principle.

Huh?

Since I know ReaR it does take away control from the user at so many places
so basically it is ReaR's normal behaviour to take away control from the user
and do things automatically and silently.
I tried to improve things according to my "final power to the user" principle
at several places where I was most hit by ReaR's silent automatisms
but overall ReaR still takes away control from the user at many places.

jsmeix commented at 2021-06-08 12:23:

@OliverO2
thank you so much for that enhancement to avoid severe loss of data!

I try to have a closer look as time permits but possibly it gets delayed until next week.

@gdha @rmetrich @pcahyna and all @rear/contributors
I would much appreciate it if you find some time and have a look here.
Perhaps you could spot this or that issue by plain looking at the code?

OliverO2 commented at 2021-06-08 12:45:

@jsmeix You wrote:

but overall ReaR still takes away control from the user at many places.

You think it's a good idea here?

gdha commented at 2021-06-11 07:28:

@OliverO2 @jsmeix A side note about control to the end-user: most end-users maybe read the man page, and probably don't look at the default.conf file. Do you think end-users look at the log files? The answer is probably most don't.
It is normal that ReaR with all its plug-ins becomes a bit complex to maintain and do good for everybody. Perhaps we should perform a grand cleanup and remove unused ones?
My point of view is - only a recover exercise is the most useful one and like most end-users don't look at the log files ;-) anymore as I know it just works for my use-cases.
PS: we have a log-file scraper to find errors and which reports it (for the many systems we maintain).

gdha commented at 2021-08-20 14:38:

@jsmeix This PR is still worthwhile to merge into our master code, no?

jsmeix commented at 2021-08-31 12:20:

My https://github.com/rear/rear/pull/2626#issuecomment-856719420
was very overoptimistic :-(

I will have a closer look as time permits
but I did not find sufficient time for a closer look
and because I am back from vacation right now
I probably don't find sufficient time for a closer look soon...

jsmeix commented at 2021-10-22 07:35:

We should not release ReaR 2.7 without having this included.

jsmeix commented at 2021-10-22 07:39:

I think (more precisely "I hope") after end of October (i.e. in November)
I will have time to have a closer look here - in particular to test it.

jsmeix commented at 2021-10-25 13:33:

I tested it on two VMs with those GPT disks

original system            replacement system
sda 8 GiB system disk      sda the 8 GiB ReaR "USB" disk from the original system
sdb 8 GiB ReaR "USB" disk  sdb 9 GiB replacement system disk

I use this etc/rear/local.conf

FIRMWARE_FILES=( 'no' )
MODULES=( 'loaded_modules' )
PROGRESS_MODE="plain"
PROGRESS_WAIT_SECONDS="3"
SSH_ROOT_PASSWORD="rear"
OUTPUT=USB
USB_DEVICE_FILESYSTEM_PERCENTAGE=90
USB_DEVICE_FILESYSTEM_LABEL="MY-DATA"
USB_BOOT_PART_SIZE=1024
USB_BOOTLOADER="grub"
USB_DEVICE_BOOT_LABEL="MY-BOOT"
OUTPUT_URL=usb:///dev/disk/by-label/MY-BOOT
USB_DEVICE_PARTED_LABEL=gpt
BACKUP=NETFS
BACKUP_URL=usb:///dev/disk/by-label/MY-DATA

Making ReaR "USB" disk:

# usr/sbin/rear -D format /dev/sdb
...
Repartitioning /dev/sdb
Creating partition table of type gpt on /dev/sdb
Creating BIOS boot partition /dev/sdb1
Setting 'bios_grub' flag on BIOS boot partition /dev/sdb1
Creating boot partition /dev/sdb2 with size 1024 MiB aligned at 8 MiB
Setting 'legacy_boot' flag on boot partition /dev/sdb2
Creating ReaR data partition /dev/sdb3 up to 90% of /dev/sdb
Creating ext2 filesystem with label 'MY-BOOT' on boot partition /dev/sdb2
Creating ext3 filesystem with label 'MY-DATA' on ReaR data partition /dev/sdb3
Adjusting filesystem parameters on ReaR data partition /dev/sdb3
Exiting rear format (PID 1554) and its descendant processes ...

The disks on the original system

# lsblk -bipo NAME,TRAN,TYPE,FSTYPE,SIZE,LABEL,MOUNTPOINT
NAME        TRAN TYPE FSTYPE       SIZE LABEL   MOUNTPOINT
/dev/sda    ata  disk        8589934592         
|-/dev/sda1      part           8388608         
`-/dev/sda2      part ext4   8580480512         /
/dev/sdb    ata  disk        8589934592         
|-/dev/sdb1      part           8364032         
|-/dev/sdb2      part ext2   1073741824 MY-BOOT 
`-/dev/sdb3      part ext3   6649020416 MY-DATA 
/dev/sr0    ata  rom         1073741312

"rear -D mkbackup"

# usr/sbin/rear -D mkbackup
...
Using build area: /var/tmp/rear.2ixQyBC3meCS35r
...

# grep WRITE_PROTECTED_ /var/tmp/rear.2ixQyBC3meCS35r/rootfs/etc/rear/rescue.conf
WRITE_PROTECTED_PARTITION_TABLE_UUIDS=( efee5e95-40a0-4a8e-bbe1-1cad7c0d255d )
WRITE_PROTECTED_FILE_SYSTEM_LABEL_PATTERNS=( )

# egrep -i 'write_protect|PTUUID' var/log/rear/rear-localhost.log
2021-10-25 14:17:43.434571282 Including prep/USB/default/480_initialize_write_protect_settings.sh
+ source /root/rear.github.OliverO2.stop-overwriting-backups/usr/share/rear/prep/USB/default/480_initialize_write_protect_settings.sh
++ WRITE_PROTECTED_PARTITION_TABLE_UUIDS+=($(lsblk --output PTUUID --noheadings --nodeps "$USB_DEVICE"))
+++ lsblk --output PTUUID --noheadings --nodeps /dev/disk/by-label/MY-DATA
2021-10-25 14:17:43.455119727 Including prep/default/490_store_write_protect_settings.sh
+ source /root/rear.github.OliverO2.stop-overwriting-backups/usr/share/rear/prep/default/490_store_write_protect_settings.sh
++ echo '# The following lines were added by 490_store_write_protect_settings.sh'
++ echo 'WRITE_PROTECTED_PARTITION_TABLE_UUIDS=( efee5e95-40a0-4a8e-bbe1-1cad7c0d255d )'
++ echo -n 'WRITE_PROTECTED_FILE_SYSTEM_LABEL_PATTERNS=('

# lsblk --output NAME,LABEL,UUID,PTUUID,PARTUUID,PARTLABEL /dev/sdb
NAME   LABEL   UUID                                 PTUUID                               PARTUUID                             PARTLABEL
sdb                                                 efee5e95-40a0-4a8e-bbe1-1cad7c0d255d                                      
├─sdb1                                              efee5e95-40a0-4a8e-bbe1-1cad7c0d255d aa3d3f8b-3adb-4ef8-a4c0-0e44cdadfff6 primary
├─sdb2 MY-BOOT ff15e0f1-b4bb-4a02-bb10-1502ec2fe0be efee5e95-40a0-4a8e-bbe1-1cad7c0d255d 281246c5-d29f-4ed2-8cfb-c7404eca010d primary
└─sdb3 MY-DATA 8bd9f9e8-9043-4a80-84e1-64d417244529 efee5e95-40a0-4a8e-bbe1-1cad7c0d255d e38c933c-d18f-4821-92c9-c99673738660 primary

"rear -D recover"
All user dialogs were accepted with the default (i.e. just proceed):

# rear -D recover
...
Comparing disks
Device sda is designated as write-protected (needs manual configuration)
Switching to manual disk layout configuration
Cannot use /dev/sda (same name and same size) for recreating /dev/sda (/dev/sda is write-protected)
Cannot use /dev/sda (same size) for recreating /dev/sda (/dev/sda is write-protected)
Could not automap /dev/sda (no disk with same size 8589934592 found)
Original disk /dev/sda does not exist (with same size) in the target system
Using /dev/sdb (the only appropriate) for recreating /dev/sda
Current disk mapping table (source => target):
  /dev/sda => /dev/sdb
...
Start system layout restoration.
Disk '/dev/sdb': creating 'gpt' partition table
Disk '/dev/sdb': creating partition number 1 with name ''sdb1''
Disk '/dev/sdb': creating partition number 2 with name ''sdb2''
Creating filesystem of type ext4 with mount point / on /dev/sdb2.
Mounting filesystem /
Disk layout created.
...
Backup restore program 'tar' started in subshell (PID=3483)
Restored 155 MiB [avg. 53170 KiB/sec] 
...
Restored 2482 MiB [avg. 28249 KiB/sec] 
OK
...
Running mkinitrd...
Recreated initrd (/sbin/mkinitrd).
Installing GRUB2 boot loader...
Determining where to install GRUB2 (no GRUB2_INSTALL_DEVICES specified)
Found possible boot disk /dev/sdb - installing GRUB2 there

After "rear recover" the disks in the still running Rear recovery system:

# lsblk --output NAME,LABEL,UUID,PTUUID,PARTUUID,PARTLABEL /dev/sda
NAME   LABEL   UUID                                 PTUUID                               PARTUUID                             PARTLABEL
sda                                                 efee5e95-40a0-4a8e-bbe1-1cad7c0d255d                                      
|-sda1                                              efee5e95-40a0-4a8e-bbe1-1cad7c0d255d aa3d3f8b-3adb-4ef8-a4c0-0e44cdadfff6 primary
|-sda2 MY-BOOT ff15e0f1-b4bb-4a02-bb10-1502ec2fe0be efee5e95-40a0-4a8e-bbe1-1cad7c0d255d 281246c5-d29f-4ed2-8cfb-c7404eca010d primary
`-sda3 MY-DATA 8bd9f9e8-9043-4a80-84e1-64d417244529 efee5e95-40a0-4a8e-bbe1-1cad7c0d255d e38c933c-d18f-4821-92c9-c99673738660 primary

# lsblk --output NAME,LABEL,UUID,PTUUID,PARTUUID,PARTLABEL /dev/sdb
NAME   LABEL UUID                                 PTUUID                               PARTUUID                             PARTLABEL
sdb                                               258d559c-1206-43f9-871f-c9e2176c2fa6                                      
|-sdb1                                            258d559c-1206-43f9-871f-c9e2176c2fa6 5c9604e7-f0c6-43f2-bee0-96810abb6b79 sdb1
`-sdb2       83a3f413-6e7d-4591-bd60-c87803613166 258d559c-1206-43f9-871f-c9e2176c2fa6 01dd5ee2-1f33-4f04-8e15-1cefead42437 sdb2

The recreated system boots well from sdb and seems to work OK.

jsmeix commented at 2021-10-25 13:36:

@rear/contributors
I would like to merge it tomorrow afternoon as is
unless there are objections.

jsmeix commented at 2021-10-25 13:54:

I am thinking about to enhance it after it was merged so that
the 480_initialize_write_protect_settings.sh scripts also initialize
WRITE_PROTECTED_FILE_SYSTEM_LABEL_PATTERNS.

Additionally I am thinking about to rename the specific
WRITE_PROTECTED_PARTITION_TABLE_UUIDS
into a generic WRITE_PROTECTED_UUIDS
and let in particular USB/default/480_initialize_write_protect_settings.sh
add all UUIDs that exist on $USB_DEVICE to WRITE_PROTECTED_UUIDS

I think it cannot cause harm to remember too many UUIDs
from a disk that must be sacrosanct during "rear recover"
because UUIDs are sufficiently universally/globally unique
so that collisions are so unlikely that they can be ignored.
And even if a collision happens the user can edit his
WRITE_PROTECTED_UUIDS in etc/rear/rescue.conf

The reason behind why I like to not only rely on partition table UUIDs is that
https://unix.stackexchange.com/questions/375548/what-is-uuid-partuuid-and-ptuuid
reads (excerpt)

UUID is a filesystem-level UUID, which is retrieved
from the filesystem metadata inside the partition.
It can only be read if the filesystem type is known and readable.

PARTUUID is a partition-table-level UUID for the partition,
a standard feature for all partitions on GPT-partitioned disks.
Since it is retrieved from the partition table, it is accessible
without making any assumptions at all about the actual
contents of the partition. If the partition is encrypted
using some unknown encryption method, this might be
the only accessible unique identifier for that particular partition.

PTUUID is the UUID of the partition table itself,
a unique identifier for the entire disk assigned
at the time the disk was partitioned. It is the equivalent
of the disk signature on MBR-partitioned disks but with more bits
and a standardized procedure for its generation.

On MBR-partitioned disks, there are no UUIDs in the partition table.
The 32-bit disk signature is used in place of a PTUUID,
and PARTUUIDs are created by adding a dash and a
two-digit partition number to the end of the disk signature.

So PTUUID may only be available on GPT partitioned disks and
I don't know if on all Linux systems the 32-bit MBR disk signature
is used in place of a PTUUID.

jsmeix commented at 2021-11-10 12:46:

With https://github.com/rear/rear/pull/2703 merged

In etc/rear/local.conf WRITE_PROTECTED_ID_TYPES="PTUUID"
results basically the same behaviour as before this pull request
where only PTUUID was used as ID,
cf. https://github.com/rear/rear/pull/2703#issuecomment-965088175

[Export of Github issue for rear/rear.]