#2785 Issue closed: No error is printed if UUID are not updated during recover (no sed output)

Labels: enhancement, fixed / solved / done

bwelterl opened issue at 2022-04-07 11:43:

Relax-and-Recover (ReaR) Issue Template

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

  • ReaR version ("/usr/sbin/rear -V"):
    rear 2.6

  • OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):
    Red Hat Enterprise Linux release 8.5 (Ootpa)

  • ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

BACKUP_PROG=rsync
BACKUP_PROG_EXCLUDE=('/tmp/*' '/dev/shm/*' '/mnt2/*' '/var/*')
BACKUP=RSYNC
BACKUP_URL=rsync://192.168.122.15/rear/rsyncbackup/

  • Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR):
    KVM

  • System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
    x86

  • Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
    UEFI

  • Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
    local

  • Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"):

NAME                      KNAME     PKNAME    TRAN   TYPE FSTYPE      LABEL  SIZE MOUNTPOINT
/dev/sr0                  /dev/sr0            sata   rom                    1024M 
/dev/vda                  /dev/vda                   disk                     20G 
|-/dev/vda1               /dev/vda1 /dev/vda         part vfat               600M /boot/efi
|-/dev/vda2               /dev/vda2 /dev/vda         part xfs                  1G /boot
`-/dev/vda3               /dev/vda3 /dev/vda         part LVM2_member       18.4G 
  |-/dev/mapper/rhel-root /dev/dm-0 /dev/vda3        lvm  xfs               16.4G /
  `-/dev/mapper/rhel-swap /dev/dm-1 /dev/vda3        lvm  swap                 2G [SWAP]
  • Description of the issue (ideally so that others can reproduce it):

In certain circumstances (need to clarify exactly the workflow,
certainly a rescue not run after a UUID changed and backuponly on the source system),
the UUID are not updated during the recover, because sed can't find old UUID.
The issue is that sed does not print an error or a different return code
if the file is not patched, thus we don't know if the sed script had an effect.

    sed  "$sed_script" "$restored_file" || LogPrintError "Migrating filesystem UUIDs in $restored_file to current UUIDs failed";

=> the error is printed only if the sed encounters an error, not if the patch is not applied

A PR will be submitted to discuss how this can be improved.

Thanks !

  • Workaround, if any:
    Manually check the UUID

jsmeix commented at 2022-04-07 11:55:

@bwelterl
thank you for your explanatory description - this is much appreciated!

pcahyna commented at 2022-04-07 13:01:

In certain circumstances (need to clarify exactly the workflow, certainly a rescue not run after a UUID changed and backuponly on the source system)

I believe that changing the system between mkrescue and mkbackuponly should not be supported, and this should be considered a user error. @jsmeix do we have such a statement anywhere in the docs? There is a checklayout command, does it report that the layout has changed before you run mkbackuponly? If so, this indicates that mkrescue needs to be rerun. If not, that would be a bug. (I wonder whether it would make sense to add a checklayout at the start of the mkbackuponly workflow, so that the likelihood of broken backups is reduced?)

jsmeix commented at 2022-04-07 13:14:

It is a user error when "rear mkbackup" was not run
after each change of the basic system in particular
after a change of the disk layout that results a change in disklayout.conf
which is what "rear checklayout" is meant to detect.

As far as I know (but I don't know our whole documentation well)
we do not document basics like when one should make a new backup
or things like that.

pcahyna commented at 2022-04-07 13:22:

If you don't run mkbackup after each change, you get an older state of the system when you recover. This might be undesired, but might be actually desired as well.
But if you split mkbackup into mkrescue and mkbackuponly and you change the system between them, the result is for sure inconsistent and would be good if ReaR detected this situation automatically.

jsmeix commented at 2022-04-07 13:27:

I need to investigate how FS_UUID_MAP is meant to work.

As far as I see on first glance it seem FS_UUID_MAP is written in
layout/prepare/GNU/Linux/131_include_filesystem_code.sh
only in case of btrfs and vfat
but a filessystem UUID is also set in case of ext* and xfs and reiserfs
which does not look right.

jsmeix commented at 2022-04-07 13:34:

@pcahyna
for most external backup methods "rear mkbackup" does not make any backup
so for most external backup methods only "rear mkrescue" is useful
which means for most external backup methods it is up to the user to ensure
that the files in his backup match sufficiently what there is in his recovery system.

On the other hand finalize scripts like
finalize/GNU/Linux/280_migrate_uuid_tags.sh
are there to migrate e.g. UUIDs in outdated restored files
to the current UUIDs of the recreated system.

pcahyna commented at 2022-04-07 13:39:

or most external backup methods it is up to the user to ensure
that the files in his backup match sufficiently what there is in his recovery system

Sure, I understand that, but at least in the internal backup case the check could be automated.

On the other hand finalize scripts like
finalize/GNU/Linux/280_migrate_uuid_tags.sh
are there to migrate e.g. UUIDs in outdated restored files
to the current UUIDs of the recreated system.

I believe this is for the case that recovery process changes a UUID and therefore the UUIDs in files are outdated and not for the case that the backup is old, or is it a misunderstanding?

bwelterl commented at 2022-04-07 13:43:

Thanks a lot for your comments !

I opened another issue (sorry...) to discuss a more general issue that rescue ISO is dependant of data in the backup:
https://github.com/rear/rear/issues/2787
And that crond job automatically creates a new ISO without checking if the required data is in the backup.
If we can avoid such dependency, the issue will not occur anymore.

Thanks !

pcahyna commented at 2022-04-07 13:45:

And that crond job automatically creates a new ISO without checking if the required data is in the backup.

Oops, I think I see the problem now... so it was first mkbackuponly and then mkrescue?

If we can avoid such dependency, the issue will not occur anymore.

I am afraid this won't be feasible in general...

jsmeix commented at 2022-04-07 13:57:

I think I understand now how the issue happens:

At some time "rear mkrescue" was run that stored
filesystem UUIDs of that time into disklayout.conf.

Then filesystem UUIDs were changed and a backup was made.

This means the filesystem UUIDs in disklayout.conf are outdated.

When then "rear recover" is run,
the old UUIDs in FS_UUID_MAP are the ones from disklayout.conf and
normally the filesystems get recreated with the UUPDs from disklayout.conf
(because nowadays mkfs.... tools support setting the filesystem UUID)
but the recreated UUIDs cannot be found in the restored config files
because the backup contained already config files with newer UUIDs.

When the filesystems get recreated with the UUPDs from disklayout.conf
old and new UUIDs are the same in FS_UUID_MAP (which is the usual case)
so finalize/GNU/Linux/280_migrate_uuid_tags.sh won't change anything.

Accordingly - from my current point of view - I think the sed part in
finalize/GNU/Linux/280_migrate_uuid_tags.sh
is not the right place to deal with this issue.

I will have to think more about it...

bwelterl commented at 2022-04-07 14:05:

@jsmeix Yes, but I think it's the opposite. The rescue is built everytime the layout changes (due to crond job), thus if you change the uuid, you will automatically have a new rescue ISO, which will be unusable ...
Rear is checking some files to update the UUID during the recovery, these files should be embedded in the rescue ISO.

Thanks !

pcahyna commented at 2022-04-07 14:20:

As far as I see on first glance it seem FS_UUID_MAP is written in
layout/prepare/GNU/Linux/131_include_filesystem_code.sh
only in case of btrfs and vfat

In this case the problem was in vfat - the EFI System Partition.

jsmeix commented at 2022-04-07 14:20:

All files that get included in the recovery system (what you call "rescue ISO")
are files from the system at the time when "rear mkrescue" is run
so in this case those files would contain the new UUIDs (as in disklayout.conf).

When during "rear recover" a backup with outdated UUIDs in files gets restored
one could compare the files that were included in the recovery system
with the restored files to find out what changed between backup and "mkrescue"
(regardless what was earlier backup or "mkrescue").

jsmeix commented at 2022-05-05 10:37:

With https://github.com/rear/rear/pull/2795 merged
this issue should be solved.

When restored basic system files do not match the recreated system
there should now be a message

Error: Restored files do not match the recreated system in /mnt/local

during "rear recover" after the backup was restored.

What basic system files are checked can be specified via
CHECK_CONFIG_FILES and the new FILES_TO_PATCH_PATTERNS
see their descriptions in conf/default.conf

@bwelterl
if possible for you please test our current ReaR GitHub master code,
see the section
"Testing current ReaR upstream GitHub master code" in
https://en.opensuse.org/SDB:Disaster_Recovery

[Export of Github issue for rear/rear.]