#2889 Issue closed: Test restore failed - recovery environment trashed

Labels: support / question, fixed / solved / done

ZENAdmin-Ops opened issue at 2022-11-13 04:21:

Relax-and-Recover (ReaR) Issue Template

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

  • ReaR version ("/usr/sbin/rear -V"):
    2.6

  • OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):
    Ubuntu Jammy

  • ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

OUTPUT=USB
BACKUP=NETFS
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp' '/var/crash')
BACKUP_URL=usb:///dev/disk/by-label/REAR-002
USB_UEFI_PART_SIZE="1000"
USB_RETAIN_BACKUP_NR=10

  • Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR):
    Hyper-V

  • System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):

  • Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):

  • Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
    Local USB

  • Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"):

  • Description of the issue (ideally so that others can reproduce it):

I think the issue was that the system being restored had
a 40GB ext4 partition and 86.50GB unallocated.

While the destination had an empty volume of around 127GB,
so more than enough for the EFI partition and the 40GB ext4 partition.
But the unallocated space would have been less.

In any case, when I did rear recover Rear reported
that space on the destination volume didn't match
(I can't recall the exact message).
But I accepted the default option of 1. Which was to use /dev/sda

The restore subsequently failed.
But the real issue for me was that the rear filesystem
on the USB drive was trashed.
I was unable to reboot the USB drive to attempt another restore.

And when I attached the USB drive to a working Ubuntu system,
that system was unable to start while the USB drive was attached.

I had to reformat the USB drive to make it usable,
which means that I have lost the rear recover logs
and if I had important backups on this drive,
I would have lost them as well.

I have subsequently re-setup the USB drive,
and re-sized the volume on the destination VM,
so that there was sufficient space for the unallocated space
to be recreated as per the source volume.
I then performed another test restore which succeeded.

So, I have a couple of questions,
based on what I have described / observed

  1. By default, does Rear attempt to clone the source volume configuration? By "clone", I mean does their need to be sufficient free space so that the unallocated space on the destination system can match the source?
  2. If that is the case, I'd like to understand how to either
  3. Only backup / restore partitions that are in use (e.g. EFI System, and the system itself (e.g. /dev/sda2) )

  4. Or how I can exit to the rear recovery shell and manually create the needed partitions on the destination system

  5. Workaround, if any:

  6. Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):

A screenshot of the source system partitions is attached

On the initial restore (that failed) the destination system
had unallocated space around 125GB.
So, plenty of space in terms of used partitions.
But unallocated would have been less than on the source system.

On the second restore, (that worked) I had increased the size
of the unallocated space to around 129GB.

Source_system_partition_layout

jsmeix commented at 2022-11-14 13:40:

@ZENAdmin-Ops

"rear recover" completely recreates a system from scratch, see
"Disaster recovery means installation (reinstalling from scratch)" at
https://en.opensuse.org/SDB:Disaster_Recovery

ReaR works on whole disks.
"rear recover" either completely recreates a disk
by overwriting all what there already is on a disk
or it leaves a disk unchanged.

When the ReaR recovery system and the backup is on a disk
and during "rear recover" you get asked what disk(s)
should be used to recreate the system
and accidentally you reply with your ReaR recovery system USB disk
then "rear recover" will completely destroy all what there already is
on that disk (i.e. ReaR can destroy its own disk if you select it), see
https://github.com/rear/rear/issues/1271

In particular when you boot the ReaR recovery system from a USB disk
that USB disk could get the kernel device node name /dev/sda and
the actual system disk that should be used to recreate the system
then usually gets the subsequent kernel device node name /dev/sdb
(kernel device node names are not really predictable).

To see what kernel device node each disk has
after you booted the ReaR recovery system
and when you logged in there as 'root' call lsblk or
e.g. lsblk -ipo NAME,KNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE
and remember what it shows before you run "rear recover".

Alternatively during "rear recover" in the user input dialogs
you can Use Relax-and-Recover shell and return back to here
(i.e. return back to the particular user input dialog)
to run lsblk to see what kernel device node each disk has.

Current ReaR 2.7 has better protection against
possibly overwriting its own (USB) disk and backup
via new WRITE_PROTECTED_... config variables
see 'WRITE_PROTECTED_...' in default.conf at
https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/conf/default.conf#L566

ZENAdmin-Ops commented at 2022-11-14 23:28:

Thanks for this info.

So, in the example above, where the disk that I'm trying to restore has a 40GB sda partition and 86.50GB unallocated

If the destination volume has 80GB free, you're saying that Rear should be able to handle that automatically provided that I ensure that I'm restoring to the correct device?

Your suspicion is that the issue in my case was that I restored over the USB disk, rather than the issue being due to restoring the backup image to a smaller disk.

jsmeix commented at 2022-11-15 07:59:

Because I wrote above ReaR works on whole disks
it is irrelevant how much space on a disk
is allocated or unallocated or whatever,
see what I wrote above:

"rear recover" either completely recreates a disk
by overwriting all what there already is on a disk
or it leaves a disk unchanged

So when you select during "rear recover"
to recreate an original system on a disk
with kernel device node '/dev/sda'
then "rear recover" will completely use that disk
by overwriting all what there already is on that disk and
"rear recover" will recreate things (partitions, filesystems, ...)
on that disk as they have been before on the original system
and "rear recover" will restore the files from the backup
into that recreated partitions and filesystems and
finally "rear recover" will install a bootloader on that disk.

I.e. "rear recover" does a whole system installation from scratch.

ZENAdmin-Ops commented at 2022-11-15 08:37:

So, if the "destination" disk is smaller than the "source" disk, taking into account un-allocated partitions; is that a supported scenario?

Or, must the destination disk always be at least the same size or larger than the source?

jsmeix commented at 2022-11-15 09:39:

When the size of the whole destination disk is smaller
than the size of the whole source disk, see the section about
Resizing partitions in MIGRATION_MODE during "rear recover"
in usr/share/rear/conf/default.conf
e.g. online for your ReaR 2.6 version at
https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/conf/default.conf#L475

When your original system disk layout is as in your
https://github.com/rear/rear/issues/2888#issue-1446676874
(excerpts)

NAME        TYPE FSTYPE SIZE MOUNTPOINT
...
/dev/sdc    disk        127G 
|-/dev/sdc1 part vfat   512M /boot/efi
`-/dev/sdc2 part ext4    40G /

the 40G root '/' partition and the 0.5G EFI partition
will fit on a disk with about 41G whole disk size
without a need to shrink the last partition
which is the 40G root '/' partition.

Then those excerpts of the
Resizing partitions in MIGRATION_MODE during "rear recover"
in default.conf apply

When the new disk is not much bigger
(less than AUTOINCREASE_DISK_SIZE_THRESHOLD_PERCENTAGE percent),
no partition gets increased (which leaves the bigger disk space
at the end of the disk unused).
When the new disk is substantially bigger
(at least AUTOINCREASE_DISK_SIZE_THRESHOLD_PERCENTAGE percent),
only the last (active) partition gets increased
but all other partitions are not changed.

The default AUTOINCREASE_DISK_SIZE_THRESHOLD_PERCENTAGE=10
means the last partition will not be increased when the
destination disk size is less than the source disk size + 10%
so when your source disk size is 127G the last partition
will not be increased when the destination disk size
is less than 139.7G (by default).

When you know your destination disk size is big enough
to recreate all the partitions on your source disk as is
without any change in any partition size
the simplest way is to specify

AUTORESIZE_PARTITIONS='false'

to enforce that no partition gets resized
during "rear recover".

Alternatively you could specify a huge value for
AUTOINCREASE_DISK_SIZE_THRESHOLD_PERCENTAGE
to avoid that the last partition will be increased
even if the destination disk size is much bigger
than the source disk is like

AUTOINCREASE_DISK_SIZE_THRESHOLD_PERCENTAGE=10000

so the last partition gets not resized
even if the destination disk size would be 100 times bigger
than the source disk size.

ZENAdmin-Ops commented at 2022-11-15 10:29:

@jsmeix

A heap of really useful info

Thanks!

jsmeix commented at 2022-11-15 10:37:

@ZENAdmin-Ops
you are welcome!

Please provide feedback how far things work for your use case.

We at ReaR upstream cannot test all possible use cases.
For example I never tested myself how things behave
when only a small part of the disk is actually used
(as in your case 40.5G used on a 127G disk).

E.g. for what cases I had tested myself
regarding AUTORESIZE_PARTITIONS see
https://github.com/rear/rear/pull/1733

ZENAdmin-Ops commented at 2022-11-15 10:43:

I have since performed a successful test restore,
based on your advice and using:
AUTORESIZE_PARTITIONS='false'

Regards,
Vaughan

[Export of Github issue for rear/rear.]