#1854 Issue closed: "Pseudo Migration Mode" needed during "rear recover" with a ReaR recovery disk¶
Labels: enhancement, documentation, discuss / RFC,
no-issue-activity
jsmeix opened issue at 2018-07-06 10:20:¶
I think we have a general problem in current ReaR master code
during "rear recover" when the ReaR recovery system
was booted from a normal disk device like /dev/sda.
See
https://github.com/rear/rear/issues/1852#issuecomment-402727869
and the section about
... a possible problem with the disk migration ...
in
https://github.com/rear/rear/issues/1852#issuecomment-402512410
I think the following suituation causes a general problem:
1.)
On the original system /dev/sda is the first system disk
(i.e. the disk that usually contains the bootloader and the root
filesystem).
For "rear mkrescue/mkbackup" an additional normal disk device is used
that is
a removable disk (e.g. a USB disk) that appears in the original system
as /dev/sdb
where to the ReaR recovery system (and perhaps also the backup) is
written
so that in particular the disklayout.conf file contains entries for
the system disk /dev/sda.
2.)
On the replacement hardware the ReaR recovery disk is connected
and the replacement hardware is booted from that disk so that
on the replacement hardware the recovery disk could be /dev/sda
and the first target system disk could be /dev/sdb so that
during "rear recover" /dev/sda and /dev/sdb are interchanged
compared to what there was on the original system.
3.)
Because of the interchanged /dev/sda and /dev/sdb "rear recover"
goes into MIGRATION_MODE where the right disk layout mapping
table (source => target) is /dev/sda => /dev/sdb and that
disk mapping must be applied to the disklayout.conf file
to recreate the system disk layout on the right target disk /dev/sdb
(otherwise the ReaR recovery disk content would be destroyed).
4.)
After "rear recover" recreated the system disk layout on /dev/sdb
the backup needs to be restored into the filesystems on /dev/sdb
and the bootloader needs to be installed on /dev/sdb.
5.)
After the backup was restored but before the bootloader gets installed
"rear recover" runs in its finalize stage several scripts that
modify
certain restored files where the above disk layout mapping gets
applied
so that in certain restored files /dev/sda gets replaced by
/dev/sdb,
cf. the rear -D recover terminal output at
https://github.com/rear/rear/issues/1852#issuecomment-402512410
that contains in particular
Applying disk layout mappings in /var/lib/rear/layout/disk_mappings to certain restored files... The original restored files get saved in var/lib/rear/saved_original_files/ (in /mnt/local) Applied disk layout mappings to restored 'boot/grub2/grub.cfg' (in /mnt/local) Applied disk layout mappings to restored 'boot/grub2/device.map' (in /mnt/local) Applied disk layout mappings to restored 'etc/sysconfig/bootloader' (in /mnt/local) Applied disk layout mappings to restored 'etc/fstab' (in /mnt/local) Applied disk layout mappings to restored 'etc/mtools.conf' (in /mnt/local) Applied disk layout mappings to restored 'etc/smartd.conf' (in /mnt/local) Applied disk layout mappings to restored 'etc/sysconfig/smartmontools' (in /mnt/local) Patching /etc/default/grub_installdevice: Replacing [/dev/disk/by-id/ata-QEMU_HARDDISK_QM00001] by [/dev/disk/by-id/ata-QEMU_HARDDISK_QM00003] Updating udev configuration (70-persistent-net.rules) Running mkinitrd... Recreated initrd (/sbin/mkinitrd). Installing GRUB2 boot loader...
Modifying certain restored files could be needed because
the bootloader needs to be installed within the target system
environment, cf.
https://github.com/rear/rear/issues/1828#issuecomment-398717889
chroot /mnt/local grub2-mkconfig -o /boot/grub2/grub.cfg chroot /mnt/local grub2-install /dev/sdb
so that in particular bootloader related config files may need to have
the right values that match the target system.
6.)
After "rear recover" finished and before the recreated system is
rebooted
the no longer needed ReaR recovery disk gets disconnected
so that then on the recreated system /dev/sda is again the first system
disk
(i.e. the same disk device node as it was on the original system).
7.)
But now the recreated system boots with the above modified config
files
where in particular /dev/sda was replaced by /dev/sdb which is now
wrong.
Conclusion:
I think a "Pseudo Migration Mode" is needed during "rear recover"
when the ReaR recovery system was booted from a normal disk.
I think because during "rear recover" some config files in the recreated
system
may need to have /dev/sda replaced by /dev/sdb (e.g. for bootloader
installation)
one cannot "just skip" that modification step.
Accordingly my immediate idea how to solve this is to add an optional
very final re-replacement step that re-replaces /dev/sdb by
/dev/sda
in those config files in the recreated system to get things right
again
to boot the recreated system without the ReaR recovery disk.
I think it is impossible to automate such a re-replacement step
so that the user must explicitly specify when he wants that
re-replacement
because only the user knows if the recreated system will be booted
with or without the ReaR recovery disk.
I think when the ReaR recovery disk gets a 'late' device node like
/dev/sdb
so that the target system disk is still /dev/sda as it was on the
original system
everything would "just work" because then the right disk layout
mapping
table (source => target) is /dev/sda => /dev/sda (i.e. the
identical mapping)
which does nothing and will be no longer applied to disklayout.conf or
other files
when
https://github.com/rear/rear/issues/1847
gets solved
by merging in
https://github.com/rear/rear/pull/1848
the commit
https://github.com/rear/rear/commit/f23c601857a9b8e6cb08892b1e3b67257ad5314a
OliverO2 commented at 2018-07-06 11:00:¶
In my view this is a good overview of problems with temporary hardware configurations used during recovery, which might lead to an unbootable system after recovery.
Is it correct to say that ReaR's mispatching of system files only applies to systems which still use bus-based naming for devices in their configuration while systems using Persistent block device naming are not affected as ReaR will preserve UUIDs and labels?
The above explanation uses the USB device as an example. As far as I am
aware, USB storage devices will be initialized after internal (SCSI)
hard disks and always receive "later" device nodes. So a USB storage
device will never displace the original /dev/sda. Or do we have
evidence to the contrary?
jsmeix commented at 2018-07-06 11:59:¶
@OliverO2
I never had a closer look at ReaR's modifications of restored files
until
https://github.com/rear/rear/pull/1843
where I got a bit scared because ReaR did that silently
and without saving the original files, cf.
https://github.com/rear/rear/pull/1843#issuecomment-401042614
For me things had "just worked" because I never had the ReaR recovery
system
on a normal disk /dev/sda - if at all I had it as /dev/sdb on the
replacement hardware
where "hardware" actually was a virtual KVM/QEMU machine where I
attached
such a ReaR recovery system disk as second IDE disk that became
/dev/sdb
so that /dev/sda was always the system disk both on the original
system
and on the replacement hardware.
Because I got no SUSE customer request about this issue here
I assume in business environments it does usually not happen
that the ReaR recovery system is on a normal disk /dev/sda
or perhaps by accident or by some luck things "somehow work"
even in this case or - scaring! - users who have the ReaR recovery
system on a normal disk do not verify that "rear recover" actually
works for them on their replacement hardware, cf.
"No disaster recovery without testing and continuous validation" in
https://en.opensuse.org/SDB:Disaster_Recovery
Now since I know about this issue here I can test how current ReaR
behaves
when I attach such a ReaR recovery system disk as first IDE disk
/dev/sda
on the replacement hardware.
I fear the current way how ReaR modifies restored files may need
to be completely overhauled because from my point of view
automated modifying restored files is a very delicate issue
because in general the user's data has to be sacrosanct.
I used the term USB disk only as a template for any removable disk.
To make things more clear I will replace USB disk by removable disk
in
https://github.com/rear/rear/issues/1854#issue-338886055
jsmeix commented at 2018-07-06 12:08:¶
@OliverO2
my losssy mind forgot that we do have evidence that a USB device
can become /dev/sda on the replacement hardware:
https://github.com/rear/rear/issues/1271
I even mentioned that issue in my description
about MIGRATION_MODE in default.conf.
jsmeix commented at 2018-07-06 12:43:¶
FYI:
Currently only a GRUB2 bootloader installation gets migrated
according to the disk layout mapping, cf.
https://github.com/rear/rear/issues/1437#issuecomment-403022440
OliverO2 commented at 2018-07-06 14:11:¶
@jsmeix Maybe ReaR could detect when the recovery system was booted from
a device (primarily /dev/sda), which is among the original system's
regular devices. In this case, ReaR could assume that this is only a
temporary situation during recovery and react as follows:
- Automatically create a temporary mapping which excludes the recovery
boot device (e.g., mapping
/dev/sdbto/dev/sda,/dev/sdcto/dev/sdb, ...) even before deciding further about migration mode. - Do not apply that temporary mapping when patching files on the recovered system.
In this case ReaR would just behave as if its own recovery system's boot device just wasn't there.
Note that /dev/sda may be the most common case, but it is also
conceivable that the recovery boot device could insert itself somewhere
else (e.g. where an original system has sda, sdb, sdc, ReaR might
boot from sdb, displacing sdb -> sdc and sdc -> sdd).
jsmeix commented at 2018-07-09 10:01:¶
@OliverO2
regarding your above
Do not apply that temporary mapping when patching files on the recovered system
see item 5.) in my initial comment:
Modifying certain restored files could be needed because the bootloader needs to be installed within the target system environment, cf. https://github.com/rear/rear/issues/1828#issuecomment-398717889 chroot /mnt/local grub2-mkconfig -o /boot/grub2/grub.cfg chroot /mnt/local grub2-install /dev/sdb so that in particular bootloader related config files may need to have the right values that match the target system.
Also the initrd/initramfs is built within the target system environment via
chroot $TARGET_FS_ROOT ...
See the usr/share/rear/finalize/ scripts for chroot commands.
Currently I don't know which of them may somehow need to have
right disk layout values in some config files in the target system
while they run within the target system environment.
For example I don't know how exactly
chroot /mnt/local grub2-mkconfig -o /boot/grub2/grub.cfg chroot /mnt/local grub2-install /dev/sdb
behaves.
The first command should create /boot/grub2/grub.cfg anew
so that the before running
Applied disk layout mappings to restored 'boot/grub2/grub.cfg' (in /mnt/local)
should no longer matter here and
the second command should install GRUB2 explicitly on /dev/sdb
but currently I don't know if that installed GRUB2 still boots
when /dev/sdb has become /dev/sda.
I will try it out - but not this week because this week is Hack Week,
see
https://hackweek.suse.com/
and
https://hackweek.suse.com/17/projects
and I will work on
https://hackweek.suse.com/17/projects/get-a-basic-understanding-about-md-software-raid-setup
OliverO2 commented at 2018-07-09 12:29:¶
@jsmeix When guessing based on my experiences with grub, I would assume that
- grub does not depend on configuration files already present on the target system (the bootloader may be installed at a point where there are no files at all on the target system, just partitions with file systems),
- grub/Legacy will probably accept switching device nodes between installation and boot without any trouble as the target's boot device order will be configured in the BIOS,
- with grub/EFI, as long as there is only one disk with an EFI partition around when the target system boots, it should also work without trouble. When there is more than one such disk, I am not sure how exactly grub will set up the EFI boot manager and whether the correct boot device will be chosen (cf. The EFI System Partition and the Default Boot Behavior - The Uncoöperative Organization).
Good luck when trying it out and now: Enjoy your hack week!
jsmeix commented at 2019-02-22 15:49:¶
For the fun of it:
Such generic issues do not only appear with ReaR.
It also happens with other installers when their installation system
was booted from a normal disk device (e.g. a USB stick) that is
/dev/sda,
e.g. with AutoYaST, see
https://lists.opensuse.org/opensuse-autoinstall/2019-02/msg00003.html
github-actions commented at 2020-06-30 01:33:¶
Stale issue message
[Export of Github issue for rear/rear.]