#1852 Issue closed: In RAWDISK recovery mode, auto disk mapping is not working

Labels: support / question, fixed / solved / done

GreenBlood opened issue at 2018-07-04 13:52:

full log.log

  • ReaR version:
    Relax-and-Recover 2.4-git.0.26e6eec.unknown / 2018-07-03
    (From PR #1850 )
  • OS version:
    CentOS 6.9
  • ReaR configuration files:
BACKUP=NETFS
OUTPUT=RAWDISK
BACKUP_TYPE=incremental
FULLBACKUPDAY="Sun"


SSH_ROOT_PASSWORD="XXXXXX"
USE_DHCLIENT=yes

BACKUP_URL=cifs://XXXXX/XXXXX
BACKUP_OPTIONS="cred=/etc/rear/.cifs_credentials"
  • System architecture:
    x86_64 (amd64)
  • Are you using BIOS or UEFI or another way to boot?
    BIOS
  • Brief description of the issue:
    When using RAWDISK recovery environment to restore a server, the disk layout configuration (mapping) does not work correctly.
    Because RAWDISK acts as a block device, it is recognized as (in my case) sda, so the root partition of the disk to be restored becomes sdb. When running rear recover, it should automagically find the appropriate disk based on its size. But here it does not work.
    As an information, source disk and the destination disk are litteraly the same, I merely boot the virtual machine on the recovery environnement. So size is the same.
    Logs for rear -D recover :
RESCUE centos6:~ # rear -D recover
Relax-and-Recover 2.4-git.0.26e6eec.unknown / 2018-07-03
Using log file: /var/log/rear/rear-centos6.log
Running workflow recover within the ReaR rescue/recovery system
For backup restore using  2018-07-04-1726-F.tar.gz
Calculating backup archive size
Backup archive size is 568M /tmp/rear.jl5vVgf9lHAyfBh/outputfs/centos6/2018-07-04-1726-F.tar.gz (compressed)
Comparing disks
Device sda has size 95420416 but 21474836480 is expected (needs manual configuration)
Switching to manual disk layout configuration
Using /dev/sdb (same size) for recreating /dev/sda
Current disk mapping table (source -> target):
    /dev/sda /dev/sdb
UserInput -I LAYOUT_MIGRATION_CONFIRM_MAPPINGS needed in /usr/share/rear/layout/prepare/default/300_map_disks.sh line 211
Confirm or edit the disk mapping
1) Confirm disk mapping and continue 'rear recover'
2) Edit disk mapping (/var/lib/rear/layout/disk_mappings)
3) Use Relax-and-Recover shell and return back to here
4) Abort 'rear recover'
(default '1' timeout 300 seconds)

(Was asked by @jsmeix to post this issue)

Regards,
Green

EDIT : Added debug log, 300_map_disks.sh begins at line 15345

jsmeix commented at 2018-07-04 14:27:

@GreenBlood
as a quick first reply about what you get on your terminal:

All looks perfectly fine to me and works exactly as it should
and as it also works for other output methods (like OUTPUT=ISO)
because the output method must not make a difference how
"rear recover" behaves regarding disk layout mapping.

It automatically detects that sda is not the expected one

Comparing disks
Device sda has size 95420416 but 21474836480 is expected (needs manual configuration)
Switching to manual disk layout configuration

and then it automatically finds that sdb has the expected size

Using /dev/sdb (same size) for recreating /dev/sda

so that it automatically creates the disk mapping proposal
which it shows to you

Current disk mapping table (source -> target):
    /dev/sda /dev/sdb

and asks for your confirmation that this is really
the right disk mapping in your particular case

Confirm or edit the disk mapping
1) Confirm disk mapping and continue 'rear recover'
2) Edit disk mapping (/var/lib/rear/layout/disk_mappings)
3) Use Relax-and-Recover shell and return back to here
4) Abort 'rear recover'
(default '1' timeout 300 seconds)

where you could use item 3) to verify what sda and sdb actually are
at that particular state to make a decision based on the actual facts.

@GreenBlood
can you describe in more detail what your expectation is and/or
how that disk mapping /dev/sda => /dev/sdb does not work for you?

FYI:
If you do not like the default timeout 300 seconds
or if you like to avoid manual input, see the section about

# Relax-and-Recover UserInput function default behaviour
...
# USER_INPUT_TIMEOUT
# USER_INPUT_INTERRUPT_TIMEOUT
# USER_INPUT_PROMPT
# USER_INPUT_MAX_CHARS
# USER_INPUT_user_input_ID

in usr/share/rear/conf/default.conf

GreenBlood commented at 2018-07-04 14:44:

@jsmeix Well let me briefly explain you my use case. I use rear to automate the recovery processes of linux servers in a cloud environnement. I used to generate a rescue ISO, boot it on a VM with all the disks expected, and using the unattended mode and some fiddling, restored the server automatically.

Now I recently changed my cloud environment and was not able anymore to boot on ISO, hence my use of RAWDISK. The issue I'm having is that because of the sda=>sdb mapping thing, the automated workflow is interrupted.

I have no issue with how rear builds the mapping, but I'd wish to know if it's possible to, by default, accept all changes, because the source sda and dest sdb are the same size, imho it would make sense to me to be able to just say to rear "do your thing"

EDIT : As I read the UserInput section in default.conf, I think I understand how I would do it. I'd have to set a default value for all input that might come up using USER_INPUT_user_input_ID ?

jsmeix commented at 2018-07-04 15:39:

@GreenBlood
I think in practice the easiest way to "blindly accept"
all UserInput dialogs with their defaults is to set

USER_INPUT_TIMEOUT=1

in your etc/rear/local.conf
(USER_INPUT_TIMEOUT=0 does not work)
for the details how it is implemented see the UserInput function
in usr/share/rear/lib/_input-output-functions.sh

In your case you cannot use MIGRATION_MODE='false'
because that would skip the automated disk migration
and "blindly" recreate on what is stored in your disklayout.conf
which means "blindly" recreate on /dev/sda which would
destroy your ReaR recovery disk.

For example how things look for me with USER_INPUT_TIMEOUT=1
on a SLES15 KVM/QEMU test system with two (virtual) disks
with identical size (only one is used for the system):

# export USER_INPUT_TIMEOUT=1

# rear -D recover
Relax-and-Recover 2.4 / Git
Using log file: /var/log/rear/rear-d228.log
Running workflow recover within the ReaR rescue/recovery system
Starting required daemons for NFS: RPC portmapper (portmap or rpcbind) and rpc.statd if available.
Started RPC portmapper 'rpcbind'.
RPC portmapper 'rpcbind' available.
RPC status rpc.statd available.
Using backup archive '/tmp/rear.Tt2PwYPVHGbJsLD/outputfs/d228/backup.tar.gz'
Will do driver migration (recreating initramfs/initrd)
Calculating backup archive size
Backup archive size is 1.1G     /tmp/rear.Tt2PwYPVHGbJsLD/outputfs/d228/backup.tar.gz (compressed)
Comparing disks
Ambiguous possible target disks need manual configuration (more than one with same size found)
Switching to manual disk layout configuration
Using /dev/sda (same name and same size) for recreating /dev/sda
Current disk mapping table (source -> target):
    /dev/sda /dev/sda
UserInput -I LAYOUT_MIGRATION_CONFIRM_MAPPINGS needed in /usr/share/rear/layout/prepare/default/300_map_disks.sh line 211
Confirm or edit the disk mapping
1) Confirm disk mapping and continue 'rear recover'
2) Edit disk mapping (/var/lib/rear/layout/disk_mappings)
3) Use Relax-and-Recover shell and return back to here
4) Abort 'rear recover'
(default '1' timeout 1 seconds)
UserInput: No real user input (empty or only spaces) - using default input
UserInput: Valid choice number result 'Confirm disk mapping and continue 'rear recover''
Continuing 'rear recover' by default
Examining /dev/sda to automatically resize its last active partition
Skipping /dev/sda (size of new disk same as size of old disk)
UserInput -I LAYOUT_FILE_CONFIRMATION needed in /usr/share/rear/layout/prepare/default/500_confirm_layout_file.sh line 26
Confirm or edit the disk layout file
1) Confirm disk layout and continue 'rear recover'
2) Edit disk layout (/var/lib/rear/layout/disklayout.conf)
3) View disk layout (/var/lib/rear/layout/disklayout.conf)
4) View original disk space usage (/var/lib/rear/layout/config/df.txt)
5) Use Relax-and-Recover shell and return back to here
6) Abort 'rear recover'
(default '1' timeout 1 seconds)
UserInput: No real user input (empty or only spaces) - using default input
UserInput: Valid choice number result 'Confirm disk layout and continue 'rear recover''
Continuing 'rear recover' by default
Doing SLES12-SP1 (and later) btrfs subvolumes setup because the default subvolume path contains '@/.snapshots/'
UserInput -I LAYOUT_CODE_CONFIRMATION needed in /usr/share/rear/layout/recreate/default/100_confirm_layout_code.sh line 26
Confirm or edit the disk recreation script
1) Confirm disk recreation script and continue 'rear recover'
2) Edit disk recreation script (/var/lib/rear/layout/diskrestore.sh)
3) View disk recreation script (/var/lib/rear/layout/diskrestore.sh)
4) View original disk space usage (/var/lib/rear/layout/config/df.txt)
5) Use Relax-and-Recover shell and return back to here
6) Abort 'rear recover'
(default '1' timeout 1 seconds)
UserInput: No real user input (empty or only spaces) - using default input
UserInput: Valid choice number result 'Confirm disk recreation script and continue 'rear recover''
Continuing 'rear recover' by default
Start system layout restoration.
Creating partitions for disk /dev/sda (gpt)
Creating filesystem of type btrfs with mount point / on /dev/sda2.
Mounting filesystem /
Running snapper/installation-helper
Creating filesystem of type xfs with mount point /home on /dev/sda4.
Mounting filesystem /home
Creating swap on /dev/sda3
Disk layout created.
UserInput -I LAYOUT_MIGRATED_CONFIRMATION needed in /usr/share/rear/layout/recreate/default/200_run_layout_code.sh line 98
Confirm the recreated disk layout or go back one step
1) Confirm recreated disk layout and continue 'rear recover'
2) Go back one step to redo disk layout recreation
3) Use Relax-and-Recover shell and return back to here
4) Abort 'rear recover'
(default '1' timeout 1 seconds)
UserInput: No real user input (empty or only spaces) - using default input
UserInput: Valid choice number result 'Confirm recreated disk layout and continue 'rear recover''
Continuing with recreated disk layout by default
Restoring from '/tmp/rear.Tt2PwYPVHGbJsLD/outputfs/d228/backup.tar.gz' (restore log in /var/lib/rear/restore/recover.backup.tar.gz.1866.restore.log) ...
Backup restore program 'tar' started in subshell (PID=4795)
Restored 12 MiB [avg. 4298 KiB/sec] 
Restored 116 MiB [avg. 19859 KiB/sec] 
Restored 251 MiB [avg. 28627 KiB/sec] 
Restored 453 MiB [avg. 38658 KiB/sec] 
Restored 636 MiB [avg. 43443 KiB/sec] 
Restored 853 MiB [avg. 48551 KiB/sec] 
Restored 1008 MiB [avg. 49171 KiB/sec] 
Restored 1219 MiB [avg. 52035 KiB/sec] 
Restored 1428 MiB [avg. 54162 KiB/sec] 
Restored 1582 MiB [avg. 54009 KiB/sec] 
Restored 1792 MiB [avg. 55631 KiB/sec] 
Restored 2008 MiB [avg. 57123 KiB/sec] 
Restored 2190 MiB [avg. 57508 KiB/sec] 
Restored 2433 MiB [avg. 59330 KiB/sec] 
OK
Restored 2444 MiB in 45 seconds [avg. 55630 KiB/sec]
Restoring finished (verify backup restore log messages in /var/lib/rear/restore/recover.backup.tar.gz.1866.restore.log)
Recreating directories (with permissions) from /var/lib/rear/recovery/directories_permissions_owner_group
UserInput -I RESTORED_FILES_CONFIRMATION needed in /usr/share/rear/finalize/default/020_confirm_finalize.sh line 35
Confirm restored config files or edit them
1) Confirm it is OK to recreate initrd and reinstall bootloader and continue 'rear recover'
2) Edit restored etc/fstab (/mnt/local/etc/fstab)
3) View restored etc/fstab (/mnt/local/etc/fstab)
4) Use Relax-and-Recover shell and return back to here
5) Abort 'rear recover'
(default '1' timeout 1 seconds)
UserInput: No real user input (empty or only spaces) - using default input
UserInput: Valid choice number result 'Confirm it is OK to recreate initrd and reinstall bootloader and continue 'rear recover''
Continuing 'rear recover' by default
Applying disk layout mappings in /var/lib/rear/layout/disk_mappings to certain restored files...
The original restored files get saved in var/lib/rear/saved_original_files/ (in /mnt/local)
Applied disk layout mappings to restored 'boot/grub2/grub.cfg' (in /mnt/local)
Applied disk layout mappings to restored 'boot/grub2/device.map' (in /mnt/local)
Applied disk layout mappings to restored 'etc/sysconfig/bootloader' (in /mnt/local)
Applied disk layout mappings to restored 'etc/fstab' (in /mnt/local)
Applied disk layout mappings to restored 'etc/mtools.conf' (in /mnt/local)
Applied disk layout mappings to restored 'etc/smartd.conf' (in /mnt/local)
Applied disk layout mappings to restored 'etc/sysconfig/smartmontools' (in /mnt/local)
Patching /etc/default/grub_installdevice: Replacing [/dev/disk/by-id/ata-QEMU_HARDDISK_QM00001] by [/dev/disk/by-id/ata-QEMU_HARDDISK_QM00003]
Updating udev configuration (70-persistent-net.rules)
Running mkinitrd...
Recreated initrd (/sbin/mkinitrd).
Installing GRUB2 boot loader...
Determining where to install GRUB2 (no GRUB2_INSTALL_DEVICES specified)
Found possible boot disk /dev/sda - installing GRUB2 there
Finished recovering your system. You can explore it under '/mnt/local'.
Exiting rear recover (PID 1866) and its descendant processes
Running exit tasks

I.e. all those UserInput dialogs proceed automatically
with the default choice after one second.

Perhaps I found a possible problem with the disk migration
in your particular case:

In the above terminal output (I use latest GitHub master code)
the following part worries me:

Applying disk layout mappings in /var/lib/rear/layout/disk_mappings to certain restored files...
The original restored files get saved in var/lib/rear/saved_original_files/ (in /mnt/local)
Applied disk layout mappings to restored 'boot/grub2/grub.cfg' (in /mnt/local)
Applied disk layout mappings to restored 'boot/grub2/device.map' (in /mnt/local)
Applied disk layout mappings to restored 'etc/sysconfig/bootloader' (in /mnt/local)
Applied disk layout mappings to restored 'etc/fstab' (in /mnt/local)
Applied disk layout mappings to restored 'etc/mtools.conf' (in /mnt/local)
Applied disk layout mappings to restored 'etc/smartd.conf' (in /mnt/local)
Applied disk layout mappings to restored 'etc/sysconfig/smartmontools' (in /mnt/local)
Patching /etc/default/grub_installdevice: Replacing [/dev/disk/by-id/ata-QEMU_HARDDISK_QM00001] by [/dev/disk/by-id/ata-QEMU_HARDDISK_QM00003]

Those changes had happened all the time but since
https://github.com/rear/rear/pull/1843 is merged therein the commit
https://github.com/rear/rear/pull/1843/commits/632bc2dee4d5b82cfe91416c81254f0bc9919a70
makes that obvious to the user via LogPrint that tell about what is done, cf.
https://github.com/rear/rear/issues/1847

That part is from scripts like
usr/share/rear/finalize/GNU/Linux/250_migrate_disk_devices_layout.sh
and subsequent scripts like
usr/share/rear/finalize/GNU/Linux/280_migrate_uuid_tags.sh
usr/share/rear/finalize/GNU/Linux/250_migrate_lun_wwid.sh
usr/share/rear/finalize/GNU/Linux/260_rename_diskbyid.sh
usr/share/rear/finalize/GNU/Linux/320_migrate_network_configuration_files.sh
which modify files in the recreated system.

But in your particular case I assume
sda is the ReaR recovery system only during "rear recover"
but later during normal operation of the recreated system
the ReaR recovery system disk is no longer there so that then
sda becomes the usual system disk of the recreated system.

But the above scripts had modified certain files on the recreated system.
In particular the disk mapping /dev/sda => /dev/sdb was applied to some files
so that now those files contain /dev/sdb instead of /dev/sda which is
probably wrong during normal operation of the recreated system
where actually /dev/sda is the first system disk (and not /dev/sdb).

OliverO2 commented at 2018-07-05 13:49:

@GreenBlood Some remarks:

  1. The difference between your old and new configuration is something like this:
    • The optical boot drive (based on an ISO image) was controlled by the kernel's CD-ROM driver (possibly sr), so it may have appeared under /dev/sr0.
    • The hard disk boot drive (based on a RAWDISK image) is contolled by the kernel's SCSI disk driver, so it appears under /dev/sda.
  2. The optical drive's device node /dev/sr0 could never clash with that of the first regular hard disk /dev/sda. So your regular hard disks were assigned the same device nodes during recovery as during normal operation.
  3. Now, your new recovery boot device, being presented as the first hard disk by QEMU, becomes /dev/sda, moving your regular hard disks to /dev/sdb and beyond.

Block device names (and by the way, the virtual ISO optical drive is also a block device) are assigned by the respective kernel driver in initialization order. (For some information on bus-based naming, see Device Names and Persistent block device naming - ArchWiki.)

In my view the easiest option for you would be to present your recovery boot device to the kernel either

  • a) as a disk device, which is initialized after the internal hard disks, but still can serve as a boot device, or
  • b) as a non-disk device (like before).

It looks like QEMU has very limited boot options but may boot anyway from secondary devices if the first hard disk is not bootable, so some ideas:

  • Could you make QEMU present your recovery (RAWDISK) drive as an attached USB device via -device usb-storage (see qemu usb storage emulation)?
  • Could you make QEMU present your recovery (RAWDISK) drive as a secondary (or tertiary or later) hard disk via -drive file=FILENAME,index=2,media=disk?
  • What is the reason that you cannot make QEMU attach an ISO image as a bootable optical drive via its command line options -cdrom FILENAME.iso -boot order=d?

jsmeix commented at 2018-07-06 10:21:

I made https://github.com/rear/rear/issues/1854
where we can discuss about that issue.

jsmeix commented at 2018-07-09 09:36:

I think the initial question in this issue how to say to rear "do your thing"
is answered so that I can close this issue.

The possible problems that could happen when
current ReaR does its thing in this particular case
can be further discussed in https://github.com/rear/rear/issues/1854

[Export of Github issue for rear/rear.]