#1380 Issue closed: ReaR recovery fails when the OS contains a Thin Pool/Volume

Labels: enhancement, fixed / solved / done

gdha opened issue at 2017-06-09 15:11:

  • rear version (/usr/sbin/rear -V): 1.17.2_3 (and 2.0)
  • OS version (cat /etc/rear/os.conf or lsb_release -a): RHEL 7.3
  • rear configuration files (cat /etc/rear/site.conf or cat /etc/rear/local.conf): BACKUP=NETFS
  • Are you using legacy BIOS or UEFI boot? no
  • Brief description of the issue:
    When using ReaR, the recovery process of a system which has a VG which contains a Thin Pool/Volume fails and the system is no longer recoverable. This has a high impact because ReaR will not be able to restore the system back to an operational state. The location of the Thin Pool is independent of the VG which contains it, that is, it does not matter if the Thin Pool is contained in the rootvg or in an independent VG.
  • Work-around, if any: see RH bug report https://bugzilla.redhat.com/show_bug.cgi?id=1450667

gdha commented at 2017-07-19 15:07:

post-pone to release 2.3

jmazanek commented at 2017-09-04 13:16:

With the rear-2.00 the system recovery works, but the thin pool is not preserved.
Here are the details of the testing we did:

Details of a new reproducer in my lab using the latest version of ReaR:

Version installed: rear-2.00-2.el7.x86_64
Hypervisor: ofamera-devel.usersys.redhat.com
Original machine: fvm-rhel-7-3-34 <-- Executed 'rear mkbackup'
Recovery machine: fvm-rhel-7-3-38 <-- Executed 'rear recover'
NFS backup server: fvm-rhel-7-3-44
User: root
Pass: testtest

*** BACKUP TEST ***

RESULT: Backup taken successfully. ISO + backup.tar.gz were sent to the NFS server

-> Original system (fvm-rhel-7-3-34):

[root@fvm-rhel-7-3-34 ~]# rear -d -v mkbackup
Relax-and-Recover 2.00 / Git
Using log file: /var/log/rear/rear-fvm-rhel-7-3-34.log
Using backup archive 'backup.tar.gz'
Creating disk layout
Creating root filesystem layout
Copying logfile /var/log/rear/rear-fvm-rhel-7-3-34.log into initramfs as '/tmp/rear-fvm-rhel-7-3-34-partial-2017-08-21T10:14:10+0200.log'
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Creating initramfs
Making ISO image
Wrote ISO image: /var/lib/rear/output/rear-fvm-rhel-7-3-34.iso (134M)
Copying resulting files to nfs location
Saving /var/log/rear/rear-fvm-rhel-7-3-34.log as rear-fvm-rhel-7-3-34.log to nfs location
Creating tar archive '/tmp/rear.WaTt8XedE2CpdJ3/outputfs/fvm-rhel-7-3-34/backup.tar.gz'
Archived 850 MiB [avg 3349 KiB/sec] OK
Archived 850 MiB in 261 seconds [avg 3336 KiB/sec]
You should also rm -Rf /tmp/rear.WaTt8XedE2CpdJ3
[root@fvm-rhel-7-3-34 ~]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root_lv r7vg -wi-ao---- 4.88g
swap_lv r7vg -wi-ao---- 256.00m
lv_thin vg_thin Vwi-a-tz-- 1.00g lv_thinpool 0.00
lv_thinpool vg_thin twi-aotz-- 92.00m 0.00 0.98

-> NFS backup server (fvm-rhel-7-3-44):

[root@fvm-rhel-7-3-44 ~]# ls -l /media/backups/fvm-rhel-7-3-34/
total 1010332
-rw-------. 1 nfsnobody nfsnobody 2252128 Aug 21 10:20 backup.log
-rw-------. 1 nfsnobody nfsnobody 891757428 Aug 21 10:20 backup.tar.gz
-rw-------. 1 nfsnobody nfsnobody 202 Aug 21 10:16 README
-rw-------. 1 nfsnobody nfsnobody 140369920 Aug 21 10:15 rear-fvm-rhel-7-3-34.iso
-rw-------. 1 nfsnobody nfsnobody 183744 Aug 21 10:16 rear-fvm-rhel-7-3-34.log
-rw-------. 1 nfsnobody nfsnobody 0 Aug 21 10:20 selinux.autorelabel
-rw-------. 1 nfsnobody nfsnobody 273 Aug 21 10:16 VERSION

*** RECOVERY TEST ***

RESULT: It recovered the system (it was able to boot afterwards) but only the OS itself (root VG/LVs), not the LVM Thin Pool/Volumes in the second disk

-> Original system (fvm-rhel-7-3-34):

[jserrano@ofamera-devel ~]$ fast-vm ssh 34
[inf] checking the 192.168.33.34 for active SSH connection (ctrl+c to interrupt)
[inf]
SSH ready
Warning: Permanently added '192.168.33.34' (ECDSA) to the list of known hosts.

System is booting up. See pam_nologin(8)
Last login: Mon Aug 21 12:35:16 2017 from gateway

[root@fvm-rhel-7-3-34 ~]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root_lv r7vg -wi-ao---- 4.88g
swap_lv r7vg -wi-ao---- 256.00m
lv_thin vg_thin Vwi-a-tz-- 1.00g lv_thinpool 0.00
lv_thinpool vg_thin twi-aotz-- 92.00m 0.00 0.98
[root@fvm-rhel-7-3-34 ~]#
[root@fvm-rhel-7-3-34 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 6G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 5.5G 0 part
├─r7vg-root_lv 253:0 0 4.9G 0 lvm /
└─r7vg-swap_lv 253:1 0 256M 0 lvm [SWAP]
sdb 8:16 0 102.4M 0 disk
├─vg_thin-lv_thinpool_tmeta 253:2 0 4M 0 lvm
│ └─vg_thin-lv_thinpool-tpool 253:4 0 92M 0 lvm
│ ├─vg_thin-lv_thinpool 253:5 0 92M 0 lvm
│ └─vg_thin-lv_thin 253:6 0 1G 0 lvm
└─vg_thin-lv_thinpool_tdata 253:3 0 92M 0 lvm
└─vg_thin-lv_thinpool-tpool 253:4 0 92M 0 lvm
├─vg_thin-lv_thinpool 253:5 0 92M 0 lvm
└─vg_thin-lv_thin 253:6 0 1G 0 lvm
sr0 11:0 1 1024M 0 rom

-> Recovery machine (fvm-rhel-7-3-38) -after 'rear recover'-:

[jserrano@ofamera-devel ~]$ fast-vm ssh 38
[inf] checking the 192.168.33.38 for active SSH connection (ctrl+c to interrupt)
[inf]
SSH ready
Warning: Permanently added '192.168.33.38' (ECDSA) to the list of known hosts.
Last login: Mon Aug 21 12:26:46 2017

[root@fvm-rhel-7-3-34 ~]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root_lv r7vg -wi-ao---- 4.88g
swap_lv r7vg -wi-ao---- 256.00m

[root@fvm-rhel-7-3-34 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 6G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 5.5G 0 part
├─r7vg-root_lv 253:0 0 4.9G 0 lvm /
└─r7vg-swap_lv 253:1 0 256M 0 lvm [SWAP]
sdb 8:16 0 102.4M 0 disk

gdha commented at 2017-10-31 16:05:

@jmazanek Thank you for your comments (it is also part of the RH BZ report). As I wrote in the BZ we should start with a prep script to capture the required binaries in the rescue image. Afterwards it will be easier to add the missing pieces

gdha commented at 2017-11-27 16:47:

We are very sorry but we will postpone this for a later release

gdha commented at 2018-05-15 06:59:

@rmetrich Are you able to assist us with this issue? Is was originally reported via RedHat. Would be nice if it could be added for ReaR v2.5?

rmetrich commented at 2018-05-15 07:11:

@gdha Hello, I will assist, but for now we do not have a fix on the RH side as well. I need to sync with LVM specialists to have that move forward.
It's in my TODO now :-)

rmetrich commented at 2018-05-16 14:56:

@gdha See above pull request

gdha commented at 2018-09-19 13:25:

as it is part of ReaR 2.4 already this one can be closed - thanks to Renaud @rmetrich

[Export of Github issue for rear/rear.]