#2294 Issue closed: SUSE 12 (ppc64le) Patch level 4 OS Backup & restore through rear utility on IBM Power system¶
Labels: support / question, fixed / solved / done
Ajju1 opened issue at 2019-12-09 04:08:¶
Relax-and-Recover (ReaR) Issue Template¶
Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):
-
ReaR version ("/usr/sbin/rear -V"):
Relax-and-Recover 2.4 / Git -
OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"):
SUSE Linux Enterprise Server 12 (ppc64le)
VERSION = 12
PATCHLEVEL = 4
# This file is deprecated and will be removed in a future service pack or release.
# Please check /etc/os-release for details about this release.
cat /etc/rear/os.conf
OS_VENDOR=SUSE_LINUX
OS_VERSION=12
- ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat
/etc/rear/local.conf"):
cat /etc/rear/local.conf
# Default is to create Relax-and-Recover rescue media as ISO image
# set OUTPUT to change that
# set BACKUP to activate an automated (backup and) restore of your data
# Possible configuration values can be found in /usr/share/rear/conf/default.conf
#
# This file (local.conf) is intended for manual configuration. For configuration
# through packages and other automated means we recommend creating a new
# file named site.conf next to this file and to leave the local.conf as it is.
# Our packages will never ship with a site.conf.
BOOT_FROM_SAN=y
OUTPUT=ISO
OUTPUT_URL=nfs://10.3.3.101/mksysb/suse_bkp
BACKUP=NETFS
BACKUP_URL=nfs://10.3.3.101/mksysb/suse_bkp
#BACKUP_OPTIONS="nfsvers=3,nolock"
AUTOEXCLUDE_MULTIPATH=n
#ONLY_INCLUDE_VG=( "system" )
EXCLUDE_COMPONENTS=(/dev/mapper/36005076305ffd1f40000000000005a01 /dev/mapper/36005076305ffd1f40000000000005a02 /dev/mapper/36005076305ffd1f40000000000005a03 /dev/mapper/36005076305ffd1f40000000000005a04 /dev/mapper/36005076305ffd1f40000000000005a05 /dev/mapper/36005076305ffd1f40000000000005a06 /dev/mapper/36005076305ffd1f40000000000005a07 /dev/mapper/36005076305ffd1f40000000000005a08 /dev/mapper/36005076305ffd1f40000000000005b00 /dev/mapper/36005076305ffd1f40000000000005b01 /dev/mapper/36005076305ffd1f40000000000005b02 /dev/mapper/36005076305ffd1f40000000000005b03 /dev/mapper/36005076305ffd1f40000000000005b04 /dev/mapper/36005076305ffd1f40000000000005b05 /dev/mapper/36005076305ffd1f40000000000005b06 /dev/mapper/36005076305ffd1f40000000000005b07)
#EXCLUDE_DEVICE_MAPPING=( "loop*" "ram*" )
BACKUP_PROG_INCLUDE=('/' '/var/log' '/tmp' '/srv' '/boot/grub2/powerpc-ieee1275' '/.snapshots' '/home' '/var/lib/mailman' '/var/cache' '/var/crash' '/var/lib/machines' '/var/lib/named' '/var/lib/pgsql' '/var/opt' '/var/lib/libvirt/images' '/var/lib/mysql' '/usr/local' '/opt' '/var/lib/mariadb' '/var/tmp' '/var/spool' )
-
Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR):
PowerVM Lpar on IBM Power 9 Server -
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
ppc64le -
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
bootloader: Grub -
Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
San bootable and multipathing -
Description of the issue (ideally so that others can reproduce it):
We are taking only system/rootvg backup from source and trying to restore on another lpar for restoration testing. During the restoration we are changing IP of the target system after booting from ISO file created by rear backup of source through HMC and then map the disk only for system vg.
Since, we have to just restore the system vg/rootvg only, so we have assigned the same size disk to destination VG and rest disks required for datavg are not assigned.
To achieve this we have done some configuration in the source system /etc/rear/local.conf file for excluding the mapping of disk other than system VG.
While restoring the disk mapping is getting failed and thus results in restoration failure.
Few more things regarding the environment.
The systems are same hardware but different lpar on IBM Power server
Nfs is used for taking backup and restoration of data
Booting lpar in SMS mode through iso file created by the rear backup of
source system and then assigning different IP
After assigning new IP, we are using rear -d D restore command
PFA are files of source configuration, rear local.conf file and rear
backup log; along with the destination steps followed and logs.
-
Workaround, if any:
-
Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
rear backup.zip
To paste verbatim text like command output or file content,
include it between a leading and a closing line of three backticks like
```
verbatim content
```
jsmeix commented at 2019-12-09 09:27:¶
@Ajju1
Your log is only rear -d -v recover but usually
we need full debug logs i.e. with the -D option.
In your ...hmciecchdb02s-target.log
the diskrestore.sh script fails at its beginning with
2019-12-05 06:55:16.328139872 Start system layout restoration.
WARNING: Failed to connect to lvmetad. Falling back to device scanning.
Logical volume system/root contains a filesystem in use.
Can't deactivate volume group "system" with 1 open logical volume(s)
+++ create_component /dev/mapper/36005076305ffd1f40000000000000001 multipath
+++ local device=/dev/mapper/36005076305ffd1f40000000000000001
+++ local type=multipath
+++ local touchfile=multipath--dev-mapper-36005076305ffd1f40000000000000001
+++ '[' -e /tmp/rear.3k6jeleDwzKz3CJ/tmp/touch/multipath--dev-mapper-36005076305ffd1f40000000000000001 ']'
+++ return 0
+++ LogPrint 'Creating partitions for disk /dev/mapper/36005076305ffd1f40000000000000001 (msdos)'
+++ Log 'Creating partitions for disk /dev/mapper/36005076305ffd1f40000000000000001 (msdos)'
++++ date '+%Y-%m-%d %H:%M:%S.%N '
+++ local 'timestamp=2019-12-05 06:55:16.572869572 '
+++ test 1 -gt 0
+++ echo '2019-12-05 06:55:16.572869572 Creating partitions for disk /dev/mapper/36005076305ffd1f40000000000000001 (msdos)'
2019-12-05 06:55:16.572869572 Creating partitions for disk /dev/mapper/36005076305ffd1f40000000000000001 (msdos)
+++ Print 'Creating partitions for disk /dev/mapper/36005076305ffd1f40000000000000001 (msdos)'
+++ test 1
+++ echo -e 'Creating partitions for disk /dev/mapper/36005076305ffd1f40000000000000001 (msdos)'
+++ my_udevsettle
+++ has_binary udevadm
+++ for bin in '$@'
+++ type udevadm
+++ return 0
+++ udevadm settle
+++ return 0
+++ parted -s /dev/mapper/36005076305ffd1f40000000000000001 mklabel msdos
Error: Partition(s) 2 on /dev/mapper/36005076305ffd1f40000000000000001
have been written, but we have been unable to inform the kernel of the change,
probably because it/they are in use. As a result, the old partition(s) will
remain in use. You should reboot now before making further changes.
...
2019-12-05 06:55:18.796366577 The disk layout recreation script failed
but I cannot imagine what the actual root cause behind is.
Could it be that you did something in the running ReaR recovery system
before you started "rear recover" and what you did results that the
target system storage components logical volume system/root
and/or dev/mapper/36005076305ffd1f40000000000000001
are actually in use (e.g. is something perhaps even mounted)?
For example after a previous (possibly failed) run of "rear recover"
some target system filesystems could be mounted under /mnt/local
so you need to umount whatever is mounted below /mnt/local
before you could re-run "rear recover".
In your ...hmciecchdb02s-target.log I see lots of lines
about btrfsmountedsubvol which indicate you use SLES12
with its (rather complicated) default btrfs structure
but I don't see in your etc/rear/local.conf the matching
entries for the SLES12 default btrfs structure,
cf. the SLE12*btrfs example config files in ReaR
https://github.com/rear/rear/tree/master/usr/share/rear/conf/examples
So when you use SLES12 with its default btrfs structure
you need first and foremost a matching etc/rear/local.conf
see also our SLE12-HA documentation Administration Guide
the section about "Disaster Recovery with Relax-and-Recover (Rear)"
https://documentation.suse.com/sle-ha/12-SP4/html/SLE-HA-all/cha-ha-rear.html
that reads (excerpt)
Limitations with Btrfs
...
Your SLE12 System Needs Matching Rear Configuration
The setup in SLE12 GA, SLE12 SP1, and SLE12 SP2
have several incompatible Btrfs default structures.
As such, it is crucial to use a matching Rear
configuration file. See the example files
/usr/share/rear/conf/examples/SLE12*-btrfs-example.conf.
Provided your etc/rear/local.conf is right for your SLES12 system
and "rear recover" still fails in your case, we need for further
analysis full debug logs i.e. with the -D option:
rear -D mkrescue/mkbackup and rear -D recover
for the latter cf. "Debugging issues with Relax-and-Recover" in
https://en.opensuse.org/SDB:Disaster_Recovery
Furthermore (also see that section) we would like to get
in particular your /var/lib/rear/layout/disklayout.conf
and your /var/lib/rear/layout/diskrestore.sh files.
Additionally to get an easier overview about your storage layout
please provide on your original system the output of the command
lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT
Ajju1 commented at 2019-12-13 11:22:¶
Hi Johannes
Please find the attached
Logs
updated.zip
log. I also updated the local.conf file as you suggested, but still no
luck
jsmeix commented at 2019-12-16 15:23:¶
Follow what I had written in
https://github.com/rear/rear/issues/2294#issuecomment-563142611
in particular (excerpts):
When you use SLES12 with its default btrfs structure
you need first and foremost a matching etc/rear/local.conf
... it is crucial to use a matching Rear configuration file.
See the example files
/usr/share/rear/conf/examples/SLE12*-btrfs-example.conf
Provided your etc/rear/local.conf is right for your SLES12 system
and "rear recover" still fails in your case ... we need for further
analysis ...
rear -D mkrescue/mkbackup and rear -D recover
(we need both - really).
Furthermore ... we would like to get
... your /var/lib/rear/layout/disklayout.conf
and your /var/lib/rear/layout/diskrestore.sh files
(both files from one same rear mkrescue/recover).
Additionally to get an easier overview about your storage layout
please provide on your original system the output of the command
lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT
(that command as is - not split up into pieces).
Ajju1 commented at 2019-12-23 11:07:¶
Hi Johannes
Please find the attached logs
Logs
updated.zip
jsmeix commented at 2020-01-09 14:46:¶
@Ajju1
in your latest
https://github.com/rear/rear/files/3994700/Logs.updated.zip
therein your rear-D-mkbackup_hmciecchdb02s.log still contains
+ source /etc/rear/local.conf
++ BOOT_FROM_SAN=y
++ OUTPUT=ISO
++ OUTPUT_URL=nfs://10.3.3.101/mksysb/suse_bkp
++ BACKUP=NETFS
++ BACKUP_URL=nfs://10.3.3.101/mksysb/suse_bkp
++ AUTOEXCLUDE_MULTIPATH=n
++ EXCLUDE_COMPONENTS=(/dev/mapper/36005076305ffd1f40000000000005a01 /dev/mapper/36005076305ffd1f40000000000005a02 /dev/mapper/36005076305ffd1f40000000000005a03 /dev/mapper/36005076305ffd1f40000000000005a04 /dev/mapper/36005076305ffd1f40000000000005a05 /dev/mapper/36005076305ffd1f40000000000005a06 /dev/mapper/36005076305ffd1f40000000000005a07 /dev/mapper/36005076305ffd1f40000000000005a08 /dev/mapper/36005076305ffd1f40000000000005b00 /dev/mapper/36005076305ffd1f40000000000005b01 /dev/mapper/36005076305ffd1f40000000000005b02 /dev/mapper/36005076305ffd1f40000000000005b03 /dev/mapper/36005076305ffd1f40000000000005b04 /dev/mapper/36005076305ffd1f40000000000005b05 /dev/mapper/36005076305ffd1f40000000000005b06 /dev/mapper/36005076305ffd1f40000000000005b07)
++ BACKUP_PROG_INCLUDE=(/var/log /tmp /srv /boot/grub2/powerpc-ieee1275 /home /var/lib/mailman /var/cache /var/lib/machines /var/lib/named /var/lib/pgsql /var/opt /var/lib/libvirt/images /var/lib/mysql /usr/local /opt /var/lib/mariadb /var/tmp /var/spool)
which is still insufficient for SLE12 with btrfs.
For SLE12 SP2 and later with btrfs start with the template in
https://raw.githubusercontent.com/rear/rear/master/usr/share/rear/conf/examples/SLE12-SP2-btrfs-example.conf
and adapt is as needed to make your specific /etc/rear/local.conf file
provided you had initially installed your system as SLE12 SP2 or later.
If you had initially installed your system as SLE12 SP1 with btrfs
you would have to make your specific /etc/rear/local.conf file based
on
https://raw.githubusercontent.com/rear/rear/master/usr/share/rear/conf/examples/SLE12-SP1-btrfs-example.conf
If you had initially installed your system as SLE12 GA (SP0) with
btrfs
you would have to make your specific /etc/rear/local.conf file based
on
https://raw.githubusercontent.com/rear/rear/master/usr/share/rear/conf/examples/SLE12-btrfs-example.conf
For the reason behind why different SLE12 service packs need
different /etc/rear/local.conf files depending on what SLE12 service
pack
was initially installed (i.e. depending on what SLE12 service pack did
the
disk layout and btrfs structure setup) see
https://github.com/rear/rear/issues/1368#issuecomment-302410707
Regarding
Error: Partition(s) 2 on /dev/mapper/36005076305ffd1f40000000000000001
have been written, but we have been unable to inform the kernel of the change,
probably because it/they are in use
the reason could be that you use perhaps an already used disk
where old partitioning and LVM meta-data is still on the disk, cf.
"Prepare replacement hardware for disaster recovery" at
https://en.opensuse.org/SDB:Disaster_Recovery
that reads in particular (excerpt)
When your replacement storage is not pristine new storage
(i.e. when it had been ever used before), you must completely
zero out your replacement storage.
Otherwise when you try to reinstall your system ... on a disk
that had been used before, various kind of unexpected
weird issues could get in your way because of whatever
kind of remainder data on an already used disk
(for example remainders of RAID
or partition-table signatures and other kind of
"magic strings" like LVM metadata and whatever else).
For some more details and background information see
https://github.com/rear/rear/issues/799
jsmeix commented at 2020-01-09 15:05:¶
I know I had seen such an parted/partprobe inform the kernel
error message before and - voila! - here you are:
https://github.com/rear/rear/issues/793#issue-139355299
jsmeix commented at 2020-06-24 11:29:¶
Because "no news is good news" I assume this issue can be closed.
[Export of Github issue for rear/rear.]