#1925 Issue `closed`: SLES 12 SP3 on IBM Power using the SAN disks¶

Labels: support / question, fixed / solved / done

cychung1 opened issue at 2018-10-10 05:55:¶

rear-mgilxps2hd99.log
disklayout.conf.txt
rear-recovery-run-screen-output.txt
diskrestore.sh.txt
make_rear_diskrestore_script-20181009-1759 (1).log

ReaR version ("/usr/sbin/rear -V"):
Relax-and-Recover 2.4 / 2018-06-21
OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"):
NAME="SLES"
VERSION="12-SP3"
VERSION_ID="12.3"
PRETTY_NAME="SUSE Linux Enterprise Server 12 SP3"
ID="sles"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles_sap:12:sp3"
ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=4,nolock"
BACKUP_URL=nfs://192.168.125.94/Solman_Backups
NETFS_KEEP_OLD_BACKUP_COPY=yes
BACKUP_PROG_INCLUDE=( /var/lib/named /var/opt /var/lib/machines /var/log /var/lib/mysql /var/lib/mariadb /boot/grub2/powerpc-ieee1275 /usr/local /opt /var/lib/pgsql /var/lib/mailman /var/lib/libvirt/images /var/spool /tmp /var/tmp /srv /home /hana/backup /hana/log /hana/shared /usr/sap)
EXCLUDE_VG=( saphanadatavg )
SSH_ROOT_PASSWORD="password_on_the_rear_recovery_system"
Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR):
PowerVM LPAR (E880C)
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
PPC64LE
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
BIOS and GRUB2
Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
SAN (IBM Storwize V7000) via FC with multipath-tools v0.4.9 (05/33, 2016)
Description of the issue (ideally so that others can reproduce it):
The source and destination servers (LPARs) are different with the same disk layout and size.
Prior to the run, setting the environment variable MIGRATION_MODE='true'.
But failing during the layout recreation.
ERROR: No filesystem mounted on '/mnt/local'. Stopping
Work-around, if any:
none

jsmeix commented at 2018-10-10 07:13:¶

@cychung1
I am neither a SAN nor a multipath user so that I cannot provide
educated help but I try nevertheless - @schabrolles is our ReaR upstream
expert for IBM POWER in general and for SAN and multipath in particular.

As far as I see in your disklayout.conf basically all entries are disabled
(i.e. commented with a leading '#' character) which means nothing
will be recreated and accordingly your diskrestore.sh script
is basically empty.

As far as I remember the reason is that by default disk and partition
and filesystem entries that belong to SAN storage get disabled
because usually one does not want to recreate non-local things
that exist on a remote SAN storage because when the local
hardware gets destroyed, the remote SAN storage is usually
not affected so that the remote SAN storage should not be touched
by a "rear recover" run on one of the SAN storage client systems.
Simply put:
By default "rear recover" is meant to recreate only
the local system (in particular local disks).

Accordingly the default setting in usr/share/rear/conf/default.conf is

# Automatically exclude multipath disks and their dependent components
AUTOEXCLUDE_MULTIPATH=y

When you set in your /etc/rear/local.conf AUTOEXCLUDE_MULTIPATH=n
then "rear recover" will do disk partitioning, creating filesystems and so on
on your remote SAN storage multipath devices so that you need to
carefully inspect the disklayout.conf file that gets created during
"rear mkrescue/mkbackup" if the actually intended things will happen
later during "rear recover" on your remote SAN storage (i.e. be careful
to not destroy unwanted things on your remote SAN storage).

Regarding SAN and multipath see also the similar looking issue
https://github.com/rear/rear/issues/1899
which mentiones additionally boot over SAN.

jsmeix commented at 2018-10-10 07:25:¶

@cychung1
by the way:
In your disklayout.conf I see you use btrfs subvolumes but
I do not see in your /etc/rear/local.conf the matching entries
that are usually needed to recreate the special SLES12 btrfs
subvolume structure with its special interdependency with snapper
plus btrfs quota setup for snapper, see our SLE12 btrfs example
configuration files in usr/share/rear/conf/examples/
in particular since SLES12-SP2
https://raw.githubusercontent.com/rear/rear/master/usr/share/rear/conf/examples/SLE12-SP2-btrfs-example.conf

Additionally:
Regarding SAN and multipath and boot on SAN specific stuff on PPC64/PPC64LE see
https://raw.githubusercontent.com/rear/rear/master/usr/share/rear/conf/examples/RHEL7-PPC64LE-Mulitpath-PXE-GRUB.conf

jsmeix commented at 2018-10-10 08:12:¶

@cychung1
another side note regarding your
BACKUP_PROG_INCLUDE=( ... /hana/backup /hana/log /hana/shared ... )
and the Backup archive size is 35G in your rear-recovery-run-screen-output.txt

I assume things in /hana/ belong to a SAP HANA database.
I think that HANA stuff makes the backup huge.

I would thoroughly recommend as far as possible
to Keep Separated Items Separated ( 'KSIS' ;-)
to Keep It Simple and Straightforward,
see 'KISS' at https://en.wikipedia.org/wiki/KISS_principle
and cf. RFC 1925 item (5) at https://tools.ietf.org/html/rfc1925

The ReaR backup is primarily meant to restore the files of the basic
operating system but not to also restore big data like a whole database, see
the sections "Basics" and "Relax-and-Recover versus backup and restore" in
https://en.opensuse.org/SDB:Disaster_Recovery

The reason is that with a small ReaR backup that contains only
what is needed for the basic operating system you get your
basic operating system up and running again much faster
during "rear recover" compared to restoring a huge
'all-in-one' backup with "rear recover" because restoring the
backup is the main time consuming task during "rear recover".

Related to that how to speed up backup restore you may have a look at
https://github.com/rear/rear/blob/master/doc/user-guide/11-multiple-backups.adoc

Normally you would restore big data as a second separated step
after "rear recover" from within the rebooted recreated system.

This way you can restore a database with its specific tool
e.g. a specific database restore tool from the manufacturer
or vendor of your particular database.

You cannot do that easily during "rear recover" because
"rear recover" runs within the special (rather minimal)
ReaR recovery system which does not contain such specific tools
unless you manually adapt and enhance ReaR to get that in which
means you have to add one more external backup method to ReaR, cf.
https://github.com/rear/rear/blob/master/doc/user-guide/10-integrating-external-backup.adoc

cychung1 commented at 2018-10-11 04:57:¶

Thank you very much for your prompt reply.
Once proper parameters were set and reduced the scope of backup (only rootvg), I was able to recover successfully (/etc/fstab was modified).
One question though, if one of VGs on the source system made up with a multiple PVs and the target has only one PV with the equal size, how do you map four LUNs (from source) to one LUN (to target)?

jsmeix commented at 2018-10-11 08:04:¶

Offhandedly (I did not check the details in the code right now) but
I assume that ReaR does not automatically map logical volumes.

My assumption is based on when I fixed in ReaR version 2.4
how ReaR automatically resizes partitions, see in the
default.conf file the description of the config variables
AUTORESIZE_PARTITIONS
AUTORESIZE_EXCLUDE_PARTITIONS
AUTOSHRINK_DISK_SIZE_LIMIT_PERCENTAGE
AUTOINCREASE_DISK_SIZE_THRESHOLD_PERCENTAGE
which reads (excerpts):

Resizing partitions in MIGRATION_MODE during "rear recover"
... this does not resize volumes on top of the affected partitions.
To migrate volumes on a disk where the disk size had changed
the user must in advance manually adapt his disklayout.conf file
before he runs "rear recover".

Accordingly I think ReaR does not automatically migrate
logical volumes in general.

This leads to the general topic what ReaR supports
to migrate a system to somewhat different new hardware
(where "hardware" could be also a virtual machine)
or to migrate a system to a different disk layout.

In general migrating a system to different hardware
or migrating a system to a different disk layout
does not "just work", cf. "Inappropriate expectations" at
https://en.opensuse.org/SDB:Disaster_Recovery

On the other hand ReaR supports to migrate a system
but here "supports" means that ReaR provides a lot
that should help you to get such a task done
but it does not mean that it would "just work" without
possibly laborious manual settings and adaptions
with trial and error legwork until you made things work
for you in your particular case, cf.
https://github.com/rear/rear/issues/1822#issuecomment-405505923

In particular for an example how ReaR can help to
migrate a system to a different disk layout
(a SLES12 btrfs to XFS migration), see
http://lists.relax-and-recover.org/pipermail/rear-users/2018-April/003547.html

In general regarding migrating a system you may have a look at
http://lists.relax-and-recover.org/pipermail/rear-users/2018-February/003528.html
and follow the links therein.

I know that the current syntax of the disklayout.conf file entries
is not really user friendly (e.g. one must specify byte values), cf.
https://raw.githubusercontent.com/rear/rear/master/doc/user-guide/06-layout-configuration.adoc
but improvements here require tons of changes in many ReaR scripts.

With current ReaR I think RECOVERY_UPDATE_URL could help here see
http://lists.relax-and-recover.org/pipermail/rear-users/2018-February/003528.html
so that one could do the laborious work to edit disklayout.conf
in a more relaxed way not within the running ReaR recovery system
but in advance on your normally used system and provide the right
prepared disklayout.conf via RECOVERY_UPDATE_URL, cf.
"A note on the meaning of 'Relax' in 'Relax-and-Recover'" in
https://en.opensuse.org/SDB:Disaster_Recovery

jsmeix commented at 2018-10-16 13:51:¶

According to https://github.com/rear/rear/issues/1925#issuecomment-428820780
this issue is solved so that I close it hereby.

cychung1 commented at 2018-10-17 08:07:¶

Thank you very much, Johannes. I got sidetracked on this project, but plan to get back and test further before this can be used in a production environment.

Regards,
Chris

On Oct 16, 2018, at 6:51 AM, Johannes Meixner notifications@github.com wrote:

According to #1925 (comment)
this issue is solved so that I close it hereby.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[Export of Github issue for rear/rear.]

#1925 Issue closed: SLES 12 SP3 on IBM Power using the SAN disks¶