#1973 Issue closed: ReaR 2.00 not working for SLES 12 SP3 on PowerVM LPAR with MPIO SAN device¶
Labels: waiting for info, support / question,
no-issue-activity
gsasidharan opened issue at 2018-11-20 09:14:¶
Relax-and-Recover (ReaR) Issue Template¶
Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):
-
ReaR version ("/usr/sbin/rear -V"): [root@rheltest ~]# rear -V
Relax-and-Recover 2.00 / Git -
OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"): SUSE 12 SP3
-
ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
-
Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR): PowerVM LPAR
-
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):ppc64
-
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
-
Storage (lokal disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):MPIO SAN device
-
Description of the issue (ideally so that others can reproduce it):
when rear recovery is initiated exiting either with NFS issue or GRUB issues..
No code has been generated to recreate swap:/dev/mapper/20017380035360016-part2 (swap).
To recreate it manually add code to /var/lib/rear/layout/diskrestore.sh or abort.
Manually add code that recreates swap:/dev/mapper/20017380035360016-part2 (swap) -
Workaround, if any: NO,failed multiple times
-
Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
RESCUE linux-yb9c:~ # rear recover
Relax-and-Recover 2.3 / 2017-12-20
Using log file: /var/log/rear/rear-linux-yb9c.log
Running workflow recover within the ReaR rescue/recovery system
Starting required daemons for NFS: RPC portmapper (portmap or rpcbind) and rpc.statd if available.
Started RPC portmapper 'rpcbind'.
RPC portmapper 'rpcbind' available.
Started rpc.statd.
RPC status rpc.statd available.
Starting rpc.idmapd failed.
Using backup archive '/tmp/rear.JG1gpxKHl4TeKTV/outputfs/linux-yb9c/backup.tar.gz'
Will do driver migration (recreating initramfs/initrd)
Calculating backup archive size
Backup archive size is 40K /tmp/rear.JG1gpxKHl4TeKTV/outputfs/linux-yb9c/backup.tar.gz (compressed)
Comparing disks
Disk configuration looks identical
Proceed with recovery (yes) otherwise manual disk layout configuration is enforced
(default 'yes' timeout 30 seconds)
yes
User confirmed to proceed with recovery
No code has been generated to recreate swap:/dev/mapper/20017380035360016-part2 (swap).
To recreate it manually add code to /var/lib/rear/layout/diskrestore.sh or abort.
Manually add code that recreates swap:/dev/mapper/20017380035360016-part2 (swap)
1) View /var/lib/rear/layout/diskrestore.sh
2) Edit /var/lib/rear/layout/diskrestore.sh
3) Go to Relax-and-Recover shell
4) Continue 'rear recover'
5) Abort 'rear recover'
(default '4' timeout 300 seconds)
4
Start system layout restoration.
Disk layout created.
ERROR: No filesystem mounted on '/mnt/local'. Stopping.
Aborting due to an error, check /var/log/rear/rear-linux-yb9c.log for details
Terminated
jsmeix commented at 2018-11-20 10:33:¶
See
https://github.com/rear/rear/blob/master/.github/ISSUE_TEMPLATE.md
what info is missing here and in general see
https://en.opensuse.org/SDB:Disaster_Recovery
On a first glance
Backup archive size is 40K ...
does not look promising...
jsmeix commented at 2018-11-20 10:37:¶
@schabrolles
because it happens on PowerVM LPAR with MPIO SAN device
perhaps you might be able to guess what happened here
(even without details).
jsmeix commented at 2018-11-20 10:43:¶
@gsasidharan
you use Relax-and-Recover 2.00 which is a bit old, from January 2017,
cf.
http://relax-and-recover.org/documentation/release-notes-2-4
Try out if a more current version works better for you, cf.
http://relax-and-recover.org/download/
See
http://relax-and-recover.org/documentation/release-notes-2-4
what has changed from ReaR 2.0 to the current ReaR 2.4.
There are some possibly backward incompatible changes.
I would also recommend to try out our current
ReaR upstream GitHub master code as follows:
Basically "git clone" our current ReaR upstream GitHub master code
into a separated directory and then configure and run ReaR
from within that directory like:
# git clone https://github.com/rear/rear.git # mv rear rear.github.master # cd rear.github.master # vi etc/rear/local.conf # usr/sbin/rear -D mkbackup
Note the relative paths "etc/rear/" and "usr/sbin/".
I recommend to try out our current ReaR upstream GitHub master code
because that is the only place where we fix bugs - i.e. bugs in older
ReaR versions are not fixed by us (i.e. by ReaR upstream).
If the issue also happens with current ReaR upstream GitHub master
code
please provide us complete ReaR debug log files of both
"rear -D mkbackup" and then "rear -D recover"
plus your 'var/lib/rear/layout/disklayout.conf' file
so that we can have a look how it behaves in your particular
environment
cf. "Debugging issues with Relax-and-Recover" at
https://en.opensuse.org/SDB:Disaster_Recovery
If it works with the current ReaR 2.4 or our current ReaR upstream
GitHub
master code we would really appreciate an explicit positive feedback.
For SUSE Linux Enterprise there should be official "rear23a" RPM
packages
that contain ReaR upstream version 2.4 plus some later ReaR upstream
commits,
see the section "rear / rear116 / rear1172a / rear118a / rear23a" in
https://en.opensuse.org/SDB:Disaster_Recovery
gsasidharan commented at 2018-11-20 11:27:¶
Thanks for your quick response...
already tried version 2.4 and got failures for SUSE.
RHEL worked perfectly fine.
will retry with below steps and confirm the status.
thanksSasi.G
Sent from my Samsung Galaxy smartphone.
jsmeix commented at 2018-11-20 11:54:¶
When things fail for SLES but work for RHEL my blind guess is
that you use SLES with its default btrfs structure which requires
special settings in your /etc/rear/local.conf for btrfs.
When you use SLES with its default btrfs structure
use a well matching *.btrfs-example.conf file from
/usr/share/rear/conf/examples/ as a starting point,
cf. the ones in our current ReaR upstream master code
https://github.com/rear/rear/tree/master/usr/share/rear/conf/examples
and read
https://github.com/rear/rear/issues/1368#issuecomment-302410707
because for SLE12 there are several different
incompatible btrfs default setups where each one
has its own specific *.btrfs-example.conf file.
schlomo commented at 2018-11-20 12:11:¶
@jsmeix looking at those examples I wonder why we don't include the snapper and other tools by default? Wouldn't that make ReaR work on all SLES versions without "special" configuration?
jsmeix commented at 2018-11-20 12:36:¶
@schlomo
as far as I see SLES12 with its several different incompatible default
btrfs structures
will always require an experienced admin who knows what specific kind
of
SLES12 btrfs structure there actually is on his particular system
so that he can specify the right things in his etc/rear/local.conf
that match his particular kind of SLES12 btrfs structure.
Of course in theory with unreasonable effort it might be even possible
to let ReaR autodetermine what specific kind of SLES12 btrfs structure
there actually is on a particular system but at least I won't implement
that
unless I am forced by SUSE who is convinced by one or more customers
who pay us for all those efforts ;-)
I think in practice it is more reasonable to leave it as it is now
in particular because at least for now it seems things have reasonably
stabilized since SLE12-SP2 because I am not aware of any new different
incompatible default btrfs structures after SLE12-SP2.
SLE12-SP2-btrfs-example.conf also works on SLE12-SP3 and SLE12-SP4
and it should also work on SLE15 (at least it worked for me for my
tests).
schlomo commented at 2018-11-20 12:46:¶
AFAIKT the only really relevant parts of that example configuration are
the REQUIRED_PROGS, COPY_AS_IS and POST_RECOVERY_SCRIPT variables.
The BACKUP_PROG_INCLUDES seems extremely specific as it mentions
MariaDB.
As it took me 3 times going over that example before I noticed those
relevant variables I can imagine that our users also will struggle with
that. Therefore I suggest to merge the REQUIRED_PROGS and COPY_AS_IS
fragments into the SLES specific configuration and to convert the
POST_RECOVERY_SCRIPT to a SLES specific script that checks if the
system is recent enough to need that treatment.
Wrong? Why? I'd like to know specifically.
@gsasidharan sorry for highjacking your issue with this general discussion.
gsasidharan commented at 2018-11-20 13:13:¶
Team..
pls share if you have the SLES12-SP3-PPC64.CONF file...
able to use rear 2.4 now and beleive the conf file is my issue now.
i use the default partitioning for filesystems .
my os version is SLES 12-SP3 for Power platform.
ThanksSasi.G
Sent from my Samsung Galaxy smartphone.
jsmeix commented at 2018-11-20 14:00:¶
@gsasidharan
I do not understand what you mean with
pls share if you have the SLES12-SP3-PPC64.CONF file
There is no SLES12-SP3-PPC64.CONF file in our ReaR upstream files,
e.g. what I get right now in a new git clone:
# git clone https://github.com/rear/rear.git # mv rear rear.github.master # cd /root/rear.github.master # find . | grep -i SLE ./usr/share/rear/conf/examples/SLE11-SLE12-SAP-HANA-UEFI-example.conf ./usr/share/rear/conf/examples/SLE12-SP2-btrfs-example.conf ./usr/share/rear/conf/examples/SLE11-ext3-example.conf ./usr/share/rear/conf/examples/SLE12-btrfs-example.conf ./usr/share/rear/conf/examples/SLE12-SP1-btrfs-example.conf
but I do not run SLES 12-SP3 on POWER platform.
gsasidharan commented at 2018-11-20 15:31:¶
Hi John,
for Sap Hana purposes we are using SLES V12SP3 in power platform.
this is not available below samples given.
Will try below suitable one and revert back.
thanksSasi.G
Sent from my Samsung Galaxy smartphone.
jsmeix commented at 2018-11-21 11:10:¶
@gsasidharan
in general regarding SAP HANA with SLES 12 SP3 on IBM Power
you may also have a look at
https://github.com/rear/rear/issues/1925
perhaps something therein could be also of interest for you.
hejharald commented at 2019-02-04 12:56:¶
I installed rear 2.4 on my Power E880, running SLES 12.3, and it allmost
restored flawless.
at first, I had the same errors with the disk layout.
During recover I had to edit /var/lib/rear/layout/disklayout.conf, exchanging all 'p1' with '-part1' and so on:
:g/p1/s//-part1/g
:g/p2/s//-part2/g
:g/p3/s//-part3/g
here's my rear dump:
Relax-and-Recover 2.4-git.3097.1e9aa97d.master.changed / 2018-08-31
Running rear dump (PID 124879)
Using log file: /var/log/rear/rear-dalbsrv.log.lockless
Dumping out configuration and system information
System definition:
ARCH = Linux-ppc64le
OS = GNU/Linux
OS_MASTER_VENDOR = SUSE
OS_MASTER_VERSION = 12
OS_MASTER_VENDOR_ARCH = SUSE/ppc64le
OS_MASTER_VENDOR_VERSION = SUSE/12
OS_MASTER_VENDOR_VERSION_ARCH = SUSE/12/ppc64le
OS_VENDOR = SUSE_LINUX
OS_VERSION = 12.3
OS_VENDOR_ARCH = SUSE_LINUX/ppc64le
OS_VENDOR_VERSION = SUSE_LINUX/12.3
OS_VENDOR_VERSION_ARCH = SUSE_LINUX/12.3/ppc64le
Configuration tree:
Linux-ppc64le.conf : OK
GNU/Linux.conf : OK
SUSE.conf : missing/empty
SUSE/ppc64le.conf : missing/empty
SUSE/12.conf : missing/empty
SUSE/12/ppc64le.conf : missing/empty
SUSE_LINUX.conf : OK
SUSE_LINUX/ppc64le.conf : missing/empty
SUSE_LINUX/12.3.conf : missing/empty
SUSE_LINUX/12.3/ppc64le.conf : missing/empty
site.conf : missing/empty
local.conf : OK
Backup with NETFS
NETFS_KEEP_OLD_BACKUP_COPY =
NETFS_PREFIX = dalbsrv
NETFS_RESTORE_CAPABILITIES = No
BACKUP_DUPLICITY_NAME = rear-backup
BACKUP_INTEGRITY_CHECK =
BACKUP_MOUNTCMD =
BACKUP_ONLY_EXCLUDE = no
BACKUP_ONLY_INCLUDE = no
BACKUP_OPTIONS =
BACKUP_RESTORE_MOVE_AWAY_DIRECTORY = /var/lib/rear/moved_away_after_backup_restore/
BACKUP_RESTORE_MOVE_AWAY_FILES = /boot/grub/grubenv /boot/grub2/grubenv
BACKUP_RSYNC_OPTIONS = --sparse --archive --hard-links --numeric-ids --stats
BACKUP_SELINUX_DISABLE = 1
BACKUP_TYPE =
BACKUP_UMOUNTCMD =
BACKUP_URL = file:///usr/common/rear-backup
Backup program is 'tar':
BACKUP_PROG = tar
BACKUP_PROG_ARCHIVE = backup
BACKUP_PROG_COMPRESS_OPTIONS = --gzip
BACKUP_PROG_COMPRESS_SUFFIX = .gz
BACKUP_PROG_CRYPT_ENABLED = 0
BACKUP_PROG_CRYPT_KEY =
BACKUP_PROG_CRYPT_OPTIONS = /usr/bin/openssl des3 -salt -k
BACKUP_PROG_DECRYPT_OPTIONS = /usr/bin/openssl des3 -d -k
BACKUP_PROG_EXCLUDE = /tmp/* /dev/shm/* /var/lib/rear/output/*
BACKUP_PROG_INCLUDE =
BACKUP_PROG_OPTIONS = --anchored
BACKUP_PROG_OPTIONS_CREATE_ARCHIVE =
BACKUP_PROG_OPTIONS_RESTORE_ARCHIVE =
BACKUP_PROG_SUFFIX = .tar
BACKUP_PROG_WARN_PARTIAL_TRANSFER = 1
Output to ISO
ISO_DEFAULT = boothd
ISO_DIR = /var/lib/rear/output
ISO_FILES =
ISO_ISOLINUX_BIN =
ISO_MAX_SIZE =
ISO_MKISOFS_BIN = /usr/bin/xorrisofs
ISO_PREFIX = rear-dalbsrv
ISO_RECOVER_MODE =
ISO_VOLID = RELAXRECOVER
OUTPUT_MOUNTCMD =
OUTPUT_OPTIONS =
OUTPUT_PREFIX = dalbsrv
OUTPUT_PREFIX_PXE =
OUTPUT_UMOUNTCMD =
OUTPUT_URL = file:///usr/common/rear-backup
RESULT_MAILTO =
jsmeix commented at 2019-02-12 12:45:¶
@hejharald
your
had to edit /var/lib/rear/layout/disklayout.conf,
exchanging all 'p1' with '-part1'
reminds me of things like
https://github.com/rear/rear/issues/1796#issuecomment-386632589
and the resulting
https://github.com/rear/rear/pull/1802
which is included in the ReaR 2.4 release
so that this issue here could be a new kind of
"multipath partition naming convention" issue.
For a detailed analysis we need
- your
var/lib/rear/layout/disklayout.conffile - the ReaR log file with debug output of
rear -D mkrescue - the content of your
/etc/multipath.conffile - the output of
ls -l /dev/mapper/
See also
https://github.com/rear/rear/issues/1796#issuecomment-386996844
what the usual multipath partition naming in SLES12 should be
and subsequent comments about kpartx and storix in particular
https://github.com/rear/rear/issues/1796#issuecomment-387461461
and storix' evil udev rule /etc/udev/rules.d/99-storixmpath.rules
that changed multipath partition naming in unexpeced ways.
Perhaps in your case there is also something unexpected
that changed the usual SLES12 multipath partition naming rules
in unexpeced ways?
github-actions commented at 2020-06-28 01:33:¶
Stale issue message
[Export of Github issue for rear/rear.]