#1973 Issue closed: ReaR 2.00 not working for SLES 12 SP3 on PowerVM LPAR with MPIO SAN device

Labels: waiting for info, support / question, no-issue-activity

gsasidharan opened issue at 2018-11-20 09:14:

Relax-and-Recover (ReaR) Issue Template

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

  • ReaR version ("/usr/sbin/rear -V"): [root@rheltest ~]# rear -V
    Relax-and-Recover 2.00 / Git

  • OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"): SUSE 12 SP3

  • ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

  • Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR): PowerVM LPAR

  • System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):ppc64

  • Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):

  • Storage (lokal disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):MPIO SAN device

  • Description of the issue (ideally so that others can reproduce it):
    when rear recovery is initiated exiting either with NFS issue or GRUB issues..
    No code has been generated to recreate swap:/dev/mapper/20017380035360016-part2 (swap).
    To recreate it manually add code to /var/lib/rear/layout/diskrestore.sh or abort.
    Manually add code that recreates swap:/dev/mapper/20017380035360016-part2 (swap)

  • Workaround, if any: NO,failed multiple times

  • Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):

RESCUE linux-yb9c:~ # rear recover
Relax-and-Recover 2.3 / 2017-12-20
Using log file: /var/log/rear/rear-linux-yb9c.log
Running workflow recover within the ReaR rescue/recovery system
Starting required daemons for NFS: RPC portmapper (portmap or rpcbind) and rpc.statd if available.
Started RPC portmapper 'rpcbind'.
RPC portmapper 'rpcbind' available.
Started rpc.statd.
RPC status rpc.statd available.
Starting rpc.idmapd failed.
Using backup archive '/tmp/rear.JG1gpxKHl4TeKTV/outputfs/linux-yb9c/backup.tar.gz'
Will do driver migration (recreating initramfs/initrd)
Calculating backup archive size
Backup archive size is 40K      /tmp/rear.JG1gpxKHl4TeKTV/outputfs/linux-yb9c/backup.tar.gz (compressed)
Comparing disks
Disk configuration looks identical
Proceed with recovery (yes) otherwise manual disk layout configuration is enforced
(default 'yes' timeout 30 seconds)
User confirmed to proceed with recovery
No code has been generated to recreate swap:/dev/mapper/20017380035360016-part2 (swap).
    To recreate it manually add code to /var/lib/rear/layout/diskrestore.sh or abort.
Manually add code that recreates swap:/dev/mapper/20017380035360016-part2 (swap)
1) View /var/lib/rear/layout/diskrestore.sh
2) Edit /var/lib/rear/layout/diskrestore.sh
3) Go to Relax-and-Recover shell
4) Continue 'rear recover'
5) Abort 'rear recover'
(default '4' timeout 300 seconds)
Start system layout restoration.
Disk layout created.
ERROR: No filesystem mounted on '/mnt/local'. Stopping.
Aborting due to an error, check /var/log/rear/rear-linux-yb9c.log for details

jsmeix commented at 2018-11-20 10:33:

what info is missing here and in general see

On a first glance

Backup archive size is 40K ...

does not look promising...

jsmeix commented at 2018-11-20 10:37:

because it happens on PowerVM LPAR with MPIO SAN device
perhaps you might be able to guess what happened here
(even without details).

jsmeix commented at 2018-11-20 10:43:

you use Relax-and-Recover 2.00 which is a bit old, from January 2017, cf.

Try out if a more current version works better for you, cf.

what has changed from ReaR 2.0 to the current ReaR 2.4.
There are some possibly backward incompatible changes.

I would also recommend to try out our current
ReaR upstream GitHub master code as follows:

Basically "git clone" our current ReaR upstream GitHub master code
into a separated directory and then configure and run ReaR
from within that directory like:

# git clone https://github.com/rear/rear.git

# mv rear rear.github.master

# cd rear.github.master

# vi etc/rear/local.conf

# usr/sbin/rear -D mkbackup

Note the relative paths "etc/rear/" and "usr/sbin/".

I recommend to try out our current ReaR upstream GitHub master code
because that is the only place where we fix bugs - i.e. bugs in older
ReaR versions are not fixed by us (i.e. by ReaR upstream).

If the issue also happens with current ReaR upstream GitHub master code
please provide us complete ReaR debug log files of both
"rear -D mkbackup" and then "rear -D recover"
plus your 'var/lib/rear/layout/disklayout.conf' file
so that we can have a look how it behaves in your particular environment
cf. "Debugging issues with Relax-and-Recover" at

If it works with the current ReaR 2.4 or our current ReaR upstream GitHub
master code we would really appreciate an explicit positive feedback.

For SUSE Linux Enterprise there should be official "rear23a" RPM packages
that contain ReaR upstream version 2.4 plus some later ReaR upstream commits,
see the section "rear / rear116 / rear1172a / rear118a / rear23a" in

gsasidharan commented at 2018-11-20 11:27:

Thanks for your quick response...
already tried version 2.4 and got failures for SUSE.
RHEL worked perfectly fine.
will retry with below steps and confirm the status.
Sent from my Samsung Galaxy smartphone.

jsmeix commented at 2018-11-20 11:54:

When things fail for SLES but work for RHEL my blind guess is
that you use SLES with its default btrfs structure which requires
special settings in your /etc/rear/local.conf for btrfs.

When you use SLES with its default btrfs structure
use a well matching *.btrfs-example.conf file from
/usr/share/rear/conf/examples/ as a starting point,
cf. the ones in our current ReaR upstream master code
and read
because for SLE12 there are several different
incompatible btrfs default setups where each one
has its own specific *.btrfs-example.conf file.

schlomo commented at 2018-11-20 12:11:

@jsmeix looking at those examples I wonder why we don't include the snapper and other tools by default? Wouldn't that make ReaR work on all SLES versions without "special" configuration?

jsmeix commented at 2018-11-20 12:36:

as far as I see SLES12 with its several different incompatible default btrfs structures
will always require an experienced admin who knows what specific kind of
SLES12 btrfs structure there actually is on his particular system
so that he can specify the right things in his etc/rear/local.conf
that match his particular kind of SLES12 btrfs structure.

Of course in theory with unreasonable effort it might be even possible
to let ReaR autodetermine what specific kind of SLES12 btrfs structure
there actually is on a particular system but at least I won't implement that
unless I am forced by SUSE who is convinced by one or more customers
who pay us for all those efforts ;-)

I think in practice it is more reasonable to leave it as it is now
in particular because at least for now it seems things have reasonably
stabilized since SLE12-SP2 because I am not aware of any new different
incompatible default btrfs structures after SLE12-SP2.

SLE12-SP2-btrfs-example.conf also works on SLE12-SP3 and SLE12-SP4
and it should also work on SLE15 (at least it worked for me for my tests).

schlomo commented at 2018-11-20 12:46:

AFAIKT the only really relevant parts of that example configuration are the REQUIRED_PROGS, COPY_AS_IS and POST_RECOVERY_SCRIPT variables. The BACKUP_PROG_INCLUDES seems extremely specific as it mentions MariaDB.

As it took me 3 times going over that example before I noticed those relevant variables I can imagine that our users also will struggle with that. Therefore I suggest to merge the REQUIRED_PROGS and COPY_AS_IS fragments into the SLES specific configuration and to convert the POST_RECOVERY_SCRIPT to a SLES specific script that checks if the system is recent enough to need that treatment.

Wrong? Why? I'd like to know specifically.

@gsasidharan sorry for highjacking your issue with this general discussion.

gsasidharan commented at 2018-11-20 13:13:

pls share if you have the SLES12-SP3-PPC64.CONF file...
able to use rear 2.4 now and beleive the conf file is my issue now.
i use the default partitioning for filesystems .
my os version is SLES 12-SP3 for Power platform.
Sent from my Samsung Galaxy smartphone.

jsmeix commented at 2018-11-20 14:00:

I do not understand what you mean with
pls share if you have the SLES12-SP3-PPC64.CONF file

There is no SLES12-SP3-PPC64.CONF file in our ReaR upstream files,
e.g. what I get right now in a new git clone:

# git clone https://github.com/rear/rear.git

# mv rear rear.github.master

# cd /root/rear.github.master

# find . | grep -i SLE

but I do not run SLES 12-SP3 on POWER platform.

gsasidharan commented at 2018-11-20 15:31:

Hi John,
for Sap Hana purposes we are using SLES V12SP3 in power platform.
this is not available below samples given.
Will try below suitable one and revert back.
Sent from my Samsung Galaxy smartphone.

jsmeix commented at 2018-11-21 11:10:

in general regarding SAP HANA with SLES 12 SP3 on IBM Power
you may also have a look at https://github.com/rear/rear/issues/1925
perhaps something therein could be also of interest for you.

hejharald commented at 2019-02-04 12:56:

I installed rear 2.4 on my Power E880, running SLES 12.3, and it allmost restored flawless.
at first, I had the same errors with the disk layout.

During recover I had to edit /var/lib/rear/layout/disklayout.conf, exchanging all 'p1' with '-part1' and so on:


here's my rear dump:

Relax-and-Recover 2.4-git.3097.1e9aa97d.master.changed / 2018-08-31
Running rear dump (PID 124879)
Using log file: /var/log/rear/rear-dalbsrv.log.lockless
Dumping out configuration and system information
System definition:
                                    ARCH = Linux-ppc64le
                                      OS = GNU/Linux
                        OS_MASTER_VENDOR = SUSE
                       OS_MASTER_VERSION = 12
                   OS_MASTER_VENDOR_ARCH = SUSE/ppc64le
                OS_MASTER_VENDOR_VERSION = SUSE/12
           OS_MASTER_VENDOR_VERSION_ARCH = SUSE/12/ppc64le
                               OS_VENDOR = SUSE_LINUX
                              OS_VERSION = 12.3
                          OS_VENDOR_ARCH = SUSE_LINUX/ppc64le
                       OS_VENDOR_VERSION = SUSE_LINUX/12.3
                  OS_VENDOR_VERSION_ARCH = SUSE_LINUX/12.3/ppc64le
Configuration tree:
                      Linux-ppc64le.conf : OK
                          GNU/Linux.conf : OK
                               SUSE.conf : missing/empty
                       SUSE/ppc64le.conf : missing/empty
                            SUSE/12.conf : missing/empty
                    SUSE/12/ppc64le.conf : missing/empty
                         SUSE_LINUX.conf : OK
                 SUSE_LINUX/ppc64le.conf : missing/empty
                    SUSE_LINUX/12.3.conf : missing/empty
            SUSE_LINUX/12.3/ppc64le.conf : missing/empty
                               site.conf : missing/empty
                              local.conf : OK
Backup with NETFS
                            NETFS_PREFIX = dalbsrv
                   BACKUP_DUPLICITY_NAME = rear-backup
                  BACKUP_INTEGRITY_CHECK =
                         BACKUP_MOUNTCMD =
                     BACKUP_ONLY_EXCLUDE = no
                     BACKUP_ONLY_INCLUDE = no
                          BACKUP_OPTIONS =
      BACKUP_RESTORE_MOVE_AWAY_DIRECTORY = /var/lib/rear/moved_away_after_backup_restore/
          BACKUP_RESTORE_MOVE_AWAY_FILES = /boot/grub/grubenv /boot/grub2/grubenv
                    BACKUP_RSYNC_OPTIONS = --sparse --archive --hard-links --numeric-ids --stats
                  BACKUP_SELINUX_DISABLE = 1
                             BACKUP_TYPE =
                        BACKUP_UMOUNTCMD =
                              BACKUP_URL = file:///usr/common/rear-backup
Backup program is 'tar':
                             BACKUP_PROG = tar
                     BACKUP_PROG_ARCHIVE = backup
               BACKUP_PROG_CRYPT_ENABLED = 0
                   BACKUP_PROG_CRYPT_KEY =
               BACKUP_PROG_CRYPT_OPTIONS = /usr/bin/openssl des3 -salt -k
             BACKUP_PROG_DECRYPT_OPTIONS = /usr/bin/openssl des3 -d -k
                     BACKUP_PROG_EXCLUDE = /tmp/* /dev/shm/* /var/lib/rear/output/*
                     BACKUP_PROG_INCLUDE =
                     BACKUP_PROG_OPTIONS = --anchored
                      BACKUP_PROG_SUFFIX = .tar
Output to ISO
                             ISO_DEFAULT = boothd
                                 ISO_DIR = /var/lib/rear/output
                               ISO_FILES =
                        ISO_ISOLINUX_BIN =
                            ISO_MAX_SIZE =
                         ISO_MKISOFS_BIN = /usr/bin/xorrisofs
                              ISO_PREFIX = rear-dalbsrv
                        ISO_RECOVER_MODE =
                               ISO_VOLID = RELAXRECOVER
                         OUTPUT_MOUNTCMD =
                          OUTPUT_OPTIONS =
                           OUTPUT_PREFIX = dalbsrv
                       OUTPUT_PREFIX_PXE =
                        OUTPUT_UMOUNTCMD =
                              OUTPUT_URL = file:///usr/common/rear-backup
                           RESULT_MAILTO =

jsmeix commented at 2019-02-12 12:45:


had to edit /var/lib/rear/layout/disklayout.conf,
exchanging all 'p1' with '-part1'

reminds me of things like
and the resulting
which is included in the ReaR 2.4 release
so that this issue here could be a new kind of
"multipath partition naming convention" issue.

For a detailed analysis we need

  • your var/lib/rear/layout/disklayout.conf file
  • the ReaR log file with debug output of rear -D mkrescue
  • the content of your /etc/multipath.conf file
  • the output of ls -l /dev/mapper/

See also
what the usual multipath partition naming in SLES12 should be
and subsequent comments about kpartx and storix in particular
and storix' evil udev rule /etc/udev/rules.d/99-storixmpath.rules
that changed multipath partition naming in unexpeced ways.

Perhaps in your case there is also something unexpected
that changed the usual SLES12 multipath partition naming rules
in unexpeced ways?

github-actions commented at 2020-06-28 01:33:

Stale issue message

[Export of Github issue for rear/rear.]