#2019 Issue closed: Update to closed issue #2016 (Unable to clone Power lpar from rear backup SLES 12)

Labels: bug, support / question, special hardware or VM, no-issue-activity

dewagner1 opened issue at 2019-01-17 21:59:

Relax-and-Recover (ReaR) Issue Template

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

Please refer to #2016 for this information
Rear Problem.txt

  • ReaR version ("/usr/sbin/rear -V"):

  • OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"):

  • ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

  • Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR):

  • System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):

  • Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):

  • Storage (lokal disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):

  • Description of the issue (ideally so that others can reproduce it):
    There was an issue with SAN zoning not being unique across all adapters. Now those issues have been resolved and we are still having problems cloning lpars. Most of the time the old lpar disk is mapped to /dev/sda. However if we try rebooting and running rear recover multiple times, it works occasionally. When it fails to map properly, the output shows no multipath device being found.

  • Workaround, if any:

  • Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
    Rear Problem.txt

schabrolles commented at 2019-01-18 08:35:

@dewagner1,

My first guess is still a SAN / FC issue. To validate, the best would be to boot your LPAR in SMS and run ioinfo via the OK prompt.

1- From SMS menu to OK prompt
Press 0 when entering SMS menu 1st page.

                                             X = eXit System Management Services
 -------------------------------------------------------------------------------
 Type menu item number and press Enter or select Navigation key:
 Open Completed.

 PowerPC Firmware
 Version FW860.51 (SV860_165)
 SMS (c) Copyright IBM Corp. 2000,2016 All rights reserved.
-------------------------------------------------------------------------------
 Main Menu
 1.   Select Language
 2.   Setup Remote IPL (Initial Program Load)
 3.   I/O Device Information
 4.   Select Console
 5.   Select Boot Options









 -------------------------------------------------------------------------------
 Navigation Keys:

                                             X = eXit System Management Services
 -------------------------------------------------------------------------------
 Type menu item number and press Enter or select Navigation key:0









                    .--------------------------------------.
                    |  Exit SMS to OK> Prompt? (Y)es/(N)o  |
                    `--------------------------------------'












 ok
0 >

2-Run ioinfo on each fiber adapter to get the list of disk/partitions path over the SAN network

 ok
0 > ioinfo

!!! IOINFO: FOR IBM INTERNAL USE ONLY !!!
This tool gives you information about SCSI,SATA,SAS,and USB devices attached to the system

Select a tool from the following

 1. SATAINFO
 2. SASINFO
 3. USBINFO
 4. FCINFO
 5. VSCSIINFO

q - quit/exit

==> 4


FCINFO Main Menu
Select a FC Node from the following list:
  #  Location Code                Pathname
---------------------------------------------------------------
  1. U8284.22A.217097V-V21-C13-T1   /vdevice/vfc-client@3000000d
  2. U8284.22A.217097V-V21-C14-T1   /vdevice/vfc-client@3000000e

  q - Quit/Exit

==> 1

Press 1 to list devices seen by the first Adapter

==> 1

1. 500507680c21099f,0                - 51200 MB Disk drive (bootable)
2. 500507680c21099f,1000000000000    - 10240 MB Disk drive
3. 500507680c21099f,2000000000000    - 10240 MB Disk drive
4. 500507680c51099f,0                - 51200 MB Disk drive (bootable)
5. 500507680c51099f,1000000000000    - 10240 MB Disk drive
6. 500507680c51099f,2000000000000    - 10240 MB Disk drive
7. 500507680c2109a6,0                - 51200 MB Disk drive (bootable)
8. 500507680c2109a6,1000000000000    - 10240 MB Disk drive
9. 500507680c2109a6,2000000000000    - 10240 MB Disk drive
10. 500507680c5109a6,0                - 51200 MB Disk drive (bootable)
11. 500507680c5109a6,1000000000000    - 10240 MB Disk drive
12. 500507680c5109a6,2000000000000    - 10240 MB Disk drive

Hit a key to continue...

Then Quit q and run the same thing on the other adapter

dewagner1 commented at 2019-01-19 12:25:

Hi Sebastien,
We are using XIV for our storage. We worked with the SAN team yesterday
and thought we found the problem. They made a change and we tried a
restore and it worked perfectly. Then we rebooted and tried two more
restores on the same lpar with the same image and they failed.

The first restore that worked had the following output for the multipath
device info. I don't have console output so I don't remember the
exact /dev/mapper id. That's why I used question marks.
The two times that didn't work (logs below including ioinfo output) did not
have any entry under "Listing multipath device found"

Listing multipath device found
/dev/mapper/20017380034??????
Comparing disks
Ambiguous possible target disks need manual configuration (more than one
with same size found)
Switching to manual disk layout configuration
Using /dev/sda (same size) for recreating /dev/mapper/200173800348802b3
Current disk mapping table (source => target):
/dev/mapper/200173800348802b3 => /dev/mapper/20017380034??????

Here is the ioinfo output

0 > ioinfo

!!! IOINFO: FOR IBM INTERNAL USE ONLY !!!
This tool gives you information about SCSI,SATA,SAS,and USB devices
attached to the system

Select a tool from the following

  1. SATAINFO
  2. SASINFO
  3. USBINFO
  4. FCINFO
  5. VSCSIINFO

q - quit/exit

==> 4

FCINFO Main Menu
Select a FC Node from the following list:

Location Code Pathname

  1. U9080.MHE.78F42D7-V7-C2-T1 /vdevice/vfc-client@30000002
  2. U9080.MHE.78F42D7-V7-C3-T1 /vdevice/vfc-client@30000003
  3. U9080.MHE.78F42D7-V7-C4-T1 /vdevice/vfc-client@30000004
  4. U9080.MHE.78F42D7-V7-C5-T1 /vdevice/vfc-client@30000005

q - Quit/Exit

==> 1

FC Node Menu FC Node String: /vdevice/vfc-client@30000002 FC Node WorldWidePortName: c0507609f1e20020

  1. List Attached FC Devices
  2. Select a FC Device
  3. Enable/Disable FC Adapter Debug flags

q - Quit/Exit

==> 1

  1. 5001738034880150,4000000000000000 - 114877 MB Disk drive
  2. 5001738034880141,4000000000000000 - 114877 MB Disk drive

Hit a key to continue...

FC Node Menu FC Node String: /vdevice/vfc-client@30000002 FC Node WorldWidePortName: c0507609f1e20020

  1. List Attached FC Devices
  2. Select a FC Device
  3. Enable/Disable FC Adapter Debug flags

q - Quit/Exit

==> q

FCINFO Main Menu
Select a FC Node from the following list:

Location Code Pathname

  1. U9080.MHE.78F42D7-V7-C2-T1 /vdevice/vfc-client@30000002
  2. U9080.MHE.78F42D7-V7-C3-T1 /vdevice/vfc-client@30000003
  3. U9080.MHE.78F42D7-V7-C4-T1 /vdevice/vfc-client@30000004
  4. U9080.MHE.78F42D7-V7-C5-T1 /vdevice/vfc-client@30000005

q - Quit/Exit

==> 2

FC Node Menu FC Node String: /vdevice/vfc-client@30000003 FC Node WorldWidePortName: c0507609f1e20022

  1. List Attached FC Devices
  2. Select a FC Device
  3. Enable/Disable FC Adapter Debug flags

q - Quit/Exit

==> 1

NO FC DEVICES FOUND

Hit a key to continue...

FC Node Menu FC Node String: /vdevice/vfc-client@30000003 FC Node WorldWidePortName: c0507609f1e20022

  1. List Attached FC Devices
  2. Select a FC Device
  3. Enable/Disable FC Adapter Debug flags

q - Quit/Exit

==> q

FCINFO Main Menu
Select a FC Node from the following list:

Location Code Pathname

  1. U9080.MHE.78F42D7-V7-C2-T1 /vdevice/vfc-client@30000002
  2. U9080.MHE.78F42D7-V7-C3-T1 /vdevice/vfc-client@30000003
  3. U9080.MHE.78F42D7-V7-C4-T1 /vdevice/vfc-client@30000004
  4. U9080.MHE.78F42D7-V7-C5-T1 /vdevice/vfc-client@30000005

q - Quit/Exit

==> 3

FC Node Menu FC Node String: /vdevice/vfc-client@30000004 FC Node WorldWidePortName: c0507609f1e20024

  1. List Attached FC Devices
  2. Select a FC Device
  3. Enable/Disable FC Adapter Debug flags

q - Quit/Exit

==> 1

NO FC DEVICES FOUND

Hit a key to continue...

FC Node Menu FC Node String: /vdevice/vfc-client@30000004 FC Node WorldWidePortName: c0507609f1e20024

  1. List Attached FC Devices
  2. Select a FC Device
  3. Enable/Disable FC Adapter Debug flags

q - Quit/Exit

==> q

FCINFO Main Menu
Select a FC Node from the following list:

Location Code Pathname

  1. U9080.MHE.78F42D7-V7-C2-T1 /vdevice/vfc-client@30000002
  2. U9080.MHE.78F42D7-V7-C3-T1 /vdevice/vfc-client@30000003
  3. U9080.MHE.78F42D7-V7-C4-T1 /vdevice/vfc-client@30000004
  4. U9080.MHE.78F42D7-V7-C5-T1 /vdevice/vfc-client@30000005

q - Quit/Exit

==> 4

FC Node Menu FC Node String: /vdevice/vfc-client@30000005 FC Node WorldWidePortName: c0507609f1e20026

  1. List Attached FC Devices
  2. Select a FC Device
  3. Enable/Disable FC Adapter Debug flags

q - Quit/Exit

==> 1

NO FC DEVICES FOUND

Hit a key to continue...

FC Node Menu FC Node String: /vdevice/vfc-client@30000005 FC Node WorldWidePortName: c0507609f1e20026

  1. List Attached FC Devices
  2. Select a FC Device
  3. Enable/Disable FC Adapter Debug flags

q - Quit/Exit

==> q

FCINFO Main Menu
Select a FC Node from the following list:

Location Code Pathname

  1. U9080.MHE.78F42D7-V7-C2-T1 /vdevice/vfc-client@30000002
  2. U9080.MHE.78F42D7-V7-C3-T1 /vdevice/vfc-client@30000003
  3. U9080.MHE.78F42D7-V7-C4-T1 /vdevice/vfc-client@30000004
  4. U9080.MHE.78F42D7-V7-C5-T1 /vdevice/vfc-client@30000005

q - Quit/Exit

Here is the console log of the two different restores that did not work.

Configuring Relax-and-Recover rescue system

Running 00-functions.sh...
Running 01-run-ldconfig.sh...
Running 10-console-setup.sh...
Using keymap of the original system
Running 20-check-boot-options.sh...
Running 40-start-udev-or-load-modules.sh...
Waiting for udev ... done.
Running 41-load-special-modules.sh...
Running 42-engage-scsi.sh...
Running 45-serial-console.sh...
Running 55-migrate-network-devices.sh...
No network interface mapping is specified in /etc/rear/mappings/mac
The original network interface eth0 d6:8b:e6:36:62:08 is not available

  1. eth3 d6:8b:e5:4d:db:08 ibmveth
  2. eth4 d6:8b:e5:4d:db:09 ibmveth
  3. eth5 d6:8b:e5:4d:db:0a ibmveth
  4. Skip replacing eth0 d6:8b:e6:36:62:08
    Choose replacement for eth0 d6:8b:e6:36:62:08 1
    Mapping eth0 d6:8b:e6:36:62:08 to eth3 d6:8b:e5:4d:db:08
    The original network interface eth1 d6:8b:e6:36:62:09 is not available
    1. eth4 d6:8b:e5:4d:db:09 ibmveth
  6. eth5 d6:8b:e5:4d:db:0a ibmveth
  7. Skip replacing eth1 d6:8b:e6:36:62:09
    Choose replacement for eth1 d6:8b:e6:36:62:09 2
    Mapping eth1 d6:8b:e6:36:62:09 to eth4 d6:8b:e5:4d:db:09
    The original network interface eth2 d6:8b:e6:36:62:0a is not available
      1. eth5 d6:8b:e5:4d:db:0a ibmveth
  9. Skip replacing eth2 d6:8b:e6:36:62:0a
    Choose replacement for eth2 d6:8b:e6:36:62:0a 3
    Mapping eth2 d6:8b:e6:36:62:0a to eth5 d6:8b:e5:4d:db:0a
    Reloading udev ... done.
    Running 58-start-dhclient.sh...
    Running 60-network-devices.sh...
    Running 62-routing.sh...
    Running 65-sysctl.sh...
    net.ipv6.conf.all.disable_ipv6 = 1
    net.ipv4.ip_forward = 1
    net.ipv6.conf.all.forwarding = 0
    vm.pagecache_limit_ignore_dirty = 2
    net.core.somaxconn = 4096
    net.ipv4.tcp_max_syn_backlog = 8192
    net.ipv4.tcp_tw_reuse = 1
    net.ipv4.tcp_tw_recycle = 1
    net.ipv4.tcp_syn_retries = 8
    net.ipv4.tcp_wmem = 4096 16384 4194304
    vm.min_free_kbytes = 10485760
    net.ipv6.conf.all.disable_ipv6 = 1
    net.ipv4.ip_forward = 1
    net.ipv6.conf.all.forwarding = 0
    vm.pagecache_limit_ignore_dirty = 2
    net.core.somaxconn = 4096
    net.ipv4.tcp_max_syn_backlog = 8192
    net.ipv4.tcp_tw_reuse = 1
    net.ipv4.tcp_tw_recycle = 1
    net.ipv4.tcp_syn_retries = 8
    net.ipv4.tcp_wmem = 4096 16384 4194304
    vm.min_free_kbytes = 10485760
    Running 99-makedev.sh...

Relax-and-Recover rescue system is ready

[ OK ] Started Initialize Rescue System.
[ OK ] Started Getty on tty0.
[ OK ] Reached target Login Prompts.
[ OK ] Started Serial Getty on hvc0.
[ OK ] Started Serial Getty on ttyS0.
[ OK ] Reached target Multi-User.

Relax-and-Recover 2.4-git.0.6ec9075.unknown / 2018-12-05

Relax-and-Recover comes with ABSOLUTELY NO WARRANTY; for details see
the GNU General Public License at: http://www.gnu.org/licenses/gpl.html

Host eniesdbp104p using Backup NETFS and Output ISO
Build date: Thu, 17 Jan 2019 14:22:20 -0500

Welcome to SUSE Linux Enterprise Server for SAP Applications 12 SP3
(ppc64le) - Kernel 4.4.162-94.72-default (hvc0).

SSH fingerprint: 2048 SHA256:bn+LTuNXpbRAhjgEBjQZ6rZv29K8anvt2wL7R2uXgqw
root@eniesdbp104p (RSA)

eniesdbp104p login: root

Welcome to Relax-and-Recover. Run "rear recover" to restore your system !

RESCUE eniesdbp104p:~ # rear recover
Relax-and-Recover 2.4-git.0.6ec9075.unknown / 2018-12-05
Running rear recover (PID 24570)
Using log file: /var/log/rear/rear-eniesdbp104p.log
Running workflow recover within the ReaR rescue/recovery system
Using backup archive
'/tmp/rear.Hs00F3ALINE1ns3/outputfs/iso_fs/REAR_BACKUP/backup.tar.gz'
Will do driver migration (recreating initramfs/initrd)
Calculating backup archive size
Backup archive size is 1909.38M (compressed)
Setting up multipathing
Activating multipath
multipath activated
Starting multipath daemon
multipathd started
Listing multipath device found

Comparing disks
Ambiguous possible target disks need manual configuration (more than one
with same size found)
Switching to manual disk layout configuration
Using /dev/sda (same size) for recreating /dev/mapper/200173800348802b3
Current disk mapping table (source => target):
/dev/mapper/200173800348802b3 => /dev/sda

Configuring Relax-and-Recover rescue system

Running 00-functions.sh...
Running 01-run-ldconfig.sh...
Running 10-console-setup.sh...
Using keymap of the original system
Running 20-check-boot-options.sh...
Running 40-start-udev-or-load-modules.sh...
Waiting for udev ... done.
Running 41-load-special-modules.sh...
Running 42-engage-scsi.sh...
Running 45-serial-console.sh...
Running 55-migrate-network-devices.sh...
No network interface mapping is specified in /etc/rear/mappings/mac
The original network interface eth0 d6:8b:e6:36:62:08 is not available

  1. eth1 d6:8b:e5:4d:db:09 ibmveth
  2. eth2 d6:8b:e5:4d:db:0a ibmveth
  3. eth3 d6:8b:e5:4d:db:08 ibmveth
  4. Skip replacing eth0 d6:8b:e6:36:62:08
    Choose replacement for eth0 d6:8b:e6:36:62:08 3
    Mapping eth0 d6:8b:e6:36:62:08 to eth3 d6:8b:e5:4d:db:08
    The original network interface eth1 d6:8b:e6:36:62:09 is not available
  5. eth1 d6:8b:e5:4d:db:09 ibmveth
  6. eth2 d6:8b:e5:4d:db:0a ibmveth
    1. Skip replacing eth1 d6:8b:e6:36:62:09
      Choose replacement for eth1 d6:8b:e6:36:62:09 1
      Mapping eth1 d6:8b:e6:36:62:09 to eth1 d6:8b:e5:4d:db:09
      The original network interface eth2 d6:8b:e6:36:62:0a is not available
    1. eth2 d6:8b:e5:4d:db:0a ibmveth
    1. Skip replacing eth2 d6:8b:e6:36:62:0a
      Choose replacement for eth2 d6:8b:e6:36:62:0a 2
      Mapping eth2 d6:8b:e6:36:62:0a to eth2 d6:8b:e5:4d:db:0a
      Reloading udev ... done.
      Running 58-start-dhclient.sh...
      Running 60-network-devices.sh...
      Running 62-routing.sh...
      Running 65-sysctl.sh...
      net.ipv6.conf.all.disable_ipv6 = 1
      net.ipv4.ip_forward = 1
      net.ipv6.conf.all.forwarding = 0
      vm.pagecache_limit_ignore_dirty = 2
      net.core.somaxconn = 4096
      net.ipv4.tcp_max_syn_backlog = 8192
      net.ipv4.tcp_tw_reuse = 1
      net.ipv4.tcp_tw_recycle = 1
      net.ipv4.tcp_syn_retries = 8
      net.ipv4.tcp_wmem = 4096 16384 4194304
      vm.min_free_kbytes = 10485760
      net.ipv6.conf.all.disable_ipv6 = 1
      net.ipv4.ip_forward = 1
      net.ipv6.conf.all.forwarding = 0
      vm.pagecache_limit_ignore_dirty = 2
      net.core.somaxconn = 4096
      net.ipv4.tcp_max_syn_backlog = 8192
      net.ipv4.tcp_tw_reuse = 1
      net.ipv4.tcp_tw_recycle = 1
      net.ipv4.tcp_syn_retries = 8
      net.ipv4.tcp_wmem = 4096 16384 4194304
      vm.min_free_kbytes = 10485760
      Running 99-makedev.sh...

Relax-and-Recover rescue system is ready

[ OK ] Started Initialize Rescue System.
[ OK ] Started Getty on tty0.
[ OK ] Reached target Login Prompts.
[ OK ] Reached target Multi-User.
[ OK ] Started Serial Getty on ttyS0.
[ OK ] Started Serial Getty on hvc0.

Relax-and-Recover 2.4-git.0.6ec9075.unknown / 2018-12-05

Relax-and-Recover comes with ABSOLUTELY NO WARRANTY; for details see
the GNU General Public License at: http://www.gnu.org/licenses/gpl.html

Host eniesdbp104p using Backup NETFS and Output ISO
Build date: Thu, 17 Jan 2019 14:22:20 -0500

Welcome to SUSE Linux Enterprise Server for SAP Applications 12 SP3
(ppc64le) - Kernel 4.4.162-94.72-default (hvc0).

SSH fingerprint: 2048 SHA256:DNyikRzuULk3bpE1wwyoXX/6b9qV0kBUW0j32NOrdkY
root@eniesdbp104p (RSA)

eniesdbp104p login: root

Welcome to Relax-and-Recover. Run "rear recover" to restore your system !

RESCUE eniesdbp104p:~ # rear recover
Relax-and-Recover 2.4-git.0.6ec9075.unknown / 2018-12-05
Running rear recover (PID 24593)
Using log file: /var/log/rear/rear-eniesdbp104p.log
Running workflow recover within the ReaR rescue/recovery system
Using backup archive
'/tmp/rear.3HmcDWcPOuSWy4R/outputfs/iso_fs/REAR_BACKUP/backup.tar.gz'
Will do driver migration (recreating initramfs/initrd)
Calculating backup archive size
Backup archive size is 1909.38M (compressed)
Setting up multipathing
Activating multipath
multipath activated
Starting multipath daemon
multipathd started
Listing multipath device found
SIBM_2810XIV_780348802C6 dm-0 IBM,2810XIV size=112G
Comparing disks
Ambiguous possible target disks need manual configuration (more than one
with same size found)
Switching to manual disk layout configuration
Using /dev/mapper/SIBM_2810XIV_780348802C6 (same size) for
recreating /dev/mapper/200173800348802b3
Current disk mapping table (source => target):
/dev/mapper/200173800348802b3 => /dev/mapper/SIBM_2810XIV_780348802C6

Dan Wagner
Senior Enterprise System Specialist

(717) 580-3844 Mobile
(717) 409-5858 Office
dwagner1@us.ibm.com

IBM Services

dewagner1 commented at 2019-01-21 10:32:

Hi Sebastien,
We eliminated all but 1 path to 1 disk on the XIV and it still is not
working. Here is the console output along with the debug log file.

(See attached file:
IES_Build_SAP-HANA2_eniesdbp101p_REAR-debug__2019Jan18.txt)

Thanks,
Dan Wagner
Senior Enterprise System Specialist

(717) 580-3844 Mobile
(717) 409-5858 Office
dwagner1@us.ibm.com

IBM Services

From: Dan Wagner/Mechanicsburg/Contr/IBM
To: rear/rear <reply
+0283324a89c9c6b22b80d4fd3f55956ed4745071464822bd92cf00000001185951f092a169ce17dedc38@reply.github.com>
Cc: Mention mention@noreply.github.com, rear/rear
rear@noreply.github.com, Susan
Steele/Mechanicsburg/IBM@IBMUS, Glenn
Lehman/Mechanicsburg/IBM@IBMUS, Cora Jung/New York/IBM@IBMUS
Date: 01/19/2019 07:24 AM
Subject: Re: [rear/rear] Update to closed issue #2016 (Unable to clone
Power lpar from rear backup SLES 12 SP3) (#2019)

Hi Sebastien,
We are using XIV for our storage. We worked with the SAN team yesterday
and thought we found the problem. They made a change and we tried a
restore and it worked perfectly. Then we rebooted and tried two more
restores on the same lpar with the same image and they failed.

The first restore that worked had the following output for the multipath
device info. I don't have console output so I don't remember the
exact /dev/mapper id. That's why I used question marks.
The two times that didn't work (logs below including ioinfo output) did not
have any entry under "Listing multipath device found"

Listing multipath device found
/dev/mapper/20017380034??????
Comparing disks
Ambiguous possible target disks need manual configuration (more than one
with same size found)
Switching to manual disk layout configuration
Using /dev/sda (same size) for recreating /dev/mapper/200173800348802b3
Current disk mapping table (source => target):
/dev/mapper/200173800348802b3 => /dev/mapper/20017380034??????

Here is the ioinfo output

0 > ioinfo

!!! IOINFO: FOR IBM INTERNAL USE ONLY !!!
This tool gives you information about SCSI,SATA,SAS,and USB devices
attached to the system

Select a tool from the following

  1. SATAINFO
  2. SASINFO
  3. USBINFO
  4. FCINFO
  5. VSCSIINFO

q - quit/exit

==> 4

FCINFO Main Menu
Select a FC Node from the following list:

Location Code Pathname

  1. U9080.MHE.78F42D7-V7-C2-T1 /vdevice/vfc-client@30000002
  2. U9080.MHE.78F42D7-V7-C3-T1 /vdevice/vfc-client@30000003
  3. U9080.MHE.78F42D7-V7-C4-T1 /vdevice/vfc-client@30000004
  4. U9080.MHE.78F42D7-V7-C5-T1 /vdevice/vfc-client@30000005

q - Quit/Exit

==> 1

FC Node Menu FC Node String: /vdevice/vfc-client@30000002 FC Node WorldWidePortName: c0507609f1e20020

  1. List Attached FC Devices
  2. Select a FC Device
  3. Enable/Disable FC Adapter Debug flags

q - Quit/Exit

==> 1

  1. 5001738034880150,4000000000000000 - 114877 MB Disk drive
  2. 5001738034880141,4000000000000000 - 114877 MB Disk drive

Hit a key to continue...

FC Node Menu FC Node String: /vdevice/vfc-client@30000002 FC Node WorldWidePortName: c0507609f1e20020

  1. List Attached FC Devices
  2. Select a FC Device
  3. Enable/Disable FC Adapter Debug flags

q - Quit/Exit

==> q

FCINFO Main Menu
Select a FC Node from the following list:

Location Code Pathname

  1. U9080.MHE.78F42D7-V7-C2-T1 /vdevice/vfc-client@30000002
  2. U9080.MHE.78F42D7-V7-C3-T1 /vdevice/vfc-client@30000003
  3. U9080.MHE.78F42D7-V7-C4-T1 /vdevice/vfc-client@30000004
  4. U9080.MHE.78F42D7-V7-C5-T1 /vdevice/vfc-client@30000005

q - Quit/Exit

==> 2

FC Node Menu FC Node String: /vdevice/vfc-client@30000003 FC Node WorldWidePortName: c0507609f1e20022

  1. List Attached FC Devices
  2. Select a FC Device
  3. Enable/Disable FC Adapter Debug flags

q - Quit/Exit

==> 1

NO FC DEVICES FOUND

Hit a key to continue...

FC Node Menu FC Node String: /vdevice/vfc-client@30000003 FC Node WorldWidePortName: c0507609f1e20022

  1. List Attached FC Devices
  2. Select a FC Device
  3. Enable/Disable FC Adapter Debug flags

q - Quit/Exit

==> q

FCINFO Main Menu
Select a FC Node from the following list:

Location Code Pathname

  1. U9080.MHE.78F42D7-V7-C2-T1 /vdevice/vfc-client@30000002
  2. U9080.MHE.78F42D7-V7-C3-T1 /vdevice/vfc-client@30000003
  3. U9080.MHE.78F42D7-V7-C4-T1 /vdevice/vfc-client@30000004
  4. U9080.MHE.78F42D7-V7-C5-T1 /vdevice/vfc-client@30000005

q - Quit/Exit

==> 3

FC Node Menu FC Node String: /vdevice/vfc-client@30000004 FC Node WorldWidePortName: c0507609f1e20024

  1. List Attached FC Devices
  2. Select a FC Device
  3. Enable/Disable FC Adapter Debug flags

q - Quit/Exit

==> 1

NO FC DEVICES FOUND

Hit a key to continue...

FC Node Menu FC Node String: /vdevice/vfc-client@30000004 FC Node WorldWidePortName: c0507609f1e20024

  1. List Attached FC Devices
  2. Select a FC Device
  3. Enable/Disable FC Adapter Debug flags

q - Quit/Exit

==> q

FCINFO Main Menu
Select a FC Node from the following list:

Location Code Pathname

  1. U9080.MHE.78F42D7-V7-C2-T1 /vdevice/vfc-client@30000002
  2. U9080.MHE.78F42D7-V7-C3-T1 /vdevice/vfc-client@30000003
  3. U9080.MHE.78F42D7-V7-C4-T1 /vdevice/vfc-client@30000004
  4. U9080.MHE.78F42D7-V7-C5-T1 /vdevice/vfc-client@30000005

q - Quit/Exit

==> 4

FC Node Menu FC Node String: /vdevice/vfc-client@30000005 FC Node WorldWidePortName: c0507609f1e20026

  1. List Attached FC Devices
  2. Select a FC Device
  3. Enable/Disable FC Adapter Debug flags

q - Quit/Exit

==> 1

NO FC DEVICES FOUND

Hit a key to continue...

FC Node Menu FC Node String: /vdevice/vfc-client@30000005 FC Node WorldWidePortName: c0507609f1e20026

  1. List Attached FC Devices
  2. Select a FC Device
  3. Enable/Disable FC Adapter Debug flags

q - Quit/Exit

==> q

FCINFO Main Menu
Select a FC Node from the following list:

Location Code Pathname

  1. U9080.MHE.78F42D7-V7-C2-T1 /vdevice/vfc-client@30000002
  2. U9080.MHE.78F42D7-V7-C3-T1 /vdevice/vfc-client@30000003
  3. U9080.MHE.78F42D7-V7-C4-T1 /vdevice/vfc-client@30000004
  4. U9080.MHE.78F42D7-V7-C5-T1 /vdevice/vfc-client@30000005

q - Quit/Exit

Here is the console log of the two different restores that did not work.

Configuring Relax-and-Recover rescue system

Running 00-functions.sh...
Running 01-run-ldconfig.sh...
Running 10-console-setup.sh...
Using keymap of the original system
Running 20-check-boot-options.sh...
Running 40-start-udev-or-load-modules.sh...
Waiting for udev ... done.
Running 41-load-special-modules.sh...
Running 42-engage-scsi.sh...
Running 45-serial-console.sh...
Running 55-migrate-network-devices.sh...
No network interface mapping is specified in /etc/rear/mappings/mac
The original network interface eth0 d6:8b:e6:36:62:08 is not available

  1. eth3 d6:8b:e5:4d:db:08 ibmveth
  2. eth4 d6:8b:e5:4d:db:09 ibmveth
  3. eth5 d6:8b:e5:4d:db:0a ibmveth
  4. Skip replacing eth0 d6:8b:e6:36:62:08
    Choose replacement for eth0 d6:8b:e6:36:62:08 1
    Mapping eth0 d6:8b:e6:36:62:08 to eth3 d6:8b:e5:4d:db:08
    The original network interface eth1 d6:8b:e6:36:62:09 is not available
    1. eth4 d6:8b:e5:4d:db:09 ibmveth
  6. eth5 d6:8b:e5:4d:db:0a ibmveth
  7. Skip replacing eth1 d6:8b:e6:36:62:09
    Choose replacement for eth1 d6:8b:e6:36:62:09 2
    Mapping eth1 d6:8b:e6:36:62:09 to eth4 d6:8b:e5:4d:db:09
    The original network interface eth2 d6:8b:e6:36:62:0a is not available
      1. eth5 d6:8b:e5:4d:db:0a ibmveth
  9. Skip replacing eth2 d6:8b:e6:36:62:0a
    Choose replacement for eth2 d6:8b:e6:36:62:0a 3
    Mapping eth2 d6:8b:e6:36:62:0a to eth5 d6:8b:e5:4d:db:0a
    Reloading udev ... done.
    Running 58-start-dhclient.sh...
    Running 60-network-devices.sh...
    Running 62-routing.sh...
    Running 65-sysctl.sh...
    net.ipv6.conf.all.disable_ipv6 = 1
    net.ipv4.ip_forward = 1
    net.ipv6.conf.all.forwarding = 0
    vm.pagecache_limit_ignore_dirty = 2
    net.core.somaxconn = 4096
    net.ipv4.tcp_max_syn_backlog = 8192
    net.ipv4.tcp_tw_reuse = 1
    net.ipv4.tcp_tw_recycle = 1
    net.ipv4.tcp_syn_retries = 8
    net.ipv4.tcp_wmem = 4096 16384 4194304
    vm.min_free_kbytes = 10485760
    net.ipv6.conf.all.disable_ipv6 = 1
    net.ipv4.ip_forward = 1
    net.ipv6.conf.all.forwarding = 0
    vm.pagecache_limit_ignore_dirty = 2
    net.core.somaxconn = 4096
    net.ipv4.tcp_max_syn_backlog = 8192
    net.ipv4.tcp_tw_reuse = 1
    net.ipv4.tcp_tw_recycle = 1
    net.ipv4.tcp_syn_retries = 8
    net.ipv4.tcp_wmem = 4096 16384 4194304
    vm.min_free_kbytes = 10485760
    Running 99-makedev.sh...

Relax-and-Recover rescue system is ready

[ OK ] Started Initialize Rescue System.
[ OK ] Started Getty on tty0.
[ OK ] Reached target Login Prompts.
[ OK ] Started Serial Getty on hvc0.
[ OK ] Started Serial Getty on ttyS0.
[ OK ] Reached target Multi-User.

Relax-and-Recover 2.4-git.0.6ec9075.unknown / 2018-12-05

Relax-and-Recover comes with ABSOLUTELY NO WARRANTY; for details see
the GNU General Public License at: http://www.gnu.org/licenses/gpl.html

Host eniesdbp104p using Backup NETFS and Output ISO
Build date: Thu, 17 Jan 2019 14:22:20 -0500

Welcome to SUSE Linux Enterprise Server for SAP Applications 12 SP3
(ppc64le) - Kernel 4.4.162-94.72-default (hvc0).

SSH fingerprint: 2048 SHA256:bn+LTuNXpbRAhjgEBjQZ6rZv29K8anvt2wL7R2uXgqw
root@eniesdbp104p (RSA)

eniesdbp104p login: root

Welcome to Relax-and-Recover. Run "rear recover" to restore your system !

RESCUE eniesdbp104p:~ # rear recover
Relax-and-Recover 2.4-git.0.6ec9075.unknown / 2018-12-05
Running rear recover (PID 24570)
Using log file: /var/log/rear/rear-eniesdbp104p.log
Running workflow recover within the ReaR rescue/recovery system
Using backup archive
'/tmp/rear.Hs00F3ALINE1ns3/outputfs/iso_fs/REAR_BACKUP/backup.tar.gz'
Will do driver migration (recreating initramfs/initrd)
Calculating backup archive size
Backup archive size is 1909.38M (compressed)
Setting up multipathing
Activating multipath
multipath activated
Starting multipath daemon
multipathd started
Listing multipath device found

Comparing disks
Ambiguous possible target disks need manual configuration (more than one
with same size found)
Switching to manual disk layout configuration
Using /dev/sda (same size) for recreating /dev/mapper/200173800348802b3
Current disk mapping table (source => target):
/dev/mapper/200173800348802b3 => /dev/sda

Configuring Relax-and-Recover rescue system

Running 00-functions.sh...
Running 01-run-ldconfig.sh...
Running 10-console-setup.sh...
Using keymap of the original system
Running 20-check-boot-options.sh...
Running 40-start-udev-or-load-modules.sh...
Waiting for udev ... done.
Running 41-load-special-modules.sh...
Running 42-engage-scsi.sh...
Running 45-serial-console.sh...
Running 55-migrate-network-devices.sh...
No network interface mapping is specified in /etc/rear/mappings/mac
The original network interface eth0 d6:8b:e6:36:62:08 is not available

  1. eth1 d6:8b:e5:4d:db:09 ibmveth
  2. eth2 d6:8b:e5:4d:db:0a ibmveth
  3. eth3 d6:8b:e5:4d:db:08 ibmveth
  4. Skip replacing eth0 d6:8b:e6:36:62:08
    Choose replacement for eth0 d6:8b:e6:36:62:08 3
    Mapping eth0 d6:8b:e6:36:62:08 to eth3 d6:8b:e5:4d:db:08
    The original network interface eth1 d6:8b:e6:36:62:09 is not available
  5. eth1 d6:8b:e5:4d:db:09 ibmveth
  6. eth2 d6:8b:e5:4d:db:0a ibmveth
    1. Skip replacing eth1 d6:8b:e6:36:62:09
      Choose replacement for eth1 d6:8b:e6:36:62:09 1
      Mapping eth1 d6:8b:e6:36:62:09 to eth1 d6:8b:e5:4d:db:09
      The original network interface eth2 d6:8b:e6:36:62:0a is not available
    1. eth2 d6:8b:e5:4d:db:0a ibmveth
    1. Skip replacing eth2 d6:8b:e6:36:62:0a
      Choose replacement for eth2 d6:8b:e6:36:62:0a 2
      Mapping eth2 d6:8b:e6:36:62:0a to eth2 d6:8b:e5:4d:db:0a
      Reloading udev ... done.
      Running 58-start-dhclient.sh...
      Running 60-network-devices.sh...
      Running 62-routing.sh...
      Running 65-sysctl.sh...
      net.ipv6.conf.all.disable_ipv6 = 1
      net.ipv4.ip_forward = 1
      net.ipv6.conf.all.forwarding = 0
      vm.pagecache_limit_ignore_dirty = 2
      net.core.somaxconn = 4096
      net.ipv4.tcp_max_syn_backlog = 8192
      net.ipv4.tcp_tw_reuse = 1
      net.ipv4.tcp_tw_recycle = 1
      net.ipv4.tcp_syn_retries = 8
      net.ipv4.tcp_wmem = 4096 16384 4194304
      vm.min_free_kbytes = 10485760
      net.ipv6.conf.all.disable_ipv6 = 1
      net.ipv4.ip_forward = 1
      net.ipv6.conf.all.forwarding = 0
      vm.pagecache_limit_ignore_dirty = 2
      net.core.somaxconn = 4096
      net.ipv4.tcp_max_syn_backlog = 8192
      net.ipv4.tcp_tw_reuse = 1
      net.ipv4.tcp_tw_recycle = 1
      net.ipv4.tcp_syn_retries = 8
      net.ipv4.tcp_wmem = 4096 16384 4194304
      vm.min_free_kbytes = 10485760
      Running 99-makedev.sh...

Relax-and-Recover rescue system is ready

[ OK ] Started Initialize Rescue System.
[ OK ] Started Getty on tty0.
[ OK ] Reached target Login Prompts.
[ OK ] Reached target Multi-User.
[ OK ] Started Serial Getty on ttyS0.
[ OK ] Started Serial Getty on hvc0.

Relax-and-Recover 2.4-git.0.6ec9075.unknown / 2018-12-05

Relax-and-Recover comes with ABSOLUTELY NO WARRANTY; for details see
the GNU General Public License at: http://www.gnu.org/licenses/gpl.html

Host eniesdbp104p using Backup NETFS and Output ISO
Build date: Thu, 17 Jan 2019 14:22:20 -0500

Welcome to SUSE Linux Enterprise Server for SAP Applications 12 SP3
(ppc64le) - Kernel 4.4.162-94.72-default (hvc0).

SSH fingerprint: 2048 SHA256:DNyikRzuULk3bpE1wwyoXX/6b9qV0kBUW0j32NOrdkY
root@eniesdbp104p (RSA)

eniesdbp104p login: root

Welcome to Relax-and-Recover. Run "rear recover" to restore your system !

RESCUE eniesdbp104p:~ # rear recover
Relax-and-Recover 2.4-git.0.6ec9075.unknown / 2018-12-05
Running rear recover (PID 24593)
Using log file: /var/log/rear/rear-eniesdbp104p.log
Running workflow recover within the ReaR rescue/recovery system
Using backup archive
'/tmp/rear.3HmcDWcPOuSWy4R/outputfs/iso_fs/REAR_BACKUP/backup.tar.gz'
Will do driver migration (recreating initramfs/initrd)
Calculating backup archive size
Backup archive size is 1909.38M (compressed)
Setting up multipathing
Activating multipath
multipath activated
Starting multipath daemon
multipathd started
Listing multipath device found
SIBM_2810XIV_780348802C6 dm-0 IBM,2810XIV size=112G
Comparing disks
Ambiguous possible target disks need manual configuration (more than one
with same size found)
Switching to manual disk layout configuration
Using /dev/mapper/SIBM_2810XIV_780348802C6 (same size) for
recreating /dev/mapper/200173800348802b3
Current disk mapping table (source => target):
/dev/mapper/200173800348802b3 => /dev/mapper/SIBM_2810XIV_780348802C6

Dan Wagner
Senior Enterprise System Specialist

(717) 580-3844 Mobile
(717) 409-5858 Office
dwagner1@us.ibm.com

IBM Services

From: "Sébastien Chabrolles" notifications@github.com
To: rear/rear rear@noreply.github.com
Cc: dewagner1 dwagner1@us.ibm.com, Mention
mention@noreply.github.com
Date: 01/18/2019 03:36 AM
Subject: Re: [rear/rear] Update to closed issue #2016 (Unable to clone
Power lpar from rear backup SLES 12 SP3) (#2019)

@dewagner1,

My first guess is still a SAN / FC issue. To validate, the best would be to
boot your LPAR in SMS and run ioinfo via the OK prompt.

1- From SMS menu to OK prompt
Press 0 when entering SMS menu 1st page.

                                         X = eXit System Management

Services

Type menu item number and press Enter or select Navigation key:
Open Completed.

Main Menu

  1. Select Language
  2. Setup Remote IPL (Initial Program Load)
  3. I/O Device Information
  4. Select Console
  5. Select Boot Options

Navigation Keys:

                                         X = eXit System Management

Services

Type menu item number and press Enter or select Navigation key:0

                .--------------------------------------.
                |  Exit SMS to OK> Prompt? (Y)es/(N)o  |
                `--------------------------------------'

ok
0 >

2-Run ioinfo on each fiber adapter to get the list of disk/partitions path
over the SAN network

ok
0 > ioinfo

!!! IOINFO: FOR IBM INTERNAL USE ONLY !!!
This tool gives you information about SCSI,SATA,SAS,and USB devices
attached to the system

Select a tool from the following

  1. SATAINFO
  2. SASINFO
  3. USBINFO
  4. FCINFO
  5. VSCSIINFO

q - quit/exit

==> 4

FCINFO Main Menu
Select a FC Node from the following list:

Location Code Pathname

  1. U8284.22A.217097V-V21-C13-T1 /vdevice/vfc-client@3000000d
  2. U8284.22A.217097V-V21-C14-T1 /vdevice/vfc-client@3000000e

q - Quit/Exit

==> 1

Press 1 to list devices seen by the first Adapter

==> 1

  1. 500507680c21099f,0 - 51200 MB Disk drive (bootable)
  2. 500507680c21099f,1000000000000 - 10240 MB Disk drive
  3. 500507680c21099f,2000000000000 - 10240 MB Disk drive
  4. 500507680c51099f,0 - 51200 MB Disk drive (bootable)
  5. 500507680c51099f,1000000000000 - 10240 MB Disk drive
  6. 500507680c51099f,2000000000000 - 10240 MB Disk drive
  7. 500507680c2109a6,0 - 51200 MB Disk drive (bootable)
  8. 500507680c2109a6,1000000000000 - 10240 MB Disk drive
  9. 500507680c2109a6,2000000000000 - 10240 MB Disk drive
  10. 500507680c5109a6,0 - 51200 MB Disk drive (bootable)
  11. 500507680c5109a6,1000000000000 - 10240 MB Disk drive
  12. 500507680c5109a6,2000000000000 - 10240 MB Disk drive

Hit a key to continue...

Then Quit q and run the same thing on the other adapter

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Only single path to boot disk - confirmed prior to starting install.

Select Attached Device
Pathname: /vdevice/vfc-client@30000002
WorldWidePortName: c0507609f1e20020

  1. 5001738034880141,4000000000000000 120 GB Disk drive

IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM

      1 = SMS Menu                          5 = Default Boot List
      8 = Open Firmware Prompt              6 = Stored Boot List


 Memory      Keyboard     Network     Speaker

Main Menu

  1. Select Language
  2. Setup Remote IPL (Initial Program Load)
  3. I/O Device Information
  4. Select Console
  5. Select Boot Options

Navigation Keys:

                                         X = eXit System Management Services

Type menu item number and press Enter or select Navigation key:3

I/O Device Information

  1. SAN Device Information
  2. SAS Device Information

Type menu item number and press Enter or select Navigation key:1

Select Media Adapter

  1. U9080.MHE.78F42D7-V7-C2-T1 /vdevice/vfc-client@30000002
  2. U9080.MHE.78F42D7-V7-C3-T1 /vdevice/vfc-client@30000003
  3. U9080.MHE.78F42D7-V7-C4-T1 /vdevice/vfc-client@30000004
  4. U9080.MHE.78F42D7-V7-C5-T1 /vdevice/vfc-client@30000005

Type menu item number and press Enter or select Navigation key:1

                          .------------------.
                          |  PLEASE WAIT.... |
                          `------------------'

/

Select Attached Device
Pathname: /vdevice/vfc-client@30000002
WorldWidePortName: c0507609f1e20020

  1. 5001738034880141,4000000000000000 120 GB Disk drive

Type menu item number and press Enter or select Navigation key:

Select Media Adapter

  1. U9080.MHE.78F42D7-V7-C2-T1 /vdevice/vfc-client@30000002
  2. U9080.MHE.78F42D7-V7-C3-T1 /vdevice/vfc-client@30000003
  3. U9080.MHE.78F42D7-V7-C4-T1 /vdevice/vfc-client@30000004
  4. U9080.MHE.78F42D7-V7-C5-T1 /vdevice/vfc-client@30000005

Type menu item number and press Enter or select Navigation key:2

                          .------------------.
                          |  PLEASE WAIT.... |
                          `------------------'






















                     .----------------------------.
                     |  No SAN devices present    |
                     |  Press any key to continue |
                     `----------------------------'

Select Media Adapter

  1. U9080.MHE.78F42D7-V7-C2-T1 /vdevice/vfc-client@30000002
  2. U9080.MHE.78F42D7-V7-C3-T1 /vdevice/vfc-client@30000003
  3. U9080.MHE.78F42D7-V7-C4-T1 /vdevice/vfc-client@30000004
  4. U9080.MHE.78F42D7-V7-C5-T1 /vdevice/vfc-client@30000005

Type menu item number and press Enter or select Navigation key:

Select Media Adapter

  1. U9080.MHE.78F42D7-V7-C2-T1 /vdevice/vfc-client@30000002
  2. U9080.MHE.78F42D7-V7-C3-T1 /vdevice/vfc-client@30000003
  3. U9080.MHE.78F42D7-V7-C4-T1 /vdevice/vfc-client@30000004
  4. U9080.MHE.78F42D7-V7-C5-T1 /vdevice/vfc-client@30000005

Type menu item number and press Enter or select Navigation key:3

                          .------------------.
                          |  PLEASE WAIT.... |
                          `------------------'






















                     .----------------------------.
                     |  No SAN devices present    |
                     |  Press any key to continue |
                     `----------------------------'

Select Media Adapter

  1. U9080.MHE.78F42D7-V7-C2-T1 /vdevice/vfc-client@30000002
  2. U9080.MHE.78F42D7-V7-C3-T1 /vdevice/vfc-client@30000003
  3. U9080.MHE.78F42D7-V7-C4-T1 /vdevice/vfc-client@30000004
  4. U9080.MHE.78F42D7-V7-C5-T1 /vdevice/vfc-client@30000005

Select Media Adapter

  1. U9080.MHE.78F42D7-V7-C2-T1 /vdevice/vfc-client@30000002
  2. U9080.MHE.78F42D7-V7-C3-T1 /vdevice/vfc-client@30000003
  3. U9080.MHE.78F42D7-V7-C4-T1 /vdevice/vfc-client@30000004
  4. U9080.MHE.78F42D7-V7-C5-T1 /vdevice/vfc-client@30000005

Type menu item number and press Enter or select Navigation key:

I/O Device Information

  1. SAN Device Information
  2. SAS Device Information

Type menu item number and press Enter or select Navigation key:

Main Menu

  1. Select Language
  2. Setup Remote IPL (Initial Program Load)
  3. I/O Device Information
  4. Select Console
  5. Select Boot Options

Navigation Keys:

                                         X = eXit System Management Services

Type menu item number and press Enter or select Navigation key:5

Multiboot

  1. Select Install/Boot Device
  2. Configure Boot Device Order
  3. Multiboot Startup
  4. SAN Zoning Support

Type menu item number and press Enter or select Navigation key:1

                          .------------------.
                          |  PLEASE WAIT.... |
                          `------------------'

Select Device Type

  1. Tape
  2. CD/DVD
  3. Hard Drive
  4. Network
  5. List all Devices

Type menu item number and press Enter or select Navigation key:2

Select Media Type

  1. SCSI
  2. SAN
  3. SAS
  4. SATA
  5. USB
  6. NVMe
  7. List All Devices

Type menu item number and press Enter or select Navigation key:1

Select Media Adapter

  1. U9080.MHE.78F42D7-V7-C6-T1 /vdevice/v-scsi@30000006

  2. U9080.MHE.78F42D7-V7-C7-T1 /vdevice/v-scsi@30000007

  3. List all devices

Type menu item number and press Enter or select Navigation key:1

                          .------------------.
                          |  PLEASE WAIT.... |
                          `------------------'

check /vdevice/v-scsi@30000006/disk@8100000000000000

Select Device
Device Current Device
Number Position Name

    • SCSI CD-ROM ( loc=U9080.MHE.78F42D7-V7-C6-T1-L8100000000000000 )

Type menu item number and press Enter or select Navigation key:1

Select Task

SCSI CD-ROM
( loc=U9080.MHE.78F42D7-V7-C6-T1-L8100000000000000 )

  1. Information
  2. Normal Mode Boot
  3. Service Mode Boot

Type menu item number and press Enter or select Navigation key:2

Are you sure you want to exit System Management Services?

  1. Yes
  2. No

Navigation Keys:

                                         X = eXit System Management Services

Type menu item number and press Enter or select Navigation key:1
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM STARTING SOFTWARE IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM PLEASE WAIT... IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM

Elapsed time since release of system processors: 766171 mins 15 secs
Welcome to GRUB!

error: unrecognized number.

                         GNU GRUB  version 2.02

+----------------------------------------------------------------------------+
|*Relax-and-Recover |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
+----------------------------------------------------------------------------+

  Use the ^ and v keys to select which entry is highlighted.
  Press enter to boot the selected OS, `e' to edit the commands
  before booting or `c' for a command-line.

OF stdout device is: /vdevice/vty@30000000 Preparing to boot Linux version 4.4.162-94.72-default (geeko@buildhost) (gcc version 4.8.5 (SUSE Linux) ) #1 SMP Mon Nov 12 18:57:45 UTC 2018 (9de753f) Detected machine type: 0000000000000101 Max number of cores passed to firmware: 256 (NR_CPUS = 2048) Calling ibm,client-architecture-support... done command line: BOOT_IMAGE=/kernel root=/dev/ram0 selinux=0 memory layout at init: memory_limit : 0000000000000000 (16 MB aligned) alloc_bottom : 000000000e840000 alloc_top : 0000000020000000 alloc_top_hi : 0000000020000000 rmo_top : 0000000020000000 ram_top : 0000000020000000 instantiating rtas at 0x000000001e9e0000... done prom_hold_cpus: skipped copying OF device tree... Building dt strings... Building dt structure... Device tree strings 0x000000000e850000 -> 0x000000000e851737 Device tree struct 0x000000000e860000 -> 0x000000000e990000 Quiescing Open Firmware ... Booting Linux via __start() ... Using pSeries machine description Page sizes from device-tree: base_shift=12: shift=12, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=0 base_shift=12: shift=16, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=7 base_shift=12: shift=24, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=56 base_shift=16: shift=16, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=1 base_shift=16: shift=24, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=8 base_shift=24: shift=24, sllp=0x0100, avpnm=0x00000001, tlbiel=0, penc=0 base_shift=34: shift=34, sllp=0x0120, avpnm=0x000007ff, tlbiel=0, penc=3 Using 1TB segments Found initrd at 0xc00000000bb00000:0xc00000000e8320f7 bootconsole [udbg0] enabled Partition configured for 1536 cpus. CPU maps initialized for 8 threads per core -> smp_release_cpus() spinning_secondaries = 887 <- smp_release_cpus() Starting Linux ppc64le #1 SMP Mon Nov 12 18:57:45 UTC 2018 (9de753f)

ppc64_pft_size = 0x24 phys_mem_size = 0xc00000000 cpu_features = 0x27fc7aec18500249 possible = 0x7fffffff18500649 always = 0x0000000018100040 cpu_user_features = 0xdc0065c2 0xef000000 mmu_features = 0x7c002001 firmware_features = 0x00000013c45bfc57 htab_hash_mask = 0x1fffffff

<- setup_system()
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Initializing cgroup subsys cpuacct
Linux version 4.4.162-94.72-default (geeko@buildhost) (gcc version 4.8.5 (SUSE Linux) ) #1 SMP Mon Nov 12 18:57:45 UTC 2018 (9de753f)
numa: Initmem setup node 0
numa: NODE_DATA [mem 0xbfcf3a100-0xbfcf43fff]
numa: NODE_DATA(0) on node 14
numa: Initmem setup node 4 [mem 0x00000000-0x3bfffffff]
numa: NODE_DATA [mem 0x3bfff6100-0x3bfffffff]
numa: Initmem setup node 5 [mem 0x3c0000000-0x84fffffff]
numa: NODE_DATA [mem 0x84fff6100-0x84fffffff]
numa: Initmem setup node 6 [mem 0x850000000-0x9dfffffff]
numa: NODE_DATA [mem 0x9dfff6100-0x9dfffffff]
numa: Initmem setup node 7 [mem 0x9e0000000-0xb3fffffff]
numa: NODE_DATA [mem 0xb3fff6100-0xb3fffffff]
numa: Initmem setup node 14 [mem 0xb40000000-0xbffffffff]
numa: NODE_DATA [mem 0xbfcf30200-0xbfcf3a0ff]
Section 3066 and 3068 (node 14) have a circular dependency on usemap and pgdat allocations
rfi-flush: fallback displacement flush available
rfi-flush: ori type flush available
barrier-nospec: using ORI speculation barrier
stf-barrier: hwsync barrier available
PPC64 nvram contains 15360 bytes
Zone ranges:
DMA [mem 0x0000000000000000-0x0000000bffffffff]
DMA32 empty
Normal empty
Movable zone start for each node
Early memory node ranges
node 4: [mem 0x0000000000000000-0x00000003bfffffff]
node 5: [mem 0x00000003c0000000-0x000000084fffffff]
node 6: [mem 0x0000000850000000-0x00000009dfffffff]
node 7: [mem 0x00000009e0000000-0x0000000b3fffffff]
node 14: [mem 0x0000000b40000000-0x0000000bffffffff]
Could not find start_pfn for node 0
Initmem setup node 0 [mem 0x0000000000000000-0x0000000000000000]
Initmem setup node 4 [mem 0x0000000000000000-0x00000003bfffffff]
Initmem setup node 5 [mem 0x00000003c0000000-0x000000084fffffff]
Initmem setup node 6 [mem 0x0000000850000000-0x00000009dfffffff]
Initmem setup node 7 [mem 0x00000009e0000000-0x0000000b3fffffff]
Initmem setup node 14 [mem 0x0000000b40000000-0x0000000bffffffff]
PERCPU: Embedded 3 pages/cpu @c000000be2f00000 s123032 r0 d73576 u262144
Built 6 zonelists in Node order, mobility grouping on. Total pages: 785664
Policy zone: DMA
Kernel command line: BOOT_IMAGE=/kernel root=/dev/ram0 selinux=0
log_buf_len individual max cpu contribution: 4096 bytes
log_buf_len total cpu_extra contributions: 6287360 bytes
log_buf_len min size: 524288 bytes
log_buf_len: 8388608 bytes
early log buf free: 501904(95%)
PID hash table entries: 4096 (order: -1, 32768 bytes)
Sorting __ex_table...
Memory: 10532800K/50331648K available (8320K kernel code, 1280K rwdata, 2560K rodata, 3456K init, 3576K bss, 552640K reserved, 0K cma-reserved)
Hierarchical RCU implementation.
RCU debugfs-based tracing is enabled.
Build-time adjustment of leaf fanout to 64.
RCU restricting CPUs from NR_CPUS=2048 to nr_cpu_ids=1536.
RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=1536
NR_IRQS:512 nr_irqs:512 16
clocksource: timebase: mask: 0xffffffffffffffff max_cycles: 0x761537d007, max_idle_ns: 440795202126 ns
clocksource: timebase mult[1f40000] shift[24] registered
Console: colour dummy device 80x25
console [hvc0] enabled
console [hvc0] enabled
bootconsole [udbg0] disabled
bootconsole [udbg0] disabled
mempolicy: Disabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl
pid_max: default: 1572864 minimum: 12288
Security Framework initialized
AppArmor: AppArmor initialized
Dentry cache hash table entries: 8388608 (order: 10, 67108864 bytes)
Inode-cache hash table entries: 4194304 (order: 9, 33554432 bytes)
Mount-cache hash table entries: 131072 (order: 4, 1048576 bytes)
Mountpoint-cache hash table entries: 131072 (order: 4, 1048576 bytes)
Initializing cgroup subsys io
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys perf_event
Initializing cgroup subsys net_prio
Initializing cgroup subsys hugetlb
Initializing cgroup subsys pids
ftrace: allocating 22528 entries in 9 pages
EEH: pSeries platform initialized
POWER8 performance monitor hardware support registered
random: nonblocking pool is initialized
Brought up 888 CPUs
node 14 initialised, 10861 pages in 0ms
node 7 initialised, 57343 pages in 0ms
node 6 initialised, 69375 pages in 0ms
node 4 initialised, 210173 pages in 0ms
node 5 initialised, 265470 pages in 0ms
devtmpfs: initialized
evm: security.selinux
evm: security.ima
evm: security.capability
EEH: devices created
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
futex hash table entries: 524288 (order: 10, 67108864 bytes)
NET: Registered protocol family 16
EEH: No capable adapters found
cpuidle: using governor ladder
cpuidle: using governor menu
pstore: Registered nvram as persistent store backend
PCI: Probing PCI hardware
pseries-rng: Registering arch random hook.
vgaarb: loaded
EDAC MC: Ver: 3.0.0
NetLabel: Initializing
NetLabel: domain hash size = 128
NetLabel: protocols = UNLABELED CIPSOv4
NetLabel: unlabeled traffic allowed by default
clocksource: Switched to clocksource timebase
AppArmor: AppArmor Filesystem Enabled
NET: Registered protocol family 2
TCP established hash table entries: 524288 (order: 6, 4194304 bytes)
TCP bind hash table entries: 65536 (order: 4, 1048576 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
UDP hash table entries: 32768 (order: 4, 1048576 bytes)
UDP-Lite hash table entries: 32768 (order: 4, 1048576 bytes)
NET: Registered protocol family 1
Unpacking initramfs...
Freeing initrd memory: 46272K
IOMMU table initialized, virtual merging enabled
hv-24x7: read 1330 catalog entries, created 337 event attrs (0 failures), 169 descs
kgr: successfully initialized
audit: initializing netlink subsys (disabled)
audit: type=2000 audit(1547843657.180:1): initialized
Initialise system trusted keyring
HugeTLB registered 16 MB page size, pre-allocated 0 pages
HugeTLB registered 16 GB page size, pre-allocated 0 pages
zbud: loaded
VFS: Disk quotas dquot_6.6.0
VFS: Dquot-cache hash table entries: 8192 (order 0, 65536 bytes)
Key type asymmetric registered
Asymmetric key parser 'x509' registered
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
pseries_rng: Registering IBM pSeries RNG driver
mousedev: PS/2 mouse device common for all mice
ledtrig-cpu: registered to indicate activity on CPUs
nx_compress_pseries ibm,compression-v1: nx842_OF_upd: device disabled
hidraw: raw HID events driver (C) Jiri Kosina
NET: Registered protocol family 10
NET: Registered protocol family 15
registered taskstats version 1
Loading compiled-in X.509 certificates
Loaded X.509 cert 'SUSE Linux Enterprise Secure Boot Signkey: 3fb077b6cebc6ff2522e1c148c57c777c788e3e7'
Loaded X.509 cert 'SUSE Linux Enterprise Secure Boot Signkey: 3fb077b6cebc6ff2522e1c148c57c777c788e3e7'
zswap: loaded using pool lzo/zbud
page_owner is disabled
Key type trusted registered
Key type encrypted registered
AppArmor: AppArmor sha1 policy hashing enabled
ima: No TPM chip found, activating TPM-bypass!
ima: Allocated hash algorithm: sha256
evm: HMAC attrs: 0x1
Freeing unused kernel memory: 3456K

Welcome to SUSE Linux Enterprise Server 12 SP3!

[ OK ] Created slice system.slice.
[ OK ] Reached target System Initialization.
[ OK ] Started Relax-and-Recover sshd service.
[ OK ] Listening on udev Kernel Socket.
[ OK ] Created slice system-getty.slice.
Starting Relax-and-Recover boot script...
[ OK ] Listening on Journal Socket.
[ OK ] Started Journal Service.
[ OK ] Listening on udev Control Socket.
Starting udev Kernel Device Manager...
Starting udev Coldplug all Devices...
[ OK ] Created slice system-serial\x2dgetty.slice.
[ OK ] Started Relax-and-Recover boot script.
[ OK ] Started udev Kernel Device Manager.
Starting Initialize Rescue System...
[ OK ] Found device /dev/ttyS0.
[ OK ] Started udev Coldplug all Devices.
[ OK ] Found device /dev/hvc0.

Verifying md5sums of the files in the Relax-and-Recover rescue system

md5sums are OK

Configuring Relax-and-Recover rescue system

Running 00-functions.sh...
Running 01-run-ldconfig.sh...
Running 10-console-setup.sh...
Using keymap of the original system
Running 20-check-boot-options.sh...
Running 40-start-udev-or-load-modules.sh...
Waiting for udev ... done.
Running 41-load-special-modules.sh...
Running 42-engage-scsi.sh...
Running 45-serial-console.sh...
Running 55-migrate-network-devices.sh...
No network interface mapping is specified in /etc/rear/mappings/mac
The original network interface eth0 d6:8b:e6:36:62:08 is not available

  1. eth0 d6:8b:e5:4d:db:08 ibmveth
  2. eth1 d6:8b:e5:4d:db:09 ibmveth
  3. eth2 d6:8b:e5:4d:db:0a ibmveth
  4. Skip replacing eth0 d6:8b:e6:36:62:08
    Choose replacement for eth0 d6:8b:e6:36:62:08 1
    Mapping eth0 d6:8b:e6:36:62:08 to eth0 d6:8b:e5:4d:db:08
    The original network interface eth1 d6:8b:e6:36:62:09 is not available
    1. eth1 d6:8b:e5:4d:db:09 ibmveth
  6. eth2 d6:8b:e5:4d:db:0a ibmveth
  7. Skip replacing eth1 d6:8b:e6:36:62:09
    Choose replacement for eth1 d6:8b:e6:36:62:09 2
    Mapping eth1 d6:8b:e6:36:62:09 to eth1 d6:8b:e5:4d:db:09
    The original network interface eth2 d6:8b:e6:36:62:0a is not available
      1. eth2 d6:8b:e5:4d:db:0a ibmveth
  9. Skip replacing eth2 d6:8b:e6:36:62:0a
    Choose replacement for eth2 d6:8b:e6:36:62:0a 3
    Mapping eth2 d6:8b:e6:36:62:0a to eth2 d6:8b:e5:4d:db:0a
    Reloading udev ... done.
    Running 58-start-dhclient.sh...
    Running 60-network-devices.sh...
    Running 62-routing.sh...
    Running 65-sysctl.sh...
    net.ipv6.conf.all.disable_ipv6 = 1
    net.ipv4.ip_forward = 1
    net.ipv6.conf.all.forwarding = 0
    vm.pagecache_limit_ignore_dirty = 2
    net.core.somaxconn = 4096
    net.ipv4.tcp_max_syn_backlog = 8192
    net.ipv4.tcp_tw_reuse = 1
    net.ipv4.tcp_tw_recycle = 1
    net.ipv4.tcp_syn_retries = 8
    net.ipv4.tcp_wmem = 4096 16384 4194304
    vm.min_free_kbytes = 10485760
    net.ipv6.conf.all.disable_ipv6 = 1
    net.ipv4.ip_forward = 1
    net.ipv6.conf.all.forwarding = 0
    vm.pagecache_limit_ignore_dirty = 2
    net.core.somaxconn = 4096
    net.ipv4.tcp_max_syn_backlog = 8192
    net.ipv4.tcp_tw_reuse = 1
    net.ipv4.tcp_tw_recycle = 1
    net.ipv4.tcp_syn_retries = 8
    net.ipv4.tcp_wmem = 4096 16384 4194304
    vm.min_free_kbytes = 10485760
    Running 99-makedev.sh...

Relax-and-Recover rescue system is ready

[ OK ] Started Initialize Rescue System.
[ OK ] Started Getty on tty0.
[ OK ] Reached target Login Prompts.
[ OK ] Started Serial Getty on hvc0.
[ OK ] Started Serial Getty on ttyS0.
[ OK ] Reached target Multi-User.

Relax-and-Recover 2.4-git.0.6ec9075.unknown / 2018-12-05

Relax-and-Recover comes with ABSOLUTELY NO WARRANTY; for details see
the GNU General Public License at: http://www.gnu.org/licenses/gpl.html

Host eniesdbp104p using Backup NETFS and Output ISO
Build date: Thu, 17 Jan 2019 14:22:20 -0500

Welcome to SUSE Linux Enterprise Server for SAP Applications 12 SP3 (ppc64le) - Kernel 4.4.162-94.72-default (hvc0).

SSH fingerprint: 2048 SHA256:23t2+0Adg3YN3JucVL4JqztjFC31d9eE8HA0F2SYMBs root@eniesdbp104p (RSA)

eniesdbp104p login: root

Welcome to Relax-and-Recover. Run "rear recover" to restore your system !

RESCUE eniesdbp104p:~ # ln -s /dev/sr0 /dev/disk/by-label/RELAXRECOVER
ln: failed to create symbolic link '/dev/disk/by-label/RELAXRECOVER': No such file or directory
RESCUE eniesdbp104p:~ # mkdir /dev/disk/by-label
RESCUE eniesdbp104p:~ # ln -s /dev/sr0 /dev/disk/by-label/RELAXRECOVER
RESCUE eniesdbp104p:~ # ln -s /dev/sr0 /dev/disk/by-label/RELAXRECOVER
ln: failed to create symbolic link '/dev/disk/by-label/RELAXRECOVER': File exists
RESCUE eniesdbp104p:~ # : # there are some predefined entries in the bash histo
ry
RESCUE eniesdbp104p:~ # ls -l /dev/disk/by-label
total 0
lrwxrwxrwx 1 root root 8 Jan 18 15:35 RELAXRECOVER -> /dev/sr0
RESCUE eniesdbp104p:~ # date
Fri Jan 18 15:35:43 EST 2019
RESCUE eniesdbp104p:~ # rear -dDv recover
Relax-and-Recover 2.4-git.0.6ec9075.unknown / 2018-12-05
Running rear recover (PID 24361)
Using log file: /var/log/rear/rear-eniesdbp104p.log
Running workflow recover within the ReaR rescue/recovery system
Using backup archive '/tmp/rear.hHdJNoKVRZfeXUB/outputfs/iso_fs/REAR_BACKUP/backup.tar.gz'
Will do driver migration (recreating initramfs/initrd)
Calculating backup archive size
Backup archive size is 1909.38M (compressed)
Setting up multipathing
Activating multipath
multipath activated
Starting multipath daemon
multipathd started
Listing multipath device found

Comparing disks
Device mapper!200173800348802b3 does not exist (manual configuration needed)
Switching to manual disk layout configuration
Using /dev/sda (same size) for recreating /dev/mapper/200173800348802b3
Current disk mapping table (source => target):
/dev/mapper/200173800348802b3 => /dev/sda

UserInput -I LAYOUT_MIGRATION_CONFIRM_MAPPINGS needed in /usr/share/rear/layout/prepare/default/300_map_disks.sh line 275
Confirm or edit the disk mapping

  1. Confirm disk mapping and continue 'rear recover'
  2. n/a
  3. Edit disk mapping (/var/lib/rear/layout/disk_mappings)
  4. Use Relax-and-Recover shell and return back to here
  5. Abort 'rear recover'
    (default '1' timeout 300 seconds)

Terminated
RESCUE eniesdbp104p:~ # cat /var/log/rear/rear-eniesdbp104p.log
2019-01-18 15:35:59.411309741 Relax-and-Recover 2.4-git.0.6ec9075.unknown / 2018-12-05
2019-01-18 15:35:59.412345248 Running rear recover (PID 24361)
2019-01-18 15:35:59.413339040 Command line options: /bin/rear -dDv recover
2019-01-18 15:35:59.414268758 Using log file: /var/log/rear/rear-eniesdbp104p.log
2019-01-18 15:35:59.415438551 Current set of flags is 'hB'
2019-01-18 15:35:59.416390895 The debugscripts flags are 'x'
2019-01-18 15:35:59.417275438 Combining configuration files
2019-01-18 15:35:59.418265733 Including /etc/rear/os.conf
2019-01-18 15:35:59.419092456 Entering debugscripts mode via 'set -x'.

  • source /etc/rear/os.conf
    ++ OS_VENDOR=SUSE_LINUX
    ++ OS_VERSION=12.3
    ++ ARCH=Linux-ppc64le
    ++ OS=GNU/Linux
    ++ OS_VERSION=12.3
    ++ OS_VENDOR=SUSE_LINUX
    ++ OS_VENDOR_VERSION=SUSE_LINUX/12.3
    ++ OS_VENDOR_ARCH=SUSE_LINUX/ppc64le
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-18 15:35:59.422509686 '
  • test 1 -gt 0
  • echo '2019-01-18 15:35:59.422509686 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-18 15:35:59.422509686 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-18 15:35:59.426007211 Including conf/Linux-ppc64le.conf
    2019-01-18 15:35:59.426882393 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/conf/Linux-ppc64le.conf
    ++ REQUIRED_PROGS=("${REQUIRED_PROGS[@]}" sfdisk)
    ++ PROGS=("${PROGS[@]}" bootlist pseries_platform nvram ofpathname bc agetty)
    +++ awk '/platform/ {print $NF}'
    ++ [[ pSeries != PowerNV ]]
    ++ test ''
    ++ FIRMWARE_FILES=('no')
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-18 15:35:59.434301636 '
  • test 1 -gt 0
  • echo '2019-01-18 15:35:59.434301636 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-18 15:35:59.434301636 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-18 15:35:59.436224578 Including conf/GNU/Linux.conf
    2019-01-18 15:35:59.437166351 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/conf/GNU/Linux.conf
    ++ REQUIRED_PROGS=("${REQUIRED_PROGS[@]:-}" chroot ip less parted route readlink)
    ++ PROGS=(${PROGS[@]:-} partprobe fdisk cfdisk sfdisk)
    ++ PROGS=(${PROGS[@]:-} rpc.statd rpcbind mknod blkid vol_id udev_volume_id portmap rpcinfo tac reboot halt shutdown killall5 killall tee ifconfig nslookup ifenslave ifrename nameif klogd syslog-ng syslogd rsyslogd date rmdir init telinit ethtool expand insmod modprobe lsmod true false mingetty agetty getty rmmod hostname uname sleep logger pstree ln dirname basename tty ping netstat free traceroute vi pico nano rmmod df dmesg du gzip netcat curl top iptraf joe getent id ldd strace rsync tail head find md5sum mkfs mkfs.ext2 mkfs.ext3 mkfs.ext4 mkfs.ext4dev mkfs.jfs mkfs.xfs mkfs.reiserfs mkfs.vfat mkfs.btrfs mkreiserfs fsck fsck.ext2 fsck.ext3 fsck.ext4 fsck.ext4dev fsck.xfs fsck.reiserfs reiserfsck fsck.btrfs findmnt btrfsck tune2fs tune4fs xfs_admin xfs_db xfs_repair xfs_info xfs_growfs btrfs jfs_tune reiserfstune egrep fgrep chmod chown stat mkswap swapon swapoff mknod touch scsi_id lsscsi logd initctl lspci usleep mktemp /bin/true strace which mkfifo seq openvt poweroff chacl getfacl setfacl attr getfattr setfattr mpath_wait xargs sg_inq env w dosfslabel sysctl blockdev lsblk)
    ++ LIBS=(${LIBS[@]:-} /lib*/libnss_dns* /lib*/libnss_files* /lib//libnss_dns /lib//libnss_files /lib*/libgcc_s* /lib*/libresolv* /usr/lib*/rsyslog/so /lib/rsyslog/so /usr/lib/syslog-ng/* /lib*/libnsspem.so* /usr/lib*/libnsspem.so* /lib*/libfreebl*.so* /usr/lib*/libfreebl*.so* /lib*/libnss3.so* /usr/lib*/libnss3.so* /lib*/libnssutil3.so* /usr/lib*/libnssutil3.so* /lib*/libsoftokn3.so* /usr/lib*/libsoftokn3.so* /lib*/libsqlite3.so* /usr/lib*/libsqlite3.so* /lib*/libfreeblpriv3.so* /usr/lib*/libfreeblpriv3.so* /lib*/libssl.so* /usr/lib*/libssl.so* /lib*/libnssdbm3.so* /usr/lib*/libnssdbm3.so*)
    ++ MODULES=(${MODULES[@]:-} vfat nls_iso8859_1 nls_utf8 nls_cp437 af_packet unix nfs nfsv4 nfsv3 lockd sunrpc cifs usbcore usb_storage usbhid sr_mod ide_cd cdrom uhci_hcd ehci_hcd xhci_hcd ohci_hcd zlib zlib-inflate zlib-deflate libcrc32c crc32c crc32c-intel)
    ++ COPY_AS_IS=(${COPY_AS_IS[@]:-} /dev /etc/inputr[c] /etc/protocols /etc/services /etc/rpc /etc/termcap /etc/terminfo /lib*/terminfo /usr/share/terminfo /etc/netconfig /etc/mke2fs.conf /etc/-release /etc/localtime /etc/magic /usr/share/misc/magic /etc/dracut.conf /etc/dracut.conf.d /usr/lib/dracut /sbin/modprobe.ksplice-orig /etc/sysctl.conf /etc/sysctl.d /etc/e2fsck.conf)
    ++ COPY_AS_IS=("${COPY_AS_IS[@]}" '/etc/ssl/certs/    ' '/etc/pki/' '/usr/lib/ssl/' '/usr/share/ca-certificates/' '/etc/ca-certificates/')
    ++ COPY_AS_IS_EXCLUDE=(${COPY_AS_IS_EXCLUDE[@]:-} dev/shm/*)
    ++ COPY_AS_IS_EXCLUDE=("${COPY_AS_IS_EXCLUDE[@]}" '/etc/pki/tls/private' '/etc/pki/CA/private' '/etc/pki/nssdb/key*.db' '/usr/lib/ssl/private')
    ++ KERNEL_CMDLINE=' selinux=0'
    ++ CLONE_USERS=("${CLONE_USERS[@]:-}" daemon rpc usbmuxd usbmux vcsa nobody dbus)
    ++ CLONE_GROUPS=("${CLONE_GROUPS[@]:-}" tty usbmuxd usbmux fuse kvm oinstall dbus)
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-18 15:35:59.443917847 '
  • test 1 -gt 0
  • echo '2019-01-18 15:35:59.443917847 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-18 15:35:59.443917847 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-18 15:35:59.445847813 Including conf/SUSE_LINUX.conf
    2019-01-18 15:35:59.446764829 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/conf/SUSE_LINUX.conf
    ++ COPY_AS_IS=("${COPY_AS_IS[@]}" /etc/sysconfig/network/config /etc/sysconfig/network/scripts/functions)
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-18 15:35:59.450620080 '
  • test 1 -gt 0
  • echo '2019-01-18 15:35:59.450620080 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-18 15:35:59.450620080 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-18 15:35:59.453817705 Including /etc/rear/local.conf
    2019-01-18 15:35:59.454697648 Entering debugscripts mode via 'set -x'.
  • source /etc/rear/local.conf
    ++ AUTOEXCLUDE_MULTIPATH=n
    ++ BOOT_OVER_SAN=y
    ++ REAR_INITRD_COMPRESSION=lzma
    ++ OUTPUT=ISO
    ++ ISO_MAX_SIZE=4000
    ++ BACKUP=NETFS
    ++ BACKUP_URL=iso:///iso_fs/REAR_BACKUP
    ++ ISO_DIR=/iso_fs/REAR_ISO
    ++ TMPDIR=/iso_fs/REAR_TEMP
    ++ OUTPUT_URL=null
    ++ BOOT_FROM_SAN=y
    ++ EXCLUDE_MOUNTPOINTS=(/iso_fs /opt/IBM/ITM /cv /opt/splunkforwarder /opt/teamquest /var/opt/BESClient /usr/sap /hana/data /hana/log /hana/shared /usr/sap/basis /usr/sap/srm /PA_backup)
    ++ EXCLUDE_COMPONENTS=(/dev/mapper/200173800348802ba /dev/mapper/200173800348802bb /dev/mapper/200173800348802bc /dev/mapper/200173800348802bd /dev/mapper/200173800348802be /dev/mapper/200173800348802bf /dev/mapper/200173800348802c0 /dev/mapper/200173800348802c1 /dev/mapper/200173800348802c2 /dev/mapper/200173800348802c3 /dev/mapper/200173800348802c4 /dev/mapper/200173800348802c5)
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-18 15:35:59.458303640 '
  • test 1 -gt 0
  • echo '2019-01-18 15:35:59.458303640 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-18 15:35:59.458303640 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-18 15:35:59.461452821 Including /etc/rear/rescue.conf
    2019-01-18 15:35:59.462357239 Entering debugscripts mode via 'set -x'.
  • source /etc/rear/rescue.conf
    ++ SHARE_DIR=/usr/share/rear
    ++ CONFIG_DIR=/etc/rear
    ++ VAR_DIR=/var/lib/rear
    ++ LOG_DIR=/var/log/rear
    ++ BACKUP_PROG_OPTIONS=(--anchored --xattrs --xattrs-include=security.capability --xattrs-include=security.selinux --acls)
    ++ NETFS_KEEP_OLD_BACKUP_COPY=
    ++ NETFS_PREFIX=eniesdbp104p
    ++ NETFS_RESTORE_CAPABILITIES=([0]="No")
    ++ unset TMPDIR
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-18 15:35:59.465967018 '
  • test 1 -gt 0
  • echo '2019-01-18 15:35:59.465967018 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-18 15:35:59.465967018 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-18 15:35:59.469147331 ======================
    2019-01-18 15:35:59.469975856 Running 'init' stage
    2019-01-18 15:35:59.470821384 ======================
    2019-01-18 15:35:59.476234568 Including init/default/005_verify_os_conf.sh
    2019-01-18 15:35:59.477086173 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/init/default/005_verify_os_conf.sh

dewagner1 commented at 2019-01-21 22:55:

Hello Sebastian,

I work with Dan and he asked me to add some more info regarding our SLES12
SP3 rear recovery issue...

We have only the 100 GB disk mapped to each server, for now - that is by
design to helps us avoid any confusion on what disk should be used; that
single disk is our desired OS-bootable disk.

When we boot into SMS/Firmware - I can verify the LUN is zoned/detected
correctly via the ioinfo menus.

We have dual SAN fabrics in our Linux server environment - using Cisco MDS
switches... 2 fabrics that are redundant (completely identical in port
layout/config,) yet separate (distinctly cabled and completely isolated).

Each of our XIV LUNs are mapped to 2 different VirtFC adapters (in unique
fabrics), and each VirtFC adapter is zoned to 2 different XIV IOports (in
same fabric) -- (2 x 2) thus configuring 4 total disk paths to the same
LUN.

This setup of 4 paths doesn't seem to be a problem, not with native the
multipathing support - multipathd - available to us.

In the past, we've seen the "Ambiguous" message in the rear recover output,
but the mutipathing was able to handle and properly detect the disk device.

Here is what a successful install outputs:

...
Setting up multipathing
Activating multipath
multipath activated
Starting multipath daemon
multipathd started
Listing multipath device found
200173800348802b2 dm-0 IBM,2810XIV size=112G
Comparing disks
Ambiguous possible target disks need manual configuration (more than one
with same size found)
Switching to manual disk layout configuration
Using /dev/mapper/200173800348802b2 (same size) for
recreating /dev/mapper/200173800348802b3
Current disk mapping table (source => target):
/dev/mapper/200173800348802b3 => /dev/mapper/200173800348802b2
...

When we have our SAN admin unzone 3 of the 4 disk connections, we no longer
see the "Ambiguous" line... so we are confident we can explain that and are
confident that is NOT the cause of our issue.

From the above messages, what seems to be key is the ability of rear to
determine the disk "source => target" mapping with the correct and proper
names.

Our issue stems from repeated attempts yielding inconsistent install
results - with success occurring intermittent & random... not very
trustworthy.

When we fail, we'll see either this type of mapping:

...
Switching to manual disk layout configuration
Using /dev/sda (same size) for recreating /dev/mapper/200173800348802b3
Current disk mapping table (source => target):
/dev/mapper/200173800348802b3 => /dev/sda

or this type of mapping:

...
Switching to manual disk layout configuration
Using /dev/mapper/SIBM_2810XIV_780348802C6 (same size) for
recreating /dev/mapper/200173800348802b3
Current disk mapping table (source => target):
/dev/mapper/200173800348802b3 => /dev/mapper/SIBM_2810XIV_780348802C6

So the BIG question is - how is rear discovering and determining the
target device's file location & name, and why with nothing being altered,
does it fail to do it consistently ??

I welcome your thoughts on understanding/resolving this... and I'd be glad
to provide you any other info - or perform any other follow-up - that you
deem would be helpful.

Thank you.

Regards,

Glenn Lehman
Unix Systems Administrator - AIX, Linux, VIO, pSeries, Storage & SAN
Global Technology Services | Distributed Systems-Unix, IBM
(717) 409-5857 - Office

From: Dan Wagner/Mechanicsburg/Contr/IBM
To: rear/rear <reply
+0283324a89c9c6b22b80d4fd3f55956ed4745071464822bd92cf00000001185951f092a169ce17dedc38@reply.github.com>
Cc: Mention mention@noreply.github.com, rear/rear
rear@noreply.github.com, Susan
Steele/Mechanicsburg/IBM@IBMUS, Glenn
Lehman/Mechanicsburg/IBM@IBMUS, Cora Jung/New York/IBM@IBMUS
Date: 01/19/2019 07:24 AM
Subject: Re: [rear/rear] Update to closed issue #2016 (Unable to clone
Power lpar from rear backup SLES 12 SP3) (#2019)

Hi Sebastien,
We are using XIV for our storage. We worked with the SAN team yesterday
and thought we found the problem. They made a change and we tried a
restore and it worked perfectly. Then we rebooted and tried two more
restores on the same lpar with the same image and they failed.

The first restore that worked had the following output for the multipath
device info. I don't have console output so I don't remember the
exact /dev/mapper id. That's why I used question marks.
The two times that didn't work (logs below including ioinfo output) did not
have any entry under "Listing multipath device found"

Listing multipath device found
/dev/mapper/20017380034??????
Comparing disks
Ambiguous possible target disks need manual configuration (more than one
with same size found)
Switching to manual disk layout configuration
Using /dev/sda (same size) for recreating /dev/mapper/200173800348802b3
Current disk mapping table (source => target):
/dev/mapper/200173800348802b3 => /dev/mapper/20017380034??????

Here is the ioinfo output

0 > ioinfo

!!! IOINFO: FOR IBM INTERNAL USE ONLY !!!
This tool gives you information about SCSI,SATA,SAS,and USB devices
attached to the system

Select a tool from the following

  1. SATAINFO
  2. SASINFO
  3. USBINFO
  4. FCINFO
  5. VSCSIINFO

q - quit/exit

==> 4

FCINFO Main Menu
Select a FC Node from the following list:

Location Code Pathname

  1. U9080.MHE.78F42D7-V7-C2-T1 /vdevice/vfc-client@30000002
  2. U9080.MHE.78F42D7-V7-C3-T1 /vdevice/vfc-client@30000003
  3. U9080.MHE.78F42D7-V7-C4-T1 /vdevice/vfc-client@30000004
  4. U9080.MHE.78F42D7-V7-C5-T1 /vdevice/vfc-client@30000005

q - Quit/Exit

==> 1

FC Node Menu FC Node String: /vdevice/vfc-client@30000002 FC Node WorldWidePortName: c0507609f1e20020

  1. List Attached FC Devices
  2. Select a FC Device
  3. Enable/Disable FC Adapter Debug flags

q - Quit/Exit

==> 1

  1. 5001738034880150,4000000000000000 - 114877 MB Disk drive
  2. 5001738034880141,4000000000000000 - 114877 MB Disk drive

Hit a key to continue...

FC Node Menu FC Node String: /vdevice/vfc-client@30000002 FC Node WorldWidePortName: c0507609f1e20020

  1. List Attached FC Devices
  2. Select a FC Device
  3. Enable/Disable FC Adapter Debug flags

q - Quit/Exit

==> q

FCINFO Main Menu
Select a FC Node from the following list:

Location Code Pathname

  1. U9080.MHE.78F42D7-V7-C2-T1 /vdevice/vfc-client@30000002
  2. U9080.MHE.78F42D7-V7-C3-T1 /vdevice/vfc-client@30000003
  3. U9080.MHE.78F42D7-V7-C4-T1 /vdevice/vfc-client@30000004
  4. U9080.MHE.78F42D7-V7-C5-T1 /vdevice/vfc-client@30000005

q - Quit/Exit

==> 2

FC Node Menu FC Node String: /vdevice/vfc-client@30000003 FC Node WorldWidePortName: c0507609f1e20022

  1. List Attached FC Devices
  2. Select a FC Device
  3. Enable/Disable FC Adapter Debug flags

q - Quit/Exit

==> 1

NO FC DEVICES FOUND

Hit a key to continue...

FC Node Menu FC Node String: /vdevice/vfc-client@30000003 FC Node WorldWidePortName: c0507609f1e20022

  1. List Attached FC Devices
  2. Select a FC Device
  3. Enable/Disable FC Adapter Debug flags

q - Quit/Exit

==> q

FCINFO Main Menu
Select a FC Node from the following list:

Location Code Pathname

  1. U9080.MHE.78F42D7-V7-C2-T1 /vdevice/vfc-client@30000002
  2. U9080.MHE.78F42D7-V7-C3-T1 /vdevice/vfc-client@30000003
  3. U9080.MHE.78F42D7-V7-C4-T1 /vdevice/vfc-client@30000004
  4. U9080.MHE.78F42D7-V7-C5-T1 /vdevice/vfc-client@30000005

q - Quit/Exit

==> 3

FC Node Menu FC Node String: /vdevice/vfc-client@30000004 FC Node WorldWidePortName: c0507609f1e20024

  1. List Attached FC Devices
  2. Select a FC Device
  3. Enable/Disable FC Adapter Debug flags

q - Quit/Exit

==> 1

NO FC DEVICES FOUND

Hit a key to continue...

FC Node Menu FC Node String: /vdevice/vfc-client@30000004 FC Node WorldWidePortName: c0507609f1e20024

  1. List Attached FC Devices
  2. Select a FC Device
  3. Enable/Disable FC Adapter Debug flags

q - Quit/Exit

==> q

FCINFO Main Menu
Select a FC Node from the following list:

Location Code Pathname

  1. U9080.MHE.78F42D7-V7-C2-T1 /vdevice/vfc-client@30000002
  2. U9080.MHE.78F42D7-V7-C3-T1 /vdevice/vfc-client@30000003
  3. U9080.MHE.78F42D7-V7-C4-T1 /vdevice/vfc-client@30000004
  4. U9080.MHE.78F42D7-V7-C5-T1 /vdevice/vfc-client@30000005

q - Quit/Exit

==> 4

FC Node Menu FC Node String: /vdevice/vfc-client@30000005 FC Node WorldWidePortName: c0507609f1e20026

  1. List Attached FC Devices
  2. Select a FC Device
  3. Enable/Disable FC Adapter Debug flags

q - Quit/Exit

==> 1

NO FC DEVICES FOUND

Hit a key to continue...

FC Node Menu FC Node String: /vdevice/vfc-client@30000005 FC Node WorldWidePortName: c0507609f1e20026

  1. List Attached FC Devices
  2. Select a FC Device
  3. Enable/Disable FC Adapter Debug flags

q - Quit/Exit

==> q

FCINFO Main Menu
Select a FC Node from the following list:

Location Code Pathname

  1. U9080.MHE.78F42D7-V7-C2-T1 /vdevice/vfc-client@30000002
  2. U9080.MHE.78F42D7-V7-C3-T1 /vdevice/vfc-client@30000003
  3. U9080.MHE.78F42D7-V7-C4-T1 /vdevice/vfc-client@30000004
  4. U9080.MHE.78F42D7-V7-C5-T1 /vdevice/vfc-client@30000005

q - Quit/Exit

Here is the console log of the two different restores that did not work.

Configuring Relax-and-Recover rescue system

Running 00-functions.sh...
Running 01-run-ldconfig.sh...
Running 10-console-setup.sh...
Using keymap of the original system
Running 20-check-boot-options.sh...
Running 40-start-udev-or-load-modules.sh...
Waiting for udev ... done.
Running 41-load-special-modules.sh...
Running 42-engage-scsi.sh...
Running 45-serial-console.sh...
Running 55-migrate-network-devices.sh...
No network interface mapping is specified in /etc/rear/mappings/mac
The original network interface eth0 d6:8b:e6:36:62:08 is not available

  1. eth3 d6:8b:e5:4d:db:08 ibmveth
  2. eth4 d6:8b:e5:4d:db:09 ibmveth
  3. eth5 d6:8b:e5:4d:db:0a ibmveth
  4. Skip replacing eth0 d6:8b:e6:36:62:08
    Choose replacement for eth0 d6:8b:e6:36:62:08 1
    Mapping eth0 d6:8b:e6:36:62:08 to eth3 d6:8b:e5:4d:db:08
    The original network interface eth1 d6:8b:e6:36:62:09 is not available
    1. eth4 d6:8b:e5:4d:db:09 ibmveth
  6. eth5 d6:8b:e5:4d:db:0a ibmveth
  7. Skip replacing eth1 d6:8b:e6:36:62:09
    Choose replacement for eth1 d6:8b:e6:36:62:09 2
    Mapping eth1 d6:8b:e6:36:62:09 to eth4 d6:8b:e5:4d:db:09
    The original network interface eth2 d6:8b:e6:36:62:0a is not available
      1. eth5 d6:8b:e5:4d:db:0a ibmveth
  9. Skip replacing eth2 d6:8b:e6:36:62:0a
    Choose replacement for eth2 d6:8b:e6:36:62:0a 3
    Mapping eth2 d6:8b:e6:36:62:0a to eth5 d6:8b:e5:4d:db:0a
    Reloading udev ... done.
    Running 58-start-dhclient.sh...
    Running 60-network-devices.sh...
    Running 62-routing.sh...
    Running 65-sysctl.sh...
    net.ipv6.conf.all.disable_ipv6 = 1
    net.ipv4.ip_forward = 1
    net.ipv6.conf.all.forwarding = 0
    vm.pagecache_limit_ignore_dirty = 2
    net.core.somaxconn = 4096
    net.ipv4.tcp_max_syn_backlog = 8192
    net.ipv4.tcp_tw_reuse = 1
    net.ipv4.tcp_tw_recycle = 1
    net.ipv4.tcp_syn_retries = 8
    net.ipv4.tcp_wmem = 4096 16384 4194304
    vm.min_free_kbytes = 10485760
    net.ipv6.conf.all.disable_ipv6 = 1
    net.ipv4.ip_forward = 1
    net.ipv6.conf.all.forwarding = 0
    vm.pagecache_limit_ignore_dirty = 2
    net.core.somaxconn = 4096
    net.ipv4.tcp_max_syn_backlog = 8192
    net.ipv4.tcp_tw_reuse = 1
    net.ipv4.tcp_tw_recycle = 1
    net.ipv4.tcp_syn_retries = 8
    net.ipv4.tcp_wmem = 4096 16384 4194304
    vm.min_free_kbytes = 10485760
    Running 99-makedev.sh...

Relax-and-Recover rescue system is ready

[ OK ] Started Initialize Rescue System.
[ OK ] Started Getty on tty0.
[ OK ] Reached target Login Prompts.
[ OK ] Started Serial Getty on hvc0.
[ OK ] Started Serial Getty on ttyS0.
[ OK ] Reached target Multi-User.

Relax-and-Recover 2.4-git.0.6ec9075.unknown / 2018-12-05

Relax-and-Recover comes with ABSOLUTELY NO WARRANTY; for details see
the GNU General Public License at: http://www.gnu.org/licenses/gpl.html

Host eniesdbp104p using Backup NETFS and Output ISO
Build date: Thu, 17 Jan 2019 14:22:20 -0500

Welcome to SUSE Linux Enterprise Server for SAP Applications 12 SP3
(ppc64le) - Kernel 4.4.162-94.72-default (hvc0).

SSH fingerprint: 2048 SHA256:bn+LTuNXpbRAhjgEBjQZ6rZv29K8anvt2wL7R2uXgqw
root@eniesdbp104p (RSA)

eniesdbp104p login: root

Welcome to Relax-and-Recover. Run "rear recover" to restore your system !

RESCUE eniesdbp104p:~ # rear recover
Relax-and-Recover 2.4-git.0.6ec9075.unknown / 2018-12-05
Running rear recover (PID 24570)
Using log file: /var/log/rear/rear-eniesdbp104p.log
Running workflow recover within the ReaR rescue/recovery system
Using backup archive
'/tmp/rear.Hs00F3ALINE1ns3/outputfs/iso_fs/REAR_BACKUP/backup.tar.gz'
Will do driver migration (recreating initramfs/initrd)
Calculating backup archive size
Backup archive size is 1909.38M (compressed)
Setting up multipathing
Activating multipath
multipath activated
Starting multipath daemon
multipathd started
Listing multipath device found

Comparing disks
Ambiguous possible target disks need manual configuration (more than one
with same size found)
Switching to manual disk layout configuration
Using /dev/sda (same size) for recreating /dev/mapper/200173800348802b3
Current disk mapping table (source => target):
/dev/mapper/200173800348802b3 => /dev/sda

Configuring Relax-and-Recover rescue system

Running 00-functions.sh...
Running 01-run-ldconfig.sh...
Running 10-console-setup.sh...
Using keymap of the original system
Running 20-check-boot-options.sh...
Running 40-start-udev-or-load-modules.sh...
Waiting for udev ... done.
Running 41-load-special-modules.sh...
Running 42-engage-scsi.sh...
Running 45-serial-console.sh...
Running 55-migrate-network-devices.sh...
No network interface mapping is specified in /etc/rear/mappings/mac
The original network interface eth0 d6:8b:e6:36:62:08 is not available

  1. eth1 d6:8b:e5:4d:db:09 ibmveth
  2. eth2 d6:8b:e5:4d:db:0a ibmveth
  3. eth3 d6:8b:e5:4d:db:08 ibmveth
  4. Skip replacing eth0 d6:8b:e6:36:62:08
    Choose replacement for eth0 d6:8b:e6:36:62:08 3
    Mapping eth0 d6:8b:e6:36:62:08 to eth3 d6:8b:e5:4d:db:08
    The original network interface eth1 d6:8b:e6:36:62:09 is not available
  5. eth1 d6:8b:e5:4d:db:09 ibmveth
  6. eth2 d6:8b:e5:4d:db:0a ibmveth
    1. Skip replacing eth1 d6:8b:e6:36:62:09
      Choose replacement for eth1 d6:8b:e6:36:62:09 1
      Mapping eth1 d6:8b:e6:36:62:09 to eth1 d6:8b:e5:4d:db:09
      The original network interface eth2 d6:8b:e6:36:62:0a is not available
    1. eth2 d6:8b:e5:4d:db:0a ibmveth
    1. Skip replacing eth2 d6:8b:e6:36:62:0a
      Choose replacement for eth2 d6:8b:e6:36:62:0a 2
      Mapping eth2 d6:8b:e6:36:62:0a to eth2 d6:8b:e5:4d:db:0a
      Reloading udev ... done.
      Running 58-start-dhclient.sh...
      Running 60-network-devices.sh...
      Running 62-routing.sh...
      Running 65-sysctl.sh...
      net.ipv6.conf.all.disable_ipv6 = 1
      net.ipv4.ip_forward = 1
      net.ipv6.conf.all.forwarding = 0
      vm.pagecache_limit_ignore_dirty = 2
      net.core.somaxconn = 4096
      net.ipv4.tcp_max_syn_backlog = 8192
      net.ipv4.tcp_tw_reuse = 1
      net.ipv4.tcp_tw_recycle = 1
      net.ipv4.tcp_syn_retries = 8
      net.ipv4.tcp_wmem = 4096 16384 4194304
      vm.min_free_kbytes = 10485760
      net.ipv6.conf.all.disable_ipv6 = 1
      net.ipv4.ip_forward = 1
      net.ipv6.conf.all.forwarding = 0
      vm.pagecache_limit_ignore_dirty = 2
      net.core.somaxconn = 4096
      net.ipv4.tcp_max_syn_backlog = 8192
      net.ipv4.tcp_tw_reuse = 1
      net.ipv4.tcp_tw_recycle = 1
      net.ipv4.tcp_syn_retries = 8
      net.ipv4.tcp_wmem = 4096 16384 4194304
      vm.min_free_kbytes = 10485760
      Running 99-makedev.sh...

Relax-and-Recover rescue system is ready

[ OK ] Started Initialize Rescue System.
[ OK ] Started Getty on tty0.
[ OK ] Reached target Login Prompts.
[ OK ] Reached target Multi-User.
[ OK ] Started Serial Getty on ttyS0.
[ OK ] Started Serial Getty on hvc0.

Relax-and-Recover 2.4-git.0.6ec9075.unknown / 2018-12-05

Relax-and-Recover comes with ABSOLUTELY NO WARRANTY; for details see
the GNU General Public License at: http://www.gnu.org/licenses/gpl.html

Host eniesdbp104p using Backup NETFS and Output ISO
Build date: Thu, 17 Jan 2019 14:22:20 -0500

Welcome to SUSE Linux Enterprise Server for SAP Applications 12 SP3
(ppc64le) - Kernel 4.4.162-94.72-default (hvc0).

SSH fingerprint: 2048 SHA256:DNyikRzuULk3bpE1wwyoXX/6b9qV0kBUW0j32NOrdkY
root@eniesdbp104p (RSA)

eniesdbp104p login: root

Welcome to Relax-and-Recover. Run "rear recover" to restore your system !

RESCUE eniesdbp104p:~ # rear recover
Relax-and-Recover 2.4-git.0.6ec9075.unknown / 2018-12-05
Running rear recover (PID 24593)
Using log file: /var/log/rear/rear-eniesdbp104p.log
Running workflow recover within the ReaR rescue/recovery system
Using backup archive
'/tmp/rear.3HmcDWcPOuSWy4R/outputfs/iso_fs/REAR_BACKUP/backup.tar.gz'
Will do driver migration (recreating initramfs/initrd)
Calculating backup archive size
Backup archive size is 1909.38M (compressed)
Setting up multipathing
Activating multipath
multipath activated
Starting multipath daemon
multipathd started
Listing multipath device found
SIBM_2810XIV_780348802C6 dm-0 IBM,2810XIV size=112G
Comparing disks
Ambiguous possible target disks need manual configuration (more than one
with same size found)
Switching to manual disk layout configuration
Using /dev/mapper/SIBM_2810XIV_780348802C6 (same size) for
recreating /dev/mapper/200173800348802b3
Current disk mapping table (source => target):
/dev/mapper/200173800348802b3 => /dev/mapper/SIBM_2810XIV_780348802C6

Dan Wagner
Senior Enterprise System Specialist

(717) 580-3844 Mobile
(717) 409-5858 Office
dwagner1@us.ibm.com

IBM Services

From: "Sébastien Chabrolles" notifications@github.com
To: rear/rear rear@noreply.github.com
Cc: dewagner1 dwagner1@us.ibm.com, Mention
mention@noreply.github.com
Date: 01/18/2019 03:36 AM
Subject: Re: [rear/rear] Update to closed issue #2016 (Unable to clone
Power lpar from rear backup SLES 12 SP3) (#2019)

@dewagner1,

My first guess is still a SAN / FC issue. To validate, the best would be to
boot your LPAR in SMS and run ioinfo via the OK prompt.

1- From SMS menu to OK prompt
Press 0 when entering SMS menu 1st page.

                                         X = eXit System Management

Services

Type menu item number and press Enter or select Navigation key:
Open Completed.

Main Menu

  1. Select Language
  2. Setup Remote IPL (Initial Program Load)
  3. I/O Device Information
  4. Select Console
  5. Select Boot Options

Navigation Keys:

                                         X = eXit System Management

Services

Type menu item number and press Enter or select Navigation key:0

                .--------------------------------------.
                |  Exit SMS to OK> Prompt? (Y)es/(N)o  |
                `--------------------------------------'

ok
0 >

2-Run ioinfo on each fiber adapter to get the list of disk/partitions path
over the SAN network

ok
0 > ioinfo

!!! IOINFO: FOR IBM INTERNAL USE ONLY !!!
This tool gives you information about SCSI,SATA,SAS,and USB devices
attached to the system

Select a tool from the following

  1. SATAINFO
  2. SASINFO
  3. USBINFO
  4. FCINFO
  5. VSCSIINFO

q - quit/exit

==> 4

FCINFO Main Menu
Select a FC Node from the following list:

Location Code Pathname

  1. U8284.22A.217097V-V21-C13-T1 /vdevice/vfc-client@3000000d
  2. U8284.22A.217097V-V21-C14-T1 /vdevice/vfc-client@3000000e

q - Quit/Exit

==> 1

Press 1 to list devices seen by the first Adapter

==> 1

  1. 500507680c21099f,0 - 51200 MB Disk drive (bootable)
  2. 500507680c21099f,1000000000000 - 10240 MB Disk drive
  3. 500507680c21099f,2000000000000 - 10240 MB Disk drive
  4. 500507680c51099f,0 - 51200 MB Disk drive (bootable)
  5. 500507680c51099f,1000000000000 - 10240 MB Disk drive
  6. 500507680c51099f,2000000000000 - 10240 MB Disk drive
  7. 500507680c2109a6,0 - 51200 MB Disk drive (bootable)
  8. 500507680c2109a6,1000000000000 - 10240 MB Disk drive
  9. 500507680c2109a6,2000000000000 - 10240 MB Disk drive
  10. 500507680c5109a6,0 - 51200 MB Disk drive (bootable)
  11. 500507680c5109a6,1000000000000 - 10240 MB Disk drive
  12. 500507680c5109a6,2000000000000 - 10240 MB Disk drive

Hit a key to continue...

Then Quit q and run the same thing on the other adapter

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

schabrolles commented at 2019-01-22 10:16:

@dewagner1,

The "Ambiguous possible target disks need manual configuration (more than one with same size found)" line is not a problem here. ReaR informs you that it can't find the exact same disk and founds several possible targets.... simply because you are running on a different server (migration) with multipath (several possible target devices). ReaR just need you to validate the target proposed.

I recommend to not unmap the disk to reduce the number of the possible targets. When you do that, multipath will switch automatically to single Path (/dev/sdX).

Could you please tell me if everything works now (even with the "Ambiguous possible target disks..." line)

...
Setting up multipathing
Activating multipath
multipath activated
Starting multipath daemon
multipathd started
Listing multipath device found
200173800348802b2 dm-0 IBM,2810XIV size=112G
Comparing disks
Ambiguous possible target disks need manual configuration (more than one
with same size found)
Switching to manual disk layout configuration
Using /dev/mapper/200173800348802b2 (same size) for
recreating /dev/mapper/200173800348802b3
Current disk mapping table (source => target):
/dev/mapper/200173800348802b3 => /dev/mapper/200173800348802b2
...

dewagner1 commented at 2019-01-22 12:18:

Sebastian,
There is only one disk mapped to this server. I went through the procedure
again this morning and have attached the console and debug log output.

(See attached file: 20190122 Eniesdbp102 Restore.txt)

The first attempt did not find a multipath device so I didn't continue
because I knew it would fail. You can see that multipath was not running
and the pvs command gave some strange errors.

However, the second attempt did work and you can see the "Ambiuous possible
target disks..." message is there.

Also, the rear restore failed on another server we configured with only one
path to one disk. See console output below.

Incidently, we were able to do a clean install from the SLES DVD on both of
these servers. Let me know if I can provide any additional information to
help debug this.

Console output

multipathd started
Listing multipath device found

Comparing disks
Device mapper!200173800348802b3 does not exist (manual configuration
needed)
Switching to manual disk layout configuration
Using /dev/sda (same size) for recreating /dev/mapper/200173800348802b3
Current disk mapping table (source => target):
/dev/mapper/200173800348802b3 => /dev/sda

UserInput -I LAYOUT_MIGRATION_CONFIRM_MAPPINGS needed
in /usr/share/rear/layout/prepare/default/300_map_disks.sh line 275
Confirm or edit the disk mapping

  1. Confirm disk mapping and continue 'rear recover'
  2. n/a
  3. Edit disk mapping (/var/lib/rear/layout/disk_mappings)
  4. Use Relax-and-Recover shell and return back to here
  5. Abort 'rear recover'
    (default '1' timeout 300 seconds)
    1
    UserInput: Valid choice number result 'Confirm disk mapping and continue
    'rear recover''
    User confirmed disk mapping
    Failed to apply layout mappings to /var/lib/rear/layout/disklayout.conf
    for /dev/sda (probably no mapping for /dev/sda
    in /var/lib/rear/layout/disk_mappings)
    ERROR: Failed to apply disklayout mappings
    to /var/lib/rear/layout/disklayout.conf
    Some latest log messages since the last called script
    320_apply_mappings.sh:
    2019-01-22 06:33:31.601636240 Including
    layout/prepare/default/320_apply_mappings.sh
    2019-01-22 06:33:31.602645375 Entering debugscripts mode via 'set -x'.
    2019-01-22 06:33:31.622237844 Failed to apply layout mappings
    to /var/lib/rear/layout/disklayout.conf for /dev/sda (probably no mapping
    for /dev/sda in /var/lib/rear/layout/disk_mappings)
    Aborting due to an error, check /var/log/rear/rear-eniesdbp104p.log for
    details
    Exiting rear recover (PID 24346) and its descendant processes
    Running exit tasks
    You should also rm -Rf /tmp/rear.3QbnZnCC5EtVl9F
    Terminated
    RESCUE eniesdbp104p:~ #

Dan Wagner
Senior Enterprise System Specialist

(717) 580-3844 Mobile
(717) 409-5858 Office
dwagner1@us.ibm.com

IBM Services

From: "Sébastien Chabrolles" notifications@github.com
To: rear/rear rear@noreply.github.com
Cc: dewagner1 dwagner1@us.ibm.com, Mention
mention@noreply.github.com
Date: 01/22/2019 05:17 AM
Subject: Re: [rear/rear] Update to closed issue #2016 (Unable to clone
Power lpar from rear backup SLES 12 SP3) (#2019)

@dewagner1,

The "Ambiguous possible target disks need manual configuration (more than
one with same size found)" line is not a problem here. ReaR informs you
that it can't find the exact same disk and founds several possible
targets.... simply because you are running on a different server
(migration) with multipath (several possible target devices). ReaR just
need you to validate the target proposed.

I recommend to not unmap the disk to reduce the number of the possible
targets. When you do that, multipath will switch automatically to single
Path (/dev/sdX).

Could you please tell me if everything works now (even with the "Ambiguous
possible target disks..." line)

...
Setting up multipathing
Activating multipath
multipath activated
Starting multipath daemon
multipathd started
Listing multipath device found
200173800348802b2 dm-0 IBM,2810XIV size=112G
Comparing disks
Ambiguous possible target disks need manual configuration (more than one
with same size found)
Switching to manual disk layout configuration
Using /dev/mapper/200173800348802b2 (same size) for
recreating /dev/mapper/200173800348802b3
Current disk mapping table (source => target):
/dev/mapper/200173800348802b3 => /dev/mapper/200173800348802b2
...

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

1ST TRY

RESCUE eniesdbp104p:~ # rear -dDv recover
Relax-and-Recover 2.4-git.0.6ec9075.unknown / 2018-12-05
Running rear recover (PID 24958)
Using log file: /var/log/rear/rear-eniesdbp104p.log
Running workflow recover within the ReaR rescue/recovery system
Using backup archive '/tmp/rear.U4Lsmb4pp4DEyYu/outputfs/iso_fs/REAR_BACKUP/backup.tar.gz'
Will do driver migration (recreating initramfs/initrd)
Calculating backup archive size
Backup archive size is 1909.38M (compressed)
Setting up multipathing
Activating multipath
multipath activated
Starting multipath daemon
multipathd started
Listing multipath device found

Comparing disks
Ambiguous possible target disks need manual configuration (more than one with same size found)
Switching to manual disk layout configuration
Using /dev/sda (same size) for recreating /dev/mapper/200173800348802b3
Current disk mapping table (source => target):
/dev/mapper/200173800348802b3 => /dev/sda

UserInput -I LAYOUT_MIGRATION_CONFIRM_MAPPINGS needed in /usr/share/rear/layout/prepare/default/300_map_disks.sh line 275
Confirm or edit the disk mapping

  1. Confirm disk mapping and continue 'rear recover'
  2. n/a
  3. Edit disk mapping (/var/lib/rear/layout/disk_mappings)
  4. Use Relax-and-Recover shell and return back to here
  5. Abort 'rear recover'
    (default '1' timeout 300 seconds)
    4
    UserInput: Valid choice number result 'Use Relax-and-Recover shell and return back to here'

Welcome to Relax-and-Recover.

rear> pvs
/run/lvm/lvmetad.socket: connect failed: No such file or directory
WARNING: Failed to connect to lvmetad. Falling back to internal scanning.
Found duplicate PV emSwP2fCyavSeeBOCALUdOqYCw41Lc01: using /dev/sdb3 not /dev/sda3
Using duplicate PV /dev/sdb3 which is last seen, replacing /dev/sda3
Found duplicate PV emSwP2fCyavSeeBOCALUdOqYCw41Lc01: using /dev/sdc3 not /dev/sdb3
Using duplicate PV /dev/sdc3 which is last seen, replacing /dev/sdb3
Found duplicate PV emSwP2fCyavSeeBOCALUdOqYCw41Lc01: using /dev/sdd3 not /dev/sdc3
Using duplicate PV /dev/sdd3 which is last seen, replacing /dev/sdc3
PV VG Fmt Attr PSize PFree
/dev/sdd3 system_vg lvm2 a-- 111.43g 26.43g
rear> multipath -ll
rear>

RESCUE eniesdbp104p:~ # cat /var/log/rear/rear-eniesdbp104p.log
2019-01-22 06:29:00.825940022 Relax-and-Recover 2.4-git.0.6ec9075.unknown / 2018-12-05
2019-01-22 06:29:00.827183229 Running rear recover (PID 24958)
2019-01-22 06:29:00.828387464 Command line options: /bin/rear -dDv recover
2019-01-22 06:29:00.829454059 Using log file: /var/log/rear/rear-eniesdbp104p.log
2019-01-22 06:29:00.830809668 Current set of flags is 'hB'
2019-01-22 06:29:00.831892086 The debugscripts flags are 'x'
2019-01-22 06:29:00.833015547 Combining configuration files
2019-01-22 06:29:00.834747961 Including /etc/rear/os.conf
2019-01-22 06:29:00.836303845 Entering debugscripts mode via 'set -x'.

  • source /etc/rear/os.conf
    ++ OS_VENDOR=SUSE_LINUX
    ++ OS_VERSION=12.3
    ++ ARCH=Linux-ppc64le
    ++ OS=GNU/Linux
    ++ OS_VERSION=12.3
    ++ OS_VENDOR=SUSE_LINUX
    ++ OS_VENDOR_VERSION=SUSE_LINUX/12.3
    ++ OS_VENDOR_ARCH=SUSE_LINUX/ppc64le
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:00.841023300 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:00.841023300 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-22 06:29:00.841023300 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-22 06:29:00.844941438 Including conf/Linux-ppc64le.conf
    2019-01-22 06:29:00.846111294 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/conf/Linux-ppc64le.conf
    ++ REQUIRED_PROGS=("${REQUIRED_PROGS[@]}" sfdisk)
    ++ PROGS=("${PROGS[@]}" bootlist pseries_platform nvram ofpathname bc agetty)
    +++ awk '/platform/ {print $NF}'
    ++ [[ pSeries != PowerNV ]]
    ++ test ''
    ++ FIRMWARE_FILES=('no')
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:00.854737254 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:00.854737254 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-22 06:29:00.854737254 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-22 06:29:00.857159566 Including conf/GNU/Linux.conf
    2019-01-22 06:29:00.858372545 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/conf/GNU/Linux.conf
    ++ REQUIRED_PROGS=("${REQUIRED_PROGS[@]:-}" chroot ip less parted route readlink)
    ++ PROGS=(${PROGS[@]:-} partprobe fdisk cfdisk sfdisk)
    ++ PROGS=(${PROGS[@]:-} rpc.statd rpcbind mknod blkid vol_id udev_volume_id portmap rpcinfo tac reboot halt shutdown killall5 killall tee ifconfig nslookup ifenslave ifrename nameif klogd syslog-ng syslogd rsyslogd date rmdir init telinit ethtool expand insmod modprobe lsmod true false mingetty agetty getty rmmod hostname uname sleep logger pstree ln dirname basename tty ping netstat free traceroute vi pico nano rmmod df dmesg du gzip netcat curl top iptraf joe getent id ldd strace rsync tail head find md5sum mkfs mkfs.ext2 mkfs.ext3 mkfs.ext4 mkfs.ext4dev mkfs.jfs mkfs.xfs mkfs.reiserfs mkfs.vfat mkfs.btrfs mkreiserfs fsck fsck.ext2 fsck.ext3 fsck.ext4 fsck.ext4dev fsck.xfs fsck.reiserfs reiserfsck fsck.btrfs findmnt btrfsck tune2fs tune4fs xfs_admin xfs_db xfs_repair xfs_info xfs_growfs btrfs jfs_tune reiserfstune egrep fgrep chmod chown stat mkswap swapon swapoff mknod touch scsi_id lsscsi logd initctl lspci usleep mktemp /bin/true strace which mkfifo seq openvt poweroff chacl getfacl setfacl attr getfattr setfattr mpath_wait xargs sg_inq env w dosfslabel sysctl blockdev lsblk)
    ++ LIBS=(${LIBS[@]:-} /lib*/libnss_dns* /lib*/libnss_files* /lib//libnss_dns /lib//libnss_files /lib*/libgcc_s* /lib*/libresolv* /usr/lib*/rsyslog/so /lib/rsyslog/so /usr/lib/syslog-ng/* /lib*/libnsspem.so* /usr/lib*/libnsspem.so* /lib*/libfreebl*.so* /usr/lib*/libfreebl*.so* /lib*/libnss3.so* /usr/lib*/libnss3.so* /lib*/libnssutil3.so* /usr/lib*/libnssutil3.so* /lib*/libsoftokn3.so* /usr/lib*/libsoftokn3.so* /lib*/libsqlite3.so* /usr/lib*/libsqlite3.so* /lib*/libfreeblpriv3.so* /usr/lib*/libfreeblpriv3.so* /lib*/libssl.so* /usr/lib*/libssl.so* /lib*/libnssdbm3.so* /usr/lib*/libnssdbm3.so*)
    ++ MODULES=(${MODULES[@]:-} vfat nls_iso8859_1 nls_utf8 nls_cp437 af_packet unix nfs nfsv4 nfsv3 lockd sunrpc cifs usbcore usb_storage usbhid sr_mod ide_cd cdrom uhci_hcd ehci_hcd xhci_hcd ohci_hcd zlib zlib-inflate zlib-deflate libcrc32c crc32c crc32c-intel)
    ++ COPY_AS_IS=(${COPY_AS_IS[@]:-} /dev /etc/inputr[c] /etc/protocols /etc/services /etc/rpc /etc/termcap /etc/terminfo /lib*/terminfo /usr/share/terminfo /etc/netconfig /etc/mke2fs.conf /etc/-release /etc/localtime /etc/magic /usr/share/misc/magic /etc/dracut.conf /etc/dracut.conf.d /usr/lib/dracut /sbin/modprobe.ksplice-orig /etc/sysctl.conf /etc/sysctl.d /etc/e2fsck.conf)
    ++ COPY_AS_IS=("${COPY_AS_IS[@]}" '/etc/ssl/certs/    ' '/etc/pki/' '/usr/lib/ssl/' '/usr/share/ca-certificates/' '/etc/ca-certificates/')
    ++ COPY_AS_IS_EXCLUDE=(${COPY_AS_IS_EXCLUDE[@]:-} dev/shm/*)
    ++ COPY_AS_IS_EXCLUDE=("${COPY_AS_IS_EXCLUDE[@]}" '/etc/pki/tls/private' '/etc/pki/CA/private' '/etc/pki/nssdb/key*.db' '/usr/lib/ssl/private')
    ++ KERNEL_CMDLINE=' selinux=0'
    ++ CLONE_USERS=("${CLONE_USERS[@]:-}" daemon rpc usbmuxd usbmux vcsa nobody dbus)
    ++ CLONE_GROUPS=("${CLONE_GROUPS[@]:-}" tty usbmuxd usbmux fuse kvm oinstall dbus)
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:00.866052417 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:00.866052417 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-22 06:29:00.866052417 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-22 06:29:00.868108887 Including conf/SUSE_LINUX.conf
    2019-01-22 06:29:00.869305604 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/conf/SUSE_LINUX.conf
    ++ COPY_AS_IS=("${COPY_AS_IS[@]}" /etc/sysconfig/network/config /etc/sysconfig/network/scripts/functions)
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:00.874792185 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:00.874792185 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-22 06:29:00.874792185 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-22 06:29:00.879437191 Including /etc/rear/local.conf
    2019-01-22 06:29:00.880548745 Entering debugscripts mode via 'set -x'.
  • source /etc/rear/local.conf
    ++ AUTOEXCLUDE_MULTIPATH=n
    ++ BOOT_OVER_SAN=y
    ++ REAR_INITRD_COMPRESSION=lzma
    ++ OUTPUT=ISO
    ++ ISO_MAX_SIZE=4000
    ++ BACKUP=NETFS
    ++ BACKUP_URL=iso:///iso_fs/REAR_BACKUP
    ++ ISO_DIR=/iso_fs/REAR_ISO
    ++ TMPDIR=/iso_fs/REAR_TEMP
    ++ OUTPUT_URL=null
    ++ BOOT_FROM_SAN=y
    ++ EXCLUDE_MOUNTPOINTS=(/iso_fs /opt/IBM/ITM /cv /opt/splunkforwarder /opt/teamquest /var/opt/BESClient /usr/sap /hana/data /hana/log /hana/shared /usr/sap/basis /usr/sap/srm /PA_backup)
    ++ EXCLUDE_COMPONENTS=(/dev/mapper/200173800348802ba /dev/mapper/200173800348802bb /dev/mapper/200173800348802bc /dev/mapper/200173800348802bd /dev/mapper/200173800348802be /dev/mapper/200173800348802bf /dev/mapper/200173800348802c0 /dev/mapper/200173800348802c1 /dev/mapper/200173800348802c2 /dev/mapper/200173800348802c3 /dev/mapper/200173800348802c4 /dev/mapper/200173800348802c5)
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:00.885043567 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:00.885043567 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-22 06:29:00.885043567 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-22 06:29:00.889299429 Including /etc/rear/rescue.conf
    2019-01-22 06:29:00.890593853 Entering debugscripts mode via 'set -x'.
  • source /etc/rear/rescue.conf
    ++ SHARE_DIR=/usr/share/rear
    ++ CONFIG_DIR=/etc/rear
    ++ VAR_DIR=/var/lib/rear
    ++ LOG_DIR=/var/log/rear
    ++ BACKUP_PROG_OPTIONS=(--anchored --xattrs --xattrs-include=security.capability --xattrs-include=security.selinux --acls)
    ++ NETFS_KEEP_OLD_BACKUP_COPY=
    ++ NETFS_PREFIX=eniesdbp104p
    ++ NETFS_RESTORE_CAPABILITIES=([0]="No")
    ++ unset TMPDIR
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:00.894968955 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:00.894968955 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-22 06:29:00.894968955 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-22 06:29:00.898620935 ======================
    2019-01-22 06:29:00.899618142 Running 'init' stage
    2019-01-22 06:29:00.900684054 ======================
    2019-01-22 06:29:00.907303505 Including init/default/005_verify_os_conf.sh
    2019-01-22 06:29:00.908463468 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/init/default/005_verify_os_conf.sh
    ++ [[ ! -f /etc/rear/os.conf ]]
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:00.912613019 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:00.912613019 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-22 06:29:00.912613019 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-22 06:29:00.916981871 Including init/default/010_set_drlm_env.sh
    2019-01-22 06:29:00.918103758 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/init/default/010_set_drlm_env.sh
    ++ is_true n
    ++ case "$1" in
    ++ return 1
    ++ return 0
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:00.923246239 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:00.923246239 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-22 06:29:00.923246239 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-22 06:29:00.927440114 Including init/default/030_update_recovery_system.sh
    2019-01-22 06:29:00.928530927 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/init/default/030_update_recovery_system.sh
    ++ test ''
    ++ return 0
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:00.932676242 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:00.932676242 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-22 06:29:00.932676242 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-22 06:29:00.937227943 Including init/default/050_check_rear_recover_mode.sh
    2019-01-22 06:29:00.938275187 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/init/default/050_check_rear_recover_mode.sh
    ++ test -f /etc/rear-release
    ++ case "$WORKFLOW" in
    ++ LogPrint 'Running workflow recover within the ReaR rescue/recovery system'
    ++ Log 'Running workflow recover within the ReaR rescue/recovery system'
    +++ date '+%Y-%m-%d %H:%M:%S.%N '
    ++ local 'timestamp=2019-01-22 06:29:00.942418241 '
    ++ test 1 -gt 0
    ++ echo '2019-01-22 06:29:00.942418241 Running workflow recover within the ReaR rescue/recovery system'
    2019-01-22 06:29:00.942418241 Running workflow recover within the ReaR rescue/recovery system
    ++ Print 'Running workflow recover within the ReaR rescue/recovery system'
    ++ test 1
    ++ echo -e 'Running workflow recover within the ReaR rescue/recovery system'
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:00.943881554 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:00.943881554 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-22 06:29:00.943881554 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-22 06:29:00.948368214 Including init/default/950_check_missing_programs.sh
    2019-01-22 06:29:00.949468073 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/init/default/950_check_missing_programs.sh
    ++ missing_progs=('')
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary /bin/rear
    ++ for bin in '$@'
    ++ type /bin/rear
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary awk
    ++ for bin in '$@'
    ++ type awk
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary bash
    ++ for bin in '$@'
    ++ type bash
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary bc
    ++ for bin in '$@'
    ++ type bc
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary cat
    ++ for bin in '$@'
    ++ type cat
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary cmp
    ++ for bin in '$@'
    ++ type cmp
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary cp
    ++ for bin in '$@'
    ++ type cp
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary cpio
    ++ for bin in '$@'
    ++ type cpio
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary cut
    ++ for bin in '$@'
    ++ type cut
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary dd
    ++ for bin in '$@'
    ++ type dd
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary diff
    ++ for bin in '$@'
    ++ type diff
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary df
    ++ for bin in '$@'
    ++ type df
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary dumpkeys
    ++ for bin in '$@'
    ++ type dumpkeys
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary echo
    ++ for bin in '$@'
    ++ type echo
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary expr
    ++ for bin in '$@'
    ++ type expr
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary file
    ++ for bin in '$@'
    ++ type file
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary getopt
    ++ for bin in '$@'
    ++ type getopt
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary grep
    ++ for bin in '$@'
    ++ type grep
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary join
    ++ for bin in '$@'
    ++ type join
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary kbd_mode
    ++ for bin in '$@'
    ++ type kbd_mode
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary loadkeys
    ++ for bin in '$@'
    ++ type loadkeys
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary ls
    ++ for bin in '$@'
    ++ type ls
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary mkdir
    ++ for bin in '$@'
    ++ type mkdir
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary mount
    ++ for bin in '$@'
    ++ type mount
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary mv
    ++ for bin in '$@'
    ++ type mv
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary pidof
    ++ for bin in '$@'
    ++ type pidof
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary ps
    ++ for bin in '$@'
    ++ type ps
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary pwd
    ++ for bin in '$@'
    ++ type pwd
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary rm
    ++ for bin in '$@'
    ++ type rm
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary sed
    ++ for bin in '$@'
    ++ type sed
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary sort
    ++ for bin in '$@'
    ++ type sort
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary strings
    ++ for bin in '$@'
    ++ type strings
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary sync
    ++ for bin in '$@'
    ++ type sync
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary tar
    ++ for bin in '$@'
    ++ type tar
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary test
    ++ for bin in '$@'
    ++ type test
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary tr
    ++ for bin in '$@'
    ++ type tr
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary umount
    ++ for bin in '$@'
    ++ type umount
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary uniq
    ++ for bin in '$@'
    ++ type uniq
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary wc
    ++ for bin in '$@'
    ++ type wc
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary sfdisk
    ++ for bin in '$@'
    ++ type sfdisk
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary chroot
    ++ for bin in '$@'
    ++ type chroot
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary ip
    ++ for bin in '$@'
    ++ type ip
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary less
    ++ for bin in '$@'
    ++ type less
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary parted
    ++ for bin in '$@'
    ++ type parted
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary route
    ++ for bin in '$@'
    ++ type route
    ++ return 0
    ++ for prog in '"${REQUIRED_PROGS[@]}"'
    ++ has_binary readlink
    ++ for bin in '$@'
    ++ type readlink
    ++ return 0
    ++ contains_visible_char ''
    +++ tr -d -c '[:graph:]'
    ++ test ''
  • source_return_code=1
  • test 0 -eq 1
  • Debug 'Source function: '''source /usr/share/rear/init/default/950_check_missing_programs.sh''' returns 1'
  • test 1
  • Log 'Source function: '''source /usr/share/rear/init/default/950_check_missing_programs.sh''' returns 1'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:00.959127844 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:00.959127844 Source function: '''source /usr/share/rear/init/default/950_check_missing_programs.sh''' returns 1'
    2019-01-22 06:29:00.959127844 Source function: 'source /usr/share/rear/init/default/950_check_missing_programs.sh' returns 1
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:00.960282421 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:00.960282421 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-22 06:29:00.960282421 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-22 06:29:00.962259470 Finished running 'init' stage in 0 seconds
    2019-01-22 06:29:00.967082899 Using build area '/tmp/rear.U4Lsmb4pp4DEyYu'
    mkdir: created directory '/tmp/rear.U4Lsmb4pp4DEyYu/rootfs'
    mkdir: created directory '/tmp/rear.U4Lsmb4pp4DEyYu/tmp'
    2019-01-22 06:29:00.970448787 Running recover workflow
    2019-01-22 06:29:00.971677021 ======================
    2019-01-22 06:29:00.972615488 Running 'setup' stage
    2019-01-22 06:29:00.973661994 ======================
    2019-01-22 06:29:00.980233933 Including setup/default/005_ssh_agent_start.sh
    2019-01-22 06:29:00.981282103 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/setup/default/005_ssh_agent_start.sh
    ++ has_binary ssh-agent
    ++ for bin in '$@'
    ++ type ssh-agent
    ++ return 0
    ++ has_binary ssh
    ++ for bin in '$@'
    ++ type ssh
    ++ return 0
    +++ type -P ssh
    ++ grep -iq AddKeysToAgent /sbin/ssh
    ++ Log 'Starting up ssh-agent'
    +++ date '+%Y-%m-%d %H:%M:%S.%N '
    ++ local 'timestamp=2019-01-22 06:29:00.988634630 '
    ++ test 1 -gt 0
    ++ echo '2019-01-22 06:29:00.988634630 Starting up ssh-agent'
    2019-01-22 06:29:00.988634630 Starting up ssh-agent
    ++ AddExitTask 'ssh-agent -k >/dev/null'
    ++ EXIT_TASKS=("$*" "${EXIT_TASKS[@]}")
    ++ Debug 'Added '''ssh-agent -k >/dev/null''' as an exit task'
    ++ test 1
    ++ Log 'Added '''ssh-agent -k >/dev/null''' as an exit task'
    +++ date '+%Y-%m-%d %H:%M:%S.%N '
    ++ local 'timestamp=2019-01-22 06:29:00.990209783 '
    ++ test 1 -gt 0
    ++ echo '2019-01-22 06:29:00.990209783 Added '''ssh-agent -k >/dev/null''' as an exit task'
    2019-01-22 06:29:00.990209783 Added 'ssh-agent -k >/dev/null' as an exit task
    +++ ssh-agent -s
    ++ eval 'SSH_AUTH_SOCK=/tmp/ssh-KdbYToEibAhs/agent.25238; export SSH_AUTH_SOCK;
    SSH_AGENT_PID=25239; export SSH_AGENT_PID;
    echo Agent pid 25239;'
    +++ SSH_AUTH_SOCK=/tmp/ssh-KdbYToEibAhs/agent.25238
    +++ export SSH_AUTH_SOCK
    +++ SSH_AGENT_PID=25239
    +++ export SSH_AGENT_PID
    +++ echo Agent pid 25239
    Agent pid 25239
    ++ echo -e '\nHost *\nAddKeysToAgent yes\n'
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:00.994568959 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:00.994568959 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-22 06:29:00.994568959 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-22 06:29:00.999096152 Including setup/default/010_pre_recovery_script.sh
    2019-01-22 06:29:01.000373915 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/setup/default/010_pre_recovery_script.sh
    ++ test ''
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:01.004634810 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:01.004634810 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-22 06:29:01.004634810 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-22 06:29:01.006751702 Finished running 'setup' stage in 1 seconds
    2019-01-22 06:29:01.008388935 ======================
    2019-01-22 06:29:01.009570271 Running 'verify' stage
    2019-01-22 06:29:01.010687680 ======================
    2019-01-22 06:29:01.017697516 Including verify/default/020_cciss_scsi_engage.sh
    2019-01-22 06:29:01.018774559 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/verify/default/020_cciss_scsi_engage.sh
    ++ grep -q '^cciss ' /proc/modules
    ++ return
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:01.023797872 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:01.023797872 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-22 06:29:01.023797872 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-22 06:29:01.028789165 Including verify/default/020_translate_url.sh
    2019-01-22 06:29:01.030347510 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/verify/default/020_translate_url.sh
    ++ [[ -n '' ]]
    ++ [[ -n '' ]]
    ++ [[ -n '' ]]
    ++ [[ -n '' ]]
    ++ [[ -n '' ]]
    ++ [[ -n '' ]]
    ++ [[ -n '' ]]
    ++ [[ -n '' ]]
    ++ [[ -n '' ]]
    ++ [[ -n '' ]]
    ++ [[ -z '' ]]
    ++ [[ -z null ]]
    ++ [[ -z null ]]
    ++ [[ -z '' ]]
    ++ [[ -z null ]]
    ++ [[ -z '' ]]
    ++ [[ -z null ]]
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:01.035003881 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:01.035003881 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-22 06:29:01.035003881 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-22 06:29:01.039580311 Including verify/default/030_translate_tape.sh
    2019-01-22 06:29:01.040706257 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/verify/default/030_translate_tape.sh
    ++ [[ -n iso:///iso_fs/REAR_BACKUP ]]
    +++ url_scheme iso:///iso_fs/REAR_BACKUP
    +++ local url=iso:///iso_fs/REAR_BACKUP
    +++ local scheme=iso
    +++ echo iso
    +++ grep -q :
    +++ echo iso
    ++ backup_scheme=iso
    ++ [[ iso == tape ]]
    ++ [[ iso == obdr ]]
    ++ [[ -z iso:///iso_fs/REAR_BACKUP ]]
    ++ [[ -z null ]]
    ++ [[ -n '' ]]
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:01.047018075 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:01.047018075 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-22 06:29:01.047018075 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-22 06:29:01.052010584 Including verify/default/040_validate_variables.sh
    2019-01-22 06:29:01.053189067 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/verify/default/040_validate_variables.sh
    ++ is_positive_integer 30
    ++ local argument=30
    ++ test 30 -gt 0
    ++ echo 30
    ++ return 0
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:01.058221096 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:01.058221096 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-22 06:29:01.058221096 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-22 06:29:01.062525909 Including verify/NETFS/default/050_check_NETFS_requirements.sh
    2019-01-22 06:29:01.063722618 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/verify/NETFS/default/050_check_NETFS_requirements.sh
    ++ [[ -n iso:///iso_fs/REAR_BACKUP ]]
    ++ StopIfError 'You must specify either BACKUP_URL or BACKUP_MOUNTCMD and BACKUP_UMOUNTCMD !'
    ++ (( 0 != 0 ))
    ++ [[ -n iso:///iso_fs/REAR_BACKUP ]]
    +++ url_scheme iso:///iso_fs/REAR_BACKUP
    +++ local url=iso:///iso_fs/REAR_BACKUP
    +++ local scheme=iso
    +++ echo iso
    +++ grep -q :
    +++ echo iso
    ++ local scheme=iso
    +++ url_hostname iso:///iso_fs/REAR_BACKUP
    +++ local url=iso:///iso_fs/REAR_BACKUP
    +++ local url_without_scheme=/iso_fs/REAR_BACKUP
    +++ local authority_part=
    +++ local host_and_port=
    +++ echo
    ++ local hostname=
    +++ url_path iso:///iso_fs/REAR_BACKUP
    +++ local url=iso:///iso_fs/REAR_BACKUP
    +++ local url_without_scheme=/iso_fs/REAR_BACKUP
    +++ echo /iso_fs/REAR_BACKUP
    ++ local path=/iso_fs/REAR_BACKUP
    ++ case $scheme in
    ++ :
    ++ [[ usb = \i\s\o ]]
    ++ [[ -n '' ]]
    ++ Log 'Skipping '''ping''' test for host '''''' in BACKUP_URL '''iso:///iso_fs/REAR_BACKUP''''
    +++ date '+%Y-%m-%d %H:%M:%S.%N '
    ++ local 'timestamp=2019-01-22 06:29:01.071966422 '
    ++ test 1 -gt 0
    ++ echo '2019-01-22 06:29:01.071966422 Skipping '''ping''' test for host '''''' in BACKUP_URL '''iso:///iso_fs/REAR_BACKUP''''
    2019-01-22 06:29:01.071966422 Skipping 'ping' test for host '' in BACKUP_URL 'iso:///iso_fs/REAR_BACKUP'
    ++ case "$(basename $BACKUP_PROG)" in
    +++ basename tar
    ++ :
    ++ PROGS=("${PROGS[@]}" showmount mount.$(url_scheme $BACKUP_URL) umount.$(url_scheme $BACKUP_URL) $( test "$BACKUP_MOUNTCMD" && echo "${BACKUP_MOUNTCMD%% *}" ) $( test "$BACKUP_UMOUNTCMD" && echo "${BACKUP_UMOUNTCMD%% *}" ) $BACKUP_PROG gzip bzip2 xz)
    +++ url_scheme iso:///iso_fs/REAR_BACKUP
    +++ local url=iso:///iso_fs/REAR_BACKUP
    +++ local scheme=iso
    +++ echo iso
    +++ grep -q :
    +++ echo iso
    +++ url_scheme iso:///iso_fs/REAR_BACKUP
    +++ local url=iso:///iso_fs/REAR_BACKUP
    +++ local scheme=iso
    +++ echo iso
    +++ grep -q :
    +++ echo iso
    +++ test ''
    +++ test ''
    ++ [[ sshfs = \i\s\o ]]
    ++ [[ ftpfs = \i\s\o ]]
    ++ MODULES=("${MODULES[@]}" $(url_scheme $BACKUP_URL))
    +++ url_scheme iso:///iso_fs/REAR_BACKUP
    +++ local url=iso:///iso_fs/REAR_BACKUP
    +++ local scheme=iso
    +++ echo iso
    +++ grep -q :
    +++ echo iso
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:01.083659470 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:01.083659470 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-22 06:29:01.083659470 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-22 06:29:01.088560198 Including verify/default/050_create_mappings_dir.sh
    2019-01-22 06:29:01.089786618 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/verify/default/050_create_mappings_dir.sh
    ++ [[ ! -d /etc/rear/mappings ]]
  • source_return_code=1
  • test 0 -eq 1
  • Debug 'Source function: '''source /usr/share/rear/verify/default/050_create_mappings_dir.sh''' returns 1'
  • test 1
  • Log 'Source function: '''source /usr/share/rear/verify/default/050_create_mappings_dir.sh''' returns 1'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:01.094238350 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:01.094238350 Source function: '''source /usr/share/rear/verify/default/050_create_mappings_dir.sh''' returns 1'
    2019-01-22 06:29:01.094238350 Source function: 'source /usr/share/rear/verify/default/050_create_mappings_dir.sh' returns 1
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:01.095960909 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:01.095960909 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-22 06:29:01.095960909 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-22 06:29:01.101356134 Including verify/GNU/Linux/050_sane_recovery_check.sh
    2019-01-22 06:29:01.103001570 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/verify/GNU/Linux/050_sane_recovery_check.sh
    ++ '[' -s /etc/scripts/system-setup ']'
    ++ StopIfError 'This it not a Relax-and-Recover rescue system.'
    ++ (( 0 != 0 ))
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:01.108915232 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:01.108915232 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-22 06:29:01.108915232 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-22 06:29:01.113623039 Including verify/NETFS/default/050_start_required_nfs_daemons.sh
    2019-01-22 06:29:01.114708602 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/verify/NETFS/default/050_start_required_nfs_daemons.sh
    +++ url_scheme iso:///iso_fs/REAR_BACKUP
    +++ local url=iso:///iso_fs/REAR_BACKUP
    +++ local scheme=iso
    +++ echo iso
    +++ grep -q :
    +++ echo iso
    ++ local backup_url_scheme=iso
    ++ test nfs = iso
    ++ return 0
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:01.121223839 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:01.121223839 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-22 06:29:01.121223839 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-22 06:29:01.126842747 Including verify/NETFS/default/060_mount_NETFS_path.sh
    2019-01-22 06:29:01.128495349 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/verify/NETFS/default/060_mount_NETFS_path.sh
    ++ mkdir -p -v /tmp/rear.U4Lsmb4pp4DEyYu/outputfs
    mkdir: created directory '/tmp/rear.U4Lsmb4pp4DEyYu/outputfs'
    ++ StopIfError 'Could not mkdir '''/tmp/rear.U4Lsmb4pp4DEyYu/outputfs''''
    ++ (( 0 != 0 ))
    ++ AddExitTask 'rmdir -v /tmp/rear.U4Lsmb4pp4DEyYu/outputfs >&2'
    ++ EXIT_TASKS=("$" "${EXIT_TASKS[@]}")
    ++ Debug 'Added '''rmdir -v /tmp/rear.U4Lsmb4pp4DEyYu/outputfs >&2''' as an exit task'
    ++ test 1
    ++ Log 'Added '''rmdir -v /tmp/rear.U4Lsmb4pp4DEyYu/outputfs >&2''' as an exit task'
    +++ date '+%Y-%m-%d %H:%M:%S.%N '
    ++ local 'timestamp=2019-01-22 06:29:01.134124305 '
    ++ test 1 -gt 0
    ++ echo '2019-01-22 06:29:01.134124305 Added '''rmdir -v /tmp/rear.U4Lsmb4pp4DEyYu/outputfs >&2''' as an exit task'
    2019-01-22 06:29:01.134124305 Added 'rmdir -v /tmp/rear.U4Lsmb4pp4DEyYu/outputfs >&2' as an exit task
    ++ [[ -n '' ]]
    ++ mount_url iso:///iso_fs/REAR_BACKUP /tmp/rear.U4Lsmb4pp4DEyYu/outputfs
    ++ local url=iso:///iso_fs/REAR_BACKUP
    ++ local mountpoint=/tmp/rear.U4Lsmb4pp4DEyYu/outputfs
    ++ local defaultoptions=rw,noatime
    ++ local options=rw,noatime
    ++ local mount_cmd
    ++ case $(url_scheme $url) in
    +++ url_scheme iso:///iso_fs/REAR_BACKUP
    +++ local url=iso:///iso_fs/REAR_BACKUP
    +++ local scheme=iso
    +++ echo iso
    +++ grep -q :
    +++ echo iso
    ++ test recover = recover
    +++ blkid -l -o device -t LABEL=RELAXRECOVER
    ++ local relaxrecover_block_device=/dev/sr0
    ++ local relaxrecover_symlink_name=/dev/disk/by-label/RELAXRECOVER
    +++ readlink --verbose -e /dev/disk/by-label/RELAXRECOVER
    readlink: /dev/disk/by-label/RELAXRECOVER: No such file or directory
    ++ local relaxrecover_symlink_target=
    ++ test /dev/sr0 -a '' -a '' = /dev/sr0
    ++ Log 'Symlink '''/dev/disk/by-label/RELAXRECOVER''' does not exist or does not point to a block device with '''RELAXRECOVER''' filesystem label'
    +++ date '+%Y-%m-%d %H:%M:%S.%N '
    ++ local 'timestamp=2019-01-22 06:29:01.246007452 '
    ++ test 1 -gt 0
    ++ echo '2019-01-22 06:29:01.246007452 Symlink '''/dev/disk/by-label/RELAXRECOVER''' does not exist or does not point to a block device with '''RELAXRECOVER''' filesystem label'
    2019-01-22 06:29:01.246007452 Symlink '/dev/disk/by-label/RELAXRECOVER' does not exist or does not point to a block device with 'RELAXRECOVER' filesystem label
    ++ mkdir --verbose -p /dev/disk/by-label
    mkdir: created directory '/dev/disk/by-label'
    ++ test -b /dev/sr0
    ++ Log 'Making symlink '''/dev/disk/by-label/RELAXRECOVER''' point to '''/dev/sr0''' because it has filesystem label '''RELAXRECOVER''''
    +++ date '+%Y-%m-%d %H:%M:%S.%N '
    ++ local 'timestamp=2019-01-22 06:29:01.248549903 '
    ++ test 1 -gt 0
    ++ echo '2019-01-22 06:29:01.248549903 Making symlink '''/dev/disk/by-label/RELAXRECOVER''' point to '''/dev/sr0''' because it has filesystem label '''RELAXRECOVER''''
    2019-01-22 06:29:01.248549903 Making symlink '/dev/disk/by-label/RELAXRECOVER' point to '/dev/sr0' because it has filesystem label 'RELAXRECOVER'
    ++ ln --verbose -sf /dev/sr0 /dev/disk/by-label/RELAXRECOVER
    '/dev/disk/by-label/RELAXRECOVER' -> '/dev/sr0'
    ++ test -e /dev/disk/by-label/RELAXRECOVER
    ++ mount_cmd='mount /dev/disk/by-label/RELAXRECOVER /tmp/rear.U4Lsmb4pp4DEyYu/outputfs'
    ++ Log 'Mounting with '''mount /dev/disk/by-label/RELAXRECOVER /tmp/rear.U4Lsmb4pp4DEyYu/outputfs''''
    +++ date '+%Y-%m-%d %H:%M:%S.%N '
    ++ local 'timestamp=2019-01-22 06:29:01.251314878 '
    ++ test 1 -gt 0
    ++ echo '2019-01-22 06:29:01.251314878 Mounting with '''mount /dev/disk/by-label/RELAXRECOVER /tmp/rear.U4Lsmb4pp4DEyYu/outputfs''''
    2019-01-22 06:29:01.251314878 Mounting with 'mount /dev/disk/by-label/RELAXRECOVER /tmp/rear.U4Lsmb4pp4DEyYu/outputfs'
    ++ eval mount /dev/disk/by-label/RELAXRECOVER /tmp/rear.U4Lsmb4pp4DEyYu/outputfs
    +++ mount /dev/disk/by-label/RELAXRECOVER /tmp/rear.U4Lsmb4pp4DEyYu/outputfs
    mount: /dev/sr0 is write-protected, mounting read-only
    ++ AddExitTask 'umount -f -v '''/tmp/rear.U4Lsmb4pp4DEyYu/outputfs''' >&2'
    ++ EXIT_TASKS=("$    " "${EXIT_TASKS[@]}")
    ++ Debug 'Added '''umount -f -v '''/tmp/rear.U4Lsmb4pp4DEyYu/outputfs''' >&2''' as an exit task'
    ++ test 1
    ++ Log 'Added '''umount -f -v '''/tmp/rear.U4Lsmb4pp4DEyYu/outputfs''' >&2''' as an exit task'
    +++ date '+%Y-%m-%d %H:%M:%S.%N '
    ++ local 'timestamp=2019-01-22 06:29:01.274404951 '
    ++ test 1 -gt 0
    ++ echo '2019-01-22 06:29:01.274404951 Added '''umount -f -v '''/tmp/rear.U4Lsmb4pp4DEyYu/outputfs''' >&2''' as an exit task'
    2019-01-22 06:29:01.274404951 Added 'umount -f -v '/tmp/rear.U4Lsmb4pp4DEyYu/outputfs' >&2' as an exit task
    ++ return 0
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:01.275950230 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:01.275950230 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-22 06:29:01.275950230 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-22 06:29:01.280754977 Including verify/NETFS/default/070_set_backup_archive.sh
    2019-01-22 06:29:01.282023305 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/verify/NETFS/default/070_set_backup_archive.sh
    ++ test ''
    ++ local backup_file_suffix=.tar.gz
    ++ local backup_file_name=backup.tar.gz
    +++ url_scheme iso:///iso_fs/REAR_BACKUP
    +++ local url=iso:///iso_fs/REAR_BACKUP
    +++ local scheme=iso
    +++ echo iso
    +++ grep -q :
    +++ echo iso
    ++ local scheme=iso
    +++ url_path iso:///iso_fs/REAR_BACKUP
    +++ local url=iso:///iso_fs/REAR_BACKUP
    +++ local url_without_scheme=/iso_fs/REAR_BACKUP
    +++ echo /iso_fs/REAR_BACKUP
    ++ local path=/iso_fs/REAR_BACKUP
    ++ case "$scheme" in
    +++ backup_path iso /iso_fs/REAR_BACKUP
    +++ local scheme=iso
    +++ local path=/iso_fs/REAR_BACKUP
    +++ case $scheme in
    +++ [[ recover = \r\e\c\o\v\e\r ]]
    +++ path=/tmp/rear.U4Lsmb4pp4DEyYu/outputfs/iso_fs/REAR_BACKUP
    +++ echo /tmp/rear.U4Lsmb4pp4DEyYu/outputfs/iso_fs/REAR_BACKUP
    ++ local outputpath=/tmp/rear.U4Lsmb4pp4DEyYu/outputfs/iso_fs/REAR_BACKUP
    ++ backuparchive=/tmp/rear.U4Lsmb4pp4DEyYu/outputfs/iso_fs/REAR_BACKUP/backup.tar.gz
    ++ LogPrint 'Using backup archive '''/tmp/rear.U4Lsmb4pp4DEyYu/outputfs/iso_fs/REAR_BACKUP/backup.tar.gz''''
    ++ Log 'Using backup archive '''/tmp/rear.U4Lsmb4pp4DEyYu/outputfs/iso_fs/REAR_BACKUP/backup.tar.gz''''
    +++ date '+%Y-%m-%d %H:%M:%S.%N '
    ++ local 'timestamp=2019-01-22 06:29:01.290993322 '
    ++ test 1 -gt 0
    ++ echo '2019-01-22 06:29:01.290993322 Using backup archive '''/tmp/rear.U4Lsmb4pp4DEyYu/outputfs/iso_fs/REAR_BACKUP/backup.tar.gz''''
    2019-01-22 06:29:01.290993322 Using backup archive '/tmp/rear.U4Lsmb4pp4DEyYu/outputfs/iso_fs/REAR_BACKUP/backup.tar.gz'
    ++ Print 'Using backup archive '''/tmp/rear.U4Lsmb4pp4DEyYu/outputfs/iso_fs/REAR_BACKUP/backup.tar.gz''''
    ++ test 1
    ++ echo -e 'Using backup archive '''/tmp/rear.U4Lsmb4pp4DEyYu/outputfs/iso_fs/REAR_BACKUP/backup.tar.gz''''
    ++ return
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:01.292502181 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:01.292502181 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-22 06:29:01.292502181 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-22 06:29:01.297016379 Including verify/NETFS/default/090_set_readonly_options.sh
    2019-01-22 06:29:01.298107632 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/verify/NETFS/default/090_set_readonly_options.sh
    ++ BACKUP_OPTIONS=ro
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:01.302477003 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:01.302477003 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-22 06:29:01.302477003 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-22 06:29:01.306853087 Including verify/GNU/Linux/230_storage_and_network_modules.sh
    2019-01-22 06:29:01.307929063 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/verify/GNU/Linux/230_storage_and_network_modules.sh
    ++ Log 'Including storage drivers'
    +++ date '+%Y-%m-%d %H:%M:%S.%N '
    ++ local 'timestamp=2019-01-22 06:29:01.312737492 '
    ++ test 1 -gt 0
    ++ echo '2019-01-22 06:29:01.312737492 Including storage drivers'
    2019-01-22 06:29:01.312737492 Including storage drivers
    ++ STORAGE_DRIVERS=($( find_modules_in_dirs /lib/modules/$KERNEL_VERSION/kernel/drivers/{block,firewire,ide,ata,md,message,scsi,usb/storage} ))
    +++ find_modules_in_dirs /lib/modules/4.4.162-94.72-default/kernel/drivers/block /lib/modules/4.4.162-94.72-default/kernel/drivers/firewire /lib/modules/4.4.162-94.72-default/kernel/drivers/ide /lib/modules/4.4.162-94.72-default/kernel/drivers/ata /lib/modules/4.4.162-94.72-default/kernel/drivers/md /lib/modules/4.4.162-94.72-default/kernel/drivers/message /lib/modules/4.4.162-94.72-default/kernel/drivers/scsi /lib/modules/4.4.162-94.72-default/kernel/drivers/usb/storage
    +++ find /lib/modules/4.4.162-94.72-default/kernel/drivers/block /lib/modules/4.4.162-94.72-default/kernel/drivers/firewire /lib/modules/4.4.162-94.72-default/kernel/drivers/ide /lib/modules/4.4.162-94.72-default/kernel/drivers/ata /lib/modules/4.4.162-94.72-default/kernel/drivers/md /lib/modules/4.4.162-94.72-default/kernel/drivers/message /lib/modules/4.4.162-94.72-default/kernel/drivers/scsi /lib/modules/4.4.162-94.72-default/kernel/drivers/usb/storage -type f -name '.ko' -printf '%f\n'
    +++ sed -e 's/^(.).ko./\1/'
    ++ Log 'Including network drivers'
    +++ date '+%Y-%m-%d %H:%M:%S.%N '
    ++ local 'timestamp=2019-01-22 06:29:01.318733109 '
    ++ test 1 -gt 0
    ++ echo '2019-01-22 06:29:01.318733109 Including network drivers'
    2019-01-22 06:29:01.318733109 Including network drivers
    ++ NETWORK_DRIVERS=($( find_modules_in_dirs /lib/modules/$KERNEL_VERSION/kernel/drivers/net ))
    +++ find_modules_in_dirs /lib/modules/4.4.162-94.72-default/kernel/drivers/net
    +++ find /lib/modules/4.4.162-94.72-default/kernel/drivers/net -type f -name '.ko' -printf '%f\n'
    +++ sed -e 's/^(.).ko./\1/'
    ++ Log 'Including crypto drivers'
    +++ date '+%Y-%m-%d %H:%M:%S.%N '
    ++ local 'timestamp=2019-01-22 06:29:01.325980866 '
    ++ test 1 -gt 0
    ++ echo '2019-01-22 06:29:01.325980866 Including crypto drivers'
    2019-01-22 06:29:01.325980866 Including crypto drivers
    ++ CRYPTO_DRIVERS=($( find_modules_in_dirs /lib/modules/$KERNEL_VERSION/kernel/crypto ))
    +++ find_modules_in_dirs /lib/modules/4.4.162-94.72-default/kernel/crypto
    +++ find /lib/modules/4.4.162-94.72-default/kernel/crypto -type f -name '.ko' -printf '%f\n'
    +++ sed -e 's/^(.).ko./\1/'
    ++ Log 'Including virtualization drivers'
    +++ date '+%Y-%m-%d %H:%M:%S.%N '
    ++ local 'timestamp=2019-01-22 06:29:01.330257531 '
    ++ test 1 -gt 0
    ++ echo '2019-01-22 06:29:01.330257531 Including virtualization drivers'
    2019-01-22 06:29:01.330257531 Including virtualization drivers
    ++ VIRTUAL_DRIVERS=($( find_modules_in_dirs /lib/modules/$KERNEL_VERSION/kernel/drivers/{virtio,xen} ))
    +++ find_modules_in_dirs /lib/modules/4.4.162-94.72-default/kernel/drivers/virtio /lib/modules/4.4.162-94.72-default/kernel/drivers/xen
    +++ find /lib/modules/4.4.162-94.72-default/kernel/drivers/virtio /lib/modules/4.4.162-94.72-default/kernel/drivers/xen -type f -name '.ko' -printf '%f\n'
    +++ sed -e 's/^(.).ko./\1/'
    ++ Log 'Including additional drivers'
    +++ date '+%Y-%m-%d %H:%M:%S.%N '
    ++ local 'timestamp=2019-01-22 06:29:01.334448810 '
    ++ test 1 -gt 0
    ++ echo '2019-01-22 06:29:01.334448810 Including additional drivers'
    2019-01-22 06:29:01.334448810 Including additional drivers
    ++ EXTRA_DRIVERS=($( find_modules_in_dirs /lib/modules/$KERNEL_VERSION/{extra,weak-updates} ))
    +++ find_modules_in_dirs /lib/modules/4.4.162-94.72-default/extra /lib/modules/4.4.162-94.72-default/weak-updates
    +++ find /lib/modules/4.4.162-94.72-default/extra /lib/modules/4.4.162-94.72-default/weak-updates -type f -name '.ko' -printf '%f\n'
    +++ sed -e 's/^(.).ko./\1/'
    ++ unset -f find_modules_in_dirs
  • source_return_code=0
  • test 0 -eq 0
  • test 1
  • Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
  • test 1
  • Log 'Leaving debugscripts mode (back to previous bash flags and options settings).'
    ++ date '+%Y-%m-%d %H:%M:%S.%N '
  • local 'timestamp=2019-01-22 06:29:01.338525922 '
  • test 1 -gt 0
  • echo '2019-01-22 06:29:01.338525922 Leaving debugscripts mode (back to previous bash flags and options settings).'
    2019-01-22 06:29:01.338525922 Leaving debugscripts mode (back to previous bash flags and options settings).
    2019-01-22 06:29:01.343180784 Including verify/GNU/Linux/260_recovery_storage_drivers.sh
    2019-01-22 06:29:01.344281530 Entering debugscripts mode via 'set -x'.
  • source /usr/share/rear/verify/GNU/Linux/260_recovery_storage_drivers.sh
    ++ have_udev
    ++ test -d /etc/udev/rules.d
    ++ has_binary udevadm udevtrigger udevsettle udevinfo udevstart
    ++ for bin in '$@'
    ++ type udevadm
    ++ return 0
    ++ return 0
    ++ FindStorageDrivers /tmp/rear.U4Lsmb4pp4DEyYu/tmp/dev
    ++ test no_modules = vfat
    ++ (( 145 == 0 ))
    ++ grep -v -E '(loop)'
    +++ lsmod
    ++ read module junk
    ++ sort -u
    ++ IsInArray Module zram virtio_blk skd rbd pktcdvd nbd mtip32xx floppy loop cryptoloop brd aoe nosy firewire-sbp2 firewire-ohci firewire-net firewire-core sata_sx4 sata_svw sata_sil24 sata_qstor sata_inic162x pdc_adma ata_generic libahci_platform ahci_platform ahci libata libahci acard-ahci md-cluster raid10 raid1 raid0 linear faulty dm-zero dm-thin-pool dm-switch dm-snapshot dm-service-time dm-round-robin raid456 md-mod dm-raid dm-queue-length dm-region-hash dm-mirror dm-log-writes dm-log dm-log-userspace dm-multipath dm-least-pending dm-flakey dm-era dm-delay dm-crypt dm-cache-smq dm-persistent-data dm-cache dm-cache-cleaner dm-mod dm-bufio dm-bio-prison bcache mptscsih mptsas mptctl mptbase virtio_scsi sym53c8xx stex st snic sg ses sd_mod qla4xxx tcm_qla2xxx qla2xxx qedi qedf pmcraid pm80xx osd libosd mvsas raid_class mpt3sas megaraid_sas lpfc iscsi_tcp ipr scsi_transport_srp ibmvscsi ibmvfc esas2r scsi_dh_rdac scsi_dh_hp_sw scsi_dh_emc scsi_dh_alua cxlflash libiscsi_tcp cxgb4i libcxgbi cxgb3i csiostor ch bnx2i libfc fcoe libfcoe bnx2fc scsi_transport_fc bfa scsi_transport_iscsi libiscsi iscsi_boot_sysfs be2iscsi arcmsr scsi_transport_spi esp_scsi am53c974 libsas aic94xx aacraid 3w-sas scsi_transport_sas sr_mod scsi_mod ums-usbat ums-sddr55 ums-sddr09 ums-realtek ums-onetouch ums-karma ums-jumpshot ums-isd200 ums-freecom ums-eneub6250 ums-datafab ums-cypress ums-alauda uas usb-storage
    ++ local needle=Module
    ++ test -z Module
    ++ shift
    ++ test Module == zram
    ++ shift
    ++ test Module == virtio_blk
    ++ shift
    ++ test Module == skd
    ++ shift
    ++ test Module == rbd
    ++ shift
    ++ test Module == pktcdvd
    ++ shift
    ++ test Module == nbd
    ++ shift
    ++ test Module == mtip32xx
    ++ shift
    ++ test Module == floppy
    ++ shift
    ++ test Module == loop
    ++ shift
    ++ test Module == cryptoloop
    ++ shift
    ++ test Module == brd
    ++ shift
    ++ test Module == aoe
    ++ shift
    ++ test Module == nosy
    ++ shift
    ++ test Module == firewire-sbp2
    ++ shift
    ++ test Module == firewire-ohci
    ++ shift
    ++ test Module == firewire-net
    ++ shift
    ++ test Module == firewire-core
    ++ shift
    ++ test Module == sata_sx4
    ++ shift
    ++ test Module == sata_svw
    ++ shift
    ++ test Module == sata_sil24
    ++ shift
    ++ test Module == sata_qstor
    ++ shift
    ++ test Module == sata_inic162x
    ++ shift
    ++ test Module == pdc_adma
    ++ shift
    ++ test Module == ata_generic
    ++ shift
    ++ test Module == libahci_platform
    ++ shift
    ++ test Module == ahci_platform
    ++ shift
    ++ test Module == ahci
    ++ shift
    ++ test Module == libata
    ++ shift
    ++ test Module == libahci
    ++ shift
    ++ test Module == acard-ahci
    ++ shift
    ++ test Module == md-cluster
    ++ shift
    ++ test Module == raid10
    ++ shift
    ++ test Module == raid1
    ++ shift
    ++ test Module == raid0
    ++ shift
    ++ test Module == linear
    ++ shift
    ++ test Module == faulty
    ++ shift
    ++ test Module == dm-zero
    ++ shift
    ++ test Module == dm-thin-pool
    ++ shift
    ++ test Module == dm-switch
    ++ shift
    ++ test Module == dm-snapshot
    ++ shift
    ++ test Module == dm-service-time
    ++ shift
    ++ test Module == dm-round-robin
    ++ shift
    ++ test Module == raid456
    ++ shift
    ++ test Module == md-mod
    ++ shift
    ++ test Module == dm-raid
    ++ shift
    ++ test Module == dm-queue-length
    ++ shift
    ++ test Module == dm-region-hash
    ++ shift
    ++ test Module == dm-mirror
    ++ shift
    ++ test Module == dm-log-writes
    ++ shift
    ++ test Module == dm-log
    ++ shift
    ++ test Module == dm-log-userspace
    ++ shift
    ++ test Module == dm-multipath
    ++ shift
    ++ test Module == dm-least-pending
    ++ shift
    ++ test Module == dm-flakey
    ++ shift
    ++ test Module == dm-era
    ++ shift
    ++ test Module == dm-delay
    ++ shift
    ++ test Module == dm-crypt
    ++ shift
    ++ test Module == dm-cache-smq
    ++ shift
    ++ test Module == dm-persistent-data
    ++ shift
    ++ test Module == dm-cache
    ++ shift
    ++ test Module == dm-cache-cleaner
    ++ shift
    ++ test Module == dm-mod
    ++ shift
    ++ test Module == dm-bufio
    ++ shift
    ++ test Module == dm-bio-prison
    ++ shift
    ++ test Module == bcache
    ++ shift
    ++ test Module == mptscsih
    ++ shift
    ++ test Module == mptsas
    ++ shift
    ++ test Module == mptctl
    ++ shift
    ++ test Module == mptbase
    ++ shift
    ++ test Module == virtio_scsi
    ++ shift
    ++ test Module == sym53c8xx
    ++ shift
    ++ test Module == stex
    ++ shift
    ++ test Module == st
    ++ shift
    ++ test Module == snic
    ++ shift
    ++ test Module == sg
    ++ shift
    ++ test Module == ses
    ++ shift
    ++ test Module == sd_mod
    ++ shift
    ++ test Module == qla4xxx
    ++ shift
    ++ test Module == tcm_qla2xxx
    ++ shift
    ++ test Module == qla2xxx
    ++ shift
    ++ test Module == qedi
    ++ shift
    ++ test Module == qedf
    ++ shift
    ++ test Module == pmcraid
    ++ shift
    ++ test Module == pm80xx
    ++ shift
    ++ test Module == osd
    ++ shift
    ++ test Module == libosd
    ++ shift
    ++ test Module == mvsas
    ++ shift
    ++ test Module == raid_class
    ++ shift
    ++ test Module == mpt3sas
    ++ shift
    ++ test Module == megaraid_sas
    ++ shift
    ++ test Module == lpfc
    ++ shift
    ++ test Module == iscsi_tcp
    ++ shift
    ++ test Module == ipr
    ++ shift
    ++ test Module == scsi_transport_srp
    ++ shift
    ++ test Module == ibmvscsi
    ++ shift
    ++ test Module == ibmvfc
    ++ shift
    ++ test Module == esas2r
    ++ shift
    ++ test Module == scsi_dh_rdac
    ++ shift
    ++ test Module == scsi_dh_hp_sw
    ++ shift
    ++ test Module == scsi_dh_emc
    ++ shift
    ++ test Module == scsi_dh_alua
    ++ shift
    ++ test Module == cxlflash
    ++ shift
    ++ test Module == libiscsi_tcp
    ++ shift
    ++ test Module == cxgb4i
    ++ shift
    ++ test Module == libcxgbi
    ++ shift
    ++ test Module == cxgb3i
    ++ shift
    ++ test Module == csiostor
    ++ shift
    ++ test Module == ch
    ++ shift
    ++ test Module == bnx2i
    ++ shift
    ++ test Module == libfc
    ++ shift
    ++ test Module == fcoe
    ++ shift
    ++ test Module == libfcoe
    ++ shift
    ++ test Module == bnx2fc
    ++ shift
    ++ test Module == scsi_transport_fc
    ++ shift
    ++ test Module == bfa
    ++ shift
    ++ test Module == scsi_transport_iscsi
    ++ shift
    ++ test Module == libiscsi
    ++ shift
    ++ test Module == iscsi_boot_sysfs
    ++ shift
    ++ test Module == be2iscsi
    ++ shift
    ++ test Module == arcmsr
    ++ shift
    ++ test Module == scsi_transport_spi
    ++ shift
    ++ test Module == esp_scsi
    ++ shift
    ++ test Module == am53c974
    ++ shift
    ++ test Module == libsas
    ++ shift
    ++ test Module == aic94xx
    ++ shift
    ++ test Module == aacraid
    ++ shift
    ++ test Module == 3w-sas
    ++ shift
    ++ test Module == scsi_transport_sas
    ++ shift
    ++ test Module == sr_mod
    ++ shift
    ++ test Module == scsi_mod
    ++ shift
    ++ test Module == ums-usbat
    ++ shift
    ++ test Module == ums-sddr55
    ++ shift
    ++ test Module == ums-sddr09
    ++ shift
    ++ test Module == ums-realtek
    ++ shift
    ++ test Module == ums-onetouch
    ++ shift
    ++ test Module == ums-karma
    ++ shift
    ++ test Module == ums-jumpshot
    ++ shift
    ++ test Module == ums-isd200
    ++ shift
    ++ test Module == ums-freecom
    ++ shift
    ++ test Module == ums-eneub6250
    ++ shift
    ++ test Module == ums-datafab
    ++ shift
    ++ test Module == ums-cypress
    ++ shift
    ++ test Module == ums-alauda
    ++ shift
    ++ test Module == uas
    ++ shift
    ++ test Module == usb-storage
    ++ shift
    ++ test Module == ''
    ++ shift
    ++ return 1
    ++ read module junk
    ++ IsInArray nls_utf8 zram virtio_blk skd rbd pktcdvd nbd mtip32xx floppy loop cryptoloop brd aoe nosy firewire-sbp2 firewire-ohci firewire-net firewire-core sata_sx4 sata_svw sata_sil24 sata_qstor sata_inic162x pdc_adma ata_generic libahci_platform ahci_platform ahci libata libahci acard-ahci md-cluster raid10 raid1 raid0 linear faulty dm-zero dm-thin-pool dm-switch dm-snapshot dm-service-time dm-round-robin raid456 md-mod dm-raid dm-queue-length dm-region-hash dm-mirror dm-log-writes dm-log dm-log-userspace dm-multipath dm-least-pending dm-flakey dm-era dm-delay dm-crypt dm-cache-smq dm-persistent-data dm-cache dm-cache-cleaner dm-mod dm-bufio dm-bio-prison bcache mptscsih mptsas mptctl mptbase virtio_scsi sym53c8xx stex st snic sg ses sd_mod qla4xxx tcm_qla2xxx qla2xxx qedi qedf pmcraid pm80xx osd libosd mvsas raid_class mpt3sas megaraid_sas lpfc iscsi_tcp ipr scsi_transport_srp ibmvscsi ibmvfc esas2r scsi_dh_rdac scsi_dh_hp_sw scsi_dh_emc scsi_dh_alua cxlflash libiscsi_tcp cxgb4i libcxgbi cxgb3i csiostor ch bnx2i libfc fcoe libfcoe bnx2fc scsi_transport_fc bfa scsi_transport_iscsi libiscsi iscsi_boot_sysfs be2iscsi arcmsr scsi_transport_spi esp_scsi am53c974 libsas aic94xx aacraid 3w-sas scsi_transport_sas sr_mod scsi_mod ums-usbat ums-sddr55 ums-sddr09 ums-realtek ums-onetouch ums-karma ums-jumpshot ums-isd200 ums-freecom ums-eneub6250 ums-datafab ums-cypress ums-alauda uas usb-storage
    ++ local needle=nls_utf8
    ++ test -z nls_utf8
    ++ shift
    ++ test nls_utf8 == zram
    ++ shift
    ++ test nls_utf8 == virtio_blk
    ++ shift
    ++ test nls_utf8 == skd
    ++ shift
    ++ test nls_utf8 == rbd
    ++ shift
    ++ test nls_utf8 == pktcdvd
    ++ shift
    ++ test nls_utf8 == nbd
    ++ shift
    ++ test nls_utf8 == mtip32xx
    ++ shift
    ++ test nls_utf8 == floppy
    ++ shift
    ++ test nls_utf8 == loop
    ++ shift
    ++ test nls_utf8 == cryptoloop
    ++ shift
    ++ test nls_utf8 == brd
    ++ shift
    ++ test nls_utf8 == aoe
    ++ shift
    ++ test nls_utf8 == nosy
    ++ shift
    ++ test nls_utf8 == firewire-sbp2
    ++ shift
    ++ test nls_utf8 == firewire-ohci
    ++ shift
    ++ test nls_utf8 == firewire-net
    ++ shift
    ++ test nls_utf8 == firewire-core
    ++ shift
    ++ test nls_utf8 == sata_sx4
    ++ shift
    ++ test nls_utf8 == sata_svw
    ++ shift
    ++ test nls_utf8 == sata_sil24
    ++ shift
    ++ test nls_utf8 == sata_qstor
    ++ shift
    ++ test nls_utf8 == sata_inic162x
    ++ shift
    ++ test nls_utf8 == pdc_adma
    ++ shift
    ++ test nls_utf8 == ata_generic
    ++ shift
    ++ test nls_utf8 == libahci_platform
    ++ shift
    ++ test nls_utf8 == ahci_platform
    ++ shift
    ++ test nls_utf8 == ahci
    ++ shift
    ++ test nls_utf8 == libata
    ++ shift
    ++ test nls_utf8 == libahci
    ++ shift
    ++ test nls_utf8 == acard-ahci
    ++ shift
    ++ test nls_utf8 == md-cluster
    ++ shift
    ++ test nls_utf8 == raid10
    ++ shift
    ++ test nls_utf8 == raid1
    ++ shift
    ++ test nls_utf8 == raid0
    ++ shift
    ++ test nls_utf8 == linear
    ++ shift
    ++ test nls_utf8 == faulty
    ++ shift
    ++ test nls_utf8 == dm-zero
    ++ shift
    ++ test nls_utf8 == dm-thin-pool
    ++ shift
    ++ test nls_utf8 == dm-switch
    ++ shift
    ++ test nls_utf8 == dm-snapshot
    ++ shift
    ++ test nls_utf8 == dm-service-time
    ++ shift
    ++ test nls_utf8 == dm-round-robin
    ++ shift
    ++ test nls_utf8 == raid456
    ++ shift
    ++ test nls_utf8 == md-mod
    ++ shift
    ++ test nls_utf8 == dm-raid
    ++ shift
    ++ test nls_utf8 == dm-queue-length
    ++ shift
    ++ test nls_utf8 == dm-region-hash
    ++ shift
    ++ test nls_utf8 == dm-mirror
    ++ shift
    ++ test nls_utf8 == dm-log-writes
    ++ shift
    ++ test nls_utf8 == dm-log
    ++ shift
    ++ test nls_utf8 == dm-log-userspace
    ++ shift
    ++ test nls_utf8 == dm-multipath
    ++ shift
    ++ test nls_utf8 == dm-least-pending
    ++ shift
    ++ test nls_utf8 == dm-flakey
    ++ shift
    ++ test nls_utf8 == dm-era
    ++ shift
    ++ test nls_utf8 == dm-delay
    ++ shift
    ++ test nls_utf8 == dm-crypt
    ++ shift
    ++ test nls_utf8 == dm-cache-smq
    ++ shift
    ++ test nls_utf8 == dm-persistent-data
    ++ shift
    ++ test nls_utf8 == dm-cache
    ++ shift
    ++ test nls_utf8 == dm-cache-cleaner
    ++ shift
    ++ test nls_utf8 == dm-mod
    ++ shift
    ++ test nls_utf8 == dm-bufio
    ++ shift
    ++ test nls_utf8 == dm-bio-prison
    ++ shift
    ++ test nls_utf8 == bcache
    ++ shift
    ++ test nls_utf8 == mptscsih
    ++ shift
    ++ test nls_utf8 == mptsas
    ++ shift
    ++ test nls_utf8 == mptctl
    ++ shift
    ++ test nls_utf8 == mptbase
    ++ shift
    ++ test nls_utf8 == virtio_scsi
    ++ shift
    ++ test nls_utf8 == sym53c8xx
    ++ shift
    ++ test nls_utf8 == stex
    ++ shift
    ++ test nls_utf8 == st
    ++ shift
    ++ test nls_utf8 == snic
    ++ shift
    ++ test nls_utf8 == sg
    ++ shift
    ++ test nls_utf8 == ses
    ++ shift
    ++ test nls_utf8 == sd_mod
    ++ shift
    ++ test nls_utf8 == qla4xxx
    ++ shift
    ++ test nls_utf8 == tcm_qla2xxx
    ++ shift
    ++ test nls_utf8 == qla2xxx
    ++ shift
    ++ test nls_utf8 == qedi
    ++ shift
    ++ test nls_utf8 == qedf
    ++ shift
    ++ test nls_utf8 == pmcraid
    ++ shift
    ++ test nls_utf

dewagner1 commented at 2019-02-19 13:10:

Debug info from testing rear-2.4-1.git.3227.0f1f9dbb.master.ppc64le.rpm provided by Sebastien.

I have created a rear backup image from a SLES 12 SP4 server and tried restoring that image to another lpar three different times - one right after the other.

The first time failed, the second time required the work-around you gave us for the previous version of rear and the third time worked. I have captured console output and the debug log file for the first two scenarios and just the console output for the third.

1-fail.txt
2-required workaround.txt
3 -worked.txt

dewagner1 commented at 2019-02-19 13:12:

I closed this issue by mistake

ccjung commented at 2019-02-21 19:20:

Hello,
Dan has attached files with the different test results. It seems we get different results with SLES12 SP4.

What is the status on this ticket?

Thanks.

jsmeix commented at 2019-02-22 10:23:

I know nothing about that special hardware
(POWER LPAR with SAN and multipath)
so that I can do only some basically blind guesswork:

In your https://github.com/rear/rear/issues/2019#issuecomment-465122332
both your https://github.com/rear/rear/files/2880076/1-fail.txt
and https://github.com/rear/rear/files/2880079/2-required.workaround.txt
contain a device like /dev/mapper/SIBM_ while in contrast
https://github.com/rear/rear/files/2880081/3.-worked.txt
does not contain such a device.

According to
https://github.com/rear/rear/issues/2016#issuecomment-454333190
and
https://github.com/rear/rear/issues/2016#issuecomment-454344375
this usually means something wrong in the SAN zoning configuration

Perhaps you still have issues in your SAN zoning configuration
as you had in
https://github.com/rear/rear/issues/2016#issuecomment-454759822

On the other hand in
https://github.com/rear/rear/files/2880079/2-required.workaround.txt
there is (excerpt)

UserInput -I LAYOUT_MIGRATION_CONFIRM_MAPPINGS needed in /usr/share/rear/layout/prepare/default/300_map_disks.sh line 275
Confirm or edit the disk mapping
1) Confirm disk mapping and continue 'rear recover'
2) n/a
3) Edit disk mapping (/var/lib/rear/layout/disk_mappings)
4) Use Relax-and-Recover shell and return back to here
5) Abort 'rear recover'
(default '1' timeout 300 seconds)
4
UserInput: Valid choice number result 'Use Relax-and-Recover shell and return back to here'

Welcome to Relax-and-Recover.

rear> udevadm control --reload-rules
rear> udevadm trigger

which seems to be what you mean with the work-around you gave us
which matches
https://github.com/schabrolles/rear/commit/03228ffdbf748e16026d8776c0745b0794c74148

That change is not yet merged into the ReaR master code.
Currently it is only in https://github.com/schabrolles/rear

@schabrolles
it seems your refresh udev with trigger before activating multipath
helps in such cases so that I think you should "just merge" it
into the ReaR master code.

My assumption is that the additional commands

udevadm control --reload-rules
udevadm trigger

cannot cause bad side effects on systems where they are not really needed.

FWIW:
In general I think
Waiting for udev and "kicking udev" are wrong (both miss the point)
according to what I described in https://github.com/rear/rear/issues/791
but if it helps in this particular case here we should just do that
according to "Dirty hacks welcome" in
https://github.com/rear/rear/wiki/Coding-Style

@suseusr168 @dewagner1
you can adapt your
usr/share/rear/layout/prepare/GNU/Linux/210_load_multipath.sh
script as in
https://github.com/schabrolles/rear/commit/03228ffdbf748e16026d8776c0745b0794c74148
and then redo rear mkrescue/mkbackup to get that workaround
into your ReaR recovery system so that those udevadm commands
are run atomatically during rear recover.

The whole updated script from @schabrolles
at https://github.com/schabrolles/rear
is visible via
https://raw.githubusercontent.com/schabrolles/rear/03228ffdbf748e16026d8776c0745b0794c74148/usr/share/rear/layout/prepare/GNU/Linux/210_load_multipath.sh
so that you could better see where to add those udevadm commands
when you have an older version of
usr/share/rear/layout/prepare/GNU/Linux/210_load_multipath.sh

schabrolles commented at 2019-02-24 12:26:

@dewagner1, @suseusr168,

Sorry for the long delay in my answer, but I don't have a lot of time and will be really busy till the end of next week.

For what I can see from the logs provided by @dewagner1 is still a problem around /dev/mapper/SIBM... devices...
We still don't know what can explain such device in your SAN + XIV environment...
This is the root cause of your issues...

Some LVM objects (pv, lv, fs) are detected on both /dev/mapper/<WWID> and the strange device /dev/mapper/SIBM_<WWID> which conducts to LVM errors due to duplicates devices... You should be able to mitigates this by adding a filter in /etc/lvm.conf ( add "r|/dev/mapper/SIBM_.*|")

Here is an example of /etc/lvm.conf taken from my SLES12 SP4 with "r|/dev/mapper/SIBM_.*|" added just before "a/.*/"

...
filter = [ "r|/dev/.*/by-path/.*|", "r|/dev/.*/by-id/.*|", "r|/dev/fd.*|", "r|/dev/cdrom|", "r|/dev/mapper/SIBM_.*|", "a/.*/" ]
...

You can set this up on the server to backup (as we also don't want to have LVM detected on /dev/mapper/SIBM devices), backup with rear. /etc/lvm.conf file should be added to your ReaR iso image.

jsmeix commented at 2019-03-04 11:28:

@dewagner1 @suseusr168
right now https://github.com/rear/rear/pull/2064 was merged
so that this issue should be now somehow avoided in ReaR, cf.
https://github.com/rear/rear/issues/2019#issuecomment-466349187

Nevertheless it seems the root cause of your issues is still there, cf.
https://github.com/rear/rear/issues/2019#issuecomment-466769572

@dewagner1 @suseusr168
how you could try out our current ReaR GitHub master code
to test whether or not https://github.com/rear/rear/pull/2064
actually helps in your case:

Basically "git clone" our current ReaR upstream GitHub master code
into a separated directory and then configure and run ReaR
from within that directory like:

# git clone https://github.com/rear/rear.git

# mv rear rear.github.master

# cd rear.github.master

# vi etc/rear/local.conf

# usr/sbin/rear -D mkbackup

Note the relative paths "etc/rear/" and "usr/sbin/".

ccjung commented at 2019-03-13 13:16:

Hello,
Any status on determining the root cause of the SLES12SP4 on Power restore problem ?

Do we have the #2059 bug problem or is it fixed?

I have attached the logs in case you haven't seen them.

We have implemented this workaround but still not having 100% success rate with the restore on SLES12SP4.

Here is an example of /etc/lvm.conf taken from my SLES12 SP4 with "r|/dev/mapper/SIBM_.|" added just before "a/./"
...
filter = [ "r|/dev/./by-path/.|", "r|/dev/./by-id/.|", "r|/dev/fd.|", "r|/dev/cdrom|", "r|/dev/mapper/SIBM_.|", "a/.*/" ]
...

LPAR eniesdbp001p does not have shared disks and the Rear restored 4 times with no problems
The 5th time failed.

I have attached both failed and worked logs for comparison.

putty.2019.02.26.eniesdbp001p.log - Failed
putty.2019.02.26.eniesdbp001p.worked.log - Worked

eniesdbp011p has shared disks. It detects the lower LUN id but fails the diskrestore.sh script all the time (3 times)
putty.2019.02.26.eniesdbp011p.log - Failed

putty.2019.02.26.eniesdbp001p.log
putty.2019.02.26.eniesdbp001p.worked.log
putty.2019.02.26.eniesdbp011p.log

ccjung commented at 2019-03-18 18:45:

Hello,
Is there any information we can provide to help debug this problem?

Thank you,

schabrolles commented at 2019-03-18 19:15:

@suseusr168,
I don't have a lot of time, but I had a quick look to your files.
For eniesdbp011p I think you try to recover a PV to /dev/mapper/20017380033e70088 that already exist on the machine /dev/mapper/20017380033e700e8. Both disk cannot have the same LVM PV UUID.

If you want to use /dev/mapper/20017380033e70088 instead of /dev/mapper/20017380033e700e8, you need to unmap the last one from the server first.

+++ lvm pvcreate -ff --yes -v --uuid emSwP2-fCya-vSee-BOCA-LUdO-qYCw-41Lc01 --norestorefile /dev/mapper/20017380033e70088-part3
  WARNING: Failed to connect to lvmetad. Falling back to device scanning.
  UUID emSwP2-fCya-vSee-BOCA-LUdO-qYCw-41Lc01 already in use on "/dev/mapper/20017380033e700e8-part3".

ccjung commented at 2019-03-18 19:34:

How do I unmap the previous disk before doing the restore?

Thanks.

ccjung commented at 2019-03-18 20:23:

It looks like ReaR was able to map the target disk 112GB disk (/dev/mapper/20017380033e70088)

Comparing disks
Ambiguous possible target disks need manual configuration (more than one with same size found)
Switching to manual disk layout configuration
Using /dev/mapper/20017380033e70088 (same size) for recreating /dev/mapper/200173800348802c6
Current disk mapping table (source => target):
/dev/mapper/200173800348802c6 => /dev/mapper/20017380033e70088

The SAN person told me that this is shared disk so 112G disk showed up twice in the "Listing multipath device found" list below and I was supposed to use the 20017380033e70088 LUN ,

20017380033e70088 dm-0 IBM,2810XIV size=112G
20017380033e700e8 dm-8 IBM,2810XIV size=112G

Listing multipath device found
20017380033e700dc dm-13 IBM,2810XIV size=32G
20017380033e700db dm-12 IBM,2810XIV size=32G
20017380033e700da dm-6 IBM,2810XIV size=32G
20017380033e70088 dm-0 IBM,2810XIV size=112G
20017380033e700e2 dm-19 IBM,2810XIV size=64G
20017380033e700d9 dm-5 IBM,2810XIV size=64G
20017380033e700e1 dm-18 IBM,2810XIV size=64G
20017380033e700d8 dm-4 IBM,2810XIV size=32G
20017380033e700e0 dm-17 IBM,2810XIV size=64G
20017380033e700e8 dm-8 IBM,2810XIV size=112G
20017380033e700df dm-16 IBM,2810XIV size=64G
20017380033e700e7 dm-7 IBM,2810XIV size=112G
20017380033e700de dm-15 IBM,2810XIV size=64G
20017380033e700dd dm-14 IBM,2810XIV size=32G

Is there a something we need to do differently in order to restore to a shared disk?
Thanks.

mahespth commented at 2019-03-18 21:25:

HI, ex IBM AIX guy here and this sounds like like a typical multi path/mpio configuration where the storage is presented to the lpar from two separate virtualisation servers for redundancy using vscsi drivers, they would be presented over two separate bus numbers and would expect mutltipath to be configured to deal with this. There is a REPO of tools for this if you do not already have it connected or available.

https://www14.software.ibm.com/webapp/set2/sas/f/lopdiags/yum.html

Hope this helps somehow.

On 18 Mar 2019, at 20:23, suseusr168 notifications@github.com wrote:

It looks like ReaR was able to map the target disk 112GB disk (/dev/mapper/20017380033e70088)

Comparing disks
Ambiguous possible target disks need manual configuration (more than one with same size found)
Switching to manual disk layout configuration
Using /dev/mapper/20017380033e70088 (same size) for recreating /dev/mapper/200173800348802c6
Current disk mapping table (source => target):
/dev/mapper/200173800348802c6 => /dev/mapper/20017380033e70088

The SAN person told me that this is shared disk so 112G disk showed up twice in the "Listing multipath device found" list below and I was supposed to use the 20017380033e70088 LUN ,

20017380033e70088 dm-0 IBM,2810XIV size=112G
20017380033e700e8 dm-8 IBM,2810XIV size=112G

Listing multipath device found
20017380033e700dc dm-13 IBM,2810XIV size=32G
20017380033e700db dm-12 IBM,2810XIV size=32G
20017380033e700da dm-6 IBM,2810XIV size=32G
20017380033e70088 dm-0 IBM,2810XIV size=112G
20017380033e700e2 dm-19 IBM,2810XIV size=64G
20017380033e700d9 dm-5 IBM,2810XIV size=64G
20017380033e700e1 dm-18 IBM,2810XIV size=64G
20017380033e700d8 dm-4 IBM,2810XIV size=32G
20017380033e700e0 dm-17 IBM,2810XIV size=64G
20017380033e700e8 dm-8 IBM,2810XIV size=112G
20017380033e700df dm-16 IBM,2810XIV size=64G
20017380033e700e7 dm-7 IBM,2810XIV size=112G
20017380033e700de dm-15 IBM,2810XIV size=64G
20017380033e700dd dm-14 IBM,2810XIV size=32G

Is there a something we need to do differently in order to restore to a shared disk?
Thanks.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub https://github.com/rear/rear/issues/2019#issuecomment-474085827, or mute the thread https://github.com/notifications/unsubscribe-auth/AHNLVUDaqS7RjP9lgmq9HWDs8grA6pvwks5vX_WvgaJpZM4aG0iu.

ccjung commented at 2019-03-19 01:39:

There are actually 3 - 112G disks on this system.

20017380033e70088 dm-0 IBM,2810XIV size=112G - this is for the rootvg

20017380033e700e8 dm-8 IBM,2810XIV size=112G - this is for shared_vg

20017380033e700e7 dm-7 IBM,2810XIV size=112G - this is for shared_vg

I might have confused myself thinking the shared_vg disks were shared disks between 2 systems because of the volume group name of shared_vg. I check the other partner HANA cluster system and they have different LUN ID numbers and they both have the volume group name shared_vg.

We want to restore the Rear backup to 20017380033e70088.

You had mentioned:
If you want to use /dev/mapper/20017380033e70088 instead of /dev/mapper/20017380033e700e8, you need to unmap the last one from the server first.

Do we unmap /dev/mapper/20017380033e700e8 from the SAN first before restoring?

ccjung commented at 2019-03-19 14:06:

Hello,
I have received confirmation from the SAN admin that the 2 SLES LPAR in the HANA cluster only share 1 heartbeat disk. The shared_vg disks are NOT shared even though the VG name has "shared" in it.

Do we have duplicate UUID numbers or something causing Rear to not be able to use disk 088 and use disk 0e8?

Thanks.

schabrolles commented at 2019-03-19 15:19:

Just a guess, but I think a previous "Rear restore" operation was made by mistake on /dev/mapper/20017380033e700e8 instead of /dev/mapper/20017380033e70088 ... (they have the same size and a pretty close wwid number)

This have recreate "3 partitions" on /dev/mapper/20017380033e700e8 and recreate a PV on /dev/mapper/20017380033e700e8-part3 with UUID emSwP2-fCya-vSee-BOCA-LUdO-qYCw-41Lc01

This prevent you to recreate your PV on /dev/mapper/20017380033e70088 (because /dev/mapper/20017380033e700e8 already has the UUID)

The solution could be to erase the configuration of /dev/mapper/20017380033e700e8.
Is the shared_vg part of the rear backup ?
Be sure to have a backup of your shared_vg, because it will wipe your disk.

if yes, you can clean the disk configuration by running the following dd command

dd if=/dev/zero of=/dev/mapper/20017380033e700e8 bs=1M count=1000

then reboot and try to restore again

ccjung commented at 2019-03-19 15:49:

Hi Sebastien,
I'll give it a try on Thursday when the Change Request to do this testing begins.

eniesdbp011p was the only system that had 3 - 112GB disks. Maybe the wrong disk was picked at one time. The serial number do look alike. This might explain why it keeps failing.

Is the shared_vg part of the rear backup ? No, only rootvg is in the Rear backup.

How to clean the disk configuration by running the following dd command.?

Do I boot the target system into the Rear rescue shell and run the dd command and then reboot and then try to restore again?

Thanks,
Cora

schabrolles commented at 2019-03-19 16:05:

Is the shared_vg part of the rear backup ? No, only rootvg is in the Rear backup.
So be aware that the dd command will destroy the shared_vg with the data.... This mean you will have to restore it from another backup if you have.

Anyway, if the previous operation use the wrong disk /dev/mapper/20017380033e700e8 this mean that it already destroy the shared_vg data....

How to clean the disk configuration by running the following dd command.?
Do I boot the target system into the Rear rescue shell and run the dd command and then reboot and then try to restore again?

Yes, you can boot the system in recue mode and run the dd command from here... then reboot and try to restore again on /dev/mapper/20017380033e70088

But before, you need to know: how to recover the shared_vg data if it is not part of rear and seems to be damaged ...

ccjung commented at 2019-03-19 17:18:

This is our disaster recovery server so we can wipe out this disk and don't need to worry about restoring shared_vg. This shared_vg will get recreated/restored in the next DR test.

Thanks.

ccjung commented at 2019-03-20 10:37:

Hello Sebastien,
I forgot that we also have another issue where another SLES12 SP4 server was able to restore using your workaround but it fails periodically. The SLES12 SP4 restore doesn't work all the time like SLES11 SP4.

I had attached the logs for the working restore and not working restore.

putty.2019.02.26.eniesdbp001p.log - Failed
putty.2019.02.26.eniesdbp001p.worked.log - Worked

Could you review the logs to see what was causing it to fail.

Thanks,

ccjung commented at 2019-03-20 14:59:

@mahespth ,
Thanks for the info. I think we have these tools installed. I found out we only have one shared disk for the heartbeat for SAP HANA cluster.

ccjung commented at 2019-03-21 21:24:

Disk 0e8 got zeroed out and I was able to restore the Rear backup.

This output was before the disk was zeroed out.

The disk 0e8 was unknown.

[unknown] shared_vg lvm2 a-m 112.18g 0

Welcome to Relax-and-Recover.

rear> pvs
WARNING: Failed to connect to lvmetad. Falling back to device scanning.
Couldn't find device with uuid cPe3uI-5X3T-P92O-uXyO-e8gA-f6wQ-QBtrht.
PV VG Fmt Attr PSize PFree
/dev/mapper/20017380033e700d8 iso_vg lvm2 a-- 32.05g 0
/dev/mapper/20017380033e700d9 tools_vg lvm2 a-- 64.10g 37.10g
/dev/mapper/20017380033e700da log_vg lvm2 a-- 32.05g 0
/dev/mapper/20017380033e700db log_vg lvm2 a-- 32.05g 0
/dev/mapper/20017380033e700dc log_vg lvm2 a-- 32.05g 0
/dev/mapper/20017380033e700dd log_vg lvm2 a-- 32.05g 0
/dev/mapper/20017380033e700de sap_vg lvm2 a-- 64.10g 0
/dev/mapper/20017380033e700df data_vg lvm2 a-- 64.10g 0
/dev/mapper/20017380033e700e0 data_vg lvm2 a-- 64.10g 0
/dev/mapper/20017380033e700e1 data_vg lvm2 a-- 64.10g 0
/dev/mapper/20017380033e700e2 data_vg lvm2 a-- 64.10g 0
/dev/mapper/20017380033e700e7 shared_vg lvm2 a-- 112.18g 0
/dev/mapper/20017380033e700e8-part3 system_vg lvm2 a-- 111.43g 26.43g
[unknown] shared_vg lvm2 a-m 112.18g 0
rear>

In /dev/mapper directory, disks 0e8 and 088 had -part1, -part2, -part3.

rear> cd /dev/mapper
rear> ls -trl
total 0
crw------- 1 root root 10, 236 Mar 21 15:03 control
lrwxrwxrwx 1 root root 8 Mar 21 15:03 20017380033e700e8_part3 -> ../dm-18
lrwxrwxrwx 1 root root 8 Mar 21 15:03 20017380033e700e8-part3 -> ../dm-18
lrwxrwxrwx 1 root root 8 Mar 21 15:03 20017380033e700e8_part1 -> ../dm-16
lrwxrwxrwx 1 root root 8 Mar 21 15:03 20017380033e700e8-part1 -> ../dm-16
lrwxrwxrwx 1 root root 8 Mar 21 15:03 20017380033e700e8_part2 -> ../dm-17
lrwxrwxrwx 1 root root 8 Mar 21 15:03 20017380033e700e8-part2 -> ../dm-17
lrwxrwxrwx 1 root root 7 Mar 21 15:03 20017380033e700d8 -> ../dm-4
lrwxrwxrwx 1 root root 7 Mar 21 15:03 20017380033e70088 -> ../dm-0
lrwxrwxrwx 1 root root 7 Mar 21 15:03 20017380033e700e7 -> ../dm-7
lrwxrwxrwx 1 root root 7 Mar 21 15:03 20017380033e700d9 -> ../dm-5
lrwxrwxrwx 1 root root 7 Mar 21 15:03 20017380033e700db -> ../dm-8
lrwxrwxrwx 1 root root 7 Mar 21 15:03 20017380033e700da -> ../dm-6
lrwxrwxrwx 1 root root 8 Mar 21 15:03 20017380033e700e8 -> ../dm-15
lrwxrwxrwx 1 root root 8 Mar 21 15:03 20017380033e700e0 -> ../dm-19
lrwxrwxrwx 1 root root 8 Mar 21 15:03 20017380033e700dd -> ../dm-10
lrwxrwxrwx 1 root root 7 Mar 21 15:03 20017380033e700dc -> ../dm-9
lrwxrwxrwx 1 root root 7 Mar 21 15:03 20017380033e70088_part1 -> ../dm-1
lrwxrwxrwx 1 root root 7 Mar 21 15:03 20017380033e70088-part1 -> ../dm-1
lrwxrwxrwx 1 root root 8 Mar 21 15:03 20017380033e700e1 -> ../dm-13
lrwxrwxrwx 1 root root 8 Mar 21 15:03 20017380033e700df -> ../dm-12
lrwxrwxrwx 1 root root 8 Mar 21 15:03 20017380033e700de -> ../dm-11
lrwxrwxrwx 1 root root 7 Mar 21 15:03 20017380033e70088_part3 -> ../dm-3
lrwxrwxrwx 1 root root 7 Mar 21 15:03 20017380033e70088-part3 -> ../dm-3
lrwxrwxrwx 1 root root 8 Mar 21 15:03 20017380033e700e2 -> ../dm-14
lrwxrwxrwx 1 root root 7 Mar 21 15:03 20017380033e70088_part2 -> ../dm-2
lrwxrwxrwx 1 root root 7 Mar 21 15:03 20017380033e70088-part2 -> ../dm-2
rear> dd if=/dev/zero of=/dev/mapper/20017380033e700e8 bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 0.110047 s, 9.5 GB/s

0e8 and 088 looks similar without eyeglasses. I must have picked the wrong disk at one point and the wrong disk kept the UUID used by 088.

Thanks for the suggestion.

ccjung commented at 2019-03-25 14:53:

I am testing the restore and it worked 2 times was able to select the 088 as the correct target disk.

The third time, it is picking a different SIBM target disk.

Current disk mapping table (source => target):
  /dev/mapper/200173800348802c6 => /dev/mapper/SIBM_2810XIV_78033E700E7

Which disk should I zero out?

Listing multipath device found
20017380033e700dc dm-6 IBM,2810XIV size=32G
SIBM_2810XIV_78033E700DA dm-11 IBM,2810XIV size=32G
20017380033e700db dm-5 IBM,2810XIV size=32G
20017380033e700da dm-2 IBM,2810XIV size=32G
SIBM_2810XIV_78033E700E7 dm-12 IBM,2810XIV size=112G
20017380033e700e2 dm-1 IBM,2810XIV size=64G
20017380033e700e1 dm-0 IBM,2810XIV size=64G
20017380033e700e0 dm-10 IBM,2810XIV size=64G
20017380033e700e8 dm-4 IBM,2810XIV size=112G
20017380033e700df dm-9 IBM,2810XIV size=64G
20017380033e700e7 dm-3 IBM,2810XIV size=112G
20017380033e700de dm-8 IBM,2810XIV size=64G
20017380033e700dd dm-7 IBM,2810XIV size=32G
Comparing disks
Ambiguous possible target disks need manual configuration (more than one with same size found)
Switching to manual disk layout configuration
Using /dev/mapper/SIBM_2810XIV_78033E700E7 (same size) for recreating /dev/mapper/200173800348802c6
Current disk mapping table (source => target):
  /dev/mapper/200173800348802c6 => /dev/mapper/SIBM_2810XIV_78033E700E7

UserInput -I LAYOUT_MIGRATION_CONFIRM_MAPPINGS needed in /usr/share/rear/layout/prepare/default/300_map_disks.sh line 275
Confirm or edit the disk mapping
1) Confirm disk mapping and continue 'rear recover'
2) n/a
3) Edit disk mapping (/var/lib/rear/layout/disk_mappings)
4) Use Relax-and-Recover shell and return back to here
5) Abort 'rear recover'
(default '1' timeout 300 seconds)
4
UserInput: Valid choice number result 'Use Relax-and-Recover shell and return back to here'

Welcome to Relax-and-Recover.

rear> cd /dev/mapper
rear> ls -al
total 0
drwxr-xr-x  2 root root     540 Mar 25 10:44 .
drwxr-xr-x 11 root root    4180 Mar 25 10:44 ..
lrwxrwxrwx  1 root root       8 Mar 25 10:44 20017380033e70088 -> ../dm-15
lrwxrwxrwx  1 root root       8 Mar 25 10:44 20017380033e70088-part1 -> ../dm-17
lrwxrwxrwx  1 root root       8 Mar 25 10:44 20017380033e70088-part2 -> ../dm-18
lrwxrwxrwx  1 root root       8 Mar 25 10:44 20017380033e70088-part3 -> ../dm-19
lrwxrwxrwx  1 root root       8 Mar 25 10:44 20017380033e70088_part1 -> ../dm-17
lrwxrwxrwx  1 root root       8 Mar 25 10:44 20017380033e70088_part2 -> ../dm-18
lrwxrwxrwx  1 root root       8 Mar 25 10:44 20017380033e70088_part3 -> ../dm-19
lrwxrwxrwx  1 root root       8 Mar 25 10:44 20017380033e700d8 -> ../dm-13
lrwxrwxrwx  1 root root       8 Mar 25 10:44 20017380033e700d9 -> ../dm-14
lrwxrwxrwx  1 root root       7 Mar 25 10:44 20017380033e700da -> ../dm-2
lrwxrwxrwx  1 root root       7 Mar 25 10:44 20017380033e700db -> ../dm-5
lrwxrwxrwx  1 root root       7 Mar 25 10:44 20017380033e700dc -> ../dm-6
lrwxrwxrwx  1 root root       7 Mar 25 10:44 20017380033e700dd -> ../dm-7
lrwxrwxrwx  1 root root       7 Mar 25 10:44 20017380033e700de -> ../dm-8
lrwxrwxrwx  1 root root       7 Mar 25 10:44 20017380033e700df -> ../dm-9
lrwxrwxrwx  1 root root       8 Mar 25 10:44 20017380033e700e0 -> ../dm-10
lrwxrwxrwx  1 root root       7 Mar 25 10:44 20017380033e700e1 -> ../dm-0
lrwxrwxrwx  1 root root       7 Mar 25 10:44 20017380033e700e2 -> ../dm-1
lrwxrwxrwx  1 root root       7 Mar 25 10:44 20017380033e700e7 -> ../dm-3
lrwxrwxrwx  1 root root       7 Mar 25 10:44 20017380033e700e8 -> ../dm-4
lrwxrwxrwx  1 root root       8 Mar 25 10:44 SIBM_2810XIV_78033E700D8 -> ../dm-16
lrwxrwxrwx  1 root root       8 Mar 25 10:44 SIBM_2810XIV_78033E700D9 -> ../dm-20
lrwxrwxrwx  1 root root       8 Mar 25 10:44 SIBM_2810XIV_78033E700DA -> ../dm-11
lrwxrwxrwx  1 root root       8 Mar 25 10:44 SIBM_2810XIV_78033E700E7 -> ../dm-12
crw-------  1 root root 10, 236 Mar 25 10:44 control
rear>

ccjung commented at 2019-03-25 15:32:

The SAN admin told me the following:

That's one of the 2 X 112GB LUNs requested for the shared_vg:

20017380033e700e7 dm-3 IBM,2810XIV size=112G
20017380033e700e8 dm-4 IBM,2810XIV size=112G

Will it cause problems for Rear to select the correct target disk if there are more than 1 disk that is the same size (eg. we have three 112GB disks but one of the disk should be the OS rootvg disk)?

schabrolles commented at 2019-03-25 18:11:

Yes, because the original disk cannot be found, rear will then find a disk with the same size. Because you have 2 disks with this criteria, rear can choose one or the other.

This is why, it proposes you a mapping ... you can correct it if you want, but the warning messages are normal: "Ambiguous possible target disks...."

ccjung commented at 2019-03-25 18:39:

On the target system in the /dev/mapper directory, I see the following SIBM softlinks:

lrwxrwxrwx 1 root root 8 Mar 25 10:44 SIBM_2810XIV_78033E700D8 -> ../dm-16
lrwxrwxrwx 1 root root 8 Mar 25 10:44 SIBM_2810XIV_78033E700D9 -> ../dm-20
lrwxrwxrwx 1 root root 8 Mar 25 10:44 SIBM_2810XIV_78033E700DA -> ../dm-11
lrwxrwxrwx 1 root root 8 Mar 25 10:44 SIBM_2810XIV_78033E700E7 -> ../dm-12

I don't see these SIBM softlinks on the source system.

What are they used for?

Isn't this a problem if we see these SIBM softlinks or see /dev/sda?

ccjung commented at 2019-03-26 12:13:

Everytime the disk layout recreation script failed, I cleared the target disk with zeroes, reboot it and the next restore attempt works.

Should I be clearing out the target disk (which is not newly created and been used before) with zeros before I even attempt a restore?

What are these files below which has SIBM in the name used for on the target system?

lrwxrwxrwx 1 root root 8 Mar 25 10:44 SIBM_2810XIV_78033E700D8 -> ../dm-16
lrwxrwxrwx 1 root root 8 Mar 25 10:44 SIBM_2810XIV_78033E700D9 -> ../dm-20
lrwxrwxrwx 1 root root 8 Mar 25 10:44 SIBM_2810XIV_78033E700DA -> ../dm-11
lrwxrwxrwx 1 root root 8 Mar 25 10:44 SIBM_2810XIV_78033E700E7 -> ../dm-12

Thanks.

jsmeix commented at 2019-03-26 13:12:

@suseusr168
yes, as long as we do not have a "cleanupdisk" script in ReaR,
cf. https://github.com/rear/rear/issues/799
you should clear out your target disk if it had been ever used before
to be in general on the safe side against unexpected weird issues
because of whatever kind of remainder data on an already used disk.

You may look through the other isues that are linked in
https://github.com/rear/rear/issues/799
perhaps there was already one that matches your particular case here.

jsmeix commented at 2019-03-26 13:34:

@suseusr168
usually it should be sufficient to use wipefs to clear out your target disk,
see https://github.com/rear/rear/issues/799 for some example.

wipefs is very much faster than zeroing out a whole big disk
because wipefs clears out only some known small areas on the disk
where filesystem, raid or partition-table signatures are usually located.

On the other hand it could happen that wipefs misses to clear out
special kind of currently not yet known metadata so that zeroing out
the whole disk should be the 100% fail-safe way - in particular
when you can do that in advance when you have time for it
to prepare a disk to be later re-used for disaster recovery, cf.
A note on the meaning of 'Relax' in 'Relax-and-Recover'
in https://en.opensuse.org/SDB:Disaster_Recovery

ccjung commented at 2019-03-26 14:23:

@jsmeix ,
I ran the find command and I don't see cleanupdisk

I ran this command and don't see the cleanupdisk
rpm -ql rear-2.4-198.git.0.e9f9d4b.unknown.ppc64le|grep -i clean

How do I find out if we have the cleanupdisk script?

Do I run the wipefs command in the ReaR rescue shell and reboot and try the restore?

ccjung commented at 2019-03-26 14:47:

@jsmeix ,
What are these files below which has SIBM in the name used for on the target system?

lrwxrwxrwx 1 root root 8 Mar 25 10:44 SIBM_2810XIV_78033E700D8 -> ../dm-16
lrwxrwxrwx 1 root root 8 Mar 25 10:44 SIBM_2810XIV_78033E700D9 -> ../dm-20
lrwxrwxrwx 1 root root 8 Mar 25 10:44 SIBM_2810XIV_78033E700DA -> ../dm-11
lrwxrwxrwx 1 root root 8 Mar 25 10:44 SIBM_2810XIV_78033E700E7 -> ../dm-12

jsmeix commented at 2019-03-26 15:26:

@suseusr168
in https://github.com/rear/rear/issues/2019#issuecomment-476624517 I wrote

as long as we do not have a "cleanupdisk" script in ReaR,
cf. https://github.com/rear/rear/issues/799

i.e. we do not yet have any kind of "cleanupdisk" script in ReaR and
accordingly https://github.com/rear/rear/issues/799 is still in state Open.

I am neither a SAN nor a multipath user so that I cannot provide any real
useful help in those areas - @schabrolles is our ReaR upstream expert
for IBM POWER in general and for SAN and multipath in particular.

ccjung commented at 2019-03-26 18:57:

@jsmeix ,
For the wipefs, do we need to reboot after running it?

Is the wipefs run from the Rear Rescue shell?

Is the following correct order?

  1. Clear the used target disk (either use dd or wipefs command)
  2. Restore the backup to the used target disk

We didn't need to clear the disk for SLES11 SP4 restores. We seem to have more issues with SLES12 SP4 restores.

Thanks!

jsmeix commented at 2019-03-27 12:15:

@suseusr168
according to my experience with wipefs you do not need to reboot,
cf. https://github.com/rear/rear/issues/799#issuecomment-197286229
but I needed a final parted /dev/sda mklabel ...
to make the old/outdated partition device nodes go away.

How device nodes appear and disaapear depends on kernel, udev,
and what tools like parted, partprobe, and others... do
so that device nodes behaviour depends on the particular
currently running system - and SLES12 is certainly different
in those areas comaped to SLES11.

I did run the disk cleanup commands manually
in the ReaR recovery system directly after login as root,
cf. https://github.com/rear/rear/issues/799#issuecomment-197286229

After the disk cleanup commands succeded
one can run rear -D recover.

I don't know what is different in SLES11 versus SLES12
why SLES12 causes more issues in your particular case
when (re)installing onto a disk that was already used before
cf. "Disaster recovery means installation (reinstalling from scratch)" at
https://en.opensuse.org/SDB:Disaster_Recovery

For some basic info what I learned about what could happen
at higher levels when there is a higher stack of storage objects
you may have a look at
https://github.com/rear/rear/issues/791#issuecomment-196212960

I am not at all a kernel, udev, partitioning, MD, LVM, whatever...
expert. I only report what "I was told", see also
https://github.com/rear/rear/issues/791

ccjung commented at 2019-03-27 17:13:

@jsmeix
I went into the Rear Rescue shell:
Welcome to Relax-and-Recover.

rear> cd /dev/mapper

rear> ls -trl
total 0
crw------- 1 root root 10, 236 Mar 27 12:52 control
lrwxrwxrwx 1 root root       7 Mar 27 12:52 20017380033e70071 -> ../dm-0
lrwxrwxrwx 1 root root       7 Mar 27 12:52 20017380033e70092 -> ../dm-5
lrwxrwxrwx 1 root root       7 Mar 27 12:52 20017380033e70093 -> ../dm-6
lrwxrwxrwx 1 root root       7 Mar 27 12:52 20017380033e70091 -> ../dm-4
lrwxrwxrwx 1 root root       7 Mar 27 12:52 20017380033e700a1 -> ../dm-8
lrwxrwxrwx 1 root root       8 Mar 27 12:52 20017380033e70097 -> ../dm-12
lrwxrwxrwx 1 root root       8 Mar 27 12:52 20017380033e70096 -> ../dm-11
lrwxrwxrwx 1 root root       7 Mar 27 12:52 20017380033e700a0 -> ../dm-7
lrwxrwxrwx 1 root root       8 Mar 27 12:52 20017380033e70095 -> ../dm-10
lrwxrwxrwx 1 root root       8 Mar 27 12:52 20017380033e70098 -> ../dm-13
lrwxrwxrwx 1 root root       7 Mar 27 12:52 20017380033e70094 -> ../dm-9
lrwxrwxrwx 1 root root       8 Mar 27 12:52 20017380033e7009a -> ../dm-15
lrwxrwxrwx 1 root root       8 Mar 27 12:52 20017380033e70099 -> ../dm-14
lrwxrwxrwx 1 root root       7 Mar 27 12:52 20017380033e70071_part1 -> ../dm-1
lrwxrwxrwx 1 root root       7 Mar 27 12:52 20017380033e70071-part1 -> ../dm-1
lrwxrwxrwx 1 root root       8 Mar 27 12:52 20017380033e7009b -> ../dm-16
lrwxrwxrwx 1 root root       7 Mar 27 12:52 20017380033e70071_part2 -> ../dm-2
lrwxrwxrwx 1 root root       7 Mar 27 12:52 20017380033e70071-part2 -> ../dm-2
lrwxrwxrwx 1 root root       7 Mar 27 12:52 20017380033e70071_part3 -> ../dm-3
lrwxrwxrwx 1 root root       7 Mar 27 12:52 20017380033e70071-part3 -> ../dm-3

rear> cd /dev/mapper

rear> parted /dev/mapper/20017380033e70071 print
Model: Linux device-mapper (multipath) (dm)
Disk /dev/mapper/20017380033e70071: 120GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  8193kB  7144kB  primary               boot, prep, type=41
 2      272MB   806MB   535MB   primary  ext3         type=83
 3      806MB   120GB   120GB   primary               lvm, type=8e

Do I run the following wipefs command or do I also run the wipefs on all the -partX?

wipefs -a -f /dev/mapper/20017380033e70071

or

wipefs -a -f /dev/mapper/20017380033e70071
wipefs -a -f /dev/mapper/20017380033e70071-part1
wipefs -a -f /dev/mapper/20017380033e70071-part2
wipefs -a -f /dev/mapper/20017380033e70071-part3

gdha commented at 2019-03-27 17:41:

Is this issue a blocker for release or a very long thread on support with rear? Sorry too much to read in my free time.

ccjung commented at 2019-03-27 17:47:

This issue is a very long thread on support with Rear.

We are having intermittent problems with restore on SLES12 SP4 on Power. It works some of the time.

I am testing to see if clearing out previously used disk before doing a restore will fix the intermittent restore problems.

jsmeix commented at 2019-03-29 10:16:

@suseusr168
regarding your
https://github.com/rear/rear/issues/2019#issuecomment-477261312
do I also run the wipefs on all the -partX?

You need to run wipefs in reverse ordering on each storage object
starting from higher level storage objects (like partitions)
down to lower level storage objects (like whole disks)
for example first on all partitions on a disk, and then on the whole disk,
and finally parted mklabel could be additionally needed too
cf. https://github.com/rear/rear/issues/799#issue-141001306

wipefs -a -f /dev/sda2
wipefs -a -f /dev/sda1
wipefs -a -f /dev/sda

plus perhaps https://github.com/rear/rear/issues/799#issuecomment-197286229

parted /dev/sda mklabel gpt

so that in your case in
https://github.com/rear/rear/issues/2019#issuecomment-477261312
wipefs should be called this way

# wipefs -a -f /dev/mapper/20017380033e70071-part3
# wipefs -a -f /dev/mapper/20017380033e70071-part2
# wipefs -a -f /dev/mapper/20017380033e70071-part1
# wipefs -a -f /dev/mapper/20017380033e70071

plus perhaps a final

# parted /dev/mapper/20017380033e70071 mklabel msdos

so that in the end parted /dev/mapper/20017380033e70071 print
shows only the plain disk without any partitions on it like

# parted /dev/mapper/20017380033e70071 print
Model: Linux device-mapper (multipath) (dm)
Disk /dev/mapper/20017380033e70071: 120GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number  Start   End     Size    Type     File system  Flags

and a command like

# find /dev -ls | grep -o '/dev/.*' | grep '20017380033e70071'

shows only the plain whole disk device node and symbolic links to it
but no longer partition device nodes or symbolic links to them.

Careful!

On what device nodes you have to run wipefs depends on
what there actually is on your current already used target disk
(not on what should be installed later on that target disk).

What stack of storage objects there currently is on a disk
should be shown by the command

# lsblk -i -p -o NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT

For example on my currently running SLES12 test system with LVM I get

# lsblk -i -p -o NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT
NAME                        KNAME     PKNAME    TRAN TYPE FSTYPE       SIZE MOUNTPOINT
/dev/sda                    /dev/sda            ata  disk               30G 
|-/dev/sda1                 /dev/sda1 /dev/sda       part vfat         156M /boot/efi
`-/dev/sda2                 /dev/sda2 /dev/sda       part LVM2_member 29.9G 
  |-/dev/mapper/system-root /dev/dm-0 /dev/sda2      lvm  btrfs         12G /
  |-/dev/mapper/system-swap /dev/dm-1 /dev/sda2      lvm  swap         1.4G [SWAP]
  `-/dev/mapper/system-home /dev/dm-2 /dev/sda2      lvm  xfs         16.4G /home

If I would get such an output within the ReaR recovery system
for an already used target disk I would run wipefs on all
those storage device nodes in reverse ordering
of what the KNAME column of lsblk lists like

# wipefs -a -f /dev/dm-2
# wipefs -a -f /dev/dm-1
# wipefs -a -f /dev/dm-0
# wipefs -a -f /dev/sda2
# wipefs -a -f /dev/sda1
# wipefs -a -f /dev/sda

For me when I am in the ReaR recovery system with a target disk
that was already used as in my running SLES12 test system with LVM
I do not get the 'lvm' TYPE /dev/dm* storage device nodes listed by lsblk.

In my case I get only the 'disk' and 'part' TYPE storage device nodes
/dev/sda, /dev/sda1, and /dev/sda2 listed by lsblk
so that in my case I would do only

# wipefs -a -f /dev/sda2
# wipefs -a -f /dev/sda1
# wipefs -a -f /dev/sda

to clean up the target disk.

To get lsblk included in the ReaR recovery system
you may need to set in your /etc/rear/local.conf

PROGS=( "${PROGS[@]}" lsblk )

Since
https://github.com/rear/rear/commit/b4fd97fd47d8c15ca1383dd911ec018780dce77b
(which had happened after the ReaR 2.4 release in June 2018))
lsblk is included in the recovery system by default.

ccjung commented at 2019-03-29 11:54:

@jsmeix
Thanks for the detailed information.

I wasn't sure what was the correct way to run wipefs so I just ran the following (the lower storage object):

wipefs -a -f /dev/mapper/20017380033e70071

The LPAR has 16 GB of Memory and 10 CPUs. Is that enough to bootup?

It was able to restore but when I reboot the LPAR, I get the following errors on bootup:

FAILED] Failed to start Notify NFS peers of a restart.
See 'systemctl status rpc-statd-notify.service' for details.
[ OK ] Stopped YaST2 Firstboot.
[ OK ] Stopped LSB: Start the sapstartsrv.
[ OK ] Reached target Network is Online.
[ OK ] Reached target Remote File Systems.
[ OK ] Reached target System Time Synchronized.
[ OK ] Stopped YaST2 Second Stage.
[ OK ] Stopped target Basic System.
[ OK ] Reached target Sockets.
[ OK ] Stopped target System Initialization.
[FAILED] Failed to start Create Volatile Files and Directories.
See 'systemctl status systemd-tmpfiles-setup.service' for details.
[FAILED] Failed to start Update UTMP about System Boot/Shutdown.
See 'systemctl status systemd-update-utmp.service' for details.
[DEPEND] Dependency failed for Update UTMP about System Runlevel Changes.
[FAILED] Failed to start Tell Plymouth To Write Out Runtime Data.
See 'systemctl status plymouth-read-write.service' for details.
[FAILED] Failed to start Emergency Shell.
See 'systemctl status emergency.service' for details.
[DEPEND] Dependency failed for Emergency Mode.
[** ] (1 of 14) A start job is running fo...ce 254:17 (14min 16s / no limit)

Any idea what is causing this? Is it because I only ran wipefs on the lower storage object?

Also, for the dd command do I run on all objects in the same order as well like the wipefs?

dd if=/dev/zero of=/dev/mapper/20017380033e70071 bs=1M count=1000

jsmeix commented at 2019-03-29 13:37:

@suseusr168
I can say nothing about booting issues with a PPC64LE PowerVM LPAR
because I am not an IBM POWER user so that I cannot provide any
real useful help in those areas - @schabrolles is our ReaR upstream expert
for IBM POWER in general and for booting a PPC64LE PowerVM LPAR in particular.

I think when you only wipefs a lower-level storage object (e.g. a whole disk)
without wipefs each one of its higher-level storage objects (e.g. its partitions),
it could happen that wipefs on the lower-level storage object
misses to clear out certain kind of signatures (magic strings)
from its higher-level storage objects so that those missed
signatures are still there and could cause issues later.

This is the reason behind why I recommend - to be on the safe side - to run
wipefs in reverse ordering on each storage object (listed by lsblk as KNAME)
starting from higher-level storage objects (like partitions)
down to lower-level storage objects (like whole disks).

Because all existing storage objects on an already used target disk are
old, outdated, and useless and should be all completely overwritten by the
new storage objects that get created from scratch by "rear recover" on that disk,
it cannot cause harm if wipefs is run on more of those old storage objects
than what would be minimally needed to fully clear out the target disk.

In contrast when you use dd if=/dev/zero of=/dev/whole_disk
you zero out the whole disk from its very first byte to it very last byte
which implicitly zeroes out all bytes that belong to all its higher-level
storage objects so that dd if=/dev/zero of=/dev/whole_disk
should be the 100% fail-safe way to get any kind of unwanted
remainders on an already used target disk out of the way, cf.
https://github.com/rear/rear/issues/2019#issuecomment-476637491

@suseusr168
when you use

dd if=/dev/zero of=/dev/mapper/20017380033e70071 bs=1M count=1000

and then run "rear recover" does then reboot of the recreated LPAR work?

I ask because your
https://github.com/rear/rear/issues/2019#issuecomment-476598723

Everytime the disk layout recreation script failed,
I cleared the target disk with zeroes, reboot it
and the next restore attempt works.

indicates that when you did dd if=/dev/zero of=/dev/whole_disk
before "rear recover", then reboot of your recreated LPAR had worked.

ccjung commented at 2019-03-29 14:24:

@jsmeix
Thanks for the info.

I might just use the dd command to zero it out, reboot and then restore .

It is one command to type and less chance to make human errors.

jsmeix commented at 2019-03-29 14:49:

I added a new section
"Prepare replacement hardware for disaster recovery" to
https://en.opensuse.org/SDB:Disaster_Recovery

jsmeix commented at 2019-04-03 14:50:

According to
https://github.com/rear/rear/issues/2019#issuecomment-478016347
I think this issue is sufficiently answered so that I can close it.
@suseusr168 if not I can of course reopen it.

ccjung commented at 2019-04-03 15:29:

@jsmeix ,
We didn't have the test systems from last Wednesday till today to do any testing.

Today, I did some more repetitive restore testing and I am seeing some weird results.

Here was my steps for today:

  1. Zero out the OS target disk
  2. Reboot
  3. Do a Rear restore
    I sometimes see /dev/sda or /dev/SIBMxxxx as the the target disk.

If I reboot once or twice again, I see the correct target disk.

Last week, I just zero the disk once and was able to reboot and restore several times.

  1. Zero target disk
  2. Reboot
  3. Do a Rear restore
  4. Repeat steps 2 and step 3 several times with no problems

Not sure what is the problem.

Can you reopen this issue again?

Thanks!

ccjung commented at 2019-04-03 16:32:

Getting different errors now (can't find the disk):
The failures are at the end of the putty log.

putty_10.253.196.17_20190403_091717.log

preferably with full debug information via '''rear -D recover'''

===================='
+++ echo '2019-04-03 12:21:45.405365550 ERROR: 
====================
BUG in /usr/share/rear/lib/layout-functions.sh line 512:
'\''No device returned for major/minor 254:12'\''
--------------------
Please report this issue at https://github.com/rear/rear/issues
and include the relevant parts from /var/log/rear/rear-eniesdbp101p.log
preferably with full debug information via '\''rear -D recover'\''
===================='
2019-04-03 12:21:45.405365550 ERROR: 
====================
BUG in /usr/share/rear/lib/layout-functions.sh line 512:
'No device returned for major/minor 254:12'
--------------------
Please report this issue at https://github.com/rear/rear/issues
and include the relevant parts from /var/log/rear/rear-eniesdbp101p.log
preferably with full debug information via 'rear -D recover'
====================
+++ LogToSyslog 'ERROR: 
====================
BUG in /usr/share/rear/lib/layout-functions.sh line 512:
'\''No device returned for major/minor 254:12'\''
--------------------
Please report this issue at https://github.com/rear/rear/issues
and include the relevant parts from /var/log/rear/rear-eniesdbp101p.log
preferably with full debug information via '\''rear -D recover'\''
===================='
+++ logger -t rear -i 'ERROR: 
====================
BUG in /usr/share/rear/lib/layout-functions.sh line 512:
'\''No device returned for major/minor 254:12'\''
--------------------
Please report this issue at https://github.com/rear/rear/issues
and include the relevant parts from /var/log/rear/rear-eniesdbp101p.log
preferably with full debug information via '\''rear -D recover'\''

ccjung commented at 2019-04-03 18:48:

I used a new backup created today was able to bypass the error in the previous post.

Here are the consistent results:

  1. Boot up LPAR and see diskrestore error
  2. Clear the target OS disk
  3. Shutdown/power up LPAR and see wrong disk mapping (eg. /dev/sda)
  4. Shutdown/power up LPAR and see wrong disk mapping (eg. /dev/SIBM.xxx)
  5. Shutdown/power up PLAR and see correct disk mapping (eg. /dev/mapper/Disk_Serial_Number
  6. Able to restore backup

I am not sure what happened from last week vs this week (same backup was used for testing last week and this week).

Any suggestions on how to restore SLES12SP4 with one try instead of rebooting several times to get the correct disk mapping in order to restore?

Thank you!

jsmeix commented at 2019-04-04 08:02:

@dewagner1
I think as long as you do not see the expected disk mapping like

/dev/mapper/<Disk_Serial_Number>

but something different like /dev/sda
or even a SIBM.xxx named device
things will not work.

According to your
https://github.com/rear/rear/issues/2019#issuecomment-479613396
it looks like some obscure timing related issue during system startup
that sometimes you get the expected disk mapping
and sometimes something different.

@schabrolles
could you have a look here?
Perhaps you have an idea what timing related issue
during ReaR recovery system startup
could cause such differences?

@dewagner1
I assume when you shutdown/power up several times another
LPAR with a pristine SLES 12 system you always get
the expected disk mapping /dev/mapper/<Disk_Serial_Number>
any you never get unexpected disks like /dev/sda
or even a SIBM.xxx named device.

I.e. I would like to know if the unexpected disks only appear
in case of a ReaR recovery system startup or if that also
could happen with an original SLES 12 system.

ccjung commented at 2019-04-04 10:22:

@jsmeix
The unexpected disk mapping occurs with both pristine and previously used disk.

We haven't recently restored the backup to itself.

We restore the backup normally for the following:

  1. to clone a new server which has pristine disks
  2. to test Disaster Recovery which will have previously used disks.

Clearing out the previously used disk did help fix the diskrestore script failures. But, we see the unexpected disk mapping twice before we see the correct disk mapping on the third try.

Last week, after I zeroed out the used disk, I was able to see the expected disk mapping and do a restore 5 times with no problems on 2 different LPARs using the same backup.

This week (using the same system and backup), after I zero out the used disk, I see unexpected disk mapping. I had to shutdown/power up three times. The third time is when I can see the expected disk mapping and do the restore.

jsmeix commented at 2019-04-04 11:28:

@dewagner1
I meant that I like to know if unexpected disks also appear sometimes
when you boot such an LPAR with SLES 12 several times
(when you boot a ReaR recovery system it is clear
that the unexpected disks appear sometimes).

ccjung commented at 2019-04-04 13:54:

@jsmeix ,
Do you mean boot up an existing LPAR with SLES12SP3?

jsmeix commented at 2019-04-04 14:15:

@dewagner1
yes, of course not a production system.

You may for example use the same testing LPAR
where you do your above "rear recover" tests.
Do a successful "rear recover" to get a SLES 12 system
onto your testing LPAR and then re-boot that recreated
SLES 12 system several times and each time inspect it
if during SLES 12 startup such unexpected disks also
appear sometimes there.

I like to find out if the root cause is only inside the ReaR recovery system
or if something from outside is involved - in the latter case I would expect
that the unexpected disks also appear sometimes during SLES 12 startup.

jsmeix commented at 2019-04-04 14:18:

@dewagner1
in https://github.com/rear/rear/issues/2019#issuecomment-479613396
I noticed you use meanwhile SLES12SP4
I had SLES12 SP3 from https://github.com/rear/rear/issues/2016
I don't care about the SLES12 service pack version.
Use SLES12 SP3 or SLES12 SP4 as you like.

I removed SP3 in my above comments.

ccjung commented at 2019-04-04 16:05:

@jsmeix,
Do you mean after the restore, check if there are any unexpected disk mapping in the /dev/mapper directory?

ccjung commented at 2019-04-04 16:09:

I just did a restore and I don't see any SIBM or sda unexpected disk mappings.

eniesdbp001p:/dev/mapper # ls -al
total 0
drwxr-xr-x  2 root root     840 Apr  4 11:20 .
drwxr-xr-x 23 root root    6660 Apr  4 11:20 ..
lrwxrwxrwx  1 root root       7 Apr  4 11:20 20017380033e70071 -> ../dm-0
lrwxrwxrwx  1 root root       7 Apr  4 11:20 20017380033e70071-part1 -> ../dm-2
lrwxrwxrwx  1 root root       7 Apr  4 11:20 20017380033e70071-part2 -> ../dm-3
lrwxrwxrwx  1 root root       7 Apr  4 11:20 20017380033e70071-part3 -> ../dm-4
lrwxrwxrwx  1 root root       7 Apr  4 11:20 20017380033e70071_part1 -> ../dm-2
lrwxrwxrwx  1 root root       7 Apr  4 11:20 20017380033e70071_part2 -> ../dm-3
lrwxrwxrwx  1 root root       7 Apr  4 11:20 20017380033e70071_part3 -> ../dm-4
lrwxrwxrwx  1 root root       7 Apr  4 11:20 20017380033e70091 -> ../dm-1
lrwxrwxrwx  1 root root       7 Apr  4 11:20 20017380033e70092 -> ../dm-5
lrwxrwxrwx  1 root root       7 Apr  4 11:20 20017380033e70093 -> ../dm-6
lrwxrwxrwx  1 root root       7 Apr  4 11:20 20017380033e70094 -> ../dm-9
lrwxrwxrwx  1 root root       8 Apr  4 11:20 20017380033e70095 -> ../dm-10
lrwxrwxrwx  1 root root       8 Apr  4 11:20 20017380033e70096 -> ../dm-12
lrwxrwxrwx  1 root root       8 Apr  4 11:20 20017380033e70097 -> ../dm-13
lrwxrwxrwx  1 root root       8 Apr  4 11:20 20017380033e70098 -> ../dm-14
lrwxrwxrwx  1 root root       8 Apr  4 11:20 20017380033e70099 -> ../dm-15
lrwxrwxrwx  1 root root       8 Apr  4 11:20 20017380033e7009a -> ../dm-17
lrwxrwxrwx  1 root root       8 Apr  4 11:20 20017380033e7009b -> ../dm-18
lrwxrwxrwx  1 root root       7 Apr  4 11:20 20017380033e700a0 -> ../dm-7
lrwxrwxrwx  1 root root       7 Apr  4 11:20 20017380033e700a1 -> ../dm-8
crw-------  1 root root 10, 236 Apr  4 11:20 control
lrwxrwxrwx  1 root root       8 Apr  4 11:20 data_vg-data_lv -> ../dm-20
lrwxrwxrwx  1 root root       8 Apr  4 11:20 iso_vg-iso_lv -> ../dm-27
lrwxrwxrwx  1 root root       8 Apr  4 11:20 log_vg-log_lv -> ../dm-34
lrwxrwxrwx  1 root root       8 Apr  4 11:20 sap_vg-sap_lv -> ../dm-26
lrwxrwxrwx  1 root root       8 Apr  4 11:20 shared_vg-shared_lv -> ../dm-35
lrwxrwxrwx  1 root root       8 Apr  4 11:20 system_vg-home_lv -> ../dm-28
lrwxrwxrwx  1 root root       8 Apr  4 11:20 system_vg-opt_lv -> ../dm-29
lrwxrwxrwx  1 root root       8 Apr  4 11:20 system_vg-root_lv -> ../dm-16
lrwxrwxrwx  1 root root       8 Apr  4 11:20 system_vg-support_lv -> ../dm-30
lrwxrwxrwx  1 root root       8 Apr  4 11:20 system_vg-swap_lv -> ../dm-31
lrwxrwxrwx  1 root root       8 Apr  4 11:20 system_vg-syslogs_lv -> ../dm-32
lrwxrwxrwx  1 root root       8 Apr  4 11:20 system_vg-tmp_lv -> ../dm-33
lrwxrwxrwx  1 root root       8 Apr  4 11:20 system_vg-usr_lv -> ../dm-19
lrwxrwxrwx  1 root root       8 Apr  4 11:20 system_vg-var_lv -> ../dm-11
lrwxrwxrwx  1 root root       8 Apr  4 11:20 tools_vg-commv_lv -> ../dm-21
lrwxrwxrwx  1 root root       8 Apr  4 11:20 tools_vg-iem_lv -> ../dm-22
lrwxrwxrwx  1 root root       8 Apr  4 11:20 tools_vg-itm_lv -> ../dm-23
lrwxrwxrwx  1 root root       8 Apr  4 11:20 tools_vg-splunk_lv -> ../dm-24
lrwxrwxrwx  1 root root       8 Apr  4 11:20 tools_vg-teamq_lv -> ../dm-25

eniesdbp001p:/dev/mapper # multipath -ll
20017380033e70092 dm-5 IBM,2810XIV
size=64G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 2:0:0:3  sdc  8:32   active ready running
  |- 2:0:1:3  sdf  8:80   active ready running
  |- 3:0:0:3  sdi  8:128  active ready running
  `- 3:0:1:3  sdl  8:176  active ready running
20017380033e70091 dm-1 IBM,2810XIV
size=32G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 2:0:0:2  sdb  8:16   active ready running
  |- 2:0:1:2  sde  8:64   active ready running
  |- 3:0:0:2  sdh  8:112  active ready running
  `- 3:0:1:2  sdk  8:160  active ready running
20017380033e7009b dm-18 IBM,2810XIV
size=240G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 4:0:0:9  sdu  65:64  active ready running
  |- 4:0:1:9  sdaf 65:240 active ready running
  |- 5:0:0:9  sdaq 66:160 active ready running
  `- 5:0:1:9  sdbb 67:80  active ready running
20017380033e70071 dm-0 IBM,2810XIV
size=112G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 2:0:0:1  sda  8:0    active ready running
  |- 2:0:1:1  sdd  8:48   active ready running
  |- 3:0:0:1  sdg  8:96   active ready running
  `- 3:0:1:1  sdj  8:144  active ready running
20017380033e7009a dm-17 IBM,2810XIV
size=240G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 4:0:0:8  sdt  65:48  active ready running
  |- 4:0:1:8  sdae 65:224 active ready running
  |- 5:0:0:8  sdap 66:144 active ready running
  `- 5:0:1:8  sdba 67:64  active ready running
20017380033e70099 dm-15 IBM,2810XIV
size=240G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 4:0:0:7  sds  65:32  active ready running
  |- 4:0:1:7  sdad 65:208 active ready running
  |- 5:0:0:7  sdao 66:128 active ready running
  `- 5:0:1:7  sdaz 67:48  active ready running
20017380033e70098 dm-14 IBM,2810XIV
size=240G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 4:0:0:6  sdr  65:16  active ready running
  |- 4:0:1:6  sdac 65:192 active ready running
  |- 5:0:0:6  sdan 66:112 active ready running
  `- 5:0:1:6  sday 67:32  active ready running
20017380033e70097 dm-13 IBM,2810XIV
size=96G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 4:0:0:5  sdq  65:0   active ready running
  |- 4:0:1:5  sdab 65:176 active ready running
  |- 5:0:0:5  sdam 66:96  active ready running
  `- 5:0:1:5  sdax 67:16  active ready running
20017380033e70096 dm-12 IBM,2810XIV
size=96G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 4:0:0:4  sdp  8:240  active ready running
  |- 4:0:1:4  sdaa 65:160 active ready running
  |- 5:0:0:4  sdal 66:80  active ready running
  `- 5:0:1:4  sdaw 67:0   active ready running
20017380033e700a1 dm-8 IBM,2810XIV
size=385G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 4:0:0:15 sdw  65:96  active ready running
  |- 4:0:1:15 sdah 66:16  active ready running
  |- 5:0:0:15 sdas 66:192 active ready running
  `- 5:0:1:15 sdbd 67:112 active ready running
20017380033e70095 dm-10 IBM,2810XIV
size=96G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 4:0:0:3  sdo  8:224  active ready running
  |- 4:0:1:3  sdz  65:144 active ready running
  |- 5:0:0:3  sdak 66:64  active ready running
  `- 5:0:1:3  sdav 66:240 active ready running
20017380033e700a0 dm-7 IBM,2810XIV
size=385G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 4:0:0:14 sdv  65:80  active ready running
  |- 4:0:1:14 sdag 66:0   active ready running
  |- 5:0:0:14 sdar 66:176 active ready running
  `- 5:0:1:14 sdbc 67:96  active ready running
20017380033e70094 dm-9 IBM,2810XIV
size=96G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 4:0:0:2  sdn  8:208  active ready running
  |- 4:0:1:2  sdy  65:128 active ready running
  |- 5:0:0:2  sdaj 66:48  active ready running
  `- 5:0:1:2  sdau 66:224 active ready running
20017380033e70093 dm-6 IBM,2810XIV
size=64G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 4:0:0:1  sdm  8:192  active ready running
  |- 4:0:1:1  sdx  65:112 active ready running
  |- 5:0:0:1  sdai 66:32  active ready running
  `- 5:0:1:1  sdat 66:208 active ready running

ccjung commented at 2019-04-05 18:13:

We have put this workaround in the /etc/lvm/lvm.conf to exclude the SIBM disk.

filter = [ "r|/dev/.*/by-path/.*|", "r|/dev/.*/by-id/.*|", "r|/dev/fd.*|", "r|/dev/cdrom|", "r|/dev/mapper/SIBM|", "a/.*/" ]

We still sometimes see the unexpected disk mappings using SIBM names.

Should we not see SIBM disk mappings when we restore?

schabrolles commented at 2019-04-07 08:16:

@suseusr168

could you try to run the following command just before running rear recover

/sbin/modprobe -a scsi_dh_alua scsi_dh_emc scsi_dh_rdac dm-multipath

I never had any /dev/mapper/SIBM.... on my test environment. You should avoid recovering on those devices when rear proposes a list of potential device to recover your system (recover on a different system). /dev/mapper/SIBM.... seems to be a single path that was not grouped with other paths that point to the same target device (I don't know why). It could be related to the Storage you are using (XiV) or SAN configuration (dual fabric).

I just want to verify if loading additional scsi modules could help to avoid this situation.
(those modules are loaded by SLES12 when multipath is loaded by systemd... that's why I think we should try it)

dewagner1 commented at 2019-04-08 11:12:

Hi Sebastian,
My colleague is on vacation for 2 weeks and I will be continuing with this
issue.

I ran the command you suggested below. - /sbin/modprobe -a scsi_dh_alua
scsi_dh_emc scsi_dh_rdac dm-multipath

The mapping this time was correct but should we still see the SIBM devices
in the multipath device list?

The following line is in /etc/lvm/lvm.conf. Should this prevent us from
seeing the SIBM devices in the multipath list?
filter = [ "r|/dev/./by-path/.|", "r|/dev/./by-id/.|", "r|/dev/fd.|",
"r|/dev/cdrom|", "r|/dev/mapper/SIBM|", "a/./" ]
Is it true that the device could not be mapped if it does not appear in the
multipath device list?

Console output

RESCUE eniesdbp101p:~ # /sbin/modprobe -a scsi_dh_alua scsi_dh_emc
scsi_dh_rdac dm-multipath
RESCUE eniesdbp101p:~ # rear recover
Relax-and-Recover 2.4-git.3227.0f1f9dbb.master / 2019-02-10
Running rear recover (PID 13528)
Using log file: /var/log/rear/rear-eniesdbp101p.log
Running workflow recover within the ReaR rescue/recovery system
Using backup archive
'/tmp/rear.EXnVw8e6ixVRgqf/outputfs/iso_fs/REAR_BACKUP/backup.tar.gz'
Will do driver migration (recreating initramfs/initrd)
Calculating backup archive size
Backup archive size is 2598.19M (compressed)
Setting up multipathing
Activating multipath
multipath activated
Starting multipath daemon
multipathd started
Listing multipath device found
20017380033e7008a dm-16 IBM,2810XIV size=240G
20017380033e70075 dm-0 ##,## size=64G
20017380033e70089 dm-15 IBM,2810XIV size=240G
SIBM_2810XIV_78033E7008C dm-18 IBM,2810XIV size=240G
SIBM_2810XIV_78033E70084 dm-2 ##,## size=385G
20017380033e70084 dm-13 IBM,2810XIV size=385G
20017380033e70118 dm-11 IBM,2810XIV size=16G
SIBM_2810XIV_78033E70116 dm-1 ##,## size=16G
20017380033e70083 dm-12 IBM,2810XIV size=385G
20017380033e70117 dm-10 IBM,2810XIV size=16G
SIBM_2810XIV_78033E70115 dm-19 IBM,2810XIV size=16G
20017380033e7007a dm-14 IBM,2810XIV size=96G
20017380033e70116 dm-9 IBM,2810XIV size=16G
20017380033e70079 dm-4 IBM,2810XIV size=96G
20017380033e70115 dm-8 IBM,2810XIV size=16G
20017380033e70078 dm-17 IBM,2810XIV size=96G
20017380033e7008c dm-6 IBM,2810XIV size=240G
20017380033e70077 dm-3 IBM,2810XIV size=96G
20017380033e7008b dm-5 IBM,2810XIV size=240G
20017380033e70076 dm-7 IBM,2810XIV size=64G
Comparing disks
Ambiguous possible target disks need manual configuration (more than one
with same size found)
Switching to manual disk layout configuration
Using /dev/mapper/20017380033e70072 (same size) for
recreating /dev/mapper/200173800348802c6
Current disk mapping table (source => target):
/dev/mapper/200173800348802c6 => /dev/mapper/20017380033e70072

Confirm or edit the disk mapping

  1. Confirm disk mapping and continue 'rear recover'
  2. n/a
  3. Edit disk mapping (/var/lib/rear/layout/disk_mappings)
  4. Use Relax-and-Recover shell and return back to here
  5. Abort 'rear recover'

Dan Wagner
Senior Enterprise System Specialist

(717) 580-3844 Mobile
(717) 409-5858 Office
dwagner1@us.ibm.com

IBM Services

From: "Sébastien Chabrolles" notifications@github.com
To: rear/rear rear@noreply.github.com
Cc: dewagner1 dwagner1@us.ibm.com, Mention
mention@noreply.github.com
Date: 04/07/2019 04:16 AM
Subject: Re: [rear/rear] Update to closed issue #2016 (Unable to clone
Power lpar from rear backup SLES 12) (#2019)

@suseusr168

could you try to run the following command just before running rear recover

/sbin/modprobe -a scsi_dh_alua scsi_dh_emc scsi_dh_rdac dm-multipath

I never had any /dev/mapper/SIBM.... on my test environment. You should
avoid recovering on those devices when rear proposes a list of potential
device to recover your system (recover on a different system).
/dev/mapper/SIBM.... seems to be a single path that was not grouped with
other paths that point to the same target device (I don't know why). It
could be related to the Storage you are using (XiV) or SAN configuration
(dual fabric).

I just want to verify if loading additional scsi modules could help to
avoid this situation.
(those modules are loaded by SLES12 when multipath is loaded by systemd...
that's why I think we should try it)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

schabrolles commented at 2019-04-11 19:52:

@dewagner1 @suseusr168

If you still see SIBM devices, this mean that loading the additional scsi modules does not help here.

From my point of view, you should not have any SIBM devices in the multipath list. But you have and I still don’t know why and can’t reproduce it in my lab.

The line in the lvm.conf file help to not detect any lvm device from the SIBM device.

My recommendations is to check the binding proposed by rear and correct it if it proposes any SIBM device as target for restoration.

Tell me if you can restore in this condition and if after reboot on the real disk if you still see any SIBM devices (multipath -ll)

jsmeix commented at 2019-04-12 08:11:

Regarding @schabrolles multiparty -ll
I like to suggest
https://en.wikipedia.org/wiki/(You_Gotta)_Fight_for_Your_Right_(To_Party!)
on this sunny Friday morning - just before the weekend ;-)
(SCNR)

Seriously:
Does multiparty really exists or is it a typo?

schabrolles commented at 2019-04-12 08:22:

:D
@jsmeix, you are right !!! it is multipath -ll
It's because I used my Iphone to answer yesterday evening... the auto-corrector change automatically multipath to multiparty :)

jsmeix commented at 2019-04-12 08:50:

@schabrolles
interesting to see how communication with an iPhone "in between" actually works.

I assume there is no real difference when you use another product
from another one of those well known companies who pretend
to "care for" their users with various "needful services", cf.
https://en.wikipedia.org/wiki/Needful_Things

Enjoy the party on the weekend!

jsmeix commented at 2019-04-12 08:57:

It seems the addressee in
https://github.com/rear/rear/issues/2019#issuecomment-482281167
also got wrong:
It was @dagwieers - I corrected it to @dewagner1 @suseusr168
plus the multipath -ll typo fix.

dewagner1 commented at 2019-04-12 12:38:

Sebastian,
I have included screen captures of my most recent tests.

The SIBM devices aren't always discovered during the rear restore and they
do not exist after a restore.

If the wrong device is mapped, I have been able to edit the file, correct
the mapping and successfully restore.

I have also included another issue which we have seen occasionally. After
the filesystems are created and the restore starts, we get chown failures.

Dan Wagner
Senior Enterprise System Specialist

(717) 580-3844 Mobile
(717) 409-5858 Office
dwagner1@us.ibm.com

IBM Services

From: "Sébastien Chabrolles" notifications@github.com
To: rear/rear rear@noreply.github.com
Cc: dewagner1 dwagner1@us.ibm.com, Mention
mention@noreply.github.com
Date: 04/11/2019 03:56 PM
Subject: Re: [rear/rear] Update to closed issue #2016 (Unable to clone
Power lpar from rear backup SLES 12) (#2019)

@dagwieers

If you still see SIBM devices, this mean that loading the additional scsi
modules does not help here.

From my point of view, you should not have any SIBM devices in the
multipath list. But you have and I still don’t know why and can’t reproduce
it in my lab.

The line in the lvm.conf file help to not detect any lvm device from the
SIBM device.

My recommendations is to check the binding proposed by rear and correct it
if it proposes any SIBM device as target for restoration.

Tell me if you can restore in this condition and if after reboot on the
real disk if you still see any SIBM devices (multiparty -ll)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

dewagner1 commented at 2019-04-12 12:40:

I forgot to include the attachment

(See attached file: Rear Testing 20190412.txt)

Dan Wagner
Senior Enterprise System Specialist

(717) 580-3844 Mobile
(717) 409-5858 Office
dwagner1@us.ibm.com

IBM Services

From: Dan Wagner/Mechanicsburg/Contr/IBM
To: rear/rear <reply
+0283324ac75b125fceb87e1f1759598190aee881cbb18a3392cebabccd7e92a169ce17dedc38@reply.github.com>
Cc: Mention mention@noreply.github.com, rear/rear
rear@noreply.github.com
Date: 04/12/2019 08:38 AM
Subject: Re: [rear/rear] Update to closed issue #2016 (Unable to clone
Power lpar from rear backup SLES 12) (#2019)

Sebastian,
I have included screen captures of my most recent tests.

The SIBM devices aren't always discovered during the rear restore and they
do not exist after a restore.

If the wrong device is mapped, I have been able to edit the file, correct
the mapping and successfully restore.

I have also included another issue which we have seen occasionally. After
the filesystems are created and the restore starts, we get chown failures.

Dan Wagner
Senior Enterprise System Specialist

(717) 580-3844 Mobile
(717) 409-5858 Office
dwagner1@us.ibm.com

IBM Services

From: "Sébastien Chabrolles" notifications@github.com
To: rear/rear rear@noreply.github.com
Cc: dewagner1 dwagner1@us.ibm.com, Mention
mention@noreply.github.com
Date: 04/11/2019 03:56 PM
Subject: Re: [rear/rear] Update to closed issue #2016 (Unable to clone
Power lpar from rear backup SLES 12) (#2019)

@dagwieers

If you still see SIBM devices, this mean that loading the additional scsi
modules does not help here.

From my point of view, you should not have any SIBM devices in the
multipath list. But you have and I still don’t know why and can’t reproduce
it in my lab.

The line in the lvm.conf file help to not detect any lvm device from the
SIBM device.

My recommendations is to check the binding proposed by rear and correct it
if it proposes any SIBM device as target for restoration.

Tell me if you can restore in this condition and if after reboot on the
real disk if you still see any SIBM devices (multiparty -ll)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Search for "#@"

You will find
#@ First Activation
This activation was used to wipe the disk with dd
SIBM multipath devices were discovered
Rear mapped the correct disk
#@ Second Activation
SIBM multiapth devices were NOT discovered
Rear mapped the correct disk
This activation did not work due to chown failures
Wiped disk with dd and tried again
#@ Third Activation
SIBM multipath devices were discovered
Rear mapped the correct disk
This activation completed successfully
#@ After final reboot
After the final reboot there is no indication of SIBM devices

Rear mapped the correct disk in all activations above.
There are still cases where rear will select /dev/sda or SIBM devices but
it seems to have become less frequent

#@ First Activation

SIBM multipath devices discovered
Rear selected correct disk
wipe the disk with dd

                         GNU GRUB  version 2.02

+----------------------------------------------------------------------------+
|*Relax-and-Recover |
| |
| |
| |
| |
+----------------------------------------------------------------------------+

  Use the ^ and v keys to select which entry is highlighted.
  Press enter to boot the selected OS, `e' to edit the commands
  before booting or `c' for a command-line.

OF stdout device is: /vdevice/vty@30000000
Preparing to boot Linux version 4.12.14-95.6-default (geeko@buildhost) (gcc version 4.8.5 (SUSE Linux) ) #1 SMP Thu Jan 17 06:04:39 UTC 2019 (6af4ef8)
Detected machine type: 0000000000000101
command line: BOOT_IMAGE=/kernel root=/dev/ram0 selinux=0
Max number of cores passed to firmware: 256 (NR_CPUS = 2048)
Calling ibm,client-architecture-support...-
Elapsed time since release of system processors: 127901 mins 6 secs
Welcome to GRUB!

OF stdout device is: /vdevice/vty@30000000 Preparing to boot Linux version 4.12.14-95.6-default (geeko@buildhost) (gcc version 4.8.5 (SUSE Linux) ) #1 SMP Thu Jan 17 06:04:39 UTC 2019 (6af4ef8) Detected machine type: 0000000000000101 command line: BOOT_IMAGE=/kernel root=/dev/ram0 selinux=0 Max number of cores passed to firmware: 256 (NR_CPUS = 2048) Calling ibm,client-architecture-support... done memory layout at init: memory_limit : 0000000000000000 (16 MB aligned) alloc_bottom : 000000000f4d0000 alloc_top : 0000000020000000 alloc_top_hi : 0000000020000000 rmo_top : 0000000020000000 ram_top : 0000000020000000 instantiating rtas at 0x000000001eca0000... done prom_hold_cpus: skipped copying OF device tree... Building dt strings... Building dt structure... Device tree strings 0x000000000f4e0000 -> 0x000000000f4e1751 Device tree struct 0x000000000f4f0000 -> 0x000000000f520000 Quiescing Open Firmware ... Booting Linux via __start() @ 0x000000000a6f0000 ... Page sizes from device-tree: base_shift=12: shift=12, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=0 base_shift=12: shift=16, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=7 base_shift=12: shift=24, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=56 base_shift=16: shift=16, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=1 base_shift=16: shift=24, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=8 base_shift=24: shift=24, sllp=0x0100, avpnm=0x00000001, tlbiel=0, penc=0 base_shift=34: shift=34, sllp=0x0120, avpnm=0x000007ff, tlbiel=0, penc=3 Using 1TB segments Initializing hash mmu with SLB Linux version 4.12.14-95.6-default (geeko@buildhost) (gcc version 4.8.5 (SUSE Linux) ) #1 SMP Thu Jan 17 06:04:39 UTC 2019 (6af4ef8) Found initrd at 0xc00000000c600000:0xc00000000f4cfeda Using pSeries machine description bootconsole [udbg0] enabled Partition configured for 96 cpus. CPU maps initialized for 8 threads per core -> smp_release_cpus() spinning_secondaries = 31 <- smp_release_cpus()

ppc64_pft_size = 0x1e phys_mem_size = 0x2000000000 dcache_bsize = 0x80 icache_bsize = 0x80 cpu_features = 0x000000ff8f5d91a7 possible = 0x00007fffcf5fb1a7 always = 0x0000006f835c91a1 cpu_user_features = 0xdc0065c2 0xef000000 mmu_features = 0x7c006001 firmware_features = 0x00000017c45bfc57 htab_hash_mask = 0x7fffff

numa: NODE_DATA [mem 0x1fffcd6300-0x1fffcdffff]
numa: NODE_DATA(0) on node 5
numa: NODE_DATA [mem 0x1fffccc600-0x1fffcd62ff]
Section 8189 and 8191 (node 5) have a circular dependency on usemap and pgdat allocations
rfi-flush: fallback displacement flush available
rfi-flush: ori type flush available
barrier-nospec: using ORI speculation barrier
stf-barrier: hwsync barrier available
PPC64 nvram contains 15360 bytes
Zone ranges:
DMA [mem 0x0000000000000000-0x0000001fffffffff]
DMA32 empty
Normal empty
Movable zone start for each node
Early memory node ranges
node 5: [mem 0x0000000000000000-0x0000001fffffffff]
Could not find start_pfn for node 0
Initmem setup node 0 [mem 0x0000000000000000-0x0000000000000000]
Initmem setup node 5 [mem 0x0000000000000000-0x0000001fffffffff]
percpu: Embedded 3 pages/cpu @c000001ffe400000 s155160 r0 d41448 u262144
Built 2 zonelists in Node order, mobility grouping on. Total pages: 2095104
Policy zone: DMA
Kernel command line: BOOT_IMAGE=/kernel root=/dev/ram0 selinux=0
log_buf_len individual max cpu contribution: 4096 bytes
log_buf_len total cpu_extra contributions: 389120 bytes
log_buf_len min size: 524288 bytes
log_buf_len: 1048576 bytes
early log buf free: 519164(99%)
PID hash table entries: 4096 (order: -1, 32768 bytes)
Memory: 2108096K/134217728K available (9600K kernel code, 1792K rwdata, 2788K rodata, 3840K init, 11735K bss, 258048K reserved, 0K cma-reserved)
ftrace: allocating 26365 entries in 10 pages
Hierarchical RCU implementation.
RCU event tracing is enabled.
RCU restricting CPUs from NR_CPUS=2048 to nr_cpu_ids=96.
RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=96
NR_IRQS:512 nr_irqs:512 16
clocksource: timebase: mask: 0xffffffffffffffff max_cycles: 0x761537d007, max_idle_ns: 440795202126 ns
clocksource: timebase mult[1f40000] shift[24] registered
Console: colour dummy device 80x25
console [hvc0] enabled
console [hvc0] enabled
bootconsole [udbg0] disabled
bootconsole [udbg0] disabled
mempolicy: Disabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl
pid_max: default: 98304 minimum: 768
Security Framework initialized
AppArmor: AppArmor initialized
Dentry cache hash table entries: 8388608 (order: 10, 67108864 bytes)
Inode-cache hash table entries: 4194304 (order: 9, 33554432 bytes)
Mount-cache hash table entries: 131072 (order: 4, 1048576 bytes)
Mountpoint-cache hash table entries: 131072 (order: 4, 1048576 bytes)
EEH: pSeries platform initialized
POWER8 performance monitor hardware support registered
smp: Bringing up secondary CPUs ...
smp: Brought up 2 nodes, 32 CPUs
numa: Node 0 CPUs:
numa: Node 5 CPUs: 0-31
node 5 initialised, 2060181 pages in 40ms
devtmpfs: initialized
evm: security.selinux
evm: security.ima
evm: security.capability
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
futex hash table entries: 32768 (order: 6, 4194304 bytes)
NET: Registered protocol family 16
EEH: No capable adapters found
cpuidle: using governor ladder
cpuidle: using governor menu
pstore: using zlib compression
pstore: Registered nvram as persistent store backend
PCI: Probing PCI hardware
pseries-rng: Registering arch random hook.
HugeTLB registered 16 MB page size, pre-allocated 0 pages
HugeTLB registered 16 GB page size, pre-allocated 0 pages
vgaarb: loaded
EDAC MC: Ver: 3.0.0
NetLabel: Initializing
NetLabel: domain hash size = 128
NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO
NetLabel: unlabeled traffic allowed by default
clocksource: Switched to clocksource timebase
VFS: Disk quotas dquot_6.6.0
VFS: Dquot-cache hash table entries: 8192 (order 0, 65536 bytes)
AppArmor: AppArmor Filesystem Enabled
NET: Registered protocol family 2
TCP established hash table entries: 524288 (order: 6, 4194304 bytes)
TCP bind hash table entries: 65536 (order: 4, 1048576 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
UDP hash table entries: 65536 (order: 5, 2097152 bytes)
UDP-Lite hash table entries: 65536 (order: 5, 2097152 bytes)
NET: Registered protocol family 1
Unpacking initramfs...
random: fast init done
Freeing initrd memory: 47872K
numa: Starting topology update prrn_enabled
IOMMU table initialized, virtual merging enabled
hv-24x7: read 1330 catalog entries, created 337 event attrs (0 failures), 169 descs
audit: initializing netlink subsys (disabled)
audit: type=2000 audit(1555067111.290:1): state=initialized audit_enabled=0 res=1
Initialise system trusted keyrings
workingset: timestamp_bits=38 max_order=21 bucket_order=0
zbud: loaded
Key type asymmetric registered
Asymmetric key parser 'x509' registered
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249)
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
io scheduler mq-deadline registered
io scheduler kyber registered
io scheduler bfq registered
Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
pseries_rng: Registering IBM pSeries RNG driver
mousedev: PS/2 mouse device common for all mice
ledtrig-cpu: registered to indicate activity on CPUs
nx_compress_pseries ibm,compression-v1: nx842_OF_upd: device disabled
hidraw: raw HID events driver (C) Jiri Kosina
NET: Registered protocol family 10
Segment Routing with IPv6
NET: Registered protocol family 15
registered taskstats version 1
Loading compiled-in X.509 certificates
Loaded X.509 cert 'SUSE Linux Enterprise Secure Boot Signkey: 3fb077b6cebc6ff2522e1c148c57c777c788e3e7'
zswap: loaded using pool lzo/zbud
page_owner is disabled
Key type trusted registered
Key type encrypted registered
AppArmor: AppArmor sha1 policy hashing enabled
ima: No TPM chip found, activating TPM-bypass! (rc=-19)
ima: Allocated hash algorithm: sha256
evm: HMAC attrs: 0x1
Freeing unused kernel memory: 3840K
This architecture does not have kernel memory protection.
alg: No test for pkcs1pad(rsa,sha256) (pkcs1pad(rsa-generic,sha256))
random: systemd: uninitialized urandom read (16 bytes read)
random: systemd: uninitialized urandom read (16 bytes read)
random: systemd: uninitialized urandom read (16 bytes read)

Welcome to SUSE Linux Enterprise Server 12 SP4!

[ OK ] Reached target System Initialization.
[ OK ] Listening on Journal Socket.
[ OK ] Listening on udev Kernel Socket.
[ OK ] Created slice system.slice.
[ OK ] Created slice system-serial\x2dgetty.slice.
[ OK ] Listening on udev Control Socket.
Starting udev Kernel Device Manager...
[ OK ] Started Journal Service.
[ OK ] Created slice system-getty.slice.
Starting Relax-and-Recover boot script...
[ OK ] Started Relax-and-Recover sshd service.
Starting udev Coldplug all Devices...
[ OK ] Started udev Kernel Device Manager.
Starting Initialize Rescue System...
[ OK ] Started Relax-and-Recover boot script.
[ OK ] Found device /dev/ttyS0.
[ OK ] Started udev Coldplug all Devices.
[ OK ] Found device /dev/hvc0.

Verifying md5sums of the files in the Relax-and-Recover rescue system

md5sums are OK

Configuring Relax-and-Recover rescue system

Running 00-functions.sh...
Running 01-run-ldconfig.sh...
Running 10-console-setup.sh...
Using keymap of the original system
Running 20-check-boot-options.sh...
Running 40-start-udev-or-load-modules.sh...
Waiting for udev ... done.
Running 41-load-special-modules.sh...
Running 42-engage-scsi.sh...
Running 45-serial-console.sh...
Running 55-migrate-network-devices.sh...
No network interface mapping is specified in /etc/rear/mappings/mac
The original network interface eth0 d6:8b:e5:4d:db:08 is not available

  1. eth3 c6:c5:cb:8e:5c:04 ibmveth
  2. eth4 c6:c5:cb:8e:5c:03 ibmveth
  3. eth5 c6:c5:cb:8e:5c:05 ibmveth
  4. Skip replacing eth0 d6:8b:e5:4d:db:08
    Choose replacement for eth0 d6:8b:e5:4d:db:08 2
    Mapping eth0 d6:8b:e5:4d:db:08 to eth4 c6:c5:cb:8e:5c:03
    The original network interface eth1 d6:8b:e5:4d:db:09 is not available
  5. eth3 c6:c5:cb:8e:5c:04 ibmveth
    1. eth5 c6:c5:cb:8e:5c:05 ibmveth
  7. Skip replacing eth1 d6:8b:e5:4d:db:09
    Choose replacement for eth1 d6:8b:e5:4d:db:09 1
    Mapping eth1 d6:8b:e5:4d:db:09 to eth3 c6:c5:cb:8e:5c:04
    The original network interface eth2 d6:8b:e5:4d:db:0a is not available
      1. eth5 c6:c5:cb:8e:5c:05 ibmveth
  9. Skip replacing eth2 d6:8b:e5:4d:db:0a
    Choose replacement for eth2 d6:8b:e5:4d:db:0a 3
    Mapping eth2 d6:8b:e5:4d:db:0a to eth5 c6:c5:cb:8e:5c:05
    Reloading udev ... done.
    Running 58-start-dhclient.sh...
    Running 60-network-devices.sh...
    Running 62-routing.sh...
    Running 65-sysctl.sh...
    net.ipv6.conf.all.disable_ipv6 = 1
    net.ipv4.ip_forward = 0
    net.ipv6.conf.all.forwarding = 0
    vm.pagecache_limit_ignore_dirty = 2
    vm.min_free_kbytes = 10485760
    net.core.somaxconn = 4096
    net.ipv4.tcp_max_syn_backlog = 8192
    net.ipv4.tcp_tw_reuse = 1
    net.ipv4.tcp_timestamps = 1
    net.ipv4.tcp_slow_start_after_idle = 0
    net.ipv4.tcp_window_scaling = 1
    net.ipv4.tcp_syn_retries = 8
    net.core.rmem_max = 56623104
    net.core.wmem_max = 56623104
    net.ipv4.tcp_rmem = 65536 262088 56623104
    net.ipv4.tcp_wmem = 65536 262088 56623104
    net.ipv4.tcp_mem = 56623104 56623104 56623104
    fs.aio-max-nr = 18446744073709551615
    fs.file-max = 20000000
    vm.memory_failure_early_kill = 1
    kernel.shmmni = 65536
    net.ipv6.conf.all.disable_ipv6 = 1
    net.ipv4.ip_forward = 0
    net.ipv6.conf.all.forwarding = 0
    vm.pagecache_limit_ignore_dirty = 2
    vm.min_free_kbytes = 10485760
    net.core.somaxconn = 4096
    net.ipv4.tcp_max_syn_backlog = 8192
    net.ipv4.tcp_tw_reuse = 1
    net.ipv4.tcp_timestamps = 1
    net.ipv4.tcp_slow_start_after_idle = 0
    net.ipv4.tcp_window_scaling = 1
    net.ipv4.tcp_syn_retries = 8
    net.core.rmem_max = 56623104
    net.core.wmem_max = 56623104
    net.ipv4.tcp_rmem = 65536 262088 56623104
    net.ipv4.tcp_wmem = 65536 262088 56623104
    net.ipv4.tcp_mem = 56623104 56623104 56623104
    fs.aio-max-nr = 18446744073709551615
    fs.file-max = 20000000
    vm.memory_failure_early_kill = 1
    kernel.shmmni = 65536
    Running 99-makedev.sh...

Relax-and-Recover rescue system is ready

[ OK ] Started Initialize Rescue System.
[ OK ] Started Getty on tty0.
[ OK ] Reached target Login Prompts.
[ OK ] Started Serial Getty on ttyS0.
[ OK ] Started Serial Getty on hvc0.
[ OK ] Reached target Multi-User.

Relax-and-Recover 2.4-git.3227.0f1f9dbb.master / 2019-02-10

Relax-and-Recover comes with ABSOLUTELY NO WARRANTY; for details see
the GNU General Public License at: http://www.gnu.org/licenses/gpl.html

Host eniesdbp101p using Backup NETFS and Output ISO
Build date: Tue, 09 Apr 2019 14:15:01 -0400

Welcome to SUSE Linux Enterprise Server for SAP Applications 12 SP4 (ppc64le) - Kernel 4.12.14-95.6-default (hvc0).

SSH fingerprint: 2048 SHA256:DnwYyS1WBORjJp5MuiuGa2YSoJkCo4VvOnDK7p3dRJQ root@eniesdbp101p (RSA)

eniesdbp101p login: root

Welcome to Relax-and-Recover. Run "rear recover" to restore your system !

RESCUE eniesdbp101p:~ # rear recover
Relax-and-Recover 2.4-git.3227.0f1f9dbb.master / 2019-02-10
Running rear recover (PID 13469)
Using log file: /var/log/rear/rear-eniesdbp101p.log
Running workflow recover within the ReaR rescue/recovery system
Using backup archive '/tmp/rear.Fn3f91MQf71uuBA/outputfs/iso_fs/REAR_BACKUP/backup.tar.gz'
Will do driver migration (recreating initramfs/initrd)
Calculating backup archive size
Backup archive size is 2614.94M (compressed)
Setting up multipathing
Activating multipath
multipath activated
Starting multipath daemon
multipathd started
Listing multipath device found
20017380033e7008a dm-12 IBM,2810XIV size=240G
20017380033e70075 dm-3 ##,## size=64G
20017380033e70089 dm-11 IBM,2810XIV size=240G
20017380033e70074 dm-2 ##,## size=32G
20017380033e70073 dm-1 IBM,2810XIV size=16G
20017380033e70072 dm-0 ##,## size=112G
SIBM_2810XIV_78033E70083 dm-20 IBM,2810XIV size=385G
20017380033e70084 dm-6 IBM,2810XIV size=385G
20017380033e70118 dm-4 IBM,2810XIV size=16G
SIBM_2810XIV_78033E70116 dm-22 IBM,2810XIV size=16G
20017380033e70117 dm-17 IBM,2810XIV size=16G
20017380033e70083 dm-5 IBM,2810XIV size=385G
SIBM_2810XIV_78033E70115 dm-18 IBM,2810XIV size=16G
20017380033e7007a dm-10 IBM,2810XIV size=96G
20017380033e70116 dm-19 IBM,2810XIV size=16G
20017380033e70079 dm-9 IBM,2810XIV size=96G
20017380033e70115 dm-16 IBM,2810XIV size=16G
20017380033e70078 dm-8 IBM,2810XIV size=96G
20017380033e7008c dm-14 IBM,2810XIV size=240G
20017380033e70077 dm-7 IBM,2810XIV size=96G
20017380033e7008b dm-13 IBM,2810XIV size=240G
SIBM_2810XIV_78033E7007A dm-21 IBM,2810XIV size=96G
20017380033e70076 dm-15 IBM,2810XIV size=64G
Comparing disks
Ambiguous possible target disks need manual configuration (more than one with same size found)
Switching to manual disk layout configuration
Using /dev/mapper/20017380033e70072 (same size) for recreating /dev/mapper/200173800348802c6
Current disk mapping table (source => target):
/dev/mapper/200173800348802c6 => /dev/mapper/20017380033e70072

Confirm or edit the disk mapping

  1. Confirm disk mapping and continue 'rear recover'
  2. n/a
  3. Edit disk mapping (/var/lib/rear/layout/disk_mappings)
  4. Use Relax-and-Recover shell and return back to here
  5. Abort 'rear recover'
    (default '1' timeout 300 seconds)
    4

Welcome to Relax-and-Recover.

rear> dd if=/dev/zero of=/dev/mapper/20017380033e70072 bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 1.53403 s, 684 MB/s
rear>

#@ Second Activation

No SIBM multipath devices discovered
Rear selected correct disk
Recover didn't work - chown failures

                         GNU GRUB  version 2.02

+----------------------------------------------------------------------------+
|*Relax-and-Recover |
| |
| |
| |
| |
+----------------------------------------------------------------------------+

  Use the ^ and v keys to select which entry is highlighted.
  Press enter to boot the selected OS, `e' to edit the commands
  before booting or `c' for a command-line.

OF stdout device is: /vdevice/vty@30000000 Preparing to boot Linux version 4.12.14-95.6-default (geeko@buildhost) (gcc version 4.8.5 (SUSE Linux) ) #1 SMP Thu Jan 17 06:04:39 UTC 2019 (6af4ef8) Detected machine type: 0000000000000101 command line: BOOT_IMAGE=/kernel root=/dev/ram0 selinux=0 Max number of cores passed to firmware: 256 (NR_CPUS = 2048) Calling ibm,client-architecture-support... done memory layout at init: memory_limit : 0000000000000000 (16 MB aligned) alloc_bottom : 000000000f4d0000 alloc_top : 0000000020000000 alloc_top_hi : 0000000020000000 rmo_top : 0000000020000000 ram_top : 0000000020000000 instantiating rtas at 0x000000001eca0000... done prom_hold_cpus: skipped copying OF device tree... Building dt strings... Building dt structure... Device tree strings 0x000000000f4e0000 -> 0x000000000f4e1751 Device tree struct 0x000000000f4f0000 -> 0x000000000f520000 Quiescing Open Firmware ... Booting Linux via __start() @ 0x000000000a6f0000 ... Page sizes from device-tree: base_shift=12: shift=12, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=0 base_shift=12: shift=16, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=7 base_shift=12: shift=24, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=56 base_shift=16: shift=16, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=1 base_shift=16: shift=24, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=8 base_shift=24: shift=24, sllp=0x0100, avpnm=0x00000001, tlbiel=0, penc=0 base_shift=34: shift=34, sllp=0x0120, avpnm=0x000007ff, tlbiel=0, penc=3 Using 1TB segments Initializing hash mmu with SLB Linux version 4.12.14-95.6-default (geeko@buildhost) (gcc version 4.8.5 (SUSE Linux) ) #1 SMP Thu Jan 17 06:04:39 UTC 2019 (6af4ef8) Found initrd at 0xc00000000c600000:0xc00000000f4cfeda Using pSeries machine description bootconsole [udbg0] enabled Partition configured for 96 cpus. CPU maps initialized for 8 threads per core -> smp_release_cpus() spinning_secondaries = 31 <- smp_release_cpus()

ppc64_pft_size = 0x1e phys_mem_size = 0x2000000000 dcache_bsize = 0x80 icache_bsize = 0x80 cpu_features = 0x000000ff8f5d91a7 possible = 0x00007fffcf5fb1a7 always = 0x0000006f835c91a1 cpu_user_features = 0xdc0065c2 0xef000000 mmu_features = 0x7c006001 firmware_features = 0x00000017c45bfc57 htab_hash_mask = 0x7fffff

numa: NODE_DATA [mem 0x1fffcd6300-0x1fffcdffff]
numa: NODE_DATA(0) on node 5
numa: NODE_DATA [mem 0x1fffccc600-0x1fffcd62ff]
Section 8189 and 8191 (node 5) have a circular dependency on usemap and pgdat allocations
rfi-flush: fallback displacement flush available
rfi-flush: ori type flush available
barrier-nospec: using ORI speculation barrier
stf-barrier: hwsync barrier available
PPC64 nvram contains 15360 bytes
Zone ranges:
DMA [mem 0x0000000000000000-0x0000001fffffffff]
DMA32 empty
Normal empty
Movable zone start for each node
Early memory node ranges
node 5: [mem 0x0000000000000000-0x0000001fffffffff]
Could not find start_pfn for node 0
Initmem setup node 0 [mem 0x0000000000000000-0x0000000000000000]
Initmem setup node 5 [mem 0x0000000000000000-0x0000001fffffffff]
percpu: Embedded 3 pages/cpu @c000001ffe400000 s155160 r0 d41448 u262144
Built 2 zonelists in Node order, mobility grouping on. Total pages: 2095104
Policy zone: DMA
Kernel command line: BOOT_IMAGE=/kernel root=/dev/ram0 selinux=0
log_buf_len individual max cpu contribution: 4096 bytes
log_buf_len total cpu_extra contributions: 389120 bytes
log_buf_len min size: 524288 bytes
log_buf_len: 1048576 bytes
early log buf free: 519164(99%)
PID hash table entries: 4096 (order: -1, 32768 bytes)
Memory: 2108096K/134217728K available (9600K kernel code, 1792K rwdata, 2788K rodata, 3840K init, 11735K bss, 258048K reserved, 0K cma-reserved)
ftrace: allocating 26365 entries in 10 pages
Hierarchical RCU implementation.
RCU event tracing is enabled.
RCU restricting CPUs from NR_CPUS=2048 to nr_cpu_ids=96.
RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=96
NR_IRQS:512 nr_irqs:512 16
clocksource: timebase: mask: 0xffffffffffffffff max_cycles: 0x761537d007, max_idle_ns: 440795202126 ns
clocksource: timebase mult[1f40000] shift[24] registered
Console: colour dummy device 80x25
console [hvc0] enabled
console [hvc0] enabled
bootconsole [udbg0] disabled
bootconsole [udbg0] disabled
mempolicy: Disabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl
pid_max: default: 98304 minimum: 768
Security Framework initialized
AppArmor: AppArmor initialized
Dentry cache hash table entries: 8388608 (order: 10, 67108864 bytes)
Inode-cache hash table entries: 4194304 (order: 9, 33554432 bytes)
Mount-cache hash table entries: 131072 (order: 4, 1048576 bytes)
Mountpoint-cache hash table entries: 131072 (order: 4, 1048576 bytes)
EEH: pSeries platform initialized
POWER8 performance monitor hardware support registered
smp: Bringing up secondary CPUs ...
smp: Brought up 2 nodes, 32 CPUs
numa: Node 0 CPUs:
numa: Node 5 CPUs: 0-31
node 5 initialised, 2060181 pages in 40ms
devtmpfs: initialized
evm: security.selinux
evm: security.ima
evm: security.capability
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
futex hash table entries: 32768 (order: 6, 4194304 bytes)
NET: Registered protocol family 16
EEH: No capable adapters found
cpuidle: using governor ladder
cpuidle: using governor menu
pstore: using zlib compression
pstore: Registered nvram as persistent store backend
PCI: Probing PCI hardware
pseries-rng: Registering arch random hook.
HugeTLB registered 16 MB page size, pre-allocated 0 pages
HugeTLB registered 16 GB page size, pre-allocated 0 pages
vgaarb: loaded
EDAC MC: Ver: 3.0.0
NetLabel: Initializing
NetLabel: domain hash size = 128
NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO
NetLabel: unlabeled traffic allowed by default
clocksource: Switched to clocksource timebase
VFS: Disk quotas dquot_6.6.0
VFS: Dquot-cache hash table entries: 8192 (order 0, 65536 bytes)
AppArmor: AppArmor Filesystem Enabled
NET: Registered protocol family 2
TCP established hash table entries: 524288 (order: 6, 4194304 bytes)
TCP bind hash table entries: 65536 (order: 4, 1048576 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
UDP hash table entries: 65536 (order: 5, 2097152 bytes)
UDP-Lite hash table entries: 65536 (order: 5, 2097152 bytes)
NET: Registered protocol family 1
Unpacking initramfs...
random: fast init done
Freeing initrd memory: 47872K
numa: Starting topology update prrn_enabled
IOMMU table initialized, virtual merging enabled
hv-24x7: read 1330 catalog entries, created 337 event attrs (0 failures), 169 descs
audit: initializing netlink subsys (disabled)
audit: type=2000 audit(1555067286.280:1): state=initialized audit_enabled=0 res=1
Initialise system trusted keyrings
workingset: timestamp_bits=38 max_order=21 bucket_order=0
zbud: loaded
Key type asymmetric registered
Asymmetric key parser 'x509' registered
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249)
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
io scheduler mq-deadline registered
io scheduler kyber registered
io scheduler bfq registered
Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
pseries_rng: Registering IBM pSeries RNG driver
mousedev: PS/2 mouse device common for all mice
ledtrig-cpu: registered to indicate activity on CPUs
nx_compress_pseries ibm,compression-v1: nx842_OF_upd: device disabled
hidraw: raw HID events driver (C) Jiri Kosina
NET: Registered protocol family 10
Segment Routing with IPv6
NET: Registered protocol family 15
registered taskstats version 1
Loading compiled-in X.509 certificates
Loaded X.509 cert 'SUSE Linux Enterprise Secure Boot Signkey: 3fb077b6cebc6ff2522e1c148c57c777c788e3e7'
zswap: loaded using pool lzo/zbud
page_owner is disabled
Key type trusted registered
Key type encrypted registered
AppArmor: AppArmor sha1 policy hashing enabled
ima: No TPM chip found, activating TPM-bypass! (rc=-19)
ima: Allocated hash algorithm: sha256
evm: HMAC attrs: 0x1
Freeing unused kernel memory: 3840K
This architecture does not have kernel memory protection.
alg: No test for pkcs1pad(rsa,sha256) (pkcs1pad(rsa-generic,sha256))
random: systemd: uninitialized urandom read (16 bytes read)
random: systemd: uninitialized urandom read (16 bytes read)
random: systemd: uninitialized urandom read (16 bytes read)

Welcome to SUSE Linux Enterprise Server 12 SP4!

[ OK ] Listening on udev Control Socket.
[ OK ] Listening on Journal Socket.
[ OK ] Reached target System Initialization.
[ OK ] Listening on udev Kernel Socket.
[ OK ] Created slice system.slice.
Starting Relax-and-Recover boot script...
[ OK ] Started Relax-and-Recover sshd service.
Starting udev Coldplug all Devices...
[ OK ] Started Journal Service.
Starting udev Kernel Device Manager...
[ OK ] Created slice system-getty.slice.
[ OK ] Created slice system-serial\x2dgetty.slice.
[ OK ] Started Relax-and-Recover boot script.
[ OK ] Started udev Kernel Device Manager.
Starting Initialize Rescue System...
[ OK ] Found device /dev/ttyS0.
[ OK ] Started udev Coldplug all Devices.
[ OK ] Found device /dev/hvc0.

Verifying md5sums of the files in the Relax-and-Recover rescue system

md5sums are OK

Configuring Relax-and-Recover rescue system

Running 00-functions.sh...
Running 01-run-ldconfig.sh...
Running 10-console-setup.sh...
Using keymap of the original system
Running 20-check-boot-options.sh...
Running 40-start-udev-or-load-modules.sh...
Waiting for udev ... done.
Running 41-load-special-modules.sh...
Running 42-engage-scsi.sh...
Running 45-serial-console.sh...
Running 55-migrate-network-devices.sh...
No network interface mapping is specified in /etc/rear/mappings/mac
The original network interface eth0 d6:8b:e5:4d:db:08 is not available

  1. eth3 c6:c5:cb:8e:5c:04 ibmveth
  2. eth4 c6:c5:cb:8e:5c:03 ibmveth
  3. eth5 c6:c5:cb:8e:5c:05 ibmveth
  4. Skip replacing eth0 d6:8b:e5:4d:db:08
    Choose replacement for eth0 d6:8b:e5:4d:db:08 2
    Mapping eth0 d6:8b:e5:4d:db:08 to eth4 c6:c5:cb:8e:5c:03
    The original network interface eth1 d6:8b:e5:4d:db:09 is not available
  5. eth3 c6:c5:cb:8e:5c:04 ibmveth
    1. eth5 c6:c5:cb:8e:5c:05 ibmveth
  7. Skip replacing eth1 d6:8b:e5:4d:db:09
    Choose replacement for eth1 d6:8b:e5:4d:db:09 1
    Mapping eth1 d6:8b:e5:4d:db:09 to eth3 c6:c5:cb:8e:5c:04
    The original network interface eth2 d6:8b:e5:4d:db:0a is not available
      1. eth5 c6:c5:cb:8e:5c:05 ibmveth
  9. Skip replacing eth2 d6:8b:e5:4d:db:0a
    Choose replacement for eth2 d6:8b:e5:4d:db:0a 3
    Mapping eth2 d6:8b:e5:4d:db:0a to eth5 c6:c5:cb:8e:5c:05
    Reloading udev ... done.
    Running 58-start-dhclient.sh...
    Running 60-network-devices.sh...
    Running 62-routing.sh...
    Running 65-sysctl.sh...
    net.ipv6.conf.all.disable_ipv6 = 1
    net.ipv4.ip_forward = 0
    net.ipv6.conf.all.forwarding = 0
    vm.pagecache_limit_ignore_dirty = 2
    vm.min_free_kbytes = 10485760
    net.core.somaxconn = 4096
    net.ipv4.tcp_max_syn_backlog = 8192
    net.ipv4.tcp_tw_reuse = 1
    net.ipv4.tcp_timestamps = 1
    net.ipv4.tcp_slow_start_after_idle = 0
    net.ipv4.tcp_window_scaling = 1
    net.ipv4.tcp_syn_retries = 8
    net.core.rmem_max = 56623104
    net.core.wmem_max = 56623104
    net.ipv4.tcp_rmem = 65536 262088 56623104
    net.ipv4.tcp_wmem = 65536 262088 56623104
    net.ipv4.tcp_mem = 56623104 56623104 56623104
    fs.aio-max-nr = 18446744073709551615
    fs.file-max = 20000000
    vm.memory_failure_early_kill = 1
    kernel.shmmni = 65536
    net.ipv6.conf.all.disable_ipv6 = 1
    net.ipv4.ip_forward = 0
    net.ipv6.conf.all.forwarding = 0
    vm.pagecache_limit_ignore_dirty = 2
    vm.min_free_kbytes = 10485760
    net.core.somaxconn = 4096
    net.ipv4.tcp_max_syn_backlog = 8192
    net.ipv4.tcp_tw_reuse = 1
    net.ipv4.tcp_timestamps = 1
    net.ipv4.tcp_slow_start_after_idle = 0
    net.ipv4.tcp_window_scaling = 1
    net.ipv4.tcp_syn_retries = 8
    net.core.rmem_max = 56623104
    net.core.wmem_max = 56623104
    net.ipv4.tcp_rmem = 65536 262088 56623104
    net.ipv4.tcp_wmem = 65536 262088 56623104
    net.ipv4.tcp_mem = 56623104 56623104 56623104
    fs.aio-max-nr = 18446744073709551615
    fs.file-max = 20000000
    vm.memory_failure_early_kill = 1
    kernel.shmmni = 65536
    Running 99-makedev.sh...

Relax-and-Recover rescue system is ready

[ OK ] Started Initialize Rescue System.
[ OK ] Started Getty on tty0.
[ OK ] Reached target Login Prompts.
[ OK ] Started Serial Getty on hvc0.
[ OK ] Reached target Multi-User.
[ OK ] Started Serial Getty on ttyS0.

Relax-and-Recover 2.4-git.3227.0f1f9dbb.master / 2019-02-10

Relax-and-Recover comes with ABSOLUTELY NO WARRANTY; for details see
the GNU General Public License at: http://www.gnu.org/licenses/gpl.html

Host eniesdbp101p using Backup NETFS and Output ISO
Build date: Tue, 09 Apr 2019 14:15:01 -0400

Welcome to SUSE Linux Enterprise Server for SAP Applications 12 SP4 (ppc64le) - Kernel 4.12.14-95.6-default (hvc0).

SSH fingerprint: 2048 SHA256:F09o81EVv2vH8hduRjB9PAjclYlADlu6nEIE+8gRB8c root@eniesdbp101p (RSA)

eniesdbp101p login: root

Welcome to Relax-and-Recover. Run "rear recover" to restore your system !

RESCUE eniesdbp101p:~ # ls -l /dev/disk/by-label
ls: cannot access '/dev/disk/by-label': No such file or directory
RESCUE eniesdbp101p:~ # mkdir /dev/disk/by-label
RESCUE eniesdbp101p:~ # ln -s /dev/sr0 /dev/disk/by-label/RELAXRECOVER
RESCUE eniesdbp101p:~ # rear recover
Relax-and-Recover 2.4-git.3227.0f1f9dbb.master / 2019-02-10
Running rear recover (PID 13701)
Using log file: /var/log/rear/rear-eniesdbp101p.log
Running workflow recover within the ReaR rescue/recovery system
Using backup archive '/tmp/rear.HS3iq4f30qsvesD/outputfs/iso_fs/REAR_BACKUP/backup.tar.gz'
Will do driver migration (recreating initramfs/initrd)
Calculating backup archive size
Backup archive size is 2614.94M (compressed)
Setting up multipathing
Activating multipath
multipath activated
Starting multipath daemon
multipathd started
Listing multipath device found
20017380033e7008a dm-2 IBM,2810XIV size=240G
20017380033e70089 dm-1 IBM,2810XIV size=240G
20017380033e70072 dm-0 ##,## size=112G
20017380033e70084 dm-11 IBM,2810XIV size=385G
20017380033e70118 dm-9 IBM,2810XIV size=16G
20017380033e70083 dm-10 IBM,2810XIV size=385G
20017380033e70117 dm-8 IBM,2810XIV size=16G
20017380033e7007a dm-15 IBM,2810XIV size=96G
20017380033e70116 dm-7 IBM,2810XIV size=16G
20017380033e70079 dm-14 IBM,2810XIV size=96G
20017380033e70115 dm-6 IBM,2810XIV size=16G
20017380033e70078 dm-13 IBM,2810XIV size=96G
20017380033e7008c dm-4 IBM,2810XIV size=240G
20017380033e70077 dm-12 IBM,2810XIV size=96G
20017380033e7008b dm-3 IBM,2810XIV size=240G
20017380033e70076 dm-5 IBM,2810XIV size=64G
Comparing disks
Ambiguous possible target disks need manual configuration (more than one with same size found)
Switching to manual disk layout configuration
Using /dev/mapper/20017380033e70072 (same size) for recreating /dev/mapper/200173800348802c6
Current disk mapping table (source => target):
/dev/mapper/200173800348802c6 => /dev/mapper/20017380033e70072

Confirm or edit the disk mapping

  1. Confirm disk mapping and continue 'rear recover'
  2. n/a
  3. Edit disk mapping (/var/lib/rear/layout/disk_mappings)
  4. Use Relax-and-Recover shell and return back to here
  5. Abort 'rear recover'
    (default '1' timeout 300 seconds)
    1
    User confirmed disk mapping
    Confirm or edit the disk layout file
  6. Confirm disk layout and continue 'rear recover'
  7. Edit disk layout (/var/lib/rear/layout/disklayout.conf)
  8. View disk layout (/var/lib/rear/layout/disklayout.conf)
  9. View original disk space usage (/var/lib/rear/layout/config/df.txt)
  10. Use Relax-and-Recover shell and return back to here
  11. Abort 'rear recover'
    (default '1' timeout 300 seconds)
    1
    User confirmed disk layout file
    Confirm or edit the disk recreation script
  12. Confirm disk recreation script and continue 'rear recover'
  13. Edit disk recreation script (/var/lib/rear/layout/diskrestore.sh)
  14. View disk recreation script (/var/lib/rear/layout/diskrestore.sh)
  15. View original disk space usage (/var/lib/rear/layout/config/df.txt)
  16. Use Relax-and-Recover shell and return back to here
  17. Abort 'rear recover'
    (default '1' timeout 300 seconds)
    1
    User confirmed disk recreation script
    Start system layout restoration.
    Creating partitions for disk /dev/mapper/20017380033e70072 (msdos)
    Creating LVM PV /dev/mapper/20017380033e70072-part3
    Creating LVM VG 'system_vg'; Warning: some properties may not be preserved...
    Creating LVM volume 'system_vg/home_lv'; Warning: some properties may not be preserved...
    Creating LVM volume 'system_vg/opt_lv'; Warning: some properties may not be preserved...
    Creating LVM volume 'system_vg/root_lv'; Warning: some properties may not be preserved...
    Creating LVM volume 'system_vg/support_lv'; Warning: some properties may not be preserved...
    Creating LVM volume 'system_vg/swap_lv'; Warning: some properties may not be preserved...
    Creating LVM volume 'system_vg/syslogs_lv'; Warning: some properties may not be preserved...
    Creating LVM volume 'system_vg/tmp_lv'; Warning: some properties may not be preserved...
    Creating LVM volume 'system_vg/usr_lv'; Warning: some properties may not be preserved...
    Creating LVM volume 'system_vg/var_lv'; Warning: some properties may not be preserved...
    Creating filesystem of type xfs with mount point / on /dev/mapper/system_vg-root_lv.
    Mounting filesystem /
    Creating filesystem of type ext3 with mount point /boot on /dev/mapper/20017380033e70072-part2.
    Mounting filesystem /boot
    Creating filesystem of type xfs with mount point /home on /dev/mapper/system_vg-home_lv.
    Mounting filesystem /home
    Creating filesystem of type xfs with mount point /opt on /dev/mapper/system_vg-opt_lv.
    Mounting filesystem /opt
    Creating filesystem of type xfs with mount point /IBMsupport on /dev/mapper/system_vg-support_lv.
    Mounting filesystem /IBMsupport
    Creating filesystem of type xfs with mount point /syslogs on /dev/mapper/system_vg-syslogs_lv.
    Mounting filesystem /syslogs
    Creating filesystem of type xfs with mount point /tmp on /dev/mapper/system_vg-tmp_lv.
    Mounting filesystem /tmp
    Creating filesystem of type xfs with mount point /usr on /dev/mapper/system_vg-usr_lv.
    Mounting filesystem /usr
    Creating filesystem of type xfs with mount point /var on /dev/mapper/system_vg-var_lv.
    Mounting filesystem /var
    Creating swap on /dev/mapper/system_vg-swap_lv
    Disk layout created.
    Confirm the recreated disk layout or go back one step
  18. Confirm recreated disk layout and continue 'rear recover'
  19. Go back one step to redo disk layout recreation
  20. Use Relax-and-Recover shell and return back to here
  21. Abort 'rear recover'
    (default '1' timeout 300 seconds)
    1
    User confirmed recreated disk layout
    Restoring from '/tmp/rear.HS3iq4f30qsvesD/tmp/tar_fifo' (restore log in /var/lib/rear/restore/recover.tar_fifo.13701.restore.log) ...
    Backup restore program 'tar' started in subshell (PID=21918)
    Preparing to restore backup.tar.gz.00 ...
    Processing backup.tar.gz.00 ...
    Restoring finished (verify backup restore log messages in /var/lib/rear/restore/recover.tar_fifo.13701.restore.log)
    Recreating directories (with permissions) from /var/lib/rear/recovery/directories_permissions_owner_group
    Failed to 'chown root:root hana/data'
    Failed to 'chown root:root iso_fs'
    Failed to 'chown root:root usr/sap'
    Failed to 'chown root:root hana/shared'
    Failed to 'chown root:root var/opt/BESClient'
    Failed to 'chown root:root cv'
    Failed to 'chown root:root hana/log'
    Failed to 'chown root:root opt/teamquest'
    Failed to 'chown root:root opt/IBM/ITM'
    Failed to 'chown UNKNOWN:sudolinux opt/splunkforwarder'
    Failed to 'chown ph1adm:sapsys usr/sap/basis'
    Failed to 'chown ph1adm:UNKNOWN usr/sap/srm'
    Failed to 'chown ph1adm:sapsys PA_backup'
    Failed to 'chown root:root bin'
    Failed to 'chown root:root dev'
    Failed to 'chown root:root etc'
    Failed to 'chown root:root etc/opt'
    Failed to 'chown root:root etc/sgml'
    Failed to 'chown root:root etc/X11'
    Failed to 'chown root:root etc/xml'
    Failed to 'chown root:root lib'
    Failed to 'chown root:root lib64'
    Failed to 'chown root:root mnt'
    Failed to 'chown root:root proc'
    Failed to 'chown root:root root'
    Failed to 'chown root:root run'
    Failed to 'chown root:root sbin'
    Failed to 'chown root:root srv'
    Failed to 'chown root:root usr/bin'
    Failed to 'chown root:root usr/include'
    Failed to 'chown root:root usr/lib'
    Failed to 'chown root:root usr/lib64'
    Failed to 'chown root:root usr/local'
    Failed to 'chown root:root usr/sbin'
    Failed to 'chown root:root usr/share'
    Failed to 'chown root:root usr/src'
    Failed to 'chown root:root usr/X11R6'
    Failed to 'chown root:root var/cache'
    Failed to 'chown root:root var/lib'
    Failed to 'chown root:root var/log'
    Failed to 'chown root:root var/spool'
    Failed to 'chown root:root var/spool/mail'
    Failed to 'chown root:root var/tmp'
    ERROR: Unmounting '/tmp/rear.HS3iq4f30qsvesD/outputfs' failed.
    Some latest log messages since the last called script 980_umount_NETFS_dir.sh:
    umount: /tmp/rear.HS3iq4f30qsvesD/outputfs: target is busy
    (In some cases useful info about processes that
    use the device is found by lsof(8) or fuser(1).)
    2019-04-12 07:10:49.414246239 Forced unmount of '/tmp/rear.HS3iq4f30qsvesD/outputfs'
    umount: /tmp/rear.HS3iq4f30qsvesD/outputfs: target is busy
    (In some cases useful info about processes that
    use the device is found by lsof(8) or fuser(1).)
    2019-04-12 07:10:49.416780999 Unmounting '/tmp/rear.HS3iq4f30qsvesD/outputfs' failed.
    Aborting due to an error, check /var/log/rear/rear-eniesdbp101p.log for details
    Exiting rear recover (PID 13701) and its descendant processes
    Terminating descendant process 21935 dd if=/tmp/rear.HS3iq4f30qsvesD/outputfs
    Running exit tasks
    Terminated
    RESCUE eniesdbp101p:~ # dd if=/dev/zero of=/dev/mapper/20017380033e70072 bs=1M
    count=1000
    1000+0 records in
    1000+0 records out
    1048576000 bytes (1.0 GB, 1000 MiB) copied, 0.0904238 s, 11.6 GB/s
    RESCUE eniesdbp101p:~ #

#@ Third Activation

SIBM multipath devices discovered
rear selected correct disk
Recover worked - chown failures

                         GNU GRUB  version 2.02

+----------------------------------------------------------------------------+
|*Relax-and-Recover |
| |
| |
| |
| |
+----------------------------------------------------------------------------+

  Use the ^ and v keys to select which entry is highlighted.
  Press enter to boot the selected OS, `e' to edit the commands
  before booting or `c' for a command-line.

OF stdout device is: /vdevice/vty@30000000 Preparing to boot Linux version 4.12.14-95.6-default (geeko@buildhost) (gcc version 4.8.5 (SUSE Linux) ) #1 SMP Thu Jan 17 06:04:39 UTC 2019 (6af4ef8) Detected machine type: 0000000000000101 command line: BOOT_IMAGE=/kernel root=/dev/ram0 selinux=0 Max number of cores passed to firmware: 256 (NR_CPUS = 2048) Calling ibm,client-architecture-support... done memory layout at init: memory_limit : 0000000000000000 (16 MB aligned) alloc_bottom : 000000000f4d0000 alloc_top : 0000000020000000 alloc_top_hi : 0000000020000000 rmo_top : 0000000020000000 ram_top : 0000000020000000 instantiating rtas at 0x000000001eca0000... done prom_hold_cpus: skipped copying OF device tree... Building dt strings... Building dt structure... Device tree strings 0x000000000f4e0000 -> 0x000000000f4e1751 Device tree struct 0x000000000f4f0000 -> 0x000000000f520000 Quiescing Open Firmware ... Booting Linux via __start() @ 0x000000000a6f0000 ... Page sizes from device-tree: base_shift=12: shift=12, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=0 base_shift=12: shift=16, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=7 base_shift=12: shift=24, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=56 base_shift=16: shift=16, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=1 base_shift=16: shift=24, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=8 base_shift=24: shift=24, sllp=0x0100, avpnm=0x00000001, tlbiel=0, penc=0 base_shift=34: shift=34, sllp=0x0120, avpnm=0x000007ff, tlbiel=0, penc=3 Using 1TB segments Initializing hash mmu with SLB Linux version 4.12.14-95.6-default (geeko@buildhost) (gcc version 4.8.5 (SUSE Linux) ) #1 SMP Thu Jan 17 06:04:39 UTC 2019 (6af4ef8) Found initrd at 0xc00000000c600000:0xc00000000f4cfeda Using pSeries machine description bootconsole [udbg0] enabled Partition configured for 96 cpus. CPU maps initialized for 8 threads per core -> smp_release_cpus() spinning_secondaries = 31 <- smp_release_cpus()

ppc64_pft_size = 0x1e phys_mem_size = 0x2000000000 dcache_bsize = 0x80 icache_bsize = 0x80 cpu_features = 0x000000ff8f5d91a7 possible = 0x00007fffcf5fb1a7 always = 0x0000006f835c91a1 cpu_user_features = 0xdc0065c2 0xef000000 mmu_features = 0x7c006001 firmware_features = 0x00000017c45bfc57 htab_hash_mask = 0x7fffff

numa: NODE_DATA [mem 0x1fffcd6300-0x1fffcdffff]
numa: NODE_DATA(0) on node 5
numa: NODE_DATA [mem 0x1fffccc600-0x1fffcd62ff]
Section 8189 and 8191 (node 5) have a circular dependency on usemap and pgdat allocations
rfi-flush: fallback displacement flush available
rfi-flush: ori type flush available
barrier-nospec: using ORI speculation barrier
stf-barrier: hwsync barrier available
PPC64 nvram contains 15360 bytes
Zone ranges:
DMA [mem 0x0000000000000000-0x0000001fffffffff]
DMA32 empty
Normal empty
Movable zone start for each node
Early memory node ranges
node 5: [mem 0x0000000000000000-0x0000001fffffffff]
Could not find start_pfn for node 0
Initmem setup node 0 [mem 0x0000000000000000-0x0000000000000000]
Initmem setup node 5 [mem 0x0000000000000000-0x0000001fffffffff]
percpu: Embedded 3 pages/cpu @c000001ffe400000 s155160 r0 d41448 u262144
Built 2 zonelists in Node order, mobility grouping on. Total pages: 2095104
Policy zone: DMA
Kernel command line: BOOT_IMAGE=/kernel root=/dev/ram0 selinux=0
log_buf_len individual max cpu contribution: 4096 bytes
log_buf_len total cpu_extra contributions: 389120 bytes
log_buf_len min size: 524288 bytes
log_buf_len: 1048576 bytes
early log buf free: 519164(99%)
PID hash table entries: 4096 (order: -1, 32768 bytes)
Memory: 2108096K/134217728K available (9600K kernel code, 1792K rwdata, 2788K rodata, 3840K init, 11735K bss, 258048K reserved, 0K cma-reserved)
ftrace: allocating 26365 entries in 10 pages
Hierarchical RCU implementation.
RCU event tracing is enabled.
RCU restricting CPUs from NR_CPUS=2048 to nr_cpu_ids=96.
RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=96
NR_IRQS:512 nr_irqs:512 16
clocksource: timebase: mask: 0xffffffffffffffff max_cycles: 0x761537d007, max_idle_ns: 440795202126 ns
clocksource: timebase mult[1f40000] shift[24] registered
Console: colour dummy device 80x25
console [hvc0] enabled
console [hvc0] enabled
bootconsole [udbg0] disabled
bootconsole [udbg0] disabled
mempolicy: Disabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl
pid_max: default: 98304 minimum: 768
Security Framework initialized
AppArmor: AppArmor initialized
Dentry cache hash table entries: 8388608 (order: 10, 67108864 bytes)
Inode-cache hash table entries: 4194304 (order: 9, 33554432 bytes)
Mount-cache hash table entries: 131072 (order: 4, 1048576 bytes)
Mountpoint-cache hash table entries: 131072 (order: 4, 1048576 bytes)
EEH: pSeries platform initialized
POWER8 performance monitor hardware support registered
smp: Bringing up secondary CPUs ...
smp: Brought up 2 nodes, 32 CPUs
numa: Node 0 CPUs:
numa: Node 5 CPUs: 0-31
node 5 initialised, 2060181 pages in 40ms
devtmpfs: initialized
evm: security.selinux
evm: security.ima
evm: security.capability
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
futex hash table entries: 32768 (order: 6, 4194304 bytes)
NET: Registered protocol family 16
EEH: No capable adapters found
cpuidle: using governor ladder
cpuidle: using governor menu
pstore: using zlib compression
pstore: Registered nvram as persistent store backend
PCI: Probing PCI hardware
pseries-rng: Registering arch random hook.
HugeTLB registered 16 MB page size, pre-allocated 0 pages
HugeTLB registered 16 GB page size, pre-allocated 0 pages
vgaarb: loaded
EDAC MC: Ver: 3.0.0
NetLabel: Initializing
NetLabel: domain hash size = 128
NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO
NetLabel: unlabeled traffic allowed by default
clocksource: Switched to clocksource timebase
VFS: Disk quotas dquot_6.6.0
VFS: Dquot-cache hash table entries: 8192 (order 0, 65536 bytes)
AppArmor: AppArmor Filesystem Enabled
NET: Registered protocol family 2
TCP established hash table entries: 524288 (order: 6, 4194304 bytes)
TCP bind hash table entries: 65536 (order: 4, 1048576 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
UDP hash table entries: 65536 (order: 5, 2097152 bytes)
UDP-Lite hash table entries: 65536 (order: 5, 2097152 bytes)
NET: Registered protocol family 1
Unpacking initramfs...
random: fast init done
Freeing initrd memory: 47872K
numa: Starting topology update prrn_enabled
IOMMU table initialized, virtual merging enabled
hv-24x7: read 1330 catalog entries, created 337 event attrs (0 failures), 169 descs
audit: initializing netlink subsys (disabled)
audit: type=2000 audit(1555068164.290:1): state=initialized audit_enabled=0 res=1
Initialise system trusted keyrings
workingset: timestamp_bits=38 max_order=21 bucket_order=0
zbud: loaded
Key type asymmetric registered
Asymmetric key parser 'x509' registered
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249)
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
io scheduler mq-deadline registered
io scheduler kyber registered
io scheduler bfq registered
Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
pseries_rng: Registering IBM pSeries RNG driver
mousedev: PS/2 mouse device common for all mice
ledtrig-cpu: registered to indicate activity on CPUs
nx_compress_pseries ibm,compression-v1: nx842_OF_upd: device disabled
hidraw: raw HID events driver (C) Jiri Kosina
NET: Registered protocol family 10
Segment Routing with IPv6
NET: Registered protocol family 15
registered taskstats version 1
Loading compiled-in X.509 certificates
Loaded X.509 cert 'SUSE Linux Enterprise Secure Boot Signkey: 3fb077b6cebc6ff2522e1c148c57c777c788e3e7'
zswap: loaded using pool lzo/zbud
page_owner is disabled
Key type trusted registered
Key type encrypted registered
AppArmor: AppArmor sha1 policy hashing enabled
ima: No TPM chip found, activating TPM-bypass! (rc=-19)
ima: Allocated hash algorithm: sha256
evm: HMAC attrs: 0x1
Freeing unused kernel memory: 3840K
This architecture does not have kernel memory protection.
alg: No test for pkcs1pad(rsa,sha256) (pkcs1pad(rsa-generic,sha256))
random: systemd: uninitialized urandom read (16 bytes read)
random: systemd: uninitialized urandom read (16 bytes read)
random: systemd: uninitialized urandom read (16 bytes read)

Welcome to SUSE Linux Enterprise Server 12 SP4!

[ OK ] Created slice system.slice.
[ OK ] Started Relax-and-Recover sshd service.
[ OK ] Reached target System Initialization.
[ OK ] Created slice system-getty.slice.
[ OK ] Listening on Journal Socket.
[ OK ] Listening on udev Kernel Socket.
[ OK ] Listening on udev Control Socket.
Starting udev Coldplug all Devices...
Starting udev Kernel Device Manager...
[ OK ] Started Journal Service.
[ OK ] Created slice system-serial\x2dgetty.slice.
Starting Relax-and-Recover boot script...
[ OK ] Started udev Kernel Device Manager.
Starting Initialize Rescue System...
[ OK ] Started Relax-and-Recover boot script.
[ OK ] Found device /dev/ttyS0.
[ OK ] Started udev Coldplug all Devices.
[ OK ] Found device /dev/hvc0.

Verifying md5sums of the files in the Relax-and-Recover rescue system

md5sums are OK

Configuring Relax-and-Recover rescue system

Running 00-functions.sh...
Running 01-run-ldconfig.sh...
Running 10-console-setup.sh...
Using keymap of the original system
Running 20-check-boot-options.sh...
Running 40-start-udev-or-load-modules.sh...
Waiting for udev ... done.
Running 41-load-special-modules.sh...
Running 42-engage-scsi.sh...
Running 45-serial-console.sh...
Running 55-migrate-network-devices.sh...
No network interface mapping is specified in /etc/rear/mappings/mac
The original network interface eth0 d6:8b:e5:4d:db:08 is not available

  1. eth3 c6:c5:cb:8e:5c:03 ibmveth
  2. eth4 c6:c5:cb:8e:5c:05 ibmveth
  3. eth5 c6:c5:cb:8e:5c:04 ibmveth
  4. Skip replacing eth0 d6:8b:e5:4d:db:08
    Choose replacement for eth0 d6:8b:e5:4d:db:08 1
    Mapping eth0 d6:8b:e5:4d:db:08 to eth3 c6:c5:cb:8e:5c:03
    The original network interface eth1 d6:8b:e5:4d:db:09 is not available
    1. eth4 c6:c5:cb:8e:5c:05 ibmveth
  6. eth5 c6:c5:cb:8e:5c:04 ibmveth
  7. Skip replacing eth1 d6:8b:e5:4d:db:09
    Choose replacement for eth1 d6:8b:e5:4d:db:09 3
    Mapping eth1 d6:8b:e5:4d:db:09 to eth5 c6:c5:cb:8e:5c:04
    The original network interface eth2 d6:8b:e5:4d:db:0a is not available
    1. eth4 c6:c5:cb:8e:5c:05 ibmveth
    1. Skip replacing eth2 d6:8b:e5:4d:db:0a
      Choose replacement for eth2 d6:8b:e5:4d:db:0a 2
      Mapping eth2 d6:8b:e5:4d:db:0a to eth4 c6:c5:cb:8e:5c:05
      Reloading udev ... done.
      Running 58-start-dhclient.sh...
      Running 60-network-devices.sh...
      Running 62-routing.sh...
      Running 65-sysctl.sh...
      net.ipv6.conf.all.disable_ipv6 = 1
      net.ipv4.ip_forward = 0
      net.ipv6.conf.all.forwarding = 0
      vm.pagecache_limit_ignore_dirty = 2
      vm.min_free_kbytes = 10485760
      net.core.somaxconn = 4096
      net.ipv4.tcp_max_syn_backlog = 8192
      net.ipv4.tcp_tw_reuse = 1
      net.ipv4.tcp_timestamps = 1
      net.ipv4.tcp_slow_start_after_idle = 0
      net.ipv4.tcp_window_scaling = 1
      net.ipv4.tcp_syn_retries = 8
      net.core.rmem_max = 56623104
      net.core.wmem_max = 56623104
      net.ipv4.tcp_rmem = 65536 262088 56623104
      net.ipv4.tcp_wmem = 65536 262088 56623104
      net.ipv4.tcp_mem = 56623104 56623104 56623104
      fs.aio-max-nr = 18446744073709551615
      fs.file-max = 20000000
      vm.memory_failure_early_kill = 1
      kernel.shmmni = 65536
      net.ipv6.conf.all.disable_ipv6 = 1
      net.ipv4.ip_forward = 0
      net.ipv6.conf.all.forwarding = 0
      vm.pagecache_limit_ignore_dirty = 2
      vm.min_free_kbytes = 10485760
      net.core.somaxconn = 4096
      net.ipv4.tcp_max_syn_backlog = 8192
      net.ipv4.tcp_tw_reuse = 1
      net.ipv4.tcp_timestamps = 1
      net.ipv4.tcp_slow_start_after_idle = 0
      net.ipv4.tcp_window_scaling = 1
      net.ipv4.tcp_syn_retries = 8
      net.core.rmem_max = 56623104
      net.core.wmem_max = 56623104
      net.ipv4.tcp_rmem = 65536 262088 56623104
      net.ipv4.tcp_wmem = 65536 262088 56623104
      net.ipv4.tcp_mem = 56623104 56623104 56623104
      fs.aio-max-nr = 18446744073709551615
      fs.file-max = 20000000
      vm.memory_failure_early_kill = 1
      kernel.shmmni = 65536
      Running 99-makedev.sh...

Relax-and-Recover rescue system is ready

[ OK ] Started Initialize Rescue System.
[ OK ] Started Getty on tty0.
[ OK ] Reached target Login Prompts.
[ OK ] Started Serial Getty on ttyS0.
[ OK ] Started Serial Getty on hvc0.
[ OK ] Reached target Multi-User.

Relax-and-Recover 2.4-git.3227.0f1f9dbb.master / 2019-02-10

Relax-and-Recover comes with ABSOLUTELY NO WARRANTY; for details see
the GNU General Public License at: http://www.gnu.org/licenses/gpl.html

Host eniesdbp101p using Backup NETFS and Output ISO
Build date: Tue, 09 Apr 2019 14:15:01 -0400

Welcome to SUSE Linux Enterprise Server for SAP Applications 12 SP4 (ppc64le) - Kernel 4.12.14-95.6-default (hvc0).

SSH fingerprint: 2048 SHA256:uEJQ+W510Qrpjp4WSfutZFRqhrPJ0/3YOdxw1bgp7KE root@eniesdbp101p (RSA)

eniesdbp101p login: root

Welcome to Relax-and-Recover. Run "rear recover" to restore your system !

RESCUE eniesdbp101p:~ # ls -l /dev/disk/by-label
ls: cannot access '/dev/disk/by-label': No such file or directory
RESCUE eniesdbp101p:~ # mkdir /dev/disk/by-label
RESCUE eniesdbp101p:~ # ln -s /dev/sr0 /dev/disk/by-label/RELAXRECOVER
RESCUE eniesdbp101p:~ # rear recover
Relax-and-Recover 2.4-git.3227.0f1f9dbb.master / 2019-02-10
Running rear recover (PID 25732)
Using log file: /var/log/rear/rear-eniesdbp101p.log
Running workflow recover within the ReaR rescue/recovery system
Using backup archive '/tmp/rear.aAxYg7j9BmHnaTQ/outputfs/iso_fs/REAR_BACKUP/backup.tar.gz'
Will do driver migration (recreating initramfs/initrd)
Calculating backup archive size
Backup archive size is 2614.94M (compressed)
Setting up multipathing
Activating multipath
multipath activated
Starting multipath daemon
multipathd started
Listing multipath device found
20017380033e7008a dm-15 IBM,2810XIV size=240G
20017380033e70075 dm-21 IBM,2810XIV size=64G
SIBM_2810XIV_78033E70073 dm-0 IBM,2810XIV size=16G
20017380033e70074 dm-20 IBM,2810XIV size=32G
20017380033e70089 dm-14 IBM,2810XIV size=240G
SIBM_2810XIV_78033E70072 dm-2 ##,## size=112G
20017380033e70073 dm-19 IBM,2810XIV size=16G
20017380033e70072 dm-18 IBM,2810XIV size=112G
SIBM_2810XIV_78033E70117 dm-24 IBM,2810XIV size=16G
20017380033e70084 dm-9 IBM,2810XIV size=385G
20017380033e70118 dm-7 IBM,2810XIV size=16G
20017380033e70083 dm-8 IBM,2810XIV size=385G
20017380033e70117 dm-6 IBM,2810XIV size=16G
SIBM_2810XIV_78033E70115 dm-23 IBM,2810XIV size=16G
20017380033e7007a dm-13 IBM,2810XIV size=96G
20017380033e70116 dm-5 IBM,2810XIV size=16G
20017380033e70079 dm-12 IBM,2810XIV size=96G
20017380033e70115 dm-4 IBM,2810XIV size=16G
20017380033e70078 dm-11 IBM,2810XIV size=96G
20017380033e7008c dm-16 IBM,2810XIV size=240G
20017380033e70077 dm-10 IBM,2810XIV size=96G
SIBM_2810XIV_78033E70075 dm-1 IBM,2810XIV size=64G
20017380033e7008b dm-17 IBM,2810XIV size=240G
SIBM_2810XIV_78033E7007A dm-25 IBM,2810XIV size=96G
20017380033e70076 dm-3 IBM,2810XIV size=64G
SIBM_2810XIV_78033E70074 dm-22 IBM,2810XIV size=32G
Comparing disks
Ambiguous possible target disks need manual configuration (more than one with same size found)
Switching to manual disk layout configuration
Using /dev/mapper/20017380033e70072 (same size) for recreating /dev/mapper/200173800348802c6
Current disk mapping table (source => target):
/dev/mapper/200173800348802c6 => /dev/mapper/20017380033e70072

Confirm or edit the disk mapping

  1. Confirm disk mapping and continue 'rear recover'
  2. n/a
  3. Edit disk mapping (/var/lib/rear/layout/disk_mappings)
  4. Use Relax-and-Recover shell and return back to here
  5. Abort 'rear recover'
    (default '1' timeout 300 seconds)
    1
    User confirmed disk mapping
    Confirm or edit the disk layout file
  6. Confirm disk layout and continue 'rear recover'
  7. Edit disk layout (/var/lib/rear/layout/disklayout.conf)
  8. View disk layout (/var/lib/rear/layout/disklayout.conf)
  9. View original disk space usage (/var/lib/rear/layout/config/df.txt)
  10. Use Relax-and-Recover shell and return back to here
  11. Abort 'rear recover'
    (default '1' timeout 300 seconds)
    1
    User confirmed disk layout file
    Confirm or edit the disk recreation script
  12. Confirm disk recreation script and continue 'rear recover'
  13. Edit disk recreation script (/var/lib/rear/layout/diskrestore.sh)
  14. View disk recreation script (/var/lib/rear/layout/diskrestore.sh)
  15. View original disk space usage (/var/lib/rear/layout/config/df.txt)
  16. Use Relax-and-Recover shell and return back to here
  17. Abort 'rear recover'
    (default '1' timeout 300 seconds)
    1
    User confirmed disk recreation script
    Start system layout restoration.
    Creating partitions for disk /dev/mapper/20017380033e70072 (msdos)
    Creating LVM PV /dev/mapper/20017380033e70072-part3
    Creating LVM VG 'system_vg'; Warning: some properties may not be preserved...
    Creating LVM volume 'system_vg/home_lv'; Warning: some properties may not be preserved...
    Creating LVM volume 'system_vg/opt_lv'; Warning: some properties may not be preserved...
    Creating LVM volume 'system_vg/root_lv'; Warning: some properties may not be preserved...
    Creating LVM volume 'system_vg/support_lv'; Warning: some properties may not be preserved...
    Creating LVM volume 'system_vg/swap_lv'; Warning: some properties may not be preserved...
    Creating LVM volume 'system_vg/syslogs_lv'; Warning: some properties may not be preserved...
    Creating LVM volume 'system_vg/tmp_lv'; Warning: some properties may not be preserved...
    Creating LVM volume 'system_vg/usr_lv'; Warning: some properties may not be preserved...
    Creating LVM volume 'system_vg/var_lv'; Warning: some properties may not be preserved...
    Creating filesystem of type xfs with mount point / on /dev/mapper/system_vg-root_lv.
    Mounting filesystem /
    Creating filesystem of type ext3 with mount point /boot on /dev/mapper/20017380033e70072-part2.
    Mounting filesystem /boot
    Creating filesystem of type xfs with mount point /home on /dev/mapper/system_vg-home_lv.
    Mounting filesystem /home
    Creating filesystem of type xfs with mount point /opt on /dev/mapper/system_vg-opt_lv.
    Mounting filesystem /opt
    Creating filesystem of type xfs with mount point /IBMsupport on /dev/mapper/system_vg-support_lv.
    Mounting filesystem /IBMsupport
    Creating filesystem of type xfs with mount point /syslogs on /dev/mapper/system_vg-syslogs_lv.
    Mounting filesystem /syslogs
    Creating filesystem of type xfs with mount point /tmp on /dev/mapper/system_vg-tmp_lv.
    Mounting filesystem /tmp
    Creating filesystem of type xfs with mount point /usr on /dev/mapper/system_vg-usr_lv.
    Mounting filesystem /usr
    Creating filesystem of type xfs with mount point /var on /dev/mapper/system_vg-var_lv.
    Mounting filesystem /var
    Creating swap on /dev/mapper/system_vg-swap_lv
    Disk layout created.
    Confirm the recreated disk layout or go back one step
  18. Confirm recreated disk layout and continue 'rear recover'
  19. Go back one step to redo disk layout recreation
  20. Use Relax-and-Recover shell and return back to here
  21. Abort 'rear recover'
    (default '1' timeout 300 seconds)
    1
    User confirmed recreated disk layout
    Restoring from '/tmp/rear.aAxYg7j9BmHnaTQ/tmp/tar_fifo' (restore log in /var/lib/rear/restore/recover.tar_fifo.25732.restore.log) ...
    Backup restore program 'tar' started in subshell (PID=33713)
    Preparing to restore backup.tar.gz.00 ...
    Processing backup.tar.gz.00 ...
    Restored 7904 MiB [avg. 120808 KiB/sec] OK
    Restored 7904 MiB in 68 seconds [avg. 119031 KiB/sec]
    Restoring finished (verify backup restore log messages in /var/lib/rear/restore/recover.tar_fifo.25732.restore.log)
    Recreating directories (with permissions) from /var/lib/rear/recovery/directories_permissions_owner_group
    Failed to 'chown ph1adm:UNKNOWN usr/sap/srm'
    Confirm restored config files or edit them
  22. Confirm it is OK to recreate initrd and reinstall bootloader and continue 'rear recover'
  23. Edit restored etc/fstab (/mnt/local/etc/fstab)
  24. View restored etc/fstab (/mnt/local/etc/fstab)
  25. Use Relax-and-Recover shell and return back to here
  26. Abort 'rear recover'
    (default '1' timeout 300 seconds)
    1
    User confirmed restored files
    Applying disk layout mappings in /var/lib/rear/layout/disk_mappings to certain restored files...
    The original restored files get saved in var/lib/rear/saved_original_files/ (in /mnt/local)
    Applied disk layout mappings to restored 'boot/grub2/grub.cfg' (in /mnt/local)
    Applied disk layout mappings to restored 'boot/grub2/device.map' (in /mnt/local)
    Applied disk layout mappings to restored 'etc/sysconfig/bootloader' (in /mnt/local)
    Applied disk layout mappings to restored 'etc/fstab' (in /mnt/local)
    Applied disk layout mappings to restored 'etc/mtools.conf' (in /mnt/local)
    Applied disk layout mappings to restored 'etc/smartd.conf' (in /mnt/local)
    Applied disk layout mappings to restored 'etc/sysconfig/smartmontools' (in /mnt/local)
    Applied disk layout mappings to restored 'etc/security/pam_m

jsmeix commented at 2019-04-26 08:56:

This is a very special case that cannot be fully solved for ReaR 2.5
so I postpone it to ReaR 2.6...

dewagner1 commented at 2019-04-26 11:14:

Do you have an estimate for ReaR 2.6 availability?

jsmeix commented at 2019-04-26 13:14:

@dewagner1
not that it actually helps you, see
https://github.com/rear/rear/milestones

ReaR v2.6
No due date

I postponed this issue to ReaR 2.6 to be able to move forward
with releasing ReaR 2.5 because its intended release date is
currently Past due by about 1 month and even that had been
already postponed from beginning of February to end of March,
see https://github.com/rear/rear/issues/2033

Simply put:
This special case issue should not block releasing ReaR 2.5
that is already very late (overall past due by about 3 month).

jsmeix commented at 2020-03-13 16:18:

FYI regarding the above
https://github.com/rear/rear/issues/2019#issuecomment-476598723
and the subsequent comments:

I think I am one step closer to the general problem, see
https://github.com/rear/rear/issues/799#issuecomment-598626162

I used LVM on top of RAID1 (i.e. the PVs are RAID1 storage objects).
In this case here it seems LVM is used on top of multipath storage objects.
So it seems the common thing in both cases is that LVM PVs are
not plain disk partitions like /dev/sdXn but higher level storage objects
where wipefs -af /dev/sdXn is insufficient to remove old metadata.

ccjung commented at 2020-03-13 16:42:

Hello,
Thanks for the info.

When will the update for this issue be available?

A different question: Is there a ReaR version to support RHEL7 or 8 on Power?

jsmeix commented at 2020-03-16 15:29:

@ccjung
what do you mean with "the update for this issue"?

Did you read
https://github.com/rear/rear/issues/799#issuecomment-598626162
?

Neither is there currently nor will there be in the foreseeable future
an automatic solution within ReaR.

Currently and in the foreseeable future how to deal with it is to
"Prepare replacement hardware for disaster recovery", cf.
https://en.opensuse.org/SDB:Disaster_Recovery
therein in particular the part about
"you must completely zero out your replacement storage".

Regarding which ReaR upstream version supports
which Linux distributions and which hardware architectures see
https://raw.githubusercontent.com/rear/rear/master/doc/rear-release-notes.txt
or (the same) at
http://relax-and-recover.org/documentation/release-notes-2-5

In particular regarding RHEL specific questions about what
ReaR versions Red Hat officially supports you may also
get in contact with your particular RHEL support contact.

Additionally or alternatively see also
http://relax-and-recover.org/support/

ccjung commented at 2020-03-16 15:49:

Yes, I read the #799 so I thought there might be an update to the ReaR program.

It sounds like the workaround is to zero out the previously used disk if there is restore problem.

We have been using this zero out disk workaround and it is working so far.

jsmeix commented at 2020-03-16 16:50:

To zero out whole disks is not possible by default within ReaR
because on huge disks that can take very much (i.e. too much) time
BUT
I see now that it might be possible if explicitly requested by the user
via a new config array variable for example something like

ZERO_OUT_DISKS=( /dev/sda /dev/sdb )

BUT
that would need to be implemented carefully right
to avoid that accidentally wrong disks get zeroed out
e.g. when /dev/sda is a USB disk with the ReaR recovery system
cf. https://github.com/rear/rear/issues/1271

I will have to meditate on it...

github-actions commented at 2020-06-30 01:33:

Stale issue message

[Export of Github issue for rear/rear.]