#1786 Issue `closed`: SUSE 12.2 ReaR recovery stuck at system layout restoration¶

Labels: support / question, won't fix / can't fix / obsolete

manums1983 opened issue at 2018-04-27 06:29:¶

Relax-and-Recover (ReaR) Issue Template¶

SUSE 12.2 ReaR recovery stuck at system layout restoration
Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

ReaR version ("/usr/sbin/rear -V"): 2.31
OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"):
SUSE 12.2 (SP2) SAP Server
ReaR configuration files ("cat /etc/rear/site.conf" or "cat /etc/rear/local.conf"):
System is not booting up to get local.conf file
System architecture (x86 compatible or POWER and/or what kind of virtual machine):
Inte 64 bit
Are you using BIOS or UEFI or another way to boot?
UEFI
Brief description of the issue:
The resrore is stuck at "system layout restoration"
Work-around, if any:

gozora commented at 2018-04-27 06:39:¶

@manums1983 I'm afraid that from such vague problem description, none will be actually able to help you.
I'm suspecting that you are doing rear recover.
Can you relaunch with rear -d -D recover and provide us with log files from /var/log/rear ?

Thanks

V.

manums1983 commented at 2018-04-27 06:46:¶

Hi Vladimir,
Thank you for your response, ok I will do the recovery using the command “rear -d -D recover” . Now I am able to login to the server. Please find the details requested below. I am using a file based backup to local mount point “/mnt/backup”. I noticed this /mnt/backup is mounted in a btrfs file system. Now I deleted the volume and recreated with ext4 file system. Running the backup again now after this I will do a restore again.

################local.conf #################################################

Begin example setup for SLE12-SP2 with default btrfs subvolumes.¶

Since SLE12-SP1 what is mounted at '/' is a btrfs snapshot subvolume¶

see https://github.com/rear/rear/issues/556 ¶

and since SLE12-SP2 btrfs quota via "snapper setup-quota" is needed¶

see https://github.com/rear/rear/issues/999 ¶

You must adapt "your.NFS.server.IP/path/to/your/rear/backup" at BACKUP_URL.¶

You must decide whether or not you want to have /home/* in the backup.¶

It depends on the size of your harddisk whether or not /home is by default¶

a btrfs subvolume or a separated xfs filesystem on a separated partition.¶

You may activate SSH_ROOT_PASSWORD and adapt the "password_on_the_rear_recovery_system".¶

For basic information see the SLE12-SP2 manuals.¶

Also see the support database article "SDB:Disaster Recovery"¶

at http://en.opensuse.org/SDB:Disaster_Recovery ¶

In particular note:¶

There is no such thing as a disaster recovery solution that "just works".¶

Regarding btrfs snapshots:¶

Recovery of btrfs snapshot subvolumes is not possible.¶

Only recovery of "normal" btrfs subvolumes is possible.¶

On SLE12-SP1 and SP2 the only exception is the btrfs snapshot subvolume¶

that is mounted at '/' but that one is not recreated but instead¶

it is created anew from scratch during the recovery installation with the¶

default first btrfs snapper snapshot subvolume path "@/.snapshots/1/snapshot"¶

by the SUSE tool "installation-helper --step 1" (cf. below).¶

Other snapshots like "@/.snapshots/234/snapshot" are not recreated.¶

Create rear recovery system as ISO image:¶

OUTPUT=ISO

Store the backup file via NFS on a NFS server:¶

BACKUP=NETFS

BACKUP_OPTIONS variable contains the NFS mount options and¶

with 'mount -o nolock' no rpc.statd (plus rpcbind) are needed:¶

BACKUP_OPTIONS="nfsvers=3,nolock"

If the NFS server is not an IP address but a hostname,¶

DNS must work in the rear recovery system when the backup is restored.¶

BACKUP_URL=file:///mnt/backup

Keep an older copy of the backup in a HOSTNAME.old directory¶

provided there is no '.lockfile' in the HOSTNAME directory:¶

NETFS_KEEP_OLD_BACKUP_COPY=10

Have all modules of the original system in the recovery system with the¶

same module loading ordering as in the original system by using the output of¶

lsmod | tail -n +2 | cut -d ' ' -f 1 | tac | tr -s '[:space:]' ' '¶

as value for MODULES_LOAD (cf. https://github.com/rear/rear/issues/626):¶

#MODULES_LOAD=( )

On SLE12-SP1 and SP2 with default btrfs subvolumes what is mounted at '/' is a btrfs snapshot subvolume¶

that is controlled by snapper so that snapper is needed in the recovery system.¶

In SLE12-SP1 and SP2 some btrfs subvolume directories (/var/lib/pgsql /var/lib/libvirt/images /var/lib/mariadb)¶

have the "no copy on write (C)" file attribute set so that chattr is required in the recovery system¶

and accordingly also lsattr is useful to have in the recovery system (but not strictly required):¶

REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" snapper chattr lsattr )

Snapper setup by the recovery system uses /usr/lib/snapper/installation-helper¶

that is linked to all libraries where snapper is linked to¶

(except libdbus that is only needed by snapper).¶

"installation-helper --step 1" creates a snapper config based on /etc/snapper/config-templates/default¶

COPY_AS_IS=( "${COPY_AS_IS[@]}" /usr/lib/snapper/installation-helper /etc/snapper/config-templates/default )

Files in btrfs subvolumes are excluded by 'tar --one-file-system'¶

so that such files must be explicitly included to be in the backup.¶

Files in the following SLE12-SP2 default btrfs subvolumes are¶

in the below example not included to be in the backup¶

/.snapshots /var/crash¶

BACKUP_PROG_EXCLUDE=( "${BACKUP_PROG_EXCLUDE[@]}" "/media/backup" "/var/crash" "/export/archive" "/home/oracle" "/oracle" "/home/oraprd" "/export/Doc" "/usr/sap/hostctrl" "/sapmnt/PRD" "/usr/sap/SMD" "/usr/sap/tmp" "/usr/sap/PRD" "/sapmnt/PRD/exe" "/oracle/stage" "/oracle/PRD" "/oracle/client" "/oracle/oraprd" "/oracle/PRD/12102" "/oracle/PRD/origlogA" "/oracle/PRD/mirrlogA" "/oracle/PRD/mirrlogB" "/oracle/PRD/origlogB" "/oracle/PRD/sapreorg" "/oracle/PRD/sapdata3" "/oracle/PRD/oraarch" "/oracle/PRD/sapdata4" "/oracle/PRD/sapdata6" "/oracle/PRD/sapdata2" "/oracle/PRD/sapdata1" "/oracle/PRD/sapdata5" "/mnt/backup" )
EXCLUDE_RECREATE=( "${EXCLUDE_RECREATE[@]}" "fs:/media/backup" "fs:/var/crash" "fs:/export/archive" "fs:/home/oracle" "fs:/oracle" "fs:/home/oraprd" "fs:/export/Doc" "fs:/usr/sap/hostctrl" "fs:/sapmnt/PRD" "fs:/usr/sap/SMD" "fs:/usr/sap/tmp" "fs:/usr/sap/PRD" "fs:/sapmnt/PRD/exe" "fs:/oracle/stage" "fs:/oracle/PRD" "fs:/oracle/client" "fs:/oracle/oraprd" "fs:/oracle/PRD/12102" "fs:/oracle/PRD/origlogA" "fs:/oracle/PRD/mirrlogA" "fs:/oracle/PRD/mirrlogB" "fs:/oracle/PRD/origlogB" "fs:/oracle/PRD/sapreorg" "fs:/oracle/PRD/sapdata3" "fs:/oracle/PRD/oraarch" "fs:/oracle/PRD/sapdata4" "fs:/oracle/PRD/sapdata6" "fs:/oracle/PRD/sapdata2" "fs:/oracle/PRD/sapdata1" "fs:/oracle/PRD/sapdata5" "fs:/mnt/backup" )

but files in /home are included to be in the backup.¶

You may use a command like¶

findmnt -n -r -o TARGET -t btrfs | grep -v '^/$' | egrep -v 'snapshots|crash'¶

to generate the values:¶

BACKUP_PROG_INCLUDE=( /var/cache /var/lib/mailman /var/tmp /var/lib/pgsql /usr/local /opt /var/lib/libvirt/images /boot/grub2/i386-pc /var/opt /srv /boot/grub2/x86_64-efi /var/lib/mariadb /var/spool /var/lib/mysql /tmp /home /var/log /var/lib/named /var/lib/machines )

The following POST_RECOVERY_SCRIPT implements during "rear recover"¶

btrfs quota setup for snapper if that is used in the original system:¶

POST_RECOVERY_SCRIPT=( 'if snapper --no-dbus -r $TARGET_FS_ROOT get-config | grep -q "^QGROUP.*[0-9]/[0-9]" ; then snapper --no-dbus -r $TARGET_FS_ROOT set-config QGROUP= ; snapper --no-dbus -r $TARGET_FS_ROOT setup-quota && echo snapper setup-quota done || echo snapper setup-quota failed ; else echo snapper setup-quota not used ; fi' )

This option defines a root password to allow SSH connection¶

whithout a public/private key pair¶

#SSH_ROOT_PASSWORD="password_on_the_rear_recovery_system"

Let the rear recovery system run dhclient to get an IP address¶

instead of using the same IP address as the original system:¶

#USE_DHCLIENT="yes"

End example setup for SLE12-SP2 with default btrfs subvolumes.¶

#########################################################################
NAME="SLES"
VERSION="12-SP2"
VERSION_ID="12.2"
PRETTY_NAME="SUSE Linux Enterprise Server 12 SP2"
ID="sles"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:12:sp2"

###################################################
Linux GESPRD1 4.4.103-92.56-default #1 SMP Wed Dec 27 16:24:31 UTC 2017 (2fd2155) x86_64 x86_64 x86_64 GNU/Linux

#######################################################
Relax-and-Recover 2.3-git.0.dd982eb.unknown / 2018-04-16
#################################################

Thanks & Regards
Manu MS
Technology Consultant
HPE Pointnext
Mobile: +6591747535
Email : manu.ms@hpe.com mailto:manu.ms@hpe.com

From: Vladimir Gozora [mailto:notifications@github.com]
Sent: Friday, April 27, 2018 2:40 PM
To: rear/rear rear@noreply.github.com
Cc: MOHANAN SATHYKUMARI, MANU manu.mohanan-sathykumari@hpe.com; Mention mention@noreply.github.com
Subject: Re: [rear/rear] SUSE 12.2 ReaR recovery stuck at system layout restoration (#1786)

@manums1983https://github.com/manums1983 I'm afraid that from such vague problem description, none will be actually able to help you.
I'm suspecting that you are doing rear recover.
Can you relaunch with rear -d -D recover and provide us with log files from /var/log/rear ?

Thanks

V.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/rear/rear/issues/1786#issuecomment-384879411, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ak-0Cy7gPNKA1CrLSYA4ajGr7wbFQjftks5tsr0tgaJpZM4Tp12D.

gozora commented at 2018-04-27 07:13:¶

My humble recommendation would be "Do not use backup to local filesystem (file://) until you are 100% sure what you are doing and what the implications are". Try to go with nfs:// or smb:// for the start.

V.

manums1983 commented at 2018-04-27 08:01:¶

HI Vladimir,
Thank for the information about the file based backup implications. We have tested SUSE 12.2 on another server using file based backup and restore it works fine which is on hardware HPE GEN9 DL380. Since we do not have an NFS server we need to use file based backup and recovery.

Current hardware is HPE GEN9 DL 580, it is a SAP server. Started the restore with command “rear -d -D recover” still it is stuck at “Start System Layout Restoration” I have attached some screenshots and backup logs. Please let know what would be causing this issue.

How to stop the restore. It is in hung state.

Thanks & Regards
Manu MS
From: Vladimir Gozora [mailto:notifications@github.com]
Sent: Friday, April 27, 2018 3:13 PM
To: rear/rear rear@noreply.github.com
Cc: MOHANAN SATHYKUMARI, MANU manu.mohanan-sathykumari@hpe.com; Mention mention@noreply.github.com
Subject: Re: [rear/rear] SUSE 12.2 ReaR recovery stuck at system layout restoration (#1786)

My humble recommendation would be "Do not use backup to local filesystem (file://file:///\\) until you are 100% sure what you are doing and what the implications are". Try to go with nfs:// or smb:// for the start.

V.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/rear/rear/issues/1786#issuecomment-384885977, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ak-0CwALFVE43y4jkbg49KH5Ye03wpNHks5tssUKgaJpZM4Tp12D.

manums1983 commented at 2018-04-27 08:07:¶

rear case log .docx
Can copy the log form iLO so attached the screenshots.

manums1983 commented at 2018-04-27 08:54:¶

I just noticed the /tmp directory got full. /tmp is only 10 GB. Is there a way i can point rear log to a different directory?

jsmeix commented at 2018-04-27 08:55:¶

@manums1983
ReaR is not something where you can "just simply make settings"
what might look to you as if it works. For example
NETFS_KEEP_OLD_BACKUP_COPY=10
does not work this way.
When you use ReaR you need to carefully read the documentation
in particular read default.conf for each config variable and often
you may even have a look at the scripts to really understand
what a particular setting actually does.
In general see
https://en.opensuse.org/SDB:Disaster_Recovery

I will not even try to read any kind of "proprietary" file format like doc or docx
(even if it is possible with huge deskop applications like LibreOffice).
Please provide ReaR's plain text debug log file completely as is.
Usually screenshots won't help because plain error messages
won't tell us the root cause.
Only the ReaR debug log file may tell the root cause.

FYI:
Your TMPDIR issue is explained in default.conf.

jsmeix commented at 2018-04-27 09:19:¶

@manums1983

FYI:
For me with a SLES12-SP3 default system (with its default btrfs structure)
on a QEMU/KVM virtual machine with two virtual harddisks
a 20 GiB sda for the system plus a 2 GiB sdb for the backup
the following /etc/rear/local.conf works:

OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=file:///mnt/sdb1
OUTPUT_URL=nfs://10.160.4.244/nfs
REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" snapper chattr lsattr )
COPY_AS_IS=( "${COPY_AS_IS[@]}" /usr/lib/snapper/installation-helper /etc/snapper/config-templates/default )
BACKUP_PROG_INCLUDE=( /srv /var/lib/mailman /tmp /var/lib/libvirt/images /var/opt /var/log /boot/grub2/i386-pc /var/lib/mysql /var/tmp /opt /boot/grub2/x86_64-efi /var/spool /var/lib/pgsql /var/lib/mariadb /usr/local /home /var/lib/machines /var/lib/named /var/cache )
POST_RECOVERY_SCRIPT=( 'if snapper --no-dbus -r $TARGET_FS_ROOT get-config | grep -q "^QGROUP.*[0-9]/[0-9]" ; then snapper --no-dbus -r $TARGET_FS_ROOT set-config QGROUP= ; snapper --no-dbus -r $TARGET_FS_ROOT setup-quota && echo snapper setup-quota done || echo snapper setup-quota failed ; else echo snapper setup-quota not used ; fi' )
SSH_ROOT_PASSWORD="rear"
USE_DHCLIENT="yes"

but one must know what BACKUP_URL=file:///mnt/sdb1 means:

On the original system I did the preparation steps

# parted -s /dev/sdb mklabel msdos

# parted -s /dev/sdb unit MiB mkpart primary 1 2047

# mkfs.ext4 /dev/sdb1

# parted -s /dev/sdb unit MiB print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sdb: 2048MiB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 
Number  Start    End      Size     Type     File system  Flags
 1      1.00MiB  2047MiB  2046MiB  primary  ext4         type=83

# mkdir /mnt/sdb1

# mount /dev/sdb1 /mnt/sdb1

and after "rear -D mkbackup" I got

# find /mnt/sdb1

/mnt/sdb1
/mnt/sdb1/lost+found
/mnt/sdb1/f121
/mnt/sdb1/f121/backup.log
/mnt/sdb1/f121/backup.tar.gz

and on my NFS server (OUTPUT_URL=nfs://10.160.4.244/nfs) I got

nfs-server:/nfs # ls -lhrt /nfs/f121

total 85M
-rw------- 1 nobody nogroup  80M Apr 27 10:40 rear-f121.iso
-rw------- 1 nobody nogroup  261 Apr 27 10:40 VERSION
-rw------- 1 nobody nogroup  202 Apr 27 10:40 README
-rw------- 1 nobody nogroup 5.2M Apr 27 10:40 rear-f121.log

For the recovery I set up a second virtual machine
with a new 20 GiB sda for the system
and the existing 2 GiB sdb from the original system with the backup
and a virtual CD-ROM where I have the rear-f121.iso to boot
the ReaR recovery system on the second virtual machine.

In the running ReaR recovery system on the second virtual machine
I did the preparation steps to make the backup on its sdb accessible:

# mkdir /mnt/sdb1

# mount /dev/sdb1 /mnt/sdb1

and then I run "rear -D recover" which then "just works" for me.

But personally I find using BACKUP_URL=file:///... too complicated
from my point of view - but I don't know your particular use-case.

Personally I would perfer OUTPUT=USB to get the backup together
with the ReaR recovery system on a bootable disk medium.

I think - with probability one (https://en.wikipedia.org/wiki/Almost_surely) - this issue
is a ReaR configuration issue or a "how to use ReaR" issue.

jsmeix commented at 2018-04-27 09:59:¶

Bad typo correction in
https://github.com/rear/rear/issues/1786#issuecomment-384915337
wrong before and then I run "rear -D mkbackup" which then "just works" for me.
now corrected and then I run "rear -D recover" which then "just works" for me.

manums1983 commented at 2018-04-27 10:04:¶

Hi Valdimir
I am collecting the restore log file will update you shortly.

Below showing the backup logs
##########################################Backup Successful ###########################
GESPRD1:/mnt/backup # rear -d -D mkbackup
Relax-and-Recover 2.3-git.0.dd982eb.unknown / 2018-04-16
Using log file: /var/log/rear/rear-GESPRD1.log
Using backup archive '/mnt/backup/GESPRD1/backup.tar.gz'
Using UEFI Boot Loader for Linux (USING_UEFI_BOOTLOADER=1)
Creating disk layout
Excluding component fs:/media/backup
Excluding component fs:/var/crash
Excluding component fs:/export/archive
Excluding component fs:/home/oracle
Excluding component fs:/oracle
Excluding component fs:/home/oraprd
Excluding component fs:/export/Doc
Excluding component fs:/usr/sap/hostctrl
Excluding component fs:/sapmnt/PRD
Excluding component fs:/usr/sap/SMD
Excluding component fs:/usr/sap/tmp
Excluding component fs:/usr/sap/PRD
Excluding component fs:/sapmnt/PRD/exe
Excluding component fs:/oracle/stage
Excluding component fs:/oracle/PRD
Excluding component fs:/oracle/client
Excluding component fs:/oracle/oraprd
Excluding component fs:/oracle/PRD/12102
Excluding component fs:/oracle/PRD/origlogA
Excluding component fs:/oracle/PRD/mirrlogA
Excluding component fs:/oracle/PRD/mirrlogB
Excluding component fs:/oracle/PRD/origlogB
Excluding component fs:/oracle/PRD/sapreorg
Excluding component fs:/oracle/PRD/sapdata3
Excluding component fs:/oracle/PRD/oraarch
Excluding component fs:/oracle/PRD/sapdata4
Excluding component fs:/oracle/PRD/sapdata6
Excluding component fs:/oracle/PRD/sapdata2
Excluding component fs:/oracle/PRD/sapdata1
Excluding component fs:/oracle/PRD/sapdata5
Excluding component fs:/mnt/backup
Using sysconfig bootloader 'grub2-efi'
Creating root filesystem layout
Handling network interface 'bond0'
bond0 is a bond
bond0 has lower interface eth0
eth0 is a physical device
bond0 has lower interface eth1
eth1 is a physical device
bond0 has lower interface eth2
eth2 is a physical device
bond0 has lower interface eth3
eth3 is a physical device
Handled network interface 'bond0'
Trying to find what to use as UEFI bootloader...
Trying to find a 'well known file' to be used as UEFI bootloader...
Using '/boot/efi/EFI/sles/grubx64.efi' as UEFI bootloader file
Copying logfile /var/log/rear/rear-GESPRD1.log into initramfs as '/tmp/rear-GESPRD1-partial-2018-04-27T14:40:31+08:00.log'
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Copying all files in /lib*/firmware/
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (207576391 bytes) in 76 seconds
Making ISO image
Wrote ISO image: /var/lib/rear/output/rear-GESPRD1.iso (242M)
Copying resulting files to file location
Saving /var/log/rear/rear-GESPRD1.log as rear-GESPRD1.log to file location
Creating tar archive '/mnt/backup/GESPRD1/backup.tar.gz'
Archived 12648 MiB [avg 6759 KiB/sec] OK
Archived 12648 MiB in 1917 seconds [avg 6756 KiB/sec]
Exiting rear mkbackup (PID 9207) and its descendant processes
Running exit tasks
You should also rm -Rf /tmp/rear.u7WHb4sgvsPoLPy

Thanks & Regards

Manu MS

manums1983 commented at 2018-04-27 10:08:¶

In the /mnt/backup/GESPRD1 will have all gzip file and ISO file. I will copy ISO to my local laptop and attach this ISO to iLO console and boot from rear ISO.

GESPRD1:/mnt/backup/GESPRD1 # ls -l
total 13242768
-rw------- 1 root root 202 Apr 27 14:42 README
-rw------- 1 root root 293 Apr 27 14:42 VERSION
-rw------- 1 root root 26890796 Apr 27 15:14 backup.log
-rw------- 1 root root 13263365582 Apr 27 15:14 backup.tar.gz
-rw------- 1 root root 253259776 Apr 27 14:42 rear-GESPRD1.iso
-rw------- 1 root root 17055276 Apr 27 14:42 rear-GESPRD1.log

manums1983 commented at 2018-04-27 10:10:¶

Please find the logs attached.
rear-logs.zip.

gozora commented at 2018-04-27 14:24:¶

Hello @manums1983
As @jsmeix pointed your in his https://github.com/rear/rear/issues/1786#issuecomment-384915337 and added reasoning to my brief "no no no, don't do it" https://github.com/rear/rear/issues/1786#issuecomment-384885977, what is happening to your rear recover is following.

Either your /mnt/backup contains mounted some remote filesystem or external device, in such case you should use appropriate protocol (nfs://, smb://, usb://) instead of file://
Or you store backup on your local filesystem which is part of your OS FS structure. This is simply not a good idea and you should use remote storage that is not directly related to OS your are backing up.

V.

gozora commented at 2018-04-27 14:44:¶

As I've read your disklayout.conf I've noticed that point 2. of my https://github.com/rear/rear/issues/1786#issuecomment-384985728 is true.

Your /mnt/backup is located on /dev/sdb1 which is SmarArray logical volume spread across single disk. If this is a test server all you have to do before starting rear recover is to mount your /mnt/backup directory and maybe comment out following line from /var/lib/rear/disklayout.conf

logicaldrive /dev/sdb 0|A|1 raid=0 drives=1I:1:3, spares= sectors=32 stripesize=256

it is just a guess, but ReaR might try to re-create your SmartArray configuration with this line active, which is not something you want.

If this is however production server, you should reconsider your bare metal disaster recovery strategy, because storing backups locally on SmartArray without any redundancy will really not help you in case of disaster.

V.

jsmeix commented at 2018-04-27 14:53:¶

@gozora
as far as I see by quick googling 'SmartArray' is HP specific hardware
so that I added the "special hardware" label to this issue
(remove it if if I am wrong).

gozora commented at 2018-04-27 14:58:¶

@jsmeix yes, you are right with one tiny correction, it is HPE nowadays :-), this is however not the crux of the problem, but rather (as you already stated)

I think - with probability one (https://en.wikipedia.org/wiki/Almost_surely) - this issue
is a ReaR configuration issue or a "how to use ReaR" issue.

So I personally would avoid using "special hardware" label ...

V.

manums1983 commented at 2018-04-27 15:04:¶

Hardware Specs:

HPE iLO4 "Smart Array P830i".

Hard disk Configuration : RAID1 Array (2x600 GB) (/dev/sda) ---> This is local disk where OS is installed.
RAID-0 (1x600 GB) ---> This is also local disk is where the ReaR backup is pointed /mnt/backup. This is GPT, Primary with EXT 4 partition (/dev/sdb1).

All other disk are 3PAR multipath disks. All 3PAR disks exclude from the backup and recreation.

jsmeix commented at 2018-04-27 15:12:¶

I don't know about Smart Array but assume there is real hardware RAID here and
not a B110i software RAID solution based on the Smart Array firmware as in
https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c00687518
so that it is basically the same disk layout as in my
https://github.com/rear/rear/issues/1786#issuecomment-384915337
"sda for the system ... sdb for the backup"

@manums1983
when your backup is on local disk sdb
I wonder how you access it from a replacement server?
But I may misunderstand things because I don't know about Smart Array.

gozora commented at 2018-04-27 15:17:¶

@jsmeix

I don't know about Smart Array but assume there is real hardware RAID here and
not a B110i software RAID solution based on the Smart Array firmware as in ...

Correct ;-)

gozora commented at 2018-04-27 15:21:¶

@jsmeix

I wonder how you access it from a replacement server?

I'm afraid that there is no replacement server, SmartArray disks are presented to system as any other disk /dev/sda, /dev/sdb ... So I guess that strategy here is just to boot ReaR recovery system and restore content of /dev/sda from /dev/sdb

V.

manums1983 commented at 2018-04-27 15:31:¶

This is purely a hardware RAID array controller. Not software based. This is a production SAP server in DR site. Highly critical to make any kind of changes,
Just wondering why this is not working because the same method of backup and restore successfully done on Hardware GEN 9 DL 380 box with OS SUSE 12.2 no issues faced, but this server do not have any 3PAR LUNs, the server have local disks sda (system) and sdb (backup).

Here the difference is DL 580 with OS SUSE 12.2 with lots of 3PAR Luns. Noticed the restore getting stuck at "Start System Layout Restoration" , do we have any issue with multi-path.

I wonder how you access it from a replacement server? ---> OS is running on RAID1 (sda) if it corrupted after patching or any other reasons can restore the (sda) from the backup (sdb). Boot from ReaR ISO , in rescue mode mount /dev/sab1 /mnt/backup and start recovery.

manums1983 commented at 2018-04-27 15:34:¶

correction:
mount /dev/sdb1 /mnt/backup and start recovery.

gozora commented at 2018-04-27 15:37:¶

@manums1983 if you think you are heaving problem with multipath and you are doing recovery solely on local disks, just try to unload dm-multipath and friends kernel modules + whatever FC (lpfc, bnx, qlaxxx) driver you are using prior rear recover.
This should let you only with local disks visible to ReaR recovery system, so there shall be no interference ...

V.

manums1983 commented at 2018-04-27 16:00:¶

@gozora,
could you pls help on this how to unload multipath disks kernel modules . This is production server i am afraid some mistake may happen by doing so.

gozora commented at 2018-04-27 17:27:¶

@manums1983 if this is an critical / production server and downtime would mean any kind of SLA breach you should either escalate this task to higher support level or ask some more experienced colleague for help.
Community around ReaR can help you with ReaR related problems but I guess that no one will give you step by step guidance through every single step that can go wrong during OS recovery ...

V.

manums1983 commented at 2018-04-28 03:03:¶

Hi gozora,
ReaR can very well handle the external disks (multipath) during backup and recovery. May you help to find obstacle for the ReaR recovery to hang at disk layout restoration phase. Is there any issues with local disk configuration or any ReaR configuration problem.
We are using Data Protector in the environment i hope ReaR can integrated with DP, but do not know this is a good method. Currently Data Protector will do the backup of /mnt/backup as flat file backup as second copy.

gozora commented at 2018-04-28 08:24:¶

One of the options would be to run recovery with debug options rear -d -D recover and check logs in /var/log/rear .

V.

manums1983 commented at 2018-04-28 08:30:¶

sure i will share you the log on Monday.

manums1983 commented at 2018-04-30 05:31:¶

Hi Gozora,
Please find the logs in /var/log/rear/ and layouts. The recovery hung in the very begining.

I like to try backup to USB. Below configuration is ok to boot form USB and do recovery. pls let know.
rear format /dev/sdX
OUTPUT=USB
BACKUP=NETFS
BACKUP_URL="usb:///dev/disk/by-label/REAR-000"
rear-GESPRD1.log

manums1983 commented at 2018-04-30 06:18:¶

I try backup to USB. But getting following error in writing logs.
rear-GESPRD1.log

Trying to find what to use as UEFI bootloader...
Trying to find a 'well known file' to be used as UEFI bootloader...
Using '/boot/efi/EFI/sles/grubx64.efi' as UEFI bootloader file
Copying logfile /var/log/rear/rear-GESPRD1.log into initramfs as '/tmp/rear-GESPRD1-partial-2018-04-30T14:13:23+08:00.log'
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Copying all files in /lib*/firmware/
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (208486998 bytes) in 75 seconds
ERROR: Could not copy /mnt/backup/rear.YnLVHYO5VGq69u4/tmp/initrd.cgz to /tmp/rear-efi.yJHGT//EFI/BOOT/initrd.cgz
Aborting due to an error, check /var/log/rear/rear-GESPRD1.log for details
Exiting rear mkbackup (PID 84124) and its descendant processes
Running exit tasks
Terminated

manums1983 commented at 2018-04-30 06:45:¶

Noticed that the /dev/sdaq1 is getting full and it is failing.
/dev/sdaq1 200M 200M 0 100% /tmp/rear-efi.exLw5

manums1983 commented at 2018-04-30 09:44:¶

Hi Gozora,
I did backup as file based and USB. Both having the same issue. They are getting hung at "Disk Layout Restoration".
As you have requested i have attached the logs. could pls have a look on.
rear-GESPRD1.log

###############Logs################################

++ StopIfError 'You must specify either BACKUP_URL or BACKUP_MOUNTCMD and BACKUP_UMOUNTCMD !'
++ StopIfError 'Could not mkdir '''/mnt/backup/rear.QIWPMl9uTLsWAr5/outputfs''''
++ StopIfError 'Mount command '''mount -v -o rw,noatime /dev/disk/by-label/REAR-000 /mnt/backup/rear.QIWPMl9uTLsWAr5/outputfs''' failed.'
++ StopIfError 'Could not find file '''mbr.bin'''. Syslinux version 3.08 or newer is required, 4.x prefered!'
++ StopIfError 'Unmounting '''/mnt/backup/rear.QIWPMl9uTLsWAr5/outputfs''' failed.'
+++ StopIfError 'Partition number '''1''' of partition sda1 is not a valid number.'
+++ StopIfError 'Partition sda1 is numbered '''1'''. More than 128 partitions is not supported.'
+++ StopIfError 'Partition number '''2''' of partition sda2 is not a valid number.'
+++ StopIfError 'Partition sda2 is numbered '''2'''. More than 128 partitions is not supported.'
+++ StopIfError 'Partition number '''3''' of partition sda3 is not a valid number.'
+++ StopIfError 'Partition sda3 is numbered '''3'''. More than 128 partitions is not supported.'
+++ StopIfError 'Partition number '''4''' of partition sda4 is not a valid number.'
+++ StopIfError 'Partition sda4 is numbered '''4'''. More than 128 partitions is not supported.'
+++ StopIfError 'Partition number '''5''' of partition sda5 is not a valid number.'
+++ StopIfError 'Partition sda5 is numbered '''5'''. More than 128 partitions is not supported.'
+++ StopIfError 'Partition number '''6''' of partition sda6 is not a valid number.'
+++ StopIfError 'Partition sda6 is numbered '''6'''. More than 128 partitions is not supported.'
+++ StopIfError 'Partition number '''7''' of partition sda7 is not a valid number.'
+++ StopIfError 'Partition sda7 is numbered '''7'''. More than 128 partitions is not supported.'
+++ StopIfError 'Partition number '''8''' of partition sda8 is not a valid number.'
+++ StopIfError 'Partition sda8 is numbered '''8'''. More than 128 partitions is not supported.'
+++ StopIfError 'Partition number '''1''' of partition sdaq1 is not a valid number.'
+++ StopIfError 'Partition sdaq1 is numbered '''1'''. More than 128 partitions is not supported.'
+++ StopIfError 'Partition number '''2''' of partition sdaq2 is not a valid number.'
+++ StopIfError 'Partition sdaq2 is numbered '''2'''. More than 128 partitions is not supported.'
+++ StopIfError 'Partition number '''1''' of partition sdb1 is not a valid number.'
+++ StopIfError 'Partition sdb1 is numbered '''1'''. More than 128 partitions is not supported.'
++ StopIfError 'Failed to save XFS options of /dev/mapper/360002ac000000000000000230001e534-part1'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Failed to save XFS options of /dev/sda4'
++ StopIfError 'Failed to save XFS options of /dev/sda5'
++ StopIfError 'Failed to save XFS options of /dev/sda6'
++ StopIfError 'Failed to save XFS options of /dev/sda7'
++ StopIfError 'Failed to save XFS options of /dev/sda8'
++ StopIfError 'Divide by zero detected'
++ StopIfError 'Divide by zero detected'
++ LogIfError 'Did not find sysfs name for device 360002ac0000000000000002c0001e534 (/sys/block/dm-7)'
++ LogIfError 'Failed to get size of dm-7 with get_disk_size'
+++ StopIfError 'Partition number '''1''' of partition 360002ac0000000000000002c0001e534-part1 is not a valid number.'
+++ StopIfError 'Partition 360002ac0000000000000002c0001e534-part1 is numbered '''1'''. More than 128 partitions is not supported.'
+++ StopIfError 'Partition number '''1''' of partition 360002ac0000000000000002c0001e534-part1 is not a valid number.'
+++ StopIfError 'Partition 360002ac0000000000000002c0001e534-part1 is numbered '''1'''. More than 128 partitions is not supported.'
++ LogIfError 'Did not find sysfs name for device 360002ac000000000000000260001e534 (/sys/block/dm-3)'
++ LogIfError 'Failed to get size of dm-3 with get_disk_size'
+++ StopIfError 'Partition number '''1''' of partition 360002ac000000000000000260001e534-part1 is not a valid number.'
+++ StopIfError 'Partition 360002ac000000000000000260001e534-part1 is numbered '''1'''. More than 128 partitions is not supported.'
+++ StopIfError 'Partition number '''1''' of partition 360002ac000000000000000260001e534-part1 is not a valid number.'
+++ StopIfError 'Partition 360002ac000000000000000260001e534-part1 is numbered '''1'''. More than 128 partitions is not supported.'
++ LogIfError 'Did not find sysfs name for device 360002ac0000000000000002a0001e534 (/sys/block/dm-9)'
++ LogIfError 'Failed to get size of dm-9 with get_disk_size'
+++ StopIfError 'Partition number '''1''' of partition 360002ac0000000000000002a0001e534-part1 is not a valid number.'
+++ StopIfError 'Partition 360002ac0000000000000002a0001e534-part1 is numbered '''1'''. More than 128 partitions is not supported.'
+++ StopIfError 'Partition number '''1''' of partition 360002ac0000000000000002a0001e534-part1 is not a valid number.'
+++ StopIfError 'Partition 360002ac0000000000000002a0001e534-part1 is numbered '''1'''. More than 128 partitions is not supported.'
++ LogIfError 'Did not find sysfs name for device 360002ac000000000000000240001e534 (/sys/block/dm-1)'
++ LogIfError 'Failed to get size of dm-1 with get_disk_size'
+++ StopIfError 'Partition number '''1''' of partition 360002ac000000000000000240001e534-part1 is not a valid number.'
+++ StopIfError 'Partition 360002ac000000000000000240001e534-part1 is numbered '''1'''. More than 128 partitions is not supported.'
+++ StopIfError 'Partition number '''1''' of partition 360002ac000000000000000240001e534-part1 is not a valid number.'
+++ StopIfError 'Partition 360002ac000000000000000240001e534-part1 is numbered '''1'''. More than 128 partitions is not supported.'
++ LogIfError 'Did not find sysfs name for device 360002ac0000000000000002d0001e534 (/sys/block/dm-8)'
++ LogIfError 'Failed to get size of dm-8 with get_disk_size'
+++ StopIfError 'Partition number '''1''' of partition 360002ac0000000000000002d0001e534-part1 is not a valid number.'
+++ StopIfError 'Partition 360002ac0000000000000002d0001e534-part1 is numbered '''1'''. More than 128 partitions is not supported.'
+++ StopIfError 'Partition number '''1''' of partition 360002ac0000000000000002d0001e534-part1 is not a valid number.'
+++ StopIfError 'Partition 360002ac0000000000000002d0001e534-part1 is numbered '''1'''. More than 128 partitions is not supported.'
++ LogIfError 'Did not find sysfs name for device 360002ac000000000000000270001e534 (/sys/block/dm-4)'
++ LogIfError 'Failed to get size of dm-4 with get_disk_size'
+++ StopIfError 'Partition number '''1''' of partition 360002ac000000000000000270001e534-part1 is not a valid number.'
+++ StopIfError 'Partition 360002ac000000000000000270001e534-part1 is numbered '''1'''. More than 128 partitions is not supported.'
+++ StopIfError 'Partition number '''1''' of partition 360002ac000000000000000270001e534-part1 is not a valid number.'
+++ StopIfError 'Partition 360002ac000000000000000270001e534-part1 is numbered '''1'''. More than 128 partitions is not supported.'
++ LogIfError 'Did not find sysfs name for device 360002ac0000000000000002b0001e534 (/sys/block/dm-6)'
++ LogIfError 'Failed to get size of dm-6 with get_disk_size'
+++ StopIfError 'Partition number '''1''' of partition 360002ac0000000000000002b0001e534-part1 is not a valid number.'
+++ StopIfError 'Partition 360002ac0000000000000002b0001e534-part1 is numbered '''1'''. More than 128 partitions is not supported.'
+++ StopIfError 'Partition number '''1''' of partition 360002ac0000000000000002b0001e534-part1 is not a valid number.'
+++ StopIfError 'Partition 360002ac0000000000000002b0001e534-part1 is numbered '''1'''. More than 128 partitions is not supported.'
++ LogIfError 'Did not find sysfs name for device 360002ac000000000000000250001e534 (/sys/block/dm-2)'
++ LogIfError 'Failed to get size of dm-2 with get_disk_size'
+++ StopIfError 'Partition number '''1''' of partition 360002ac000000000000000250001e534-part1 is not a valid number.'
+++ StopIfError 'Partition 360002ac000000000000000250001e534-part1 is numbered '''1'''. More than 128 partitions is not supported.'
+++ StopIfError 'Partition number '''1''' of partition 360002ac000000000000000250001e534-part1 is not a valid number.'
+++ StopIfError 'Partition 360002ac000000000000000250001e534-part1 is numbered '''1'''. More than 128 partitions is not supported.'
++ LogIfError 'Did not find sysfs name for device 360002ac000000000000000230001e534 (/sys/block/dm-0)'
++ LogIfError 'Failed to get size of dm-0 with get_disk_size'
+++ StopIfError 'Partition number '''1''' of partition 360002ac000000000000000230001e534-part1 is not a valid number.'
+++ StopIfError 'Partition 360002ac000000000000000230001e534-part1 is numbered '''1'''. More than 128 partitions is not supported.'
+++ StopIfError 'Partition number '''1''' of partition 360002ac000000000000000230001e534-part1 is not a valid number.'
+++ StopIfError 'Partition 360002ac000000000000000230001e534-part1 is numbered '''1'''. More than 128 partitions is not supported.'
++ LogIfError 'Did not find sysfs name for device 360002ac000000000000000280001e534 (/sys/block/dm-5)'
++ LogIfError 'Failed to get size of dm-5 with get_disk_size'
+++ StopIfError 'Partition number '''1''' of partition 360002ac000000000000000280001e534-part1 is not a valid number.'
+++ StopIfError 'Partition 360002ac000000000000000280001e534-part1 is numbered '''1'''. More than 128 partitions is not supported.'
+++ StopIfError 'Partition number '''1''' of partition 360002ac000000000000000280001e534-part1 is not a valid number.'
+++ StopIfError 'Partition 360002ac000000000000000280001e534-part1 is numbered '''1'''. More than 128 partitions is not supported.'
++ StopIfError 'Could not find a suitable kernel. Maybe you have to set KERNEL_FILE [/boot/vmlinuz-4.4.103-92.56-default] ?'
++ StopIfError 'Failed to create mount point /tmp/rear-efi.MQ5RB'
++ StopIfError 'Failed to mount EFI partition /dev/disk/by-label/REAR-EFI to /tmp/rear-efi.MQ5RB'
++ StopIfError 'Failed to create /tmp/rear-efi.MQ5RB//EFI/BOOT'
++ StopIfError 'Could not copy EFI bootloader to /tmp/rear-efi.MQ5RB//EFI/BOOT/BOOTX64.efi'
++ StopIfError 'Could not copy /boot/vmlinuz-4.4.103-92.56-default to /tmp/rear-efi.MQ5RB//EFI/BOOT/kernel'
++ StopIfError 'Could not copy /mnt/backup/rear.QIWPMl9uTLsWAr5/tmp/initrd.cgz to /tmp/rear-efi.MQ5RB//EFI/BOOT/initrd.cgz'
++ StopIfError 'Failed to create BOOTX64.efi'
++ LogIfError 'Could not remove temporary directory /tmp/rear-efi.MQ5RB, please check manually'
++ StopIfError 'Could not mkdir '''/mnt/backup/rear.QIWPMl9uTLsWAr5/outputfs''''
++ StopIfError 'Mount command '''mount -v -o rw,noatime /dev/disk/by-label/REAR-000 /mnt/backup/rear.QIWPMl9uTLsWAr5/outputfs''' failed.'
++ StopIfError 'Could not mkdir /mnt/backup/rear.QIWPMl9uTLsWAr5/tmp/boot'
++ StopIfError 'Could not mkdir '''/mnt/backup/rear.QIWPMl9uTLsWAr5/outputfs/GESPRD1''''
++ StopIfError 'Could not create '''/mnt/backup/rear.QIWPMl9uTLsWAr5/outputfs/GESPRD1/.lockfile''''
++ StopIfError 'Could not find a working syslinux path.'
++ StopIfError 'Could not create USB ReaR dir [/mnt/backup/rear.QIWPMl9uTLsWAr5/outputfs/rear/GESPRD1/20180430.1537] !'
++ StopIfError 'Could not create USB syslinux dir [/mnt/backup/rear.QIWPMl9uTLsWAr5/outputfs/boot/syslinux] !'
++ StopIfError 'Could not create /mnt/backup/rear.QIWPMl9uTLsWAr5/outputfs/rear/GESPRD1/20180430.1537/kernel'
++ StopIfError 'Could not create /mnt/backup/rear.QIWPMl9uTLsWAr5/outputfs/rear/GESPRD1/20180430.1537/initrd.cgz'
++ StopIfError 'Could not copy /var/log/rear/rear-GESPRD1.log to /mnt/backup/rear.QIWPMl9uTLsWAr5/outputfs/rear/GESPRD1/20180430.1537/rear-GESPRD1.log'
++ BugIfError 'RAW_USB_DEVICE and REAL_USB_DEVICE should be already set'
++ StopIfError 'Problem with extlinux -i /mnt/backup/rear.QIWPMl9uTLsWAr5/outputfs/boot/syslinux'
++ StopIfError 'Problem with writing the mbr.bin to '''/dev/sdaq''''
++ StopIfError 'Could not copy '''/usr/share/rear/conf/templates/RESULT_usage_USB.txt''''
++ StopIfError 'Unmounting '''/mnt/backup/rear.QIWPMl9uTLsWAr5/outputfs''' failed.'
++ StopIfError 'Could not mkdir '''/mnt/backup/rear.QIWPMl9uTLsWAr5/outputfs''''
++ StopIfError 'Mount command '''mount -v -o rw,noatime /dev/disk/by-label/REAR-000 /mnt/backup/rear.QIWPMl9uTLsWAr5/outputfs''' failed.'
++ StopIfError 'Could not remove '''/mnt/backup/rear.QIWPMl9uTLsWAr5/outputfs/rear/GESPRD1/20180430.1537.old''''
++ StopIfError 'Could not move '''/mnt/backup/rear.QIWPMl9uTLsWAr5/outputfs/rear/GESPRD1/20180430.1537''''
++ StopIfError 'Could not mkdir '''/mnt/backup/rear.QIWPMl9uTLsWAr5/outputfs/rear/GESPRD1/20180430.1537''''
++ StopIfError 'Could not create '''/mnt/backup/rear.QIWPMl9uTLsWAr5/outputfs/rear/GESPRD1/20180430.1537/.lockfile''''
++ StopIfError 'Unmounting '''/mnt/backup/rear.QIWPMl9uTLsWAr5/outputfs''' failed.'
GESPRD1:/var/log/rear # ls
rear-GESPRD1.log rear-GESPRD1.log.old
GESPRD1:/var/log/rear # cat rear-GESPRD1.log.old | grep "Error"
GESPRD1:/var/log/rear # ls -l
total 17600
-rw-r----- 1 root root 18013129 Apr 30 16:35 rear-GESPRD1.log
-rw-r----- 1 root root 4455 Apr 30 15:37 rear-GESPRD1.log.old
GESPRD1:/var/log/rear # ls -l
total 17600
-rw-r----- 1 root root 18013129 Apr 30 16:35 rear-GESPRD1.log
-rw-r----- 1 root root 4455 Apr 30 15:37 rear-GESPRD1.log.old
GESPRD1:/var/log/rear # ls -l
total 17600
-rw-r----- 1 root root 18013129 Apr 30 16:35 rear-GESPRD1.log
-rw-r----- 1 root root 4455 Apr 30 15:37 rear-GESPRD1.log.old

gozora commented at 2018-04-30 10:31:¶

Again, if you are heaving trouble during restore phase, please provide log files created by rear -d -D recover

V.

manums1983 commented at 2018-04-30 10:34:¶

Hi Gozora,
i was doing it from iLO console i could not copy it from there. i believe i copied the "rear -d -D recover" logs. i copied it form location /var/log/rear/.
Ok let me try once more.

manums1983 commented at 2018-04-30 10:37:¶

I tried it copy the /var/log/rear/gesprd*** to a local drive. but the server is getting hung. Not responding to any abort commands.
I have a screen shot can i attach?

manums1983 commented at 2018-04-30 11:17:¶

I have attached restore -d -D recover error screenshot. i could not copy any logs to local directory because the whole server is hung. Really appreciate if you can see something form the screenshots.
restore_error_log_screenshot.docx

gozora commented at 2018-04-30 14:44:¶

I don't really believe that screenshot is enough ...
Are you aware that there is sshd running inside ReaR recovery system ?

V.

manums1983 commented at 2018-05-01 04:59:¶

Hi gozora,
I tried ssh to the server via putty from my laptop but it was failing.
Let me tell you the other thing i noticed while it was booting from rescue disk is that it was failing at script "40-start-udev-or-load-modules.sh". Below is the error message it was captured from screen
############Rescue disk Boot error on screen#############
Running 40-start-udev-or-load-modules.sh...
insmod /lib/modules/4.4.103-92..56-default/weak-update/updates/lpfc.ko lpfc_devloss_tmo=14 ipfc_lun_queue_depth=16 lpfc_discovery_threads=32
grep: write error.
########################################################
I understand without proper recovery log it is very difficult to analyse the problem. But situation i have is once i hit "rear -d -D recover" the server fully getting hung and only response to iLO reboot option which making no way to collect the logs. Also on screen i was not able to see any real time logs.
Please let know the above log making any sense relate to any know issue, because i observed in another recent case it seem as a bug.
https://github.com/rear/rear/issues/1207

gozora commented at 2018-05-01 07:29:¶

@manums1983 I'm sorry, but without proper logs I can't help you.

V.

[Export of Github issue for rear/rear.]

#1786 Issue closed: SUSE 12.2 ReaR recovery stuck at system layout restoration¶