#228 Issue closed: Are SAN boot disks (with "mpath") supported by "rear" ?

Labels: enhancement, fixed / solved / done

WautersPh opened issue at 2013-04-18 07:26:

Dears,

I'm getting the following error when attempting to "rear recover" a RHEL 5.6 system having its boot disk on the SAN.

Can rear handle such a configuration or is "rear" only foreseen for local system disks ?
Or am I missing a critical setup in "rear" somewhere ?

Thanks on forehand for your help.

Regards.

Philippe

WautersPh commented at 2013-04-18 07:31:

"No code has been generated to restore device pv:/dev/mpath/mpath0p2 (lvmdev).
Please add code to /var/lib/rear/layout/diskrestore.sh to manually install it or choose abort."

gdha commented at 2013-04-18 08:03:

in the default.conf file the following is defined by default AUTOEXCLUDE_MULTIPATH=y which excludes SAN disks. If you want SAN disks to be added then un-define in /etc/rear/local.conf the variable to AUTOEXCLUDE_MULTIPATH=

WautersPh commented at 2013-04-18 08:06:

Hi Gratien,

I'm afraid I've already tried this but it doesn't help (at least for the boot disk on the SAN).
It probably works for data disks on the SAN.

I can give you a copy of the local.conf file and the log files if this helps.

Thanks for helping me as this is becoming a relatively hot topic here and I have a test system to reproduce the problem right now.

Kind regards.

Philippe

gdha commented at 2013-04-18 08:24:

Ok, run a rear -d -D mkrescue and attach the log file as a gist (and link it into this issue)

WautersPh commented at 2013-04-18 08:28:

OK : I will run this.
Sorry for this probably very stupid question but ... what is a "gist" ?

gdha commented at 2013-04-18 08:45:

see https://gist.github.com/

WautersPh commented at 2013-04-18 08:48:

Found it ... and trying to upload the log file (but is is 4 MB big though ...).

WautersPh commented at 2013-04-18 08:54:

https://gist.github.com/WautersPh/9d5812c7091cec1b2f79

jhoekx commented at 2013-04-18 09:38:

Could you also provide /var/lib/rear/layout/{disklayout.conf,diskdeps.conf,disktodo.conf}? They give a better overview of what's happening.

You don't have to create gists for them as they are much smaller.

gdha commented at 2013-04-18 09:39:

was just to ask to same as @jhoekx (thanks!), but maybe also the output of df might be helpful

WautersPh commented at 2013-04-18 09:40:

Sure. You get it in a few minutes.

WautersPh commented at 2013-04-18 09:45:

[root@uxit400c rear]# cd /var/lib/rear/layout

[root@uxit400c layout]# ls
config  diskdeps.conf  disklayout.conf  disktodo.conf  lvm

[root@uxit400c layout]# cat diskdeps.conf
/dev/sda1 /dev/sda
/dev/sda2 /dev/sda
/dev/sdb1 /dev/sdb
/dev/sdc1 /dev/sdc
/dev/sdc2 /dev/sdc
/dev/sdd1 /dev/sdd
/dev/sde1 /dev/sde
/dev/sde2 /dev/sde
/dev/sdf1 /dev/sdf
/dev/sdg1 /dev/sdg
/dev/sdg2 /dev/sdg
/dev/sdh1 /dev/sdh
/dev/vg_root pv:/dev/mpath/mpath0p2
pv:/dev/mpath/mpath0p2 /dev/mpath/mpath0p2
/dev/mapper/vg_root-lv_root /dev/vg_root
/dev/mapper/vg_root-lv_tmp /dev/vg_root
/dev/mapper/vg_root-lv_opt /dev/vg_root
/dev/mapper/vg_root-lv_autosys /dev/vg_root
/dev/mapper/vg_root-lv_var /dev/vg_root
/dev/mapper/vg_root-lv_home /dev/vg_root
/dev/mapper/vg_root-lv_usr /dev/vg_root
/dev/mapper/vg_root-lv_tivoli /dev/vg_root
fs:/ /dev/mapper/vg_root-lv_root
fs:/tmp /dev/mapper/vg_root-lv_tmp
fs:/tmp fs:/
fs:/opt /dev/mapper/vg_root-lv_opt
fs:/opt fs:/
fs:/opt/CA /dev/mapper/vg_root-lv_autosys
fs:/opt/CA fs:/
fs:/opt/CA fs:/opt
fs:/var /dev/mapper/vg_root-lv_var
fs:/var fs:/
fs:/home /dev/mapper/vg_root-lv_home
fs:/home fs:/
fs:/usr /dev/mapper/vg_root-lv_usr
fs:/usr fs:/
fs:/usr/monitoring/Tivoli /dev/mapper/vg_root-lv_tivoli
fs:/usr/monitoring/Tivoli fs:/
fs:/usr/monitoring/Tivoli fs:/usr
fs:/boot /dev/mapper/mpath0p1
fs:/boot fs:/
swap:/dev/mapper/mpath1p1 /dev/mapper/mpath1p1
/dev/mapper/mpath1 /dev/sdb
/dev/mapper/mpath1 /dev/sdd
/dev/mapper/mpath1 /dev/sdf
/dev/mapper/mpath1 /dev/sdh
/dev/mapper/mpath1p1 /dev/mapper/mpath1
/dev/mapper/mpath0 /dev/sda
/dev/mapper/mpath0 /dev/sdc
/dev/mapper/mpath0 /dev/sde
/dev/mapper/mpath0 /dev/sdg
/dev/mapper/mpath0p1 /dev/mapper/mpath0
/dev/mapper/mpath0p2 /dev/mapper/mpath0

[root@uxit400c layout]# cat disklayout.conf
#disk /dev/cciss/c0d0 0
#disk /dev/sda 53687091200 msdos
#part /dev/sda 106896384 32256 primary boot /dev/sda1
#part /dev/sda 15726735360 106928640 primary lvm /dev/sda2
#disk /dev/sdb 161061273600 msdos
#part /dev/sdb 69684539904 32256 primary boot /dev/sdb1
#disk /dev/sdc 53687091200 msdos
#part /dev/sdc 106896384 32256 primary boot /dev/sdc1
#part /dev/sdc 15726735360 106928640 primary lvm /dev/sdc2
#disk /dev/sdd 161061273600 msdos
#part /dev/sdd 69684539904 32256 primary boot /dev/sdd1
#disk /dev/sde 53687091200 msdos
#part /dev/sde 106896384 32256 primary boot /dev/sde1
#part /dev/sde 15726735360 106928640 primary lvm /dev/sde2
#disk /dev/sdf 161061273600 msdos
#part /dev/sdf 69684539904 32256 primary boot /dev/sdf1
#disk /dev/sdg 53687091200 msdos
#part /dev/sdg 106896384 32256 primary boot /dev/sdg1
#part /dev/sdg 15726735360 106928640 primary lvm /dev/sdg2
#disk /dev/sdh 161061273600 msdos
#part /dev/sdh 69684539904 32256 primary boot /dev/sdh1
lvmdev /dev/vg_root /dev/mpath/mpath0p2 RMb70v-BUnQ-0Nhy-9Lpy-R8zd-s7Uh-19idnM 30716280
lvmgrp /dev/vg_root 32768 468 15335424
lvmvol /dev/vg_root lv_root 62 4063232
lvmvol /dev/vg_root lv_tmp 31 2031616
lvmvol /dev/vg_root lv_opt 46 3014656
lvmvol /dev/vg_root lv_autosys 15 983040
lvmvol /dev/vg_root lv_var 156 10223616
lvmvol /dev/vg_root lv_home 3 196608
lvmvol /dev/vg_root lv_usr 78 5111808
lvmvol /dev/vg_root lv_tivoli 62 4063232
fs /dev/mapper/vg_root-lv_root / ext3 uuid=d492b1d1-a56a-41df-9541-870f2e03c992 label= blocksize=4096 reserved_blocks=4% max_mounts=-1 check_interval=0d bytes_per_inode=4096 options=rw
fs /dev/mapper/vg_root-lv_tmp /tmp ext3 uuid=d69dc609-67c8-41ad-9da6-f04b81e43828 label= blocksize=4096 reserved_blocks=4% max_mounts=-1 check_interval=0d bytes_per_inode=4096 options=rw
fs /dev/mapper/vg_root-lv_opt /opt ext3 uuid=576be55e-789d-48d7-8387-f51f06237826 label= blocksize=4096 reserved_blocks=4% max_mounts=-1 check_interval=0d bytes_per_inode=4093 options=rw
fs /dev/mapper/vg_root-lv_autosys /opt/CA ext3 uuid=a0a70ce7-7e40-4ea8-9555-5976079397a5 label= blocksize=1024 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=4096 options=rw
fs /dev/mapper/vg_root-lv_var /var ext3 uuid=8bb95a92-d539-4343-a51d-133cc9ba1208 label= blocksize=4096 reserved_blocks=4% max_mounts=-1 check_interval=0d bytes_per_inode=4096 options=rw
fs /dev/mapper/vg_root-lv_home /home ext3 uuid=8d615552-8aa2-4171-916e-b6bf511d1b71 label= blocksize=1024 reserved_blocks=4% max_mounts=-1 check_interval=0d bytes_per_inode=4096 options=rw
fs /dev/mapper/vg_root-lv_usr /usr ext3 uuid=f4954e39-c2f3-419c-b86b-4b4fa2b605bd label= blocksize=4096 reserved_blocks=4% max_mounts=-1 check_interval=0d bytes_per_inode=4093 options=rw
fs /dev/mapper/vg_root-lv_tivoli /usr/monitoring/Tivoli ext3 uuid=d3f4b887-55cb-4cb9-929f-961ee6dd5d2d label= blocksize=4096 reserved_blocks=4% max_mounts=-1 check_interval=0d bytes_per_inode=4096 options=rw
fs /dev/mapper/mpath0p1 /boot ext3 uuid=547ccc1a-0cb9-46af-aef8-186faa7bb44b label=/boot blocksize=1024 reserved_blocks=4% max_mounts=-1 check_interval=0d bytes_per_inode=4094 options=rw
#swap /dev/mapper/mpath1p1 uuid= label=SWAP-mpath1p1
#multipath /dev/mapper/mpath1 /dev/sdb,/dev/sdd,/dev/sdf,/dev/sdh
#part /dev/mapper/mpath1 69684539904 unknown primary boot /dev/mapper/mpath1p1
multipath /dev/mapper/mpath0 /dev/sda,/dev/sdc,/dev/sde,/dev/sdg
part /dev/mapper/mpath0 106896384 unknown primary boot /dev/mapper/mpath0p1
part /dev/mapper/mpath0 15726735360 unknown primary lvm /dev/mapper/mpath0p2

[root@uxit400c layout]# cat disktodo.conf
done /dev/cciss/c0d0 disk
done /dev/sda disk
done /dev/sda1 part
done /dev/sda2 part
done /dev/sdb disk
done /dev/sdb1 part
done /dev/sdc disk
done /dev/sdc1 part
done /dev/sdc2 part
done /dev/sdd disk
done /dev/sdd1 part
done /dev/sde disk
done /dev/sde1 part
done /dev/sde2 part
done /dev/sdf disk
done /dev/sdf1 part
done /dev/sdg disk
done /dev/sdg1 part
done /dev/sdg2 part
done /dev/sdh disk
done /dev/sdh1 part
todo pv:/dev/mpath/mpath0p2 lvmdev
todo /dev/vg_root lvmgrp
todo /dev/mapper/vg_root-lv_root lvmvol
todo /dev/mapper/vg_root-lv_tmp lvmvol
todo /dev/mapper/vg_root-lv_opt lvmvol
todo /dev/mapper/vg_root-lv_autosys lvmvol
todo /dev/mapper/vg_root-lv_var lvmvol
todo /dev/mapper/vg_root-lv_home lvmvol
todo /dev/mapper/vg_root-lv_usr lvmvol
todo /dev/mapper/vg_root-lv_tivoli lvmvol
todo fs:/ fs
todo fs:/tmp fs
todo fs:/opt fs
todo fs:/opt/CA fs
todo fs:/var fs
todo fs:/home fs
todo fs:/usr fs
todo fs:/usr/monitoring/Tivoli fs
todo fs:/boot fs
done swap:/dev/mapper/mpath1p1 swap
done /dev/mapper/mpath1 multipath
done /dev/mapper/mpath1p1 part
todo /dev/mapper/mpath0 multipath
todo /dev/mapper/mpath0p1 part
todo /dev/mapper/mpath0p2 part

[root@uxit400c layout]# df -hP | column -t
Filesystem                      Size  Used  Avail  Use%  Mounted                 on
/dev/mapper/vg_root-lv_root     1.9G  349M  1.5G   20%   /
/dev/mapper/vg_root-lv_tmp      961M  201M  711M   23%   /tmp
/dev/mapper/vg_root-lv_opt      1.4G  330M  1023M  25%   /opt
/dev/mapper/vg_root-lv_autosys  465M  161M  281M   37%   /opt/CA
/dev/mapper/vg_root-lv_var      4.8G  301M  4.2G   7%    /var
/dev/mapper/vg_root-lv_home     93M   5.7M  83M    7%    /home
/dev/mapper/vg_root-lv_usr      2.4G  964M  1.3G   43%   /usr
/dev/mapper/vg_root-lv_tivoli   1.9G  296M  1.5G   17%   /usr/monitoring/Tivoli
/dev/mapper/mpath0p1            99M   64M   31M    68%   /boot
tmpfs                           32G   0     32G    0%    /dev/shm

jhoekx commented at 2013-04-18 09:52:

That looks normal.

Is it possible to get the debug log from a rear recover?

WautersPh commented at 2013-04-18 09:53:

Yes. I will reinitiate the "rear recover" and post the log file.

WautersPh commented at 2013-04-18 10:33:

2013-04-18 12:25:00 Relax-and-Recover 1.14 / Git
2013-04-18 12:25:00 Command line options: /bin/rear recover
2013-04-18 12:25:00 Using log file: /var/log/rear/rear-uxit400c.log
2013-04-18 12:25:00 Including /etc/rear/os.conf
2013-04-18 12:25:00 Including conf/Linux-i386.conf
2013-04-18 12:25:00 Including conf/GNU/Linux.conf
2013-04-18 12:25:00 Including /etc/rear/local.conf
2013-04-18 12:25:00 Including /etc/rear/rescue.conf
2013-04-18 12:25:00 Using build area '/tmp/rear.cNXPVCOSiGu4224'
mkdir: created directory `/tmp/rear.cNXPVCOSiGu4224/rootfs'
mkdir: created directory `/tmp/rear.cNXPVCOSiGu4224/tmp'
2013-04-18 12:25:00 Running recover workflow
2013-04-18 12:25:00 Running 'setup' stage
2013-04-18 12:25:00 Including setup/default/01_pre_recovery_script.sh
2013-04-18 12:25:00 Finished running 'setup' stage in 0 seconds
2013-04-18 12:25:00 Running 'verify' stage
2013-04-18 12:25:00 Including verify/default/02_cciss_scsi_engage.sh
2013-04-18 12:25:00 Engage SCSI on host /proc/driver/cciss/cciss0
2013-04-18 12:25:02 Including verify/default/02_translate_url.sh
2013-04-18 12:25:02 Including verify/default/03_translate_tape.sh
2013-04-18 12:25:02 Including verify/NETFS/default/05_check_NETFS_requirements.sh
2013-04-18 12:25:02 Skipping ping test
2013-04-18 12:25:02 Including verify/GNU/Linux/05_sane_recovery_check.sh
2013-04-18 12:25:02 Including verify/NETFS/default/07_set_backup_archive.sh
2013-04-18 12:25:02 Including verify/NETFS/default/08_start_required_daemons.sh
2013-04-18 12:25:02 Including verify/NETFS/default/09_set_readonly_options.sh
2013-04-18 12:25:02 Including verify/NETFS/default/10_mount_NETFS_path.sh
mkdir: created directory `/tmp/rear.cNXPVCOSiGu4224/outputfs'
2013-04-18 12:25:02 Mounting with 'mount -v -t nfs -o ro ks1.mux.isinfra.net:/nfsexports/rear/uxit400c /tmp/rear.cNXPVCOSiGu4224/outputfs'
mount: trying 10.91.6.201 prog 100003 vers 3 prot tcp port 2049
mount: trying 10.91.6.201 prog 100005 vers 3 prot udp port 4002
2013-04-18 12:25:02 Including verify/GNU/Linux/23_storage_and_network_modules.sh
find: /lib/modules/2.6.18-308.el5/kernel/drivers/xen: No such file or directory
find: /lib/modules/2.6.18-308.el5/weak-updates: No such file or directory
2013-04-18 12:25:02 Including verify/GNU/Linux/26_recovery_storage_drivers.sh
find: /tmp/rear.cNXPVCOSiGu4224/tmp/dev: No such file or directory
1,2d0
< be2iscsi
< bnx2i
4,8d1
< cxgb3i
< iscsi_tcp
< libcxgbi
< libiscsi2
< libiscsi_tcp
10,15d2
< mptbase
< mptctl
< scsi_dh
< scsi_dh_alua
< scsi_dh_emc
< scsi_dh_rdac
18,19d4
< scsi_transport_iscsi
< scsi_transport_iscsi2
21a7
> sr_mod
2013-04-18 12:25:02 NOTICE: Will do driver migration
2013-04-18 12:25:02 Including verify/NETFS/default/55_check_backup_archive.sh
2013-04-18 12:25:02 Calculating backup archive size
2013-04-18 12:25:02 Backup archive size is 744M (compressed)
2013-04-18 12:25:02 Finished running 'verify' stage in 2 seconds
2013-04-18 12:25:02 Running 'layout/prepare' stage
2013-04-18 12:25:02 Including layout/prepare/default/01_prepare_files.sh
2013-04-18 12:25:02 Including layout/prepare/GNU/Linux/10_include_partition_code.sh
2013-04-18 12:25:02 Including layout/prepare/GNU/Linux/11_include_lvm_code.sh
2013-04-18 12:25:02 Including layout/prepare/GNU/Linux/12_include_raid_code.sh
2013-04-18 12:25:02 Including layout/prepare/GNU/Linux/13_include_filesystem_code.sh
2013-04-18 12:25:02 Including layout/prepare/GNU/Linux/14_include_swap_code.sh
2013-04-18 12:25:02 Including layout/prepare/GNU/Linux/15_include_drbd_code.sh
2013-04-18 12:25:02 Including layout/prepare/GNU/Linux/16_include_luks_code.sh
2013-04-18 12:25:02 Including layout/prepare/GNU/Linux/17_include_hpraid_code.sh
2013-04-18 12:25:02 Including layout/prepare/default/20_recreate_hpraid.sh
2013-04-18 12:25:02 Including layout/prepare/GNU/Linux/21_load_multipath.sh
2013-04-18 12:25:02 Activating multipath
create: mpath0 (360060e80164ca90000014ca900000209)  HITACHI,OPEN-V
[size=50G][features=0][hwhandler=0][n/a]
\_ round-robin 0 [prio=4][undef]
\_ 2:0:0:0 sda 8:0   [undef][ready]
\_ 2:0:1:0 sdc 8:32  [undef][ready]
\_ 6:0:0:0 sde 8:64  [undef][ready]
\_ 6:0:1:0 sdg 8:96  [undef][ready]
create: mpath1 (360060e80164ca90000014ca90000020a)  HITACHI,OPEN-V
[size=150G][features=0][hwhandler=0][n/a]
\_ round-robin 0 [prio=4][undef]
\_ 2:0:0:1 sdb 8:16  [undef][ready]
\_ 2:0:1:1 sdd 8:48  [undef][ready]
\_ 6:0:0:1 sdf 8:80  [undef][ready]
\_ 6:0:1:1 sdh 8:112 [undef][ready]
2013-04-18 12:25:02 Including layout/prepare/default/25_compare_disks.sh
2013-04-18 12:25:02 Comparing disks.
2013-04-18 12:25:02 Disk configuration is identical, proceeding with restore.
2013-04-18 12:25:02 Including layout/prepare/default/30_map_disks.sh
2013-04-18 12:25:02 Including layout/prepare/default/31_remove_exclusions.sh
2013-04-18 12:25:02 Including layout/prepare/default/32_apply_mappings.sh
2013-04-18 12:25:02 Including layout/prepare/default/40_autoresize_disks.sh
2013-04-18 12:25:02 Including layout/prepare/default/50_confirm_layout.sh
2013-04-18 12:25:02 Including layout/prepare/default/51_list_dependencies.sh
2013-04-18 12:25:02 Including layout/prepare/default/52_exclude_components.sh
2013-04-18 12:25:02 Including layout/prepare/default/54_generate_device_code.sh
2013-04-18 12:25:02 Disk label for /dev/mapper/mpath0 detected as msdos.
2013-04-18 12:25:03 Including layout/prepare/default/55_finalize_script.sh
2013-04-18 12:25:03 Including layout/prepare/default/60_show_unprocessed.sh
2013-04-18 12:25:03 No code has been generated to restore device pv:/dev/mpath/mpath0p2 (lvmdev).
    Please add code to /var/lib/rear/layout/diskrestore.sh to manually install it or choose abort.
2013-04-18 12:25:06 Error detected during restore.
2013-04-18 12:25:06 Restoring backup of /var/lib/rear/layout/disklayout.conf
2013-04-18 12:25:06 ERROR: User chose to abort the recovery.
=== Stack trace ===
Trace 0: /bin/rear:245 main
Trace 1: /usr/share/rear/lib/recover-workflow.sh:29 WORKFLOW_recover
Trace 2: /usr/share/rear/lib/framework-functions.sh:79 SourceStage
Trace 3: /usr/share/rear/lib/framework-functions.sh:40 Source
Trace 4: /usr/share/rear/layout/prepare/default/60_show_unprocessed.sh:14 source
Message: User chose to abort the recovery.
===================
2013-04-18 12:25:06 Running exit tasks.
ks1.mux.isinfra.net:/nfsexports/rear/uxit400c umounted
rmdir: removing directory, /tmp/rear.cNXPVCOSiGu4224/outputfs
2013-04-18 12:25:06 Finished in 6 seconds
2013-04-18 12:25:06 Removing build area /tmp/rear.cNXPVCOSiGu4224
rmdir: removing directory, /tmp/rear.cNXPVCOSiGu4224
2013-04-18 12:25:06 End of program reached

jhoekx commented at 2013-04-18 10:39:

Sorry if it was not clear, I meant the -D version as a gist, not the -d...

WautersPh commented at 2013-04-18 11:13:

OK. I'm collecting the info and will post it asap.

WautersPh commented at 2013-04-18 11:31:

https://gist.github.com/WautersPh/064de48f6d4e53a3f462

jhoekx commented at 2013-04-18 13:36:

Ok, I can see the problem.

We don't translate the PV name correctly.

A workaround for now is, before you type rear restore, to change in /var/lib/rear/layout/disklayout.conf:

This line:

lvmdev /dev/vg_root /dev/mpath/mpath0p2 RMb70v-BUnQ-0Nhy-9Lpy-R8zd-s7Uh-19idnM 30716280

To this:

lvmdev /dev/vg_root /dev/mapper/mpath0p2 RMb70v-BUnQ-0Nhy-9Lpy-R8zd-s7Uh-19idnM 30716280

WautersPh commented at 2013-04-18 13:38:

Thank you for your investigations and your quick answer.
I will give it a try right now ...

WautersPh commented at 2013-04-18 13:57:

It is already better but I think there is still another remaining problem somewhere :

rear2

"0 logical volume(s) in volume group "vg_root" now active
An error occurred during layout recreation."

jhoekx commented at 2013-04-18 14:06:

Can you show the last 200 or so lines of the -D log of this restore? At least the pvcreate should be in it.

I guess it's related to the partitioning of the multipath device. Can you also choose option 3 in the screen you posted and show the output of parted -s /dev/mapper/mpath0 p?

WautersPh commented at 2013-04-18 14:08:

Yes I can :-)

Give me just a few minutes as I didn't run it with "-D" and I will restart it now.

WautersPh commented at 2013-04-18 14:33:

A bit strange but I get another error when doing a "rear -D recover" now (after the edition recommended above in the disklayout.conf file) : it proceeds further and start with restoration, but it ends with a failure as you can see in the following gits :

https://gist.github.com/WautersPh/065579d901b7ceb28bf9

WautersPh commented at 2013-04-18 14:55:

I've tried again and I get the same error (see the last gist).

Here is a screenshot :

rear3

xenlo commented at 2013-04-18 17:02:

Hi,
I'm working with WautersPh. So I also had a look on the issue.

After reading the debug output, it seems there is still an issue with the disklayout.conf.
The script is looking for a disk to install the grub in there but don't found any:

+++ grep '^disk ' /var/lib/rear/layout/disklayout.conf
+++ cut '-d ' -f2
++ disks=
++ [[ -n '' ]]
++ StopIfError 'Unable to find any disks'
++ ((  1 != 0  ))
++ Error 'Unable to find any disks'

Actually all the disk lines are commented,

RESCUE uxit400c:~ # grep '#disk ' /var/lib/rear/layout/disklayout.conf
#disk /dev/cciss/c0d0 0
#disk /dev/sda 53687091200 msdos
#disk /dev/sdb 161061273600 msdos
#disk /dev/sdc 53687091200 msdos
#disk /dev/sdd 161061273600 msdos
#disk /dev/sde 53687091200 msdos
#disk /dev/sdf 161061273600 msdos
#disk /dev/sdg 53687091200 msdos
#disk /dev/sdh 161061273600 msdos

And there is only direct access devices. In my opinion, we should expect a 'multipathed disk' like /dev/mapper/mpath0.
Am I right?

xenlo commented at 2013-04-18 17:36:

I restarted a new rear -D recover test after adding this following line to the disklayout.conf:

disk /dev/mapper/mpath0 53687091200 msdos

But without more sucess. It seems not a good idea to edit the disklayout too much manually:

+++ echo -e 'Creating partitions for disk /dev/mapper/mpath0 (msdos)'
+++ parted -s /dev/mapper/mpath0 mklabel msdos
Warning: Partition(s) on /dev/mapper/mpath0 are being used.
++ ((  1 == 0  ))
++ LogPrint 'An error occured during layout recreation.'
++ Log 'An error occured during layout recreation.'

full ouput on: https://gist.github.com/xenlo/5414617

jhoekx commented at 2013-04-19 05:57:

Looks like we will need to learn our grub install script to use multipath devices. @dagwieers what's best to use in that case, the multipath device itself or one of the paths (sda)?

For now you can chroot into /mnt/local and run grub from there.

gdha commented at 2013-04-19 06:16:

Multipath devices were to my knowledge not yet recognized properly as / and /boot within rear. We always tried to exclude these to protect the multipath devices as a matter of fact. However, times are changing and we see more and more multipath devices are the only devices on a Linux system (e.g. with blades). We also need to take in mind that cloning from internal disks to multipath devices are also coming in scope.

Guess we should consider this thoughts when going forward in our design.
@jhoekx @dagwieers I wonder if "friendly" names are the best approach? Perhaps we should consider to use only the uuid naming convention as these are persistent?

jhoekx commented at 2013-04-19 06:30:

I tend to explicitly name my multipath devices in multipath.conf. So on the systems where I tested multipath support, these names were stable. Naming is already quite messy in general and hard to get right in all distributions we care about at the same time (as shown by the first bug discovered here).

Not that I'm opposed to naming by uuid if we can get that to work reliably...

WautersPh commented at 2013-04-19 07:31:

Good morning everyone,

The situation is exactly what @gdha described and this is 100% what we are facing here.

The point is that we have to address this issue now. We have a few blades that will have a very critical role to play as of mid-May onwards. We also have a test blade (this is very new here) and we have a total freedom to perform every possible tests on it as you have seen in all my previous posts.

We are willing to help as much as we can (with the limitations of our - and certainly my - knowledge) but we would like to know if you see a possibility to address this quite rapidly.

What is your feeling ?

Thanks and regards.

Philippe

WautersPh commented at 2013-04-19 11:18:

Dears,

We managed to complete the restore by doing the following "grub" commands right after the last error reported :

RESCUE uxit400c:/mnt/local/boot/grub # grub
Probing devices to guess BIOS drives. This may take a long time.

GNU GRUB version 0.97 (640K lower / 3072K upper memory)

[ Minimal BASH-like line editing is supported. For the first word, TAB
lists possible command completions. Anywhere else TAB lists the possible
completions of a device/filename.]
grub> device (hd0) /dev/mapper/mpath0
device (hd0) /dev/mapper/mpath0
grub> root (hd0,0)
root (hd0,0)
Filesystem type is ext2fs, partition type 0x83
grub> setup (hd0)
setup (hd0)
Checking if "/boot/grub/stage1" exists... no
Checking if "/grub/stage1" exists... yes
Checking if "/grub/stage2" exists... yes
Checking if "/grub/e2fs_stage1_5" exists... yes
Running "embed /grub/e2fs_stage1_5 (hd0)"... 15 sectors are embedded.
succeeded
Running "install /grub/stage1 (hd0) (hd0)1+15 p (hd0,0)/grub/stage2 /grub/grub.conf"... succeeded
Done.
grub> quit
quit

RESCUE uxit400c:/mnt/local/boot/grub # reboot

We will document this and live with it (while hoping for a next release or "rear" to handle this automatically).

Kind regards.

Philippe

xenlo commented at 2013-04-19 14:47:

Good news, I found why we get a lvmdev line with a device /dev/mpath in place of a /dev/mapper/!

Even RedHat recommend to use the /dev/mapper/ as lvm PV (https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/DM_Multipath/multipath_logical_volumes.html), and even I provide well the /dev/mapper/ at the installation, after the reboot the LVM take the /dev/mpath device.
Actually it is due to a not enough restrictive filter by default in lvm.conf.

So I adapted the filter into the lvm.conf to reject /dev/mpath/mpath.*. Then I rebooted and pvs command provide me an nice output with /dev/mapper/.
Now the freshly re-generated disklayout.conf has the right line:

lvmdev /dev/vg_root /dev/mapper/mpath0p2 RMb70v-BUnQ-0Nhy-9Lpy-R8zd-s7Uh-19idnM 30716280

So now, that the root cause of the first issue is remove, let's hope that it will solve that first issue
I will try the restore right now. Stay tuned!

xenlo commented at 2013-04-19 15:44:

Great, as expected, the first issue is solved by using a right ffilter in the lvm.conf!

RESCUE uxit400c:~ # rear -D recover
[...]
Comparing disks.
Disk configuration is identical, proceeding with restore.
Start system layout restoration.
Creating partitions for disk /dev/mapper/mpath0 (msdos)
Creating LVM PV /dev/mapper/mpath0p2
  0 logical volume(s) in volume group "vg_root" now active
Restoring LVM VG vg_root
Creating ext3-filesystem / on /dev/mapper/vg_root-lv_root
Mounting filesystem /
Creating ext3-filesystem /tmp on /dev/mapper/vg_root-lv_tmp
Mounting filesystem /tmp
Creating ext3-filesystem /opt on /dev/mapper/vg_root-lv_opt
Mounting filesystem /opt
Creating ext3-filesystem /opt/CA on /dev/mapper/vg_root-lv_autosys
Mounting filesystem /opt/CA
Creating ext3-filesystem /var on /dev/mapper/vg_root-lv_var
Mounting filesystem /var
Creating ext3-filesystem /home on /dev/mapper/vg_root-lv_home
Mounting filesystem /home
Creating ext3-filesystem /usr on /dev/mapper/vg_root-lv_usr
Mounting filesystem /usr
Creating ext3-filesystem /usr/monitoring/Tivoli on /dev/mapper/vg_root-lv_tivoli
Mounting filesystem /usr/monitoring/Tivoli
Creating ext3-filesystem /boot on /dev/mapper/mpath0p1
Mounting filesystem /boot
Disk layout created.
Restoring from '/tmp/rear.vDbgolAFGRc4120/outputfs/uxit400c/backup.tar.gz'
Restored 1929 MiB [avg 65869 KiB/sec]OK
Restored 1929 MiB in 31 seconds [avg 63744 KiB/sec]
Updated initramfs with new drivers for this system.
Installing GRUB boot loader
ERROR: Unable to find any disks
Aborting due to an error, check /var/log/rear/rear-uxit400c.log for details
Terminated

So only the second issue with grub install is still there!
And as Philippe told, this can be solve at least manually by the following grub commands:

grub> device (hd0) /dev/mapper/mpath0
grub> root (hd0,0)
grub> setup (hd0)

I don't know exactly why the grub-install command does not work:

RESCUE uxit400c:~ # grub-install --root-directory=/mnt/local/boot/ /dev/mapper/mpath0
Probing devices to guess BIOS drives. This may take a long time.
/dev/mapper/mpath0 does not have any corresponding BIOS drive.

Anyway, I will have a look on monday how you proceed in the ReaR script to restore the Grub and see if I don't have a change proposition.

gdha commented at 2013-04-22 06:25:

Great Job! OK, to complete the process I think that if multipath devices are used as boot devices we should detect this in the prepration phase (not that difficult), and then we should check if the necessary lvm.conf filters are applied, and the multipath.conf fullfills to the minimum rules for multipathing. Then, we're sure that in recovery mode we have the correct configuration settings. Furthermore, the prep script should add the necessary multipath commands, such as multipathd, multipath and dm-multipath kernel module.
Perhaps, check out also http://lists.relax-and-recover.org/pipermail/rear-users/2013-February/002642.html for some more details on multipath issues.

gdha commented at 2013-10-04 12:11:

this case is somehow linked to issue #305

Reitrok commented at 2015-02-12 13:38:

Hello,
i know this issue is known and has a workaround but i´m facing a similar problem and have some additional findings.

I´m using SLES 11 SP3 on a HP Blade Server with SAN disks only.
after the "rear -v mkbackup" with the following local.conf:
OUTPUT=ISO
BACKUP=NETFS
BACKUP_URL=nfs://xxxxxxxxxxxxxxxxx
ONLY_INCLUDE_VG=( 'VG00' )
AUTOEXCLUDE_MULTIPATH=n
BOOT_OVER_SAN=y

the disklayout.conf file looks like this:
#disk /dev/sda 5368709120 msdos
#part /dev/sda 1077511680 2097152 primary boot /dev/sda1
#part /dev/sda 4289097728 1079611392 primary none /dev/sda2
#disk /dev/sdb 26843545600
#disk /dev/sdc 5368709120 msdos
#part /dev/sdc 1077511680 2097152 primary boot /dev/sdc1
#part /dev/sdc 4289097728 1079611392 primary none /dev/sdc2
#disk /dev/sdd 26843545600
#disk /dev/sde 5368709120 msdos
#part /dev/sde 1077511680 2097152 primary boot /dev/sde1
#part /dev/sde 4289097728 1079611392 primary none /dev/sde2
#disk /dev/sdf 26843545600
#disk /dev/sdg 5368709120 msdos
#part /dev/sdg 1077511680 2097152 primary boot /dev/sdg1
#part /dev/sdg 4289097728 1079611392 primary none /dev/sdg2
#disk /dev/sdh 26843545600
....
multipath /dev/mapper/mpathb /dev/sdb,/dev/sdd,/dev/sdf,/dev/sdh
multipath /dev/mapper/mpatha /dev/sda,/dev/sdc,/dev/sde,/dev/sdg
part /dev/mapper/mpatha 1077511680 unknown primary boot /dev/mapper/mpatha_part1
part /dev/mapper/mpatha 4289097728 unknown primary none /dev/mapper/mpatha_part2

When i restore from this iso without correcting the "unknown" start sectors for the mpatha/b partitions the system wont boot. I´m setting up grub manualy after restore.

I´m writing cause i´m not shure if you mentioned that the start sectors for the partitioning are not correctly collected during backup.
Will there be a fix for the next release?

Kind regards
Daniel

gdha commented at 2015-02-16 18:42:

@DanielBerglar I rather close this issue now as rear-1.17 will be released soon. If your problem still occurs I would propose to open a new issue for it (as now it is nested somewhere deep in another kind of issue).

[Export of Github issue for rear/rear.]