#1899 Issue closed: Rear multipath issue¶
Labels: support / question, fixed / solved / done
samurdhi opened issue at 2018-08-14 18:20:¶
Relax-and-Recover (ReaR) Issue Template¶
Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):
- ReaR version ("/usr/sbin/rear -V"):
[root@localhost ~]# /usr/sbin/rear -V Relax-and-Recover 2.00 / Git [root@localhost ~]#
- OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"):
[root@localhost ~]# cat /etc/rear/os.conf OS_VENDOR=RedHatEnterpriseServer OS_VERSION=7 [root@localhost ~]#
- ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
OUTPUT=ISO
OUTPUT_URL=nfs://122.15.40.67/opt/tftpboot/rear
BACKUP=NETFS
BACKUP_URL=nfs://122.15.40.67/opt/tftpboot/rearbackup
AUTOEXCLUDE_MULTIPATH=n
NETFS_KEEP_OLD_BACKUP_COPY=1
BACKUP_PROG_EXCLUDE=( ${BACKUP_PROG_EXCLUDE[@]} '/mnt/*' '/test01/*' '/test02/*' )
-
Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR):
Hardware Server -
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
x86 compatible -
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
BIOS or UEFI -
Description of the issue (ideally so that others can reproduce it):
This is a first time I'm going to use REAR as a backup system in my company. I have tested with servers having only internal disks, rear work well with that. But this time I'm going to create recovery iso for following server, it has both internal disks(sda) and SAN multipath disks(mpatha, mpathb, mpathc).
mpatha has been used to create a volume group (sdcsit26-openvvg)
mpathb,mpathac used to create a volume group (openvvg)
Those multipath LVM VGs are used to create LV and mounted on the server. I want to know that doest rear support recovery of multipath LVMs ? and I believe I dont need to take backup of those LVMs because those are created using LUN disks, is that true ?
[root@srv ~]# pvs PV VG Fmt Attr PSize PFree /dev/mapper/mpatha sdcsit26-openvvg lvm2 a-- <1.40t 0 /dev/mapper/mpathb openvvg lvm2 a-- <500.00g 0 /dev/mapper/mpathc openvvg lvm2 a-- <500.00g <100.00g /dev/sda2 rootvg lvm2 a-- <557.88g 470.88g [root@srv ~]# vgs VG #PV #LV #SN Attr VSize VFree openvvg 2 2 1 wz--n- 999.99g <100.00g rootvg 1 5 0 wz--n- <557.88g 470.88g sdcsit26-openvvg 1 1 0 wz--n- <1.40t 0 [root@srv ~]# [root@srv ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert openv_bak openvvg swi-a-s--- 400.00g openvlv 35.99 openvlv openvvg owi-aos--- <500.00g homelv rootvg -wi-ao---- 5.00g rootlv rootvg -wi-ao---- 23.99g swaplv rootvg -wi-ao---- 32.00g tmplv rootvg -wi-ao---- 10.00g varlv rootvg -wi-ao---- 16.00g sdcsit26-openvlv sdcsit26-openvvg -wi-ao---- <1.40t [root@srv ~]# df -kh Filesystem Size Used Avail Use% Mounted on /dev/mapper/rootvg-rootlv 24G 1.4G 23G 6% / devtmpfs 63G 0 63G 0% /dev tmpfs 63G 0 63G 0% /dev/shm tmpfs 63G 67M 63G 1% /run tmpfs 63G 0 63G 0% /sys/fs/cgroup /dev/sda1 1016M 149M 867M 15% /boot /dev/mapper/sdcsit26--openvvg-sdcsit26--openvlv 1.4T 33M 1.4T 1% /usr/openv_sync /dev/mapper/rootvg-homelv 5.0G 55M 5.0G 2% /home /dev/mapper/rootvg-tmplv 10G 44M 10G 1% /tmp /dev/mapper/rootvg-varlv 16G 547M 16G 4% /var /dev/mapper/openvvg-openvlv 500G 99G 401G 20% /usr/openv tmpfs 13G 0 13G 0% /run/user/36 tmpfs 13G 0 13G 0% /run/user/9697
- Work-around, if any:
wsdsd
jsmeix commented at 2018-08-15 08:27:¶
@samurdhi
you use Relax-and-Recover 2.00 which is meanwhile a bit old in
particular
because there have been many enhancements and fixes regarding
multipath
since ReaR 2.00 was released in January 2017, see
https://raw.githubusercontent.com/rear/rear/master/doc/rear-release-notes.txt
Could you try out our current ReaR upstream GitHub master code
so that you and we at ReaR upstream use the same code
which makes debugging easier for us when there are issues.
To use our current ReaR upstream GitHub master code do the following:
Basically "git clone" it into a separated directory and then
configure and run ReaR from within that directory like:
# git clone https://github.com/rear/rear.git # mv rear rear.github.master # cd rear.github.master # vi etc/rear/local.conf # usr/sbin/rear -D mkbackup
Note the relative paths "etc/rear/" and "usr/sbin/".
I recommend to use our current ReaR upstream GitHub master code
because that is the only place where we at ReaR upstream fix bugs.
I.e. bugs in older ReaR versions are not fixed by us at ReaR upstream.
I am not a multipath user and I have no personal experience with it
so that I cannot really help with multipath issues.
Nevertheless I try as far as I can guess how things work:
When your server has both internal disks and SAN multipath disks
then - as far as I see - by default via AUTOEXCLUDE_MULTIPATH=y
in usr/share/rear/conf/default.conf multipath disks are excluded
because in general SAN storage devicers are normally not meant
to be recovered by ReaR, see
https://raw.githubusercontent.com/rear/rear/master/doc/user-guide/06-layout-configuration.adoc
I think normally you have a separated recovery tool for your SAN
storage
e.g. a specific recovery tool from the manufacturer or vendor
of your particular SAN storage.
I think things might go even terribly wrong on your SAN storage
if "rear recover" that runs locally on one single machine would
do partitioning, creating filesystems, and things like that
on your SAN storage devices.
Perhaps @schabrolles who is a multipath user could have a look here
because likely he can answer your question much better than I can do.
samurdhi commented at 2018-08-15 13:11:¶
@jsmeix
Thanks you for the reply and it's very helpful. Actually what I need to
know is that I don't want to backup those multipath disks but after
recovering the server I want them to present in the server.
jsmeix commented at 2018-08-16 12:12:¶
I have no personal experience with SAN and multipath
so that I cannot really help with such issues.
Nevertheless I try as far as I know how things should work:
During "rear recover" the backup of the files of the original system
gets restored onto the replacement system so that in particular
all config files of the original system get restored and afterwards
the initrd gets recreated and the bootloader gets reinstalled.
Accordingly after "rear recover" finished when the recreated replacement
system
is rebooted it should boot and start up in the same way as the original
system did
so that in particular the recreated replacement system should be able to
access
the SAN storage in the same way as the original system did - provided
the
replacement system is fully compatible with the original system,
cf. the section "Fully compatible replacement hardware is needed" in
https://en.opensuse.org/SDB:Disaster_Recovery
In practice things are more complicated because there are certain kind
of
hardware specific IDs used on a system like filesystem UUIDs or MAC
addresses
and things like that so that the restored config files of the original
system could
contain some old hardware specific IDs of the original system that do
not match
the new hardware specific IDs of the replacement hardware.
Therefore during the "finalize" stage of "rear recover" ReaR adapts some
known
kind of hardware specific IDs in some known config files so that usually
things
should "just work" after "rear recover" on the recreated replacement
system
in the same way as on the original system.
For some details how that usually works but could fail under special
conditions
in particular regarding the bootloader see the initial description in
https://github.com/rear/rear/issues/1854
and in particular regarding config files like etc/fstab see
https://github.com/rear/rear/pull/1758
Accordingly there is no guarantee that all will always "just work" after
"rear recover",
cf. the section "Inappropriate expectations" and its sub-sections
"Disaster recovery does not just work" and
"No disaster recovery without testing and continuous validation" in
https://en.opensuse.org/SDB:Disaster_Recovery
samurdhi commented at 2018-08-16 12:59:¶
Should I remove AUTOEXCLUDE_MULTIPATH=n ?
jsmeix commented at 2018-08-16 13:50:¶
In general I would start with the default settings
(because one can assume the defaults are usually right)
and see how things work by default unless I know already
in advance that in my particular case the defaults are not right.
@samurdhi
in the end you need to test how it works with your particular original
system
in your particular environment on your replacement hardware, see
https://en.opensuse.org/SDB:Disaster_Recovery
jsmeix commented at 2018-08-16 14:25:¶
@schabrolles
in usr/share/rear/conf/examples/RHEL7-PPC64LE-Mulitpath-PXE-GRUB.conf
there is (excerpt)
AUTOEXCLUDE_MULTIPATH=n BOOT_OVER_SAN=y
Could you explain why AUTOEXCLUDE_MULTIPATH=n
(and BOOT_OVER_SAN=y) must be used in this particular example
in contrast to AUTOEXCLUDE_MULTIPATH=y in default.conf
because the matching issue
https://github.com/rear/rear/pull/1339
mentiones neither SAN nor multipath but is only about
PXE based on GRUB2 booting on PPC64/PPC64LE.
I guess BOOT_OVER_SAN=y requires AUTOEXCLUDE_MULTIPATH=n
because when the system should boot from a SAN device with multipath
the "SAN via multipath devices" should not be autoexcluded but I
wonder
what happens when SAN is used without multipath?
I wonder why and how PXE boot is interconnected with BOOT_OVER_SAN
and multipath or is the connection in this example config file only by
chance?
I guess that also a system with a local harddisk (i.e. no SAN)
could be booted via PXE (because I think PXE boot is not only
for thin clients) or do I misunderstand something here?
schabrolles commented at 2018-08-17 07:05:¶
@jsmeix,
You are right, there is no direct link between Multipath/Boot_on_SAN
and PXE (even on POWER).
I use them in my example because PowerVM LPARs usually need Multipath
(90% of the case in my point of view)
PowerVM LPAR with single VIOS (not common) => nothing special
PowerVM LPAR with DUAL VIOS (HA configuration) => NEED multipath for
dual disk access (vscsi over 2 VIOS) or NPIV (Fiber Channel
virtualization)
Note: NPIV (which is quite popular) needs also BOOT_ON_SAN
jsmeix commented at 2018-08-17 09:48:¶
@schabrolles
thanks for your explanation!
Accordingly I added some more explanatory comments to
ttps://github.com/rear/rear/blob/master/usr/share/rear/conf/examples/RHEL7-PPC64LE-Mulitpath-PXE-GRUB.conf
via
https://github.com/rear/rear/commit/e75b86a7d0cc5d043e0712f0d9b6846797167525
gdha commented at 2018-09-05 11:15:¶
@samurdhi is this solved?
samurdhi commented at 2018-09-05 13:20:¶
@gdha
Yes, Gratien,
It's resolved. Thank you so much for the support.
[Export of Github issue for rear/rear.]