#882 Issue `closed`: HP Superdome-X servers using multipath an booting from SAN not backing up de ROOT filesystem!¶

Labels: enhancement, fixed / solved / done

didacog opened issue at 2016-06-16 12:36:¶

Relax-and-Recover (rear) Issue Template¶

Please fill in the following items before submitting a new issue:

rear version (/usr/sbin/rear -V):
Relax-and-Recover 1.17.2 / Git
OS version (cat /etc/rear/os.conf or lsb_release -a):
LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 6.6 (Santiago)
Release: 6.6
Codename: Santiago
rear configuration files (cat /etc/rear/site.conf or cat /etc/rear/local.conf):

    DRLM_NAME=sdomeX01
    OUTPUT=PXE
    OUTPUT_PREFIX=$OUTPUT
    OUTPUT_PREFIX_PXE=$DRLM_NAME/$OUTPUT_PREFIX
    OUTPUT_URL=nfs://192.168.1.100/DRLM/STORE/${DRLM_NAME}
    BACKUP=NETFS
    BACKUP_URL=nfs://192.168.1.100/DRLM/STORE/${DRLM_NAME}
    NETFS_PREFIX=BKP
    EXCLUDE_VG=( 'vgDATA' )
    SSH_ROOT_PASSWORD=rear
    USING_UEFI_BOOTLOADER=1
    AUTOEXCLUDE_MULTIPATH=n
    BOOT_OVER_SAN=y

Brief description of the issue

We are playing with new HP Superdome-X servers. These brand new servers are using multipath an booting from SAN.

On disk layout collection never gets info from root filesystem (/) but for other partitions in same device collects all info with no problems. The problem is that is not backing up de ROOT filesystem!

There is an issue with this file: /usr/share/rear/layout/save/GNU/Linux/23_filesystem_layout.sh

this line:

read_filesystems_command="$findmnt_command **-nrv** -o SOURCE,TARGET,FSTYPE,OPTIONS -t $supported_filesystems"

# findmnt command looks on  /proc/self/mountinfo by default and gets this info of (/):

$ cat /proc/self/mountinfo | grep ext3
21 1 253:12 / / rw,relatime - ext3 /dev/mapper/**360002ac000000000000000040001a624p4** rw,errors=continue,user_xattr,acl,barrier=1,data=ordered
23 21 253:10 / /boot rw,relatime - ext3 /dev/mapper/vvbootp2 rw,errors=continue,user_xattr,acl,barrier=1,data=ordered

In this case is informing the device WWN from SAN instead of the mpath alias name used in all rear scripts.

# mount command uses /etc/mtab by default gets the correct device info:

$ cat /etc/mtab | grep ext3
/dev/mapper/**vvbootp4** / ext3 rw 0 0
/dev/mapper/vvbootp2 /boot ext3 rw 0 0

Work-around, if any

Proposal:

/usr/share/rear/layout/save/GNU/Linux/23_filesystem_layout.sh

Change this line:

read_filesystems_command="$findmnt_command **-nrv** -o SOURCE,TARGET,FSTYPE,OPTIONS -t $supported_filesystems"

to:

read_filesystems_command="$findmnt_command **-mnrv** -o SOURCE,TARGET,FSTYPE,OPTIONS -t $supported_filesystems"

This will force findmnt command to use /etc/mtab instead of /proc/self/mountinfo and provide the correct info resulting in the correct disklayout file:

fs /dev/mapper/**vvbootp4** / ext3 uuid=2cf46bd1-892b-4154-8f87-c6fc79fcc2a5 label=HANA_ROOT blocksize=4096 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=16381 default_mount_options=user_xattr,acl options=rw

I've tested it and now is getting the correct info, but prior to sending a pull request I would ask if you guys have any concern about this modification.

Kind Regards!

Didac

jsmeix commented at 2016-06-16 12:50:¶

The code that uses the findmnt command is from me
therefore I assign the issue to me.

I do not run RHEL (and I do not know what is special on RHEL)
therefore I assign the issue also to @gdha

@didacog
simply do a GitHub Pull Request (regardless how good your code
currently is) so that we have something real to discuss it
and so that I can test if it still works on SLES with your changes.
Please use the current rear GitHub master code
as base for your Pull Request.

@didacog
the code snippets in your comment avobe are very hard to read
because of the automated formatting in GitHub.
For code snippets I enclose them
in HTML

and

tags which works o.k.
for me but one has to write < as HTML <
Perhaps there is something even better how to
specify preformatted texts in GitHub?

didacog commented at 2016-06-16 13:01:¶

Hi @jsmeix @gdha,

Ok I will prepare the pull request. this is the code to change, just one findmnt option:

/usr/share/rear/layout/save/GNU/Linux/23_filesystem_layout.sh
Change this line:
read_filesystems_command="$findmnt_command -nrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t $supported_filesystems"
to:
read_filesystems_command="$findmnt_command -mnrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t $supported_filesystems"

I will also check if any modification needed for this other issue:

2016-06-16 13:22:34.103712191 Could not determine start of partition vvbootp1.
2016-06-16 13:22:34.111206812 Could not determine start of partition vvbootp2.
2016-06-16 13:22:34.118799098 Could not determine start of partition vvbootp3.
2016-06-16 13:22:34.126096733 Could not determine start of partition vvbootp4.

resulting in /var/lib/rear/layout/disklayout.conf:

part /dev/mapper/vvboot 536870912 unknown rear-noname boot /dev/mapper/vvbootp1
part /dev/mapper/vvboot 536870912 unknown rear-noname none /dev/mapper/vvbootp2
part /dev/mapper/vvboot 2147483648 unknown rear-noname none /dev/mapper/vvbootp3
part /dev/mapper/vvboot 271623127040 unknown rear-noname none /dev/mapper/vvbootp4

And provide a pull request with all modifications needed.

Thanks!

Didac

jsmeix commented at 2016-06-16 13:03:¶

I changed the initial comment to
use HTML

and

tags
and now I can better read it.

On very first glance I do not like to

force findmnt command to use /etc/mtab instead of /proc/self/mountinfo

because think using /etc/mtab is an indirection (cf. RFC1925 6a)
that can cause whatever unexpected failures instead of
asking the kernel directly (via /proc/self/mountinfo).

@didacog
nevertheless do your Pull Request.

jsmeix commented at 2016-06-16 13:17:¶

On my SLES12-SP1 system with its default btrfs structure
the output of the proposed new "findmnt -mnrv ..." is the same
as the output of the currently used "findmnt -nrv ..." so that
from my current point of view the proposed change seems
to be o.k. (i.e. it seems to cause no regression for SLE).

What I did:

# cat /etc/os-release 
NAME="SLES"
VERSION="12-SP1"
VERSION_ID="12.1"
PRETTY_NAME="SUSE Linux Enterprise Server 12 SP1"
...
# /usr/bin/findmnt -mnrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs >/tmp/findmnt-mnrv
# /usr/bin/findmnt -nrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs >/tmp/findmnt-nrv
# diff -s /tmp/findmnt-nrv /tmp/findmnt-mnrv
Files /tmp/findmnt-nrv and /tmp/findmnt-mnrv are identical

jsmeix commented at 2016-06-16 13:24:¶

Nevertheless I need to inspect what the actual difference is
between /etc/mtab which is a link to /proc/self/mounts
versus /proc/self/mountinfo

didacog commented at 2016-06-16 13:25:¶

@jsmeix you are really fast! ;)

I think this will be the same behaviour as using mount command if findmnt is not present, and should work.

I'm now deeping with the other issue with partition layouts and after that I will provide a pull request.

Thanks!

didacog commented at 2016-06-16 13:30:¶

@jsmeix i found this post regarding /proc/self/mounts vs /proc/self/mountinfo:

http://serverfault.com/questions/581178/proc-self-mountinfo-per-mount-options-vs-per-super-options/581180

jsmeix commented at 2016-06-16 13:33:¶

If it is really the exact same behaviour as the old mount command
I could not accept it because I need new findmnt features to deal
correctly with btrfs subvolumes.

See also my comment regarding "man 8 mount"
in 23_filesystem_layout.sh but this comment only
tells to use findmnt instead of mount so that
"findmnt -m..." could be the right thing.

On the other hand perhaps using "findmnt -m..."
might be not an actually right future-proof solution
but merely some kind of workaround that only hides
to deal correctly with the actual root cause?

jsmeix commented at 2016-06-16 13:40:¶

@didacog
when you are currently "playing with new HP Superdome-X servers"
could you also test if your changes also work with SLE12-SP1
and its default btrfs subvolume structure?

Obviously I cannot do such a test myself because I do not
have a HP Superdome-X server to play around ;-)

Such a test would help a lot to ensure that also btrfs subvolumes
on RHEL still work on that new hardware or to reveal the
actual root cause (e.g. something like better general support
when the root filesystem is on SAN via multipath.

I never tested rear myself with SAN and multipath.

I blindly guess that general support when the root filesystem
is on SAN via multipath might be currently poor in rear.

jsmeix commented at 2016-06-16 13:49:¶

@didacog
many thanks for the link in
https://github.com/rear/rear/issues/882#issuecomment-226485148

I do not yet fully understand what is meant in
http://serverfault.com/questions/581178/proc-self-mountinfo-per-mount-options-vs-per-super-options/581180
with (excerpt)

/proc/self/mountinfo is the most authoritative source to check your mounts.
/proc/mounts is a deprecated source to check the status of your mounts.
...
 /proc/mounts symlinks to /proc/self/mounts

As far as I understand it this means that
/proc/self/mounts (and its symlinks like /proc/mounts or /etc/mtab)
are deprecated sources to check the status of your mounts.

Accordingly it seems "findmnt -m..."
is not an actually right future-proof solution.

didacog commented at 2016-06-16 13:53:¶

@jsmeix

I think that with the root partition on mpath device, the problem is when kernel loading the root partiton booting from SAN, this partition has not yet the mpath alias defined in kernel and keeps its WWN name on the /proc/self/mountinfo file.

Maybe an exeption when booting from SAN? the problem is that if not properly identified with this name, root filesystem is not present in disklayout file and does not backup of (/).

I will figure out other solutions and use this as workarround in the meantime, and if is possible i will test with SLES, but I cannot promise because those SuperdomeX are not mine :P

didacog commented at 2016-06-16 14:03:¶

@jsmeix This case is not with Boot from SAN on SuperdomeX but same output from findmnt command with -m option with btrfs subvolumes on SLES11 SP3:

$ findmnt -nrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t btrfs
/dev/sda4 / btrfs rw,relatime,space_cache
/dev/sda4 /srv btrfs rw,relatime,space_cache
/dev/sda4 /opt btrfs rw,relatime,space_cache
/dev/sda4 /tmp btrfs rw,relatime,space_cache
/dev/sda4 /var/spool btrfs rw,relatime,space_cache
/dev/sda4 /var/run btrfs rw,relatime,space_cache
/dev/sda4 /var/log btrfs rw,relatime,space_cache
/dev/sda4 /var/crash btrfs rw,relatime,space_cache
/dev/sda4 /var/tmp btrfs rw,relatime,space_cache

$ findmnt -mnrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t btrfs
/dev/sda4 / btrfs rw,relatime,space_cache
/dev/sda4 /srv btrfs rw,relatime,space_cache
/dev/sda4 /opt btrfs rw,relatime,space_cache
/dev/sda4 /tmp btrfs rw,relatime,space_cache
/dev/sda4 /var/spool btrfs rw,relatime,space_cache
/dev/sda4 /var/run btrfs rw,relatime,space_cache
/dev/sda4 /var/log btrfs rw,relatime,space_cache
/dev/sda4 /var/crash btrfs rw,relatime,space_cache
/dev/sda4 /var/tmp btrfs rw,relatime,space_cache

Also I've seen that /etc/mtab is a softlink to /proc/self/mounts in SLES but not in RHEL.

SLES:
$ ls -l /etc/mtab
lrwxrwxrwx 1 root root 17 Feb 15 13:47 /etc/mtab -> /proc/self/mounts

RHEL6:
$ ls -l /etc/mtab
-rw-r--r-- 1 root root 990 Jun 16 13:52 /etc/mtab

Maybe we need to use a different approach for each OS.

jsmeix commented at 2016-06-16 14:20:¶

In general I am agianst using different approaches
that depend on the Linux distrubution and version
because this is again an indirection (RFC 1925 6a)
that leads to weird unexpected failures because such
code just breaks if on an older Linux distrubution/version
newer software is installed so that the older distrubution
actually behaves like a newer distrubution.

In contrast I prefer to use different approaches
that depend on real existing conditions in the
current system.

I.e. I am against code like

case "$OS_VENDOR" in
    (SUSE_LINUX)
        do_this
        ;;
    (RedHatEnterpriseServer)
        do_that
        ;;
    (*)
        do_default
        ;;
esac

In contrast I prefer code like

if CONDITION ; then
   do_this
else
    do_that
fi

where CONDITION is something real that can be directly
tested on the current system and not something vague
like a version string.

For example I tried to implement my btrfs support code
so that it works generically on SLE and RHEL without
testing things like "$OS_VENDOR" and "$OS_VERSION".

jsmeix commented at 2016-06-16 14:30:¶

@didacog
note that in layout/save/GNU/Linux/23_filesystem_layout.sh
I use findmnt at several places:

First to make "fs" entries in disklayout.conf.

Afterwards to make "btrfs..." entries in disklayout.comf.

For the first usage (make "fs" entries) see my comment

# The only difference is that the traditional mount command output has the list of options in parenthesis.

so that here - at last for now - using "findmnt -m"
should be o.k.

In contrast for the other usages (make "btrfs..." entries)
the traditional mount format is insufficient, see my
comment in 23_filesystem_layout.sh and I try to
work aound that by awkward "grepping" in /etc/fstab
which is not a reliably working solution.

Therefore to make make "btrfs..." entries I assume
"findmnt -m" cannot be used - but I did not yet chek it.

didacog commented at 2016-06-16 15:26:¶

@jsmeix

Ok, maybe tomorrow I can try to run a backup with this modified on one of the development SLES servers with btrfs subvolumes and check if working ok with findmnt -m.

I guess this should work because, as you can see on the SLES outputs I provided are the same and the origin file format Is very different, but anyway, tomorrow I would clear those doubts :P

jsmeix commented at 2016-06-17 07:33:¶

@didacog
very many thanks in advance for your testing.
This is a very valuabe cuntribution to rear!

According to what I wrote above in https://github.com/rear/rear/issues/882#issuecomment-226502639
I also think that
at least for now using (deprecated) "findmnt -m"
for making "fs" entries in disklayout.conf should be o.k.
(i.e. adding '-m' for the 'read_filesystems_command')
while in contrast
I think that one cannot use "findmnt -m"
for making "btrfsmountedsubvol" entries in disklayout.conf
(i.e. no '-m' for the 'read_mounted_btrfs_subvolumes_command').

@didacog
because I have basically no knowledge about SAN and multipath
I have a basic question regarding what you wrote in your
initial comment https://github.com/rear/rear/issues/882#issue-160650954

device WWN from SAN instead of the mpath alias name

Colud you explain (or provide an URL to explanatory info)
what 'device WWN from SAN' versus 'mpath alias name'
is about?

Via quick "Googling" I found
http://www.datadisk.co.uk/html_docs/redhat/rh_multipathing.htm
but that talks about "WWID".
Is "WWID" the same as "WWN"?

Perhaps the actually right future-proof solution is
to implement in rear a function that gets as input a
'WWID' or 'WWN' and outputs the matching 'mpath alias name'
so that we could still use findmnt without '-m'.

Must each 'WWID' or 'WWN' have exactly one
associated 'mpath alias name' in any case
or could there be none or several 'mpath alias names'
for one 'WWID' or 'WWN'?

Perhaps there is already some ready-to-use command line tool
in the system to find the matching 'mpath alias name'
because I assume when "findmnt -m" outputs
the 'mpath alias name' it may use such kind of mapping
function between WWID or WWN and mpath alias name?

didacog commented at 2016-06-17 09:27:¶

@jsmeix

I've tested with SLES and here my results:

...
SLES11 SP3 (rear log):
2016-06-17 10:57:04.940394305 Begin saving filesystem layout
2016-06-17 10:57:04.943144580 Saving filesystem layout (using the findmnt command).
/usr/share/rear/layout/save/GNU/Linux/23_filesystem_layout.sh: line 123: dosfslabel: command not found
findmnt: unknown column: FSROOT
2016-06-17 10:57:05.103735053 End saving filesystem layout
...

I've tested this on shell:

        ########################################
        # Mounted btrfs subvolumes:
        if test -x "$findmnt_command" ; then
            read_mounted_btrfs_subvolumes_command="$findmnt_command -nrv -o SOURCE,TARGET,OPTIONS,FSROOT -t btrfs"
        else
            read_mounted_btrfs_subvolumes_command="mount -t btrfs | cut -d ' ' -f 1,3,6"
        fi

SLES:
$ findmnt -nrv -o SOURCE,TARGET,OPTIONS,FSROOT -t btrfs
findmnt: unknown column: FSROOT
RHEL:
$ findmnt -nrv -o SOURCE,TARGET,OPTIONS,FSROOT -t btrfs
findmnt: unknown column: FSROOT

Seems this command is not working at this time.

the resulting disklayout have same output format (because in fact the first findmnt command is getting the fs list and the specific btrfs command is not working)

supported_filesystems="ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs"
read_filesystems_command="$findmnt_command -mnrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t $supported_filesystems"

findmnt -m ... is working also with BTRFS subvols and the dedicated btrfs command is not working.

Look at disklayout outputs:

# grep btrfssnap disklayout.conf (with findmnt -m ... )
# Format: btrfssnapshotsubvol    
#btrfssnapshotsubvol /dev/sda4 / 276 @/.snapshots/1/snapshot
#btrfssnapshotsubvol /dev/sda4 / 277 @/opt/.snapshots/1/snapshot
#btrfssnapshotsubvol /dev/sda4 / 283 @/.snapshots/4/snapshot
#btrfssnapshotsubvol /dev/sda4 / 284 @/.snapshots/5/snapshot
#btrfssnapshotsubvol /dev/sda4 / 470 @/.snapshots/98/snapshot
#btrfssnapshotsubvol /dev/sda4 / 471 @/opt/.snapshots/96/snapshot
#btrfssnapshotsubvol /dev/sda4 / 608 @/.snapshots/167/snapshot
#btrfssnapshotsubvol /dev/sda4 / 609 @/.snapshots/168/snapshot
#btrfssnapshotsubvol /dev/sda4 / 610 @/.snapshots/169/snapshot
#btrfssnapshotsubvol /dev/sda4 / 611 @/.snapshots/170/snapshot
#btrfssnapshotsubvol /dev/sda4 / 1595 @/.snapshots/662/snapshot
#btrfssnapshotsubvol /dev/sda4 / 1596 @/opt/.snapshots/656/snapshot
#btrfssnapshotsubvol /dev/sda4 / 3036 @/.snapshots/1382/snapshot
#btrfssnapshotsubvol /dev/sda4 / 3037 @/opt/.snapshots/1376/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4261 @/.snapshots/1993/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4262 @/.snapshots/1994/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4263 @/.snapshots/1995/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4265 @/.snapshots/1996/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4532 @/.snapshots/2130/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4533 @/opt/.snapshots/2120/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4820 @/.snapshots/2274/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4821 @/opt/.snapshots/2264/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4868 @/.snapshots/2298/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4869 @/opt/.snapshots/2288/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4916 @/.snapshots/2322/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4917 @/opt/.snapshots/2312/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4964 @/.snapshots/2346/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4965 @/opt/.snapshots/2336/snapshot
#btrfssnapshotsubvol /dev/sda4 / 5012 @/.snapshots/2370/snapshot
#btrfssnapshotsubvol /dev/sda4 / 5013 @/opt/.snapshots/2360/snapshot
#btrfssnapshotsubvol /dev/sda4 / 5060 @/.snapshots/2394/snapshot
#btrfssnapshotsubvol /dev/sda4 / 5061 @/opt/.snapshots/2384/snapshot
#btrfssnapshotsubvol /dev/sda4 / 5108 @/.snapshots/2418/snapshot
#btrfssnapshotsubvol /dev/sda4 / 5109 @/opt/.snapshots/2408/snapshot
...
...
$ grep btrfssnap disklayout.conf.orig (without findmnt -m ... )
# Format: btrfssnapshotsubvol    
#btrfssnapshotsubvol /dev/sda4 / 276 @/.snapshots/1/snapshot
#btrfssnapshotsubvol /dev/sda4 / 277 @/opt/.snapshots/1/snapshot
#btrfssnapshotsubvol /dev/sda4 / 283 @/.snapshots/4/snapshot
#btrfssnapshotsubvol /dev/sda4 / 284 @/.snapshots/5/snapshot
#btrfssnapshotsubvol /dev/sda4 / 470 @/.snapshots/98/snapshot
#btrfssnapshotsubvol /dev/sda4 / 471 @/opt/.snapshots/96/snapshot
#btrfssnapshotsubvol /dev/sda4 / 608 @/.snapshots/167/snapshot
#btrfssnapshotsubvol /dev/sda4 / 609 @/.snapshots/168/snapshot
#btrfssnapshotsubvol /dev/sda4 / 610 @/.snapshots/169/snapshot
#btrfssnapshotsubvol /dev/sda4 / 611 @/.snapshots/170/snapshot
#btrfssnapshotsubvol /dev/sda4 / 1595 @/.snapshots/662/snapshot
#btrfssnapshotsubvol /dev/sda4 / 1596 @/opt/.snapshots/656/snapshot
#btrfssnapshotsubvol /dev/sda4 / 3036 @/.snapshots/1382/snapshot
#btrfssnapshotsubvol /dev/sda4 / 3037 @/opt/.snapshots/1376/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4261 @/.snapshots/1993/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4262 @/.snapshots/1994/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4263 @/.snapshots/1995/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4265 @/.snapshots/1996/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4532 @/.snapshots/2130/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4533 @/opt/.snapshots/2120/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4772 @/.snapshots/2250/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4773 @/opt/.snapshots/2240/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4820 @/.snapshots/2274/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4821 @/opt/.snapshots/2264/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4868 @/.snapshots/2298/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4869 @/opt/.snapshots/2288/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4916 @/.snapshots/2322/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4917 @/opt/.snapshots/2312/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4964 @/.snapshots/2346/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4965 @/opt/.snapshots/2336/snapshot
#btrfssnapshotsubvol /dev/sda4 / 5012 @/.snapshots/2370/snapshot
#btrfssnapshotsubvol /dev/sda4 / 5013 @/opt/.snapshots/2360/snapshot
#btrfssnapshotsubvol /dev/sda4 / 5060 @/.snapshots/2394/snapshot
...
...

Regarding to WWN, yes is WWID, the unique name/ID of a Storage LUN. The problem is that after the multipath is loaded to the kernel and applied all aliases, /dev/mapper/WWIDpX no longer exists on system, only the alias /dev/mapper/vvbootp4. This is the cause rear is not able to get (/) information from system getting findmnt info from /proc/self/mountinfo because the block device is trying to get info no longer exists.

jsmeix commented at 2016-06-17 10:19:¶

@didacog
many thanks for your testing on SLE11.

You are right.
On SLE11:

# findmnt -nrv -o SOURCE,TARGET,OPTIONS,FSROOT
findmnt: unknown column: FSROOT

The reason why this was not noticed up to now is
that on SLE11 in parctice "nobody" uses btrfs
(and I never tested btrfs on SLE11).

On SLE12 findmnt knows about FSROOT
(and I tested btrfs only on SLE12).

I will provide a fix soon via https://github.com/rear/rear/issues/883

didacog commented at 2016-06-17 10:28:¶

@jsmeix
No problem! we are here to help! ;)

What about -m option? did you test it on SLES12?

On SLES11 with btrfs subvols the output with or without (-m) is the same. Anyway I cannot test it with SLES12 and not sure if something changes with FSROOT option.

jsmeix commented at 2016-06-17 11:33:¶

I merged https://github.com/rear/rear/pull/884
because it still works for me on SLE12-SP1
with its default btrfs structure.

Furthermore with https://github.com/rear/rear/pull/884
I use 'findmnt -m' to make the 'fs' entries
(but not for other findmnt commands),
cf. https://github.com/rear/rear/issues/882#issuecomment-226700826

@didacog
please upgrade to the current rear master to have https://github.com/rear/rear/pull/884
and do your further work based on that one.

I did not yet test whether or not SLE11 with btrfs
works with https://github.com/rear/rear/pull/884
because I need a bit of time to prepare a
SLE11 test system with btrfs...

didacog commented at 2016-06-17 11:43:¶

Ok, I will test it and provide some feedback ASAP ;)

jsmeix commented at 2016-06-17 13:13:¶

A quick test with SLE11-SP4 with btrfs
works for me with https://github.com/rear/rear/pull/884

During initial installation of SLE11-SP4
I selected in the YaST partitioning dialog
to use the whole harddisk and then
to use btrfs on it.

I got this partitioning:

# parted -l
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sda: 21.5GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number  Start   End     Size    Type     File system     Flags
 1      1049kB  165MB   164MB   primary  ext3            boot, type=83
 2      165MB   1735MB  1571MB  primary  linux-swap(v1)  type=82
 3      1735MB  21.5GB  19.7GB  primary  btrfs           type=83

and that btrfs subvolumes:

# btrfs subvolume list -a /        
ID 256 gen 194 top level 5 path /@
ID 258 gen 195 top level 256 path @/tmp
ID 259 gen 166 top level 256 path @/opt
ID 260 gen 166 top level 256 path @/srv
ID 261 gen 126 top level 256 path @/var/crash
ID 262 gen 209 top level 256 path @/var/spool
ID 263 gen 208 top level 256 path @/var/log
ID 264 gen 208 top level 256 path @/var/run
ID 265 gen 166 top level 256 path @/var/tmp
ID 268 gen 195 top level 256 path @/.snapshots

I used this /etc/rear/local.conf

OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://10.160.4.244/nfs
NETFS_KEEP_OLD_BACKUP_COPY=yes
BACKUP_PROG_INCLUDE=( '/var/tmp/*' '/var/run/*' '/var/spool/*' '/var/log/*' '/var/crash/*' '/tmp/*' '/srv/*' '/opt/*' )
SSH_ROOT_PASSWORD="rear"
USE_DHCLIENT="yes"
KEEP_BUILD_DIR=""

I did not fully verify it all is really correct after "rear recover"
but the recovered system works and looks o.k.

jsmeix commented at 2016-06-17 13:52:¶

@didacog
regarding your https://github.com/rear/rear/issues/882#issuecomment-226735901
"I cannot test it with SLES12"

I wonder why you cannot test it with SLES12?

We (i.e. SUSE) have SLE12 installation images available
for free download at
https://www.suse.com/download-linux

Furthermore there is the new public SUSE Beta Program, see
https://www.suse.com/support/beta-program

didacog commented at 2016-06-17 14:32:¶

@jsmeix

Tested on SLES12SP1. ;)

findmnt command shows same output with or without (-m) option. In this case, maybe using (-mnrv) on all cases could be a good solution.

here the evidence:

linux-hxww:~ # lsb_release -a
LSB Version:    n/a
Distributor ID: SUSE LINUX
Description:    SUSE Linux Enterprise Server 12 SP1
Release:    12.1
Codename:   n/a
linux-hxww:~ # findmnt -nrv -o SOURCE,TARGET,OPTIONS,FSROOT -t btrfs
/dev/sda2 / rw,relatime,space_cache,subvolid=257,subvol=/@ /@
/dev/sda2 /var/tmp rw,relatime,space_cache,subvolid=275,subvol=/@/var/tmp /@/var/tmp
/dev/sda2 /var/lib/mariadb rw,relatime,space_cache,subvolid=268,subvol=/@/var/lib/mariadb /@/var/lib/mariadb
/dev/sda2 /var/spool rw,relatime,space_cache,subvolid=274,subvol=/@/var/spool /@/var/spool
/dev/sda2 /var/lib/libvirt/images rw,relatime,space_cache,subvolid=266,subvol=/@/var/lib/libvirt/images /@/var/lib/libvirt/images
/dev/sda2 /var/opt rw,relatime,space_cache,subvolid=273,subvol=/@/var/opt /@/var/opt
/dev/sda2 /var/log rw,relatime,space_cache,subvolid=272,subvol=/@/var/log /@/var/log
/dev/sda2 /var/lib/pgsql rw,relatime,space_cache,subvolid=271,subvol=/@/var/lib/pgsql /@/var/lib/pgsql
/dev/sda2 /var/lib/named rw,relatime,space_cache,subvolid=270,subvol=/@/var/lib/named /@/var/lib/named
/dev/sda2 /var/lib/mysql rw,relatime,space_cache,subvolid=269,subvol=/@/var/lib/mysql /@/var/lib/mysql
/dev/sda2 /var/lib/mailman rw,relatime,space_cache,subvolid=267,subvol=/@/var/lib/mailman /@/var/lib/mailman
/dev/sda2 /var/crash rw,relatime,space_cache,subvolid=265,subvol=/@/var/crash /@/var/crash
/dev/sda2 /usr/local rw,relatime,space_cache,subvolid=264,subvol=/@/usr/local /@/usr/local
/dev/sda2 /tmp rw,relatime,space_cache,subvolid=263,subvol=/@/tmp /@/tmp
/dev/sda2 /opt rw,relatime,space_cache,subvolid=261,subvol=/@/opt /@/opt
/dev/sda2 /srv rw,relatime,space_cache,subvolid=262,subvol=/@/srv /@/srv
/dev/sda2 /home rw,relatime,space_cache,subvolid=260,subvol=/@/home /@/home
/dev/sda2 /boot/grub2/x86_64-efi rw,relatime,space_cache,subvolid=259,subvol=/@/boot/grub2/x86_64-efi /@/boot/grub2/x86_64-efi
/dev/sda2 /boot/grub2/i386-pc rw,relatime,space_cache,subvolid=258,subvol=/@/boot/grub2/i386-pc /@/boot/grub2/i386-pc
linux-hxww:~ # findmnt -mnrv -o SOURCE,TARGET,OPTIONS,FSROOT -t btrfs
/dev/sda2 / rw,relatime,space_cache,subvolid=257,subvol=/@ /@
/dev/sda2 /var/tmp rw,relatime,space_cache,subvolid=275,subvol=/@/var/tmp /@/var/tmp
/dev/sda2 /var/lib/mariadb rw,relatime,space_cache,subvolid=268,subvol=/@/var/lib/mariadb /@/var/lib/mariadb
/dev/sda2 /var/spool rw,relatime,space_cache,subvolid=274,subvol=/@/var/spool /@/var/spool
/dev/sda2 /var/lib/libvirt/images rw,relatime,space_cache,subvolid=266,subvol=/@/var/lib/libvirt/images /@/var/lib/libvirt/images
/dev/sda2 /var/opt rw,relatime,space_cache,subvolid=273,subvol=/@/var/opt /@/var/opt
/dev/sda2 /var/log rw,relatime,space_cache,subvolid=272,subvol=/@/var/log /@/var/log
/dev/sda2 /var/lib/pgsql rw,relatime,space_cache,subvolid=271,subvol=/@/var/lib/pgsql /@/var/lib/pgsql
/dev/sda2 /var/lib/named rw,relatime,space_cache,subvolid=270,subvol=/@/var/lib/named /@/var/lib/named
/dev/sda2 /var/lib/mysql rw,relatime,space_cache,subvolid=269,subvol=/@/var/lib/mysql /@/var/lib/mysql
/dev/sda2 /var/lib/mailman rw,relatime,space_cache,subvolid=267,subvol=/@/var/lib/mailman /@/var/lib/mailman
/dev/sda2 /var/crash rw,relatime,space_cache,subvolid=265,subvol=/@/var/crash /@/var/crash
/dev/sda2 /usr/local rw,relatime,space_cache,subvolid=264,subvol=/@/usr/local /@/usr/local
/dev/sda2 /tmp rw,relatime,space_cache,subvolid=263,subvol=/@/tmp /@/tmp
/dev/sda2 /opt rw,relatime,space_cache,subvolid=261,subvol=/@/opt /@/opt
/dev/sda2 /srv rw,relatime,space_cache,subvolid=262,subvol=/@/srv /@/srv
/dev/sda2 /home rw,relatime,space_cache,subvolid=260,subvol=/@/home /@/home
/dev/sda2 /boot/grub2/x86_64-efi rw,relatime,space_cache,subvolid=259,subvol=/@/boot/grub2/x86_64-efi /@/boot/grub2/x86_64-efi
/dev/sda2 /boot/grub2/i386-pc rw,relatime,space_cache,subvolid=258,subvol=/@/boot/grub2/i386-pc /@/boot/grub2/i386-pc
linux-hxww:~ # findmnt -nrv -o SOURCE,TARGET,OPTIONS,FSROOT -t btrfs > findmnt.out
linux-hxww:~ # findmnt -mnrv -o SOURCE,TARGET,OPTIONS,FSROOT -t btrfs > findmnt-m.out
linux-hxww:~ # diff findmnt.out findmnt-m.out

didacog commented at 2016-06-28 13:42:¶

@jsmeix

Tomorrow I will test to recover on SLES12 with your latest code for the FSROOT option, but anyway the findmnt (-m ) should be on BTRFS section also, because if not, in case of BTRFS with BOOT from SAN it will fail with same issue. With my last test the output is the same on both cases.
If you haven't any complaint about this, i will test your code with (-m) on btrfs section also and send a pull request with this and other issues i found playing with Boot from SAN these last days.

I will open an issue for them also together with the pull request.

Regards,
Didac

jsmeix commented at 2016-06-28 14:26:¶

@didacog
since I had written that new famous
"Dirty hacks welcome" section in
https://github.com/rear/rear/wiki/Coding-Style
any properly documented code that intends
to make it work everywhere is welcome.

In this particular case:
As long as "findmnt -m" works everywhere it is o.k.
and as long as it is properly documented why that '-m'
is used one could easily later change it as needed.

didacog commented at 2016-06-28 14:32:¶

@jsmeix

Ok, tomorrow I will submit a new pull request with those changes properly documented.

Regards,
Didac

didacog commented at 2016-07-05 09:14:¶

Hi @jsmeix,

I've submitted a new pull request #903 solving findmnt problems (findmnt -m).

I've tested it on SLES 12 and worked.

Also meet a problem with chattr missing on rescue image. Should be solved with the pull request.

I had problems with usr/share/rear/finalize/Linux-i386/23_run_efibootmgr.sh. But is solved in 1.18 with this code:

if [[ ${Dev/mapper//} != $Dev ]] ; then # we have 'mapper' in devname
    # we only expect mpath_partX  or mpathpX or mpath-partX
    case $Disk in
        (*p)     Disk=${Disk%p} ;;
        (*-part) Disk=${Disk%-part} ;;
        (*_part) Disk=${Disk%_part} ;;
        (*)      Log "Unsupported kpartx partition delimiter for $Dev"
    esac
fi

Regards,

jsmeix commented at 2016-07-07 11:48:¶

Damn!
I made a typo in the git commit message for https://github.com/rear/rear/pull/903
where I falsely wrote https://github.com/rear/rear/issues/822
instead of the right issue https://github.com/rear/rear/issues/882

jsmeix commented at 2016-07-07 12:01:¶

@didacog
regarding "chattr missing on rescue image" see
usr/share/rear/conf/examples/SLE12-SP1-btrfs-example.conf
With your https://github.com/rear/rear/pull/903
this should now "just work" even automatically.
Many thanks for it!

Regarding your "problems with ... 23_run_efibootmgr.sh"
in https://github.com/rear/rear/issues/882#issuecomment-230427317

I think it would be better when you submit a new separated
GitHub issue for that to avoid that several actually separated
issues get mixed up in one single big and fat monster issue.

In particular because I am not at all an expert regarding
bootloader issues I would prefer a separated issue.

Or perhaps just do a separated GitHub pull request that
fixes your "problems with ... 23_run_efibootmgr.sh"?

jsmeix commented at 2016-07-07 12:50:¶

@didacog
I think I misunderstood what you meant with
your "problems with ... 23_run_efibootmgr.sh"
in https://github.com/rear/rear/issues/882#issuecomment-230427317

In current rear master code your mentioned code
is already there so that you had problems
that are already fixed in current rear master.

FYI:

"git blame" shows for 23_run_efibootmgr.sh
that your mentioned code was provided by
Pavol Domin via git commit bf6742be
and sarching for that git commit ID in
"git log" shows that commit is about
"SAN booting related improvements" which leads to
git commit 42be8008cbd79193fbd00228aef0e005ed9d940f
that reads "Merge pull request #695 from pavoldomin/sanboot"
so that we finally know it was https://github.com/rear/rear/pull/695

@didacog
can I close this particular issue?

I.e. is this particular issue fixed for you
with https://github.com/rear/rear/pull/903
and https://github.com/rear/rear/pull/695

didacog commented at 2016-07-07 13:27:¶

@jsmeix Yes you can close this issue. Thanks! ;)

[Export of Github issue for rear/rear.]

#882 Issue closed: HP Superdome-X servers using multipath an booting from SAN not backing up de ROOT filesystem!¶

didacog opened issue at 2016-06-16 12:36:¶

Relax-and-Recover (rear) Issue Template¶

jsmeix commented at 2016-06-16 12:50:¶

didacog commented at 2016-06-16 13:01:¶

jsmeix commented at 2016-06-16 13:03:¶

jsmeix commented at 2016-06-16 13:17:¶

jsmeix commented at 2016-06-16 13:24:¶

didacog commented at 2016-06-16 13:25:¶

didacog commented at 2016-06-16 13:30:¶

jsmeix commented at 2016-06-16 13:33:¶

jsmeix commented at 2016-06-16 13:40:¶

jsmeix commented at 2016-06-16 13:49:¶

didacog commented at 2016-06-16 13:53:¶

didacog commented at 2016-06-16 14:03:¶

jsmeix commented at 2016-06-16 14:20:¶

jsmeix commented at 2016-06-16 14:30:¶

didacog commented at 2016-06-16 15:26:¶

jsmeix commented at 2016-06-17 07:33:¶

didacog commented at 2016-06-17 09:27:¶

jsmeix commented at 2016-06-17 10:19:¶

didacog commented at 2016-06-17 10:28:¶

jsmeix commented at 2016-06-17 11:33:¶

didacog commented at 2016-06-17 11:43:¶

jsmeix commented at 2016-06-17 13:13:¶

jsmeix commented at 2016-06-17 13:52:¶

didacog commented at 2016-06-17 14:32:¶

didacog commented at 2016-06-28 13:42:¶

jsmeix commented at 2016-06-28 14:26:¶

didacog commented at 2016-06-28 14:32:¶

didacog commented at 2016-07-05 09:14:¶

jsmeix commented at 2016-07-07 11:48:¶

jsmeix commented at 2016-07-07 12:01:¶

jsmeix commented at 2016-07-07 12:50:¶

didacog commented at 2016-07-07 13:27:¶

#882 Issue `closed`: HP Superdome-X servers using multipath an booting from SAN not backing up de ROOT filesystem!¶