#555 Issue closed: ReaR Recover Issue - Unrecognized mount option, wrong fs type, bad option¶

Labels: support / question

bbeaver opened issue at 2015-03-02 00:17:¶

Hi All:

Running current rear-1.16.1-1.git201502200825, and had both a Centos6.6 and RHEL6.5 box experience recover problems that had to be manually corrected. Both had very similar problems, but with different invalid options built within a mount command. My Centos6.6 box attempted a 'mount -o rw,relatime,barrier=1,data /dev/mapper/vg_centos02-LogVol00 /mnt/local/" command, and my RHEL6.5 box attempted a "mount -o rw,relatime,barrier=1,strip /dev/mapper/vg00-LogVol00 /mnt/local/" command. Both failed and had to be manually tweaked using the "Edit restore script" option. In the Centos case, had to remove the "data" option that had been added, in the case of RHEL had to remove the "strip" option that had been added. How are these invalid mount options landing in the restore script? Thanks for your help.

gdha commented at 2015-03-02 08:42:¶

@bbeaver Could you attach the disklayout.conf file? There were recent some changes in the script /usr/share/rear/layout/save/GNU/Linux/23_filesystem_layout.sh - could be related?

bbeaver commented at 2015-03-02 13:47:¶

Yes, thanks - please see disklayout.conf below. You can see it was created with an invalid "data" option within the mount_options:

disk /dev/cciss/c0d0 587127480320 msdos
part /dev/cciss/c0d0 524288000 1048576 primary boot /dev/cciss/c0d0p1
part /dev/cciss/c0d0 565628108800 525336576 primary lvm /dev/cciss/c0d0p2
part /dev/cciss/c0d0 10485760000 566153445376 primary none /dev/cciss/c0d0p3
part /dev/cciss/c0d0 1024 576639205376 extended lba /dev/cciss/c0d0p4
part /dev/cciss/c0d0 10485760000 576641302528 logical lvm /dev/cciss/c0d0p5
lvmdev /dev/VG00 /dev/cciss/c0d0p5 DqSgVf-IPJs-lW6z-VMVQ-3sbO-AoH0-iMSBgL 20480000
lvmdev /dev/vg_centos02 /dev/cciss/c0d0p2 a4Cutt-6Gnv-eeVp-5mQC-32oV-p8Fk-5Nqj71 1104742400
lvmgrp /dev/VG00 4096 2499 10235904
lvmgrp /dev/vg_centos02 4096 134856 552370176
lvmvol /dev/VG00 LVTEST01 1280 10485760
lvmvol /dev/vg_centos02 LogVol01 8000 65536000
lvmvol /dev/vg_centos02 LogVol00 12500 102400000
lvmvol /dev/vg_centos02 LogVol02 114356 936804352
# Filesystems (only ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs are supported).
# Format: fs <device> <mountpoint> <fstype> [uuid=<uuid>] [label=<label>] [<attributes>]
fs /dev/cciss/c0d0p1 /boot ext4 uuid=964edbd0-3eb2-4caa-9704-3fca7540ba71 label= blocksize=1024 fragmentsize=1024 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=4063 default_mount_options=user_xattr,acl options=rw,relatime,barrier=1,data
fs /dev/drbd0 /home/test01 ext4 uuid=5eac3567-ad8d-48d6-b519-1276f20b4ad2 label= blocksize=4096 fragmentsize=4096 reserved_blocks=4% max_mounts=21 check_interval=180d bytes_per_inode=16383 options=rw,relatime,barrier=1,data
fs /dev/mapper/vg_centos02-LogVol00 / ext4 uuid=7f82e34c-b22b-4d09-b30e-f1010a3f4b80 label= blocksize=4096 fragmentsize=4096 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=16304 default_mount_options=user_xattr,acl options=rw,relatime,barrier=1,data
fs /dev/mapper/vg_centos02-LogVol02 /home ext4 uuid=d70d94e8-30b2-4651-8648-6984e8a4e787 label= blocksize=4096 fragmentsize=4096 reserved_blocks=2% max_mounts=-1 check_interval=0d bytes_per_inode=16318 default_mount_options=user_xattr,acl options=rw,relatime,barrier=1,data
swap /dev/mapper/vg_centos02-LogVol01 uuid=6d15bf48-4298-488d-99a3-c698b4392570 label=
drbd /dev/drbd0 test01data /dev/mapper/VG00-LVTEST01

gdha commented at 2015-03-02 13:52:¶

@bbeaver ok thanks - and could you also post the mount output?

bbeaver commented at 2015-03-02 14:42:¶

Sure - plesae see mount output below:

mount¶

/dev/mapper/vg_centos02-LogVol00 on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/cciss/c0d0p1 on /boot type ext4 (rw)
/dev/mapper/vg_centos02-LogVol02 on /home type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/dev/drbd0 on /home/test01 type ext4 (rw)

gdha commented at 2015-03-02 15:47:¶

# diff disklayout.conf disklayout.conf.old
11,16c11,14
< # Filesystems (only ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs are supported).
< # Format: fs <device> <mountpoint> <fstype> [uuid=<uuid>] [label=<label>] [<attributes>]
< fs /dev/mapper/vg_noc-lv_home /home ext4 uuid=42fc2968-51b8-4f0a-89a0-f4faeffe72c2 label= blocksize=4096 fragmentsize=4096 reserved_blocks=4% max_mounts=-1 check_interval=0d bytes_per_inode=16377 default_mount_options=user_xattr,acl options=rw,relatime,barrier=1,data=ordered
< fs /dev/mapper/vg_noc-lv_root / ext4 uuid=9dcfc861-9895-4456-89cd-8aab887900f8 label= blocksize=4096 fragmentsize=4096 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=16384 default_mount_options=user_xattr,acl options=rw,relatime,barrier=1,data=ordered
< fs /dev/sda1 /boot ext4 uuid=8bfe951c-ad1c-464a-86a6-72f65f3833f4 label= blocksize=1024 fragmentsize=1024 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=4095 default_mount_options=user_xattr,acl options=rw,relatime,barrier=1,data=ordered
---
> fs /dev/mapper/vg_noc-lv_root / ext4 uuid=9dcfc861-9895-4456-89cd-8aab887900f8 label= blocksize=4096 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=16384 default_mount_options=user_xattr,acl options=rw
> fs /dev/sda1 /boot ext4 uuid=8bfe951c-ad1c-464a-86a6-72f65f3833f4 label= blocksize=1024 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=4095 default_mount_options=user_xattr,acl options=rw
> fs /dev/mapper/vg_noc-lv_home /home ext4 uuid=42fc2968-51b8-4f0a-89a0-f4faeffe72c2 label= blocksize=4096 reserved_blocks=4% max_mounts=-1 check_interval=0d bytes_per_inode=16377 default_mount_options=user_xattr,acl options=rw

The difference is coming from:

/bin/findmnt -alnv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs

before applying the patch of @jsmeix we just used mount

Alright, between rw and rw,relatime,seclabel,barrier=1,data=ordered - what should be keep (rw), but what about the rest?

bbeaver commented at 2015-03-02 16:07:¶

I manually edited the restore script in both my cases where rear recover failed. On the CentOS box related to the info above, I changed the mount command from:

"mount -o rw,relatime,barrier=1,data /dev/mapper/vg_centos02-LogVol00 /mnt/local/"

to:

"mount -o rw,relatime,barrier=1 /dev/mapper/vg_centos02-LogVol00 /mnt/local/"

On the RHEL system, I basically did the same thing, but removed a "strip" option instead of the "data" option. After doing this, I chose to continue the recovery, and it succeeded.

jsmeix commented at 2015-03-03 11:15:¶

@bbeaver

What are the exact outputs for your CentOS6.6 and RHEL6.5 systems for each of the commands

findmnt -alnv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs

versus

mount -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs

jsmeix commented at 2015-03-03 11:23:¶

@gdha

at a first glange I currenly guess that the findmnt output that results entries like

fs  ...  options=rw,relatime,barrier=1,data=ordered

is probably correct but I guess some other script might fail to correctly process that kind of options value (in particular options of the form keyword=value)?

I guess this because bbeaver reported

mount -o rw,relatime,barrier=1,data /dev/...

which shows that the mount options have been clipped at the last '='.

I assume with the whole mount options as reported by findmnt

mount -o rw,relatime,barrier=1,data=ordered /dev/...

mounting would have worked.

jsmeix commented at 2015-03-03 11:44:¶

Phew!
It seems this issue here is only about fstype "ext4" which means it should not be caused by my new btrfs code :-)

jsmeix commented at 2015-03-03 11:52:¶

I think in 13_include_mount_filesystem_code.sh the following is wrong:

    # Extract mount options.
    local option mountopts
    for option in $options ; do
        name=${option%%=*}     # options can contain more '=' signs
        value=${option#*=}

because:

$ options=rw,relatime,barrier=1,data=ordered ; echo $options ; for option in $options ; do name=${option%%=*} ; value=${option#*=} ; echo name=$name value=$value ; done

outputs

rw,relatime,barrier=1,data=ordered
name=rw,relatime,barrier value=1,data=ordered

bbeaver commented at 2015-03-03 13:15:¶

Gdha is exactly right, the "=" is getting clipped out. Here is the output from findmnt and mount:

[root@centos02]]# findmnt -alnv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs
/dev/mapper/vg_centos02-LogVol00 / ext4 rw,relatime,barrier=1,data=ordered
/dev/cciss/c0d0p1 /boot ext4 rw,relatime,barrier=1,data=ordered
/dev/mapper/vg_centos02-LogVol02 /home ext4 rw,relatime,barrier=1,data=ordered
/dev/drbd0 /home/test01 ext4 rw,relatime,barrier=1,data=ordered

[root@centos02]# mount -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs
/dev/mapper/vg_centos02-LogVol00 on / type ext4 (rw)
/dev/cciss/c0d0p1 on /boot type ext4 (rw)
/dev/mapper/vg_centos02-LogVol02 on /home type ext4 (rw)
/dev/drbd0 on /home/test01 type ext4 (rw)

jsmeix commented at 2015-03-03 13:32:¶

My previous comment is wrong because the options entry is not

rw,relatime,barrier=1,data=ordered

but instead it is

options=rw,relatime,barrier=1,data=ordered

and with the latter it works because

$ options="options=rw,relatime,barrier=1,data=ordered" ; echo $options ; for option in $options ; do name=${option%%=*} ; value=${option#*=} ; echo name=$name value=$value ; done

outputs

options=rw,relatime,barrier=1,data=ordered
name=options value=rw,relatime,barrier=1,data=ordered

Somewhere else it happens that the mount options are clipped at the last '='.

gdha commented at 2015-03-03 14:44:¶

@bbeaver @jsmeix I think problem is not the code but the length of the output. In bbeaver example I counted 274 characters before it was cut off. Could be wrong of course, but it looks we also need to investigate this path

gdha commented at 2015-03-03 14:50:¶

@bbeaver could you run rear -vD savelayout and check the code around the line fs /dev/mapper/vg_centos02-LogVol02 /home?
Please paste only that part as we should be able to see what happens with the line (being truncated or what).

bbeaver commented at 2015-03-03 15:02:¶

The "savelayout" shows proper mount options, but running "mkbackup" does not, and cuts off the string.

Here's the output, First, run "rear -vD savelayout" and check disklayout.conf:

[root@centos02] # cat /var/lib/rear/layout/disklayout.conf
disk /dev/cciss/c0d0 587127480320 msdos
part /dev/cciss/c0d0 524288000 1048576 primary boot /dev/cciss/c0d0p1
part /dev/cciss/c0d0 565628108800 525336576 primary lvm /dev/cciss/c0d0p2
part /dev/cciss/c0d0 10485760000 566153445376 primary none /dev/cciss/c0d0p3
part /dev/cciss/c0d0 1024 576639205376 extended lba /dev/cciss/c0d0p4
part /dev/cciss/c0d0 10485760000 576641302528 logical lvm /dev/cciss/c0d0p5
lvmdev /dev/VG00 /dev/cciss/c0d0p5 DqSgVf-IPJs-lW6z-VMVQ-3sbO-AoH0-iMSBgL 20480000
lvmdev /dev/vg_centos02 /dev/cciss/c0d0p2 a4Cutt-6Gnv-eeVp-5mQC-32oV-p8Fk-5Nqj71 1104742400
lvmgrp /dev/VG00 4096 2499 10235904
lvmgrp /dev/vg_centos02 4096 134856 552370176
lvmvol /dev/VG00 LVTEST01 1280 10485760
lvmvol /dev/vg_centos02 LogVol01 8000 65536000
lvmvol /dev/vg_centos02 LogVol00 12500 102400000
lvmvol /dev/vg_centos02 LogVol02 114356 936804352
# Filesystems (only ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs are supported).
# Format: fs <device> <mountpoint> <fstype> [uuid=<uuid>] [label=<label>] [<attributes>]
fs /dev/cciss/c0d0p1 /boot ext4 uuid=964edbd0-3eb2-4caa-9704-3fca7540ba71 label= blocksize=1024 fragmentsize=1024 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=4047 default_mount_options=user_xattr,acl options=rw,relatime,barrier=1,data=ordered
fs /dev/drbd0 /home/test01 ext4 uuid=5eac3567-ad8d-48d6-b519-1276f20b4ad2 label= blocksize=4096 fragmentsize=4096 reserved_blocks=3% max_mounts=21 check_interval=180d bytes_per_inode=16351 options=rw,relatime,barrier=1,data=ordered
fs /dev/mapper/vg_centos02-LogVol00 / ext4 uuid=7f82e34c-b22b-4d09-b30e-f1010a3f4b80 label= blocksize=4096 fragmentsize=4096 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=16272 default_mount_options=user_xattr,acl options=rw,relatime,barrier=1,data=ordered
fs /dev/mapper/vg_centos02-LogVol02 /home ext4 uuid=d70d94e8-30b2-4651-8648-6984e8a4e787 label= blocksize=4096 fragmentsize=4096 reserved_blocks=1% max_mounts=-1 check_interval=0d bytes_per_inode=16286 default_mount_options=user_xattr,acl options=rw,relatime,barrier=1,data=ordered
swap /dev/mapper/vg_centos02-LogVol01 uuid=6d15bf48-4298-488d-99a3-c698b4392570 label=
drbd /dev/drbd0 testdata01 /dev/mapper/VG00-LVTEST01

Then, run "rear -v mkbackup" and check disklayout.conf:

[root@centos02] # cat disklayout.conf
disk /dev/cciss/c0d0 587127480320 msdos
part /dev/cciss/c0d0 524288000 1048576 primary boot /dev/cciss/c0d0p1
part /dev/cciss/c0d0 565628108800 525336576 primary lvm /dev/cciss/c0d0p2
part /dev/cciss/c0d0 10485760000 566153445376 primary none /dev/cciss/c0d0p3
part /dev/cciss/c0d0 1024 576639205376 extended lba /dev/cciss/c0d0p4
part /dev/cciss/c0d0 10485760000 576641302528 logical lvm /dev/cciss/c0d0p5
lvmdev /dev/VG00 /dev/cciss/c0d0p5 DqSgVf-IPJs-lW6z-VMVQ-3sbO-AoH0-iMSBgL 20480000
lvmdev /dev/vg_centos02 /dev/cciss/c0d0p2 a4Cutt-6Gnv-eeVp-5mQC-32oV-p8Fk-5Nqj71 1104742400
lvmgrp /dev/VG00 4096 2499 10235904
lvmgrp /dev/vg_centos02 4096 134856 552370176
lvmvol /dev/VG00 LVTEST01 1280 10485760
lvmvol /dev/vg_centos02 LogVol01 8000 65536000
lvmvol /dev/vg_centos02 LogVol00 12500 102400000
lvmvol /dev/vg_centos02 LogVol02 114356 936804352
# Filesystems (only ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs are supported).
# Format: fs <device> <mountpoint> <fstype> [uuid=<uuid>] [label=<label>] [<attributes>]
fs /dev/cciss/c0d0p1 /boot ext4 uuid=964edbd0-3eb2-4caa-9704-3fca7540ba71 label= blocksize=1024 fragmentsize=1024 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=4047 default_mount_options=user_xattr,acl options=rw,relatime,barrier=1,data
fs /dev/drbd0 /home/test01 ext4 uuid=5eac3567-ad8d-48d6-b519-1276f20b4ad2 label= blocksize=4096 fragmentsize=4096 reserved_blocks=3% max_mounts=21 check_interval=180d bytes_per_inode=16351 options=rw,relatime,barrier=1,data
fs /dev/mapper/vg_centos02-LogVol00 / ext4 uuid=7f82e34c-b22b-4d09-b30e-f1010a3f4b80 label= blocksize=4096 fragmentsize=4096 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=16272 default_mount_options=user_xattr,acl options=rw,relatime,barrier=1,data
fs /dev/mapper/vg_centos02-LogVol02 /home ext4 uuid=d70d94e8-30b2-4651-8648-6984e8a4e787 label= blocksize=4096 fragmentsize=4096 reserved_blocks=1% max_mounts=-1 check_interval=0d bytes_per_inode=16286 default_mount_options=user_xattr,acl options=rw,relatime,barrier=1,data
swap /dev/mapper/vg_centos02-LogVol01 uuid=6d15bf48-4298-488d-99a3-c698b4392570 label=
drbd /dev/drbd0 testdata01 /dev/mapper/VG00-LVTEST01

gdha commented at 2015-03-03 15:24:¶

@bbeaver that is interesting, but I don't get it as the disklayout.conf file is written only once (during the savelayout workflow)
Could you add the rear.log file when you run rear -vD mkbackup as a gist?

bbeaver commented at 2015-03-03 15:35:¶

The dksklayout.conf file definitely gets updated when running the "rear mkbackup" command. Please see below. Anything specific you are wanting to see from the rear.log? Thanks for your help

[root@centos02 tmp]# ls -ltr /var/lib/rear/layout
total 20
drwxr-xr-x 2 root root 4096 Jan 11 13:03 config
-rw-r--r-- 1 root root 2188 Mar 3 09:53 disklayout.conf
-rw-r--r-- 1 root root 598 Mar 3 09:53 disktodo.conf
-rw-r--r-- 1 root root 824 Mar 3 09:53 diskdeps.conf
drwxr-xr-x 2 root root 4096 Mar 3 09:53 lvm

[root@centos02 tmp]# date
Tue Mar 3 10:29:21 EST 2015

root@centos02 tmp]# nohup rear -vD mkbackup &

[root@centos02 tmp]# ls -ltr /var/lib/rear/layout
total 20
drwxr-xr-x 2 root root 4096 Jan 11 13:03 config
-rw-r--r-- 1 root root 2188 Mar 3 10:29 disklayout.conf
-rw-r--r-- 1 root root 598 Mar 3 10:29 disktodo.conf
-rw-r--r-- 1 root root 824 Mar 3 10:29 diskdeps.conf
drwxr-xr-x 2 root root 4096 Mar 3 10:29 lvm

gdha commented at 2015-03-03 15:55:¶

@bbeaver that is normal as sudo rear -s mkbackup will tell you that script layout/save/GNU/Linux/23_filesystem_layout.sh will get sourced. To verify type: sudo rear -s savelayout.
I still would like to get the full debugging output - thanks.

bbeaver commented at 2015-03-03 17:28:¶

Sure - looking to see how I upload the file...

bbeaver commented at 2015-03-03 17:34:¶

Please see if this gets you there - full rear-centos02.log debug:

https://gist.github.com/bbeaver/f736f9e80c3bd0b04017

gdha commented at 2015-03-03 19:24:¶

@bbeaver thanks for the logging:

+ . /usr/share/rear/layout/save/GNU/Linux/23_filesystem_layout.sh
++ Log 'Begin saving filesystem layout'
....
+++ tune2fs -l /dev/cciss/c0d0p1
+++ grep -i 'Default mount options:'
+++ cut -d : -f 2
+++ awk '{$1=$1};1'
+++ grep -v none
+++ tr ' ' ,
++ default_mount_options=user_xattr,acl
++ [[ -n user_xattr,acl ]]
++ echo -n ' default_mount_options=user_xattr,acl'
++ options=rw,relatime,barrier=1,data
++ options=rw,relatime,barrier=1,data
++ options=rw,relatime,barrier=1,data
++ echo -n ' options=rw,relatime,barrier=1,data'

I'll dig into the code tomorrow and see what it is causing it.

jsmeix commented at 2015-03-04 09:06:¶

At https://github.com/rear/rear/issues/545 I found

OS_VENDOR=RedHatEnterpriseServer
OS_VERSION=6.4

and

+++ mount -o rw,relatime,seclabel,barrier=1,dat /dev/mapper/vg_lsrs0vio-lv_root /mnt/local/
mount: wrong fs type, bad option, bad superblock on /dev/mapper/vg_lsrs0vio-lv_root,

It is the same issue there:
There the mount options are cut at "...dat".

Because there it is cut in between the "data=ordered" (and not at the '=' character) it now seems there is really some obscure length limit somewhere.

But if it is a length limit it is not a fixed length limit.

Very strange.

gdha commented at 2015-03-04 14:18:¶

@bbeaver @jsmeix I noticed it happens on centos 6.6 at random - let's say 1/10 it is bad and 9/10 it is ok - makes it even worse to debug

jsmeix commented at 2015-03-04 14:25:¶

@gdha

do the 1/10 bad happen also with "rear savelayout" or do the 1/10 bad happen only with "rear mkbackup"?

bbeaver commented at 2015-03-04 14:28:¶

I'm seeing RedHat/Fedora/CentOS servers behave the same with this issue. Not too surprising, since these distros are all RedHat based.

jsmeix commented at 2015-03-04 14:38:¶

Only a blind guess:

Perhaps it is not rear that sometimes cuts the mount options.

Perhaps it is the findmnt command itself that sometimes reports clipped mount options?

On my SLES12 system "man findmnt" reads in particular:

-u, --notruncate
Do not truncate text in columns. The default is to not truncate
the TARGET, SOURCE, UUID, LABEL, PARTUUID, PARTLABEL
columns. This option disables text truncation also in all other columns.

and 23_filesystem_layout.sh contains

read_filesystems_command="$findmnt_command -alnv -o SOURCE,TARGET,FSTYPE,OPTIONS ...

wich means without '-u' the FSTYPE and OPTIONS could be truncated.

bbeaver commented at 2015-03-04 14:43:¶

That might explain why I just started seeing this issue recently. A few months ago I had not come accross this problem. I understand that ReaR recently changed from using "mount" to using "findmnt". If there are problems with "findmnt", then ReaR is just now seeing those issues.

jsmeix commented at 2015-03-04 14:46:¶

@bbeaver

try if it helps to replace in usr/share/rear/layout/save/GNU/Linux/23_filesystem_layout.sh

read_filesystems_command="$findmnt_command -alnv -o ...

with

read_filesystems_command="$findmnt_command -alnuv -o ...

gdha commented at 2015-03-04 14:54:¶

@bbeaver @jsmeix savelayout is sourced by mkbackup so it is running all the time (overwriting the existing disklayout.conf file)

gdha commented at 2015-03-04 15:16:¶

I think that @jsmeix is right - it has to do with findmnt command itself and in particular on older RHEL based OSes (5,6)

schlomo commented at 2015-03-04 16:17:¶

findmnt supports -u (--notruncate) also on RHEL6, I don't have RHEL5 to
check.

Without running my own tests I wold place my bets on the -u option :-)

On 4 March 2015 at 16:16, gdha notifications@github.com wrote:

I think that @jsmeix https://github.com/jsmeix is right - it has to do
with findmnt command itself and in particular on older RHEL based OSes
(5,6)

—
Reply to this email directly or view it on GitHub
https://github.com/rear/rear/issues/555#issuecomment-77175752.

bbeaver commented at 2015-03-04 16:26:¶

I just added the "-u" option, such that 23_filesystem_layout.sh now looks like:

read_filesystems_command="$findmnt_command -alnuv -o ...

I've not tried tested a ReaR recovery, but, appears good now. The /var/lib/rear/layout/disklayout.conf file now shows "data=ordered" instead of "data" at the end of the fs options line.

schlomo commented at 2015-03-04 16:30:¶

I guess somebody should try to find out if RHEL5 supports that as well.
Otherwise we would have to discontinue RHEL5 support now??

On 4 March 2015 at 17:26, bbeaver notifications@github.com wrote:

I just added the "-u" option, such that 23_filesystem_layout.sh now looks
like:

read_filesystems_command="$findmnt_command -alnuv -o ...

I've not tried tested a ReaR recovery, but, appears good now. The
/var/lib/rear/layout/disklayout.conf file now shows "data=ordered" instead
of "data" at the end of the fs options line.

—
Reply to this email directly or view it on GitHub
https://github.com/rear/rear/issues/555#issuecomment-77190576.

bbeaver commented at 2015-03-04 17:09:¶

I'll hunt around and see if I can dig up a RHEL5 or Fedora equivalent box, check the findmnt options, and report back.

bbeaver commented at 2015-03-04 17:25:¶

I've made it back as far as Fedora 14, (which supports findmnt), but don't have easy access to a box with an older rhel5'ish o/s. I will continue looking though.

In the meantime, will this "-u" option to the findmnt command be added to the latest rear code?

bbeaver commented at 2015-03-04 18:45:¶

Thank you much for your help and quick response

jsmeix commented at 2015-03-05 10:48:¶

On RHEL 5.1 there is no findmnt command (at least not by default):

# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.1 (Tikanga)
# findmnt -alnv -o SOURCE,TARGET,FSTYPE,OPTIONS
-bash: findmnt: command not found
# type -a findmnt
-bash: type: findmnt: not found

In this case rear falls back using "mount" in 23_filesystem_layout.sh as follows:

# mount -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs | tr -s '[:blank:]' ' ' | cut -d ' ' -f 1,3,5,6
/dev/mapper/VolGroup00-LogVol00 / ext3 (rw)
/dev/hda1 /boot ext3 (rw)

schlomo commented at 2015-03-05 10:54:¶

The we should probably use this fallback code for all cases where
"findmount -u" will not work.

Are there any downsides to using mount instead of findmount? If yes, then
we should warn the user about them, e..g

WARNING: On your system ReaR cannot determine XYZ because findmnt is not
available.

On 5 March 2015 at 11:48, Johannes Meixner notifications@github.com wrote:

On RHEL 5.1 there is no findmnt command (at least not by default):

cat /etc/redhat-release¶

Red Hat Enterprise Linux Server release 5.1 (Tikanga)

findmnt -alnv -o SOURCE,TARGET,FSTYPE,OPTIONS¶

-bash: findmnt: command not found

type -a findmnt¶

-bash: type: findmnt: not found

In this case rear falls back using "mount" in 23_filesystem_layout.sh as
folows:

mount -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs | tr -s '[:blank:]' ' ' | cut -d ' ' -f 1,3,5,6¶

/dev/mapper/VolGroup00-LogVol00 / ext3 (rw)
/dev/hda1 /boot ext3 (rw)

—
Reply to this email directly or view it on GitHub
https://github.com/rear/rear/issues/555#issuecomment-77343473.

jsmeix commented at 2015-03-05 11:08:¶

One downside to using mount instead of findmount is documented in 23_filesystem_layout.sh:

# If the findmnt command is available use it instead of the traditional mount command
# because (since SLE12) "man 8 mount" reads:
#   The listing mode is maintained for backward compatibility only.
#   For more robust and customizable output use findmnt(8), especially in your scripts.

Another downside seems to be shown here in this issue:
findmnt reports more mount options compared to mount which means with findmnt rear could recreate the system more exactly as it was originally.

Finally findmnt is basically required for btrfs subvolumes to get the mounted btrfs subvolume path (i.e. the subvolume name), see "Mounted btrfs subvolumes" in 23_filesystem_layout.sh - if findmnt is not available it falls back using awkward hacks to determine the mounted btrfs subvolume path.

In practice I think on old systems without findmnt there will be no btrfs used and then the old systems get recreated as all the time before by plain "mount".

jsmeix commented at 2015-03-05 11:31:¶

On RHEL 5.9 it is (at least by default) the same as on RHEL 5.1:

# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.9 (Tikanga)
# findmnt -alnv -o SOURCE,TARGET,FSTYPE,OPTIONS
-bash: findmnt: command not found
# type -a findmnt
-bash: type: findmnt: not found
# mount -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs | tr -s '[:blank:]' ' ' | cut -d ' ' -f 1,3,5,6
/dev/mapper/VolGroup00-LogVol00 / ext3 (rw)
/dev/hda1 /boot ext3 (rw)

schlomo commented at 2015-03-05 11:33:¶

I guess systems without findmnt won't notice much difference then?

On 5 March 2015 at 12:31, Johannes Meixner notifications@github.com wrote:

On RHEL 5.9 it is (at least by default) the same as on RHEL 5.1:

cat /etc/redhat-release¶

Red Hat Enterprise Linux Server release 5.9 (Tikanga)

findmnt -alnv -o SOURCE,TARGET,FSTYPE,OPTIONS¶

-bash: findmnt: command not found

type -a findmnt¶

-bash: type: findmnt: not found

mount -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs | tr -s '[:blank:]' ' ' | cut -d ' ' -f 1,3,5,6¶

/dev/mapper/VolGroup00-LogVol00 / ext3 (rw)
/dev/hda1 /boot ext3 (rw)

—
Reply to this email directly or view it on GitHub
https://github.com/rear/rear/issues/555#issuecomment-77348964.

jsmeix commented at 2015-03-05 11:41:¶

I did my changes in 23_filesystem_layout.sh so that on systems without findmnt everything should work as before.

But on systems with findmnt it now uses findmnt and then issues like this one appear.

schlomo commented at 2015-03-05 11:45:¶

That is why I suggest that we use findmnt only if it supports the -u option.

On 5 March 2015 at 12:41, Johannes Meixner notifications@github.com wrote:

I did my changes in 23_filesystem_layout.sh so that on systems without
findmnt everything should work as before.

But on systems with findmnt it now uses findmnt and then issues like this
one appear.

—
Reply to this email directly or view it on GitHub
https://github.com/rear/rear/issues/555#issuecomment-77350188.

jsmeix commented at 2015-03-05 11:49:¶

@schlomo
do you know a Linux distribution with findmnt where findmnt does not support the 'u' option?

schlomo commented at 2015-03-05 12:00:¶

No, but I find it simpler to require that findmnt can handle -u than to
worry about the case of meeting a findmnt that does not.

Actually, before this thread I was not consciously aware of findmnt, so
thanks a lot for that.

On 5 March 2015 at 12:49, Johannes Meixner notifications@github.com wrote:

@schlomo https://github.com/schlomo
do you know a Linux disrtibution with findmnt where findmnt does not
support the 'u' option?

—
Reply to this email directly or view it on GitHub
https://github.com/rear/rear/issues/555#issuecomment-77351120.

jsmeix commented at 2015-03-05 12:10:¶

@schlomo
how would you implement a test to find out whether or not findmnt can handle the '-u' option?

I am currently installing RHEL 6 to find out about findmnt there...

FYI, regarding findmnt in general:
With btrfs and subvolumes everyone basically must use it because plain mount reports insufficient/misleading information because mount only reports the plain device like /dev/sda2 mounted multiple times at various mount points but not what subvolume of the btrfs on /dev/sda2 is mounted cf. https://github.com/rear/rear/issues/497 in particular https://github.com/rear/rear/issues/497#issuecomment-66615699

jsmeix commented at 2015-03-05 13:09:¶

On RHEL 6.5:

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.5 (Santiago)
# findmnt -alnv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs | tr -s '[:blank:]' ' '
/dev/mapper/VolGroup-lv_root / ext4 rw,relatime,seclabel,barrier=1,data=ordered
/dev/sda1 /boot ext4 rw,relatime,seclabel,barrier=1,data=ordered
# findmnt -alnuv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs | tr -s '[:blank:]' ' '
/dev/mapper/VolGroup-lv_root / ext4 rw,relatime,seclabel,barrier=1,data=ordered
/dev/sda1 /boot ext4 rw,relatime,seclabel,barrier=1,data=ordered
# mount -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs | tr -s '[:blank:]' ' ' | cut -d ' ' -f 1,3,5,6
/dev/mapper/VolGroup-lv_root / ext4 (rw)
/dev/sda1 /boot ext4 (rw)

Currently I don't know how to trigger that findmnt truncates FSTYPE or OPTIONS output.

jsmeix commented at 2015-03-05 13:12:¶

It depends on the width of the terminal wherein findmnt is run whether or not it truncates output.

On RHEL 6.5 in a smaller-width xterm:

# findmnt -alnv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs | tr -s '[:blank:]' ' '
/dev/mapper/VolGroup-lv_root / ext4 rw,relatime,seclabel,barrier=1,data=o
/dev/sda1 /boot ext4 rw,relatime,seclabel,barrier=1,data=o
# findmnt -alnuv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs | tr -s '[:blank:]' ' '
/dev/mapper/VolGroup-lv_root
 / ext4 rw,relatime,seclabel,barrier=1,data=ordered
/dev/sda1 /boot ext4 rw,relatime,seclabel,barrier=1,data=ordered
# findmnt -alnuv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs | tr -s '[:blank:]' ' ' | od -a
0000000   /   d   e   v   /   m   a   p   p   e   r   /   V   o   l   G
0000020   r   o   u   p   -   l   v   _   r   o   o   t  nl  sp   /  sp
0000040   e   x   t   4  sp   r   w   ,   r   e   l   a   t   i   m   e
0000060   ,   s   e   c   l   a   b   e   l   ,   b   a   r   r   i   e
0000100   r   =   1   ,   d   a   t   a   =   o   r   d   e   r   e   d
0000120  nl   /   d   e   v   /   s   d   a   1  sp   /   b   o   o   t
0000140  sp   e   x   t   4  sp   r   w   ,   r   e   l   a   t   i   m
0000160   e   ,   s   e   c   l   a   b   e   l   ,   b   a   r   r   i
0000200   e   r   =   1   ,   d   a   t   a   =   o   r   d   e   r   e
0000220   d  nl
0000222

Scaring!

With '-u' findmnt inserts a newline character when the output is too long for one line.

jsmeix commented at 2015-03-05 13:16:¶

@gdha
the issue is not yet fixed!

jsmeix commented at 2015-03-05 13:25:¶

I think using the '-r' option alone helps - at least on RHEL 6.5:

# findmnt -nrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs
/dev/mapper/VolGroup-lv_root / ext4 rw,relatime,seclabel,barrier=1,data=ordered
/dev/sda1 /boot ext4 rw,relatime,seclabel,barrier=1,data=ordered

jsmeix commented at 2015-03-05 13:40:¶

Also on SLE12 "findmnt -nrv -o .." seems to output the right thing:

# findmnt -nrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs
/dev/sda2 / btrfs rw,relatime,space_cache
/dev/sda2 /.snapshots btrfs rw,relatime,space_cache
/dev/sda2 /var/tmp btrfs rw,relatime,space_cache
/dev/sda2 /var/spool btrfs rw,relatime,space_cache
/dev/sda2 /var/opt btrfs rw,relatime,space_cache
/dev/sda2 /var/log btrfs rw,relatime,space_cache
/dev/sda2 /var/lib/named btrfs rw,relatime,space_cache
/dev/sda2 /var/lib/pgsql btrfs rw,relatime,space_cache
/dev/sda2 /var/lib/mailman btrfs rw,relatime,space_cache
/dev/sda2 /usr/local btrfs rw,relatime,space_cache
/dev/sda2 /var/crash btrfs rw,relatime,space_cache
/dev/sda2 /opt btrfs rw,relatime,space_cache
/dev/sda2 /tmp btrfs rw,relatime,space_cache
/dev/sda2 /srv btrfs rw,relatime,space_cache
/dev/sda2 /boot/grub2/x86_64-efi btrfs rw,relatime,space_cache
/dev/sda2 /home btrfs rw,relatime,space_cache
/dev/sda2 /boot/grub2/i386-pc btrfs rw,relatime,space_cache
# findmnt -nrv -o SOURCE,TARGET,OPTIONS,FSROOT -t btrfs
/dev/sda2 / rw,relatime,space_cache /@
/dev/sda2 /.snapshots rw,relatime,space_cache /@/.snapshots
/dev/sda2 /var/tmp rw,relatime,space_cache /@/var/tmp
/dev/sda2 /var/spool rw,relatime,space_cache /@/var/spool
/dev/sda2 /var/opt rw,relatime,space_cache /@/var/opt
/dev/sda2 /var/log rw,relatime,space_cache /@/var/log
/dev/sda2 /var/lib/named rw,relatime,space_cache /@/var/lib/named
/dev/sda2 /var/lib/pgsql rw,relatime,space_cache /@/var/lib/pgsql
/dev/sda2 /var/lib/mailman rw,relatime,space_cache /@/var/lib/mailman
/dev/sda2 /usr/local rw,relatime,space_cache /@/usr/local
/dev/sda2 /var/crash rw,relatime,space_cache /@/var/crash
/dev/sda2 /opt rw,relatime,space_cache /@/opt
/dev/sda2 /tmp rw,relatime,space_cache /@/tmp
/dev/sda2 /srv rw,relatime,space_cache /@/srv
/dev/sda2 /boot/grub2/x86_64-efi rw,relatime,space_cache /@/boot/grub2/x86_64-efi
/dev/sda2 /home rw,relatime,space_cache /@/home
/dev/sda2 /boot/grub2/i386-pc rw,relatime,space_cache /@/boot/grub2/i386-pc

jsmeix commented at 2015-03-05 14:06:¶

On RHEL 6.0 there is no findmnt (in contrast to RHEL 6.5) - at least not by default:

# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.0 (Santiago)
# type -a findmnt
-bash: type: findmnt: not found
# mount -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs | cut -d ' ' -f 1,3,5,6
/dev/mapper/VolGroup-lv_root / ext4 (rw)
/dev/sda1 /boot ext4 (rw)

jsmeix commented at 2015-03-05 14:12:¶

On RHEL 6.x findmnt was added to the util-linux-ng RPM
on "Thu Jan 27 2011":

# rpm -q --changelog util-linux-ng
.
.
.
* Thu Jan 27 2011 Karel Zak  2.17.2-8
...
- fix #616325 - Include findmnt in util-linux-ng package

jsmeix commented at 2015-03-05 14:59:¶

findmnt is available since RHEL 6.1 and works there as needed:

# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.1 (Santiago)
# findmnt -nrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs
/dev/mapper/VolGroup-lv_root / ext4 rw,relatime,seclabel,barrier=1,data=ordered
/dev/sda1 /boot ext4 rw,relatime,seclabel,barrier=1,data=ordered
# mount -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs | cut -d ' ' -f 1,3,5,6
/dev/mapper/VolGroup-lv_root / ext4 (rw)
/dev/sda1 /boot ext4 (rw)

jsmeix commented at 2015-03-05 15:17:¶

I submitted https://github.com/rear/rear/pull/559 that should hopefully fix it.

gdha commented at 2015-03-05 15:23:¶

@bbeaver @jsmeix and what if we do the following:

COLUMNS=254 findmnt -alnuv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs
/dev/mapper/vg00-lv00 /          ext3   rw,relatime,errors=continue,user_xattr,acl,barrier=1,data=ordered

or

COLUMNS=254 findmnt -nrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs
/dev/mapper/vg00-lv00 / ext3 rw,relatime,errors=continue,user_xattr,acl,barrier=1,data=ordered

jsmeix commented at 2015-03-05 15:32:¶

With '-r' findmnt results the right output regardless of the terminal width.

I had tried "COLUMNS=900" before but that did not make a difference for me.

I think the whole issue is that by default findmnt tries (too?) hard to make the output nice looking for interactive usage and only with '-r' one gets the real hard facts for real hard men ;-)

With RHEL 6.1 in a smaller xterm:

# COLUMNS=254 findmnt -alnuv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs        
/dev/mapper/VolGroup-lv_root
                          /      ext4   rw,relatime,seclabel,barrier=1,data=ordered
/dev/sda1                 /boot  ext4   rw,relatime,seclabel,barrier=1,data=ordered
# findmnt -nrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs
/dev/mapper/VolGroup-lv_root / ext4 rw,relatime,seclabel,barrier=1,data=ordered
/dev/sda1 /boot ext4 rw,relatime,seclabel,barrier=1,data=ordered

Note the additional newline in the first findmnt output that breaks what belongs to /dev/mapper/VolGroup-lv_root

---

[Export of Github issue for rear/rear.]