#545 Issue closed: Restoring from NFS on RHEL (ppc64)- mounting error on recreated filesystems¶
Labels: bug, support / question
jayfurmanek opened issue at 2015-02-06 23:33:¶
This is on RHEL ppc64, but it may not be specific to ppc64.
This very well could be a configuration issue.
local.conf:
BACKUP=NETFS
BACKUP_URL="nfs://<MY_IP>/local/export/rear/chef-server/"
USE_SERIAL_CONSOLE=y
BACKUP_SELINUX_DISABLE=y
os.conf:
OS_VENDOR=RedHatEnterpriseServer
OS_VERSION=6.4
System boots into REAR. I execute rear recover
<snip>
This filesystem will be automatically checked every 27 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
+++ tune2fs -U 420c1768-e3b5-4b18-a67e-162f49d8e24e /dev/mapper/vg_lsrs0vio-lv_root
tune2fs 1.41.12 (17-May-2010)
+++ tune2fs -m 5 -c -1 -i 0d -o user_xattr,acl /dev/mapper/vg_lsrs0vio-lv_root
tune2fs 1.41.12 (17-May-2010)
Setting maximal mount count to -1
Setting interval between checks to 0 seconds
Setting reserved blocks percentage to 5% (655360 blocks)
+++ LogPrint 'Mounting filesystem /'
+++ Log 'Mounting filesystem /'
+++ test 1 -gt 0
++++ Stamp
++++ date '+%Y-%m-%d %H:%M:%S '
+++ echo '2015-02-06 17:06:44 Mounting filesystem /'
2015-02-06 17:06:44 Mounting filesystem /
+++ Print 'Mounting filesystem /'
+++ test 1
+++ echo -e 'Mounting filesystem /'
+++ mkdir -p /mnt/local/
+++ mount -o rw,relatime,seclabel,barrier=1,dat /dev/mapper/vg_lsrs0vio-lv_root /mnt/local/
mount: wrong fs type, bad option, bad superblock on /dev/mapper/vg_lsrs0vio-lv_root,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
2015-02-06 17:06:44 An error occurred during layout recreation.
dmesg | tail:
sd 0:0:1:0: [sda] Cache data unavailable
sd 0:0:1:0: [sda] Assuming drive cache: write through
sda: sda1 sda2 sda3
eth1: no IPv6 routers present
EXT4-fs (dm-0): Unrecognized mount option "seclabel" or missing value
I used option 4 to go in and edit the restore script
(diskrestore.sh)
and deleted the references to "seclabel".
It also looks like the data=ordered parameter is truncated.
After fixing the diskrestore script on the fly, the restore completes.
cat /proc/mounts from the original system:
# cat /proc/mounts
rootfs / rootfs rw 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,seclabel,relatime 0 0
devtmpfs /dev devtmpfs rw,seclabel,relatime,size=4029248k,nr_inodes=62957,mode=755 0 0
devpts /dev/pts devpts rw,seclabel,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /dev/shm tmpfs rw,seclabel,relatime 0 0
/dev/mapper/vg_lsrs0vio-lv_root / ext4 rw,seclabel,relatime,barrier=1,data=ordered 0 0
none /selinux selinuxfs rw,relatime 0 0
devtmpfs /dev devtmpfs rw,seclabel,relatime,size=4029248k,nr_inodes=62957,mode=755 0 0
/proc/bus/usb /proc/bus/usb usbfs rw,relatime 0 0
/dev/sda2 /boot ext4 rw,seclabel,relatime,barrier=1,data=ordered 0 0
/dev/mapper/vg_lsrs0vio-lv_home /home ext4 rw,seclabel,relatime,barrier=1,data=ordered 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
cgroup /cgroup/cpuset cgroup rw,relatime,cpuset 0 0
cgroup /cgroup/cpu cgroup rw,relatime,cpu 0 0
cgroup /cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0
cgroup /cgroup/memory cgroup rw,relatime,memory 0 0
cgroup /cgroup/devices cgroup rw,relatime,devices 0 0
cgroup /cgroup/freezer cgroup rw,relatime,freezer 0 0
cgroup /cgroup/net_cls cgroup rw,relatime,net_cls 0 0
cgroup /cgroup/blkio cgroup rw,relatime,blkio 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0
/etc/auto.misc /misc autofs rw,relatime,fd=7,pgrp=1603,timeout=300,minproto=5,maxproto=5,indirect 0 0
-hosts /net autofs rw,relatime,fd=13,pgrp=1603,timeout=300,minproto=5,maxproto=5,indirect 0 0
does show the seclabel parameter.
I assume, like all things evil, this is related to SELinux. Is there
are way to configure around this?
gdha commented at 2015-02-07 08:43:¶
SElinux is disabled by rear in recovery mode (see /proc/cmdline).
Hum, the
mount -o rw,relatime,seclabel,barrier=1,dat /dev/mapper/vg_lsrs0vio-lv_root /mnt/local/
seems like ,dat is truncated? Could you paste the
/var/lib/rear/layout/disklayout.conf file?
jayfurmanek commented at 2015-02-10 23:57:¶
RESCUE chefserver:~ # cat /var/lib/rear/layout/disklayout.conf
disk /dev/sda 107374182400 msdos
part /dev/sda 4194304 1048576 primary boot,prep /dev/sda1
part /dev/sda 524288000 5242880 primary none /dev/sda2
part /dev/sda 106844651520 529530880 primary lvm /dev/sda3
lvmdev /dev/vg_lsrs0vio /dev/sda3 VzXJfz-1fSj-uz7p-Tf3E-8w5x-RoNQ-bVVIUo 208680960
lvmgrp /dev/vg_lsrs0vio 4096 25473 104337408
lvmvol /dev/vg_lsrs0vio lv_root 12800 104857600
lvmvol /dev/vg_lsrs0vio lv_home 10641 87171072
lvmvol /dev/vg_lsrs0vio lv_swap 2032 16646144
# Filesystems (only ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs are supported).
# Format: fs <device> <mountpoint> <fstype> [uuid=<uuid>] [label=<label>] [<attributes>]
fs /dev/mapper/vg_lsrs0vio-lv_home /home ext4 uuid=a5ec9996-ad9a-4975-9231-50187d1ba50b label= blocksize=4096 reserved_blocks=4% max_mounts=-1 check_interval=0d bytes_per_inode=16360 default_mount_options=user_xattr,acl options=rw,relatime,seclabel,barrier=1,dat
fs /dev/mapper/vg_lsrs0vio-lv_root / ext4 uuid=420c1768-e3b5-4b18-a67e-162f49d8e24e label= blocksize=4096 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=16384 default_mount_options=user_xattr,acl options=rw,relatime,seclabel,barrier=1,dat
fs /dev/sda2 /boot ext4 uuid=7df6992a-5802-478d-9b86-a1045ebeedde label= blocksize=1024 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=4095 default_mount_options=user_xattr,acl options=rw,relatime,seclabel,barrier=1,dat
swap /dev/mapper/vg_lsrs0vio-lv_swap uuid=0fa4fade-6441-4f5e-b2a8-9a4dbc5000d3 label=
jayfurmanek commented at 2015-02-11 00:03:¶
Yup, "dat" is truncated (see previous comment), but the mount actually fails on the "seclabel" option first, which is strange.
jayfurmanek commented at 2015-02-11 01:11:¶
My original system has SELinux installed, but it is set to permissive:
# getenforce
Permissive
Also, on the original server, I can mount things on the command line using the "-o seclabel" option
# mount -o seclabel /dev/sr0 /mnt
mount: block device /dev/sr0 is write-protected, mounting read-only
On the REAR recover console, I cannot mount with the "-o seclabel"
option. The version of the mount command is the same, of course.
Perhaps SELinux has to be present for that mount option to work?
Perhaps that option needs to be stripped if BACKUP_SELINUX_DISABLE is
"yes"?
jayfurmanek commented at 2015-02-11 02:15:¶
I've redone the mkrescue and rebooted a few times and have not been able to reacreate that truncated "data=ordered" issue.
disk /dev/sda 107374182400 msdos
part /dev/sda 4194304 1048576 primary boot,prep /dev/sda1
part /dev/sda 524288000 5242880 primary none /dev/sda2
part /dev/sda 106844651520 529530880 primary lvm /dev/sda3
lvmdev /dev/vg_lsrs0vio /dev/sda3 VzXJfz-1fSj-uz7p-Tf3E-8w5x-RoNQ-bVVIUo 208680960
lvmgrp /dev/vg_lsrs0vio 4096 25473 104337408
lvmvol /dev/vg_lsrs0vio lv_root 12800 104857600
lvmvol /dev/vg_lsrs0vio lv_home 10641 87171072
lvmvol /dev/vg_lsrs0vio lv_swap 2032 16646144
# Filesystems (only ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs are supported).
# Format: fs <device> <mountpoint> <fstype> [uuid=<uuid>] [label=<label>] [<attributes>]
fs /dev/mapper/vg_lsrs0vio-lv_home /home ext4 uuid=a5ec9996-ad9a-4975-9231-50187d1ba50b label= blocksize=4096 rese
rved_blocks=4% max_mounts=-1 check_interval=0d bytes_per_inode=16360 default_mount_options=user_xattr,acl options=
rw,relatime,seclabel,barrier=1,data=ordered
fs /dev/mapper/vg_lsrs0vio-lv_root / ext4 uuid=420c1768-e3b5-4b18-a67e-162f49d8e24e label= blocksize=4096 reserved
_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=16384 default_mount_options=user_xattr,acl options=rw,r
elatime,seclabel,barrier=1,data=ordered
fs /dev/sda2 /boot ext4 uuid=7df6992a-5802-478d-9b86-a1045ebeedde label= blocksize=1024 reserved_blocks=5% max_mou
nts=-1 check_interval=0d bytes_per_inode=4095 default_mount_options=user_xattr,acl options=rw,relatime,seclabel,ba
rrier=1,data=ordered
swap /dev/mapper/vg_lsrs0vio-lv_swap uuid=0fa4fade-6441-4f5e-b2a8-9a4dbc5000d3 label=
It looks ok now. It seems, that was an anomoly of sorts.
The "seclabel" option continues to be a problem, however.
I've tried just enabling SELinux on my original server (setting it to
"enforcing") and in the backup (setting BACKUP_SELINUX_DISABLE to 0),
but I must need more than that as the recover system always comes up
with selinux=0 in the cmdline. The thought was that perhaps SELinux
needs to be enabled for that seclabel option to work. I haven't been
able to confirm that.
jayfurmanek commented at 2015-02-12 19:55:¶
I can attempt a patch that will clip out that "seclabel" mount option
when SELinux has been disabled in the rescue environment.
And once the system reboots, post restore, the filesystem should get
mounted properly by the OS init facilities.
That "seclabel" mount option shouldn't really be needed in the rescue
enivornment as far as I can see.
What do you think? I've not tried this on RHEL x86 yet, but "seclabel" will show up as a mount option there too. I'd be surprised if this is ppc64 specific.
jayfurmanek commented at 2015-02-13 06:54:¶
This system does have the findmnt command, which seems to give all of the mount options, while the mount command doesn't seem to output as many:
# mount -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs | tr -s '[:blank:]' ' ' | cut -d ' ' -f 1,3,5,6
/dev/mapper/vg_lsrs0vio-lv_root / ext4 (rw)
/dev/sda2 /boot ext4 (rw)
/dev/mapper/vg_lsrs0vio-lv_home /home ext4 (rw)
# findmnt -alnv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs
/dev/mapper/vg_lsrs0vio-lv_root / ext4 rw,relatime,seclabel,barrier=1,data=ordered
/dev/sda2 /boot ext4 rw,relatime,seclabel,barrier=1,data=ordered
/dev/mapper/vg_lsrs0vio-lv_home /home ext4 rw,relatime,seclabel,barrier=1,data=ordered
I didn't want to just mask the problem by removing findmnt, so I added a line to./usr/share/rear/layout/save/GNU/Linux/23_filesystem_layout.sh that clips out the 'seclabel' option.
esac
# Remove parenthesis (from the traditional mount command output) from the list of options:
options=${options#(}
options=${options%)}
### Clip out seclabel
options=${options//seclabel,/} <---
echo -n " options=$options"
# Finish the current filesystem layout line with a newline character:
echo
done < <( eval $read_filesystems_command )
With this change, I can now recover just fine.
What do you think? Does this addition need to be conditional (on SELinux
settings, for example)?
jayfurmanek commented at 2015-02-17 18:19:¶
Any thoughts on my previous post?
I can create a pull request for that change, but I'd like some feedback.
It's a bit of a hack, but it avoids the problem in the rear recovery
console when findmnt lists the seclabel option, and it's pretty
harmless.
gdha commented at 2015-02-17 20:10:¶
@jayfurmanek as rear comes by default 'SELinux disabled' I think it does
not hurt to remove the seclabel from the options line. It is only used
to mount with the listed options in rear's recovery mode.
By aware that I've updated today the
usr/share/rear/layout/save/GNU/Linux/23_filesystem_layout.sh script
gdha commented at 2015-02-21 10:56:¶
@jayfurmanek is your problem fixed by the change in code?
gdha commented at 2015-02-23 13:53:¶
added to the release notes so we can close this issue
jayfurmanek commented at 2015-02-23 14:06:¶
It is indeed fixed on RHEL6, thanks.
[Export of Github issue for rear/rear.]