#2777 Issue closed: XFS: diskrestore.sh failed at 'mount: /mnt/local: ...' because of wrong mount options¶
Labels: support / question, no-issue-activity
JKuhns5739 opened issue at 2022-03-21 19:56:¶
Relax-and-Recover (ReaR) Issue Template¶
Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):
-
ReaR version ("/usr/sbin/rear -V"):
Relax-and-Recover 2.4 / Git -
OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):
OS_VENDOR=RedHatEnterpriseServer
OS_VERSION=8
- ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
OUTPUT=ISO
OUTPUT_URL=nfs://172.29.73.228/backups/rearMateBackup/
BACKUP=NETFS
BACKUP_URL=iso:///backup/
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/backups/tmp/*' '/backups/rearMateBackup/*' '/backups/rearLocalBackup/*' '/var/log/*' '/var/lib/libvirt/images/*' '/home/storage/loads/*' '/home/storage/patches/*' '/home/storage/backups/*')
export TMPDIR=/backups/tmp
TMPDIR=/backups/tmp
KEEP_BUILD_DIR=
KEEP_OLD_OUTPUT_COPY=y
ISO_DIR=/backups/rearLocalBackup
-
Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR):
PC -
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
x86 -
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
UEFI -
Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
local disk -
Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"):
# lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT
NAME KNAME PKNAME TRAN TYPE FSTYPE LABEL SIZE MOUNTPOINT
/dev/sda /dev/sda disk 2.2T
|-/dev/sda1 /dev/sda1 /dev/sda part vfat 512M /boot/efi
|-/dev/sda2 /dev/sda2 /dev/sda part xfs 1G /boot
`-/dev/sda3 /dev/sda3 /dev/sda part LVM2_member 2.2T
|-/dev/mapper/vg00-root /dev/dm-0 /dev/sda3 lvm xfs 20G /
|-/dev/mapper/vg00-opt /dev/dm-1 /dev/sda3 lvm xfs 10G /opt
|-/dev/mapper/vg00-var_log_audit /dev/dm-2 /dev/sda3 lvm xfs 4G /var/log/audit
|-/dev/mapper/vg00-var_log /dev/dm-3 /dev/sda3 lvm xfs 10G /var/log
|-/dev/mapper/vg00-var_tmp /dev/dm-4 /dev/sda3 lvm xfs 10G /var/tmp
|-/dev/mapper/vg00-var_lib_libvirt_images /dev/dm-5 /dev/sda3 lvm xfs 1.8T /var/lib/libvirt/images
|-/dev/mapper/vg00-var /dev/dm-6 /dev/sda3 lvm xfs 10G /var
|-/dev/mapper/vg00-home /dev/dm-7 /dev/sda3 lvm xfs 50G /home
|-/dev/mapper/vg00-tmp /dev/dm-8 /dev/sda3 lvm xfs 10G /tmp
|-/dev/mapper/vg00-backups /dev/dm-9 /dev/sda3 lvm xfs 50G /backups
`-/dev/mapper/vg00-upgrades /dev/dm-10 /dev/sda3 lvm xfs 200G /upgrades
- Description of the issue (ideally so that others can reproduce
it):
rear recover fails with the following error:
+++ echo -e 'Mounting filesystem /'
+++ mkdir -p /mnt/local/
+++ mount -o rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquota /dev/mapper/vg00-root /mnt/local/
mount: /mnt/local: wrong fs type, bad option, bad superblock on /dev/mapper/vg00-root, missing codepage or helper program, or other error.
++ (( 32 == 0 ))
-
Workaround, if any:
None -
Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
rear-OTT50-Host-1B-mkbackup.log (log of mkbackup used for recovery)
rear-OTT50-Host-1B-recover.log (recovery log with failure)
rear-OTT50-Host-1B-mkbackup.log
rear-OTT50-Host-1B-recover.log
jsmeix commented at 2022-03-22 08:15:¶
ReaR version 2.4 is rather old.
It was released in June 2018, see
https://github.com/rear/rear/blob/master/doc/rear-release-notes.txt
@JKuhns5739
please test if it works for you when you use
our current ReaR GitHub master code.
See the section
"Testing current ReaR upstream GitHub master code" in
https://en.opensuse.org/SDB:Disaster_Recovery
how you can try out our current ReaR GitHub master code
without conflicts with your already installed ReaR version.
In general I recommend to try out our latest GitHub master code
because the GitHub master code is the only place where we fix things
and if there are issues it helps when you use exactly the code
where we could fix things.
In general we at ReaR upstream do not support older ReaR versions.
We at ReaR upstream do not plain reject issues with older ReaR
versions
(e.g. we may answer easy to solve questions also for older ReaR
versions)
but we do not spend much time on issues with older ReaR versions
because
we do not (and cannot) fix issues in released ReaR versions.
Issues in released ReaR versions are not fixed by us (by ReaR
upstream).
Issues in released ReaR versions that got fixed in current ReaR
upstream
GitHub master code might be fixed (if the fix can be backported with
reasonable effort) by the Linux distributor wherefrom you got ReaR.
In case of RHEL you may contact Red Hat directly
provided you have an appropriate support contract.
The disk layout recreation script "diskrestore.sh" failed at
creating filesystem of type xfs with mount point / on
/dev/mapper/vg00-root
according to the following excerpt from
https://github.com/rear/rear/files/8318871/rear-OTT50-Host-1B-recover.log
+++ echo -e 'Creating filesystem of type xfs with mount point / on /dev/mapper/vg00-root.'
+++ wipefs --all --force /dev/mapper/vg00-root
/dev/mapper/vg00-root: 4 bytes were erased at offset 0x00000000 (xfs): 58 46 53 42
+++ mkfs.xfs -f -m uuid=07cb88d0-a9af-4bef-8114-f493be852c76 -i size=512 -d agcount=32 -s size=512 -i attr=2 -i projid32bit=1 -m crc=1 -m finobt=1 -b size=4096 -i maxpct=25 -d sunit=512 -d swidth=512 -l version=2 -l lazy-count=1 -n size=4096 -n version=2 -r extsize=4096 /dev/mapper/vg00-root
meta-data=/dev/mapper/vg00-root isize=512 agcount=32, agsize=163840 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1
data = bsize=4096 blocks=5242880, imaxpct=25
= sunit=64 swidth=64 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=5184, version=2
= sectsz=512 sunit=64 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
+++ LogPrint 'Mounting filesystem /'
+++ Log 'Mounting filesystem /'
++++ date '+%Y-%m-%d %H:%M:%S.%N '
+++ local 'timestamp=2022-03-21 18:40:50.013377686 '
+++ test 1 -gt 0
+++ echo '2022-03-21 18:40:50.013377686 Mounting filesystem /'
2022-03-21 18:40:50.013377686 Mounting filesystem /
+++ Print 'Mounting filesystem /'
+++ test 1
+++ echo -e 'Mounting filesystem /'
+++ mkdir -p /mnt/local/
+++ mount -o rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquota /dev/mapper/vg00-root /mnt/local/
mount: /mnt/local: wrong fs type, bad option, bad superblock on /dev/mapper/vg00-root, missing codepage or helper program, or other error.
++ (( 32 == 0 ))
++ true
+++ UserInput -I LAYOUT_CODE_RUN -p 'The disk layout recreation script failed' ...
So mkfs.xfs finished with exit code 0 so the created XFS should be
OK
but then mounting it failed so I guess mount fails because of some
"bad option".
I am not at all a sufficient XFS expert to understand the XFS specific
mount options.
Perhaps those mount options do not match how the XFS filesystem was
created?
jsmeix commented at 2022-03-22 08:21:¶
Since ReaR 2.5 we have MKFS_XFS_OPTIONS config
variables which could help in special cases,
see
https://github.com/rear/rear/pull/2005
and
https://github.com/rear/rear/issues/1998
and the issues mentioned therein for some
"not so funny fun with XFS" that we had in the past.
Regarding this issue here see in particuIar
https://github.com/rear/rear/issues/1998#issuecomment-857622163
JKuhns5739 commented at 2022-03-22 17:34:¶
I was able to recover the system by editing the disklayout.conf file and removing most of the xfs options. I first tried removing just the 'noquota' option as that is the only option not listed in the mount man page for xfs; however, the restore failed for the same reason. I went back to the original system and found that the RAID stripe unit and width were 64, not 512 as specified in the disklayout.conf file. After removing the stripe values the xfs was successfully created.
The suggestion to move to a new version could help since we could specify a limited set of XFS options. I would still like to understand why ReaR would create options in the disklayout.conf file that do align with the original system.
Failed options:
mount -o rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquota
Successful options:
mount -o rw,relatime,attr2,inode64
pcahyna commented at 2022-03-22 19:13:¶
- rear recover fails with the following error:
+++ echo -e 'Mounting filesystem /' +++ mkdir -p /mnt/local/ +++ mount -o rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquota /dev/mapper/vg00-root /mnt/local/ mount: /mnt/local: wrong fs type, bad option, bad superblock on /dev/mapper/vg00-root, missing codepage or helper program, or other error.
Unfortunately, the error message from mount is not very informative, but when this happens, the kernel sometimes prints a more specific error message to the log. Is there anything xfs-related in dmesg after this happens?
I went back to the original system and found that the RAID stripe unit and width were 64, not 512 as specified in the disklayout.conf file.
I suspect this is a misunderstanding. I tried creating a filesystem with
mkfs.xfs -d sunit=512 -d swidth=512 /dev/vdb
and it printed:
meta-data=/dev/vdb isize=512 agcount=16, agsize=163840 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1 bigtime=0 inobtcount=0
data = bsize=4096 blocks=2621440, imaxpct=25
= sunit=64 swidth=64 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=5184, version=2
= sectsz=512 sunit=64 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
It looks wrong, until one realizes that the message actually reads
sunit=64 swidth=64 blks. A block is 4096 bytes. 64 blks is thus
64*4096 = 262144. sunit on command line is specified in multiples of
512 bytes. 512*512 = 262144, so it is consistent.
JKuhns5739 commented at 2022-03-22 19:40:¶
Thanks, it is a misunderstanding on my part, which unfortunately makes the issue more confusing.
I did not see anything in dmesg that indicated an error. These are the only xfs specific log messages in the file.
kern :info : [ +3.953741] SGI XFS with ACLs, security attributes, quota, no debug enabled
kern :warn : [ +0.002342] XFS (dm-8): logbuf size must be greater than or equal to log stripe size
pcahyna commented at 2022-03-22 19:51:¶
kern :warn : [ +0.002342] XFS (dm-8): logbuf size must be greater than or equal to log stripe size
And that's it.
pcahyna commented at 2022-03-22 19:53:¶
What are the mount option on the original system? I mean the options that really get passed to mount (in fstab), not the filesystem creation options. Do you use the sunit= ,swidth= etc. options in normal operation?
JKuhns5739 commented at 2022-03-22 20:31:¶
The options are not specified in fstab.
[root@OTT50-Host-1B ~]# cat /etc/fstab
/dev/mapper/vg00-root / xfs defaults 0 0
UUID=ea026af3-32f8-4b7e-8d9c-a2ade4a00206 /boot xfs defaults 0 0
UUID=F316-EA90 /boot/efi vfat defaults,uid=0,gid=0,umask=077,shortname=winnt 0 2
/dev/mapper/vg00-home /home xfs defaults,nodev 0 0
/dev/mapper/vg00-opt /opt xfs defaults 0 0
/dev/mapper/vg00-tmp /tmp xfs defaults,nodev,nosuid,noexec 0 0
/dev/mapper/vg00-var /var xfs defaults 0 0
/dev/mapper/vg00-var_lib_libvirt_images /var/lib/libvirt/images xfs defaults 0 0
/dev/mapper/vg00-var_log /var/log xfs defaults,nodev,nosuid,noexec 0 0
/dev/mapper/vg00-var_log_audit /var/log/audit xfs defaults,nodev,nosuid,noexec 0 0
/dev/mapper/vg00-var_tmp /var/tmp xfs defaults,nodev,nosuid,noexec 0 0
tmpfs /dev/shm tmpfs nodev,nosuid,noexec 0 0
/dev/mapper/vg00-backups /backups xfs defaults 0 0
/dev/mapper/vg00-upgrades /upgrades xfs defaults 0 0
jsmeix commented at 2022-03-23 08:11:¶
A side note FYI regarding
https://github.com/rear/rear/issues/2777#issuecomment-1075532334
that mkfs.xfs -d sunit=512 results sunit=64 in the output, cf.
https://github.com/rear/rear/issues/1998#issuecomment-445220483
I think this is the code that deals with it (block size 4096 versus sector size 512)
function xfs_parse
...
# Handle mkfs.xfs special cases
# sunit & swidth are set in blocks
if [ $var = "sunit" ] || [ $var = "swidth" ]; then
val=$((val*$block_size/512))
fi
which is currently at
https://github.com/rear/rear/blob/master/usr/share/rear/lib/filesystems-functions.sh#L190
There is a hardcoded 512 so I wonder if that is actually always a fixed
value.
E.g. I wonder what happens on disks with 4096 sector size like my disk:
# parted -s /dev/sda unit B print
Model: ATA TOSHIBA MQ01ABF0 (scsi)
Disk /dev/sda: 500107862016B
Sector size (logical/physical): 512B/4096B
Will the logical sector size be used or the physical sector size
or will a fixed kernel internal (logical) sector size 512 be used?
In lib/layout-functions.sh we have two hardcoded
local block_size=512
in particular this one
# Linux always considers sectors to be 512 bytes long. See the note in the
# kernel source, specifically, include/linux/types.h regarding the sector_t
# type for details.
local block_size=512
currently at
https://github.com/rear/rear/blob/master/usr/share/rear/lib/layout-functions.sh#L702
That "Linux always considers sectors to be 512 bytes" is also mentioned
at
https://docs.huihoo.com/doxygen/linux/kernel/3.7/include_2linux_2types_8h.html
Linux always considers sectors to be 512 bytes long
independently of the devices real block size.
This is for "Linux Kernel 3.7.1" and it is still the same for 5.17.0
so one could assume that kernel internal (logical) sector size 512 is a
fixed value.
jsmeix commented at 2022-03-23 08:53:¶
Regarding how a filesystem is created during "rear recover"
versuse how the created filesystem is mounted during "rear recover":
A filesystem is created by the create_fs function in
layout/prepare/GNU/Linux/131_include_filesystem_code.sh
that implements different case code for different filesystems.
A created filesystem is mounted by the mount_fs function in
layout/prepare/GNU/Linux/133_include_mount_filesystem_code.sh
that extracts mount options from the matching fs entry in
disklayout.conf
and also implements different case code for certain filesystems
but for XFS the default case is used.
Regarding how the mount options for a filesystem are stored in
disklayout.conf:
This happens in layout/save/GNU/Linux/230_filesystem_layout.sh
Because
https://github.com/rear/rear/files/8318869/rear-OTT50-Host-1B-mkbackup.log
contains
2022-03-21 13:44:57.428361528 Including layout/save/GNU/Linux/230_filesystem_layout.sh
2022-03-21 13:44:57.429669431 Begin saving filesystem layout
2022-03-21 13:44:57.432069537 Saving filesystem layout (using the findmnt command).
2022-03-21 13:44:57.461560988 Processing filesystem 'xfs' on '/dev/mapper/vg00-backups' mounted at '/backups'
2022-03-21 13:44:57.640448151 Processing filesystem 'xfs' on '/dev/mapper/vg00-home' mounted at '/home'
2022-03-21 13:44:57.744970477 Processing filesystem 'xfs' on '/dev/mapper/vg00-opt' mounted at '/opt'
2022-03-21 13:44:57.887880604 Processing filesystem 'xfs' on '/dev/mapper/vg00-root' mounted at '/'
2022-03-21 13:44:58.074530745 Processing filesystem 'xfs' on '/dev/mapper/vg00-tmp' mounted at '/tmp'
2022-03-21 13:44:58.169939386 Processing filesystem 'xfs' on '/dev/mapper/vg00-upgrades' mounted at '/upgrades'
2022-03-21 13:44:58.256965993 Processing filesystem 'xfs' on '/dev/mapper/vg00-var' mounted at '/var'
2022-03-21 13:44:58.350765059 Processing filesystem 'xfs' on '/dev/mapper/vg00-var_lib_libvirt_images' mounted at '/var/lib/libvirt/images'
2022-03-21 13:44:58.548745820 Processing filesystem 'xfs' on '/dev/mapper/vg00-var_log' mounted at '/var/log'
2022-03-21 13:44:58.645453973 Processing filesystem 'xfs' on '/dev/mapper/vg00-var_log_audit' mounted at '/var/log/audit'
2022-03-21 13:44:58.702769030 Processing filesystem 'xfs' on '/dev/mapper/vg00-var_tmp' mounted at '/var/tmp'
2022-03-21 13:44:58.791070010 Processing filesystem 'vfat' on '/dev/sda1' mounted at '/boot/efi'
2022-03-21 13:44:58.828735651 Processing filesystem 'xfs' on '/dev/sda2' mounted at '/boot'
2022-03-21 13:44:58.879783279 End saving filesystem layout
the read_filesystems_command in 230_filesystem_layout.sh is
findmnt -mnrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs
so the mount options should be basically what findmnt shows as OPTIONS
as far as I currently see what the code does.
jsmeix commented at 2022-03-23 09:01:¶
@JKuhns5739
please attach your original (unmodified) disklayout.conf file
and what
findmnt -mnrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs
results on your system - if possible what it results on your original
system
if you still have your original system running - or on another
sufficiently simliar
system if you have several such systems with similar XFS running.
A debug log file of "rear -d -D mkbackup" on your original system
would also be hepful for us to better understand things.
pcahyna commented at 2022-03-23 09:29:¶
There is a hardcoded 512 so I wonder if that is actually always a fixed value.
@jsmeix mkfs.xfs(8) man page says:
sunit=value
This is used to specify the stripe unit for a RAID
device or a logical volume. The value has to be
specified in 512-byte block units.
It says nothing about any relation to the physical sector size. But you can try it, if you have such a disk - documentation can be misleading from time to time.
jsmeix commented at 2022-03-23 09:32:¶
Sigh - sorry! Again I missed to just read the matching "fine manual"
;-)
When the man page states a fixed value has to be used that vaule must be
hardcoded.
pcahyna commented at 2022-03-23 12:41:¶
I tested it anyway on a 4096-sector-size disk (both physical and logical).
# mkfs.xfs -d sunit=512 -d swidth=512
meta-data=/dev/dasdb1 isize=512 agcount=16, agsize=1126912 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1
data = bsize=4096 blocks=18030576, imaxpct=25
= sunit=64 swidth=64 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=8803, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
I.e. same values as before, despite the fact that XFS correctly
recognizes the larger sector size sectsz=4096, it is still using 512
bytes as the multiplier for sunit, otherwise the values would not
match. So I conclude that the fine manual is accurate (this time).
JKuhns5739 commented at 2022-03-23 12:49:¶
The system has been restored with Ansible scripts so it should be back to its original values. This data also aligns with similar systems we have.
[root@OTT50-Host-1B ~]# findmnt -mnrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs
/dev/mapper/vg00-root / xfs rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
/dev/sda2 /boot xfs rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
/dev/mapper/vg00-opt /opt xfs rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
/dev/sda1 /boot/efi vfat rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro
/dev/mapper/vg00-backups /backups xfs rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
/dev/mapper/vg00-var /var xfs rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
/dev/mapper/vg00-tmp /tmp xfs rw,nosuid,nodev,noexec,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
/dev/mapper/vg00-home /home xfs rw,nodev,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
/dev/mapper/vg00-upgrades /upgrades xfs rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
/dev/mapper/vg00-var_tmp /var/tmp xfs rw,nosuid,nodev,noexec,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
/dev/mapper/vg00-var_log /var/log xfs rw,nosuid,nodev,noexec,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
/dev/mapper/vg00-var_log_audit /var/log/audit xfs rw,nosuid,nodev,noexec,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
/dev/mapper/vg00-var_lib_libvirt_images /var/lib/libvirt/images xfs rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
Requested files:
disklayout.conf.txt
rear-OTT50-Host-1B-Debug.log
jsmeix commented at 2022-03-23 13:50:¶
The filesystem options in
https://github.com/rear/rear/files/8333052/disklayout.conf.txt
are (cut and sorted)
rw,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
rw,nodev,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquota
rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquota
rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquota
rw,nosuid,nodev,noexec,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquota
rw,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
rw,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
rw,nosuid,nodev,noexec,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquota
rw,nosuid,nodev,noexec,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquota
rw,nosuid,nodev,noexec,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquota
rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquota
rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro
rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquota
In layout/save/GNU/Linux/230_filesystem_layout.sh the 'seclabel'
option is cut, cf.
https://github.com/rear/rear/blob/master/usr/share/rear/layout/save/GNU/Linux/230_filesystem_layout.sh#L246
so the filesystem options in the findmnt output without the 'seclabel'
option in
https://github.com/rear/rear/issues/2777#issuecomment-1076338303
are (cut and sorted)
rw,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
rw,nodev,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
rw,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
rw,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
rw,nosuid,nodev,noexec,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
rw,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
rw,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
rw,nosuid,nodev,noexec,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
rw,nosuid,nodev,noexec,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
rw,nosuid,nodev,noexec,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
rw,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro
rw,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota
The only remaining difference is that in the findmnt output there is always
... logbufs=8,logbsize=256k ...
while in disklayout.conf there is both
... logbufs=8,logbsize=256k ...
... logbufs=8,logbsize=32k ...
In
https://github.com/rear/rear/files/8333076/rear-OTT50-Host-1B-Debug.log
there is (excerpts):
+ source /usr/share/rear/layout/save/GNU/Linux/230_filesystem_layout.sh
...
++ echo -n ' options=rw,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota'
++ echo -n ' options=rw,nodev,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota'
++ echo -n ' options=rw,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota'
++ echo -n ' options=rw,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota'
++ echo -n ' options=rw,nosuid,nodev,noexec,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota'
++ echo -n ' options=rw,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota'
++ echo -n ' options=rw,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota'
++ echo -n ' options=rw,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota'
++ echo -n ' options=rw,nosuid,nodev,noexec,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota'
++ echo -n ' options=rw,nosuid,nodev,noexec,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota'
++ echo -n ' options=rw,nosuid,nodev,noexec,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota'
++ echo -n ' options=rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro'
++ echo -n ' options=rw,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota'
so the disklayout.conf that was created by this run of "rear mkbackup"
always has
... logbufs=8,logbsize=256k ...
which matches what there is in the findmnt output.
There is no logbsize=32 in
https://github.com/rear/rear/files/8333076/rear-OTT50-Host-1B-Debug.log
I assume that
https://github.com/rear/rear/files/8333052/disklayout.conf.txt
is the original (unmodified) disklayout.conf file that
caused the "rear recover" failure in this issue here
and I assume the filesystem options in that disklayout.conf file
also match what there is in the findmnt output when it was run
at that time on that system where that disklayout.conf file was made.
I cannot explain why the filesystem options in the findmnt output
at that time on that system where that disklayout.conf file was made
seem to not have matched what the xfs_info command did output, cf.
https://github.com/rear/rear/blob/master/usr/share/rear/layout/save/GNU/Linux/230_filesystem_layout.sh#L225
During "rear mkrescue/mkbackup" the script
layout/save/GNU/Linux/230_filesystem_layout.sh
calls 'xfs_info' and stores its output in
var/lib/rear/layout/xfs/XFS_DEVICE.xfs
files where XFS_DEVICE is the basename of the device
where the XFS filesystem is (e.g. sda2), cf.
https://github.com/rear/rear/blob/master/usr/share/rear/conf/default.conf#L596
@JKuhns5739
perhaps you still have your original xfs_info command output files
that match your above attached
https://github.com/rear/rear/files/8333052/disklayout.conf.txt
If yes, could you please also attach them?
JKuhns5739 commented at 2022-03-23 15:32:¶
I don't have the original command output. I did create a backup log from Host-1B mate unit, Host-1A, which is an identical system. The log file includes both logbsizes of 32k and 256k.
+++ mount -t xfs
++ '[' -n '/dev/mapper/vg00-root on / type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquota)
/dev/sda2 on /boot type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquota)
/dev/mapper/vg00-opt on /opt type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquota)
/dev/mapper/vg00-tmp on /tmp type xfs (rw,nosuid,nodev,noexec,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquota)
/dev/mapper/vg00-var on /var type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquota)
/dev/mapper/vg00-upgrades on /upgrades type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota)
/dev/mapper/vg00-home on /home type xfs (rw,nodev,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquota)
/dev/mapper/vg00-backups on /backups type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota)
/dev/mapper/vg00-var_tmp on /var/tmp type xfs (rw,nosuid,nodev,noexec,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquota)
/dev/mapper/vg00-var_log on /var/log type xfs (rw,nosuid,nodev,noexec,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquota)
/dev/mapper/vg00-var_log_audit on /var/log/audit type xfs (rw,nosuid,nodev,noexec,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquota)
/dev/mapper/vg00-var_lib_libvirt_images on /var/lib/libvirt/images type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquota)
/dev/mapper/vg00-var on /var/lib/containers/storage/overlay type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquota)' ']'
jsmeix commented at 2022-03-24 12:58:¶
In
https://github.com/rear/rear/files/8334320/rear-OTt50-Host-1A.log
the options that mount -t xfs shows are the same as the options that
findmnt -mnrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t xfsshows
and the latter ones get stored in disklayout.conf
except the removed seclabel option
and except the duplicated mounted
/dev/mapper/vg00-var on /var ...
/dev/mapper/vg00-var on /var/lib/containers/storage/overlay ...
where the second one ("overlay") is skipped by
230_filesystem_layout.sh at
"# Remove duplicate lines for the same device" which is currently at
https://github.com/rear/rear/blob/master/usr/share/rear/layout/save/GNU/Linux/230_filesystem_layout.sh#L41
So all looks well from what I can see in
https://github.com/rear/rear/files/8334320/rear-OTt50-Host-1A.log
So it still looks as if mkfs.xfs (re)-creates the XFS with options
from what is stored in the var/lib/rear/layout/xfs/XFS_DEVICE.xfs
files
that then do not work for mounting it with the options in
disklayout.conf
as shown in the "rear recover" log excerpt from
https://github.com/rear/rear/files/8318871/rear-OTT50-Host-1B-recover.log
in
https://github.com/rear/rear/issues/2777#issuecomment-1074861999
But we don't have a dmesg output from the "rear recover" time
2022-03-21 18:40
that may show more about what actually went wrong with XFS there.
We only have
https://github.com/rear/rear/files/8327383/rear-OTT50-Host-1B-dmesg.log
that is from a later time 2022-03-22 13:45
likely after the manual recovery of the system by editing
disklayout.conf
and removing most of the xfs options in
https://github.com/rear/rear/issues/2777#issuecomment-1075428019
Perhaps
https://github.com/rear/rear/issues/2777#issuecomment-1075428019
points to a possible way to make "rear recover" more robust
against subtle things with XFS mount options:
Currently during "rear recover" it mounts XFS filesystems
with all those "fancy" options that have been reported by
mount -t xfs
and/or findmnt -mnrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t xfs
at the time when "rear mkbackup" was run on the original system.
Probably in most cases all those "fancy" options are not needed
to mount the (re)-created XFS during "rear recover"
because all what is needed during "rear recover" is to get
the filesystems mounted "sufficiently OK" so that it works
to restore the files from backup into the empty (re)-created
filesystems.
After "rear recover" the recreated system will be rebooted
and then its filesystems get mounted according to what is
stored in the restored config files from the backup.
So a possible way to make "rear recover" more robust
against subtle things with XFS mount options
could be to
first try to mount the (re)-created XFS with all those "fancy" options
but when this fails then try to mount it without any options as
fallback
and only if also this fails stop with an error,
cf. "Dirty hacks welcome" in
https://github.com/rear/rear/wiki/Coding-Style
So in the mount_fs() function in
layout/prepare/GNU/Linux/133_include_mount_filesystem_code.sh
we may add a separated case for $fstype (xfs)
for example like (totally untested offhanded proposal):
(xfs)
(
echo "mkdir -p $TARGET_FS_ROOT$mountpoint"
echo "if ! mount $mountopts $device $TARGET_FS_ROOT$mountpoint ; then"
echo " # Trying to mount without mount options as fallback"
echo " mount $device $TARGET_FS_ROOT$mountpoint"
echo "fi"
) >> "$LAYOUT_CODE"
;;
Perhaps some standard XFS mount options might be useful also in the
fallback case
but I am not at all a sufficient XFS expert to understand XFS specific
mount options.
I cannot imagine what should go worse with such a fallback attempt.
If the fallback works to get the backup restored things should be OK,
if not "rear recover" behaves as before i.e. it fails (only a bit
different).
@pcahyna
what do you think about such kind of fallback behaviour?
github-actions commented at 2022-06-05 03:00:¶
Stale issue message
[Export of Github issue for rear/rear.]