#2398 Issue `closed`: ReaR uses NFS4 with AIX mount¶

Labels: support / question, fixed / solved / done

mahmoudrafat opened issue at 2020-05-14 13:17:¶

Relax-and-Recover (ReaR) Issue Template¶

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

ReaR version ("/usr/sbin/rear -V"): 2.3
OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"):
SUSE Linux Enterprise Server 12 SP3
ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

OUTPUT=ISO
OUTPUT_URL=nfs://IP/share
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://IP/share
REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" snapper chattr lsattr )
COPY_AS_IS=( "${COPY_AS_IS[@]}" /usr/lib/snapper/installation-helper /etc/snapper/config-templates/default 
BACKUP_PROG_INCLUDE=( '/var/lib/mysql/*' '/var/tmp/*' '/var/spool/*' '/var/lib/mailman/*' '/boot/grub2/powerpc-ieee1275/*' '/var/opt/*' '/var/cache/*' '/var/lib/machines/*' '/srv/*' '/var/lib/mariadb/*' '/tmp/*' '/var/lib/pgsql/*' '/opt/*' '/var/lib/libvirt/images/*' '/usr/local/*' '/var/log/*' '/var/lib/named/*' '/home/*' )
POST_RECOVERY_SCRIPT=( 'if snapper --no-dbus -r $TARGET_FS_ROOT get-config | grep -q "^QGROUP.*[0-9]/[0-9]" ; then snapper --no-dbus -r $TARGET_FS_ROOT set-config QGROUP= ; snapper --no-dbus -r $TARGET_FS_ROOT setup-quota && echo snapper setup-quota done || echo snapper setup-quota failed ; else echo snapper setup-quota not used ; fi' )
EXCLUDE_MOUNTPOINTS=( '/MEG-sysbackup/*' '/run/user/*' '/sapmnt' '/usr/sap' )

Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR):
SUSE on IBM power
Description of the issue (ideally so that others can reproduce it):

I share a directory from AIX machine to use it as backup for ReaR,
i can mount it using command

mount -o vers=3 ip:/share /mount_point

but when i using ReaR it seems that it use NFS Version=4
and produce error. like below.

The log file in /var/log/rear/xxx.log

mkdir: created directory '/tmp/rear.rjbDv6M41RuA6rS/outputfs'
2020-05-14 16:16:38.263945559 Mounting with 'mount -v -t nfs -o rw,noatime IP:/share /tmp/rear.rjbDv6M41RuA6rS/outputfs'
mount.nfs: mount(2): Remote I/O error
mount.nfs: Remote I/O error

Can anyone help me?
thanks..

gozora commented at 2020-05-14 13:21:¶

Hi @mahmoudrafat,

NFS version 4 is now Linux default, so it is not ReaR making trouble.
Try to add BACKUP_OPTIONS="nfsvers=3,nolock" to your local.conf (site.conf).

V.

gozora commented at 2020-05-14 13:24:¶

Hmm, now reading more carefully I see that you've already did that.

Can you provide full /var/log/rear/rear.log, so we can see what is going on ?

V.

jsmeix commented at 2020-05-14 15:15:¶

@mahmoudrafat
more precisely we need the full debug log, i.e. what you get
with rear -d -D mkrescue or rear -d -D mkbackup, see also
"Debugging issues with Relax-and-Recover" in
https://en.opensuse.org/SDB:Disaster_Recovery

Note that the full debug log may contain private information
about your business environment like server names or
even some real secrets so you may as a first step cut away
what is not needed but what must stay is the full debug log
of all parts that are related to mounting your AIX NFS share.

For comparison on my SLES12-SP5 x86_64 test system
where I use this etc/rear/local.conf

OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://192.168.122.1/nfs
SSH_ROOT_PASSWORD="rear"
USE_DHCLIENT="yes"
PROGRESS_MODE=plain
PROGRESS_WAIT_SECONDS=3
KEEP_BUILD_DIR="yes"
FIRMWARE_FILES=( no )

what I get in the log for "rear -d -D mkrescue"

+ source /root/rear.github.master/usr/share/rear/prep/NETFS/default/060_mount_NETFS_path.sh
...
++ mount_url nfs://192.168.122.1/nfs /tmp/rear.3EfnQShLll41m14/outputfs nfsvers=3,nolock
...
++ mount_cmd='mount -v -t nfs -o nfsvers=3,nolock 192.168.122.1:/nfs /tmp/rear.3EfnQShLll41m14/outputfs'
++ Log 'Mounting with '\''mount -v -t nfs -o nfsvers=3,nolock 192.168.122.1:/nfs /tmp/rear.3EfnQShLll41m14/outputfs'\'''
++ echo '2020-05-14 12:06:59.728550142 Mounting with '\''mount -v -t nfs -o nfsvers=3,nolock 192.168.122.1:/nfs /tmp/rear.3EfnQShLll41m14/outputfs'\'''
2020-05-14 12:06:59.728550142 Mounting with 'mount -v -t nfs -o nfsvers=3,nolock 192.168.122.1:/nfs /tmp/rear.3EfnQShLll41m14/outputfs'
++ eval mount -v -t nfs -o nfsvers=3,nolock 192.168.122.1:/nfs /tmp/rear.3EfnQShLll41m14/outputfs
+++ mount -v -t nfs -o nfsvers=3,nolock 192.168.122.1:/nfs /tmp/rear.3EfnQShLll41m14/outputfs
mount.nfs: trying 192.168.122.1 prog 100003 vers 3 prot TCP port 2049
mount.nfs: trying 192.168.122.1 prog 100005 vers 3 prot UDP port 20048
mount.nfs: timeout set for Thu May 14 12:08:59 2020
mount.nfs: trying text-based options 'nfsvers=3,nolock,addr=192.168.122.1'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: prog 100005, trying vers=3, prot=17

where mounting a NFS share works for me
that is on an openSUSE Leap 15.1 NFS server.

mahmoudrafat commented at 2020-05-14 18:50:¶

rear-test.log

Hi, Thanks for your response,
i attached the error part only because the log is too big more than 1000 page!!
when i mount the shared directory with "mount -o nfsver=3 IP/share" it is ok.
may i change the rear mount command to use that command instead of mount -v -t nfs -o rw,noatime" ?
Note: that problem shows only when using AIX NFS server i did it when using Redhat NFS server and it is ok.

gozora commented at 2020-05-14 18:58:¶

Hi @mahmoudrafat,

i attached the error part only because the log is too big more than 1000 page!!

You can attach files directly to this conversation by drag-n-drop, no need to copy + paste file content.

may i change the rear mount command to use that command instead of mount -v -t nfs -o rw,noatime" ?

You can, this right is granted to you by GPL ;-)

I'm just wondering why BACKUP_OPTIONS does not work for you, because content of BACKUP_OPTIONS is normally passed to mount command by ReaR, so I'd still like to see full output from rear -d -D mkrescue as @jsmeix proposed in https://github.com/rear/rear/issues/2398#issuecomment-628702055.

V.

mahmoudrafat commented at 2020-05-15 13:28:¶

rear-test.log

Hi, sorry for late due to time zone,
I uploaded the log file for your review.

You can, this right is granted to you by GPL ;-)

Can you please explain how to do that?

Thanks.

gozora commented at 2020-05-15 13:52:¶

Hmm, I always thought that if OUTPUT_OPTIONS is not explicitly set, it somehow takes over content from BACKUP_OPTIONS, but this is only true if following condition https://github.com/rear/rear/blob/f6e9fbca381ffd7ba14cb3685c64ecad3f1b9585/usr/share/rear/prep/default/020_translate_url.sh#L47-L52
is matched.

@mahmoudrafat can you please add following line to your local.conf and try to re-run rear mkbackup

OUTPUT_OPTIONS="nfsvers=3,nolock"

V.

mahmoudrafat commented at 2020-05-15 23:55:¶

@gozora
I did what you say and it make ISO image but the size is only 92M for ISO and 166MB for tar file, that doesn't make sense because the used space for the root is 22GB!
Can you please tell me how to change mount option in rear configuration?
i need to test it with "mount -o nfsvers=3 " only

82104704-3dcbd800-9718-11ea-9370-20a2fc9b2a51

82104703-3c9aab00-9718-11ea-80c3-afc3013c2e53

schabrolles commented at 2020-05-16 08:14:¶

@mahmoudrafat

I think you forget to add some specific option needed by Suse when BTRFS and subvolume are used. Try to use the following as local.conf

OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
OUPUT_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://192.168.122.1/nfs
SSH_ROOT_PASSWORD="rear"
USE_DHCLIENT="yes"
PROGRESS_MODE=plain
PROGRESS_WAIT_SECONDS=3
KEEP_BUILD_DIR="yes"
FIRMWARE_FILES=( no )
REAR_INITRD_COMPRESSION=lzma

#SLES BTRFS specific
REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" snapper chattr lsattr )
COPY_AS_IS=( "${COPY_AS_IS[@]}" /usr/lib/snapper/installation-helper /etc/snapper/config-templates/default )

for subvol in $(findmnt -n -r -t btrfs | cut -d ' ' -f 1 | grep -v '^/$' | egrep -v 'snapshots|crash') ; do
        BACKUP_PROG_INCLUDE=( "${BACKUP_PROG_INCLUDE[@]}" "$subvol" )
done

POST_RECOVERY_SCRIPT=( 'if snapper --no-dbus -r $TARGET_FS_ROOT get-config | grep -q "^QGROUP.*[0-9]/[0-9]" ; then snapper --no-dbus -r $TARGET_FS_ROOT set-config QGROUP= ; snapper --no-dbus -r $TARGET_FS_ROOT setup-quota && echo snapper setup-quota done || echo snapper setup-quota failed ; else echo snapper setup-quota not used ; fi' )

mahmoudrafat commented at 2020-05-17 15:32:¶

@schabrolles
I used the example from the docs and i don't think this is the problem because i made it on VM machine with the same configuration.
the problem with NFS share from AIX to SUSE.

schabrolles commented at 2020-05-17 15:57:¶

@mahmoudrafat, compare to the one included into the doc I added OUPUT_OPTIONS="nfsvers=3,nolock" (proposed by @gozora) ...

Although, did you apply the following command on the AIX machine ?

nfso -po nfs_use_reserved_ports=1
nfso -po portcheck=1

mahmoudrafat commented at 2020-05-18 10:34:¶

@schabrolles, I did what @gozora Said and added OUPUT_OPTIONS="nfsvers=3,nolock"
But it make an 150MB ISO only, see the previous comment.

Although, did you apply the following command on the AIX machine ?

I will check it.

jsmeix commented at 2020-05-18 13:06:¶

Now I see why

OUTPUT_OPTIONS="nfsvers=3,nolock"

(it is OUTPUT_OPTIONS not BACKUP_OPTIONS)
is needed when one has

OUTPUT=ISO
OUTPUT_URL=nfs://IP/share
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://IP/share

i.e. when OUTPUT_URL is explicitly specified:

This is because prep/default/020_translate_url.sh contains

if [[ -z "$OUTPUT_OPTIONS" ]] ; then
    if [[ -z "$OUTPUT_URL" && -z "$OUTPUT_MOUNTCMD" ]] ; then
        ### There can be cases where it's intentionally empty.
        OUTPUT_OPTIONS=$BACKUP_OPTIONS
    fi
fi

https://github.com/rear/rear/blob/master/usr/share/rear/prep/default/020_translate_url.sh#L48
so OUTPUT_OPTIONS inherits from BACKUP_OPTIONS
only if OUTPUT_URL is not explicitly specified.

jsmeix commented at 2020-05-18 13:22:¶

@mahmoudrafat
to see what actually happens while the backup is made
inspect your backup.log file.
For debug information in the log file
you must run rear with debug logging enabled
like rear -d -D mkrescue or rear -d -D mkbackup

mahmoudrafat commented at 2020-05-20 08:58:¶

Hi,
Ok i will try what you said and apply it after the holidays ends (we have one week off).
Many thanks.

mahmoudrafat commented at 2020-06-04 12:11:¶

Hi Again,
Today i test the recover but it stuck at this point i don't know wheres the problem is because small size of backup or something else, i investigated the log file but didn't find anything wrong.
does anyone face the same problem?
@jsmeix

rear

jsmeix commented at 2020-06-04 13:10:¶

@mahmoudrafat
at this point "rear recover" is about to recreate the disk storage layout
which happens before the backup of the files is restored
so this new issue in your recent
https://github.com/rear/rear/issues/2398#issuecomment-638808558
has nothing to do with what is in the backup.

To get an initial basic understanding about
your particular actual disk storage layout and
what "rear mkrescue/mkbackup" finds out about it
we need the output of

lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT

or if that is not possible on SLES12 the output of

lsblk -io NAME,KNAME,FSTYPE,SIZE,MOUNTPOINT

or if that is also not possible at least the output of plain

lsblk

and additionally we need your particular

var/lib/rear/layout/disklayout.conf

file.

Because you use ReaR 2.3 which is meanwhile rather old
(from our ReaR upstream developer's point of view) and
because we at ReaR upstream do not support old ReaR releases
I would seriously recommend to try out our current ReaR upstream
GitHub master code from within a separated directory as a test
to find out if things work better with our current ReaR upstream
master code compared to your installed ReaR version,
see the section
"Testing current ReaR upstream GitHub master code" in
https://en.opensuse.org/SDB:Disaster_Recovery
how to do that.
To run git clone https://github.com/rear/rear.git you need
the RPM package git-core that provides /usr/bin/git.

mahmoudrafat commented at 2020-06-10 09:19:¶

Thanks For your help, i made the backup and restore it without any problem, here is the final config file for your attention.
if there is no question you can close it.
Thanks again for your great support.
"/etc/rear/local.conf" file

OUTPUT=ISO
OUTPUT_URL=nfs://IP/Share
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://IP/Share
OUTPUT_OPTIONS="nfsvers=3,nolock"
REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" snapper chattr lsattr )
COPY_AS_IS=( "${COPY_AS_IS[@]}" /usr/lib/snapper/installation-helper /etc/snapper/config-templates/default )
BACKUP_PROG_INCLUDE=( '/var/lib/mysql/*' '/var/tmp/*' '/var/spool/*' '/var/lib/mailman/*' '/boot/grub2/powerpc-ieee1275/*' '/var/opt/*' '/var/cache/*' '/var/lib/machines/*' '/srv/*' '/var/lib/mariadb/*' '/tmp/*' '/var/lib/pgsql/*' '/opt/*' '/var/lib/libvirt/images/*' '/usr/local/*' '/var/log/*' '/var/lib/named/*' '/home/*' )
POST_RECOVERY_SCRIPT=( 'if snapper --no-dbus -r $TARGET_FS_ROOT get-config | grep -q "^QGROUP.*[0-9]/[0-9]" ; then snapper --no-dbus -r $TARGET_FS_ROOT set-config QGROUP= ; snapper --no-dbus -r $TARGET_FS_ROOT setup-quota && echo snapper setup-quota done || echo snapper setup-quota failed ; else echo snapper setup-quota not used ; fi' )
EXCLUDE_MOUNTPOINTS=( '/POint/*' )
EXCLUDE_VG=( 'VG_NAME' )
AUTOEXCLUDE_MULTIPATH=n

jsmeix commented at 2020-06-15 08:13:¶

@mahmoudrafat
thank you for your feedeback that you got things working for your case
and for your etc/rear/local.conf that shows how you made it working.

[Export of Github issue for rear/rear.]

#2398 Issue closed: ReaR uses NFS4 with AIX mount¶