#991 Issue closed: mount_NETFS_path issue¶
Labels: support / question, fixed / solved / done
gigawatts opened issue at 2016-08-30 19:15:¶
- Relax-and-Recover 1.17.2 / Git
- OS_VENDOR=RedHatEnterpriseServer OS_VERSION=6
- Issue: We define a custom
NETFS_UMOUNTCMDvariable in our site.conf, with a function that basically spits out a backup complete message, along with a timestamp and where the backup was stored so that we can insert that result into a database. The problem is that it seems during thebackup/NETFS/default/10_mount_NETFS_path.shscript step, it is trying to run this above custom defined UMOUNT command instead of a stock MOUNT command, even though theNETFS_MOUNTCMDvariable is not defined. See the Mounting with line below from my log. Besides failing to mount, since it is running the wrong command, it does not error out, and continues writing the tar.gz file to the /tmp/rear.xxxx directory, filling up the /tmp filesystem.
2016-08-29 16:07:28 Including output/default/98_umount_output_dir.sh
2016-08-29 16:07:28 Unmounting '/tmp/rear.SEVLuCsI5908TRZ/outputfs'
NFSv4 mount point detected
nfs-server.example.com:/rear-backups/Linux umounted
rmdir: removing directory, `/tmp/rear.SEVLuCsI5908TRZ/outputfs'
2016-08-29 16:07:28 Finished running 'output' stage in 1 seconds
2016-08-29 16:07:28 Running 'backup' stage
2016-08-29 16:07:28 Including backup/default/00_valid_backup_methods.sh
2016-08-29 16:07:28 Including backup/NETFS/default/10_mount_NETFS_path.sh
mkdir: created directory `/tmp/rear.SEVLuCsI5908TRZ/outputfs'
2016-08-29 16:07:28 Mounting with 'custom_umount /tmp/rear.SEVLuCsI5908TRZ/outputfs'
2016-08-29 16:07:28 Backup_Result|nfs://nfs-server.example.com:/rear-backups/Linux/client.example.com/client.example.com-backup.tar.gz|client.example.com|2016-08-29 16:07:28
umount: /tmp/rear.SEVLuCsI5908TRZ/outputfs: not mounted
2016-08-29 16:07:28 Including backup/NETFS/default/15_save_copy_of_prefix_dir.sh
2016-08-29 16:07:28 Including backup/NETFS/default/20_make_prefix_dir.sh
mkdir: created directory `/tmp/rear.SEVLuCsI5908TRZ/outputfs/client.example.com'
2016-08-29 16:07:28 Including backup/NETFS/default/25_create_lock.sh
I even modified the backup/NETFS/default/10_mount_NETFS_path.sh
script, for debug purposes, to spit out the contents of all the MOUNTCMD
and UMOUNTCMD variables, and none of the UMOUNT variables contain
anything, so I don't know where its picking this botched mount command
up from.
2016-08-30 15:04:51 Including backup/NETFS/default/10_mount_NETFS_path.sh
mkdir: created directory `/tmp/rear.9ttQqoQPAKL9Dq2/outputfs'
BACKUP_MOUNTCMD =
BACKUP_UMOUNTCMD = custom_umount
NETFS_MOUNTCMD =
NETFS_UMOUNTCMD =
OUTPUT_MOUNTCMD =
OUTPUT_UMOUNTCMD =
2016-08-30 15:04:51 Mounting with 'custom_umount /tmp/rear.9ttQqoQPAKL9Dq2/outputfs'
...
umount: /tmp/rear.9ttQqoQPAKL9Dq2/outputfs: not mounted
It doesn't seem to matter if I set BACKUP_UMOUNTCMD or NETFS_UMOUNTCMD to my custom umount function, same result either way.
One other thought, we store both the ISO and BACKUP in the same spot on our NFS server, but during the course of the rear mkbackup, the NFS share is mounted and un-mounted 3 times, I believe. Is there any way to let it stay mounted the whole time so only 1 mount and 1 un-mount is required? The first 2 mounts and un-mounts work just fine, its only that 3rd / last mount, during the backup phase, that fails.
A better option (for us) would be a variable we could set that would run
at the very end, perhaps during
backup/NETFS/default/98_umount_NETFS_dir.sh script, to avoid having to
hijack the umount command in the way that we do. Does something like
that already exist? I didn't turn up anything obvious in my search.
Thank you in advance for any help you can provide.
gdha commented at 2016-09-15 17:31:¶
@gigawatts try running with the debug option turned on to see where it goes wrong. To be honest I never used this option myself (I believe @schlomo wrote this piece)
ajitmote commented at 2016-10-21 07:13:¶
I debug this issue further and below are my observations.
// From backup/NETFS/default/10_mount_NETFS_path.sh
`
mkdir -p $v "$BUILD_DIR/outputfs" >&2
StopIfError "Could not mkdir '$BUILD_DIR/outputfs'"
AddExitTask "rmdir $v $BUILD_DIR/outputfs >&2"
if [[ "$BACKUP_MOUNTCMD" ]] ; then
BACKUP_URL="var://BACKUP_MOUNTCMD"
fi
mount_url $BACKUP_URL $BUILD_DIR/outputfs $BACKUP_OPTIONS
<<<<<< this is where actual failure is triggered
`
So it means $BACKUP_URL is not correct.
I added debug statement here and got below :
2016-09-14 04:37:53 amote var://BACKUP_UMOUNTCMD.. /tmp/rear.iAtsBx6RCk9ODs2/outputfs.. rw,noatime ......
It means $BACKUP_URL is set to var://BACKUP_UMOUNTCMD which is causing this issue.
Checking further, I see that BACKUP_URL is set in below two places
which is relevant to our analysis :
[root@rhel7 rear]# grep -rn BACKUP_URL * | grep var backup/NETFS/default/10_mount_NETFS_path.sh:8: BACKUP_URL="var://BACKUP_MOUNTCMD" backup/NETFS/default/98_umount_NETFS_dir.sh:4: BACKUP_URL="var://BACKUP_UMOUNTCMD" prep/NETFS/default/05_check_NETFS_requirements.sh:17: ### set other variables from BACKUP_URL
I ran "rear -v mkbackup" command and observed logs from /var/log/rear/rear-rhel7.log file. Along with this, I added debug statement in other places as well.
When mkbackup command is run, it go through all below stages :
2016-09-14 05:16:31 Running 'init' stage 2016-09-14 05:16:31 Running mkbackup workflow 2016-09-14 05:16:31 Running 'prep' stage 2016-09-14 05:16:31 Running DHCP client found, enabling USE_DHCLIENT 2016-09-14 05:16:31 Running 'layout/save' stage 2016-09-14 05:16:33 Running 'rescue' stage 2016-09-14 05:16:34 Running 'build' stage 2016-09-14 05:16:42 Running 'pack' stage 2016-09-14 05:16:59 Running 'output' stage 2016-09-14 05:17:05 Running 'backup' stage 2016-09-14 05:17:05 Running exit tasks.
During 'prep' stage, I see that 98_umount_NETFS_dir.sh is sourced and that's where BACKUP_URL gets set.
// from 98_umount_NETFS_dir.sh , as we have set BACKUP_UMOUNTCMD in site.conf file, if is true and then set BACKUP_URL
if [[ "$BACKUP_UMOUNTCMD" ]] ; then BACKUP_URL="var://BACKUP_UMOUNTCMD" fi
2016-09-14 05:16:31 Running 'prep' stage
`
2016-09-14 05:16:31 amote Including
prep/NETFS/default/06_mount_NETFS_path.sh
mkdir: created directory '/tmp/rear.ydwE7wMMk8Y5fme/outputfs'
2016-09-14 05:16:31 amote nfs://192.168.122.1/home/amote/nfs..
/tmp/rear.ydwE7wMMk8Y5fme/outputfs.. rw,noatime ......
2016-09-14 05:16:31 Mounting with 'mount -v -t nfs -o rw,noatime
192.168.122.1:/home/amote/nfs /tmp/rear.ydwE7wMMk8Y5fme/outputfs'
mount.nfs: timeout set for Wed Sep 14 05:18:31 2016
mount.nfs: trying text-based options
'vers=4,addr=192.168.122.1,clientaddr=192.168.122.137'
2016-09-14 05:16:31
amoteSource.../usr/share/rear/prep/NETFS/default/98_umount_NETFS_dir.sh...
<<<<<<<<
2016-09-14 05:16:31 amote Including
prep/NETFS/default/98_umount_NETFS_dir.sh
2016-09-14 05:16:31 Unmounting with 'my_umount
/tmp/rear.ydwE7wMMk8Y5fme/outputfs'
rmdir: removing directory, '/tmp/rear.ydwE7wMMk8Y5fme/outputfs'
2016-09-14 05:16:31 Finished running 'prep' stage in 0 seconds
`
2016-09-14 05:16:31 Running 'layout/save' stage
55767 extents written (108 MB) 2016-09-14 04:37:51 Wrote ISO image: /var/lib/rear/output/rhel7-09142016-rescue.iso (128M) ... 2016-09-14 04:37:51 Copying files '/var/lib/rear/output/rhel7-09142016-rescue.iso' to nfs location '/var/lib/rear/output/rhel7-09142016-rescue.iso' -> '/tmp/rear.iAtsBx6RCk9ODs2/outputfs/rhel7/rhel7-09142016-rescue.iso' '/usr/share/rear/conf/templates/RESULT_usage_ISO.txt' -> '/tmp/rear.iAtsBx6RCk9ODs2/outputfs/rhel7/README' 2016-09-14 04:37:53 Saved /var/log/rear/rear-rhel7.log as rear.log ... removed '/tmp/rear.iAtsBx6RCk9ODs2/outputfs/rhel7/.lockfile' 2016-09-14 04:37:53 amoteSource.../usr/share/rear/output/default/98_umount_output_dir.sh... 2016-09-14 04:37:53 amote Including output/default/98_umount_output_dir.sh 2016-09-14 04:37:53 Unmounting '/tmp/rear.iAtsBx6RCk9ODs2/outputfs' /tmp/rear.iAtsBx6RCk9ODs2/outputfs: nfs4 mount point detected /tmp/rear.iAtsBx6RCk9ODs2/outputfs: umounted rmdir: removing directory, '/tmp/rear.iAtsBx6RCk9ODs2/outputfs' 2016-09-14 04:37:53 Finished running 'output' stage in 3 seconds
Now during last 'backup' stage, 10_mount_NETFS_path.sh script access
BACKUP_URL variable which is already set in 'prep' stage to
"var://BACKUP_UMOUNTCMD". Because of this, its getting failed here.
2016-09-14 04:37:53 Running 'backup' stage 2016-09-14 04:37:53 amote SourceStage... default/00_valid_backup_methods.sh.../usr/share/rear...backup 2016-09-14 04:37:53 amoteSource.../usr/share/rear/backup/default/00_valid_backup_methods.sh... 2016-09-14 04:37:53 amote Including backup/default/00_valid_backup_methods.sh 2016-09-14 04:37:53 amoteSource.../usr/share/rear/backup/NETFS/default/10_mount_NETFS_path.sh... 2016-09-14 04:37:53 amote Including backup/NETFS/default/10_mount_NETFS_path.sh mkdir: created directory '/tmp/rear.iAtsBx6RCk9ODs2/outputfs' 2016-09-14 04:37:53 amote var://BACKUP_UMOUNTCMD.. /tmp/rear.iAtsBx6RCk9ODs2/outputfs.. rw,noatime ...... 2016-09-14 04:37:53 amote1 var..my_umount /tmp/rear.iAtsBx6RCk9ODs2/outputfs.. 2016-09-14 04:37:53 Mounting with 'my_umount /tmp/rear.iAtsBx6RCk9ODs2/outputfs' TRYING TO RUN UN-MOUNT COMMAND umount: /tmp/rear.iAtsBx6RCk9ODs2/outputfs: not mounted 2016-09-14 04:37:53 ERROR: Mount command 'my_umount /tmp/rear.iAtsBx6RCk9ODs2/outputfs' failed.
=== Stack trace ===
Trace 0: /usr/sbin/rear:251 main Trace 1: /usr/share/rear/lib/mkbackup-workflow.sh:26 WORKFLOW_mkbackup Trace 2: /usr/share/rear/lib/framework-functions.sh:73 SourceStage Trace 3: /usr/share/rear/lib/framework-functions.sh:32 Source Trace 4: /usr/share/rear/backup/NETFS/default/10_mount_NETFS_path.sh:11 source Trace 5: /usr/share/rear/lib/global-functions.sh:155 mount_url Trace 6: /usr/share/rear/lib/_input-output-functions.sh:132 StopIfError Message: Mount command 'my_umount /tmp/rear.iAtsBx6RCk9ODs2/outputfs' failed.
To workaround this issue, I defined NETFS_MOUNTCMD in site.conf file
and "rear -v mkbackup" is properly executed without any failure.
$ cat /etc/rear/site.conf NETFS_URL=nfs://192.168.122.1/home/amote/nfs BACKUP_PROG_ARCHIVE=hostname
-s-backup OUTPUT=ISO ISO_PREFIX="hostname -s-date
+%m%d%Y-rescue" BACKUP=NETFS NETFS_SKIP_WARNING=1 NETFS_PREFIX="$(uname -n)" OUTPUT_PREFIX="$(uname -n)" NETFS_MOUNTCMD="mount -v -t nfs -o rw,noatime 192.168.122.1:/home/amote/nfs" NETFS_UMOUNTCMD="my_umount" my_umount() { echo "TRYING TO RUN UN-MOUNT COMMAND"; umount $1 } ...
gdha commented at 2016-12-31 09:42:¶
@gigawatts @ajitmote In the latest development release of ReaR we have
modified the exit messages so that the NETFS_UMOUNTCMD may be
unnecessary. Just give it a try.
If it was just for me I rather see the NETFS_UMOUNTCMD and
NETFS_MOUNTCMD go away. It should be enough to use BACKUP_URL
@jsmeix @schlomo Any comments or thoughts?
jsmeix commented at 2017-01-02 09:56:¶
The initial comment
https://github.com/rear/rear/issues/991#issue-174101377
reads
We define a custom NETFS_UMOUNTCMD variable in our site.conf, with a function that basically spits out a backup complete message, along with a timestamp and where the backup was stored ... A better option (for us) would be a variable we could set that would run at the very end ... to avoid having to hijack the umount command in the way that we do. Does something like that already exist?
I think one cannot expect that misuse of NETFS_UMOUNTCMD
has no unexpected side effects.
NETFS_UMOUNTCMD is deprectated.
nowadays it is BACKUP_UMOUNTCMD,
see default.conf.
BACKUP_UMOUNTCMD is meant to be used together
with a matching BACKUP_MOUNTCMD.
To run arbitrary commands at the very end use
POST_BACKUP_SCRIPT
To spit out a backup complete message see
https://github.com/rear/rear/pull/1022#issuecomment-255323450
I think this issue is sufficiently answered via
https://github.com/rear/rear/pull/1022
so that I close it.
[Export of Github issue for rear/rear.]