#146 Issue closed: Default NFS option (root_squash) causes problems, we should detect/report¶
Labels: support / question
mcamero opened issue at 2012-08-17 18:12:¶
Following Error while trying to create build dir
ERROR: Could not mkdir '/tmp/rear.fQgYU7zNLwwKWYJ/outputfs/rear'
Following is output of rear log
2012-08-17 14:11:44 Relax and Recover 1.13.0 / $Date$
2012-08-17 14:11:44 Command line options: /usr/sbin/rear -v mkbackuponly
2012-08-17 14:11:44 Including conf/Linux-i386.conf
2012-08-17 14:11:44 Including conf/GNU/Linux.conf
2012-08-17 14:11:44 Including conf/SUSE_LINUX.conf
2012-08-17 14:11:44 Including /etc/rear/local.conf
2012-08-17 14:11:44 Using build area '/tmp/rear.JmxRUMCdqcvOtqA'
mkdir: created directory `/tmp/rear.JmxRUMCdqcvOtqA/rootfs'
mkdir: created directory `/tmp/rear.JmxRUMCdqcvOtqA/tmp'
2012-08-17 14:11:44 Running mkbackuponly workflow
2012-08-17 14:11:44 Running 'prep' stage
2012-08-17 14:11:44 Including prep/default/00_remove_workflow_conf.sh
mkdir: created directory `/tmp/rear.JmxRUMCdqcvOtqA/rootfs/etc'
mkdir: created directory `/tmp/rear.JmxRUMCdqcvOtqA/rootfs/etc/rear'
2012-08-17 14:11:44 Including prep/default/02_translate_url.sh
2012-08-17 14:11:44 Including prep/NETFS/default/05_check_NETFS_requirements.sh
2012-08-17 14:11:44 Skipping ping test
2012-08-17 14:11:44 Including prep/NETFS/default/15_save_rsync_version.sh
2012-08-17 14:11:44 Including prep/GNU/Linux/20_include_agetty.sh
2012-08-17 14:11:44 Including prep/NETFS/GNU/Linux/20_selinux_in_use.sh
2012-08-17 14:11:44 Including prep/GNU/Linux/21_include_dhclient.sh
2012-08-17 14:11:44 Including prep/GNU/Linux/22_include_lvm_tools.sh
2012-08-17 14:11:44 Device mapper found enabled. Including LVM tools.
2012-08-17 14:11:44 Including prep/GNU/Linux/23_include_md_tools.sh
2012-08-17 14:11:44 Including prep/GNU/Linux/28_include_systemd.sh
2012-08-17 14:11:44 Including prep/GNU/Linux/28_include_vmware_tools.sh
2012-08-17 14:11:44 Including '/usr/lib*/vmware-tools'
2012-08-17 14:11:44 Including prep/GNU/Linux/29_include_drbd.sh
2012-08-17 14:11:44 Including prep/ISO/default/30_check_iso_dir.sh
2012-08-17 14:11:44 Including prep/GNU/Linux/30_include_grub_tools.sh
2012-08-17 14:11:44 Including prep/ISO/default/32_check_cdrom_size.sh
2012-08-17 14:11:44 ISO Directory '/var/tmp' [/dev/mapper/system-root] has 4933 MB free space
2012-08-17 14:11:44 Including prep/ISO/GNU/Linux/32_verify_mkisofs.sh
2012-08-17 14:11:44 Using '/usr/bin/mkisofs' to create ISO images
2012-08-17 14:11:44 Including prep/ISO/Linux-i386/33_find_isolinux.sh
2012-08-17 14:11:44 Including prep/NETFS/default/40_automatic_exclude_recreate.sh
2012-08-17 14:11:44 Finished running 'prep' stage in 0 seconds
2012-08-17 14:11:44 Running 'layout/save' stage
2012-08-17 14:11:44 Including layout/save/GNU/Linux/10_create_layout_file.sh
2012-08-17 14:11:44 Creating disk layout
2012-08-17 14:11:44 Preparing layout directory.
2012-08-17 14:11:44 Including layout/save/GNU/Linux/20_partition_layout.sh
2012-08-17 14:11:44 Saving disk partitions.
2012-08-17 14:11:44 Including layout/save/GNU/Linux/21_raid_layout.sh
2012-08-17 14:11:44 Including layout/save/GNU/Linux/22_lvm_layout.sh
2012-08-17 14:11:44 Saving LVM layout.
2012-08-17 14:11:44 Including layout/save/GNU/Linux/23_filesystem_layout.sh
2012-08-17 14:11:44 Saving Filesystem layout.
2012-08-17 14:11:45 Including layout/save/GNU/Linux/24_swaps_layout.sh
2012-08-17 14:11:45 Saving Swap information.
2012-08-17 14:11:45 Including layout/save/GNU/Linux/25_drbd_layout.sh
2012-08-17 14:11:45 Including layout/save/GNU/Linux/26_crypt_layout.sh
2012-08-17 14:11:45 Saving Encrypted volumes.
2012-08-17 14:11:45 Including layout/save/GNU/Linux/27_hpraid_layout.sh
2012-08-17 14:11:45 Including layout/save/GNU/Linux/28_multipath_layout.sh
2012-08-17 14:11:45 Including layout/save/default/30_list_dependencies.sh
2012-08-17 14:11:45 Including layout/save/default/31_include_exclude.sh
2012-08-17 14:11:45 Including layout/save/default/32_autoexclude.sh
2012-08-17 14:11:45 Including layout/save/default/33_remove_exclusions.sh
2012-08-17 14:11:45 Including layout/save/default/34_generate_mountpoint_device.sh
2012-08-17 14:11:45 Including layout/save/GNU/Linux/35_copy_drbdtab.sh
2012-08-17 14:11:45 Including layout/save/GNU/Linux/50_extract_vgcfg.sh
2012-08-17 14:11:45 Including layout/save/GNU/Linux/51_current_disk_usage.sh
2012-08-17 14:11:45 Including layout/save/SUSE_LINUX/60_create_mkbootloader.sh
2012-08-17 14:11:45 Using boot loader GRUB
2012-08-17 14:11:45 Including layout/save/default/60_snapshot_files.sh
2012-08-17 14:11:45 Finished running 'layout/save' stage in 1 seconds
2012-08-17 14:11:45 Running 'backup' stage
2012-08-17 14:11:45 Including backup/NETFS/default/10_mount_NETFS_path.sh
mkdir: created directory `/tmp/rear.JmxRUMCdqcvOtqA/outputfs'
2012-08-17 14:11:45 Mounting with 'mount -v -t nfs -o rw,noatime acpic-s711.rcmp-grc.gc.ca:/global/data/images/ReaR/ /tmp/rear.JmxRUMCdqcvOtqA/outputfs'
mount.nfs: trying 192.168.35.9 prog 100003 vers 3 prot TCP port 2049
mount.nfs: trying 192.168.35.9 prog 100005 vers 3 prot UDP port 950
mount.nfs: timeout set for Fri Aug 17 14:13:45 2012
mount.nfs: trying text-based options 'addr=192.168.35.9'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: prog 100005, trying vers=3, prot=17
acpic-s711.rcmp-grc.gc.ca:/global/data/images/ReaR/ on /tmp/rear.JmxRUMCdqcvOtqA/outputfs type nfs (rw,noatime)
2012-08-17 14:11:45 Including backup/NETFS/default/15_save_copy_of_prefix_dir.sh
2012-08-17 14:11:45 Including backup/NETFS/default/20_make_prefix_dir.sh
mkdir: cannot create directory `/tmp/rear.JmxRUMCdqcvOtqA/outputfs/rear': Permission denied
Trace: 122 StopIfError /usr/share/rear/lib/_input-output-functions.sh
Trace: 14 source /usr/share/rear/backup/NETFS/default/20_make_prefix_dir.sh
Trace: 40 Source /usr/share/rear/lib/framework-functions.sh
Trace: 79 SourceStage /usr/share/rear/lib/framework-functions.sh
Trace: 19 WORKFLOW_mkbackuponly /usr/share/rear/lib/mkbackuponly-workflow.sh
Trace: 238 main /usr/sbin/rear
2012-08-17 14:11:45 ERROR: Could not mkdir '/tmp/rear.JmxRUMCdqcvOtqA/outputfs/rear'
2012-08-17 14:11:45 Running exit tasks.
Legacy NFS mount point detected
umount.nfs: trying 192.168.35.9 prog 100005 vers 3 prot UDP port 950
umount.nfs: prog 100005, trying vers=3, prot=17
acpic-s711.rcmp-grc.gc.ca:/global/data/images/ReaR/ umounted
rmdir: removing directory, `/tmp/rear.JmxRUMCdqcvOtqA/outputfs'
2012-08-17 14:11:45 Finished in 1 seconds
2012-08-17 14:11:45 Removing build area /tmp/rear.JmxRUMCdqcvOtqA
rmdir: removing directory, `/tmp/rear.JmxRUMCdqcvOtqA'
2012-08-17 14:11:45 End of program reached
dagwieers commented at 2012-08-20 13:11:¶
Can this error be caused by NFS mount permissions ?
mkdir: cannot create directory `/tmp/rear.JmxRUMCdqcvOtqA/outputfs/rear': Permission denied
If you are root and root has no permissions it reminds me of root_squash problems.
mcamero commented at 2012-08-20 13:35:¶
Root cannot create a subdir prior to create parent dir. See below:
mkdir errors on creating directory with -p option.
With -p option works.
when running rear mkbackuponly it 'StopifError' on the attempt to create
whole directory.
rear:/usr/share/rear/prep/NETFS/default # mkdir /tmp/rear.JmxRUMCdqcvOtqA/outputfs/rear
mkdir: cannot create directory `/tmp/rear.JmxRUMCdqcvOtqA/outputfs/rear': No such file or directory
rear:/usr/share/rear/prep/NETFS/default # mkdir -p /tmp/rear.JmxRUMCdqcvOtqA/outputfs/rear
mcamero commented at 2012-08-20 13:36:¶
Sorry..Was not intention to close this issue. Please see comment above
dagwieers commented at 2012-08-20 13:47:¶
The parent dir is your mountpoint so it should exist (since it didn't bail out when mounting, see below).
2012-08-17 14:11:45 Mounting with 'mount -v -t nfs -o rw,noatime acpic-s711.rcmp-grc.gc.ca:/global/data/images/ReaR/ /tmp/rear.JmxRUMCdqcvOtqA/outputfs'
mount.nfs: trying 192.168.35.9 prog 100003 vers 3 prot TCP port 2049
mount.nfs: trying 192.168.35.9 prog 100005 vers 3 prot UDP port 950
mount.nfs: timeout set for Fri Aug 17 14:13:45 2012
mount.nfs: trying text-based options 'addr=192.168.35.9'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: prog 100005, trying vers=3, prot=17
acpic-s711.rcmp-grc.gc.ca:/global/data/images/ReaR/ on /tmp/rear.JmxRUMCdqcvOtqA/outputfs type nfs (rw,noatime)
The problem is a permission denied creating a subdirectory inside the NFS mount, which I think is because of permissions issues. The reason you can create the directory afterwards is because at that point it had been unmounted (and removed). Please try mounting the NFS mount yourself, and creating a directory inside the NFS mount to verify that this is indeed the problem.
mcamero commented at 2012-08-20 16:47:¶
From the client side the script 'rear mkbackuponly' is trying to create /tmp/rear.JmxRUMCdqcvOtqA/outputfs in order to use it as the mount point to the indicated nfs server. Below is the permissions of /tmp.
When I manually create the nfs mount:
acpic-s711.rcmp-grc.gc.ca:/global/data/images/ReaR 103212320 91064640 7953376 92% /tmp/rear.JmxRUMCdqcvOtqA/outputfs
And then try to create the 'rear' subdir under it:
rear:/tmp # mkdir rear /tmp/rear.JmxRUMCdqcvOtqA/outputfs
mkdir: cannot create directory `/tmp/rear.JmxRUMCdqcvOtqA/outputfs': File exists
rear:/tmp # mkdir -p rear /tmp/rear.JmxRUMCdqcvOtqA/outputfs
Only works with -p option
drwxrwxrwt 42 root root 4096 Aug 20 12:38 tmp
Thanks for help, appreciated.
mcamero commented at 2012-08-20 16:53:¶
drwxr-xr-x 2 root root 4096 Aug 15 13:29 outputfs
mkdir: created directory `/tmp/rear.LYNrvU9i66M123V/outputfs'
2012-08-20 12:51:58 Mounting with 'mount -v -t nfs -o rw,noatime acpic-s711.rcmp-grc.gc.ca:/global/data/images/ReaR/ /tmp/rear.LYNrvU9i66M123V/outputfs'
mount.nfs: trying 192.168.35.9 prog 100003 vers 3 prot TCP port 2049
mount.nfs: trying 192.168.35.9 prog 100005 vers 3 prot UDP port 950
mount.nfs: timeout set for Mon Aug 20 12:53:58 2012
mount.nfs: trying text-based options 'addr=192.168.35.9'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: prog 100005, trying vers=3, prot=17
acpic-s711.rcmp-grc.gc.ca:/global/data/images/ReaR/ on /tmp/rear.LYNrvU9i66M123V/outputfs type nfs (rw,noatime)
2012-08-20 12:51:58 Including backup/NETFS/default/15_save_copy_of_prefix_dir.sh
2012-08-20 12:51:58 Including backup/NETFS/default/20_make_prefix_dir.sh
mkdir: cannot create directory `/tmp/rear.LYNrvU9i66M123V/outputfs/rear': Permission denied
Trace: 122 StopIfError /usr/share/rear/lib/_input-output-functions.sh
mcamero commented at 2012-08-20 17:22:¶
Would a strace help?
Strace grep error only
read(255, "exec 2>\"$LOGFILE\" || echo \"ERROR"..., 6828) = 2681
read(255, "StopIfError \"Could not create bu"..., 6828) = 634
read(255, "StopIfError \"Could not create $R"..., 6828) = 410
read(255, "StopIfError \"Could not create $T"..., 6828) = 341
write(1, "2012-08-20 13:14:05 ERROR: Could"..., 85) = 85
write(1, "ERROR: Could not mkdir '/tmp/rea"..., 65ERROR: Could not mkdir '/tmp/rear.yBf6xK3q7KcibqQ/outputfs/rear'
write(1, "Aborting due to an error, check "..., 63Aborting due to an error, check /tmp/rear-rear.log for details
dagwieers commented at 2012-08-21 12:55:¶
Only works with -p option
The command mkdir -p doesn't do anything if the directory already
exists. Just do the test:
[dag@moria unoconv]$ mkdir /usr/bin
mkdir: cannot create directory `/usr/bin': File exists
[dag@moria unoconv]$ mkdir -p /usr/bin
[dag@moria unoconv]$
Also remark the error File exists which is not as the original report Permission denied.
So please try the same by doing
mkdir /tmp/rear.JmxRUMCdqcvOtqA/outputfs/rear and
mkdir -p /tmp/rear.JmxRUMCdqcvOtqA/outputfs/rear instead (mind the
appended /rear).
mcamero commented at 2012-08-21 13:16:¶
Fixed. Thanks.
Mark Cameron
613 868 2773
dagwieers commented at 2012-08-21 21:56:¶
@mcamero What exactly is fixed ? Can you provide us with an update to this ticket ?
mcamero commented at 2012-08-21 23:58:¶
Sorry. I was being a moron. Permissions on the directory of the Nfs server. Thanks for your assistance.
Mark Cameron
613 868 2773
jetwes commented at 2012-08-22 07:30:¶
I have the same error.
sudo rear mkbackup -v
Relax-and-Recover 1.13.0-git201208040834 / 2012-08-04
Using log file: /var/log/rear/rear-sm-star1.log
Creating disk layout
Creating root filesystem layout
TIP: To login as root via ssh you need to set up /root/.ssh/authorized_keys
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Creating initramfs
ERROR: Could not mkdir '/tmp/rear.wIK3JYJPpgs2r8j/outputfs/sm-star1'
Aborting due to an error, check /var/log/rear/rear-sm-star1.log for details*
Logfile:
mkdir: created directory `/tmp/rear.wIK3JYJPpgs2r8j/tmp/boot'
2012-08-22 09:24:17 Including output/NETFS/default/20_make_prefix_dir.sh
mkdir: cannot create directory `/tmp/rear.wIK3JYJPpgs2r8j/outputfs/sm-star1': Permission denied
Trace: 122 StopIfError /usr/share/rear/lib/_input-output-functions.sh
Trace: 14 source /usr/share/rear/output/NETFS/default/20_make_prefix_dir.sh
Trace: 40 Source /usr/share/rear/lib/framework-functions.sh
Trace: 79 SourceStage /usr/share/rear/lib/framework-functions.sh
Trace: 24 WORKFLOW_mkbackup /usr/share/rear/lib/mkbackup-workflow.sh
Trace: 245 main /usr/sbin/rear
2012-08-22 09:24:17 ERROR: Could not mkdir '/tmp/rear.wIK3JYJPpgs2r8j/outputfs/sm-star1'
2012-08-22 09:24:17 Running exit tasks.
NFSv4 mount point detected
sm-star1.soest-hosting.de:/var/backup umounted
rmdir: removing directory, `/tmp/rear.wIK3JYJPpgs2r8j/outputfs'
2012-08-22 09:24:17 Finished in 16 seconds
2012-08-22 09:24:17 Removing build area /tmp/rear.wIK3JYJPpgs2r8j
rmdir: removing directory, `/tmp/rear.wIK3JYJPpgs2r8j'
2012-08-22 09:24:17 End of program reached*
config:
OUTPUT=ISO
BACKUP=NETFS
NETFS_URL="nfs://sm-star1.soest-hosting.de/var/backup"*
my NFS ist working:
sudo showmount -e sm-star1.soest-hosting.de
Export list for sm-star1.soest-hosting.de:
/var/backup sm-star1.soest-hosting.de* and has write permissions
dagwieers commented at 2012-08-22 12:28:¶
@mcamero Thanks for the update. Having a resolution is important for users and future troubleshooting.
@jetwes Can you verify this is not caused by the same problem reported by @mcamero ? If your NFS share is exported as root_squash, the root user on the NFS client will not have the proper rights on the mounted NFS share (as root). All access will be reduced to anonymous access (nobody) on the NFS server (even when, or rather, specifically because you are root). You can test this by mounting the NFS share and then trying to create a directory and/or file inside the mounted share.
dagwieers commented at 2012-08-22 12:30:¶
BTW We may have to make this more obvious from the documentation and/or modify the error message specifically when NFS is used and a Permission denied error is returned. Before closing this ticket I will create a feature request for this. I first like to see if this is the same cause for @jetwes !
mcamero commented at 2012-08-22 19:29:¶
What are the permissions of nfs share ?
Mark Cameron
613 868 2773
mcamero commented at 2012-08-22 19:32:¶
That sounds like a good plan. Enabling all the permissions on the subdir rear of the nfs share eg /global/data/rear, was what fixed it for me.
Mark Cameron
613 868 2773
dagwieers commented at 2012-08-28 21:53:¶
@jetwes Do you still have this issue ?
mcamero commented at 2012-08-28 23:22:¶
No. It has been resolved.
Thanks
Mark Cameron
613 868 2773
jetwes commented at 2012-08-29 06:58:¶
Sorry for the late answer. I didn't try it again, because it reinstalled the system and didn't had the time to check it again. I will get to this as soon as i tried again.
Thx
Jens
cal-s commented at 2012-09-05 16:28:¶
Yes, this happens when the remote filesystem is exported with "root_squash", so root on the local system is unable to create dirs below the /tmp/rear.[blah]/outputfs mountpoint. The workarouind is to create a world-writeable subdir manually and register its location in local.conf. Alternately, create a "no_root_squash" filesystem/export for ReaR's use. At the risk of overcomplicating matters, perhaps ReaR could offer to spawn a shell so that the user can ssh to the remote and create the directory, then retry the mount/mkdir without being bounced out of ReaR?
schlomo commented at 2012-09-05 19:46:¶
Just wanted to note that ReaR is intended to run unattended, e.g. from
a
cron job. So spawning shell is for most use cases not a good idea.
Usually people set up ReaR to work properly and let it run by itself
after
the initial setup.
cal-s commented at 2012-09-05 20:05:¶
So i hear. However, root_squash'd nfs exports and lack of distincitve (ie: date-appaended) save targets/dirs do make unattended operation kind of unwieldy. I'd imagine that many people pull mkbackups selectively by hand as systems undergo known changes - i know i do.
regards,
- cal
dagwieers commented at 2012-09-05 21:54:¶
@cal-s Even if one does occasional backups using Relax-and-Recover, setting up NFS correctly in advance is not an unreasonable expectation. We should report to the user that root_squash is the cause (and add it as a tip in the documentation), but that's as far as our responsibility goes. If permissions are the problem a different error could indicate this, but remember selinux could also be causing us problems... To me this is a support issue, rather than a technical issue to solve and so we should make it easier for users to understand the problem. That's why I do not consider it a bug.
TODO: Escalate better to the user
cal-s commented at 2012-09-06 08:07:¶
Fully agree. Thank you!
- cal
mcamero commented at 2012-09-06 22:24:¶
Definitely not a bug.
Mark Cameron
613 868 2773
[Export of Github issue for rear/rear.]