#1875 Issue closed: mkdir: cannot create directory '/tmp/rear.aOAII02mqnI8cRX/outputfs/TestUbuntu': Permission denied¶
Labels: support / question, fixed / solved / done
langerkunal opened issue at 2018-07-19 07:31:¶
Relax-and-Recover (ReaR) Issue Template¶
Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):
-
ReaR version ("/usr/sbin/rear -V"):
usr/sbin/rear -V
Relax-and-Recover 2.4 / Git -
OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"):
Ubuntu 16.04 -
ReaR configuration files ("cat /etc/rear/site.conf" or "cat /etc/rear/local.conf"):
OUTPUT=ISO BACKUP=NETFS BACKUP_URL=nfs://10.23.172.99/nimfs/Linux_OS_Backups/
-
Real hardware (PC or PowerNV BareMetal or ARM) and/or virtual machine (KVM guest or PoverVM LPAR):
IBM POWER8 LPAR using PowerVM -
System architecture (x86 compatible or POWER PPC64/PPC64LE or what excat ARM device):
POWER pp64le -
Are you using BIOS or UEFI or another way to boot (Open Firmware or Petitboot)?
UEFI -
Brief description of the issue:
When I am trying to runusr/sbin/rear -d -D mkbackupas a root user, I am getting permission denied error. The same was observed in SLES 12 SP3 on IBM POWER8 as well, see
https://github.com/rear/rear/issues/1858#issuecomment-404490224
Its the same release downloaded from github rear version 2.4
++ mkdir -p -v -m0750 /tmp/rear.aOAII02mqnI8cRX/outputfs/TestUbuntu mkdir: cannot create directory '/tmp/rear.aOAII02mqnI8cRX/outputfs/TestUbuntu': Permission denied ++ StopIfError 'Could not mkdir '\''/tmp/rear.aOAII02mqnI8cRX/outputfs/TestUbuntu'\''' ++ (( 1 != 0 )) ++ Error 'Could not mkdir '\''/tmp/rear.aOAII02mqnI8cRX/outputfs/TestUbuntu'\''' ++ LogPrintError 'ERROR: Could not mkdir '\''/tmp/rear.aOAII02mqnI8cRX/outputfs/TestUbuntu'\''' ++ Log 'ERROR: Could not mkdir '\''/tmp/rear.aOAII02mqnI8cRX/outputfs/TestUbuntu'\''' +++ date '+%Y-%m-%d %H:%M:%S.%N ' ++ local 'timestamp=2018-07-19 12:53:11.263682133 ' ++ test 1 -gt 0 ++ echo '2018-07-19 12:53:11.263682133 ERROR: Could not mkdir '\''/tmp/rear.aOAII02mqnI8cRX/outputfs/TestUbuntu'\''' 2018-07-19 12:53:11.263682133 ERROR: Could not mkdir '/tmp/rear.aOAII02mqnI8cRX/outputfs/TestUbuntu' ++ PrintError 'ERROR: Could not mkdir '\''/tmp/rear.aOAII02mqnI8cRX/outputfs/TestUbuntu'\''' ++ echo -e 'ERROR: Could not mkdir '\''/tmp/rear.aOAII02mqnI8cRX/outputfs/TestUbuntu'\''' ++ LogToSyslog 'ERROR: Could not mkdir '\''/tmp/rear.aOAII02mqnI8cRX/outputfs/TestUbuntu'\''' ++ logger -t rear -i 'ERROR: Could not mkdir '\''/tmp/rear.aOAII02mqnI8cRX/outputfs/TestUbuntu'\''' ++ has_binary caller
However I can create the same directory manually as root user:
mkdir -p /tmp/rear.aOAII02mqnI8cRX/outputfs/TestUbuntu ls -ltr /tmp/rear.aOAII02mqnI8cRX/ total 12 drwxr-xr-x 18 root root 4096 Jul 19 12:52 rootfs drwxr-xr-x 4 root root 4096 Jul 19 12:53 tmp drwxr-xr-x 3 root root 4096 Jul 19 12:59 outputfs
- Work-around, if any:
No workaround.
schabrolles commented at 2018-07-19 10:34:¶
@langerkunal, could you please tell me exactly how you install "rear".
I would like to test the same procedure as you on my system to try to
replicate your issue and understand what happens here (as I don't have
any problem on my side with either SLES12sp3 and Ubuntu.)
jsmeix commented at 2018-07-19 10:56:¶
FYI (regardless that it does not help):
Currently I am at my wit's end.
jsmeix commented at 2018-07-19 11:05:¶
@langerkunal
some blind shots in the dark:
What results on your system things like
# grep ':0:' /etc/passwd # type -a mkdir # ls -ld /tmp # getfacl /tmp # mount | grep tmp # ( set -x ; mkdir -p -v -m0750 /tmp/foo/bar/baz )
Could you provide us a complete debug log
when you run usr/sbin/rear -d -D mkbackup as a root?
gdha commented at 2018-07-19 11:57:¶
@langerkunal Did you check the umask already?
jsmeix commented at 2018-07-19 12:36:¶
@gdha
as far as I see ReaR sets umask 0077 in
output/default/010_set_umask.sh
before it calls mkdir ... in output/default/200_make_prefix_dir.sh
langerkunal commented at 2018-07-20 04:44:¶
@jsmeix
Please find below outputs of the specific commands you shared:
grep ':0:' /etc/passwd root:x:0:0:root:/root:/bin/bash
type -a mkdir mkdir is /bin/mkdir
ls -ld /tmp drwxrwxrwt 10 root root 4096 Jul 20 09:17 /tmp
getfacl /tmp getfacl: Removing leading '/' from absolute path names # file: tmp # owner: root # group: root # flags: --t user::rwx group::rwx other::rwx
mount | grep tmp udev on /dev type devtmpfs (rw,nosuid,relatime,size=2013056k,nr_inodes=31454,mode=755) tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=413632k,mode=755) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=413632k,mode=700)
( set -x ; mkdir -p -v -m0750 /tmp/foo/bar/baz ) + mkdir -p -v -m0750 /tmp/foo/bar/baz mkdir: created directory '/tmp/foo' mkdir: created directory '/tmp/foo/bar' mkdir: created directory '/tmp/foo/bar/baz'
@gdha the umask value as of now on my system is 0022:
root@TestUbuntu:~# umask 0022
Attaching the complete log file for reference.
Please note this time I ran rear -d -D mkbackup with sudo as shown
below:
sudo usr/sbin/rear -d -D mkbackup Relax-and-Recover 2.4 / 2018-06-21 Using log file: /home/rear/rear-2.4/var/log/rear/rear-TestUbuntu.log Using backup archive '/tmp/rear.uH4gaH9T7J3EZkZ/outputfs/TestUbuntu/backup.tar.gz' Creating disk layout Using guessed bootloader 'EFI' (found in first bytes on /dev/sda) Creating root filesystem layout Handling network interface 'ibmveth14' ibmveth14 is a physical device Handled network interface 'ibmveth14' Cannot include keyboard mappings (no keymaps default directory '') To log into the recovery system via ssh set up /root/.ssh/authorized_keys or specify SSH_ROOT_PASSWORD Copying logfile /home/rear/rear-2.4/var/log/rear/rear-TestUbuntu.log into initramfs as '/tmp/rear-TestUbuntu-partial-2018-07-20T10:06:56+05:30.log' Copying files and directories Copying binaries and libraries Copying kernel modules Omit copying files in /lib*/firmware/ (FIRMWARE_FILES='no') Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression Created initrd.cgz with gzip default compression (78093699 bytes) in 12 seconds ERROR: Could not mkdir '/tmp/rear.uH4gaH9T7J3EZkZ/outputfs/TestUbuntu' Aborting due to an error, check /home/rear/rear-2.4/var/log/rear/rear-TestUbuntu.log for details Exiting rear mkbackup (PID 20944) and its descendant processes Running exit tasks You should also rm -Rf /tmp/rear.uH4gaH9T7J3EZkZ Terminated
jsmeix commented at 2018-07-23 08:56:¶
@langerkunal
I think I have something:
In your rear-TestUbuntu.log there are many successful
mkdir -v -p /tmp/rear.XXX/... calls
which shows that root can create local sub-directories inside your /tmp/
directory.
But what fails is
mkdir -p -v -m0750 /tmp/rear.XXX/outputfs/TestUbuntu
and in contrast to the other successful mkdir -v -p /tmp/rear.XXX/...
calls
this one does not create a local sub-directory inside your /tmp/
directory
because at /tmp/rear.XXX/outputfs the remote NFS share is mounted
so that the /tmp/rear.XXX/outputfs/TestUbuntu sub-directory has to
be
created on the remote NFS server there in its exported directory and
with probability one (cf.
https://en.wikipedia.org/wiki/Almost_surely
)
that fails because you are not root on your remote NFS server,
more precisely on your remote NFS server you are not a user
with sufficient permissions to create that directory there.
Excerpts from your rear-TestUbuntu.log that show what I mean:
++ mkdir -p -v /tmp/rear.aOAII02mqnI8cRX/outputfs mkdir: created directory '/tmp/rear.aOAII02mqnI8cRX/outputfs' ... +++ mount -v -t nfs -o rw,noatime 10.23.172.99:/nimfs/Linux_OS_Backups/ /tmp/rear.aOAII02mqnI8cRX/outputfs mount.nfs: mount(2): No such file or directory mount.nfs: trying 10.23.172.99 prog 100003 vers 3 prot TCP port 2049 mount.nfs: trying 10.23.172.99 prog 100005 vers 3 prot UDP port 33010 mount.nfs: timeout set for Thu Jul 19 12:55:11 2018 mount.nfs: trying text-based options 'vers=4,addr=10.23.172.99,clientaddr=10.23.172.53' mount.nfs: trying text-based options 'addr=10.23.172.99' mount.nfs: prog 100003, trying vers=3, prot=6 mount.nfs: prog 100005, trying vers=3, prot=17 ... ++ mkdir -p -v -m0750 /tmp/rear.aOAII02mqnI8cRX/outputfs/TestUbuntu mkdir: cannot create directory '/tmp/rear.aOAII02mqnI8cRX/outputfs/TestUbuntu': Permission denied
The solution is to configure your NFS server appropriately
so that it allows to create that directory.
I.e. when you work as root on a NFS client machine
the NFS server must allow that user to create sub-directories
on the NFS server below its .../nimfs/Linux_OS_Backups/ directory.
Cf. the section
"First steps with Relax-and-Recover" in
https://en.opensuse.org/SDB:Disaster_Recovery
therein item (1) that has a few hints how one could set up the NFS
server.
Of course that section is not meant as explanatory description about NFS
server setup.
FYI:
For me things "just work" with that /etc/exports
on my NFS server (which is openSUSE Leap 42.3):
/nfs *(rw,root_squash,sync,no_subtree_check)
and with that I use in etc/rear/local.conf
BACKUP=NETFS BACKUP_OPTIONS="nfsvers=3,nolock" BACKUP_URL=nfs://NFS.server.IP.address/nfs
The exported '/nfs' directory on my NFS server has those permissions
# ls -ld /nfs drwxrwxrwx ... root root ... /nfs
so that with root_squash the user:group nobody:nogroup
can create/write files and sub-directories there and
access files in those sub-directories.
I reproduced it:
With default directory permissions on my NFS server
drwxr-xr-x ... root root ... /nfs
rear mkbackup fails in the same way for me.
In contrast with sufficiently permissive permissions (as above) it
works.
langerkunal commented at 2018-07-23 09:28:¶
After giving right permissions on NFS server I was able to successfully create the backup. Thanks for the help.
jsmeix commented at 2018-07-23 09:29:¶
@langerkunal
thanks for the prompt feedback.
I consider this issue sufficiently analyzed and solved.
gdha commented at 2018-07-23 09:41:¶
@jsmeix perhaps we could add some extra text when writing fails on the NFS server? Such as "make sure root has write permissions on NFS server (no_root_squash)"
jsmeix commented at 2018-07-23 09:57:¶
@gdha
I also had this idea but I think it gets a bit complicated to be
implemented
because at the place where mkdir /tmp/rear.XXX/outputfs/HOSTNAME
fails
it is not clear where that /tmp/rear.XXX/outputfs/HOSTNAME directory
actually is (i.e. if it is a local directory or on a remote machine and
in the
latter case whia what kind of remote connection - NFS is only one
case)
because mounting the remote NFS share at /tmp/rear.XXX/outputfs/
happens in output/default/100_mount_output_path.sh
via the mount_url function in lib/global-functions.sh
while 'mkdir /tmp/rear.XXX/outputfs/HOSTNAME`
happens in output/default/200_make_prefix_dir.sh
Perhaps the mount_url function is the right central place
where such tests could be generically added and where a failure of such
a test
could be even reported with a meaningful error message that depends on
the
url_scheme value so that for each different url_scheme an appropriate
test
plus message could be shown.
But it seems the url_scheme nfs is just the (*) default case in
mount_url
so that currently there won't be a NFS specific test plus message.
jsmeix commented at 2018-07-23 10:14:¶
@gdha
I think it gets overcomplicated to implement such tests in mount_url
because then the tests would need to depend on the workflow:
For mkrescue, mkbackup, and mkbackuponly write (and read?) permissions
are needed
while in contrast recover and restoreonly need (only?) read permissions.
schabrolles commented at 2018-07-23 10:41:¶
@jsmeix good catch !!!!!
Maybe a simple "touch file" test to validate it is writable could be
enough.
jsmeix commented at 2018-07-23 12:21:¶
@schabrolles
I think if we test we should better just try the real thing
and currently in output/default/200_make_prefix_dir.sh
it tries already the real thing via
mkdir -p $v -m0750 "${opath}" >&2
StopIfError "Could not mkdir '${opath}'"
but that does not make it obvious what the root cause is
neither for the user nor for us without analyzing the ReaR log.
Accordingly I think what would really help is a more meaningful
message
but currently I don't see how to implement that with reasonable effort
because that code is used in too many different cases.
For example how it looks on the terminal
when I reproduce it how it fails on my system:
# usr/sbin/rear -D mkrescue ... ERROR: Could not mkdir '/tmp/rear.2uvvEzDMiEq8JGa/outputfs/f144' Aborting due to an error, check /root/rear.github.master/var/log/rear/rear-f144.log for details
where the original mkdir error message
mkdir: cannot create directory '/tmp/rear.2uvvEzDMiEq8JGa/outputfs/f144': Permission denied
is hidden from the user in the log file.
If the user could see the original mkdir error message on his
terminal
he would get a hint that points to the root cause Permission denied
but currently I have no good idea how to show the original error
message
on the terminal only in case of an error.
As a quick test hwo to do that I changed the code in output/default/200_make_prefix_dir.sh to
# create $OUTPUT_PREFIX sub-directory under the mounted network filesystem share mkdir -p $v -m0750 "$opath" >&2 && return 0 LogPrintError "Could not mkdir '$opath' - some last log file lines:" # skip 'set -x' lines (i.e. lines that start with one or more '+' characters) and # skip lines of the 'Log' function (i.e. lines that have a timestamp that contains 'date +%Y-%m-%d') # so that normally only the normal stdout and stderr messages of the called programs are shown here: LogUserOutput "$( grep -E -v "^\+|$( date +%Y-%m-%d )" $RUNTIME_LOGFILE | tail -n 10 | sed -e 's/^/ /' )" Error "Failed to create $OUTPUT_PREFIX under the mounted network filesystem share"
Now things look more meaningful to me on the terminal:
Could not mkdir '/tmp/rear.1N5Gvjc6ewWUARM/outputfs/f144' - some last log file lines: ~/rear.github.master mkdir: created directory '/tmp/rear.1N5Gvjc6ewWUARM/outputfs' mount.nfs: trying 10.160.4.244 prog 100003 vers 3 prot TCP port 2049 mount.nfs: trying 10.160.4.244 prog 100005 vers 3 prot UDP port 20048 mount.nfs: timeout set for Mon Jul 23 14:21:11 2018 mount.nfs: trying text-based options 'nfsvers=3,nolock,addr=10.160.4.244' mount.nfs: prog 100003, trying vers=3, prot=6 mount.nfs: prog 100005, trying vers=3, prot=17 mkdir: created directory '/tmp/rear.1N5Gvjc6ewWUARM/tmp/boot' mkdir: cannot create directory '/tmp/rear.1N5Gvjc6ewWUARM/outputfs/f144': Permission denied ERROR: Failed to create f144 under the mounted network filesystem share Aborting due to an error, check /root/rear.github.master/var/log/rear/rear-f144.log for details
But it also shows how complicated it could become to implement
meaningful error output for each particular case in practice...
jsmeix commented at 2018-07-23 12:24:¶
I will have to meditate on it...
Perhaps the Error function could be the right place where
such more meaningful error output on the user's terminal
could be implemented in a generic way...
jsmeix commented at 2018-07-23 15:32:¶
I think the Error function is really the right place where such more
meaningful
error output on the user's terminal can be shown in a generic way
(i.e. where things can be implemented with reasonable effort):
https://github.com/rear/rear/pull/1877
jsmeix commented at 2018-07-26 11:37:¶
With
https://github.com/rear/rear/pull/1877
merged
the Error function shows now some of the last log messages
since the last actual script got sourced that contain in particular
the normal stdout and stderr messages of the last called programs
to make the root cause of an error more obvious to the user
without the need to analyze the log file in any case.
[Export of Github issue for rear/rear.]