#150 Issue closed: Relax-and-Recover fails when /tmp is mounted noexec

Labels: enhancement, support / question

schoekek opened issue at 2012-09-01 09:59:

Hi,

i have this error since version 1.12...
I installed today the 1.13.0 from suse Buildservice on Debian 6.0.5.

many thx for ideas,

Karsten

dagwieers commented at 2012-09-01 21:12:

Could you please test the snapshot release ? (Which is incidently also called 1.13.0 with a git-timestamp)

Also, we would need more information in order to help you.

@jhoekx I think we need to make a small troubleshooting guide that instructs people to try the most recent release from the master branch, perform some tests and to send more detailed information (configuration, version and logs).

schoekek commented at 2012-09-02 08:30:

I try the shapshot release rear_1.13.0-0git201208292128_all.deb frome suse built service.
The error ist the same.
Here is the output:

rear -dD mkbackup
.
.
Copying binaries and libraries
Copying kernel modules
ERROR: BUG BUG BUG!  ROOTFS_DIR '/tmp/rear.WTjmuRD2dQETSQc/rootfs' is broken, chroot bash test failed.
=== Issue report ===
Please report this unexpected issue at: https://github.com/rear/rear/issues
Also include the relevant bits from /var/log/rear/rear-legolas.log

HINT: If you can reproduce the issue, try using the -d or -D option !
====================
Aborting due to an error, check /var/log/rear/rear-legolas.log for details
You should also rm -Rf /tmp/rear.WTjmuRD2dQETSQc
Beendet

The local.conf file ist emty.
The file /var/log/rear/rear-legolas.log is very log, can you mail it?

Whis infos need you??

Karsten

dagwieers commented at 2012-09-02 11:52:

I try the shapshot release rear_1.13.0-0git201208292128_all.deb frome suse built service.
The error ist the same.

Thanks for testing.

In order to debug Relax-and-Recover retains the temporary directory in /tmp/, can you try to do a: chroot /tmp/rear.WTjmuRD2dQETSQc /bin/bash

This is in fact what is probably failing. I hope the output of that command to be useful.

PS You can upload log files through gist: https://gist.github.com/

jhoekx commented at 2012-09-02 14:15:

I will try to reproduce this in a VM. Are you on Debian 64-bit?

schoekek commented at 2012-09-02 14:22:

thx for your Time on sunday!

I use Debian on 32 Bit

legolas:~# uname -a
Linux legolas 2.6.32-5-686 #1 SMP Sun May 6 04:01:19 UTC 2012 i686 GNU/Linux
legolas:~# cat /etc/debian_version
6.0.5

Here ist the output:

# chroot /tmp/rear.qPmcLixJUznp1E7 /bin/bash
chroot: failed to run command `/bin/bash': No such file or directory

I try strace -f chroot /tmp/rear.qPmcLixJUznp1E7 /bin/bash 2>&1
Here is the output:

execve("/usr/sbin/chroot", ["chroot", "/tmp/rear.qPmcLixJUznp1E7", "/bin/bash"], [/* 16 vars */]) = 0
brk(0)                                  = 0x9445000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb77d1000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=76406, ...}) = 0
mmap2(NULL, 76406, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb77be000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/i686/cmov/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\0n\1\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1327556, ...}) = 0
mmap2(NULL, 1337704, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7677000
mprotect(0xb77b7000, 4096, PROT_NONE)   = 0
mmap2(0xb77b8000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x140) = 0xb77b8000
mmap2(0xb77bb000, 10600, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb77bb000
close(3)                                = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7676000
set_thread_area({entry_number:-1 -> 6, base_addr:0xb76768d0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
mprotect(0xb77b8000, 8192, PROT_READ)   = 0
mprotect(0xb77ef000, 4096, PROT_READ)   = 0
munmap(0xb77be000, 76406)               = 0
brk(0)                                  = 0x9445000
brk(0x9466000)                          = 0x9466000
open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=1527680, ...}) = 0
mmap2(NULL, 1527680, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7501000
close(3)                                = 0
chroot("/tmp/rear.qPmcLixJUznp1E7")     = 0
chdir("/")                              = 0
execve("/bin/bash", ["/bin/bash"], [/* 16 vars */]) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/locale.alias", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/de_DE.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/de_DE.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/de_DE/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/de.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/de.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/de/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, "chroot: ", 8chroot: )                 = 8
write(2, "failed to run command `/bin/bash"..., 33failed to run command `/bin/bash') = 33
open("/usr/share/locale/de_DE.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/de_DE.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/de_DE/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/de.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/de.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/de/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, ": No such file or directory", 27: No such file or directory) = 27
write(2, "\n", 1
)                       = 1
close(1)                                = 0
close(2)                                = 0
exit_group(127)                         = ?

Can help this?

jhoekx commented at 2012-09-02 15:16:

I did only test with 64 bit debian. Tested it again right now and that works. I will install a 32 bit version.

jhoekx commented at 2012-09-02 16:00:

So I tested in debian 6.0.5 i386 and everything worked great.

Can you check that there are no files from a previous Relax-and-Recover installation present?

Otherwise, please show ls -l /.

Mine looks like:

root@debianrear32:~# ls -l /
total 128
drwxr-xr-x  2 root   root    4096 Sep  2 17:41 bin
drwxr-xr-x  3 root   root    4096 Sep  2 17:42 boot
drwxr-xr-x 14 root   root    2940 Sep  2 17:54 dev
drwxr-xr-x 67 root   root    4096 Sep  2 17:54 etc
drwxr-xr-x  3 root   root    4096 Sep  2 17:43 home
lrwxrwxrwx  1 root   root      28 Sep  2 17:52 initrd.img -> boot/initrd.img-2.6.32-5-686
drwxr-xr-x 12 root   root    4096 Sep  2 17:47 lib
drwx------  2 root   root   16384 Sep  2 17:27 lost+found
drwxr-xr-x  3 root   root    4096 Sep  2 17:27 media
drwxr-xr-x  2 root   root    4096 May  7 16:55 mnt
drwxr-xr-x  2 root   root    4096 Sep  2 17:28 opt
dr-xr-xr-x 69 root   root       0 Sep  2 17:54 proc
drwx------  4 root   root    4096 Sep  2 17:52 root
drwxr-xr-x  2 root   root    4096 Sep  2 17:47 sbin
drwxr-xr-x  2 root   root    4096 Jul 21  2010 selinux
drwxr-xr-x  2 root   root    4096 Sep  2 17:28 srv
drwxr-xr-x 12 root   root       0 Sep  2 17:54 sys
drwxrwxrwt  2 root   root    4096 Sep  2 17:54 tmp
drwxr-xr-x 10 root   root    4096 Sep  2 17:28 usr
drwxr-xr-x 13 root   root    4096 Sep  2 17:28 var
lrwxrwxrwx  1 root   root      25 Sep  2 17:52 vmlinuz -> boot/vmlinuz-2.6.32-5-686

And ldd /bin/bash:

root@debianrear32:~# ldd /bin/bash 
        linux-gate.so.1 =>  (0xb7778000)
        libncurses.so.5 => /lib/libncurses.so.5 (0xb7738000)
        libdl.so.2 => /lib/i686/cmov/libdl.so.2 (0xb7734000)
        libc.so.6 => /lib/i686/cmov/libc.so.6 (0xb75ec000)
        /lib/ld-linux.so.2 (0xb7779000)

schoekek commented at 2012-09-02 16:30:

I search for old file at any time...
All old files was deleted.

legolas:~# ldd /bin/bash
        linux-gate.so.1 =>  (0xb778d000)
        libncurses.so.5 => /lib/libncurses.so.5 (0xb773e000)
        libdl.so.2 => /lib/i686/cmov/libdl.so.2 (0xb773a000)
        libc.so.6 => /lib/i686/cmov/libc.so.6 (0xb75f2000)
        /lib/ld-linux.so.2 (0xb778e000)

Mein Debian was updated from Etch to Lenny to Squeezy...

schoekek commented at 2012-09-02 16:34:

sorry i press the Close button ;-(

dagwieers commented at 2012-09-02 19:57:

Ok, I think we started off on the wrong foot because I gave you the wrong command, it should have been: chroot /tmp/rear.WTjmuRD2dQETSQc/rootfs /bin/bash.

jhoekx commented at 2012-09-03 06:37:

Can you run rear -D mkrescue and copy the logfile in a gist. That will allow us to figure out what happened that makes the chroot test fail.

schoekek commented at 2012-09-03 16:34:

oh, no prblem, here ist the output of the command:

legolas:~# chroot /tmp/rear.qPmcLixJUznp1E7/rootfs /bin/bash
chroot: failed to run command `/bin/bash': Permission denied

schoekek commented at 2012-09-03 16:52:

and here is rear output:

legolas:~# rear -dD mkbackup
Relax-and-Recover 1.13.0-git201208292128 / 2012-08-29
Using log file: /var/log/rear/rear-legolas.log
Creating disk layout
Creating root filesystem layout
TIP: To login as root via ssh you need to set up /root/.ssh/authorized_keys
Copying files and directories
Copying binaries and libraries
Copying kernel modules
ERROR: BUG BUG BUG!  ROOTFS_DIR '/tmp/rear.BoILxSWM3SFKL5R/rootfs' is broken, chroot bash test failed.
=== Issue report ===
Please report this unexpected issue at: https://github.com/rear/rear/issues
Also include the relevant bits from /var/log/rear/rear-legolas.log

HINT: If you can reproduce the issue, try using the -d or -D option !
====================
Aborting due to an error, check /var/log/rear/rear-legolas.log for details
You should also rm -Rf /tmp/rear.BoILxSWM3SFKL5R
Beendet

and the logfile /var/log/rear/rear-legolas.log https://gist.github.com/3610710

gdha commented at 2012-09-04 06:39:

what is the output of ll /tmp/rear.qPmcLixJUznp1E7/rootfs/bin/bash ?

jhoekx commented at 2012-09-04 11:46:

Is your /tmp mounted noexec? grep /tmp /proc/mounts

I can get the exact error message as you've got when I change my /tmp to be noexec.

schoekek commented at 2012-09-04 15:36:

Yes /tmp ist noexec mounted...
this is for a long time

dagwieers commented at 2012-09-04 16:42:

@lgbff Thanks for the clarification !
@jhoekx Thanks for the detective work (or lucky guess ? :-)

Ok, so Relax-and-Recover expects /tmp to be exec mounted. How are we going to handle this ?

  • Check /tmp for noexec, halt and escalate problem ?
  • Do we use another location that is less likely to be noexec (e.g. /var/lib/rear) ?
  • Do we temporarily disable noexec by remounting and then go back to exec ?
  • Are there any other workarounds ? We do not actually need exec other than testing bash :-( The irony here is that it's not bash that is hurting us, but the test to see if it works correctly (which it probably will !)

schlomo commented at 2012-09-04 16:46:

Why not use /dev/shm?

/tmp is probably the worst possible location and pure "legacy"...

Or even better: Use /dev/shm by default and allow user to change the
location.

gdha commented at 2012-09-04 16:51:

Or, if /tmp is noexec mounted skip the test (saying testing is not possible)?

schoekek commented at 2012-09-04 17:04:

I remount /tmp without noexec option and rear run fine ;-)

(my /var is mounted noexec too)

dagwieers commented at 2012-09-04 18:45:

@schlomo I don't like /dev/shm for this, I fear a risk of trashing the system because we pressure memory too high. (default is 50%of RAM)

@gdha That's probably the best suggestion. Just alert the condition, in 99% of the cases there is no issue and it is up to the user (in this case) to ensure the recovery process works as expected. Maybe we can combine this together with a check for other possible location (before giving up, disabling the check and alerting the user).

jhoekx commented at 2012-09-05 06:28:

I found this by thinking how you could get a permission denied error when running bash after you've written it.

Well, /tmp is also a tmpfs these days on non-enterprise systems, so memory pressure is not really an argument.

I think a combination of making TMP_DIR configurable with a check and warning if /tmp is mounted noexec would be the best solution for now.

dagwieers commented at 2012-09-07 09:02:

I am closing this issue. Feel free to report if this feature can be improved or fails to work in your case.

dagwieers commented at 2012-09-09 19:48:

I am wondering about one thing with my implementation though. If Relax-and-Recover is started in quiet mode (default) no warning messages are shown. So in case you run rear via cron, it won't escalate this warning. I can see cases where this is expected (i.e. the user tested it himself thoroughly and wants to leave /tmp noexec) but also cases where it is not (i.e. user is unaware due to this and might be at risk).

Not sure what is best, the logfile does have the warning so if the user checked the logfile for warnings or errors, he should have known. But you know how ignorant users can be ;-))

schlomo commented at 2012-09-09 20:05:

IMHO this should be a fatal error. Let the user give ReaR another work dir
where it can do the chroot test. IMHO this test is very important to ensure
that the resulting initrd has a chance to function.

Who knows. Maybe a users changes /tmp to noexec long time after setting up
ReaR?

dagwieers commented at 2012-09-09 20:17:

@schlomo Maybe we can change it to a fatal error once we have issue #157 solved. I am open to that.

kollyma commented at 2019-04-17 13:19:

We are facing the same issue, because /tmp mouted with noexec permissions:

ERROR: Cannot test if the ReaR recovery system is usable because /tmp is mounted 'noexec'
Aborting due to an error, check /var/log/rear/rear-serverxy.log for details

set the TMDIR variable to fix the issue:

export TMPDIR='/var/tmp/' ; /usr/sbin/rear mkbackup

[Export of Github issue for rear/rear.]