#236 Issue closed: ROOTFS_DIR '/tmp/rear.H6HxCSznsf34ump/rootfs' is broken, chroot bash test failed.¶
Labels: waiting for info, discuss / RFC
rapmancz opened issue at 2013-06-07 12:24:¶
While running
rear -vd mkbackup
In the log, I can see:
====================
=== Stack trace ===
Trace 0: /usr/sbin/rear:245 main
Trace 1: /usr/share/rear/lib/mkbackup-workflow.sh:20 WORKFLOW_mkbackup
Trace 2: /usr/share/rear/lib/framework-functions.sh:79 SourceStage
Trace 3: /usr/share/rear/lib/framework-functions.sh:40 Source
Trace 4: /usr/share/rear/build/default/98_verify_rootfs.sh:17 source
Trace 5: /usr/share/rear/lib/_input-output-functions.sh:143 BugError
Message: BUG BUG BUG! ROOTFS_DIR '/tmp/rear.rVUxzrBDE3PPVzb/rootfs' is broken, chroot bash test failed.
=== Issue report ===
Please report this unexpected issue at: https://github.com/rear/rear/issues
Also include the relevant bits from /var/log/rear/rear-clu-node1.log
HINT: If you can reproduce the issue, try using the -d or -D option !
====================
My configuration:
OUTPUT=ISO
BACKUP=NETFS
BACKUP_URL=cifs://10.60.2.101/PRimages/czzdlc020
BACKUP_OPTIONS="cred=/etc/rear/.cifs"
#MODULES_LOAD=( vmxnet )
#DHCLIENT_BIN=dhclient
I can mount the path by hand using the same command as rear is using.
Do you have any idea, what could be wrong? This is real CentOS machine
with raid, Relax-and-Recover 1.14 / Git
When I use this configuration on Virtual machine, it is able to create the backup.
dagwieers commented at 2013-06-07 16:51:¶
The test that failed with this BUG BUG BUG error is in fact a test to see if your chroot environment (which will be used for the recovery) spawns a bash shell sucessfully. This is a very basic test that needs to work and in your case it fails for some reason.
It would be nice to know what happened (what was in the output) right before the stack trace. It might give us a clue. Another option is that you check the initrd yourself by doing:
chroot $ROOTFS_DIR bash -c true
using the correct ROOTFS_DIR (where the initrd is mounted).
PS Maybe the error should not be as loud as it is, as it masks a bit the
real error message:
Message: BUG BUG BUG! ROOTFS_DIR '/tmp/rear.rVUxzrBDE3PPVzb/rootfs'
is broken, chroot bash test failed.
rapmancz commented at 2013-06-10 12:39:¶
Here is a cut from the log file
2013-06-10 14:37:34.175875512 WARNING: unmatched external call to 'mpath-iosched.sh' in lib/udev/rules.d/80-mpath-iosched.rules
2013-06-10 14:37:34.179659526 WARNING: unmatched external call to 'scsi_id' in lib/udev/rules.d/60-persistent-storage.rules
2013-06-10 14:37:34.183945461 WARNING: unmatched external call to 'udev-acl' in lib/udev/rules.d/70-acl.rules
2013-06-10 14:37:34.187722266 WARNING: unmatched external call to 'write_cd_rules' in lib/udev/rules.d/75-cd-aliases-generator.rules
2013-06-10 14:37:34.283917279 Including build/default/97_add_rear_release.sh
2013-06-10 14:37:34.288750383 Including build/default/98_verify_rootfs.sh
chroot: failed to run command `bash': Permission denied
2013-06-10 14:37:34.300571559 ERROR: BUG BUG BUG! ROOTFS_DIR '/tmp/rear.VPoU6igWoz7riJG/rootfs' is broken, chroot bash test failed.
dagwieers commented at 2013-06-10 16:25:¶
All right, so up to you to find out why the bash returns permission denied. Using the procedure mentioned above.
rapmancz commented at 2013-06-11 08:59:¶
This is a production machine, I am affraid to experiment too much. What do you mean with correct ROOTFS_DIR?
Mount output:
[root@clu-node1 ~]# mount
/dev/mapper/vg_slave-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/cciss/c0d0p1 on /boot type ext4 (rw,noexec,nosuid,nodev)
/dev/mapper/vg_slave-lv_home on /home type ext4 (rw,nosuid,nodev,noatime)
/dev/mapper/vg_slave-lv_tmp on /tmp type ext4 (rw,noexec,nosuid,nodev)
/dev/mapper/vg_slave-lv_var on /var type ext4 (rw,nosuid)
/dev/mapper/vg_slave-lv_varlog on /var/log type ext4 (rw,noexec,nosuid,nodev)
/tmp on /var/tmp type none (rw,noexec,nosuid,nodev,bind)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
none on /sys/kernel/config type configfs (rw)
locate initrd output:
[root@clu-node1 ~]# locate initrd
/boot/initrd-2.6.32-131.0.15.el6.x86_64kdump.img
/boot/initrd-2.6.32-220.2.1.el6.x86_64kdump.img
/sbin/lsinitrd
/sbin/mkinitrd
/usr/lib/python2.6/site-packages/sos/plugins/initrd.py
/usr/lib/python2.6/site-packages/sos/plugins/initrd.pyc
/usr/lib/python2.6/site-packages/sos/plugins/initrd.pyo
/usr/libexec/plymouth/plymouth-generate-initrd
/usr/libexec/plymouth/plymouth-populate-initrd
/usr/libexec/plymouth/plymouth-update-initrd
/usr/share/doc/kernel-doc-2.6.32/Documentation/initrd.txt
/usr/share/dracut/modules.d/50plymouth/plymouth-populate-initrd
/usr/share/man/man4/initrd.4.gz
/usr/share/rear/output/USB/Linux-i386/83_copy_kernel_initrd.sh
/usr/share/rear/rescue/GNU/Linux/26_collect_initrd_modules.sh
/var/lib/rear/recovery/initrd_modules
gdha commented at 2013-06-11 12:33:¶
run the command rear -v -d -D mkbackup which will not remove the
ROOTFS_DIR which can be found under
/tmp/rear.<random-stuff>/rootfs
Then execute the chroot command as explained by @dagwieers
rapmancz commented at 2013-06-11 15:55:¶
OK, here it is:
[root@clu-node1 tmp]# rear -v -d -D mkbackup
Relax-and-Recover 1.14 / Git
Using log file: /var/log/rear/rear-clu-node1.log
Creating disk layout
Creating root filesystem layout
Copying files and directories
Copying binaries and libraries
Copying kernel modules
ERROR: BUG BUG BUG! ROOTFS_DIR '/tmp/rear.WXQtDTE6x3W6FQH/rootfs' is broken, chroot bash test failed.
=== Issue report ===
Please report this unexpected issue at: https://github.com/rear/rear/issues
Also include the relevant bits from /var/log/rear/rear-clu-node1.log
HINT: If you can reproduce the issue, try using the -d or -D option !
====================
Aborting due to an error, check /var/log/rear/rear-clu-node1.log for details
You should also rm -Rf /tmp/rear.WXQtDTE6x3W6FQH
Terminated
[root@clu-node1 tmp]# chroot /tmp/rear.WXQtDTE6x3W6FQH/rootfs/ bash -c true
chroot: failed to run command `bash': Permission denied
rapmancz commented at 2013-06-11 16:14:¶
could it be that the /tmp is mounted with noexec?
Here is a cut from the fstab
[root@clu-node1 etc]# cat fstab |grep /tmp
/dev/mapper/vg_slave-lv_tmp /tmp ext4 defaults,nodev,noexec,nosuid 1 2
/tmp /var/tmp none defaults,nodev,noexec,nosuid,bind 0 0
gdha commented at 2013-06-12 06:45:¶
@rapmancz Indeed that is the culprit "noexec" on /tmp which means
bash cannot execute.
This is what script usr/share/rear/build/default/98_verify_rootfs.sh
tests.
Could you copy/paste the output from the rear.log file which was
executed by script usr/share/rear/build/default/98_verify_rootfs.sh?
Would like to see what exactly happened...
rapmancz commented at 2013-06-12 14:24:¶
When I run the script by hand, the error is the same:
[root@clu-node1 /]#
/usr/share/rear/build/default/98_verify_rootfs.sh
-bash: /usr/share/rear/build/default/98_verify_rootfs.sh: Permission
denied
I don't see anything in the logfile.
But I can confirm that after mounting the partition executable ( using
mount -o remount,rw,exec,nosuid /dev/mapper/vg_slave-lv_tmp ), I can
create the backup
dagwieers commented at 2013-06-19 19:13:¶
Alright, so the solution here is to inform the user about this issue by adding a check.
Shouldn't be hard, we have a function for getting the mountpoint of a
path, and a single grep in /proc/mounts would be very conclusive.
Should we also check for nodev ?
gdha commented at 2014-04-16 12:08:¶
to my best knowledge an additional test has been added in
build/default/98_verify_rootfs.sh script. The nodev check: not sure
we need that?
gdha commented at 2014-05-06 07:32:¶
rear-1.16 is out and fix is included
gamezdaniel commented at 2015-04-23 15:47:¶
Hello Gratien,
I'm using "Relax-and-Recover 1.17.0" version on RHEL5.5 (x86_64 -
2.6.18-348.el5 GNU/Linux), and I'm getting the same problem issued in
this thread.
Was it fixed at all?
Many thanks,
Dan.
[Export of Github issue for rear/rear.]