#1555 Issue closed: rear mkrescue fails to copy files and later complains about ldd /bin/bash

Labels: bug, fixed / solved / done

efwe opened issue at 2017-10-28 12:34:

  • rear version (/usr/sbin/rear -V):
    Relax-and-Recover 2.2 / Git
  • OS version (cat /etc/rear/os.conf or lsb_release -a):
    LSB Version: 1.4
    Distributor ID: Arch
    Description: Arch Linux
    Release: rolling
    Codename: n/a
    OUTPUT=USB
    BACKUP=NETFS
    BACKUP_URL=usb:///dev/disk/by-label/REAR-000
  • Are you using legacy BIOS or UEFI boot?
    BIOS
  • Brief description of the issue:
    During rear -d -D mkrescue I see failures during copy and it eventually aborts with
    BUG in /usr/share/rear/build/default/980_verify_rootfs.sh line 29
  • Debug information
    rear-osa.log
  • Work-around, if any:
    Not found yet

Sorry if this is a simple configuration problem, but I don't see it now.
Greetings
~fw

efwe commented at 2017-10-28 12:59:

I think problems start here

++ local dir=/usr/lib64
++ test -d /tmp/rear.WiCMxKkJL0E5UG0/rootfs//usr/lib64
++ mkdir -v -p /tmp/rear.WiCMxKkJL0E5UG0/rootfs//usr/lib64
mkdir: cannot create directory '/tmp/rear.WiCMxKkJL0E5UG0/rootfs//usr/lib64': File exists
++ test -e /tmp/rear.WiCMxKkJL0E5UG0/rootfs//usr/lib64/libfreebl3.so

this double-slash thing is throughout the log. but here it fails
test returns 1 and mkdir fails later. stat returns 0 for this dir.
How did I mess things up here? I feel slightly guilty for not seeing somethig obivous :)

~fw

gozora commented at 2017-10-28 14:44:

Hello @efwe,
Just by quick checking your logs, I guess something strange is happening here.
In your log file:

++ chroot /tmp/rear.WiCMxKkJL0E5UG0/rootfs /bin/ldd /bin/bash
    not a dynamic executable

Which could mean that whatever ReaR copied into its rootfs as bash, can't be examined with ldd.

Can you please run following command on your original system: ldd /bin/bash and post output here?
If above command works fine (you don't get any error) could you pack contents of /tmp/rear.WiCMxKkJL0E5UG0/rootfs and upload it here or any other location so I can examine it a bit closer?

Thanks

V.

efwe commented at 2017-10-28 15:16:

Hello @gozora,
thanks for your quick reply. So here is some more information.
first the reproducer from the log

[root@osa ~]# chroot /tmp/rear.WiCMxKkJL0E5UG0/rootfs
Welcome to Relax-and-Recover. Run "rear recover" to restore your system !
RESCUE osa:/ # /bin/ldd /bin/bash
    not a dynamic executable
RESCUE osa:/ # file /bin/bash
file: could not find any valid magic files!
RESCUE osa:/ # ls -lrst /bin/bash
812 -rwxr-xr-x 1 root root 828320 Feb 14  2017 /bin/bash

RESCUE osa:/ # exit

The original version looks better of course

[root@osa ~]# ls -l /bin/bash
-rwxr-xr-x 1 root root 828320 Feb 14  2017 /bin/bash
[root@osa ~]# ldd /bin/bash
    linux-vdso.so.1 (0x00007ffd7e523000)
    libreadline.so.7 => /usr/lib/libreadline.so.7 (0x00007f4358814000)
    libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f4358610000)
    libc.so.6 => /usr/lib/libc.so.6 (0x00007f4358259000)
    libncursesw.so.6 => /usr/lib/libncursesw.so.6 (0x00007f4358021000)
    /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f4358a62000)
    libtinfo.so.6 => /usr/lib/libtinfo.so.6 (0x00007f4357df5000)
[root@osa ~]#

so maybe if the copying for the /usr/lib64 fails it makes ldd failing along as the ld-linux would not be avail?
I'll work on the debug-package a little bit later.

Again, thanks
~fw

gozora commented at 2017-10-28 16:05:

thanks for your quick reply

No problem ...

Talking about failed to copy missing libs, I really can't understand one thing.
First ReaR makes test whether destination directory exists

++ test -d /tmp/rear.WiCMxKkJL0E5UG0/rootfs//usr/lib64

This test fails and ReaR tries to create it

++ mkdir -v -p /tmp/rear.WiCMxKkJL0E5UG0/rootfs//usr/lib64

But here mkdir fails with:

mkdir: cannot create directory '/tmp/rear.WiCMxKkJL0E5UG0/rootfs//usr/lib64': File exists

This IMHO can mean only one thing, namespace /tmp/rear.WiCMxKkJL0E5UG0/rootfs//usr/lib64 exists and it is NOT a directory. My best guess would be that it exists as symlink to some file (or maybe broken symlink).

V.

efwe commented at 2017-10-28 16:27:

Hello @gozora,
you're right

host-system:

[root@osa ~]# ls -l /usr/lib64
lrwxrwxrwx 1 root root 3 Mar 26  2017 /usr/lib64 -> lib

chroot:

RESCUE osa:~ # ls -l /usr/lib64
lrwxrwxrwx 1 root root 7 Oct 28 12:27 /usr/lib64 -> usr/lib

I was completely unaware of the fact, that this is a link.
But it looks like rear chose a not so good link-target.

Hope, that helps
~fw

P.S.: https://123k.work/rootfs.tar.gz

gozora commented at 2017-10-28 16:40:

Does link destination (usr/lib) exists in your chrooted environment?

efwe commented at 2017-10-28 16:55:

ehh, no. That would make /usr/usr/lib later?

gozora commented at 2017-10-28 17:00:

Let me be a bit more accurate.
After you run chroot like chroot /tmp/rear.WiCMxKkJL0E5UG0/rootfs what will be output of ls -al /usr/lib ?

efwe commented at 2017-10-28 17:12:

Ok :)

first contents of ls -l /usr/

[root@osa lib64]#  chroot /tmp/rear.WiCMxKkJL0E5UG0/rootfs

Welcome to Relax-and-Recover. Run "rear recover" to restore your system !

RESCUE osa:/ # ls -l /usr/
total 0
lrwxrwxrwx 1 root root    6 Oct 28 12:27 bin -> ../bin
drwxr-xr-x 2 root root   40 Oct 27 18:44 include
drwxr-xr-x 8 root root 4340 Oct 28 12:28 lib
lrwxrwxrwx 1 root root    7 Oct 28 12:27 lib64 -> usr/lib
lrwxrwxrwx 1 root root    6 Oct 28 12:27 sbin -> ../bin
drwxr-xr-x 9 root root  180 Oct 28 12:27 share
RESCUE osa:/ #

and here the output of ls -al /usr/lib

ls.txt

gozora commented at 2017-10-28 17:30:

Well, this looks really like a bug to me, because accessing /usr/lib64 in your chroot would be actually a try to access /usr/usr/lib64 (which I assume does not exist).
I'd need to install Arch and check this behavior deeper.
I'll keep you posted about the progress.

@efwe Thanks for reporting this!

V.

jsmeix commented at 2017-11-07 08:52:

With https://github.com/rear/rear/pull/1557 merged
I consider this issue to be fixed (if not it can be reopended).

aochkintr commented at 2019-05-08 16:07:

got same ( similar) failure with rear v2.4 and then v.2.3 on RHEL 7.6
+++++++ excerpt from /var/log/rear/rear-aragorn.log
...

ReaR2019-05-08 09:29:48.969211462 Including build/default/960_remove_encryption_keys.sh ReaR2019-05-08 09:29:48.974476692 Including build/default/970_add_rear_release.sh ReaR2019-05-08 09:29:48.981849145 Including build/default/980_verify_rootfs.sh ReaR2019-05-08 09:29:48.990485974 Testing that /tmp/rear.xFM4znPxHfr26uX/rootfs contains a usable s ystem ReaR2019-05-08 09:29:48.996730694 Testing 'ldd /bin/bash' to ensure 'ldd' works for the subsequent 'ldd' tests chroot: failed to run command '/bin/ldd': No such file or directory ReaR2019-05-08 09:29:49.002383819 ERROR:

BUG in /usr/share/rear/build/default/980_verify_rootfs.sh line 29:
'ReaR recovery system in '/tmp/rear.xFM4znPxHfr26uX/rootfs' is broken: 'ldd /bin/bash' failed'

[Export of Github issue for rear/rear.]