#2266 Issue closed: In CentOS 7.7 and 8.0 certain programs (e.g. dhclient) require additional libraries¶
Labels: enhancement, documentation, fixed / solved / done
DamaniN opened issue at 2019-10-25 16:31:¶
Relax-and-Recover (ReaR) Issue Template¶
Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):
- ReaR version ("/usr/sbin/rear -V"):
Relax-and-Recover 2.5 / Git
- OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"):
CentOS Linux release 7.7.1908 (Core)
- ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
OUTPUT=ISO
BACKUP=CDM
- Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR):
vSphere Virtual Machine
- System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
x86 compatible
- Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
UEFI and Grub
- Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
vSphere virtual disks.
- Description of the issue (ideally so that others can reproduce it):
When rear mkrescue is run on a CentOS 7.7 system as a pre-script in
Rubrik CDM it exits with a return code of 15. The logs indicate that the
libisc-export.so.169 and libdns-export.so.1102 libraries needed for
/bin/dhclient are not found. I looked for /bin/dhclient on this
system but found that it does not exist. Instead, there is a
/sbin/dhclinet. Further analysis of the logs shows that Rear copies
/sbin/dhclient to /tmp/rear.vOpjUNKmz87uhqQ/rootfs/bin/dhclient. I
assume this is where dhclient is being tested for libraries with
ldd.
When I tested /sbin/dhclient using ldd the libisc-export.so.169
and libdns-export.so.1102 libraries were found. They are, however,
listed as being in /usr/lib64//bind9-export/ which is different than
/lib64 where the other libraries are. I also noticed that the path has
// in it. This may be causing ReaR some issues. I then copied
/sbin/dhclient to /tmp/bin/dhclient and tested it there. ldd still
found the same thing.
I tried adding /usr/lib64//bind9-export/ to COPY_AS_IS and
COPY_AS_IS_CDM but this still resulted in the same error. If I run
rear mkrescue from the OS command line it completes successfully. It
seems that there's some issue with the environment that CDM's pre-script
runs in but I'm not sure what that may be or how to proceed. I did see
the notes in #1562 about setting the LD_LIBRARY_PATH, however,
dhclient is not a CDM binary but an OS one.
- Workaround, if any:
None
- Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
Error in CDM:
Failed executing pre-backup script for Fileset 'dn-allthethings-with-rear(dn-rear-centos-77)'. Script finished with error code '15'
Error in ReaR log:
2019-10-25 05:40:48.452331193 Testing 'ldd /bin/bash' to ensure 'ldd' works for the subsequent 'ldd' tests within the recovery system
linux-vdso.so.1 => (0x00007ffecf323000)
libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00007f9dd2c81000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f9dd2a7d000)
libc.so.6 => /lib64/libc.so.6 (0x00007f9dd26af000)
/lib64/ld-linux-x86-64.so.2 (0x00007f9dd2eab000)
2019-10-25 05:40:48.464352730 Testing each binary (except links) with ldd and look for 'not found' libraries within the recovery system
2019-10-25 05:41:09.717055114 There are binaries or libraries in the ReaR recovery system that need additional libraries
2019-10-25 05:41:09.721113715 /bin/dhclient requires additional libraries (fatal error)
2019-10-25 05:41:09.731578550 linux-vdso.so.1 => (0x00007ffd98ddd000)
libomapi.so.0 => /lib64/libomapi.so.0 (0x00007fd3e8ab6000)
libcap-ng.so.0 => /lib64/libcap-ng.so.0 (0x00007fd3e88b0000)
libdns-export.so.1102 => not found
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fd3e8663000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fd3e837a000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fd3e8147000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fd3e7f43000)
libisc-export.so.169 => not found
libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007fd3e7ae0000)
libcap.so.2 => /lib64/libcap.so.2 (0x00007fd3e78db000)
libsystemd-daemon.so.0 => /lib64/libsystemd-daemon.so.0 (0x00007fd3e76d4000)
liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007fd3e74c5000)
libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007fd3e7270000)
libc.so.6 => /lib64/libc.so.6 (0x00007fd3e6ea2000)
/lib64/ld-linux-x86-64.so.2 (0x00007fd3e8f44000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fd3e6c92000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fd3e6a8e000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fd3e688a000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fd3e6671000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fd3e6455000)
libz.so.1 => /lib64/libz.so.1 (0x00007fd3e623f000)
libattr.so.1 => /lib64/libattr.so.1 (0x00007fd3e603a000)
librt.so.1 => /lib64/librt.so.1 (0x00007fd3e5e32000)
libm.so.6 => /lib64/libm.so.6 (0x00007fd3e5b30000)
libdw.so.1 => /lib64/libdw.so.1 (0x00007fd3e58df000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fd3e56c9000)
libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007fd3e54ac000)
libssl.so.10 => /lib64/libssl.so.10 (0x00007fd3e523a000)
libssl3.so => /lib64/libssl3.so (0x00007fd3e4fe1000)
libsmime3.so => /lib64/libsmime3.so (0x00007fd3e4db9000)
libnss3.so => /lib64/libnss3.so (0x00007fd3e4a8a000)
libnssutil3.so => /lib64/libnssutil3.so (0x00007fd3e485a000)
libplds4.so => /lib64/libplds4.so (0x00007fd3e4656000)
libplc4.so => /lib64/libplc4.so (0x00007fd3e4451000)
libnspr4.so => /lib64/libnspr4.so (0x00007fd3e4213000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fd3e3fec000)
libelf.so.1 => /lib64/libelf.so.1 (0x00007fd3e3dd4000)
liblzma.so.5 => /lib64/liblzma.so.5 (0x00007fd3e3bae000)
libbz2.so.1 => /lib64/libbz2.so.1 (0x00007fd3e399e000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fd3e3767000)
libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fd3e3505000)
libfreebl3.so => /lib64/libfreebl3.so (0x00007fd3e3302000)
2019-10-25 05:41:09.736426108 ReaR recovery system in '/tmp/rear.vOpjUNKmz87uhqQ/rootfs' needs additional libraries, check /var/log/rear/rear-dn-rear-centos-77.log for details
2019-10-25 05:41:09.738604320 Build area /tmp/rear.vOpjUNKmz87uhqQ will be removed
.
.
.
2019-10-25 05:41:10.813700090 ERROR: ReaR recovery system in '/tmp/rear.vOpjUNKmz87uhqQ/rootfs' not usable (required libraries are missing)
===== Stack trace =====
Trace 0: /usr/sbin/rear:541 main
Trace 1: /usr/share/rear/lib/mkrescue-workflow.sh:18 WORKFLOW_mkrescue
Trace 2: /usr/share/rear/lib/framework-functions.sh:116 SourceStage
Trace 3: /usr/share/rear/lib/framework-functions.sh:56 Source
Trace 4: /usr/share/rear/build/default/990_verify_rootfs.sh:214 source
=== End stack trace ===
2019-10-25 05:41:10.824754683 Exiting rear mkrescue (PID 23414) and its descendant processes ...
/usr/share/rear/lib/_input-output-functions.sh: line 122: pstree: command not found
2019-10-25 05:41:13.855986910 PID COMMAND
PID COMMAND
21556 /bin/bash /usr/sbin/rear -v mkrescue
/usr/share/rear/lib/_input-output-functions.sh: line 151: kill: (21562) - No such process
2019-10-25 05:41:13.878009219 Running exit tasks
2019-10-25 05:41:13.880200300 Finished in 53 seconds
2019-10-25 05:41:13.882141581 Removing build area /tmp/rear.vOpjUNKmz87uhqQ
removed directory: '/tmp/rear.vOpjUNKmz87uhqQ'
2019-10-25 05:41:14.367021766 End of program reached
The output of ldd /sbin/dhclient from the OS prompt:
ldd /sbin/dhclient
linux-vdso.so.1 => (0x00007ffc60345000)
libomapi.so.0 => /lib64/libomapi.so.0 (0x00007f79a07bb000)
libcap-ng.so.0 => /lib64/libcap-ng.so.0 (0x00007f79a05b5000)
libdns-export.so.1102 => /usr/lib64//bind9-export/libdns-export.so.1102 (0x00007f79a0187000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f799ff3a000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f799fc51000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f799fa1e000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f799f81a000)
libisc-export.so.169 => /usr/lib64//bind9-export/libisc-export.so.169 (0x00007f799f5af000)
libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f799f14c000)
libcap.so.2 => /lib64/libcap.so.2 (0x00007f799ef47000)
libsystemd-daemon.so.0 => /lib64/libsystemd-daemon.so.0 (0x00007f799ed40000)
liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007f799eb31000)
libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007f799e8dc000)
libc.so.6 => /lib64/libc.so.6 (0x00007f799e50e000)
/lib64/ld-linux-x86-64.so.2 (0x00007f79a0c49000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f799e2fe000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f799e0fa000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f799def6000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f799dcdd000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f799dac1000)
libz.so.1 => /lib64/libz.so.1 (0x00007f799d8ab000)
libattr.so.1 => /lib64/libattr.so.1 (0x00007f799d6a6000)
librt.so.1 => /lib64/librt.so.1 (0x00007f799d49e000)
libm.so.6 => /lib64/libm.so.6 (0x00007f799d19c000)
libdw.so.1 => /lib64/libdw.so.1 (0x00007f799cf4b000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f799cd35000)
libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007f799cb18000)
libssl.so.10 => /lib64/libssl.so.10 (0x00007f799c8a6000)
libssl3.so => /lib64/libssl3.so (0x00007f799c64d000)
libsmime3.so => /lib64/libsmime3.so (0x00007f799c425000)
libnss3.so => /lib64/libnss3.so (0x00007f799c0f6000)
libnssutil3.so => /lib64/libnssutil3.so (0x00007f799bec6000)
libplds4.so => /lib64/libplds4.so (0x00007f799bcc2000)
libplc4.so => /lib64/libplc4.so (0x00007f799babd000)
libnspr4.so => /lib64/libnspr4.so (0x00007f799b87f000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f799b658000)
libelf.so.1 => /lib64/libelf.so.1 (0x00007f799b440000)
liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f799b21a000)
libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f799b00a000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f799add3000)
libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f799ab71000)
libfreebl3.so => /lib64/libfreebl3.so (0x00007f799a96e000)
DamaniN commented at 2019-10-30 04:17:¶
I was able to fix this by adding this line to the /etc/rear/local.conf
file:
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/lib64/bind9-export"
I was also able to resolve a similar issue with CentOS 8.0 by adding
this line to the /etc/rear/local.conf file:
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/lib64/bind9-export:/usr/lib64/eog:/usr/lib64/python3.6/site-packages:/usr/lib64/samba:/usr/lib64/firefox"
I'll update the CDM documentation for this.
jsmeix commented at 2019-11-04 12:50:¶
I think this is not an issue only for CDM
but a generic issue with CentOS 7.7 and 8.0.
I am not a CentOS user so I cannot reproduce issues with CentOS
but I guess export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:..."
is not how things are done on original CentOS 7.7 and 8.0 systems
so I guess export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:..."
is not yet the right solution but only a workaround for now, cf.
https://github.com/rear/rear/pull/2270#issuecomment-549336774
gdha commented at 2019-11-04 13:57:¶
@jsmeix perhaps ldconfig -p could of be some use to find the correct
paths of the missing libraries?
jsmeix commented at 2019-11-04 14:39:¶
My probably too optimistic expectation was that
running ldconfig in the recovery system via
ldconfig $v -r "$ROOTFS_DIR
in
https://github.com/rear/rear/blob/master/usr/share/rear/build/GNU/Linux/390_copy_binaries_libraries.sh#L116
would do "the right things" so that normal programs in the recovery
system
could find their libraries in the recovery system.
jsmeix commented at 2019-11-18 11:17:¶
@DamaniN
I am neither a CentOS user so I cannot reproduce your issue
nor am I a dynamic linker expert so I cannot provide really
authoritative instructions how to debug dynamic linker issues
so all I can do is some naive info how the dynamic linker config
looks on my openSUSE Leap 15.0 system as an example
what you could do on your CentOS system to find out how
the dynamic linker config looks there that may finally help
to see what is special with the dynamic linker config on your
CentOS system that leads to this issue in the ReaR recovery system:
On my openSUSE Leap 15.0 system I have:
# type -a dhclient
dhclient is /sbin/dhclient
# ldd /sbin/dhclient
linux-vdso.so.1 (0x00007ffcc78eb000)
libc.so.6 => /lib64/libc.so.6 (0x00007f1cc270a000)
/lib64/ld-linux-x86-64.so.2 (0x00007f1cc2eb5000)
# rpm -qf /sbin/dhclient
dhcp-client-4.3.5-lp150.5.3.1.x86_64
# rpm -qi dhcp-client
...
URL : http://www.isc.org/software/dhcp
Summary : ISC DHCP Client
Description :
This is an alternative DHCP client, the ISC DHCP client for Linux.
Like "dhcpcd" (the client that is installed by default) ...
so dhclient on my openSUSE Leap 15.0 system
does not need any special libraries.
General dynamic linker things on my openSUSE Leap 15.0 system:
# ldconfig -v 2>/dev/null | grep ':$'
/usr/local/lib64:
/usr/local/lib:
/usr/lib64/graphviz:
/lib:
/lib64:
/usr/lib:
/usr/lib64:
those are the directories that are used by the run-time linker
to look for libraries.
The whole ldconfig -v output shows all those libraries
so that you could use a command like
ldconfig -v 2>/dev/null | egrep ':$|libdns-export|libisc-export'
to see where the run-time linker would find the
libraries libdns-export and libisc-export that are needed
by /sbin/dhclient on your CentOS system.
For example on my openSUSE Leap 15.0 system
where the run-time linker would find libparted
that is needed by /usr/sbin/parted
# ldd /usr/sbin/parted | grep parted
libparted.so.2 => /usr/lib64/libparted.so.2 (0x00007f3c972d6000)
# ldconfig -v 2>/dev/null | egrep ':$|libparted'
/usr/local/lib64:
/usr/local/lib:
/usr/lib64/graphviz:
/lib:
/lib64:
/usr/lib:
/usr/lib64:
libparted.so.2 -> libparted.so.2.0.1
g243:~ # find /usr/lib64 -ls | grep libparted
... lrwxrwxrwx ... /usr/lib64/libparted.so.2 -> libparted.so.2.0.1
... -rwxr-xr-x ... /usr/lib64/libparted.so.2.0.1
What directories are used by ldconfig are set in config files
like /etc/ld.so.conf and /etc/ld.so.conf.d/*.conf
for example on my openSUSE Leap 15.0 system
# cat /etc/ld.so.conf
/usr/local/lib64
/usr/local/lib
include /etc/ld.so.conf.d/*.conf
# /lib64, /lib, /usr/lib64 and /usr/lib gets added
# automatically by ldconfig after parsing this file.
# So, they do not need to be listed.
and because of the include /etc/ld.so.conf.d/*.conf:
# find /etc/ld.so.conf.d/
/etc/ld.so.conf.d/
/etc/ld.so.conf.d/graphviz.conf
# cat /etc/ld.so.conf.d/graphviz.conf
/usr/lib64/graphviz
/usr/lib64/graphviz/sharp
/usr/lib64/graphviz/java
/usr/lib64/graphviz/perl
/usr/lib64/graphviz/php
/usr/lib64/graphviz/ocaml
/usr/lib64/graphviz/python
/usr/lib64/graphviz/lua
/usr/lib64/graphviz/tcl
/usr/lib64/graphviz/guile
/usr/lib64/graphviz/ruby
but because actually there is none of the directories
listed in /etc/ld.so.conf.d/graphviz.conf
calling ldconfig -v reports them on its stderr
# ldconfig -v 1>/dev/null
ldconfig: Can't stat /usr/lib64/graphviz/sharp: No such file or directory
ldconfig: Can't stat /usr/lib64/graphviz/java: No such file or directory
ldconfig: Can't stat /usr/lib64/graphviz/perl: No such file or directory
ldconfig: Can't stat /usr/lib64/graphviz/php: No such file or directory
ldconfig: Can't stat /usr/lib64/graphviz/ocaml: No such file or directory
ldconfig: Can't stat /usr/lib64/graphviz/python: No such file or directory
ldconfig: Can't stat /usr/lib64/graphviz/lua: No such file or directory
ldconfig: Can't stat /usr/lib64/graphviz/tcl: No such file or directory
ldconfig: Can't stat /usr/lib64/graphviz/guile: No such file or directory
ldconfig: Can't stat /usr/lib64/graphviz/ruby: No such file or directory
but that is not a real error.
The first thing what I would like to know is how the config
of the run-time linker for /sbin/dhclient looks like
on your original CentOS system.
I guess there is something "unusual" with that.
The second thing what I would like to know is how the config
of the run-time linker for /sbin/dhclient looks like
inside your ReaR recovery system.
To inspect things inside the ReaR recovery system
set in your /etc/rear/local.conf
KEEP_BUILD_DIR="yes"
then create the recovery system with "rear -D mkrescue"
and finally chroot into your recovery system in
$TMPDIR/rear.XXXXXXXXXXXXXXX/rootfs/
and run commands like the above to find out
how the config of the run-time linker for /sbin/dhclient
looks like inside your recovery system.
DamaniN commented at 2019-11-23 03:15:¶
Re-submitted the old solution as a #2284.
I'll still research a new solution that doesn't use LD_LIBRARY_PATH as
time allows.
jsmeix commented at 2020-01-21 09:31:¶
With
https://github.com/rear/rear/pull/2284
merged via
https://github.com/rear/rear/commit/5e52c99ee8cf22a8d043bc304a74c3504c0e7d62
this issue and possible (dirty) workarounds via LD_LIBRARY_PATH
cf.
https://github.com/rear/rear/pull/2270#issuecomment-551832257
is at least documented as one of the "Known Issues" in
https://github.com/rear/rear/blob/master/doc/user-guide/16-Rubrik-CDM.adoc
jsmeix commented at 2022-01-26 13:03:¶
This issue is likely avoided via
https://github.com/rear/rear/pull/2747
but the actual root cause might be related to
https://github.com/rear/rear/issues/2743#issuecomment-1022133392
on the other hand when the whole /usr/lib64 is included
there should be nothing missing in /usr/lib64
[Export of Github issue for rear/rear.]