#1533 Issue closed: ERROR: ReaR recovery system not usable (missing libraries for TSM libraries)

Labels: bug, external tool

dcz01 opened issue at 2017-10-13 07:54:

ERROR: ReaR recovery system not usable (TSM)

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

  • rear version (/usr/sbin/rear -V): Relax-and-Recover 2.2-git.0.3286708.unknown / 2017-10-10
  • OS version (cat /etc/rear/os.conf or lsb_release -a):
OS_VENDOR=RedHatEnterpriseServer
OS_VERSION=6
  • rear configuration files (cat /etc/rear/site.conf or cat /etc/rear/local.conf):
OUTPUT=ISO
OUTPUT_URL=file:///tmp/rear
#BACKUP=NETFS
BACKUP=TSM
#BACKUP_PROG=tar
#BACKUP_PROG_CRYPT_ENABLED=1
#BACKUP_PROG_CRYPT_KEY=<Verschluesselungskennwort>
#BACKUP_PROG_CRYPT_OPTIONS="/usr/bin/openssl aes256 -salt -k"
#BACKUP_PROG_DECRYPT_OPTIONS="/usr/bin/openssl aes256 -d -k"
#BACKUP_URL=nfs://<IP-Adresse oder DNS-Name>/<Freigabepfad>
#BACKUP_URL=cifs://<Server>/<Freigabe>
#BACKUP_OPTIONS="cred=/etc/rear/cifs,sec=ntlmsspi"
#BACKUP_TYPE=incremental
#FULLBACKUPDAY="Sat"
#BACKUP_PROG_EXCLUDE=( '/tmp/*' '/dev/shm/*' $VAR_DIR/output/\* '/opt/tivoli/tsm/rear/*' '/mnt/*' '/media/*' '/var/lib/pgsql/*/data/base/*' '/var/lib/pgsql/*/data/global/*' '/var/lib/pgsql/*/data/pg*/*' )
SSH_ROOT_PASSWORD='$1$HGjk3XUV$lid3Nd3k01Kht1mpMscLw1'
  • Are you using legacy BIOS or UEFI boot?
    BIOS.
  • Brief description of the issue:
    The ReaR command "rear mkrescue" fails because of the need of additional libraries.
    The rescue image wasn't build.
  • Work-around, if any:
    None.

Full detailed log is attached.
Have there been changed anything about the creation of the rescue image in combination with TSM?

rear-brp-server.log

jsmeix commented at 2017-10-13 09:04:

Actually this is great news!
Such issues happen since I added the test for missing libraries in
build/default/980_verify_rootfs.sh
which is intentionally there to ensure the programs and libraries
in the recovery system are consistent, cf.
https://github.com/rear/rear/pull/1514
for its initial pull request.

At first glance this issue here looks as if TSM needs to be
enhanced to get all needed libraries for TSM programs
included in the recovery system (on unneeded TSM programs
excluded from the recovery system).

schabrolles commented at 2017-10-13 09:12:

Ok, I really need to add rhel6 in my test env. I’ll try to find time to do that next week to reproduce the issue.

jsmeix commented at 2017-10-13 09:15:

The most relevant excerpt from
https://github.com/rear/rear/files/1381778/rear-brp-server.log

There are binaries or libraries in the ReaR recovery system that need additional libraries
/opt/tivoli/tsm/client/api/bin64/libTsmViSdk.so requires additional libraries
        libcrypto.so.1.0.1 => not found
/opt/tivoli/tsm/client/api/bin64/libssl.so.1.0.1 requires additional libraries
        libcrypto.so.1.0.1 => not found
/opt/tivoli/tsm/client/ba/bin/libTsmViSdk.so requires additional libraries
        libcrypto.so.1.0.1 => not found
/opt/tivoli/tsm/client/ba/bin/libvmacore.so requires additional libraries
        libexpat.so.0 => not found
/opt/tivoli/tsm/client/ba/bin/libssl.so.1.0.1 requires additional libraries
        libcrypto.so.1.0.1 => not found
/opt/tivoli/tsm/client/ba/bin/lib64/libdiskLibPlugin.so requires additional libraries
        libcrypto.so.1.0.1 => not found
/opt/tivoli/tsm/client/ba/bin/libvixDiskLibVim.so.6.0.0 requires additional libraries
        libgvmomi.so => not found
/opt/tivoli/tsm/client/ba/bin/libvmomi.so requires additional libraries
        libexpat.so.0 => not found
/opt/tivoli/tsm/client/ba/bin/libvim-types.so requires additional libraries
        libvmacore.so => not found
/opt/tivoli/tsm/client/ba/bin/libvixMntapi.so.1.1.0 requires additional libraries
        libvixDiskLib.so.6 => not found
/opt/tivoli/tsm/client/ba/bin/libcurl.so.4 requires additional libraries
        libssl.so.1.0.1 => not found
/opt/tivoli/tsm/client/ba/bin/libssoclient.so requires additional libraries
        libvmacore.so => not found
ERROR: ReaR recovery system in '/tmp/rear.qF5N138Tn9IzFlJ/rootfs' not usable

which boils down to

        libcrypto.so.1.0.1 => not found
        libexpat.so.0 => not found
        libgvmomi.so => not found
        libssl.so.1.0.1 => not found
        libvixDiskLib.so.6 => not found
        libvmacore.so => not found

jsmeix commented at 2017-10-13 09:35:

FYI
what I have on my SLES12 test system

# for l in libcrypto.so libexpat.so libgvmomi.so libssl.so libvixDiskLib.so libvmacore.so ; do find /lib /lib64 /usr/lib /usr/lib64 | grep "$l" ; done
/lib64/libcrypto.so.1.0.0
/usr/lib64/libexpat.so.1
/usr/lib64/libexpat.so.1.6.0
/lib64/libssl.so.1.0.0

and what I get in the recovery system by default

# for l in libcrypto.so libexpat.so libgvmomi.so libssl.so libvixDiskLib.so libvmacore.so ; do find /tmp/rear.0eRYb1dejfVaBS5/rootfs | grep "$l" ; done
/tmp/rear.0eRYb1dejfVaBS5/rootfs/lib64/libcrypto.so.1.0.0
/tmp/rear.0eRYb1dejfVaBS5/rootfs/usr/lib64/libexpat.so.1
/tmp/rear.0eRYb1dejfVaBS5/rootfs/usr/lib64/libexpat.so.1.6.0
/tmp/rear.0eRYb1dejfVaBS5/rootfs/lib64/libssl.so.1.0.0

I.e. for me things "just work" for
libcrypto.so libexpat.so libssl.so

Regarding libgvmomi.so libvixDiskLib.so libvmacore.so:

I do not have libgvmomi.so libvixDiskLib.so libvmacore.so
on my SLES12 test system.

In
https://github.com/rear/rear/files/1381778/rear-brp-server.log

... /bin/ldd /opt/tivoli/tsm/client/ba/bin/libgvmomi.so
... /bin/ldd /opt/tivoli/tsm/client/ba/bin/libvixDiskLib.so.6.0.0
... /bin/ldd /opt/tivoli/tsm/client/ba/bin/libvmacore.so

indicates that libgvmomi.so libvixDiskLib.so libvmacore.so
belong to 'tivoli' i.e. to the Tivoli Storage Manager (TSM).

MatthiasMerk commented at 2017-10-13 13:01:

i can confirm the same issue on SLES12-SP2
rear version: Relax-and-Recover 2.2-git.0.02aefae.unknown / 2017-10-09

my workaround was to only include the dsmc binary and not the whole client directory. but i haven't had a chance to test a restore yet.

changes in /etc/rear/local.conf

-COPY_AS_IS_TSM=( /opt/tivoli/tsm/client /usr/local/ibm/gsk8* )
+COPY_AS_IS_TSM=( /opt/tivoli/tsm/client/ba/bin/dsmc /usr/local/ibm/gsk8* )

and the rear log with /opt/tivoli/tsm/client in COPY_AS_IS_TSM:

2017-10-13 14:39:48.029011652 There are binaries or libraries in the ReaR recovery system that need additional libraries
2017-10-13 14:39:48.033063622 /opt/tivoli/tsm/client/api/bin64/libssl.so.1.0.1 requires additional libraries
2017-10-13 14:39:48.042425193   linux-vdso.so.1 (0x00007ffd12580000)
        libcrypto.so.1.0.1 => not found
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f9022a26000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f9022683000)
        /lib64/ld-linux-x86-64.so.2 (0x0000561fc9899000)
2017-10-13 14:39:48.049771397 /opt/tivoli/tsm/client/api/bin64/libTsmViSdk.so requires additional libraries
2017-10-13 14:39:48.058939291   linux-vdso.so.1 (0x00007fff6d5bc000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007faa0aa69000)
        libcrypto.so.1.0.1 => not found
        libssl.so.1.0.1 => not found
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007faa0a6df000)
        libm.so.6 => /lib64/libm.so.6 (0x00007faa0a3e2000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007faa0a1ca000)
        libc.so.6 => /lib64/libc.so.6 (0x00007faa09e27000)
        /lib64/ld-linux-x86-64.so.2 (0x000055db9d6aa000)
2017-10-13 14:39:48.066069355 /opt/tivoli/tsm/client/ba/bin/libvixMntapi.so.1.1.0 requires additional libraries
2017-10-13 14:39:48.074423545   linux-vdso.so.1 (0x00007ffc99970000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fcf720e3000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fcf71ec6000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fcf71c8a000)
        libz.so.1 => /lib64/libz.so.1 (0x00007fcf71a74000)
        libvixDiskLib.so.6 => not found
        libfuse.so.2 => /lib64/libfuse.so.2 (0x00007fcf71836000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fcf7161f000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fcf7127c000)
        /lib64/ld-linux-x86-64.so.2 (0x000056021b1ba000)
2017-10-13 14:39:48.080726365 /opt/tivoli/tsm/client/ba/bin/libssoclient.so requires additional libraries
2017-10-13 14:39:48.088315726   linux-vdso.so.1 (0x00007ffd2ccfe000)
        libvmacore.so => not found
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007fc9f92a5000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fc9f8fa8000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fc9f8d91000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fc9f8b73000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fc9f87d0000)
        /lib64/ld-linux-x86-64.so.2 (0x000055774fe23000)
2017-10-13 14:39:48.094834344 /opt/tivoli/tsm/client/ba/bin/libvmacore.so requires additional libraries
2017-10-13 14:39:48.106086848   linux-vdso.so.1 (0x00007ffe762e4000)
        libexpat.so.0 => not found
        libz.so.1 => /lib64/libz.so.1 (0x00007fc2b5287000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fc2b5083000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fc2b4e48000)
        librt.so.1 => /lib64/librt.so.1 (0x00007fc2b4c3f000)
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007fc2b48b6000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fc2b45b9000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fc2b43a1000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fc2b4184000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fc2b3de1000)
        /lib64/ld-linux-x86-64.so.2 (0x000055774a787000)
2017-10-13 14:39:48.113702755 /opt/tivoli/tsm/client/ba/bin/libssl.so.1.0.1 requires additional libraries
2017-10-13 14:39:48.120953912   linux-vdso.so.1 (0x00007ffd1e1a2000)
        libcrypto.so.1.0.1 => not found
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f388a7e0000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f388a43d000)
        /lib64/ld-linux-x86-64.so.2 (0x0000558d2b1d3000)
2017-10-13 14:39:48.127221785 /opt/tivoli/tsm/client/ba/bin/libvixDiskLibVim.so.6.0.0 requires additional libraries
2017-10-13 14:39:48.134698991   linux-vdso.so.1 (0x00007ffc98b12000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f92b11da000)
        /lib64/ld-linux-x86-64.so.2 (0x000056140bc28000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f92b0edd000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f92b0cd8000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f92b0abb000)
        libglib-2.0.so.0 => /usr/lib64/libglib-2.0.so.0 (0x00007f92b07ab000)
        libgmodule-2.0.so.0 => /usr/lib64/libgmodule-2.0.so.0 (0x00007f92b05a6000)
        libgobject-2.0.so.0 => /usr/lib64/libgobject-2.0.so.0 (0x00007f92b0355000)
        libgthread-2.0.so.0 => /usr/lib64/libgthread-2.0.so.0 (0x00007f92b0153000)
        libxml2.so.2 => /usr/lib64/libxml2.so.2 (0x00007f92afde7000)
        libgvmomi.so => not found
        libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007f92afb78000)
        libffi.so.4 => /usr/lib64/libffi.so.4 (0x00007f92af96e000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f92af758000)
        liblzma.so.5 => /usr/lib64/liblzma.so.5 (0x00007f92af532000)
2017-10-13 14:39:48.141176994 /opt/tivoli/tsm/client/ba/bin/libcurl.so.4 requires additional libraries
2017-10-13 14:39:48.148808500   linux-vdso.so.1 (0x00007ffde2fba000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f080273d000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f0802539000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f0802322000)
        libssl.so.1.0.1 => not found
        libcrypto.so.1.0.1 => not found
        libc.so.6 => /lib64/libc.so.6 (0x00007f0801f7e000)
        /lib64/ld-linux-x86-64.so.2 (0x0000562cadf82000)
2017-10-13 14:39:48.154471546 /opt/tivoli/tsm/client/ba/bin/libvim-types.so requires additional libraries
2017-10-13 14:39:48.161376799   linux-vdso.so.1 (0x00007fff6f741000)
        libvmacore.so => not found
        libvmomi.so => not found
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f798f91c000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f798f61f000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f798f407000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f798f1ea000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f798ee47000)
        /lib64/ld-linux-x86-64.so.2 (0x000055fb06e77000)
2017-10-13 14:39:48.166908094 /opt/tivoli/tsm/client/ba/bin/libvmomi.so requires additional libraries
2017-10-13 14:39:48.174645608   linux-vdso.so.1 (0x00007ffcc9d3f000)
        libexpat.so.0 => not found
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f4f7d9c0000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f4f7d785000)
        libssoclient.so => not found
        libvmacore.so => not found
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f4f7d3fb000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f4f7d0fe000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f4f7cee6000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f4f7ccc9000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f4f7c926000)
        /lib64/ld-linux-x86-64.so.2 (0x0000555a86fb1000)
2017-10-13 14:39:48.180473857 /opt/tivoli/tsm/client/ba/bin/lib64/libdiskLibPlugin.so requires additional libraries
2017-10-13 14:39:48.188157374   linux-vdso.so.1 (0x00007ffcd71a4000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3bf4b23000)
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f3bf479a000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f3bf4595000)
        libcrypto.so.1.0.1 => not found
        libssl.so.1.0.1 => not found
        libexpat.so.0 => not found
        libz.so.1 => /lib64/libz.so.1 (0x00007f3bf437e000)
        libvmacore.so => not found
        libvmomi.so => not found
        libvim-types.so => not found
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f3bf4166000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f3bf3dc2000)
        /lib64/ld-linux-x86-64.so.2 (0x00005599236bb000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f3bf3ac5000)
2017-10-13 14:39:48.194501401 /opt/tivoli/tsm/client/ba/bin/libTsmViSdk.so requires additional libraries
2017-10-13 14:39:48.202275308   linux-vdso.so.1 (0x00007fff18e60000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8b93401000)
        libcrypto.so.1.0.1 => not found
        libssl.so.1.0.1 => not found
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f8b93077000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f8b92d7a000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f8b92b62000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f8b927bf000)
        /lib64/ld-linux-x86-64.so.2 (0x000056113c33f000)
2017-10-13 14:39:48.207708907 ERROR: ReaR recovery system in '/var/tmp/sysb/rear.m1D2pF6PNVe06VB/rootfs' not usable
==== Stack trace ====

jsmeix commented at 2017-10-13 13:26:

The "log with /opt/tivoli/tsm/client in COPY_AS_IS_TSM" in
https://github.com/rear/rear/issues/1533#issuecomment-336445871
shows that only libraries required by TSM libraries are missing.

I do not have TSM so that I cannot help to find out
where that libraries required by TSM libraries are stored
(perhaps an unusual library directory needs to be included?)
or why they appear as "not found" in the "ldd" output
(could there be something wrong with "ldconfig"?).

MatthiasMerk commented at 2017-10-13 14:53:

after adding

/opt/tivoli/tsm/client/api/bin64
/opt/tivoli/tsm/client/ba/bin

to /etc/ld.so.conf and running ldconfig the image gets created. i'm no linker expert but looks fine to me.

jsmeix commented at 2017-10-13 15:02:

@MatthiasMerk
if I understand your
https://github.com/rear/rear/issues/1533#issuecomment-336476185
correctly you did that in your original system
which even seems to indicate there could be
an issue during TSM installation that is now
(indirectly) detected by ReaR?

In other words:
If you undo your changes in your /etc/ld.so.conf
https://github.com/rear/rear/issues/1533#issuecomment-336476185
and re-run ldconfig and then inspect all libraries and binaries
in /opt/tivoli/tsm/ with 'ldd', are then some "not found"?
If yes, TSM might be not properly installed on your original system.

schabrolles commented at 2017-10-13 15:33:

@MatthiasMerk, I just test again a rear mkrescue with TSM on my SLES12 and I cannot reproduce your issue.

rear mkrescue terminate correctly, and I don't have any message regarding lib missing for TSM in rear.log

My question:

  • Do you have the following package installed on your system (they are dependencies for TSM):
rpm -qa | grep gsk
gskcrypt64-8.0-50.66
gskssl64-8.0-50.66

https://www.ibm.com/support/knowledgecenter/SSEQVQ_8.1.2/client/t_inst_linuxx86client.html

  • I also noticed you removed /etc/adsm from COPY_AS_IS_TSM, any reason ? (I know it should not be related to the issue, but still ask...)

MatthiasMerk commented at 2017-10-13 15:59:

@jsmeix yes, without the ld.so.conf entries ldd shows some libs as "not found". TSM seems to want libopenssl-1.0.1 and we have 1.0.0 installed. The TIVsm-{API,BA} packages from IBM are including 1.0.1 IIRC.
@schabrolles gskcrypt, gskssl64, TIVsm-{API,BA} are installed. i skipped the adsm bit in COPY_AS_IS_TSM on purpose for some reason ...
Which version of libopenssl is on your system?

schabrolles commented at 2017-10-13 16:25:

@MatthiasMerk here are the outputs:

#rpm -qa | grep openssl
openssl-1.0.2j-55.1.ppc64le
libopenssl1_0_0-1.0.2j-55.1.ppc64le

Here is what I got from ldd on dsmc binary (TIVsm-BA-8.1.0-2.ppc64le)

#ldd $(which dsmc)
    linux-vdso64.so.1 (0x00003fffb21b0000)
    libgsk8ssl_64.so => /usr/lib64/libgsk8ssl_64.so (0x00003fffb1f10000)
    libgsk8iccs_64.so => /usr/lib64/libgsk8iccs_64.so (0x00003fffb1e90000)
    libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00003fffb1e40000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00003fffb1e00000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00003fffb1dd0000)
    libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00003fffb1bb0000)
    libgpfs.so => /usr/lib64/libgpfs.so (0x00003fffb1b80000)
    libdmapi.so => /usr/lib64/libdmapi.so (0x00003fffb1b60000)
    librt.so.1 => /lib64/librt.so.1 (0x00003fffb1b30000)
    libm.so.6 => /lib64/libm.so.6 (0x00003fffb1a50000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00003fffb1a10000)
    libc.so.6 => /lib64/libc.so.6 (0x00003fffb1840000)
    libgsk8cms_64.so => /usr/lib64/libgsk8cms_64.so (0x00003fffb14a0000)
    /lib64/ld64.so.2 (0x0000000034cc0000)

MatthiasMerk commented at 2017-10-14 06:28:

@schabrolles i have:

$ rpm -qa *openssl*
libopenssl1_0_0-1.0.2j-60.11.2.x86_64
openssl-1.0.2j-60.11.2.x86_64

TSM stuff (older version- maybe thats the problem)

$ rpm -qa *TIV*
TIVsm-API64-7.1.6-3.x86_64
TIVsm-BA-7.1.6-3.x86_64

$ rpm -ql libopenssl1_0_0-1.0.2j-60.11.2.x86_64
/lib64/engines
/lib64/engines/libgost.so
/lib64/engines/libpadlock.so
/lib64/libcrypto.so.1.0.0
/lib64/libssl.so.1.0.0

ldd ldd $(which dsmc) (TIVsm-BA-7.1.6-3.x86_64)

    linux-vdso.so.1 (0x00007ffe03b7e000)
    libgsk8ssl_64.so => /usr/lib64/libgsk8ssl_64.so (0x00007ff36167a000)
    libgsk8iccs_64.so => /usr/lib64/libgsk8iccs_64.so (0x00007ff36151c000)
    libxmlutil-7.1.6.0.so => /usr/lib64/libxmlutil-7.1.6.0.so (0x00007ff361404000)
    libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007ff3611c9000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff360fac000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007ff360da7000)
    libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007ff360a1e000)
    libgpfs.so => /usr/lib64/libgpfs.so (0x00007ff360911000)
    libdmapi.so => /usr/lib64/libdmapi.so (0x00007ff360809000)
    librt.so.1 => /lib64/librt.so.1 (0x00007ff360601000)
    libm.so.6 => /lib64/libm.so.6 (0x00007ff360304000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ff3600ec000)
    libc.so.6 => /lib64/libc.so.6 (0x00007ff35fd49000)
    libgsk8cms_64.so => /usr/lib64/libgsk8cms_64.so (0x00007ff35f898000)
    libtsmxerces-depdom.so.28 => /usr/lib64/libtsmxerces-depdom.so.28 (0x00007ff35f724000)
    libtsmxerces-c.so.28 => /usr/lib64/libtsmxerces-c.so.28 (0x00007ff35f209000)
    /lib64/ld-linux-x86-64.so.2 (0x00005590e57b2000)

also the changes i made to ld.so.conf break zypper 👯

$ sudo zypper search  libopenssl*
zypper: /opt/tivoli/tsm/client/ba/bin/libstdc++.so.6: version `GLIBCXX_3.4.18' not found (required by zypper)
zypper: /opt/tivoli/tsm/client/ba/bin/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by zypper)
zypper: /opt/tivoli/tsm/client/ba/bin/libstdc++.so.6: version `GLIBCXX_3.4.18' not found (required by /usr/lib64/libzypp.so.1600)
zypper: /opt/tivoli/tsm/client/ba/bin/libstdc++.so.6: version `CXXABI_1.3.5' not found (required by /usr/lib64/libzypp.so.1600)
zypper: /opt/tivoli/tsm/client/ba/bin/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by /usr/lib64/libzypp.so.1600)
1

jsmeix commented at 2017-10-14 09:18:

@MatthiasMerk
regarding your
https://github.com/rear/rear/issues/1533#issuecomment-336476185
and "the changes i made to ld.so.conf break zypper" in
https://github.com/rear/rear/issues/1533#issuecomment-336614197

In general:
Caution when adding "foreign" libraries directories to /etc/ld.so.conf
and running 'ldconfig' with such a changed /etc/ld.so.conf:
This may result that then all programs get linked during
program startup by the dynamic linker with that "foreign" libraries
which may result that this or that program does no longer work
because it only works when the program is run with the "usual" libraries.

jsmeix commented at 2017-10-14 09:58:

@schabrolles
does TSM perhaps use special LD_* environment variables
(e.g. things like LD_LIBRARY_PATH or LD_PRELOAD)
so that TSM programs are run with the TSM-specific libraries?

schabrolles commented at 2017-10-31 12:42:

@MatthiasMerk I've reproduced your issue. The reason why I don't hit it before is because I use COPY_AS_IS_TSM with a reduce set of binaries/libraries to reduce the size of my initrd.

 COPY_AS_IS_TSM=( /etc/adsm/TSM.PWD /opt/tivoli/tsm/client/ba/bin/dsmc /opt/tivoli/tsm/client/ba/bin/tsmbench_inclexcl /opt/tivoli/tsm/client/ba/bin/dsm.sys /opt/tivoli/tsm/client/ba/bin/dsm.opt /opt/tivoli/tsm/client/api/bin64/libgpfs.so /opt/tivoli/tsm/client/api/bin64/libdmapi.so /opt/tivoli/tsm/client/ba/bin/EN_US/dsmclientV3.cat /usr/local/ibm/gsk8* )

@jsmeix you were right. setting LD_LIBRARY_PATH in chroot during ldd test seems to solve the issue. I propose the following (I'm not sure about the best way to implement this).

  1. create a script build/TSM/900_tsm_update_ld_library_path.sh
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/tivoli/tsm/client/ba/bin:/opt/tivoli/tsm/client/ba/bin64:/opt/tivoli/tsm/client/api/bin64:/opt/tivoli/tsm/client/api/bin
  1. Change build/default/980_verify_rootfs.sh to add LD_LIBRARY_PATH
for binary in $( find $ROOTFS_DIR -type f -executable -printf '/%P\n' ) ; do
    chroot $ROOTFS_DIR LD_LIBRARY_PATH=$LD_LIBRARY_PATH /bin/ldd $binary | grep -q 'not found' && broken_binaries="$broken_binaries $binary"
done

schabrolles commented at 2017-10-31 13:54:

@jsmeix forget my previous post,
chroot $ROOTFS_DIR LD_LIBRARY_PATH=$LD_LIBRARY_PATH /bin/ldd $binary is not working

schabrolles commented at 2017-10-31 18:13:

@jsmeix an alternative would be to add the following line into /usr/share/rear/prep/TSM/default/400_prep_tsm.sh

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/tivoli/tsm/client/ba/bin:/opt/tivoli/tsm/client/api/bin64:/opt/tivoli/tsm/client/api/bin:/opt/tivoli/tsm/client/api/bin64/cit/bin

using that line help 980_verify_rootfs.sh to find libraries for TSM with chroot $ROOTFS_DIR /bin/ldd $binary.

schabrolles commented at 2017-11-06 14:00:

@MatthiasMerk @dcz01, #1562 should solve your issue regarding missing TSM libraries. it is now merged into master branch.
Could you please test and confirm ?
Thanks.

MatthiasMerk commented at 2017-11-06 15:06:

Thanks!
I'm going to test as soon as the rpm on OBS/Archiving:Backup:Rear:Snapshot/rear is built.

jsmeix commented at 2017-11-07 08:58:

@MatthiasMerk
in general - as far as I know- building on
https://build.opensuse.org/package/show/Archiving:Backup:Rear:Snapshot/rear
does not happen automatically for each merged
pull request.

To test the current ReaR GitHub master code at any time
basically "git clone" it into a directory and then
configure and run it from within that directory like:

# git clone https://github.com/rear/rear.git

# cd rear

# vi etc/rear/local.conf

# usr/sbin/rear -d -D mkbackup

(note the relative paths "etc/rear/" and "usr/sbin/").

MatthiasMerk commented at 2017-11-08 12:21:

with this etc/rear/local.conf:

COPY_AS_IS_TSM=( /etc/opt/tb017/security/TSM.PWD /opt/tivoli/tsm/client /usr/local/ibm/gsk8* )
TSM_LD_LIBRARY_PATH="/opt/tivoli/tsm/client/ba/bin:/opt/tivoli/tsm/client/api/bin64:/opt/tivoli/tsm/client/api/bin"
COPY_AS_IS_EXCLUDE_TSM=( )
PROGS_TSM=(dsmc)

i get one missing library:

There are binaries or libraries in the ReaR recovery system that need additional libraries
/opt/tivoli/tsm/client/ba/bin/libvixMntapi.so.1.1.0 requires additional libraries
    libvixDiskLib.so.6 => not found

to restore i guess i only need the dsmc binary so i changed COPY_AS_IS_TSM to

COPY_AS_IS_TSM=( /etc/opt/tb017/security/TSM.PWD /opt/tivoli/tsm/client/ba/bin/dsmc /usr/local/ibm/gsk8* )

and then the iso is being built without errors. but that also worked before #1562

schabrolles commented at 2017-11-08 13:23:

@MatthiasMerk, I'm gonna have a look.

You should try this

COPY_AS_IS_TSM=( /etc/adsm /opt/tivoli/tsm/client/ba/bin/dsmc /opt/tivoli/tsm/client/ba/bin/dsm.sys /opt/tivoli/tsm/client/ba/bin/dsm.opt /opt/tivoli/tsm/client/api/bin64/libgpfs.so /opt/tivoli/tsm/client/api/bin64/libdmapi.so /opt/tivoli/tsm/client/ba/bin/EN_US/dsmclientV3.cat /usr/local/ibm/gsk8* )

This is the minimum to have dsmc working properly.
Don't forget to also add any of your configuration files (PASSWD, ou Include_Exclude files).

jsmeix commented at 2017-11-08 13:32:

Only a side note:
I wonder why there is

COPY_AS_IS_TSM=( ... /opt/tivoli/tsm/client/ba/bin/dsmc ... )

when there is also in default.conf

PROGS_TSM=(dsmc)

I would expect that because of PROGS_TSM=(dsmc)
there is no need for /opt/tivoli/tsm/client/ba/bin/dsmc
in COPY_AS_IS_TSM?

schabrolles commented at 2017-11-08 13:56:

@jsmeix you are right. I'm gonna test without /opt/tivoli/tsm/client/ba/bin/dsmc

schabrolles commented at 2017-11-08 14:05:

@MatthiasMerk
could you please share you local.conf file ?

MatthiasMerk commented at 2017-11-08 14:30:

@schabrolles
local.conf:

# Default is to create Relax-and-Recover rescue media as ISO image
# set OUTPUT to change that
# set BACKUP to activate an automated (backup and) restore of your data
# Possible configuration values can be found in /usr/share/rear/conf/default.conf
#
# This file (local.conf) is intended for manual configuration. For configuration
# through packages and other automated means we recommend creating a new
# file named site.conf next to this file and to leave the local.conf as it is. 
# Our packages will never ship with a site.conf.
BACKUP=TSM
OUTPUT=ISO
KERNEL_CMDLINE="console=ttyS0,115200"

export TMPDIR="/var/tmp/sysb"

##
# BACKUP=TSM stuff
##
#
COPY_AS_IS_TSM=( /etc/opt/tb017/security/TSM.PWD /opt/tivoli/tsm/client/ba/bin/dsmc /opt/tivoli/tsm/client/ba/bin/inclexcl.lst /opt/tivoli/tsm/client/ba/bin/dsm.sys /opt/tivoli/tsm/client/ba/bin/dsm.opt /usr/local/ibm/gsk8* )
COPY_AS_IS_EXCLUDE_TSM=( )
PROGS_TSM=(dsmc)
# where to copy the resulting files to and save them with TSM
TSM_RESULT_FILE_PATH="$TMPDIR"
#
# Additional dsmc options for restore. Point-in-time read from user
# input is also added to this array.
TSM_DSMC_RESTORE_OPTIONS=( )
#
# Point-in-time date to use, calculated from user input during recovery.
# User input is read as YYYY-MM-DD, but internally MM/DD/YYYY is used.
# (Used internally)
TSM_RESTORE_PIT_DATE=
#
# Point-in-time time to use, calculated from user input during recovery.
# Format is HH:mm:ss.
# (Used internally)
TSM_RESTORE_PIT_TIME=
#
# Should the result from mkrecover/backup saved via TSM
# You can disable these saving when the result is saved on an different way (ISO_URL....)
# (y/n) default to y
TSM_RESULT_SAVE=y

# TSM archive management class definition
TSM_ARCHIVE_MGMT_CLASS=

# Say "y", "Yes" or "1" to remove the ISO file from local system (in the ISO_DIR location)
# if TSM server confirms the backup was successful (to preserve space on the local system)
TSM_RM_ISOFILE=

MatthiasMerk commented at 2017-11-08 14:44:

looks like my COPY_AS_IS_TSM didn't contain enough.
it works when using your minimal TSM version @schabrolles.
Here it is with inclexl and PWD etc:

COPY_AS_IS_TSM=( /etc/opt/tb017/security/TSM.PWD /opt/tivoli/tsm/client/ba/bin/dsm.sys /opt/tivoli/tsm/client/ba/bin/dsm.opt /opt/tivoli/tsm/client/ba/bin/inclexcl.lst /opt/tivoli/tsm/client/api/bin64/libgpfs.so /opt/tivoli/tsm/client/api/bin64/libdmapi.so /opt/tivoli/tsm/client/ba/bin/EN_US/dsmclientV3.cat /usr/local/ibm/gsk8* )

dcz01 commented at 2017-11-09 07:16:

@schabrolles Sorry for my late reply...
I've tested it today (right now) and there are two librarys missing at this time.
The full log is attached.
rear-brp-server.log

jsmeix commented at 2017-11-20 15:35:

With https://github.com/rear/rear/pull/1585 merged
there is now the new config variable
NON_FATAL_BINARIES_WITH_MISSING_LIBRARY
that provides a generic method how one could avoid
issues like his one, see default.conf how to use it.

schabrolles commented at 2017-11-29 06:43:

@dcz01, I cannot reproduce your issue ad I don't have any libvixMntapi.so. It seems to be related to VMWare.
Could you try ldd /opt/tivoli/tsm/client/ba/bin/libvixMntapi.so on your system to backup please ?

dcz01 commented at 2017-11-29 07:28:

@schabrolles I tried your command and got the following output:

[root@brp-server ~]# ldd /opt/tivoli/tsm/client/ba/bin/libvixMntapi.so
        linux-vdso.so.1 =>  (0x00007ffd92d7f000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fb03c7b8000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb03c59b000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fb03c363000)
        libz.so.1 => /lib64/libz.so.1 (0x00007fb03c14d000)
        libvixDiskLib.so.6 => not found
        libfuse.so.2 => not found
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fb03bf36000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fb03bba2000)
        /lib64/ld-linux-x86-64.so.2 (0x0000003414400000)
        libfreebl3.so => /lib64/libfreebl3.so (0x00007fb03b99e000)

And yes, the test machine runs in an VMware environment.

dcz01 commented at 2017-11-29 07:29:

@schabrolles I tested to install the VMware tools to the machine but the error still exists.

dcz01 commented at 2017-11-29 08:43:

@schabrolles But i get the same error on an physical machine.
So it can't be an VMware related error with these librarys.
Here is the log from the physical machine:
rear-FBD0D99R.log

And the same output like on the VMware machine:

[root@FBD0D99R ~]# ldd /opt/tivoli/tsm/client/ba/bin/libvixMntapi.so
        linux-vdso.so.1 =>  (0x00007ffe4cfae000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fa6c5217000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa6c4ffa000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fa6c4dc2000)
        libz.so.1 => /lib64/libz.so.1 (0x00007fa6c4bac000)
        libvixDiskLib.so.6 => not found
        libfuse.so.2 => not found
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fa6c4995000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fa6c4601000)
        /lib64/ld-linux-x86-64.so.2 (0x0000003d5e800000)
        libfreebl3.so => /lib64/libfreebl3.so (0x00007fa6c43fd000)

dcz01 commented at 2017-11-29 09:09:

@schabrolles Now i found one library to install with yum install -y fuse-libs.
Now only one library is missing.

[root@FBD0D99R ~]# ldd /opt/tivoli/tsm/client/ba/bin/libvixMntapi.so
        linux-vdso.so.1 =>  (0x00007ffdc38fb000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f3398458000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f339823b000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f3398003000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f3397ded000)
        libvixDiskLib.so.6 => not found
        libfuse.so.2 => /lib64/libfuse.so.2 (0x00007f3397bb6000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f33979a0000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f339760c000)
        /lib64/ld-linux-x86-64.so.2 (0x0000003d5e800000)
        libfreebl3.so => /lib64/libfreebl3.so (0x00007f3397408000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f3397200000)

But i searched on the system for that library and found it in the TSM folder:

[root@FBD0D99R ~]# find / -name libvixDiskLib.so.6.0.0
/tmp/rear.MaIKxKAFIZ3zXU8/rootfs/opt/tivoli/tsm/client/ba/bin/libvixDiskLib.so.6.0.0
/tmp/rear.xnItMsUXQ5G4DRq/rootfs/opt/tivoli/tsm/client/ba/bin/libvixDiskLib.so.6.0.0
/opt/tivoli/tsm/client/ba/bin/libvixDiskLib.so.6.0.0

gdha commented at 2017-12-11 08:45:

@jsmeix @schabrolles Shall we move the milestone from 2.3 to 2.4?

dcz01 commented at 2017-12-11 09:24:

@gdha If this bug is moved to ReaR 2.4 all users with an TSM system will have an problem...
This is my opinion.
Or does the new variable from @jsmeix suffice?

schabrolles commented at 2017-12-11 10:21:

@dcz01,
I cannot reproduce your issue as libvixMntapi.so seems to be related to a VMWare environment.
Could you please tell me the version of TSM tyou are using.

As you told me you also have the issue on the system to backup, it looks like to be a problem in you setup. You can have a look here : http://www-01.ibm.com/support/docview.wss?uid=swg21577631

[root@FBD0D99R ~]# ldd /opt/tivoli/tsm/client/ba/bin/libvixMntapi.so
        linux-vdso.so.1 =>  (0x00007ffdc38fb000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f3398458000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f339823b000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f3398003000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f3397ded000)
        libvixDiskLib.so.6 => not found
        libfuse.so.2 => /lib64/libfuse.so.2 (0x00007f3397bb6000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f33979a0000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f339760c000)
        /lib64/ld-linux-x86-64.so.2 (0x0000003d5e800000)
        libfreebl3.so => /lib64/libfreebl3.so (0x00007f3397408000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f3397200000)

Could you please verify that your: /usr/share/rear/conf/default.conf contains the following line:

TSM_LD_LIBRARY_PATH="/opt/tivoli/tsm/client/ba/bin:/opt/tivoli/tsm/client/api/bin64:/opt/tivoli/tsm/client/api/bin:/opt/tivoli/tsm/client/api/bin64/cit/bin"

dcz01 commented at 2017-12-11 11:03:

@schabrolles No, the libvixMntapi.so library is part of the TSM client installation and you can find it in the installation directory /opt/tivoli/tsm/client/ba/bin.
My TSM client version is 7.1.3.0.

On my system and the client systems i don't use an 'Backup VM' command or something else.
We're just normal backuping the physical system.

And yes my variable TSM_LD_LIBRARY_PATH in the /usr/share/rear/conf/default.conf file is default, so like yours.

jsmeix commented at 2017-12-11 11:09:

@dcz01
what is the output of your 'ld' config files, i.e. what results

# for f in $( find /etc/ld.so.conf* -type f ) ; do echo Content of $f ; cat $f ; echo ====== ; done

on your system?

jsmeix commented at 2017-12-11 11:11:

@dcz01
I missed your
https://github.com/rear/rear/issues/1533#issuecomment-350692299
so that your 'ld' config files are probably not needed
when the libvixMntapi.so library is part of TSM.

jsmeix commented at 2017-12-11 11:28:

According to
https://github.com/rear/rear/issues/1534#issuecomment-347825583
it seems this issue here could be at least avoided via

NON_FATAL_BINARIES_WITH_MISSING_LIBRARY='/opt/tivoli/tsm/client/ba/bin/libvixMntapi.so.1.1.0'

dcz01 commented at 2017-12-11 11:35:

@jsmeix Yes, that is what i tested and meant in my comment https://github.com/rear/rear/issues/1534#issuecomment-347825583.
But there must be another option to solve this or not?

jsmeix commented at 2017-12-11 12:36:

I do not have TSM (or any other third-party backup tool)
so that I cannot help with actual TSM issues.

I noticed that third-party backup tools sometimes
have unexpected ways how they use their libraries
so that I implemented NON_FATAL_BINARIES_WITH_MISSING_LIBRARY
as a generic way out so that the user can specify
when something is actually "false alarm".

I cannot decide whether or not the failing 'ldd' test for
/opt/tivoli/tsm/client/ba/bin/libvixMntapi.so.1.1.0
is "false alarm" or if perhaps there is a real issue
with the TSM files in the recovery system that
needs to be fixed.

dcz01 commented at 2017-12-11 13:37:

@jsmeix @schabrolles Well i can't find another solution to fix the last missing library of the TSM client so i will use now the new variable NON_FATAL_BINARIES_WITH_MISSING_LIBRARY and we can close this issue.

[Export of Github issue for rear/rear.]