#2669 Issue closed: ERROR: gdisk is missing librarys

Labels: support / question, fixed / solved / done

dcz01 opened issue at 2021-08-12 13:52:

ERROR: gdisk is missing librarys

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

  • ReaR version ("/usr/sbin/rear -V"):
    Relax-and-Recover 2.6 / 2020-06-17

  • OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):

NAME="CentOS Linux"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
  • ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
OUTPUT=ISO
OUTPUT_URL=file:///tmp/rear
BACKUP=NETFS
#BACKUP=TSM
BACKUP_PROG=tar
BACKUP_PROG_CRYPT_ENABLED=1
BACKUP_PROG_CRYPT_KEY=XXXXXXXXXXXXXXXXXXXX
BACKUP_PROG_CRYPT_OPTIONS="/usr/bin/openssl aes-256-cbc -salt -k"
BACKUP_PROG_DECRYPT_OPTIONS="/usr/bin/openssl aes-256-cbc -d -k"
BACKUP_URL=nfs://XXXXXXXX/Backups
#BACKUP_URL=cifs://<Server>/<Freigabe>
#BACKUP_OPTIONS="cred=/etc/rear/cifs,sec=ntlmsspi"
BACKUP_OPTIONS="nfsvers=4,nolock"
BACKUP_TYPE=incremental
FULLBACKUPDAY="Sat"
BACKUP_PROG_EXCLUDE=( '/tmp/*' '/dev/shm/*' $VAR_DIR/output/\* '/opt/tivoli/tsm/rear/*' '/mnt/*' '/media/*' '/var/lib/pgsql/*/data/base/*' '/var/lib/pgsql/*/data/global/*' '/var/lib/pgsql/*/data/pg*/*' )
SSH_ROOT_PASSWORD='$6$CFQoHxuu57fA8oWc$RCLWE/ZiSlKFAjNADp6ob.feRYxy/zk1Hch/QK9awCMhtTwPmEdddza/w5WlpnK85pcCVhh/MJ4evHTg73sl//'
BOOTLOADER="GRUB2-EFI"

  • Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR):
    Hardware Server

  • System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
    x86_64

  • Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
    UEFI with GRUB2

  • Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
    Local Disk with RAID

  • Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT" or "lsblk" as makeshift):

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 893,3G  0 disk
├─sda1   8:1    0   200M  0 part /boot/efi
├─sda2   8:2    0   500M  0 part /boot
├─sda3   8:3    0  97,7G  0 part /
├─sda4   8:4    0  31,4G  0 part [SWAP]
├─sda5   8:5    0     1M  0 part
└─sda6   8:6    0 763,5G  0 part /var/lib/pgsql
sr0     11:0    1  1024M  0 rom

NAME        KNAME     PKNAME   TRAN   TYPE FSTYPE   SIZE MOUNTPOINT
/dev/sda    /dev/sda                  disk        893,3G
|-/dev/sda1 /dev/sda1 /dev/sda        part vfat     200M /boot/efi
|-/dev/sda2 /dev/sda2 /dev/sda        part xfs      500M /boot
|-/dev/sda3 /dev/sda3 /dev/sda        part xfs     97,7G /
|-/dev/sda4 /dev/sda4 /dev/sda        part swap    31,4G [SWAP]
|-/dev/sda5 /dev/sda5 /dev/sda        part            1M
`-/dev/sda6 /dev/sda6 /dev/sda        part xfs    763,5G /var/lib/pgsql
/dev/sr0    /dev/sr0           sata   rom          1024M

  • Description of the issue (ideally so that others can reproduce it):
    When trying to manually build a first or new ReaR recovery image it stops with the error that gdisk would have missing librarys.

  • Workaround, if any:
    None.

  • Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
    rear-brp-server1.log

To paste verbatim text like command output or file content,
include it between a leading and a closing line of three backticks like

```
verbatim content
```

gdha commented at 2021-08-13 13:14:

@dcz01 From the log:
/bin/ldd /bin/gdisk output:
linux-vdso.so.1 (0x00007fff9a36b000) << in which path do you find the library? Seems the missing one.
libuuid.so.1 => /lib64/libuuid.so.1 (0x00007fb75e7d2000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fb75e43d000)
libm.so.6 => /lib64/libm.so.6 (0x00007fb75e0bb000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fb75dea3000)
libc.so.6 => /lib64/libc.so.6 (0x00007fb75dade000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb75ec0d000)

dcz01 commented at 2021-08-16 14:53:

@gdha I haven't got a tool under /bin/gdisk...
The tool is under /sbin/gdisk:

/bin/ldd /bin/gdisk
ldd: /bin/gdisk: Datei oder Verzeichnis nicht gefunden

And then:

/bin/ldd /sbin/gdisk
        linux-vdso.so.1 (0x00007ffd05b97000)
        libuuid.so.1 => /lib64/libuuid.so.1 (0x00007fdede5e9000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fdede254000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fdedded2000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fdeddcba000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fdedd8f5000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fdedea24000)

And i can't find the library missing on the system:
find / -name linux-vdso.so.1
Gives no output...

dcz01 commented at 2021-08-16 14:58:

@gdha @jsmeix Does this library exist on any system?
https://man7.org/linux/man-pages/man7/vdso.7.html

gdha commented at 2021-08-27 07:41:

@dcz01 Indeed you are right about linux-vdso.so.1 library. My best advise is to use chroot on a /tmp/rear.xxx/rootfs directory and try to find out if gdisk is working or not. Use the option -d with rear -vd mkrescue

jsmeix commented at 2021-08-31 12:34:

Regarding https://github.com/rear/rear/issues/2669#issuecomment-899575624
/bin/gdisk versus /sbin/gdisk

The ldd test is run inside the ReaR recovery system (via chroot $ROOTFS_DIR ... ldd)
and inside the ReaR recovery system all binaries are actually in /bin/
because the other directories are only symlinks to /bin/

localhost:/tmp/rear.XXXXX/rootfs # ls -ld bin sbin usr/bin usr/sbin
drwxr-xr-x ... bin
lrwxrwxrwx ... sbin -> bin
lrwxrwxrwx ... usr/bin -> ../bin
lrwxrwxrwx ... usr/sbin -> ../bin

@dcz01
see the KEEP_BUILD_DIR description in default.conf that is currently at
https://github.com/rear/rear/blob/master/usr/share/rear/conf/default.conf#L163
how you could inspect the ReaR recovery system contents
to find out what is missing for gdisk inside the ReaR recovery system.

dcz01 commented at 2021-09-06 12:49:

@gdha @jsmeix Well so after my vacations i had the time to finally test all you said.
But for me it looks normally:

[root@build ~]# rear -vd mkrescue
Relax-and-Recover 2.6 / 2020-06-17
Running rear mkrescue (PID 530630)
Using log file: /var/log/rear/rear-build.log
Running workflow mkrescue on the normal/original system
Found EFI system partition /dev/sda1 on /boot/efi type vfat
Using UEFI Boot Loader for Linux (USING_UEFI_BOOTLOADER=1)
Using autodetected kernel '/boot/vmlinuz-4.18.0-305.12.1.el8_4.x86_64' as kernel in the recovery system
Creating disk layout
Overwriting existing disk layout file /var/lib/rear/layout/disklayout.conf
Using specified bootloader 'GRUB2-EFI'
Verifying that the entries in /var/lib/rear/layout/disklayout.conf are correct ...
Creating recovery system root filesystem skeleton layout
Adding biosdevname=0 to KERNEL_CMDLINE
Adding net.ifnames=0 to KERNEL_CMDLINE
Handling network interface 'eth0'
eth0 is a physical device
Handled network interface 'eth0'
Trying to find what to use as UEFI bootloader...
Trying to find a 'well known file' to be used as UEFI bootloader...
Using '/boot/efi/EFI/centos/grubx64.efi' as UEFI bootloader file
Copying logfile /var/log/rear/rear-build.log into initramfs as '/tmp/rear-build-partial-2021-09-06T14:45:34+02:00.log'
Copying files and directories
Copying binaries and libraries
Copying all kernel modules in /lib/modules/4.18.0-305.12.1.el8_4.x86_64 (MODULES contains 'all_modules')
Copying all files in /lib*/firmware/
Skip copying broken symlink '/etc/mtab' target '/proc/542096/mounts' on /proc/ /sys/ /dev/ or /run/
Broken symlink '/usr/lib/modules/4.18.0-305.12.1.el8_4.x86_64/source' in recovery system because 'readlink' cannot determine its link target
Broken symlink '/usr/lib/modules/4.18.0-305.12.1.el8_4.x86_64/build' in recovery system because 'readlink' cannot determine its link target
Testing that the recovery system in /tmp/rear.LHNgAtmVUYIU9Cz/rootfs contains a usable system
There are binaries or libraries in the ReaR recovery system that need additional libraries
/bin/gdisk requires additional libraries (fatal error)

ReaR recovery system in '/tmp/rear.LHNgAtmVUYIU9Cz/rootfs' needs additional libraries, check /var/log/rear/rear-build.log for details
Build area kept for investigation in /tmp/rear.LHNgAtmVUYIU9Cz, remove it when not needed
ERROR: ReaR recovery system in '/tmp/rear.LHNgAtmVUYIU9Cz/rootfs' not usable (required libraries are missing)
Some latest log messages since the last called script 990_verify_rootfs.sh:
  partprobe is /bin/partprobe
  wipefs is /bin/wipefs
  mkfs is /bin/mkfs
  mkfs.vfat is /bin/mkfs.vfat
  mkfs.xfs is /bin/mkfs.xfs
  xfs_admin is /bin/xfs_admin
  mkswap is /bin/mkswap
  ldconfig is /bin/ldconfig
Aborting due to an error, check /var/log/rear/rear-build.log for details
Exiting rear mkrescue (PID 530630) and its descendant processes ...
Running exit tasks
You should also rm -Rf /tmp/rear.LHNgAtmVUYIU9Cz
Beendet
[root@build ~]# chroot /tmp/rear.LHNgAtmVUYIU9Cz/
rootfs/ tmp/
[root@build ~]# chroot /tmp/rear.LHNgAtmVUYIU9Cz/rootfs/
bash-4.4# /bin/ldd /sbin/gdisk
        linux-vdso.so.1 (0x00007fff25d46000)
        libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f63ba846000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f63ba4b1000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f63ba12f000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f63b9f17000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f63b9b52000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f63bac81000)
bash-4.4# ls -ld bin sbin usr/bin usr/sbin
drwxr-xr-x. 2 root root 8192 Sep  6 14:45 bin
lrwxrwxrwx. 1 root root    3 Sep  6 14:45 sbin -> bin
lrwxrwxrwx. 1 root root    6 Sep  6 14:45 usr/bin -> ../bin
lrwxrwxrwx. 1 root root    6 Sep  6 14:45 usr/sbin -> ../bin

What could be this thing?

dcz01 commented at 2021-09-21 08:48:

@jsmeix @gdha
Now i think i found the problem and mybe the solution for you...
I changed the option/value BACKUP=TSM to BACKUP=NETFS and the ISO was created successfully like before.
But with the option/value BACKUP=TSM there is always the same problem on some machines.

Could you now find the problem here?

jsmeix commented at 2021-09-21 14:28:

@dcz01
thank you for your feedback.
It helps a lot to understand where the root cause likely is.

We (at ReaR upstream) know since a long time
that third party tools often do "special" things with their libraries.
Sometimes their "special things" are so very special that it only works
in their very special way in their specificially set up runtime environment
but e.g. not in the ReaR recovery system environment.

Because we (at ReaR upstream) do not have those third party tools
we cannot help you with issues that are "inside" third party tools.
In particular we cannot find out what special things would have to be
set up in the ReaR recovery system environment to make some
special third party tool work within the ReaR recovery system.

All you can do is to contact the vendor or manufacturer of that
third party tool for help and support with their third party tool.
When you paid them they should provide you what you paid for.

When the vendor or manufacturer of that third party tool
opens an issue here at ReaR upstream we would help him
to integrate his third party tool into our ReaR recovery system
or we may find out together that this is impossible in practice
with reasonable effort.

jsmeix commented at 2021-09-21 14:50:

@dcz01
now I see that you had already "a lot of no so fun" with TSM in the past
https://github.com/rear/rear/issues/1533

So it seems TSM uses LD_LIBRARY_PATH
which is known to cause troubles, cf.
https://github.com/rear/rear/issues/1907#issuecomment-434215448
which links to
https://www.hpc.dtu.dk/?page_id=1180
which further links to
http://xahlee.info/UnixResource_dir/_/ldpath.html

@dcz01
could it be that the user 'root' on your original system
(more precisely the exact 'root' that runs "rear mkrescue")
has LD_LIBRARY_PATH set to a non-empty value
and if yes what is LD_LIBRARY_PATH set to?

dcz01 commented at 2021-09-23 06:03:

@jsmeix Thanks for your fast reply.
I've tested the LD_LIBRARY_PATH variable with echo $LD_LIBRARY_PATH but it was empty.
So where could this variable been set outside of ReaR?

jsmeix commented at 2021-09-23 12:53:

Empty LD_LIBRARY_PATH for 'root' who runs rear is good.

I have another idea:
In your initial comment you wrote
Relax-and-Recover 2.6 / 2020-06-17

On Nov 26 2020 I merged
https://github.com/rear/rear/commit/6b804a5cfcc45c1e2dc33110e33113d4b2db613f
where I completely overhauled the ldd test in 990_verify_rootfs.sh via
https://github.com/rear/rear/pull/2523
because of
https://github.com/rear/rear/issues/2508#issuecomment-726711801
which could be the description of the root cause of this issue here.

@dcz01
to test if this issue here is fixed in current ReaR master code
do what is described in the section
"Testing current ReaR upstream GitHub master code" in
https://en.opensuse.org/SDB:Disaster_Recovery
and provide feedback how current ReaR master code
behaves in your particular case here.

dcz01 commented at 2021-09-27 18:19:

@jsmeix You're right.
I tested the actual master code and it worked perfectly for me and all my clients with problems.
Thanks a lot.
I used only this build: https://download.opensuse.org/repositories/Archiving:/Backup:/Rear:/Snapshot/CentOS_8/x86_64/rear-2.6.5-1.el8.x86_64.rpm

jsmeix commented at 2021-09-28 07:48:

@dcz01
thank you for your feedback!
It helps me so much to understand real world examples of special cases
where older code had failed and that it is actually fixed in current code.

[Export of Github issue for rear/rear.]