#2144 Issue closed: Additional libraries needed for NBU¶
Labels: support / question, external tool, no-issue-activity
KKeXX opened issue at 2019-05-14 12:54:¶
Additional libraries needed for NBU¶
-
ReaR version:
Relax-and-Recover 2.5 / 2019-05-10 -
OS version:
cat /etc/rear/os.conf
OS_VENDOR=RedHatEnterpriseServer
OS_VERSION=7
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.6 (Maipo)
- ReaR configuration files:
cat /etc/rear/site.conf
export TMPDIR=/var/tmp
export BACKUP=NBU
export OUTPUT=ISO
export OUTPUT_URL="file:///root/rear"
export TYMESYNC=NTP
export USE_CFG2HTML=Y
-
Hardware:
HP ProLiant DL380 G7 -
System architecture:
x86_64 -
Firmware:
Legacy BIOS P67 05/21/2018 and grub 2.02-0.76 -
Storage:
local disk with lvm -
Description of the issue:
Yesterday I upgraded from Relax-and-Recover 2.4 / Git (from rhel-7-server-rpms repo) to Relax-and-Recover 2.5 / 2019-05-10 (from OpenSUSE repo -> http://download.opensuse.org/repositories/Archiving:/Backup:/Rear/RHEL_7/).
When I try with Version 2.5, this Error occures:
/usr/sbin/rear mkrescue
There are binaries or libraries in the ReaR recovery system that need additional libraries
/usr/openv/pdde/pdshared/lib/libstspipd.so requires additional libraries
libpdvfs.so => not found
libmetab.so => not found
libmtstrmco.so => not found
libmtstrmcl.so => not found
libpd.so => not found
libcr++.so => not found
libcr.so => not found
libfc.so => not found
libdct.so => not found
libboost_system.so.1.58.0 => not found
libprotobuf.so.9 => not found
libssl.so.1.0.0 => not found
libcrypto.so.1.0.0 => not found
libisal.so.2 => not found
libcilkrts.so.5 => not found
These are the dynamic libraries for
/usr/openv/pdde/pdshared/lib/libstspipd.so
ldd /usr/openv/pdde/pdshared/lib/libstspipd.so
linux-vdso.so.1 => (0x00007fffd7b3f000)
libpdvfs.so => /opt/pdshared/lib/libpdvfs.so (0x00007f7d2c531000)
libmetab.so => /opt/pdshared/lib/libmetab.so (0x00007f7d2c2e9000)
libmtstrmco.so => /opt/pdshared/lib/libmtstrmco.so (0x00007f7d2c0bc000)
libmtstrmcl.so => /opt/pdshared/lib/libmtstrmcl.so (0x00007f7d2be39000)
libpd.so => /opt/pdshared/lib/libpd.so (0x00007f7d2bbe7000)
libcr++.so => /opt/pdshared/lib/libcr++.so (0x00007f7d2b9be000)
libcr.so => /opt/pdshared/lib/libcr.so (0x00007f7d2b69b000)
libfc.so => /opt/pdshared/lib/libfc.so (0x00007f7d2b48e000)
libdct.so => /opt/pdshared/lib/libdct.so (0x00007f7d2b197000)
libboost_system.so.1.58.0 => /opt/pdag/lib/libboost_system.so.1.58.0 (0x00007f7d2af94000)
libprotobuf.so.9 => /opt/pdag/lib/libprotobuf.so.9 (0x00007f7d2aae3000)
libxml2.so.2 => /opt/pdag/lib/libxml2.so.2 (0x00007f7d2a736000)
libcurl.so.4 => /opt/pdag/lib/libcurl.so.4 (0x00007f7d2a4e1000)
libssl.so.1.0.0 => /opt/pdag/lib/libssl.so.1.0.0 (0x00007f7d2a277000)
libcrypto.so.1.0.0 => /opt/pdag/lib/libcrypto.so.1.0.0 (0x00007f7d29e65000)
libz.so.1 => /opt/pdag/lib/libz.so.1 (0x00007f7d29c50000)
libisal.so.2 => /opt/pdshared/lib/libisal.so.2 (0x00007f7d29991000)
libcilkrts.so.5 => /opt/pdshared/lib/libcilkrts.so.5 (0x00007f7d29750000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f7d29534000)
libm.so.6 => /lib64/libm.so.6 (0x00007f7d29232000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f7d28f2b000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f7d28d15000)
libc.so.6 => /lib64/libc.so.6 (0x00007f7d28948000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f7d28744000)
librt.so.1 => /lib64/librt.so.1 (0x00007f7d2853c000)
/lib64/ld-linux-x86-64.so.2 (0x00007f7d2cf2a000)
All the libraries exist on the system. Adding /opt/pdshared to the
NBU_LD_LIBRARY_PATH Variable does not help.
Version 2.4 worked with the fix from #2105
Hope someone can help out with that issue?
Best regards,
Chris...
jsmeix commented at 2019-05-14 13:43:¶
I do not use NBU (Symantec/Veritas NetBackup)
because I do not have any kind of proprietary software
like third-party backup tools.
Therefore I can neither reproduce such issues
nor can I verify what could actually fix it.
In general to get missing things into the ReaR recovery system
you need to add what is missing to the generic variables like
COPY_AS_IS, REQUIRED_PROGS, and LIBS
e.g. in this particular case via something like
COPY_AS_IS+=( /opt/pdshared /opt/pdag )
For NBU there is the specific variable COPY_AS_IS_NBU but in the
end
it should not matter if COPY_AS_IS or COPY_AS_IS_NBU is used
because the only usage of COPY_AS_IS_NBU is in
usr/share/rear/prep/NBU/default/400_prep_nbu.sh
COPY_AS_IS=( "${COPY_AS_IS[@]}" "${COPY_AS_IS_NBU[@]}" )
KKeXX commented at 2019-05-14 13:49:¶
Thanks for the info. I'll try.
But I'm wondering, why works 2.4 (with the described fix) and 2.5 does
not? Just downgraded rear on this machine and made an rescue image
without any problems!
jsmeix commented at 2019-05-14 13:51:¶
@rmetrich
I dared to assign this issue to you because
you had worked on other NBU issues
like
https://github.com/rear/rear/issues/2105
and
https://github.com/rear/rear/issues/1974
Perhaps you can have a look what goes non here.
rmetrich commented at 2019-05-14 15:32:¶
Try adding /usr/openv/pdde/pdshared/lib to $NBU_LD_LIBRARY_PATH:
NBU_LD_LIBRARY_PATH="$NBU_LD_LIBRARY_PATH:/usr/openv/pdde/pdshared/lib"
KKeXX commented at 2019-05-16 08:10:¶
After a lot of testing I got it working.
I add the directories /usr/openv/pdde/pdopensource/lib and
/usr/openv/pdde/pdshared/lib to the COPY_AS_IS_NBU variable:
COPY_AS_IS_NBU=( /usr/openv/bin/vnetd /usr/openv/bin/vopied /usr/openv/lib /usr/openv/netbackup /usr/openv/var/auth/[mn]*.txt /usr/openv/pdde/pdopensource/lib /usr/openv/pdde/pdshared/lib )
AND added /usr/openv/pdde/pdshared/lib and
/usr/openv/pdde/pdshared/lib to NBU_LD_LIBRARY_PATH variable.
NBU_LD_LIBRARY_PATH="/usr/openv/lib:/usr/openv/netbackup/sec/at/lib/:/usr/openv/pdde/pdshared/lib/:/usr/openv/pdde/pdopensource/lib/"
But, why is this change needed? With verison 2.4 it works without those two directories.
jsmeix commented at 2019-05-16 09:18:¶
On the one hand
the 'ldd' test that runs inside the ReaR recovery system
in usr/share/rear/build/default/990_verify_rootfs.sh as
chroot $ROOTFS_DIR /bin/bash --login -c "cd $( dirname $binary ) && ldd $binary" ...
finds that
/usr/openv/pdde/pdshared/lib/libstspipd.so requires additional libraries
cf. the initial description https://github.com/rear/rear/issues/2144#issue-443898795
On the other hand the RequiredSharedObjects() function
in usr/share/rear/lib/linux-functions.sh should find (also
transitively)
all additional libraries by calling 'ldd' inside the original system.
The RequiredSharedObjects() function is called
in usr/share/rear/build/GNU/Linux/390_copy_binaries_libraries.sh
for all binaries and all libraries in the LIBS array that get
included
in the ReaR recovery system
local all_libs=( "${LIBS[@]}" $( RequiredSharedObjects "${all_binaries[@]}" "${LIBS[@]}" ) )
When for /usr/openv/pdde/pdshared/lib/libstspipd.so dependant
libraries
are missing in the ReaR recovery system it means something with this
automatism does not work as normally intended in this particular case.
Perhaps /usr/openv/pdde/pdshared/lib/libstspipd.so is not in the
LIBS array?
Perhaps the RequiredSharedObjects() function somehow failed to
determine dependant libraries for
/usr/openv/pdde/pdshared/lib/libstspipd.so?
Perhaps something else?
@KKeXX
please attach a complete debug log file when you run
rear -D mkrescue
on your system without the directories
/usr/openv/pdde/pdopensource/lib
and /usr/openv/pdde/pdshared/lib added to the COPY_AS_IS_NBU
variable
so that we have a chance to see what in the above described automatism
may not work as normally intended.
FYI:
Third-party backup tools are meanwhile well known to do any kind
of special libraries stuff where our generic automatisms do not work.
Fortunately since the 'ldd' verification step that runs inside the
ReaR recovery system via build/default/990_verify_rootfs.sh
such issues are no longer silently ignored during rear mkrescue.
Before it could have happened that later - when it is too late -
during
'rear recover' you may learn it the hard way that this or that library
that could be actually needed to restore the backup is missing
in the ReaR recovery system.
jsmeix commented at 2019-05-16 09:25:¶
@KKeXX
you may also compare what you get in the ReaR recovery system
with ReaR version 2.4 versus with ReaR version 2.5.
Use KEEP_BUILD_DIR="yes" cf. usr/share/rear/conf/default.conf
and compare the two $TMPDIR/rear.XXXXXXXXXXXXXXX/rootfs/
that you get with ReaR version 2.4 versus with ReaR version 2.5.
KKeXX commented at 2019-05-17 08:40:¶
Here is a debug log from ReaR 2.5 without modifications:
rear-hostname.log
I compared the working version 2.5 with the working version 2.4. The
directories /opt/pdag/, /opt/pdshared/ and /usr/openv/pdde/ are
not included in the iso from 2.4. So there is no
/usr/openv/pdde/pdshared/lib/libstspipd.so which causes the problems.
pdde is for mediaserver deduplication. I guess, this is not needed for the iso to restore a Server.
We'v version 2.4 in production, without any issues. Startet a restore a few times with that iso!
jsmeix commented at 2019-05-17 09:39:¶
@KKeXX
thank you for your prompt and explanatory reply.
From my (non NBU user's) point of view it looks now
as if you could alternatively add the unwanted stuff
to the COPY_AS_IS_EXCLUDE_NBU array.
Offhandedly I don't know why with ReaR 2.4 the directories
/opt/pdag/, /opt/pdshared/ and /usr/openv/pdde/
are not included in the ReaR recovery system
while in contrast with ReaR 2.5 they are.
jsmeix commented at 2019-05-17 10:08:¶
@KKeXX
your debug log from ReaR 2.5 without modifications
https://github.com/rear/rear/files/3190620/rear-hostname.log
looks like things are somehow terribly broken with your ReaR
bcause it starts with
2019-05-16 14:06:56.882737847 Relax-and-Recover 2.5 / 2019-05-10
2019-05-16 14:06:56.883978722 Running rear dump (PID 12897)
2019-05-16 14:06:56.885127046 Command line options: /usr/sbin/rear dump
and it runs the dump workflow
but then it runs those additional scripts
# grep '^+ source ' rear-hostname.log
+ source /usr/share/rear/rescue/GNU/Linux/960_collect_MC_serviceguard_infos.sh
+ source /usr/share/rear/rescue/GNU/Linux/990_sysreqs.sh
+ source /usr/share/rear/build/GNU/Linux/005_create_symlinks.sh
+ source /usr/share/rear/build/GNU/Linux/090_create_lib_directories_and_symlinks.sh
+ source /usr/share/rear/build/GNU/Linux/100_copy_as_is.sh
+ source /usr/share/rear/build/GNU/Linux/110_touch_empty_files.sh
+ source /usr/share/rear/build/GNU/Linux/130_create_dotfiles.sh
+ source /usr/share/rear/build/GNU/Linux/150_adjust_permissions.sh
+ source /usr/share/rear/build/GNU/Linux/390_copy_binaries_libraries.sh
+ source /usr/share/rear/build/GNU/Linux/400_copy_modules.sh
+ source /usr/share/rear/build/GNU/Linux/420_copy_firmware_files.sh
+ source /usr/share/rear/build/GNU/Linux/450_symlink_mingetty.sh
+ source /usr/share/rear/build/default/500_ssh_setup.sh
+ source /usr/share/rear/build/default/501_check_ssh_keys.sh
+ source /usr/share/rear/build/default/502_include_mdadm_conf.sh
+ source /usr/share/rear/build/GNU/Linux/600_verify_and_adjust_udev.sh
+ source /usr/share/rear/build/GNU/Linux/610_verify_and_adjust_udev_systemd.sh
+ source /usr/share/rear/build/GNU/Linux/620_verify_os_release_file.sh
+ source /usr/share/rear/build/GNU/Linux/630_simplify_systemd_reboot_halt_poweroff_shutdown.sh
+ source /usr/share/rear/build/GNU/Linux/630_verify_resolv_conf_file.sh
+ source /usr/share/rear/build/GNU/Linux/640_verify_lvm_conf.sh
+ source /usr/share/rear/build/default/950_check_missing_programs.sh
+ source /usr/share/rear/build/default/960_remove_encryption_keys.sh
+ source /usr/share/rear/build/default/970_add_rear_release.sh
+ source /usr/share/rear/build/default/975_update_os_conf.sh
+ source /usr/share/rear/build/default/985_fix_broken_links.sh
+ source /usr/share/rear/build/default/990_verify_rootfs.sh
+ source /usr/share/rear/build/default/995_md5sums_rootfs.sh
+ source /usr/share/rear/pack/GNU/Linux/900_create_initramfs.sh
+ source /usr/share/rear/output/default/010_set_umask.sh
+ source /usr/share/rear/output/default/100_mount_output_path.sh
+ source /usr/share/rear/output/default/150_save_copy_of_prefix_dir.sh
+ source /usr/share/rear/output/default/200_make_boot_dir.sh
+ source /usr/share/rear/output/default/200_make_prefix_dir.sh
+ source /usr/share/rear/output/default/250_create_lock.sh
+ source /usr/share/rear/output/ISO/Linux-i386/250_populate_efibootimg.sh
+ source /usr/share/rear/output/ISO/Linux-i386/260_EFISTUB_populate.sh
+ source /usr/share/rear/output/ISO/Linux-i386/300_create_isolinux.sh
+ source /usr/share/rear/output/default/400_copy_disk_struct_files.sh
+ source /usr/share/rear/output/ISO/Linux-i386/700_create_efibootimg.sh
+ source /usr/share/rear/output/ISO/Linux-i386/800_create_isofs.sh
+ source /usr/share/rear/output/ISO/Linux-i386/810_prepare_multiple_iso.sh
+ source /usr/share/rear/output/ISO/Linux-i386/820_create_iso_image.sh
+ source /usr/share/rear/output/ISO/Linux-i386/830_create_iso_image_EFISTUB.sh
+ source /usr/share/rear/output/ISO/Linux-i386/850_check_for_errors.sh
+ source /usr/share/rear/output/default/940_grub2_rescue.sh
+ source /usr/share/rear/output/default/940_grub_rescue.sh
+ source /usr/share/rear/output/default/950_copy_result_files.sh
But those additionally run scripts differ very much
from what scripts get sourced when I run rear -D mkrescue
on my openSUSE Leap 15.0 system.
So currently it looks to me as if your ReaR 2.5 installation
is somehow terribly broken.
KKeXX commented at 2019-05-20 08:00:¶
I've removed rear completely and removed all leftovers by hand. Then I reinstalled rear and made a new mkrescue. With the same results:
# /usr/sbin/rear -D mkrescue
# head /var/log/rear/rear-hostname.log
2019-05-20 08:47:18.738812817 Relax-and-Recover 2.5 / 2019-05-10
2019-05-20 08:47:18.740208702 Running rear dump (PID 25323)
2019-05-20 08:47:18.741515178 Command line options: /usr/sbin/rear dump
# grep '^+ source ' /var/log/rear/rear-hostname.log
+ source /usr/share/rear/rescue/GNU/Linux/960_collect_MC_serviceguard_infos.sh
+ source /usr/share/rear/rescue/GNU/Linux/990_sysreqs.sh
+ source /usr/share/rear/build/GNU/Linux/005_create_symlinks.sh
+ source /usr/share/rear/build/GNU/Linux/090_create_lib_directories_and_symlinks.sh
+ source /usr/share/rear/build/GNU/Linux/100_copy_as_is.sh
+ source /usr/share/rear/build/GNU/Linux/110_touch_empty_files.sh
+ source /usr/share/rear/build/GNU/Linux/130_create_dotfiles.sh
+ source /usr/share/rear/build/GNU/Linux/150_adjust_permissions.sh
+ source /usr/share/rear/build/GNU/Linux/390_copy_binaries_libraries.sh
+ source /usr/share/rear/build/GNU/Linux/400_copy_modules.sh
+ source /usr/share/rear/build/GNU/Linux/420_copy_firmware_files.sh
+ source /usr/share/rear/build/GNU/Linux/450_symlink_mingetty.sh
+ source /usr/share/rear/build/default/500_ssh_setup.sh
+ source /usr/share/rear/build/default/501_check_ssh_keys.sh
+ source /usr/share/rear/build/default/502_include_mdadm_conf.sh
+ source /usr/share/rear/build/GNU/Linux/600_verify_and_adjust_udev.sh
+ source /usr/share/rear/build/GNU/Linux/610_verify_and_adjust_udev_systemd.sh
+ source /usr/share/rear/build/GNU/Linux/620_verify_os_release_file.sh
+ source /usr/share/rear/build/GNU/Linux/630_simplify_systemd_reboot_halt_poweroff_shutdown.sh
+ source /usr/share/rear/build/GNU/Linux/630_verify_resolv_conf_file.sh
+ source /usr/share/rear/build/GNU/Linux/640_verify_lvm_conf.sh
+ source /usr/share/rear/build/default/950_check_missing_programs.sh
+ source /usr/share/rear/build/default/960_remove_encryption_keys.sh
+ source /usr/share/rear/build/default/970_add_rear_release.sh
+ source /usr/share/rear/build/default/975_update_os_conf.sh
+ source /usr/share/rear/build/default/985_fix_broken_links.sh
+ source /usr/share/rear/build/default/990_verify_rootfs.sh
+ source /usr/share/rear/build/default/995_md5sums_rootfs.sh
+ source /usr/share/rear/pack/GNU/Linux/900_create_initramfs.sh
+ source /usr/share/rear/output/default/010_set_umask.sh
+ source /usr/share/rear/output/default/100_mount_output_path.sh
+ source /usr/share/rear/output/default/150_save_copy_of_prefix_dir.sh
+ source /usr/share/rear/output/default/200_make_boot_dir.sh
+ source /usr/share/rear/output/default/200_make_prefix_dir.sh
+ source /usr/share/rear/output/default/250_create_lock.sh
+ source /usr/share/rear/output/ISO/Linux-i386/250_populate_efibootimg.sh
+ source /usr/share/rear/output/ISO/Linux-i386/260_EFISTUB_populate.sh
+ source /usr/share/rear/output/ISO/Linux-i386/300_create_isolinux.sh
+ source /usr/share/rear/output/default/400_copy_disk_struct_files.sh
+ source /usr/share/rear/output/ISO/Linux-i386/700_create_efibootimg.sh
+ source /usr/share/rear/output/ISO/Linux-i386/800_create_isofs.sh
+ source /usr/share/rear/output/ISO/Linux-i386/810_prepare_multiple_iso.sh
+ source /usr/share/rear/output/ISO/Linux-i386/820_create_iso_image.sh
+ source /usr/share/rear/output/ISO/Linux-i386/830_create_iso_image_EFISTUB.sh
+ source /usr/share/rear/output/ISO/Linux-i386/850_check_for_errors.sh
+ source /usr/share/rear/output/default/940_grub2_rescue.sh
+ source /usr/share/rear/output/default/940_grub_rescue.sh
+ source /usr/share/rear/output/default/950_copy_result_files.sh
+ source /usr/share/rear/output/default/950_email_result_files.sh
+ source /usr/share/rear/output/default/970_remove_lock.sh
+ source /usr/share/rear/output/default/980_umount_output_dir.sh
here is the new dump log, with unmodified rear 2.5:
rear-hostname.log
Adding /usr/openv/pdde/ to COPY_AS_IS_EXCLUDE_NBU does not fix the
problem. The directory does get included as well. Can't even find the
Variable in the logfile ...
rmetrich commented at 2019-05-20 09:25:¶
Hi @KKeXX,
Could you send output of
bash -x /usr/sbin/rear -D mkrescue 2>/tmp/rear.stderr, your rear is
still showing workflow "dump" which is not expected.
KKeXX commented at 2019-05-20 12:03:¶
# bash -x /usr/sbin/rear -D mkrescue 2>/tmp/rear.stderr
Relax-and-Recover 2.5 / 2019-05-10
Running rear mkrescue (PID 15932)
Using log file: /var/log/rear/rear-hostname.log
Using autodetected kernel '/boot/vmlinuz-3.10.0-957.12.1.el7.x86_64' as kernel in the recovery system
Creating disk layout
Using guessed bootloader 'GRUB' (found in first bytes on /dev/sda)
Verifying that the entries in /var/lib/rear/layout/disklayout.conf are correct ...
Creating root filesystem layout
Handling network interface 'bond0'
bond0 is a bond
bond0 has lower interface enp3s0f0
enp3s0f0 is a physical device
bond0 has lower interface enp3s0f1
enp3s0f1 is a physical device
Handled network interface 'bond0'
Copying logfile /var/log/rear/rear-hostname.log into initramfs as '/tmp/rear-hostname-partial-2019-05-20T13:24:11+0200.log'
Copying files and directories
Copying binaries and libraries
Copying all kernel modules in /lib/modules/3.10.0-957.12.1.el7.x86_64 (MODULES contains 'all_modules')
Copying all files in /lib*/firmware/
Skip copying broken symlink '/etc/mtab' target '/proc/27552/mounts' on /proc/ /sys/ /dev/ or /run/
Testing that the recovery system in /var/tmp/rear.44bRY0Toc1UtVkD/rootfs contains a usable system
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (2783426150 bytes) in 229 seconds
Making ISO image
Wrote ISO image: /var/lib/rear/output/rear-hostname.iso (2.7G)
Copying resulting files to file location
Saving /var/log/rear/rear-hostname.log as rear-hostname.log to file location
Copying result files '/var/lib/rear/recovery/cfg2html/hostname.html /var/lib/rear/output/rear-hostname.iso /var/tmp/rear.44bRY0Toc1UtVkD/tmp/VERSION /var/tmp/rear.44bRY0Toc1UtVkD/tmp/README /var/tmp/rear.44bRY0Toc1UtVkD/tmp/rear-hostname.log' to /root/rear/hostname at file location
Exiting rear mkrescue (PID 15932) and its descendant processes ...
Running exit tasks
You should also rm -Rf /var/tmp/rear.44bRY0Toc1UtVkD
And here is the output log:
rear.stderr.log
rmetrich commented at 2019-05-20 12:15:¶
Looks like it worked, we can see "Running rear mkrescue (PID 15932)" message and rear exited successfully. Whereas your last log showed "rear dump"
KKeXX commented at 2019-05-20 12:55:¶
But I haven't chaned anything. Here is the logfile, there is the dump
cmd in it:
rear-hostname.log
jsmeix commented at 2019-05-21 06:51:¶
What looks strange in
https://github.com/rear/rear/issues/2144#issuecomment-493957860
is the literal hostname instead of a real name of the host as in
Using log file: /var/log/rear/rear-hostname.log
...
Wrote ISO image: /var/lib/rear/output/rear-hostname.iso (2.7G)
and the huge ISO image size.
For me with OUTPUT=ISO and BACKUP=NETFS things look like
g243:~/rear.github.master # usr/sbin/rear -D mkrescue
Relax-and-Recover 2.5 / Git
Running rear mkrescue (PID 24765)
Using log file: /root/rear.github.master/var/log/rear/rear-g243.log
Found EFI system partition /dev/sda1 on /boot/efi type vfat
Using UEFI Boot Loader for Linux (USING_UEFI_BOOTLOADER=1)
Using autodetected kernel '/boot/vmlinuz-4.12.14-lp150.12.61-default' as kernel in the recovery system
Creating disk layout
Using sysconfig bootloader 'grub2-efi'
Verifying that the entries in /root/rear.github.master/var/lib/rear/layout/disklayout.conf are correct ...
Creating root filesystem layout
Handling network interface 'eth0'
eth0 is a physical device
Handled network interface 'eth0'
Skipping 'virbr0': not bound to any physical interface.
Trying to find what to use as UEFI bootloader...
Trying to find a 'well known file' to be used as UEFI bootloader...
Using '/boot/efi/EFI/opensuse/grubx64.efi' as UEFI bootloader file
Copying logfile /root/rear.github.master/var/log/rear/rear-g243.log into initramfs as '/tmp/rear-g243-partial-2019-05-17T11:55:59+02:00.log'
Copying files and directories
Copying binaries and libraries
Copying all kernel modules in /lib/modules/4.12.14-lp150.12.61-default (MODULES contains 'all_modules')
Omit copying files in /lib*/firmware/ (FIRMWARE_FILES='no')
Symlink '/mystuffdir/myotherstuffsubdirsymlink' -> '/myotherstuff/subdir' refers to a non-existing directory on the recovery system.
It will not be copied by default. You can include '/myotherstuff/subdir' via the 'COPY_AS_IS' configuration variable.
Skip copying broken symlink '/etc/mtab' target '/proc/1065/mounts' on /proc/ /sys/ /dev/ or /run/
Testing that the recovery system in /tmp/rear.GSm1qjXb9ZRPQnh/rootfs contains a usable system
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (131710282 bytes) in 17 seconds
Making ISO image
Wrote ISO image: /root/rear.github.master/var/lib/rear/output/rear-g243.iso (172M)
Copying resulting files to nfs location
Saving /root/rear.github.master/var/log/rear/rear-g243.log as rear-g243.log to nfs location
Copying result files '/root/rear.github.master/var/lib/rear/output/rear-g243.iso /tmp/rear.GSm1qjXb9ZRPQnh/tmp/VERSION /tmp/rear.GSm1qjXb9ZRPQnh/tmp/README /tmp/rear.GSm1qjXb9ZRPQnh/tmp/rear-g243.log' to /tmp/rear.GSm1qjXb9ZRPQnh/outputfs/g243 at nfs location
Exiting rear mkrescue (PID 24765) and its descendant processes ...
Running exit tasks
You should also rm -Rf /tmp/rear.GSm1qjXb9ZRPQnh
It still looks as if something goes terribly wrong for @KKeXX
jsmeix commented at 2019-05-21 06:57:¶
@KKeXX
I think you should run our current ReaR GitHub master code
from within a separated directory as a test to compare and find out
whether or not something is wrong with your current ReaR installation.
Basically "git clone" our current ReaR upstream GitHub master code
into a separated directory and then configure and run ReaR
from within that directory like:
# git clone https://github.com/rear/rear.git
# mv rear rear.github.master
# cd rear.github.master
# vi etc/rear/local.conf
# usr/sbin/rear -D mkbackup
Note the relative paths "etc/rear/" and "usr/sbin/".
And of course provide us your complete etc/rear/local.conf - cf.
https://raw.githubusercontent.com/rear/rear/master/.github/ISSUE_TEMPLATE.md
KKeXX commented at 2019-05-21 08:28:¶
@jsmeix
I've changed the actual name of the server to generic name hostname by
hand. The size of the image is because of netbackup. The server is a
netbackup media server and there the /usr/openv/netbackup directory is
almost 3GB and gets partly included into the iso.
Now I've exclued one file with NetBackup Client install files (/usr/openv/netbackup/client)
I've testet your git master:
tstn1 /var/tmp/rear.github.master # bash -x usr/sbin/rear -D mkrescue 2>/tmp/rear.stderr.log
Relax-and-Recover 2.5 / Git
Running rear mkrescue (PID 7024)
Using log file: /var/tmp/rear.github.master/var/log/rear/rear-tstn1.log
Using autodetected kernel '/boot/vmlinuz-3.10.0-957.12.1.el7.x86_64' as kernel in the recovery system
Creating disk layout
Using guessed bootloader 'GRUB' (found in first bytes on /dev/sda)
Verifying that the entries in /var/tmp/rear.github.master/var/lib/rear/layout/disklayout.conf are correct ...
Creating root filesystem layout
Handling network interface 'bond0'
bond0 is a bond
bond0 has lower interface enp3s0f0
enp3s0f0 is a physical device
bond0 has lower interface enp3s0f1
enp3s0f1 is a physical device
Handled network interface 'bond0'
Copying logfile /var/tmp/rear.github.master/var/log/rear/rear-tstn1.log into initramfs as '/tmp/rear-tstn1-partial-2019-05-21T09:56:43+0200.log'
Copying files and directories
Copying binaries and libraries
Copying all kernel modules in /lib/modules/3.10.0-957.12.1.el7.x86_64 (MODULES contains 'all_modules')
Copying all files in /lib*/firmware/
Skip copying broken symlink '/etc/mtab' target '/proc/14995/mounts' on /proc/ /sys/ /dev/ or /run/
Testing that the recovery system in /var/tmp/rear.FKHQMCXSPTAZxYn/rootfs contains a usable system
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (775526816 bytes) in 132 seconds
Making ISO image
Wrote ISO image: /var/tmp/rear.github.master/var/lib/rear/output/rear-tstn1.iso (748M)
Copying resulting files to file location
Saving /var/tmp/rear.github.master/var/log/rear/rear-tstn1.log as rear-tstn1.log to file location
Copying result files '/var/tmp/rear.github.master/var/lib/rear/recovery/cfg2html/tstn1.html /var/tmp/rear.github.master/var/lib/rear/output/rear-tstn1.iso /var/tmp/rear.FKHQMCXSPTAZxYn/tmp/VERSION /var/tmp/rear.FKHQMCXSPTAZxYn/tmp/README /var/tmp/rear.FKHQMCXSPTAZxYn/tmp/rear-tstn1.log' to /root/rear/tstn1 at file location
Exiting rear mkrescue (PID 7024) and its descendant processes ...
Running exit tasks
You should also rm -Rf /var/tmp/rear.FKHQMCXSPTAZxYn
And here are the configs:
tstn1 /var/tmp/rear.github.master # cat etc/rear/local.conf
# Default is to create Relax-and-Recover rescue media as ISO image
# set OUTPUT to change that
# set BACKUP to activate an automated (backup and) restore of your data
# Possible configuration values can be found in /usr/share/rear/conf/default.conf
#
# This file (local.conf) is intended for manual configuration. For configuration
# through packages and other automated means we recommend creating a new
# file named site.conf next to this file and to leave the local.conf as it is.
# Our packages will never ship with a site.conf.
COPY_AS_IS+=( /usr/src/kernels )
tstn1 /var/tmp/rear.github.master # cat etc/rear/os.conf
OS_VENDOR=RedHatEnterpriseServer
OS_VERSION=7
tstn1 /var/tmp/rear.github.master # cat etc/rear/site.conf
export TMPDIR=/var/tmp
export BACKUP=NBU
export OUTPUT=ISO
export OUTPUT_URL="file:///root/rear"
export TYMESYNC=NTP
tstn1 /var/tmp/rear.github.master # grep NBU usr/share/rear/conf/default.conf
# BACKUP=NBU stuff (Symantec/Veritas NetBackup)
COPY_AS_IS_NBU=( /usr/openv/bin/vnetd /usr/openv/bin/vopied /usr/openv/lib /usr/openv/netbackup /usr/openv/var/auth/[mn]*.txt /usr/openv/pdde/pdopensource/lib /usr/openv/pdde/pdshared/lib )
COPY_AS_IS_EXCLUDE_NBU=( /usr/openv/netbackup/logs "/usr/openv/netbackup/bin/bpjava*" /usr/openv/netbackup/bin/xbp /usr/openv/netbackup/bin/private /usr/openv/lib/java /usr/openv/lib/shared/vddk /usr/openv/netbackup/baremetal /usr/openv/netbackup/client )
NBU_LD_LIBRARY_PATH="/usr/openv/lib:/usr/openv/netbackup/sec/at/lib/:/usr/openv/pdde/pdshared/lib/:/usr/openv/pdde/pdopensource/lib/"
PROGS_NBU=( )
And the logs:
/tmp/rear.stderr.log
rear.stderr.log
/var/tmp/rear.github.master/var/log/rear/rear-tstn1.log
rear_github_master_rear-tstn1.log
/var/log/rear/rear-tstn1.log
rear-tstn1.log
does this run look as it should?
jsmeix commented at 2019-05-21 10:02:¶
@KKeXX
your
/var/tmp/rear.github.master/var/log/rear/rear-tstn1.log
rear_github_master_rear-tstn1.log
https://github.com/rear/rear/files/3201465/rear_github_master_rear-tstn1.log
looks normal to me with some additional set -x log messages because of
bash -x /usr/sbin/rear -D mkrescue 2>/tmp/rear.stderr
You can concatenate your /tmp/rear.stderr.log and your
/var/tmp/rear.github.master/var/log/rear/rear-tstn1.log
to get the full continuous log of your
bash -x /usr/sbin/rear -D mkrescue 2>/tmp/rear.stderr
call.
Your
/var/log/rear/rear-tstn1.log
rear-tstn1.log
https://github.com/rear/rear/files/3201470/rear-tstn1.log
is a normal "rear dump" log which is usually not stored
as var/log/rear/rear-HOSTNAME.log
but as var/log/rear/rear-HOSTNAME.log.lockless
KKeXX commented at 2019-05-21 12:16:¶
@jsmeix
So, is this as it should be? Is this different to my installed version
2.5 / 2019-05-10?
jsmeix commented at 2019-05-22 07:29:¶
@KKeXX
your
/var/tmp/rear.github.master/var/log/rear/rear-tstn1.log
rear_github_master_rear-tstn1.log
https://github.com/rear/rear/files/3201465/rear_github_master_rear-tstn1.log
looks as it should be.
Your
/var/log/rear/rear-tstn1.log
rear-tstn1.log
https://github.com/rear/rear/files/3201470/rear-tstn1.log
is a normal "rear dump" log which looks as it should be
(but its filename should be rear-HOSTNAME.log.lockless)
provided you called "rear dump" but if you didn't call "rear dump"
but something different like "rear mkrescue" then that log file
is plain wrong for "rear mkrescue".
I cannot know how ReaR is installed on your particular system.
A side note:
When your ReaR recovery system ISO image is 2.7G big
I wonder how it works to run the recovery system on
your replacement hardware (or replacement virtual machine)
because a 2.7G recovery system size means one needs
at least that amount of RAM on the replacement hardware
because the ReaR recovery system is in a RAM disk
plus some additional RAM that can be normally used
to run the recovery system and its programs.
I.e. with a 2.7G recovery system size I assume one needs
at least about 4G RAM (perhaps even more) on the
replacement hardware to run the recovery system there.
KKeXX commented at 2019-05-29 13:08:¶
@jsmeix
I run rear again with the installed verson 2.5, from the opensuse
Repository:
tstn1 /var/tmp # bash -x /usr/sbin/rear -D mkrescue 2>/tmp/rear.stderr.log
Relax-and-Recover 2.5 / 2019-05-10
Running rear mkrescue (PID 22969)
Using log file: /var/log/rear/rear-tstn1.log
Using autodetected kernel '/boot/vmlinuz-3.10.0-957.12.1.el7.x86_64' as kernel in the recovery system
Creating disk layout
Using guessed bootloader 'GRUB' (found in first bytes on /dev/sda)
Verifying that the entries in /var/lib/rear/layout/disklayout.conf are correct ...
Creating root filesystem layout
Handling network interface 'bond0'
bond0 is a bond
bond0 has lower interface enp3s0f0
enp3s0f0 is a physical device
bond0 has lower interface enp3s0f1
enp3s0f1 is a physical device
Handled network interface 'bond0'
Copying logfile /var/log/rear/rear-tstn1.log into initramfs as '/tmp/rear-tstn1-partial-2019-05-29T14:16:13+0200.log'
Copying files and directories
Copying binaries and libraries
Copying all kernel modules in /lib/modules/3.10.0-957.12.1.el7.x86_64 (MODULES contains 'all_modules')
Copying all files in /lib*/firmware/
Skip copying broken symlink '/etc/mtab' target '/proc/340/mounts' on /proc/ /sys/ /dev/ or /run/
Testing that the recovery system in /var/tmp/rear.iEvZv4I9c1yXvi1/rootfs contains a usable system
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (775749828 bytes) in 127 seconds
Making ISO image
Wrote ISO image: /var/lib/rear/output/rear-tstn1.iso (749M)
Copying resulting files to file location
Saving /var/log/rear/rear-tstn1.log as rear-tstn1.log to file location
Copying result files '/var/lib/rear/recovery/cfg2html/tstn1.html /var/lib/rear/output/rear-tstn1.iso /var/tmp/rear.iEvZv4I9c1yXvi1/tmp/VERSION /var/tmp/rear.iEvZv4I9c1yXvi1/tmp/README /var/tmp/rear.iEvZv4I9c1yXvi1/tmp/rear-tstn1.log' to /root/rear/tstn1 at file location
Exiting rear mkrescue (PID 22969) and its descendant processes ...
Running exit tasks
You should also rm -Rf /var/tmp/rear.iEvZv4I9c1yXvi1
Config is the same as in comment https://github.com/rear/rear/issues/2144#issuecomment-494293351
And here the logs:
/tmp/rear.stderr.log
rear.stderr.log
/var/tmp/rear.iEvZv4I9c1yXvi1/tmp/rear-tstn1.log
rear-tstn1.log
/var/log/rear/rear-tstn1.log
rear-tstn1.log
I run the installed rear version like the one from git, see comment
https://github.com/rear/rear/issues/2144#issuecomment-494293351.
rear -D mkrescue. I never run rear dump neither with the version
from git nor with the installed version 2.5 from the opensuse Repo. Do
these two runs show any differences?
This test system is a physical Server with 12GB RAM, so the ISO size s not a problem :-)
jsmeix commented at 2019-06-03 12:22:¶
@KKeXX
in your
https://github.com/rear/rear/issues/2144#issuecomment-496929055
you /tmp/rear.stderr.log contains WORKFLOW=mkrescue
but your /var/log/rear/rear-tstn1.log seems to contradict in itself
because it contains at its beginning
2019-05-29 14:17:06.961806370 Running rear dump (PID 7837)
but then at its end
2019-05-29 14:25:03.573210010 Exiting rear mkrescue (PID 22969) and its descendant processes ...
It seems as if your /var/log/rear/rear-tstn1.log is initially written
by that obscure unwanted rear dump that seems to run in parallel
to rear mkrescue and when that rear dump had finished
the rear mkrescue is writing the rest of your
/var/log/rear/rear-tstn1.log because I get (excerpts)
# grep PID rear.stderr.log
...
++ MASTER_PID=22969
and
# egrep ' dump | mkrescue ' rear-tstn1.log
2019-05-29 14:17:06.961806370 Running rear dump (PID 7837)
2019-05-29 14:17:07.166508413 Running dump workflow
2019-05-29 14:17:07.405183620 # End of dump configuration and system information.
2019-05-29 14:17:07.406558511 Finished running dump workflow
...
2019-05-29 14:25:03.484481761 Finished running mkrescue workflow
...
2019-05-29 14:25:03.573210010 Exiting rear mkrescue (PID 22969) and its descendant processes ...
KKeXX commented at 2019-06-03 12:29:¶
@jsmeix
... and what does that mean? I've never run rear dump. How does that
happen? Is there another config option so that the dump is run in
parallel? I startet rear with the command i provided:
bash -x /usr/sbin/rear -D mkrescue 2>/tmp/rear.stderr.log
jsmeix commented at 2019-06-03 12:55:¶
I don't know how that obscure unwanted rear dump
happens to run in your particular environment.
Nothing like that happens or had ever happened in my environment.
I have no idea how it could happen at all because as far as I can tell
there is no code in ReaR that would automatically run rear dump
or any other workflow that was not specified.
JJ-at-github commented at 2019-08-12 18:00:¶
Hello
I got it running with NBU 8.1.2 and rear 2.5. After adding some libraries I can build the rescue image without a problem. It is also possible to start with the iso. The problem I have is that the master server cannot contact the client. It is possible to get the certificates from the master but if I try to do a communication with the client I get a 25 Error . Name resolution etc. is working. I think it is not rear related but maybe someone with NBU can give me a hint??
github-actions commented at 2020-06-27 01:33:¶
Stale issue message
[Export of Github issue for rear/rear.]