#2925 Issue closed: "rear mkrescue" does not error out when COPY_AS_IS="/some/dir" results broken recovery system

Labels: enhancement, support / question, fixed / solved / done

Scooty-Puff-Jr opened issue at 2023-02-13 16:57:

Relax-and-Recover (ReaR) Issue Template

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

  • ReaR version: 2.6

  • OS version: RHEL 8.5

  • ReaR configuration files:
    OUTPUT=USB
    OUTPUT_URL="usb:///dev/disk/by-label/REAR-000"
    BACKUP=NETFS
    BACKUP_URL="usb:///dev/disk/by-label/REAR-000"
    BACKUP_SELINUX_DISABLE=1
    CLONE_ALL_USERS=y
    USE_SERIAL_CONSOLE=n
    SSH_ROOT_PASSWORD=
    USE_SERIAL_CONSOLE=n
    COPY_AS_IS="/usr/src/kernals/4.18.0-348.el8.x86_64"
    USB_UEFI_PART_SIZE="650"
    EOF

  • Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR):

  • System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):

  • Firmware: GRUB

  • Storage: SSD

  • Description of the issue: So I was able create a mkBACKUP on a external lockable SSD that I had available. I am able to boot to the drive by plugging the device and the SSD into a docking station to keep the lockable ssd powered so it won't auto lock during the reboot process, however once booted to the drive and it enters ReaR the drive umounts and instantly locks after loading into rear. Once ReaR loads and asks for the login, I do that and attempt a rear recover, but says it can't find /bin/rear to find the default.conf file. I unlock the drive, do a lsblk to make sure that it shows that it is mounted on the system, but I get the same result.

Is it impossible to use a lockable drive as the USB device or am I missing something to make this work correctly?
20230213_100912

schlomo commented at 2023-02-13 18:32:

Can you please use version 2.7? We can't support older version, sorry.

Also, how did you install ReaR? @jsmeix do we have RHEL 8 repos with current packages somewhere?

Is there a specific reason for copying a kernel source to the rescue system? What would you do with that during recovery?

IMHO the lockable USB storage device should not make a difference as long as it is unlocked. The rescue systems runs completely out of a RAM disk (created by the initrd so that at this moment nothing should be mounted in any case.

Bottom line: This looks like an installation gone wrong or a very unusual system layout. After upgrading to 2.7 please try again and also provide us the log file from the rear mkrescue and the output of rear dump

Scooty-Puff-Jr commented at 2023-02-13 19:38:

I can try the newer version. Didn't realize I was using an older version.

yum install rear

I had to do the copy as is with that kernel because the mkbackup kept erroring out with that specific line as the cause, when I went looking for solutions that copy as is line was what I found and once I included it, I could at the very least create the backup.

That makes sense, but why would it reach out for the default.conf file during that phase?

This was a second go round, I uninstalled rear and all of the packages with it. Reinstalled it and had the same error at the same point.

I will attempt with the newer version and try again.

schlomo commented at 2023-02-13 19:46:

/usr/share/rear/conf/default.conf is always read by rear, which btw is copied in its entirety into the rescue system.

The fact that it is missing shows that there is some fundamental problem.

Please try with a recent version and als show us the output and log of rear dump, rear -v mkrescue or rear -v mkbackup, whatever you used.

schlomo commented at 2023-02-13 19:47:

BTW, you can configure SSH for the rescue system to make your life simpler.

jsmeix commented at 2023-02-14 07:52:

The cause is

COPY_AS_IS="/usr/src/kernals/4.18.0-348.el8.x86_64"

in etc/rear/local.conf (or etc/rear/site.conf)
because this overwrites

COPY_AS_IS=( $SHARE_DIR $VAR_DIR )

in usr/share/rear/conf/default.conf that is sourced in usr/sbin/rear
before etc/rear/local.conf and etc/rear/site.conf get sourced
so nothing in usr/share/rear/ and var/lib/rear/ gets copied
into the ReaR rescue/recovery system.

Use

COPY_AS_IS+=( /usr/src/kernals/4.18.0-348.el8.x86_64 )

instead.

Note that COPY_AS_IS is an array,
see the initial comments in our ReaR 2.6 upstream
usr/share/rear/conf/default.conf
https://github.com/rear/rear/blob/rear-2.6/usr/share/rear/conf/default.conf

See also the comments in our original
ReaR 2.6 upstream etc/rear/local.conf
https://github.com/rear/rear/blob/rear-2.6/etc/rear/local.conf

By the way:
Is it really '/usr/src/kernals/' and not '/usr/src/kernels/'?

jsmeix commented at 2023-02-14 08:00:

In the openSUSE Build Service (OBS)
there is no RHEL 8 repository avialable for building on
https://build.opensuse.org/projects/Archiving:Backup:Rear/distributions/new
Login at OBS is needed to see that - without login at OBS one gets
"Sorry, you are not authorized to update this project."

schlomo commented at 2023-02-14 08:09:

Nice catch @jsmeix!

Maybe some day we can do a simple check for such common mistakes by validating that variables that should be an array are actually still an array, or at least the ones where we always expect more than one value in them, like this COPY_AS_IS.

jsmeix commented at 2023-02-14 08:38:

Interestingly "rear mkrescue" does not error out with

COPY_AS_IS="/some/directory"

I tested it right now on a test VM with

COPY_AS_IS="/home/johannes"

I would have expected that something fails with that.
But it looks to the user as if it had "just worked" ok:

# usr/sbin/rear -D mkrescue
Relax-and-Recover 2.6 / Git
Running rear mkrescue (PID 2369)
Using log file: /root/rear.github.master/var/log/rear/rear-localhost.log
Running workflow mkrescue on the normal/original system
Using autodetected kernel '/boot/vmlinuz-5.14.21-150400.22-default' as kernel in the recovery system
Creating disk layout
Overwriting existing disk layout file /root/rear.github.master/var/lib/rear/layout/disklayout.conf
Using sysconfig bootloader 'grub2'
Verifying that the entries in /root/rear.github.master/var/lib/rear/layout/disklayout.conf are correct ...
Creating recovery system root filesystem skeleton layout
Handling network interface 'eth0'
eth0 is a physical device
Handled network interface 'eth0'
Copying logfile /root/rear.github.master/var/log/rear/rear-localhost.log into initramfs as '/tmp/rear-localhost-partial-2023-02-14T09:18:32+01:00.log'
Copying files and directories
Copying binaries and libraries
Copying only currently loaded kernel modules (MODULES contains 'loaded_modules')
Omit copying files in /lib*/firmware/ (FIRMWARE_FILES='no')
Skip copying broken symlink '/etc/resolv.conf' target '/run/netconfig/resolv.conf' on /proc/ /sys/ /dev/ or /run/
Skip copying broken symlink '/etc/mtab' target '/proc/13197/mounts' on /proc/ /sys/ /dev/ or /run/
Broken symlink '/usr/share/rear' in recovery system because 'readlink' cannot determine its link target
Testing that the recovery system in /tmp/rear.BaBizTjSCHecwck/rootfs contains a usable system
Skipped ldd test for '/home/johannes/.xinitrc.template' (owner 'johannes' not in TRUSTED_FILE_OWNERS)
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (63550540 bytes) in 12 seconds
Making ISO image
Wrote ISO image: /root/rear.github.master/var/lib/rear/output/rear-localhost.iso (74M)
Copying resulting files to nfs location
Saving /root/rear.github.master/var/log/rear/rear-localhost.log as rear-localhost.log to nfs location
Copying result files '/root/rear.github.master/var/lib/rear/output/rear-localhost.iso /tmp/rear.BaBizTjSCHecwck/tmp/VERSION /tmp/rear.BaBizTjSCHecwck/tmp/README /tmp/rear.BaBizTjSCHecwck/tmp/rear-localhost.log' to /tmp/rear.BaBizTjSCHecwck/outputfs/localhost at nfs location
Exiting rear mkrescue (PID 2369) and its descendant processes ...
Running exit tasks
You should also rm -Rf /tmp/rear.BaBizTjSCHecwck

I think I will enhance
usr/share/rear/build/default/990_verify_rootfs.sh
to verify that 'rear' can be run inside the recovery system
e.g. by calling

rear dump 1>/dev/null

or something similar.

When I call this inside the ReaR recovery system
with 'COPY_AS_IS="/some/directory"' I get

# chroot /tmp/rear.BaBizTjSCHecwck/rootfs/

bash-4.4# rear dump 1>/dev/null
/sbin/rear: line 312: /usr/share/rear/conf/default.conf: No such file or directory
/sbin/rear: line 377: /usr/share/rear/lib/_input-output-functions.sh: No such file or directory
/sbin/rear: line 384: : No such file or directory
/sbin/rear: line 390: : No such file or directory
/sbin/rear: line 407: LogPrint: command not found
/sbin/rear: line 408: LogPrint: command not found
/sbin/rear: line 409: Log: command not found
/sbin/rear: line 410: LogPrint: command not found
/sbin/rear: line 424: Debug: command not found
/sbin/rear: line 425: Debug: command not found
/sbin/rear: line 430: Debug: command not found
/sbin/rear: line 433: Source: command not found
/sbin/rear: line 435: SetOSVendorAndVersion: command not found
/sbin/rear: line 447: Source: command not found
/sbin/rear: line 447: Source: command not found
/sbin/rear: line 509: SourceStage: command not found
/sbin/rear: line 528: QuietAddExitTask: command not found
/sbin/rear: line 529: Log: command not found
/sbin/rear: line 537: has_binary: command not found
/sbin/rear: line 544: LogPrintError: command not found
/sbin/rear: line 576: LogToSyslog: command not found

bash-4.4# echo $?
1

For now I prefer a simple test for what is actually needed
inside the ReaR recovery system over a more abstract
verification of config variable settings because currently
I don't have a good idea how to implement that with reasonable
effort (which does of course not mean a general verification
of config variables cannot be implemented at a later time).

schlomo commented at 2023-02-14 09:46:

Should we keep this issue open till we actually add a sanity check?

I was thinking along the lines of parsing default.conf to find all variables that should be an array:
image

And then we can add a check like this to validate that all those variables are actually still arrays:

[[ "$(declare -p EXCLUDE_BACKUP)" =~ "declare -a" ]] && echo array

(and this code is not very robust, see https://stackoverflow.com/a/42877229 for the probably most "correct" implementation)

What do you think. Should I add that to ReaR?

jsmeix commented at 2023-02-14 10:34:

I did
https://github.com/rear/rear/issues/2930
for the generic test to have things separated.

By the way:

COPY_AS_IS=( /some/directory )

would be syntactically correct but still results
the same useless recovery system.

[Export of Github issue for rear/rear.]