#2972 Issue open: Cannot create rescue image on Ubuntu Server 23.04 - Confirmed bug in cp 9.x¶
Labels: external tool, not ReaR / invalid
schlomo opened issue at 2023-04-23 10:47:¶
Trying out ReaR on Ubuntu 23.04 and I stumble over a very strange error:
Context¶
-
my configuration contains
MODULES=( 'loaded_modules' )otherwise this problem is not triggered.
-
running from source tree mounted via vboxsf but the problem is not related to that either, also not to
REAR_VAR.
Observable Problem¶
root@rear-u2304:/src/rear# REAR_VAR=/tmp/rear_var usr/sbin/rear -C /etc/rear/local.conf -D mkrescue
Relax-and-Recover 2.7 / Git
Running rear mkrescue (PID 55846 date 2023-04-23 10:32:31)
Command line options: usr/sbin/rear -C /etc/rear/local.conf -D mkrescue
Using log file: /tmp/rear_var/log/rear/rear-rear-u2304.log
Using build area: /var/tmp/rear.Acrvl84FH5J2jWn
Sourcing additional configuration file '/etc/rear/local.conf'
...
... lots more
...
Copying binaries and libraries
Copying only currently loaded kernel modules (MODULES contains 'loaded_modules') and those in MODULES_LOAD
ERROR: Failed to copy '/lib/modules/6.2.0-20-generic/kernel/arch/x86/crypto/aesni-intel.ko /lib/modules/6.2.0-20-generic/kernel/arch/x86/crypto/crc32-pclmul.ko /lib/modules/6.2.0-20-generic/kernel/arch/x86/crypto/crct10dif-pclmul.ko /lib/modules/6.2.0-20-generic/kernel/arch/x86/crypto/ghash-clmulni-intel.ko /lib/modules/6.2.0-20-generic/kernel/arch/x86/crypto/polyval-clmulni.ko /lib/modules/6.2.0-20-generic/kernel/arch/x86/crypto/sha512-ssse3.ko /lib/modules/6.2.0-20-generic/kernel/arch/x86/events/rapl.ko /lib/modules/6.2.0-20-generic/kernel/arch/x86/kernel/msr.ko /lib/modules/6.2.0-20-generic/kernel/crypto/async_tx/async_memcpy.ko /lib/modules/6.2.0-20-generic/kernel/crypto/async_tx/async_pq.ko /lib/modules/6.2.0-20-generic/kernel/crypto/async_tx/async_raid6_recov.ko /lib/modules/6.2.0-20-generic/kernel/crypto/async_tx/async_tx.ko /lib/modules/6.2.0-20-generic/kernel/crypto/async_tx/async_xor.ko /lib/modules/6.2.0-20-generic/kernel/crypto/blake2b_generic.ko /lib/modules/6.2.0-20-generic/kernel/crypto/cryptd.ko /lib/modules/6.2.0-20-generic/kernel/crypto/crypto_simd.ko /lib/modules/6.2.0-20-generic/kernel/crypto/polyval-generic.ko /lib/modules/6.2.0-20-generic/kernel/crypto/xor.ko /lib/modules/6.2.0-20-generic/kernel/drivers/acpi/video.ko /lib/modules/6.2.0-20-generic/kernel/drivers/ata/ahci.ko /lib/modules/6.2.0-20-generic/kernel/drivers/ata/libahci.ko /lib/modules/6.2.0-20-generic/kernel/drivers/ata/pata_acpi.ko /lib/modules/6.2.0-20-generic/kernel/drivers/firmware/dmi-sysfs.ko /lib/modules/6.2.0-20-generic/kernel/drivers/firmware/efi/efi-pstore.ko /lib/modules/6.2.0-20-generic/kernel/drivers/gpu/drm/drm.ko /lib/modules/6.2.0-20-generic/kernel/drivers/gpu/drm/drm_kms_helper.ko /lib/modules/6.2.0-20-generic/kernel/drivers/gpu/drm/drm_ttm_helper.ko /lib/modules/6.2.0-20-generic/kernel/drivers/gpu/drm/ttm/ttm.ko /lib/modules/6.2.0-20-generic/kernel/drivers/gpu/drm/vmwgfx/vmwgfx.ko /lib/modules/6.2.0-20-generic/kernel/drivers/hid/hid-generic.ko /lib/modules/6.2.0-20-generic/kernel/drivers/hid/hid.ko /lib/modules/6.2.0-20-generic/kernel/drivers/hid/usbhid/usbhid.ko /lib/modules/6.2.0-20-generic/kernel/drivers/i2c/busses/i2c-piix4.ko /lib/modules/6.2.0-20-generic/kernel/drivers/input/input-leds.ko /lib/modules/6.2.0-20-generic/kernel/drivers/input/joydev.ko /lib/modules/6.2.0-20-generic/kernel/drivers/input/mouse/psmouse.ko /lib/modules/6.2.0-20-generic/kernel/drivers/input/serio/serio_raw.ko /lib/modules/6.2.0-20-generic/kernel/drivers/macintosh/mac_hid.ko /lib/modules/6.2.0-20-generic/kernel/drivers/md/dm-multipath.ko /lib/modules/6.2.0-20-generic/kernel/drivers/md/linear.ko /lib/modules/6.2.0-20-generic/kernel/drivers/md/multipath.ko /lib/modules/6.2.0-20-generic/kernel/drivers/md/raid0.ko /lib/modules/6.2.0-20-generic/kernel/drivers/md/raid1.ko /lib/modules/6.2.0-20-generic/kernel/drivers/md/raid10.ko /lib/modules/6.2.0-20-generic/kernel/drivers/md/raid456.ko /lib/modules/6.2.0-20-generic/kernel/drivers/net/ethernet/intel/e1000/e1000.ko /lib/modules/6.2.0-20-generic/kernel/drivers/platform/x86/wmi.ko /lib/modules/6.2.0-20-generic/kernel/drivers/powercap/intel_rapl_common.ko /lib/modules/6.2.0-20-generic/kernel/drivers/powercap/intel_rapl_msr.ko /lib/modules/6.2.0-20-generic/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko /lib/modules/6.2.0-20-generic/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko /lib/modules/6.2.0-20-generic/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko /lib/modules/6.2.0-20-generic/kernel/drivers/video/fbdev/core/syscopyarea.ko /lib/modules/6.2.0-20-generic/kernel/drivers/video/fbdev/core/sysfillrect.ko /lib/modules/6.2.0-20-generic/kernel/drivers/video/fbdev/core/sysimgblt.ko /lib/modules/6.2.0-20-generic/kernel/drivers/virt/vboxguest/vboxguest.ko /lib/modules/6.2.0-20-generic/kernel/fs/autofs/autofs4.ko /lib/modules/6.2.0-20-generic/kernel/fs/binfmt_misc.ko /lib/modules/6.2.0-20-generic/kernel/fs/btrfs/btrfs.ko /lib/modules/6.2.0-20-generic/kernel/fs/fscache/fscache.ko /lib/modules/6.2.0-20-generic/kernel/fs/lockd/lockd.ko /lib/modules/6.2.0-20-generic/kernel/fs/netfs/netfs.ko /lib/modules/6.2.0-20-generic/kernel/fs/nfs/nfs.ko /lib/modules/6.2.0-20-generic/kernel/fs/nfs/nfsv4.ko /lib/modules/6.2.0-20-generic/kernel/fs/nfs_common/grace.ko /lib/modules/6.2.0-20-generic/kernel/fs/nls/nls_iso8859-1.ko /lib/modules/6.2.0-20-generic/kernel/fs/vboxsf/vboxsf.ko /lib/modules/6.2.0-20-generic/kernel/lib/libcrc32c.ko /lib/modules/6.2.0-20-generic/kernel/lib/raid6/raid6_pq.ko /lib/modules/6.2.0-20-generic/kernel/net/ipv4/netfilter/ip_tables.ko /lib/modules/6.2.0-20-generic/kernel/net/netfilter/x_tables.ko /lib/modules/6.2.0-20-generic/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko /lib/modules/6.2.0-20-generic/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko /lib/modules/6.2.0-20-generic/kernel/net/sunrpc/sunrpc.ko /lib/modules/6.2.0-20-generic/kernel/net/wireless/cfg80211.ko /lib/modules/6.2.0-20-generic/kernel/sound/ac97_bus.ko /lib/modules/6.2.0-20-generic/kernel/sound/core/snd-pcm.ko /lib/modules/6.2.0-20-generic/kernel/sound/core/snd-timer.ko /lib/modules/6.2.0-20-generic/kernel/sound/core/snd.ko /lib/modules/6.2.0-20-generic/kernel/sound/pci/ac97/snd-ac97-codec.ko /lib/modules/6.2.0-20-generic/kernel/sound/pci/snd-intel8x0.ko /lib/modules/6.2.0-20-generic/kernel/sound/soundcore.ko '
Some latest log messages since the last called script 400_copy_modules.sh:
'/lib/modules/6.2.0-20-generic/kernel/sound/core/snd-timer.ko' -> '/var/tmp/rear.Acrvl84FH5J2jWn/rootfs/lib/modules/6.2.0-20-generic/kernel/sound/core/snd-timer.ko'
'/lib/modules/6.2.0-20-generic/kernel/sound/core/snd.ko' -> '/var/tmp/rear.Acrvl84FH5J2jWn/rootfs/lib/modules/6.2.0-20-generic/kernel/sound/core/snd.ko'
/lib/modules/6.2.0-20-generic/kernel/sound/pci -> /var/tmp/rear.Acrvl84FH5J2jWn/rootfs/lib/modules/6.2.0-20-generic/kernel/sound/pci
/lib/modules/6.2.0-20-generic/kernel/sound/pci/ac97 -> /var/tmp/rear.Acrvl84FH5J2jWn/rootfs/lib/modules/6.2.0-20-generic/kernel/sound/pci/ac97
'/lib/modules/6.2.0-20-generic/kernel/sound/pci/ac97/snd-ac97-codec.ko' -> '/var/tmp/rear.Acrvl84FH5J2jWn/rootfs/lib/modules/6.2.0-20-generic/kernel/sound/pci/ac97/snd-ac97-codec.ko'
cp: 'lib/modules/6.2.0-20-generic/kernel/sound/pci/ac97': No such file or directory
'/lib/modules/6.2.0-20-generic/kernel/sound/pci/snd-intel8x0.ko' -> '/var/tmp/rear.Acrvl84FH5J2jWn/rootfs/lib/modules/6.2.0-20-generic/kernel/sound/pci/snd-intel8x0.ko'
'/lib/modules/6.2.0-20-generic/kernel/sound/soundcore.ko' -> '/var/tmp/rear.Acrvl84FH5J2jWn/rootfs/lib/modules/6.2.0-20-generic/kernel/sound/soundcore.ko'
Aborting due to an error, check /tmp/rear_var/log/rear/rear-rear-u2304.log for details
Exiting rear mkrescue (PID 55846) and its descendant processes ...
Running exit tasks
To remove the build area you may use (with caution): rm -Rf --one-file-system /var/tmp/rear.Acrvl84FH5J2jWn
Terminated
Root Cause Analysis¶
A bit further digging leads to the observation that the cp utility
fails even though it actually copied all files. Here is an extract to
illustrate the problem:
root@rear-u2304:~# cat /etc/os-release
PRETTY_NAME="Ubuntu 23.04"
NAME="Ubuntu"
VERSION_ID="23.04"
VERSION="23.04 (Lunar Lobster)"
VERSION_CODENAME=lunar
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=lunar
LOGO=ubuntu-logo
root@rear-u2304:~# rm -Rf /tmp/f && mkdir /tmp/f && cp --verbose -t /tmp/f -L --preserve=all --parents /etc/apt/sources.list && echo yes ; ls -lR /tmp/f/etc/apt/sources.list /etc/apt/sources.list
/etc -> /tmp/f/etc
/etc/apt -> /tmp/f/etc/apt
'/etc/apt/sources.list' -> '/tmp/f/etc/apt/sources.list'
cp: ‘etc/apt’: No such file or directory
-rw-r--r-- 1 root root 2437 Apr 23 09:53 /etc/apt/sources.list
-rw-r--r-- 1 root root 2437 Apr 23 09:53 /tmp/f/etc/apt/sources.list
root@rear-u2304:~#
Anybody has an idea what could be the problem here?
On an Ubuntu 22.04 system this works as expected:
schlomo@brho:~$ rm -Rf /tmp/f && mkdir /tmp/f && cp --verbose -t /tmp/f -L --preserve=all --parents /etc/apt/sources.list && echo yes ; ls -lR /tmp/f/etc/apt/sources.list /etc/apt/sources.list
/etc -> /tmp/f/etc
/etc/apt -> /tmp/f/etc/apt
'/etc/apt/sources.list' -> '/tmp/f/etc/apt/sources.list'
yes
-rw-r--r-- 1 root root 263 Mär 26 15:20 /etc/apt/sources.list
-rw-r--r-- 1 schlomo schlomo 263 Mär 26 15:20 /tmp/f/etc/apt/sources.list
schlomo@brho:~$
I reported this bug to Ubuntu via https://bugs.launchpad.net/ubuntu/+source/coreutils/+bug/2017414 and upstream via https://debbugs.gnu.org/cgi/bugreport.cgi?bug=63245
abbbi commented at 2023-05-03 08:31:¶
this will hit with upcoming debian/bookworm release, too.
Works with debian/bullseye:
rm -Rf /tmp/f && mkdir /tmp/f && cp --verbose -t /tmp/f -L --preserve=all --parents /etc/apt/sources.list && echo yes ; ls -lR /tmp/f/etc/apt/sources.list /etc/apt/sources.list
/etc -> /tmp/f/etc
/etc/apt -> /tmp/f/etc/apt
'/etc/apt/sources.list' -> '/tmp/f/etc/apt/sources.list'
yes
-rw-r--r-- 1 root root 47 May 3 10:29 /etc/apt/sources.list
-rw-r--r-- 1 root root 47 May 3 10:29 /tmp/f/etc/apt/sources.list
fails in bookworm and debian/sid:
rm -Rf /tmp/f && mkdir /tmp/f && cp --verbose -t /tmp/f -L --preserve=all --parents /etc/apt/sources.list && echo yes ; ls -lR /tmp/f/etc/apt/sources.list /etc/apt/sources.list
/etc -> /tmp/f/etc
/etc/apt -> /tmp/f/etc/apt
'/etc/apt/sources.list' -> '/tmp/f/etc/apt/sources.list'
cp: 'etc/apt': No such file or directory
-rw-r--r-- 1 root root 47 May 3 10:28 /etc/apt/sources.list
-rw-r--r-- 1 root root 47 May 3 10:28 /tmp/f/etc/apt/sources.list
schlomo commented at 2023-05-03 08:42:¶
Any idea how to escalate this to the upstream projects (coreutils)
abbbi commented at 2023-05-03 08:48:¶
i just tried with the latest coreutils source and the behavior is the same with the latest checkout, but theres a different error message, which suggests the issue is related to preserving
rm -Rf /tmp/f && mkdir /tmp/f && src/cp --verbose -t /tmp/f -L --preserve=all --parents /etc/apt/sources.list && echo yes ; ls -lR /tmp/f/etc/apt/sources.list /etc/apt/sources.list
/etc -> /tmp/f/etc
/etc/apt -> /tmp/f/etc/apt
'/etc/apt/sources.list' -> '/tmp/f/etc/apt/sources.list'
src/cp: preserving permissions for ‘/tmp/f/etc/apt’: No such file or directory
-rw-r--r-- 1 root root 278 Jun 7 2022 /etc/apt/sources.list
-rw-r--r-- 1 abi abi 278 Jun 7 2022 /tmp/f/etc/apt/sources.list
maybe its best to report this to https://debbugs.gnu.org/cgi/pkgreport.cgi?pkg=coreutils or the debian project directly.
abbbi commented at 2023-05-03 10:17:¶
i just attempted to mkrescue on debian/bookworm which ships rear 2.7 by default, and this worked like expected, so i guess this issue only hits if REAR_VAR=/tmp/rear_var is used or some other condition is met?
schlomo commented at 2023-05-03 10:22:¶
We use cp --preserve=all in these three places
https://github.com/rear/rear/blob/54df9693402bb1b81ed1633d33be4d3f772c071a/usr/share/rear/build/GNU/Linux/400_copy_modules.sh#L138
https://github.com/rear/rear/blob/54df9693402bb1b81ed1633d33be4d3f772c071a/usr/share/rear/build/GNU/Linux/400_copy_modules.sh#L223
https://github.com/rear/rear/blob/54df9693402bb1b81ed1633d33be4d3f772c071a/usr/share/rear/build/default/490_fix_broken_links.sh#L78
IMHO this code doesn't depend on REAR_VAR at all.
abbbi commented at 2023-05-03 10:25:¶
yes, i missed the (MODULES contains 'loaded_modules') part. Default is all_modules.
jsmeix commented at 2023-05-03 11:15:¶
At other places we use '--preserve=all' implicitly via cp ... -a:
# find usr/sbin/rear usr/share/rear -type f | xargs grep 'cp .* -a '
usr/share/rear/build/GNU/Linux/420_copy_firmware_files.sh:
cp $verbose -t $ROOTFS_DIR -a --parents /lib*/firmware 2>>/dev/$DISPENSABLE_OUTPUT_DEV 1>&2
usr/share/rear/build/GNU/Linux/400_copy_modules.sh:
if ! cp $verbose -t $ROOTFS_DIR -a -L --parents /lib/modules/$KERNEL_VERSION 2>>/dev/$DISPENSABLE_OUTPUT_DEV 1>&2 ; then
usr/share/rear/build/GNU/Linux/390_copy_binaries_libraries.sh:
test -e $ROOTFS_DIR/$lib || cp $v -a -f $lib $ROOTFS_DIR/$lib 1>&2
usr/share/rear/finalize/GNU/Linux/250_migrate_disk_devices_layout.sh:
cp $v -a --parents "$restored_file" $save_original_file_dir
I see right now that
# find usr/sbin/rear usr/share/rear -type f | xargs grep 'cp .*-a '
finds some more places:
usr/share/rear/output/RAWDISK/Linux-i386/270_create_grub2_efi_bootloader.sh:
cp -a $v "$original_efi_root/." "$RAWDISK_BOOT_EFI_STAGING_ROOT" || Error "Could not copy EFI configuration"
usr/share/rear/output/RAWDISK/Linux-i386/270_create_grub2_efi_bootloader.sh:
cp -a $v "$additional_grub_directory/$grub_modules_directory/$module" "$grub_target_directory/$grub_modules_directory"
usr/share/rear/build/Debian/i386/600_fix_debian_stuff.sh:
cp -a $v /lib/libdevmapper* $ROOTFS_DIR/lib/ >&2
usr/share/rear/rescue/default/910_copy_logfile.sh:
cp -a $v $RUNTIME_LOGFILE $ROOTFS_DIR/tmp/${logfile_basename%.*}-partial-$(date -Iseconds).log >&2
usr/share/rear/restore/Fedora/050_copy_dev_files.sh:
cp -a /dev/. $TARGET_FS_ROOT/dev/
usr/share/rear/restore/YUM/default/950_grub_mkconfig.sh:
cp -a $TARGET_FS_ROOT/boot/grub/grub.conf $TARGET_FS_ROOT/boot/grub/grub.conf.bak
usr/share/rear/restore/YUM/default/400_restore_packages.sh:
cp -a /etc/yum.repos.d/* $TARGET_FS_ROOT/etc/yum.repos.d/
usr/share/rear/restore/YUM/default/410_restore_backup.sh:
cp -a $TMP_DIR/restore-exclude-list.txt $TARGET_FS_ROOT/tmp
usr/share/rear/restore/default/990_move_away_restored_files.sh:
cp -a --parents $file_relative $move_away_dir || continue
usr/share/rear/restore/GALAXY11/default/500_migrate_commvault_certificates.sh:
cp -a $certs "$TARGET_FS_ROOT/$GALAXY11_HOME_DIRECTORY" || Error "Could not copy $certs"
usr/share/rear/finalize/default/110_bind_mount_proc_sys_dev_run.sh:
cp -a /dev/. $TARGET_FS_ROOT/dev || LogPrintError "Failed to copy /dev contents from ReaR recovery system to $TARGET_FS_ROOT/dev"
schlomo commented at 2023-05-03 11:29:¶
It seems related to -t i think, couldn't reproduce with regular cp -a
schlomo commented at 2023-05-03 17:40:¶
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=63245
is the upstream bugreport with a patch proposal
https://debbugs.gnu.org/cgi/bugreport.cgi?att=1;filename=cp-abs-parents.patch;msg=8;bug=63245
jsmeix commented at 2023-05-04 08:34:¶
openSUSE Bugzilla report:
https://bugzilla.opensuse.org/show_bug.cgi?id=1211081
SUSE SLE15 and openSUSE Leap 15 are not affected because
# osc list SUSE:SLE-15:Update coreutils | grep tar
coreutils-8.29.tar.xz
So it seems only openSUSE Tumbleweed is affected because
# osc list openSUSE:Factory coreutils | grep tar
coreutils-9.3.tar.xz
jsmeix commented at 2023-05-04 08:43:¶
@pcahyna
please have a look here regarding
whether or not RHEL or Fedora are affected
i.e. if RHEL or Fedora contain coreutils 9.1-9.3 inclusive.
pcahyna commented at 2023-05-04 08:49:¶
RHEL does not (coreutils 8.*), but Fedora does (f37 has coreutils 9.1).
jsmeix commented at 2023-05-31 12:51:¶
@rear/contributors
is there something simple, fail-safe, and backward compatible
that we could do with reasonable effort in ReaR
to avoid this issue in ReaR?
In general I think it does not belong to ReaR
to implement special stuff to avoid generic bugs
in standard programs that are called by ReaR.
[Export of Github issue for rear/rear.]