#3539 PR merged: SourceStage() : Use 'sort -k 2 | uniq' instead of 'sort -k 2 -u'

Labels: bug, fixed / solved / done

jsmeix opened issue at 2025-11-26 13:53:

In the SourceStage function
use 'sort -k 2 | uniq' instead of 'sort -k 2 -u'
to avoid that scripts with same number_name
but under different paths get ignored.

jsmeix commented at 2025-11-27 12:02:

Quick initial test in a git clone
with BACKUP=NETFS:

# find usr/share/rear | grep 500_selinux_autorelabel.sh

usr/share/rear/restore/default/500_selinux_autorelabel.sh
usr/share/rear/restore/NETFS/default/500_selinux_autorelabel.sh
usr/share/rear/restore/BLOCKCLONE/default/500_selinux_autorelabel.sh

Without the change in this pull request:

# usr/sbin/rear -s recover | grep 500_selinux_autorelabel

Source restore/NETFS/default/500_selinux_autorelabel.sh

With the change in this pull request:

# usr/sbin/rear -s recover | grep 500_selinux_autorelabel

Source restore/NETFS/default/500_selinux_autorelabel.sh
Source restore/default/500_selinux_autorelabel.sh

pcahyna commented at 2025-11-28 10:47:

I did an investigation of what might be affected. I investigated all combinations (Cartesian product) of the following:

DISTROS=( debian-x86_64 fedora-x86_64 rhel-x86_64 rhel-s390x rhel-ppc64le rhel-aarch64 )

BACKUPS=( AVA
BACULA
BAREOS
BLOCKCLONE
BORG
CDM
DP
DUPLICITY
EXTERNAL
FDRUPSTREAM
GALAXY
GALAXY10
GALAXY11
GALAXY7
NBKDC
NBU
NETFS
NFS4SERVER
NSR
OBDR
OPALPBA
PPDM
RBME
REQUESTRESTORE
RSYNC
SESAM
TSM
VEEAM
YUM
ZYPPER
)

OUTPUTS=(
RAMDISK
ISO
PXE
OBDR
USB
RAWDISK
PORTABLE
)

Apart from SELinux, which we already covered, I found that

  • restore/default/900_create_missing_directories.sh gets skipped if BACKUP=SESAM
  • output/default/200_make_prefix_dir.sh gets skipped if BACKUP=RSYNC

The latter may have impact if one uses RSYNC backup and an OUPTUT_URL with a mountable destination (usb:, nfs:, file:, probably cifs:). Not sure what could be the impact of the former on SESAM users.

@jsmeix I can repeat the investigation with SUSE distros added.

schlomo commented at 2025-11-28 10:52:

@pcahyna can it be that this PR actually fixes your concern too? We do uniq on the full path names of the scripts, which should filter out duplicates that are the result of OS_VENDOR==OS_MASTER_VENDOR but it should not filter out duplicate filenames in separate paths.

Or did I misunderstand the code?

pcahyna commented at 2025-11-28 10:54:

@schlomo which concern?

pcahyna commented at 2025-11-28 10:55:

We do uniq on the full path names of the scripts, which should filter out duplicates that are the result of OS_VENDOR==OS_MASTER_VENDOR but it should not filter out duplicate filenames in separate paths.

yes, that's the point.

schlomo commented at 2025-11-28 11:39:

@schlomo which concern?

I have no concern, what did you mean?

jsmeix commented at 2025-11-28 11:52:

@pcahyna regarding your
https://github.com/rear/rear/pull/3539#issuecomment-3588811404

Apart from SELinux, which we already covered,
I found that

* restore/default/900_create_missing_directories.sh
  gets skipped if BACKUP=SESAM

* output/default/200_make_prefix_dir.sh
  gets skipped if BACKUP=RSYNC

Thank you for your exhaustive test.

I can confirm in my git clone the following:

With BACKUP=SESAM:

Without the change in this pull request

# usr/sbin/rear -s recover | grep create_missing_directories

Source restore/SESAM/default/900_create_missing_directories.sh
Source restore/SUSE_LINUX/910_create_missing_directories.sh

With the change in this pull request

# usr/sbin/rear -s recover | grep create_missing_directories

Source restore/SESAM/default/900_create_missing_directories.sh
Source restore/default/900_create_missing_directories.sh
Source restore/SUSE_LINUX/910_create_missing_directories.sh

With BACKUP=RSYNC:

Without the change in this pull request:

# usr/sbin/rear -s mkbackup | grep make_prefix_dir

Source output/RSYNC/default/200_make_prefix_dir.sh
Source backup/RSYNC/default/200_make_prefix_dir.sh

With the change in this pull request:

# usr/sbin/rear -s mkbackup | grep make_prefix_dir

Source output/RSYNC/default/200_make_prefix_dir.sh
Source output/default/200_make_prefix_dir.sh
Source backup/RSYNC/default/200_make_prefix_dir.sh

pcahyna commented at 2025-11-28 12:36:

@schlomo which concern?

I have no concern, what did you mean?

I meant "can it be that this PR actually fixes your concern too?" If you meant the concern in https://github.com/rear/rear/pull/3171#issuecomment-3581232814 , fixing it is the whole point.

pcahyna commented at 2025-11-28 13:54:

@jsmeix minor point: can you please squash all the commits together and force-push?

pcahyna commented at 2025-11-28 14:53:

@jsmeix do you want me to produce the SUSE report? If so I would need a list of variables like

  ARCH="Linux-i386"
  OS="GNU/Linux"
  OS_MASTER_VENDOR="Fedora"
  OS_MASTER_VERSION="43"
  OS_MASTER_VENDOR_ARCH="Fedora/i386"
  OS_MASTER_VENDOR_VERSION="Fedora/43"
  OS_MASTER_VENDOR_VERSION_ARCH="Fedora/43/i386"
  OS_VENDOR="Fedora"
  OS_VERSION="43"
  OS_VENDOR_ARCH="Fedora/i386"
  OS_VENDOR_VERSION="Fedora/43"
  OS_VENDOR_VERSION_ARCH="Fedora/43/i386"

(produced by rear dump).

pcahyna commented at 2025-11-30 21:38:

I described the bug that would be fixed by this PR with BACKUP=RSYNC in https://github.com/rear/rear/issues/3522#issuecomment-3590066618 .

schlomo commented at 2025-12-01 09:20:

What is missing for us to merge this?

jsmeix commented at 2025-12-02 09:26:

I will merge it right now
here in the GitHub web fronted
via its [Squash and merge] button.

[Export of Github issue for rear/rear.]