#3473 Issue `open`: ReaR automatisms overwrite sacrosanct user specified values¶

Labels: bug

jsmeix opened issue at 2025-05-21 13:12:¶

By accident I found at least one case where
ReaR autmatisms overwrite sacrosanct user specified values:

See
https://github.com/rear/rear/pull/3471#discussion_r2100254872

Fullquote of that comment:¶

I found one case where disk mappings are applied
to sacrosanct user config variable values:

It happens during "rear recover" for WRITE_PROTECTED_IDS
by calling the apply_layout_mappings() function for the
autogenerated ReaR config file 'rescue.conf' in
layout/prepare/default/320_apply_mappings.sh
via

for file_to_migrate in "$LAYOUT_FILE" "$original_disk_space_usage_file" "$rescue_config_file" ; do
...
    ... apply_layout_mappings "$file_to_migrate" ...

where WRITE_PROTECTED_IDS from the original system
had been written there during "rear mkrescue" in
prep/default/490_store_write_protect_settings.sh
via

{
...
    echo "WRITE_PROTECTED_IDS=( ${WRITE_PROTECTED_IDS[*]} )"
...
} >> "$ROOTFS_DIR/etc/rear/rescue.conf"

The likely plain wrong thing in this case here is
that in sbin/rear /etc/rear/local.conf is sourced
before /etc/rear/rescue.conf

# User configuration files, last thing is to overwrite variables if we are in the rescue system:
for config in site local rescue ; do
...
    ... Source "$CONFIG_DIR/$config.conf" ...

so as the comment states sbin/rear intentionally overwrites
sacrosanct user specified values in /etc/rear/local.conf
by autogenerated ReaR values in /etc/rear/rescue.conf
here in particular any user specified WRITE_PROTECTED_IDS
in his /etc/rear/local.conf will get completey replaced
by autogenerated ReaR's WRITE_PROTECTED_IDS=( ... )
in /etc/rear/rescue.conf so the user would have to adapt
/etc/rear/rescue.conf instead of /etc/rear/local.conf
but this may not help him because later during "rear recover"
his WRITE_PROTECTED_IDS in /etc/rear/rescue.conf may get
again modified by the apply_layout_mappings() function
in layout/prepare/default/320_apply_mappings.sh

WHAT A HORRIBLE PILE OF INCONSISTENT MESSY MESS!

End of fullquote of that comment.

Addendum:

In particular /etc/rear/rescue.conf is not mentioned
in "man rear" as a user config file because currently
"man rear" reads:

FILES
       ...

       /etc/rear/local.conf
           System specific configuration can be set here.

       /etc/rear/site.conf
           Site specific configuration can be set here
           (not created by default).

so only /etc/rear/local.conf and /etc/rear/site.conf
are meant to be user config files.

jsmeix commented at 2025-05-21 13:23:¶

My above
"WHAT A HORRIBLE PILE OF INCONSISTENT MESSY MESS!"
is not meant as a complaint or accusation.

It is meant as helpless cry how unsolvable things are in practice.

I do understand why for ReaR by default

last thing is to overwrite variables if we are in the rescue system

as an attempt to automatically make things "just work"
for the user when things are different in the rescue system
compared to the original system.
BUT
At the same time this totally contradicts that
ReaR should provide "final power to the user"
which means it should work that the user adapts
his config variable values in his /etc/rear/local.conf
in his ReaR recovery system before he runs "rear recover"
and then ReaR obeys what his user had told it to do, cf.
https://github.com/rear/rear/pull/3471#discussion_r2100010738

So the dilemma is how to do both
make things automatically "just work" for the user
and still provide "final power to the user".

Currently I have no good idea how to solve that
in practice with reasonable effort.
Therefore I set the milestone to 'ReaR 999 (future)'
"Those issues need more discussion, or are not tied into a specific release."

pcahyna commented at 2025-05-21 13:45:¶

quote of my reply:

@jsmeix I actually thought the opposite - I thought that mappings are being applied to configuration values at more places, I am now surprised that there is only one.
I would actually argue for applying them more consistently to more values. What is the value of being able to specify DISKS_TO_BE_WIPED, for example, if you don't know what will the names will look like in the rescue system? You tell ReaR during rescue image creation that it should wipe /dev/sda, this gets embedded into the rescue image, but when the image boots, the devices swap and ReaR now wipes what you thought was /dev/sdb instead, as it has become /dev/sda now. Oops.

because the user can adapt his config variable values in /etc/rear/local.conf in the ReaR recovery system before he runs "rear recover"

is a valid argument for sure, but not very convincing to me. How often do users want do do this? Usually they want to automate, which means to configure stuff during (or before) rescue image creation, not during recovery.
More importantly, the mappings get applied to the layout file (that's the main point of mappings) before starting the recovery process. What if the user has edited the layout file in the rescue image before running "rear recover"? The user's values will now get mapped as well, yet we don't mind.

during "rear recover"
his WRITE_PROTECTED_IDS in /etc/rear/rescue.conf may get
again modified by the apply_layout_mappings() function
in layout/prepare/default/320_apply_mappings.sh

And what if "rear recover" fails and the user reruns it? Will rescue.conf get mapped AGAIN? :) So what was originally /dev/sda will get first mapped to /dev/sdb, but in the second run will get mapped again to /dev/sda if /dev/sda swaps with /dev/sdb ?

jsmeix commented at 2025-05-21 14:53:¶

@pcahyna
I did not test what happens in such cases.
I only tested "usual known cases".
I fear anything may happen when "rear recover" is rerun.

In particular because the apply_layout_mappings() function
is "just run" on the original file (via 'sed -i')
and no backup of the original file is made
so after the apply_layout_mappings() function was run
one cannot tell by looking at e.g. /etc/rear/local.conf
if it is still the original one or a changed one
because there is only one /etc/rear/local.conf

According to my personal experience in practice
it often fails in weird ways when "rear recover" is rerun
depending on how things are after "rear recover" failed.
So in practice when "rear recover" failed
it is usually more simple and straightforward
to find out why "rear recover" failed,
then reboot the ReaR recovery system anew
and adapt things as needed in the rebooted ReaR recovery system
before starting "rear recover".
What can help in sufficiently simple cases
to successfully run "rear recover" again
in the same ReaR recovery system is
to umount all on /mnt/local and
to sufficiently wipe the target system disks
but in complicated cases this is not enough.

jsmeix commented at 2025-05-22 06:30:¶

Currently I think the only generic way out is
to set all user config variables 'readonly'
in /sbin/rear after the user config files were sourced
to ensure that ReaR cannot change what his user specified
and from the user config variable values ReaR must set
its own ReaR-internal variables which ReaR can change as needed.

I.e. user config variables are owned by the user
and user property must be sacrosanct for ReaR.

When ReaR uses its own ReaR-internal variables
it only helps to be able to know at any time
while ReaR is running what the user had specified
but it cannot help against that ReaR may behave
different (via its own ReaR-internal variables)
than what his user had specified.

jsmeix commented at 2025-05-22 07:18:¶

FYI how often we modify user config variables in ReaR:

# USER_OWNED_VARIABLES=( $( grep -v '^[[:space:]]*#' usr/share/rear/conf/default.conf | grep -o '[A-Z][[:alnum:]_]*=' | cut -d '=' -f1 | sort -u ) )

# for v in "${USER_OWNED_VARIABLES[@]}" ; \
  do echo -n "$v " ; \
    find usr/sbin/rear usr/share/rear/ -type f \
    | xargs egrep -w "$v=|$v+=" \
    | egrep -v ': *#|usr/share/rear/conf/' \
    | wc -l ; \
  done | grep -v ' 0$' | sort -n -k2 -r

BACKUP_URL 15
UEFI_BOOTLOADER 11
MIGRATION_MODE 11
OUTPUT_URL 10
KERNEL_FILE 8
BACKUP 8
OUTPUT 7
LD_LIBRARY_PATH_FOR_BACKUP_TOOL 7
YUM_BACKUP_FILES 6
USB_DEVICE 6
OS_VERSION 6
USE_SERIAL_CONSOLE 5
NSRSERVER 5
FIRMWARE_FILES 5
YUM_BACKUP_SELINUX_CONTEXTS 4
USE_DHCLIENT 4
TMPDIR 4
KERNEL_VERSION 4
BACKUP_PROG_COMPRESS_SUFFIX 4
USB_UEFI_PART_SIZE 3
USB_PARTITION_ALIGN_BLOCK_SIZE 3
USB_DEVICE_PARTED_LABEL 3
USB_DEVICE_FILESYSTEM_LABEL 3
USB_DEVICE_BOOT_LABEL 3
TRUSTED_PATHS 3
REQUIRED_PROGS 3
PROGS 3
GRUB2_SEARCH_ROOT_COMMAND 3
BAREOS_CLIENT 3
BACKUP_PROG 3
BACKUP_OPTIONS 3
SSH_FILES 2
RBME_BACKUP 2
PXE_TFTP_IP 2
PPDM_RESTORE_PIT 2
OUTPUT_UMOUNTCMD 2
OUTPUT_OPTIONS 2
OUTPUT_MOUNTCMD 2
OS_VENDOR 2
OPAL_PBA_UNLOCK_MODE 2
NSR_RETENTION_TIME 2
NETFS_RESTORE_CAPABILITIES 2
NBKDC_DIR 2
MODULES 2
KERNEL_CMDLINE 2
KEEP_BUILD_DIR 2
ISO_ISOLINUX_BIN 2
GRUB2_SET_ROOT_COMMAND 2
GALAXY7_BACKUPSET 2
GALAXY11_PIT_RECOVERY 2
GALAXY11_BACKUPSET 2
GALAXY10_BACKUPSET 2
EXCLUDE_RUNTIME_LOGFILE 2
DUPLY_PROFILE 2
DISKS_TO_BE_WIPED 2
BORGBACKUP_RESTORE_ARCHIVES_SHOW_MAX 2
BAREOS_RESTORE_JOB 2
BAREOS_FILESET 2
BACULA_CONF_DIR 2
BACULA_BIN_DIR 2
BACKUP_RSYNC_OPTIONS 2
BACKUP_PROG_SUFFIX 2
BACKUP_PROG_CRYPT_KEY 2
BACKUP_PROG_COMPRESS_OPTIONS 2
WRITE_PROTECTED_IDS 1
WRITE_PROTECTED_FS_LABEL_PATTERNS 1
WAIT_SECS 1
VEEAM_BACKUPID 1
USING_UEFI_BOOTLOADER 1
USE_RESOLV_CONF 1
USE_RAMDISK 1
USB_DEVICE_FILESYSTEM_PERCENTAGE 1
USB_DEVICE_FILESYSTEM 1
USB_BOOT_PART_SIZE 1
USB_BOOTLOADER 1
TSM_RESULT_FILE_PATH 1
TSM_LD_LIBRARY_PATH 1
TRUSTED_OWNERS 1
SSH_UNPROTECTED_PRIVATE_KEYS 1
SSH_ROOT_PASSWORD 1
SECURE_BOOT_BOOTLOADER 1
RSYNC_PROTOCOL_VERSION 1
RESULT_MAILSUBJECT 1
RESULT_FILES 1
REBUILD_INITRAMFS 1
REAR_INITRD_COMPRESSION 1
RAWDISK_INSTALL_GPT_PARTITION_NAME 1
RAWDISK_IMAGE_NAME 1
RAWDISK_IMAGE_COMPRESSION_COMMAND 1
RAWDISK_GPT_PARTITION_NAME 1
RAWDISK_FAT_VOLUME_LABEL 1
PXE_TFTP_UPLOAD_URL 1
PXE_HTTP_DOWNLOAD_URL 1
PPDM_ASSETS_AND_SSIDS 1
OPAL_PBA_TKNPATH 1
OPAL_PBA_TKNOFFSET 1
OPAL_PBA_TKNKEY 1
OPAL_PBA_TKNDBG 1
OPAL_PBA_TKNBIND 1
OPAL_PBA_TKN2FAMAXTRIES 1
OPAL_PBA_TKN2FAFAILWIPE 1
OPAL_PBA_SBWARN 1
OPAL_PBA_NOSUCCESSMSG 1
OPAL_PBA_DEBUG_PASSWORD 1
OPAL_PBA_DEBUG_DEVICE_COUNT 1
NETFS_PREFIX 1
NETFS_KEEP_OLD_BACKUP_COPY 1
ISO_FILE_SIZE_LIMIT 1
ISO_DEFAULT 1
HOSTNAME 1
GALAXY_PORT 1
GALAXY_LOGONID 1
GALAXY_INSTANCE 1
GALAXY_COMMCELL 1
GALAXY_BACKUPSET 1
GALAXY11_TEMP_DIRECTORY 1
GALAXY11_LOG_DIRECTORY 1
GALAXY11_JOBS_RESULTS_DIRECTORY 1
GALAXY11_HOME_DIRECTORY 1
GALAXY11_CORE_DIRECTORY 1
FULLBACKUP_OUTDATED_DAYS 1
FORMAT_DASDS 1
ELILO_BIN 1
EFI_STUB_EFIBOOTMGR_ARGS 1
EFI_STUB 1
COPY_AS_IS 1
CLONE_ALL_USERS_GROUPS 1
BORGBACKUP_PRUNE_SHOW_STATS 1
BORGBACKUP_PRUNE_SHOW_RC 1
BORGBACKUP_PRUNE_SHOW_PROGRESS 1
BORGBACKUP_PRUNE_SHOW_LIST 1
BORGBACKUP_EXTRACT_SHOW_RC 1
BORGBACKUP_EXTRACT_SHOW_PROGRESS 1
BORGBACKUP_EXTRACT_SHOW_LIST 1
BORGBACKUP_CREATE_SHOW_STATS 1
BORGBACKUP_CREATE_SHOW_RC 1
BORGBACKUP_CREATE_SHOW_PROGRESS 1
BORGBACKUP_CREATE_SHOW_LIST 1
BOOTLOADER 1
BLOCKCLONE_STRICT_PARTITIONING 1
BEXTRACT_VOLUME 1
BACKUP_UMOUNTCMD 1
BACKUP_PROG_OPTIONS 1
BACKUP_MOUNTCMD 1

There are very many false positives like
BLOCKCLONE_STRICT_PARTITIONING= in a message text
usr/share/rear/layout/prepare/GNU/Linux/100_include_partition_code.sh:

        echo "# If you want avoid this, set BLOCKCLONE_STRICT_PARTITIONING=\"no\""

or
CLONE_ALL_USERS_GROUPS= in a trailing comment
usr/share/rear/prep/BACULA/default/400_prep_bacula.sh:

CLONE_GROUPS+=( bacula ) # default CLONE_ALL_USERS_GROUPS="true" in default.conf, but just in case...

or
MIGRATION_MODE='true'
at several places in usr/share/rear/layout/prepare/default/250_compare_disks.sh
to let ReaR set it when the user has not specified it.

I think there are only a few exceptional cases
where ReaR overwrites what his user specified
because otherwise we should have got user reports
when ReaR does not behave as specified so I think
in practice things work reasonably well.

Currently I don't have a good idea how to find
those exceptional cases where ReaR overwrites
what his user has specified.

jsmeix commented at 2025-05-26 15:43:¶

https://github.com/rear/rear/issues/2181
is a somewhat related issue where apply_layout_mappings()
was not called for some files that need to get migrated
but it seems those few cases were only the tip of the iceberg

jsmeix commented at 2025-06-03 07:31:¶

Regarding
https://github.com/rear/rear/issues/3473#issuecomment-2898030954

mappings are being applied to configuration values at more places

I created the new separated issue
https://github.com/rear/rear/issues/3477

[Export of Github issue for rear/rear.]

#3473 Issue open: ReaR automatisms overwrite sacrosanct user specified values¶