#2978 Issue open: BACKUP=RSYNC: BACKUP_PROG_ARCHIVE does not set backup destination (but sets backup logfile)

Labels: documentation, support / question, old version

milleld1 opened issue at 2023-05-03 21:11:

  • ReaR version ("/usr/sbin/rear -V"): 2.4

  • If your ReaR version is not the current version, explain why you can't upgrade: The system in question is required to use CentOS 7, and ReaR version 2.4 is the most up to date for that system.

  • OS version: CentOS Linux release 7.9.2009 (Core)

  • ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"): Is this part of the form asking me to paste my configuration file here? I'm not sure what to put here. I'm attaching at the end.

  • Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR): VM

  • System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device): I'm not sure

  • Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot): I'm not sure

  • Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe): I'm not sure

  • Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"):

NAME                                       KNAME     PKNAME    TRAN   TYPE FSTYPE     LABEL  SIZE MOUNTPOINT
/dev/sda                                   /dev/sda                   disk                   100G 
|-/dev/sda1                                /dev/sda1 /dev/sda         part vfat              200M /boot/efi
|-/dev/sda2                                /dev/sda2 /dev/sda         part xfs                 1G /boot
`-/dev/sda3                                /dev/sda3 /dev/sda         part LVM2_membe       98.8G 
  |-/dev/mapper/centos_dgvlocaldomain-root /dev/dm-0 /dev/sda3        lvm  xfs                50G /
  |-/dev/mapper/centos_dgvlocaldomain-swap /dev/dm-1 /dev/sda3        lvm  swap              7.9G [SWAP]
  `-/dev/mapper/centos_dgvlocaldomain-home /dev/dm-2 /dev/sda3        lvm  xfs              40.9G /home
/dev/sr0                                   /dev/sr0            sata   rom                   1024M
  • Description of the issue (ideally so that others can reproduce it):

This note (below) in deafult.conf seems to say that I can change the name of my backup to include the date, as is already done with differential and incremental backups. However, when I set BACKUP_PROG_ARCHIVE to the value written there, it only has the effect of changing the name of the resulting backup log file. It does not change the name of the backup itself.

# One might even create a dynamic name like BACKUP_PROG_ARCHIVE="backup_$( date -Iseconds )"
# in particular one can use BACKUP_PROG_ARCHIVE="$( date '+%Y-%m-%d-%H%M' )-F"
# to get the same full backup file name for BACKUP_TYPE="" as what is used
# for BACKUP_TYPE=incremental and/or BACKUP_TYPE=differential (see below).

With BACKUP_PROG_ARCHIVE unset or set to "" and after running rear mkbackuponly the result is:

  • the backup destination folder will be location/systemname/backup
  • the backup logfile will be backup-date.time.log.gz

With BACKUP_PROG_ARCHIVE set to "backup_$( date '+%Y-%m-%d-%H%M' )-F-anytexthere" and after running rear mkbackuponly the expected result is:

  • the backup destination folder will be location/systemname/backup_date-time-F-anytexthere
  • the backup logfile will be backup_date-time-F-anytexthere.log.gz

However the actual result is:

  • the backup destination folder will be location/systemname/backup
  • the backup logfile will be backup_date-time-F-anytexthere**-date.time**.log.gz

In my understanding, the comments in default.conf imply that setting BACKUP_PROG_ARCHIVE will allow one to change the name of the backup file, but this does not seem to happen, even with the provided example used exactly. Without dynamically named backups, when rear mkbackuponly is run, the previous backup is overwritten. This is not the behavior I want.

I was also expecting the backup log to inherit the string in BACKUP_PROG_ARCHIVE exactly, but that doesn't seem to be the case. If this is as intended, I suppose it's alright to have a redundant datetime stamp, but it would be nice to be able to control this exactly.

How can I exactly specify the name of each:

  • full backup file
  • backup log

?

Thank you for your time and consideration.

  • Workaround, if any: None found

  • Attachments, as applicable: local.conf

local.txt

jsmeix commented at 2023-05-04 06:09:

@milleld1

your https://github.com/rear/rear/files/11391181/local.txt
contains (without comments):

OUTPUT=ISO
BACKUP_PROG_ARCHIVE="backup_$( date '+%Y-%m-%d-%H%M' )-F"
BACKUP=RSYNC
BACKUP_URL=rsync://otherhost/home/rearbackupslocation
EXCLUDE_MOUNTPOINTS=( '/mnt/16G_thumb_drive' /SOMEDISKIDONTNEED )}

I am not a 'BACKUP=RSYNC' user but as far as I see in
https://github.com/rear/rear/blob/rear-2.4/usr/share/rear/backup/RSYNC/default/500_make_rsync_backup.sh
the BACKUP_PROG_ARCHIVE value is used in case of BACKUP=RSYNC
only to store the rsync log file as ${BACKUP_PROG_ARCHIVE}.log
but the rsync destination is hardcoded as

... ${RSYNC_PATH}/${RSYNC_PREFIX}/backup

So it seems you have to set the config variable
RSYNC_PREFIX to specify the rsync destination prefix
but with current code you cannot configure
the hardcoded 'backup' sub-directory.

See the section about "BACKUP=RSYNC" in conf/default.conf
https://github.com/rear/rear/blob/rear-2.4/usr/share/rear/conf/default.conf#L1793

jsmeix commented at 2023-05-04 12:00:

The current behaviour is now better explained in default.conf via
https://github.com/rear/rear/commit/586293c291d4e2f6864dfac7a1233bbdfe787126

milleld1 commented at 2023-05-04 19:59:

The hardcoding of directories to always end with /backup is problematic for me because when rsync tries to mkdir, it will fail without the --parents or -p option. I don't believe there's a way to specify to rsync that you want it to use that option with mkdir. But more generally, I simply want to control where rear puts the backup file. Is there a different way users are meant to do this? I can't have rear overwriting every backup it makes.

milleld1 commented at 2023-05-04 20:27:

P.S. If not to change the location of the backup, what is the RSYNC_PREFIX option intended to be used for?

jsmeix commented at 2023-05-05 12:46:

As far as I understand it RSYNC_PREFIX is meant
to specify the location of the backup,
but with current code only the prefix can be specified
and the actual location is ${RSYNC_PREFIX}/backup

I am not a 'rsync' user so I cannot properly answer
questions about particular 'rsync' behaviour as in
https://github.com/rear/rear/issues/2978#issuecomment-1535338662

I am wondering why different RSYNC_PREFIX values
like 'this' and 'that' would not work
to get different 'rsync' destinations like
'location/this/backup' and 'location/that/backup'
i.e. I am wondering why the hardcoded suffix 'backup'
causes a problem - but I am not a 'rsync' user
so I may misunderstand things.

milleld1 commented at 2023-05-09 20:09:

I can explain why the hard coded suffix /backup is problematic.

If you set RSYNC_PREFIX="this" then the final directory will be location/systemname/this/backup

If location/systemname/this already exsits, but .../backup does not, it's okay. Rysnc will internally use mkdir to create .../backup inside .../this
However, if location/systemname/this doesn't exist yet, the rsync command will fail. It cannot create more than one directory layer at a time. I think you'll see this is the case if you try to use the configuration file I provided. The error log says exactly the problem.

Now mkdir actually has an option to cover this case. It is --parents or -p. But rsync doesn't offer us an option to include that mkdir option. We cannot access the mkdir command internal to rsync's operation.

You might think we can just do this step ourselves before ReaR calls rsync. But we aren't actually making a directory with a static name like location/systemname/this/backup. We want a dynamic name that includes a datetime stamp. This also makes sure that ReaR is using a different every time, so that the backups are not overwritten.

I think If I end my configuration file with some arbitrary script that ensures the directory that needs to exist has already been made, it should work. But how will I know what its name is? ReaR is going to generate that name dynamically, but I don't know where or when. If I only include the hour in the directory name, there stands a good chance that this may work as long as we aren't near the top of the hour. But this is just not good enough precision. I need it to work every time so that this can be set on a schedule and left alone.

Do you understand now the problem I am trying to solve?

jsmeix commented at 2023-05-10 07:50:

@milleld1
only an offhanded and untested idea:

Does it perhaps somehow improve things for you
when you replace in your
usr/share/rear/backup/RSYNC/default/500_make_rsync_backup.sh
script the hardcoded '/backup' with '$BACKUP_PROG_ARCHIVE'?

In theory that should result that with

BACKUP_PROG_ARCHIVE="backup_$( date '+%Y-%m-%d-%H%M' )-F-anytexthere"

you get the backup destination folder

location/systemname/backup_date-time-F-anytexthere

Likely there are some more code places where
such a hardcoded '/backup' needs to be replaced
with '$BACKUP_PROG_ARCHIVE' to make this actually work.
For example it seems such a hardcoded '/backup' is also used in
usr/share/rear/output/RSYNC/default/200_make_prefix_dir.sh
https://github.com/rear/rear/blob/rear-2.4/usr/share/rear/output/RSYNC/default/200_make_prefix_dir.sh
In current GitHub master code this script is moved to
usr/share/rear/backup/RSYNC/default/200_make_prefix_dir.sh

The backup logfile will be (as before)

backup_date-time-F-anytexthere-date.time.log.gz

The additionally appended second 'date.time' comes from
usr/share/rear/backup/RSYNC/default/700_copy_backup_log.sh

# copy the backup.log & rear.log file to remote destination with timestamp added
Timestamp=$(date +%Y%m%d.%H%M)
...
... ${RSYNC_PATH}/${RSYNC_PREFIX}/${BACKUP_PROG_ARCHIVE}-${Timestamp}.log.gz

https://github.com/rear/rear/blob/rear-2.4/usr/share/rear/backup/RSYNC/default/700_copy_backup_log.sh

But when BACKUP_PROG_ARCHIVE="something_non_static"
works to get a rsync backup into a
'location/systemname/something_non_static'
directory, then there is the problem how to restore
from a non-static directory during "rear recover".
Having during "rear recover" something like

BACKUP_PROG_ARCHIVE="backup_$( date '+%Y-%m-%d-%H%M' )-F-anytexthere"

in etc/rear/local.conf cannot work because there is no directory
with the current timestamp when "rear recover" is running.
So to restore from a non-static directory you need to edit
etc/rear/local.conf before you call "rear recover" and specify
the right directory with an existing static timestamp like

BACKUP_PROG_ARCHIVE="backup_existing_timestmp-F-anytexthere"

jsmeix commented at 2023-05-10 07:57:

@pcahyna
could you perhaps have a look here as time permits?

I ask you in particular because some time ago
you did some code changes also for BACKUP=RSYNC
(in particular https://github.com/rear/rear/pull/2831)
so perhaps you are more familiar with how that actually works?

[Export of Github issue for rear/rear.]