#2875 Issue `open`: Optimizing for speed when rsync is in use¶

Labels: enhancement

fadamo opened issue at 2022-09-30 11:05:¶

By backing up the system via rsync, the process involves:

if test -d "${opath}" ; then
    rm -rf $v "${opath}.old" || Error "Could not remove '${opath}.old'"
    mv -f $v "${opath}" "${opath}.old" || Error "Could not move '${opath}'"
fi

So, basically, we have to wait for minutes (we save on NAS) due to "rm -rf" and many minutes due to rsync starting from scratch.
We found that modifying the code with something like:

if [ ! -d "${opath}" ]; then
    mkdir -p $v -m 0750 "${opath}"
    rm -Rf "${opath}.old"                # if exists
elif [ -d "${opath}.old" ]; then
    mv "${opath}.old" "${opath}.veryold"
    mv "${opath}" "${opath}.old"
    # better to update an old copy rather then starting from scratch (rsync --delete --delete-excluded is in use)
    mv "${opath}.veryold" "${opath}"
else
    mv "${opath}" "${opath}.old"
    mkdir -p $v -m 0750 "${opath}"
fi

we are able to backup a server in a couple of minutes.
What do you think about? Can the modification be considered?

pcahyna commented at 2022-09-30 11:09:¶

@fadamo could you please be more specific? What backup method and BACKUP_URL are you using? And which file does "the code" refer to?

fadamo commented at 2022-09-30 12:18:¶

We use this config:

OUTPUT=ISO
OUTPUT_URL=rsync://USER@NAS::/rear-nfs
BACKUP=RSYNC
BACKUP_URL=rsync://USER@NAS::/rear-nfs
RSYNC_PROTO=rsync
BACKUP_PROG=rsync

and actually we do this:

mount $BASEPATH
if [ ! -d $BASEPATH/$SRV ]; then
    mkdir -m 750 $BASEPATH/$SRV
    rm -Rf "$BASEPATH"/"$SRV".old
elif [ -d $BASEPATH/"$SRV".old ]; then
    mv "$BASEPATH"/"$SRV".old "$BASEPATH"/"$SRV".veryold
    mv "$BASEPATH"/"$SRV" "$BASEPATH"/"$SRV".old
    # meglio aggiornare una vecchia copia che copiare tutto
    mv "$BASEPATH"/"$SRV".veryold "$BASEPATH"/"$SRV"
else
    mv "$BASEPATH"/"$SRV" "$BASEPATH"/"$SRV".old
    mkdir -m 750 $BASEPATH/$SRV
fi
umount $BASEPATH

for every server before calling rear.

We have a lot of limitation by our security deparment, there is a firewall between servers and NAS..... a long story. We had to implement a complex configuration.
BTW, in the past I did some test on lab servers, modifying /usr/share/rear/output/default/150_save_copy_of_prefix_dir.sh and probably /usr/share/rear/backup/NETFS/default/150_save_copy_of_prefix_dir.sh, too.
The concept is not to start from scratch when rsync is in use.
FYI, this is the log of the backup of an Oracle server:

############################################################################
#  2022/09/24 17:40:53     orasrv1
############################################################################

Cleaning yum cache...

Relax-and-Recover 2.4 / Git
Using log file: /var/log/rear/rear-orasrv1.log
Creating disk layout
Excluding Volume Group vg01
Using guessed bootloader 'GRUB' (found in first bytes on /dev/sdar)
Creating root filesystem layout
Handling network interface 'p4p2'
p4p2 is a physical device
Handled network interface 'p4p2'
Handling network interface 'p5p2'
p5p2 is a physical device
Handled network interface 'p5p2'
Handling network interface 'pubteam0'
pubteam0 is a team
pubteam0 has lower interface p4p1
p4p1 is a physical device
pubteam0 has lower interface p5p1
p5p1 is a physical device
Handled network interface 'pubteam0'
Copying logfile /var/log/rear/rear-orasrv1.log into initramfs as '/tmp/rear-orasrv1-partial-2022-09-24T17:41:04+0200.log'
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Copying all files in /lib*/firmware/
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (316515506 bytes) in 36 seconds
Making ISO image
Wrote ISO image: /var/lib/rear/output/rear-orasrv1.iso (311M)
Copying resulting files to rsync://REARUSER@NAS::/rear-nfs location
Copying resulting files to rsync location
Saving /var/log/rear/rear-orasrv1.log as rear-orasrv1.log to rsync location
Calculating size of /boot
Calculating size of /u01/app/ahf
Calculating size of /u01/app/grid
Calculating size of /u01/app/19/grid
Calculating size of /home
Calculating size of /opt
Calculating size of /u01/app/oracle
Calculating size of /
Calculating size of /tmp
Calculating size of /var
Estimated size of local file systems is 107420 MB   <<<<<<<<
Creating rsync archive on 'NAS:rear-nfs'
Running archive operationOK
Number of deleted files: 0
Number of regular files transferred: 98
Total file size: 80,111,505,068 bytes
Total transferred file size: 1,904,123,937 bytes
Literal data: 1,264,759,071 bytes
Matched data: 639,364,866 bytes
File list size: 327,635
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 1,275,508,629
Total bytes received: 2,109,272

sent 1,275,508,629 bytes  received 2,109,272 bytes  5,120,713.03 bytes/sec
total size is 80,111,505,068  speedup is 62.70 in 250 seconds.
Exiting rear mkbackup (PID 58731) and its descendant processes
Running exit tasks


############################################################################
#  2022/09/24 17:46:32     END
############################################################################

Less then 6 minutes.

fadamo commented at 2022-09-30 12:35:¶

We have also another kind of configuration for which I'm extracting some info:

OUTPUT=ISO
OUTPUT_URL=nfs://NAS/VOL06/LNX/REAR
OUTPUT_OPTIONS="rsize=65536,wsize=65536,tcp,hard"
BACKUP=NETFS
BACKUP_URL=nfs://NAS/VOL06/LNX/REAR
BACKUP_OPTIONS="rsize=65536,wsize=65536,tcp,hard"
BACKUP_PROG=rsync

fadamo commented at 2022-09-30 13:16:¶

OK, try this configuration:

OUTPUT=ISO
OUTPUT_URL=nfs://NAS/VOL06/LNX/REAR
OUTPUT_OPTIONS="rsize=65536,wsize=65536,tcp,hard"
BACKUP=NETFS
BACKUP_URL=nfs://NAS/VOL06/LNX/REAR
BACKUP_OPTIONS="rsize=65536,wsize=65536,tcp,hard"
BACKUP_PROG=rsync
NETFS_KEEP_OLD_BACKUP_COPY=y

Modify these files:

[root@server2backup:~]# cat /usr/share/rear/output/default/150_save_copy_of_prefix_dir.sh  
# if KEEP_OLD_OUTPUT_COPY is not empty then move old OUTPUT_PREFIX directory to OUTPUT_PREFIX.old

[ -z "${KEEP_OLD_OUTPUT_COPY}" ] && return

# do not do this for tapes and special attention for file:///path
local scheme=$( url_scheme $OUTPUT_URL )
local path=$( url_path $OUTPUT_URL )

# if filesystem access to url is unsupported return silently (e.g. scheme tape)
scheme_supports_filesystem $scheme || return 0

local opath=$( output_path $scheme $path )

# an old lockfile from a previous run not cleaned up by output is possible
[[ -f ${opath}/.lockfile ]] && rm -f ${opath}/.lockfile >&2

#if test -d "${opath}" ; then
#    rm -rf $v "${opath}.old" || Error "Could not remove '${opath}.old'"
#    # below statement was 'cp -af' instead of 'mv -f' (see issue #192)
#    mv -f $v "${opath}" "${opath}.old" || Error "Could not move '${opath}'"
#fi

set -ex
if [ ! -d "${opath}" ]; then
    mkdir -p $v -m 0750 "${opath}"
    rm -Rf "${opath}.old"                # if exists
elif [ -d "${opath}.old" ]; then
    mv "${opath}.old" "${opath}.veryold"
    mv "${opath}" "${opath}.old"
    # better to update an old copy rather then starting from scratch
    mv "${opath}.veryold" "${opath}"
else
    mv "${opath}" "${opath}.old"
    mkdir -p $v -m 0750 "${opath}"
fi
set +ex

# the ${BUILD_DIR}/outputfs/${OUTPUT_PREFIX} will be created by output/default/200_make_prefix_dir.sh

[root@server2backup:/var/log/rear]# cat  /usr/share/rear/output/default/200_make_prefix_dir.sh

# Create $OUTPUT_PREFIX directory.
# The $OUTPUT_PREFIX directory defaults to $HOSTNAME.
#
# This happens usually under a mounted network filesystem share
# e.g. in case of OUTPUT_URL=nfs://NFS.server.IP.address/remote/nfs/share
# but it is also happens for local stuff like OUTPUT_URL=usb:///dev/disk/by-label/REAR-000
#
# Do not do this for tapes and special attention for file:///path
local scheme=$( url_scheme $OUTPUT_URL )
local path=$( url_path $OUTPUT_URL )

# If filesystem access to url is unsupported return silently (e.g. scheme tape)
scheme_supports_filesystem $scheme || return 0

local opath=$( output_path $scheme $path )

# Create $OUTPUT_PREFIX sub-directory:
if [ ! -d "${opath}" ]; then
    mkdir -p $v -m0750 "${opath}" && return
else
    return 0
fi

# A failure to create the $OUTPUT_PREFIX sub-directory is fatal:
Error "Failed to create '$opath' directory for OUTPUT_URL=$OUTPUT_URL"

then run 3 times

time rear -Ddv mkbackup

In my case:

1st:    # No backup present
real    12m8.235s
user    1m34.092s
sys     1m17.382s

2nd:    # Only 1 backup present
real    15m10.871s
user    1m33.559s
sys     1m17.733s

3rd:    # 2 backup presents
real    1m52.441s
user    1m0.933s
sys     0m30.103s

I don't know if it's correct to modify files in /usr/share/rear/output/default, probably it should be better to use something like /usr/share/rear/output/NETFS (or RSYNC). BTW updating an old backup is much more convenient when using rsync.

pcahyna commented at 2022-09-30 13:41:¶

Thank you for the information, and what version of ReaR is this? A log above shows 2.4, is that the case also for the version you are experimenting with?