#974 Issue `closed`: No Real Incremental Backups¶

Labels: enhancement, documentation, support / question, fixed / solved / done

dcz01 opened issue at 2016-08-22 07:50:¶

rear version (/usr/sbin/rear -V): 1.18 / Git
OS version (cat /etc/rear/os.conf or lsb_release -a):

OS_VENDOR=RedHatEnterpriseServer
OS_VERSION=6

rear configuration files (cat /etc/rear/site.conf or cat /etc/rear/local.conf):

OUTPUT=ISO
OUTPUT_URL=file:///opt/brp/rear
BACKUP=NETFS
BACKUP_PROG=tar
#BACKUP_PROG_CRYPT_ENABLED=1
#BACKUP_PROG_CRYPT_KEY=brpbackup123
#BACKUP_PROG_CRYPT_OPTIONS="/usr/bin/openssl aes256 -salt -k"
#BACKUP_PROG_DECRYPT_OPTIONS="/usr/bin/openssl aes256 -d -k"
#BACKUP_URL=nfs://
BACKUP_URL=cifs://notesrechte/BRP-Backup
BACKUP_OPTIONS="cred=/etc/rear/cifs"
BACKUP_TYPE=incremental
FULLBACKUPDAY="Sat"
BACKUP_PROG_EXCLUDE=( '/tmp/*' '/dev/shm/*' $VAR_DIR/output/\* '/mnt/*' '/media/*' '/var/lib/pgsql/9.3/data/base/*' '/var/lib/pgsql/9.3/data/global/*' '/var/lib/pgsql/9.3/data/pg*/*' )

I found a little bug in the incremental solution of rear.
rear is actual working with differential backups based on the last full backup. so it needs only to restore the last full and the newest differential.
this is not the right incremental backup and restore because rear is so backing up every day a lot more than needed.
is this so intended?

because we are backing up every day an postgres database (which does an self full backup every day) so that the backup every day gets to large to transmit it from the client to the data center.

here i found some info about that if needed:
http://typesofbackup.com/incremental-vs-differential-vs-full-backup/
http://paulwhippconsulting.com/blog/using-tar-for-full-and-incremental-backups/

dcz01 commented at 2016-08-30 12:54:¶

@gdha @jsmeix Or maybe you could change the comparison in the file /usr/share/rear/prep/NETFS/default/07_set_backup_archive.sh from

if [ -f "${BUILD_DIR}/outputfs/${NETFS_PREFIX}/timestamp.txt" ]; then
backuparchive="${BUILD_DIR}/outputfs/${NETFS_PREFIX}/$(date +"%Y-%m-%d-%H%M")-I${BACKUP_PROG_SUFFIX}${BACKUP_PROG_COMPRESS_SUFFIX}"
BACKUP_PROG_X_OPTIONS="$BACKUP_PROG_X_OPTIONS --newer=$(cat ${BUILD_DIR}/outputfs/${NETFS_PREFIX}/timestamp.txt)"
BACKUP_PROG_X_OPTIONS="$BACKUP_PROG_X_OPTIONS -V $(cat ${BUILD_DIR}/outputfs/${NETFS_PREFIX}/basebackup.txt)"
Log "Performing Incremental-Backup $backuparchive"

to something of that to backup only files newer than the last backup date +%Y-%m-%d -d yesterday --newer-mtime '1 days ago'

gdha commented at 2016-09-22 09:19:¶

@dcz01 The code is now made that it only needs to restore one full backup and one incremental backup (an incremental since last full backup - it would perhaps be better if we called it differential instead of incremental).
We have no intention to change this in the near future.

jsmeix commented at 2016-09-22 10:21:¶

@dcz01
in your https://github.com/rear/rear/issues/974#issue-172388615
I noticed "we are backing up every day an postgres database".

FYI regarding huge backup because it contains huge files
like databases you may have a look at my personal opinion in
https://github.com/rear/rear/issues/1006#issuecomment-248862040

jsmeix commented at 2016-09-22 10:49:¶

@gdha
regarding your "it would perhaps be better if we called
it differential instead of incremental)":
Do you intend to enhance the documentation?

If not I could try to to it.

If I do it I would not change any code
(e.g. I would keep things like "BACKUP_TYPE=incremental")
but I would try to make it clear in the documentation
that "incremental" actually is "differential".

Or should I even try to also adapt the code e.g.
change "BACKUP_TYPE=incremental"
to "BACKUP_TYPE=differential"
or - if needed - keep it backward compatible
that BACKUP_TYPE could be "incremental" or "differential"
and both lead to the same result?

jsmeix commented at 2016-09-22 12:53:¶

I have a plan:

I will only enhance the documentation to make it clear
that currently "BACKUP_TYPE=incremental" functionality
actually implements differential backup behaviour.

This way the "BACKUP_TYPE=incremental" functionality
can be enhanced at a later time by a contributor to actually
implement real incremental backup behaviour.

jsmeix commented at 2016-09-22 14:04:¶

Have a look at my pull request https://github.com/rear/rear/pull/1007
whether or not it is o.k. for you this way.

jsmeix commented at 2016-09-23 07:55:¶

With https://github.com/rear/rear/issues/974#issuecomment-248851517
plus the merged https://github.com/rear/rear/pull/1007
I close this issue as "fixed".

@dcz01
That "we have no intention to change this in the near future"
does of course not mean that another contributor (e.g. you)
could not enhance the current "BACKUP_TYPE=incremental"
functionality to actually implement real incremental backup.

FYI:
In general regarding contributing to Relax-and-Recover see
http://relax-and-recover.org/development/
and the section
"How to contribute to Relax-and-Recover" in
https://en.opensuse.org/SDB:Disaster_Recovery

jsmeix commented at 2016-09-23 08:27:¶

Typo correction in my https://github.com/rear/rear/issues/974#issuecomment-249126761
I meant:

That "we have no intention to change this in the near future"
does of course not mean that another contributor (e.g. you)
could not enhance the current "BACKUP_TYPE=incremental"
functionality to actually implement real incremental backup.

I.e. of course we do appreciate GitHub pull requests
that enhance Relax-and-Recover functionality.

jsmeix commented at 2016-11-21 12:21:¶

With
https://github.com/rear/rear/pull/1071
merged, there is now support for multiple restore archives
which is a precondition to support real incremental backups
so that there is now support for real incremantal backups
and still support for differential backups.

There is a small behavioural change:
Now BACKUP_TYPE=incremental means real incremental backup
and the new BACKUP_TYPE=differential means what it tells.
This is all described in default.conf

With that above pull request merged
I consider this issue to be really fixed now.

[Export of Github issue for rear/rear.]

#974 Issue closed: No Real Incremental Backups¶