#1037 PR `merged`: Support for Borg¶

Labels: enhancement, fixed / solved / done

gozora opened issue at 2016-10-15 17:17:¶

Hello all,

This pull request is for issue #1030 (add support for Borg Backup).
It was tested on following distros, with latest stable Borg release 1.0.7:

OS	Release	Kernel	Notes
SLES 11	SP3	3.0.101-0.40-default	LEGACY BOOT
SLES 11	SP3	3.0.76-0.11-default	UEFI BOOT
SLES 12	SP1	3.12.49-11-default	UEFI BOOT, btrfs
CentOS	7.2.1511 (Core)	3.10.0-327.36.2.el7.x86_64	LEGACY BOOT
CentOS	7.2.1511 (Core)	3.10.0-327.18.2.el7.x86_64	UEFI BOOT
Debian GNU/Linux	8.6 (jessie)	3.16.0-4-amd64	UEFI BOOT

I'll update documentation soon, for now here are some hits on setup:

Borg installation manual recommends to install Borg to /usr/local/bin. I'd however recommend to use /usr/sbin instead, due reasons mentioned in point one in comment.
Before using ReaR with Borg, setup ssh keys for user that will be running backup. In other words, issuing following command must work without any password prompts or remote host identity confirmation.
ssh <borg_user>@<borg_server>
I've used following configuration to make Borg work with ReaR:

BACKUP=BORG
OUTPUT=ISO

OUTPUT_URL=nfs://<server_to_store_backups>/mnt/rear/iso
BACKUP_PROG_EXCLUDE=( ${BACKUP_PROG_EXCLUDE[@]} /mnt /test/exclude )

BORG_HOST=<server_to_store_backups>
BORG_USERNAME=root
BORG_REPO=/mnt/rear/borg/$HOSTNAME
BORG_PRUNE_WEEKLY=2

GRUB_RESCUE=n

#BTRFS stuff
REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" snapper chattr lsattr )
COPY_AS_IS=( "${COPY_AS_IS[@]}" /usr/lib/snapper/installation-helper /etc/snapper/config-templates/default )

jsmeix commented at 2016-10-17 08:30:¶

In usr/share/rear/restore/BORG/default/40_restore_backup.sh
I found hardcoded 'mnt/local'.
Nowadays $TARGET_FS_ROOT should be used, cf.
https://github.com/rear/rear/issues/708#issuecomment-184246702
unless there is a good reason why hardcoded 'mnt/local'
must be used in your particular case.

In your documentation you sometimes write "borg"
and sometimes "Borg",
cf. https://github.com/borgbackup/borg that reads:

BorgBackup (short: Borg) is a deduplicating backup program.

gozora commented at 2016-10-17 08:41:¶

Didn't know about TARGET_FS_ROOT... I'll correct it.
Maybe I could update 09-design-concepts.adoc as well?

I've try to used "borg" (small 'b') when referencing command and Borg (capital 'B') everywhere else. I'll review documentation one more time.

Thanks for your inputs.

jsmeix commented at 2016-10-17 10:08:¶

If @gdha does not object, I will merge it tomorrow.

FYI: I missed
https://github.com/rear/rear/blob/master/doc/user-guide/09-design-concepts.adoc#inter-module-communication
because 'mnt/local' was not mentioned at that place and I only
checked all places where 'mnt/local' was already used.

gozora commented at 2016-10-17 10:55:¶

@jsmeix , @gdha
Did you noticed my change in usr/share/rear/build/GNU/Linux/10_copy_as_is.sh ?

> cp $v -r $CONFIG_DIR/. $ROOTFS_DIR/etc/rear/ >&2

< cp $v -r $CONFIG_DIR/* $ROOTFS_DIR/etc/rear/ >&2
< cp $v -r $CONFIG_DIR/.[a-z]* $ROOTFS_DIR/etc/rear/ >&2

Do you think it is safe?

gdha commented at 2016-10-17 10:57:¶

@gozora sure no problem with copying dot files (already did the same for CIFS in 1.19) to initrd rootfs

gdha commented at 2016-10-17 11:00:¶

@gozora Now that you got a fresh integration in your fingertips why don't you update the https://github.com/rear/rear/blob/master/doc/user-guide/10-integrating-external-backup.adoc document?

gozora commented at 2016-10-17 11:35:¶

10-integrating-external-backup.adoc looks fine for me. Unfortunately I've discovered it after coding was done (typical me ...).
Do you have some some specific part that needs to be updated on your mind? Because it looks to be quite accurate to me.

gdha commented at 2016-10-17 11:50:¶

@gozora No, I was wondering if it could be improved then now was the right moment to update it so it makes more sense for (new) developers. However, if it is OK for you then it I'm glad to hear that.

gozora commented at 2016-10-18 19:20:¶

@ThomasWaldmann And one more thanks for all the typos corrections. I'm obviously lost without spellchecker!

jsmeix commented at 2016-10-19 09:00:¶

@gozora
don't worry about a spellchecker - I could read it well without
noticing all those typos - I'm obviously running some kind of
silently working full automated autocorrection in my mind ;-)

gozora commented at 2016-10-19 09:09:¶

@jsmeix :-) I basically don't, but @ThomasWaldmann gave me wake up call, and I realized how much work is actually done by spell checker for me!
Before I didn't even think of how many mistakes I'm doing ...
I need to be more careful.

jsmeix commented at 2016-10-19 10:09:¶

@gozora only a side note:
What I would like to have is an
actually useful "spell checker" for bash
(not only the simple "bash -n" syntax check)
that could detect bugs like

CLEANUP_COMMAND="..."
... || $CLEANUP_C0MMAND

Even with "set -eu" the bug would usually not appear
during runtime (assuming the cleanup is usually not needed).
I know that it is impossible to really validate a bash script
but I would be already happy if I had some basic testing
and get warnings for some usual things that "look suspicious".
Cf. https://github.com/rear/rear/issues/549

gozora commented at 2016-10-19 10:46:¶

@jsmeix heh, this might be tricky one, I guess you are looking for similar functionality like gcc have, which reports unused variable and (static) functions ...

alxgu commented at 2016-10-19 11:13:¶

@jsmeix Maybe have a look on https://github.com/koalaman/shellcheck / https://www.shellcheck.net/

jsmeix commented at 2016-10-19 12:30:¶

@alxgu
many thanks for the hint!

Some time ago I had a look at ShellCheck
but as it is written in Haskell I did not try to
compile it on my own.
But now it seems there are ready-made packages
available for some recent openSUSE systems:

$ osc search ShellCheck | grep ^openSUSE
openSUSE:Backports:SLE-12         ShellCheck
openSUSE:Factory                  ShellCheck
openSUSE:Leap:42.2                ShellCheck
$ osc results -v devel:languages:haskell ShellCheck | grep succeeded
openSUSE_Leap_42.1   x86_64     succeeded
openSUSE_Factory_PowerPC_standard ppc64le    succeeded
openSUSE_Factory_PowerPC_standard ppc64      succeeded
openSUSE_13.2_Ports_ppc ppc64le    succeeded
openSUSE_13.2_Ports_ppc ppc64      succeeded
openSUSE_13.2        i586       succeeded
openSUSE_13.2        x86_64     succeeded
SLE_12_SP1_Backports ppc64le    succeeded
SLE_12_SP1_Backports x86_64     succeeded
SLE_12_Backports     ppc64le    succeeded
SLE_12_Backports     x86_64     succeeded

I will try it out soon...

gozora commented at 2016-10-19 20:25:¶

@ThomasWaldmann maybe one more off topic question. Is there any difference in doing:
borg delete TARGET and rm -rf TARGET ?

ThomasWaldmann commented at 2016-10-19 23:33:¶

yes, it also deletes the .cache/borg/id. see borg delete --help.

[Export of Github issue for rear/rear.]

#1037 PR merged: Support for Borg¶