#1030 Issue closed: add support for Borg Backup

Labels: enhancement, waiting for info, fixed / solved / done

phirestalker opened issue at 2016-10-09 02:29:

Borg Backup here: https://borgbackup.readthedocs.io/en/stable/index.html

This backup solution uses data deduplication which is good for people backing up disk images or other large files. Plus it seems to be simple and file based and not disk image based, so would be a great fit as rear would bring bare metal restore capability to a space saving backup solution.

schlomo commented at 2016-10-09 12:12:

Interesting software, would you consider to provide a pull request?

I added an issue to borg to keep track of this: https://github.com/borgbackup/borg/issues/1702

gozora commented at 2016-10-09 18:38:

Really a nice tool!
If you guys are can't find anyone to implement it, I'll gladly do my best and integrate Borg into ReaR.

phirestalker commented at 2016-10-09 19:44:

​What do you mean pull request?​

On Sun, Oct 9, 2016 at 11:38 AM, Vladimir Gozora notifications@github.com
wrote:

Really a nice tool!
If you guys are can't find anyone to implement it, I'll gladly do my best
and integrate Borg into ReaR.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/rear/rear/issues/1030#issuecomment-252503930, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AHIXZ5c48c_vZCZ4k-mHgwGBS-VR-qOaks5qyTSNgaJpZM4KR4eN
.

schlomo commented at 2016-10-09 20:38:

@phirestalker see https://help.github.com/categories/collaborating-with-issues-and-pull-requests/

gozora commented at 2016-10-10 12:42:

Small first success (still far from being perfect though) :-)
Backup/restore using Borg done ...

...
2016-10-10 14:27:36 Disk layout created.
2016-10-10 14:27:36 Including layout/recreate/default/25_verify_mount.sh
2016-10-10 14:27:36 Finished running 'layout/recreate' stage in 5 seconds
2016-10-10 14:27:36 Running 'restore' stage
2016-10-10 14:27:36 Including restore/BORG/default/10_restore_borg.sh
++ export LANG=en_US.UTF-8
++ LANG=en_US.UTF-8
++ export LC_ALL=en_US.UTF-8
++ LC_ALL=en_US.UTF-8
++ borg extract root@node2:/mnt/rear/node1::rear_1
++ locale
++ set +x
2016-10-10 14:30:57 Including restore/default/50_selinux_autorelabel.sh
2016-10-10 14:30:57 Created /.autorelabel file : after reboot SELinux will relab
el all files
...

I'll publish code today or tomorrow on my Github page for more discussion ...

V.

gozora commented at 2016-10-11 18:17:

Hello all,

I have first sample of Borg implementation ready on my Github page. Code is not yet ready for pull request, as I need to solve couple of things first. Any comments and testing are (as always) more than welcome.

Couple of notes if you decide to test:

  1. Borg installation manual recommends to install Borg to /usr/local/bin.
    I however, had some hard times with this location. (not sure what was the reason, but I guess that ssh client have some hard coded paths, which don't cover /usr/local/bin), so either install Borg to /usr/sbin; /usr/bin; /bin; /sbin or create symlink to these locations.
  2. Before using ReaR with Borg, setup ssh keys for user that will be running backup. In other words, issuing following command must work without any password prompts or remote host identity confirmation.
    ssh <borg_user>@<borg_server>
  3. I did test ONLY for client/server scenario over ssh (backup was running from client to remote server over SSH). No manual mounting of remote filesystems or anything else was tested yet.
  4. Borg seems to be sensitive to locales setup. I did just quick and dirty configuration of locales on my test system (SLES11 SP3 :-)), So if you see error during recovery that LANG should be set to certain value, try to setup your locales correctly (can be a bit hard though). I've managed to succeed with en_US.UTF-8 ...
  5. For now, use this code for testing purposes only!!!

And just bit of stats:
I've managed to save 3 backup of base OS (without Xorg stuff) in ~1.5GB
Output statistics made after 3rd backup can be found here.

I'll continue to work on this topic and keep you posted ...

V.

jsmeix commented at 2016-10-12 07:32:

@gozora
regarding "Borg seems to be sensitive to locales setup":

In
https://github.com/rear/rear/compare/master...gozora:rear_borg?diff=unified&name=rear_borg
I noticed in your usr/share/rear/conf/default.conf

# Investigate propper locales setup in ReaR recovery media
COPY_AS_IS_BORG=( /usr/lib/locale/en_US.utf8 /usr/share/locale/en_US )

Side note: Typo: Not "ReaR recovery media"
(cf. https://github.com/rear/rear/issues/1008 and https://github.com/rear/rear/issues/1033)
but either "rear rescue/recovery system" or even
better "Relax-and-Recover rescue/recovery system".

I know about issues with locale when doing ssh
from a remote host onto a rear recovery system.

Basically the locale that you use on the remote host wherefrom
you do things like "ssh root@rear.recovery.system"
is also used for the comands that you run afterwards (via ssh)
on the rear recovery system.

But on the rear recovery system there is no localization support.
The only locale that works on the rear recovery system is
plain 'POSIX' (or 'C'), cf. /usr/sbin/rear

# Make sure that we use only English:
export LC_CTYPE=C LC_ALL=C LANG=C

(I don't know the reason behind why also LC_CTYPE=C is set
because - as far as I know and as far as I experienced - only
"export LC_ALL=C LANG=C" should be sufficient.)

Accordingly the solution for working from remote on a
rear recovery system is to set the POSIX/C locale via

export LC_ALL=POSIX LANG=POSIX

either on the remote system before "ssh root@rear.recovery.system"
or in the rear recovery system before running any other command.
To be on the safe side I would suggest to set the POSIX/C locale
on the remote system before "ssh root@rear.recovery.system"
to ensure also the bash and all what profile/bashrc initially runs
on the rear recovery system is run in the POSIX/C locale.

For the fun of it:
Running commands in a non-POSIX locale on a system where
only POSIX is supported can result arbitrarily nasty issues.
For example very basic commands may badly abort
with a locale-related error from glibc.
Obviously local related issues are (nowadays?) sufficiently
severe to just abort programs ;-)

gozora commented at 2016-10-12 08:35:

Hello @jsmeix,

Thanks for you inputs. I'll correct that misspelling and will try to use new agreed style for naming things.
That locale thing with Borg would be tricky. I guess that Borg requires use of same locales for recovery as was used for backup.
The problem I'v encounter is described here: https://github.com/borgbackup/borg/issues/382
So far I was only able to successfully restore, if my locales were set to en_US.UTF-8 in my Relax-and-Recover rescue/recovery system.
I'll do some more experiments on this topic for sure ..

jsmeix commented at 2016-10-12 10:27:

In general when a software fails in POSIX/C locale
it is usually a bug in that software (an exception
might be things like an editor that is specifically
intended to be only used for e.g. UTF-8 files).

In particular system tools like backup/restore software
must not fails in POSIX/C locale.

No backup/restore software should depend
on localization support.

In particular no backup/restore software should depend
on localization support for UTF-8 - regardless how much
UTF-8 is "usually used nowadays", it must not fail if it is
run in plain traditional POSIX/C locale.

gozora commented at 2016-10-12 10:32:

100% agree, give me some time to gather some more data on this localization issue. Once I've learned a bit more about this topic, I'll ask Borg guys about their point of view.

jsmeix commented at 2016-10-12 10:38:

From what I read in https://github.com/borgbackup/borg/issues/382

File system encoding is "ascii", extracting
non-ascii filenames will not be supported

it seems - at least on first glance - that perhaps
Borg deals wrongly with filenames, cf.
"Non-ASCII characters in file names"
in https://en.opensuse.org/SDB:Plain_Text_versus_Locale
that reads in particular:

For the operating system a filename is a
plain sequence of bytes without any additional
information what characters are meant by
this sequence of bytes

Accordingly I think that backup/restore software should
not treat filenames as text in whatever encoding
but as plain sequences of bytes.

In particular it would be plain wrong when a backup/restore software
would store a filename as text in the encoding that matches the
locale when the backup was run and restore it under a filename
as text in the encoding that matches the locale when the restore
is run.

If backup/restore software would work this way (but I never
tested how e.g. 'tar' behaves under different locales)
then one could get a filename /tmp/binär on the disk
in ISO-8859-15 encoding restored under a filename in
UTF-8 encoding which results a different file name, cf.
"Non-ASCII characters in file names"
in https://en.opensuse.org/SDB:Plain_Text_versus_Locale

gozora commented at 2016-10-12 11:16:

Let's ask Borg guys: https://github.com/borgbackup/borg/issues/1702#issuecomment-253185826

jsmeix commented at 2016-10-12 11:26:

Puuhhh! (that is my native German - in English it is "Phew!")

Good old traditional 'tar' works for any filenames:

According to what I described in
"Non-ASCII characters in file names"
in https://en.opensuse.org/SDB:Plain_Text_versus_Locale
I created /tmp/tartest with two
subdirs /tmp/tartest/iso885915 and /tmp/tartest/utf8

In /tmp/tartest/iso885915 I did in the en_GB.iso885915 locale
echo binär >binär

In /tmp/tartest/utf8 I did in the en_GB.utf8 locale
echo binär >binär

I got this:

$ ls /tmp/tartest/utf8 | od -c
0000000   b   i   n 303 244   r  \n
0000007
$ cat /tmp/tartest/utf8/* | od -c
0000000   b   i   n 303 244   r  \n
0000007
$ ls /tmp/tartest/iso885915 | od -c
0000000   b   i   n 344   r  \n
0000006
$ cat /tmp/tartest/iso885915/* | od -c
0000000   b   i   n 344   r  \n
0000006

Now I created a tar archive under POSIX/C locale:

$ pushd /tmp/tartest
$ tar -cf testy.tar *
$ popd

Let's see what filenames tar has stored:

$ tar -tf /tmp/tartest/testy.tar 
iso885915/
iso885915/bin\344r
utf8/
utf8/bin\303\244r

Now I restore it under POSIX/C locale:

$ mkdir /tmp/restore
$ pushd /tmp/restore
$ tar -xf /tmp/tartest/testy.tar
$popd

Let's see what tar has restored:

$ ls /tmp/restore/utf8 | od -c
0000000   b   i   n 303 244   r  \n
0000007
$ cat /tmp/restore/utf8/* | od -c
0000000   b   i   n 303 244   r  \n
0000007
$ ls /tmp/restore/iso885915 | od -c
0000000   b   i   n 344   r  \n
0000006
$ cat /tmp/restore/iso885915/* | od -c
0000000   b   i   n 344   r  \n
0000006

Summary:

Good old traditional 'tar' works for any filenames
and for any file content.

jsmeix commented at 2016-10-12 12:18:

@gozora
could you test that the above also works with Borg?
I.e. that after restore you got the two files again
with same bytes in their file names as it was before, i.e.
one file with this (hexadecimal) bytes for its name

62 69 6e c3 a4 72

( 'binär' in utf8 encoding)
and one file with that (hexadecimal) bytes for its name

62 69 6e e4 72

( 'binär' in iso885915 encoding)

gozora commented at 2016-10-12 13:04:

@jsmeix will do ...

gozora commented at 2016-10-12 16:26:

Hello @jsmeix,

I've try to reproduce your test with help of Borg best I could (hope I did it right, all this encoding stuff don't feel so natural and smooth to me :-( ).

Here are the results:

Test environment

alfa:(/)(root)# ls /tmp/borgtest/utf8 | od -c
0000000   b   i   n 303 244   r  \n
0000007
alfa:(/)(root)# cat /tmp/borgtest/utf8/* | od -c
0000000   b   i   n 303 244   r  \n
0000007
alfa:(/)(root)# ls /tmp/borgtest/iso885915 | od -c
0000000   b   i   n 344   r  \n
0000006
alfa:(/)(root)# cat /tmp/borgtest/iso885915/* | od -c
0000000   b   i   n 344   r  \n
0000006

initialize Borg repository, make backup and list of backup content

alfa:(/)(root)# borg init -e none root@beta.lan12.sk:/mnt/rear/borg/enc_test 
alfa:(/)(root)# borg create --stats --compression zlib,9 root@beta.lan12.sk:/mnt/rear/borg/enc_test::first /tmp/borgtest

alfa:(/)(root)# borg list root@beta.lan12.sk:/mnt/rear/borg/enc_test::first
drwxr-xr-x root   root          0 Wed, 2016-10-12 17:24:31 tmp/borgtest
drwxr-xr-x root   root          0 Wed, 2016-10-12 17:58:40 tmp/borgtest/iso885915
-rw-r--r-- root   root          6 Wed, 2016-10-12 17:58:40 tmp/borgtest/iso885915/bin?r
drwxr-xr-x root   root          0 Wed, 2016-10-12 17:57:59 tmp/borgtest/utf8
-rw-r--r-- root   root          7 Wed, 2016-10-12 17:57:59 tmp/borgtest/utf8/binär

Restore and compare with original data

alfa:(/)(root)# cd /tmp/restore/
alfa:(/tmp/restore)(root)# borg extract root@beta.lan12.sk:/mnt/rear/borg/enc_test::first

alfa:(/tmp/restore)(root)# ls tmp/borgtest/utf8 | od -c                                                        
0000000   b   i   n 303 244   r  \n
0000007
alfa:(/tmp/restore)(root)# cat tmp/borgtest/utf8/* | od -c
0000000   b   i   n 303 244   r  \n
0000007
alfa:(/tmp/restore)(root)# ls tmp/borgtest/iso885915 | od -c 
0000000   b   i   n 344   r  \n
0000006
alfa:(/tmp/restore)(root)# cat tmp/borgtest/iso885915/* | od -c
0000000   b   i   n 344   r  \n
0000006

alfa:(/tmp/borgtest)(root)# find . -type f -exec hexdump -C {} \;
00000000  62 69 6e e4 72 0a                                 |bin.r.|
00000006
00000000  62 69 6e c3 a4 72 0a                              |bin..r.|
00000007

alfa:(/tmp/restore/tmp/borgtest)(root)# find . -type f -exec hexdump -C {} \;
00000000  62 69 6e e4 72 0a                                 |bin.r.|
00000006
00000000  62 69 6e c3 a4 72 0a                              |bin..r.|
00000007

alfa:(/)(root)# diff -r /tmp/borgtest /tmp/restore/tmp/borgtest; echo $?
0

If I've understood your test intentions correctly, Borg handles different encodings fine.

V.

jsmeix commented at 2016-10-13 14:12:

@gozora
yes, it seems Borg backup and restore work correctly.

You have
one file named 'binär' in utf8 encoding
with content 'binär' in utf8 encoding
(62 69 6e c3 a4 72 in hex values) and
another file named 'binär' in iso885915 encoding
with content 'binär' in iso885915 encoding
(62 69 6e e4 72 in hex values)
and after backup and restore you get exactly
the same (no byte has changed - neither
in the file names nor in the contents).

In particular it seems (hopefully) Borg does not try to
"handle encodings" for file names (or even file contents)
for its actual backup and restore operations.

For the actual backup and restore operations any
backup and restore software must backup file names
(and file contents) as plain sequences of bytes and
restore them as exactly the same plain sequences of bytes.

It would be a different task if a software provides
some kind of "migration" feature where file names
(and perhaps also file contents) could be re-encoded
provided the user explicitly wants that.
But then the user must specify both the source encoding
and the target encoding to get predictable results because
in general it is not possible to autodetect which encoding
is meant for the bytes in a file name or the content bytes
in a so called "plain text" file, cf.
https://en.opensuse.org/SDB:Plain_Text_versus_Locale

gozora commented at 2016-10-13 14:59:

@jsmeix so far I did several backup/restores with Borg as backend and did not observed any anomalies so far. I was browsing Borg issues today, and it looks that Borg guys are really taking close attention to data integrity and testing, so I think that Borg will not be a weak point here.

For that locale issue, I don't have anything new in https://github.com/borgbackup/borg/issues/1702, so I've decided to go for plan B and made some improvements in code.
A quick summary of commit:

  1. Create rear own locales on running system with localedef during prep phase, and add them to Relax-and-Recover rescue/recovery system.
  2. Set LC_ALL=rear.UTF-8 but only for duration of borg restore.

I think that this should have no impact on remaining ReaR flows ...

V.

jsmeix commented at 2016-10-14 07:23:

@gozora
as far as I understand
https://github.com/gozora/rear/commit/c7dc42da35cd1f106829fcc882369265efc3c722
I like your 'plan B' very much because
it is in perfect compliance
with "Dirty hacks welcome" in
https://github.com/rear/rear/wiki/Coding-Style
and
it is a nice initial step how we might add UTF-8 locale
support in general to the rear recovery system:
https://github.com/rear/rear/issues/1035

@gozora
as always I appreciate your continuous valuable
contributions to Relax-and-Recover very much!

gozora commented at 2016-10-14 07:46:

@jsmeix
As always, you are welcome, it is fun working on ReaR.

At the time I was reading Coding-Style, I've skipped "Dirty hacks welcome" section thinking that I'd never attempt to do something like that, and here we are :-). Life is strange sometimes...

I'm planning to do some tests with other distributions over weekend and if all goes fine, create "official" pull request starting of next week. Hopefully there will be more volunteers for testing once the code is in main tree.

jsmeix commented at 2016-10-14 08:20:

When you do your "official pull request" please mention
on what systems it has worked for you (and if you know
about systems where it fails). This would increase
the probability that I can "just blindly accept it"
to almost 100% ;-)

gozora commented at 2016-10-14 08:25:

sure thing ...

jsmeix commented at 2016-10-18 12:08:

With
https://github.com/rear/rear/pull/1037
merged, there is now initial support for BorgBackup
in Relax-and-Recover.

@gozora
many thanks for your (always) valuable contributions
to Relax-and-Recover!

@phirestalker
please provide feedback here how it works for you.
You can still add comments regardless that
this issue is closed.

If there are issues with current support for BorgBackup
in Relax-and-Recover please submit new separated
GitHub issues for each separated issue.

gozora commented at 2016-10-18 12:17:

@jsmeix no problem!

phirestalker commented at 2017-01-02 17:43:

been preoccupied with other stuff. I noticed there were 2 more commits related to borg. I wanted to know if there is somewhere I can find how to configure rear for borg and also if I have to do the initial setup of a repository in borg first.

gozora commented at 2017-01-02 17:47:

You can read some basics here.
No, if all goes fine you don't need to do repository initialization.

phirestalker commented at 2017-01-02 23:56:

I have borg and rear on the same machine, do I still need to mess with ssh? If not which variables can I leave out?

phirestalker commented at 2017-01-03 03:51:

ok I read the make backup script for borg. It looks like I can fix it with a few checks for blank user and host variables. According to the borg documentation if you just use a path with no user and host it will use the local system.

jsmeix commented at 2017-01-03 07:56:

@phirestalker
you wrote that you have "borg and rear on the same machine".
Does this mean the Borg backup is stored on that machine?
If yes, I wonder how the backup could be restored after a
disaster happened on that machine?
I am not a Borg user but as far as I understand
https://borgbackup.readthedocs.io/en/stable/quickstart.html
"using SSH ... is fastest and easiest" to get the Borg backup
safe on another host.

gozora commented at 2017-01-03 09:53:

@phirestalker,
Implementation for Borg was written for one simple scenario: Client - Server over ssh.
As you've already mentioned, it should not be a big problem to adapt the code not to use ssh.

I'd be however interested in your setup. Currently I can imagine 2 kind of configurations:

  1. You have Borg repository on your internal disk, which is not a good idea
  2. You have Borg repository mounted using some external filesystem (NFS, Samba, CIFS ..), in such case you will also need to adapt REQUIRED_PROGS with binaries that will actually mount your external filesystem in ReaR rescue/recovery system.

V.

phirestalker commented at 2017-01-03 15:33:

I have a SMB share mounted for the backup. Could you point me to where to
modify REQUIRED_PROGS?

On Tue, Jan 3, 2017 at 2:53 AM, Vladimir Gozora notifications@github.com
wrote:

@phirestalker https://github.com/phirestalker,
Implementation for Borg was written for one simple scenario: Client -
Server over ssh.
As you've already mentioned, it should not be a big problem to adapt the
code not to use ssh.

I'd be however interested in your setup. Currently I can imagine 2 kind of
configurations:

  1. You have Borg repository on your internal disk, which is not a good
    idea
  2. You have Borg repository mounted using some external filesystem
    (NFS, Samba, CIFS ..), in such case you will also need to adapt
    REQUIRED_PROGS with binaries that will actually mount your external
    filesystem in ReaR rescue/recovery system.

V.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rear/rear/issues/1030#issuecomment-270080078, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AHIXZ_3tpkwKDnn0_4afwS9zBqlLrGgZks5rOhqKgaJpZM4KR4eN
.

gozora commented at 2017-01-03 17:13:

In your local.conf | site.conf ....

Something like:

REQUIRED_PROGS=( $REQUIRED_PROGS smbmount )

leader32 commented at 2017-11-02 11:02:

Trying to recover using ReaR bootable ISO image.
Recover fails with error:
ERROR: Could not list Borg archive
I think there is no python binaries with required libraries for borg

Probably need to include python support to the rear livecd

gozora commented at 2017-11-02 11:25:

@leader32 if you are having trouble with ReaR, please open new issue and include all relevant details that might help us find out what is wrong...

V.

leader32 commented at 2017-11-02 12:19:

@gozora I decided to write here, because this error is relevant to borg

Using COPY_AS_IS_BORG as a workaround:

COPY_AS_IS_BORG=( '/usr/lib64/python3.4*' '/usr/bin/python3*' '/usr/bin/pyvenv*' '/usr/lib/python3.4*' '/usr/lib64/libpython3*' )

List of files consolidated from rpm -qa | egrep 'borg|python34' command, maybe there is no need pyenv binary, but now it works fine for me, and rear recover command suceed.

gozora commented at 2017-11-22 17:16:

Hello @leader32,

Trying to recover using ReaR bootable ISO image.
Recover fails with error:
ERROR: Could not list Borg archive
I think there is no python binaries with required libraries for borg

Probably need to include python support to the rear livecd

Yesterday I've find out why I did not encounter this problem when writing support for Borg.
The thing is, that I've used Borg binaries generated with pyinstaller which seems to contain all the libs needed for Borg run.

V.

[Export of Github issue for rear/rear.]