#769 Issue closed: Fundamental enhancements for the backup¶
Labels: enhancement, fixed / solved / done
jsmeix opened issue at 2016-02-12 10:24:¶
This issue is meant as some kind of generic container
or parent issue of various specific issues to collect the
basic ideas about fundamental enhancements regarding
how the backup and restore of the files is done in rear.
For an explanation about the basic design ideas behind
how curently the backup of the files is done in rear see
https://github.com/rear/rear/issues/733#issuecomment-183205395
(excerpts):
By design ReaR operates on the filesystem level. That means that anything not mounted is out of scope for ReaR. The design goal of ReaR was to take a backup / layout snapshot of a running system
From "anything not mounted is out of scope for ReaR" results
one idea for a fundamental enhancements for the backup:
Add support to also backup files from non-mounted volumes.
Another basic design how curently the backup of the files
is done in rear is:
Only one single backup method can be used at runtime. Rear supports many different backup methods but one must select a single one that is actually used. One cannot backup (and restore) some files with backup method "A" and other files with another backup method "B".
From "one single backup method at runtime" results
another idea for a fundamental enhancements for the backup:
Add support to use multiple backup methods.
schlomo commented at 2016-02-12 11:18:¶
@jsmeix thanks for opening up this issue.
Improving the NETFS Backup¶
IMHO we would be better off to extract the backup functionality from ReaR into a separate Open Source project which deals only with backup and restore.
That way we can stay true to ReaRs goal of doing the "bare metal" part while also delivering value with a new backup solution that can focus only on that part.
Unmounted Volumes¶
This is a real challange because there is usually a very good reason why they are not mounted. Most likely because other systems use those volumes and the filesystem on the volume is not multi-host capable (e.g. a cluster FS).
I don't think that a truly generic solution will come easily. Maybe a long chain of special solutions for well-defined use cases.
Another thought: ReaR operates on block devices, block device layers (partitions, LVM ...) and filesystems. If you can "wrap" other stuff into this format then it would be probably easy to support in ReaR.
Multiple Backup Methods¶
The wish to support more than one backup method is also very interesting. I think the solution for that would be to create a new backup method called "META" which would then in turn use a mapping between files and dirs and backup methods to do the actual work.
If there is a specific user request then I think that this is a good start for somebody who wants to code for ReaR.
jsmeix commented at 2016-02-12 12:49:¶
A possible use case for "Multiple Backup Methods":
Assume a 100GB btrfs with snapshot subvolumes is used for the basic system and a huge 10TB xfs is used for a database application.
Then one may like to backup the btrfs subvolume that contains the currently used basic system with one method (e.g. plain "tar") and the btrfs snapshot subvolumes with another method (e.g. "btrfs send" or whatever is "in vogue" for btrfs snapshot subvolumes) and for the database application a special third-party backup solution is used.
During "rear recover" first the basic system would be restored from the backup.tar.gz, then the restore of btrfs snapshot subvolumes may be just skipped by the admin (because they would be restored only during runtime if actually needed), and finally the third-party backup solution restores the database.
For the third-party backup restore it might be even posible to "chroot" into "/mnt/local" and run the third-party backup solution in the already restored basic system. The advantage would be that the third-party backup restore software does no longer need to be included in the rear recovery system so that using a simple generic rear recovery system also for third-party backup solutions might be possible.
jsmeix commented at 2016-02-12 12:59:¶
@schlomo regarding "extract the backup functionality from ReaR into a separate Open Source project":
I fully agree with that future goal.
I prefer so much to Keep Separated Issues Separated ( KSIS ;-)
@schlomo regarding "specific user request":
Yes. My intent here is only brainstorming and discussion without obligation - in particular no "proactive implementation without actual user/customer request".
didacog commented at 2016-02-12 13:14:¶
@schlomo @jsmeix
Hi, regarding this: "extract the backup functionality from ReaR into a separate Open Source project"
We rely on these features to manage ReaR deployments, we take care of Rescue images and required services and provide store for DR backups also. This could break integration with DRLM if those big changes are not well planned.
schlomo commented at 2016-02-12 13:17:¶
With regard to the 10TB data volume with the 3rd party backup beeing
called via chroot after restoring the base system: I don't see anything
in ReaR that prevents you from doing that with a few lines placed either
in POST_RECOVERY_SCRIPT or dropped into the relevant script directory
within ReaR.
If you have the need, why not simply create a rear-foobar tool that
provides the glue code between the foobar 3rd party backup tool and
ReaR. That package will add some files in /usr/share/rear and
integrate itself into ReaR.
It seems to me that many people don't see the modularized design of ReaR as an asset. It was written like that specifically to make this kind of deeply integrated extensions simple.
@didacog good that you are here and participate in the discssion. Rest assured that we value your opinions highly.
jsmeix commented at 2016-02-12 13:34:¶
@schlomo I was not talking about how to implement something and/or how complicated that could be. Of course because rear is bash scripts each user can always implement his own adaptions and enhancements as he needs. The question here is more what rear might provide ready-made "out of the box" in the future - provided a user really needs it ( and when he also likes to pay for it, I or anybody else could even implement it for him ;-)
@didacog if the backup functionality would be split from ReaR into a separate Open Source project I would expect that all what is needed by the user is to install one more software package (e.g. a RPM package) that provides the backup functionality. Just as it is now: Rear does not provide any actual backup software (e.g "tar").
didacog commented at 2016-02-12 13:43:¶
@jsmeix
True, ReaR does not provide the backup software (tar, bareOS,
Netbackup,...), but provides the required methods to backup and recover
the OS properly with these tools.
Maybe putting this methods outside from ReaR will be more probelmatic
than keep them inside
jsmeix commented at 2016-02-12 15:20:¶
@didacog regarding "probelmatic":
For example I wonder how to get third-party backup restore software
included in the rear recovery system if the backup functionality would
be split from ReaR. Certainly doable but probably "problematic". With
third-party backup software I mean basically all the real backup
solutions that are currently supported by rear (i.e. basically all
except "tar").
On the other hand if the backup functionality would be split from rear it requires a clear and stable interface between rear and the backup software. Such an interface could make it easier for the user to integrate any backup software. Currently that is a bit complicated, see doc/user-guide/10-integrating-external-backup.adoc and perhaps even more important: Currently integrating a backup software means code hacking in rear.
didacog commented at 2016-02-12 15:39:¶
@jsmeix
Ok, your point is keep standard OS backup tools (tar, rsync,..), methods
in rear but extract to external tool the third-party backup solutions?
In this case I understand better your point. I understood to exclude all
methots, including standard linux os backups tools, this was my concern.
didacog commented at 2016-02-12 22:01:¶
@schlomo @jsmeix
Please correct me if i'm wrong regarding to split backup methods to separate project.
The objective will be to keep only the BACKUP=NETFS (tar, rsync) in ReaR and move all other methods to a separate project? this way focusing only in the OS backup/recovery (bare metal) using just the standard OS tools?
If this is the goal I'm totally agree and also think this will reduce complexity.
Sorry for my previous misunderstanding.
gdha commented at 2016-02-13 11:15:¶
@didacog Small Note Nothing will change for the moment, and maybe for a long time if no-one contributes to this or sponsor the new way of integration. I prefer getting a stable rear package for the moment.
didacog commented at 2016-02-13 21:24:¶
@gdha
Of course, I was talking in the context of "ReaR future", i understand
this is not an inmediate goal. I just wanted to give my point of view
about those proposed changes for future.
On the other hand, can take us into account (DRLM Team) for helping to
contribute in those goals in future.
gdha commented at 2017-01-17 12:47:¶
@jsmeix something to discuss at Fosdem
jsmeix commented at 2017-01-17 13:12:¶
Yes!
But I think we can even close it as "fixed" right now
because multiple backup methods is already implemented
and for unmounted volumes there will be the new kind of
"backup" method: BACKUP=BLOCKCLONE from @gozora
https://github.com/rear/rear/issues/1162
which will be in ReaR 2.1 (with probability one ;-)
If something new appears on FOSDEM we could better
open a new separated issue for that.
[Export of Github issue for rear/rear.]