#81 Issue closed: Offer possibility to get consistent backup (regarding applications)

Labels: enhancement, discuss / RFC

dagwieers opened issue at 2012-05-24 12:56:

When making backups using Rear, we need to make sure that running applications or deamons write consistent data out to the file system. This is impossible, except when stopping services this can be guaranteed in all cases.

For this we would like to add the capability to stop and start services before and after running the backup.

We need to make sure that:

  • Only effectively stopped services (those that were running) are being started again
  • All applications are effectively being started (we can guarantee and it's not Rear's problem ! Document this risk !)
  • If Rear fails, make sure we start all services anyway (as exit tasks)

The implementation would use an array as a list of services to stop/start, and by default it would be empty (so this functionality is effectively disabled).

It is the task of the system administrator to decide what applications require this and what the effects are of stopping the service and the resulting downtime.

jsmeix commented at 2012-05-29 11:22:

Currently I use rear-1.10.0 (on a SUSE Linux Enterprise 11 SP1 system) and there I have /usr/share/rear/backup/readme which reads:

00-09: initialization
10-19: mount NETFS
20-29: create prefix dir, etc.
30-39: specials like SELinux (stop)
40-49: create include/exclude files for backup
50-59: backup itself
60-69: specials like SELinux (start)
70-79: umount NETFS
80-89:
90-99:

Accordingly I have in particular those files /usr/share/rear/backup/NETFS/GNU/Linux/30_stop_selinux.sh /usr/share/rear/backup/NETFS/GNU/Linux/60_start_selinux.sh

I.e. there is already something implemented at the right time to stop something before backup and re-start it after backup.

I suggest to only enhance this as follows:

Provide two directories where executables (usually bash scripts) can be placed by the system admin which are run before the backup and after the backup.

Why directories?

This way the system admin (or even several system admins) can maintain separated executables to stop and re-start
separarted services.

Separated scripts for separated services should be run by default simultaneously to speed up the overall time untill all services are stopped and re-started.

If different services depend on each other the system admin must implement in his scripts that one waits until another one has completed.

Perhaps ReaR could provide functionality to enforce running them one after the other only if the executable file names have leading two-digit numbers and all executables with none or same numbers are launched simultaneously.

It may take a longer time until a service can be stopped in a reasonable/friendly way in the running system.

Examples:

  • One would let the printing service finish the printout of all currently printing jobs before the printing service is stopped.
  • Imagine at 3 o'clock in the night whatever process had submitted a 200 pages job which is now printing at page 190 when the printing service is suddenly stopped. After re-start of the printing service it would re-print the whole 200 pages job from the beginning.
  • One would let the mail service finish to receive all mails which are currently received before it is stopped.
  • Imagine at 3 o'clock in the night from somewhere a mail with a huge 5GB data attachment arrives where right now 4GB was already received when suddenly the mail service is stopped.

In such cases it would be good if the scripts for the printing and mail service would run by default simultaneously (and also wait simultaneosly until the 200 pages print job finished and the huge mail was completely received).

I think it belongs to the system admin to make appropriate scripts to stop and re-start his particular services in a way which is reasonable in his particular environment. In particular only the system admin could know how long to wait for a service to complete its current active tasks before the service is forced to stop. Think about an active printing 200 pages job at 3 o'clock in the night where the printer runs out of paper at page 190.

I think all what ReaR should provide are those directories and the functionality to run all executables therein but not the actual executables therein.

Perhaps ReaR might provide example scripts as documentation but no actual executables in those directories.

baccenfutter commented at 2012-06-03 14:21:

How about this approach?

We introduce a new directory /etc/rear/services. In here the admin can place init-scripts. He can either simply symlink his local /etc/init.d/* init-scripts, or write wrappers to these containing any additionally required logic. I believe, most logic for determining the local init-daemon already exists within ReaR, thus all we would have to do is trigger the init-scripts in the according shell-syntax (e.g. /etc/rear/services/service start or service service start, etc).

The big advantages I see, is that we would be able to leverage init-script logic, like start-stop dependencies, start-stop-status shell-syntax and such, while keeping all these complexities out of rear. I also believe, that this would be a very intuitive abstraction layer to the user, making it easy to add and remove backup-safe services for any particular execution of rear. Writing a separate rear script for every service one wants to be backup-safe would be (in many cases) reduced to ln -s /etc/init.d/service /etc/rear/services. In case one does require very special logic while stopping and starting services, he can simply provide a wrapper to any of his init-scripts or of course any stand-alone script, as well.

Additionally we could limit rear to only call start/stop on files in /etc/rear/services that are set +x.

dagwieers commented at 2012-06-03 20:46:

Let me clarify the two main opinions in the development team.

  1. We do not want to make Rear more distribution specific and making it harder to support Rear correctly. Rear should be a basic tool doing the basics right. More specific detailed functionality that's not widely needed should be added by end-users/administrators and Rear is designed to be modular and extendable... The solution is to make use of pre-backup and post-backup scripts and leave it up to the administrator to define what is needed.
  2. This functionality is something that administrators should not have to invent on a case-by-case basis and we can make it easier for everybody if we can provide it for everyone to use. Not enabled by default, but integrated with little need to customize by making use of the distribution's way of starting and stopping services. It is up to the administrator to decide what services need this special treatment.

Both views make sense. Nothing has been decided, however one thing is clear. At this point this is not a priority, so unless someone is doing the implementation or sponsoring an implementation this is unlikely to happen in the short term. However, the ability to hook scripts into different parts of Rear is more generally useful and is therefor more likely to see the light of day. But that doesn't mean we cannot have both at some point.

jsmeix commented at 2012-07-03 12:57:

Regarding SysVinit scripts and "leverage init-script logic" and any kind of "abstraction layer":

Nowadays there is no longer only SysVinit but also systemd so that there is no longer a single generic way how to stop and re-start services.

This means it becomes complicated if Rear would try to provide some kind of ready-to-use functionality to stop and re-start services.

gdha commented at 2013-06-03 09:53:

I agree with statement of @jsmeix - it is not the task of rear to stop/start services.
Remember everything within rear is a script, so anybody can write a small script (and drop it somewhere in prep directory) to avoid interruptions of critical services.
I gonna close this case - it can always be re-opened if there is an urgent need for it.

[Export of Github issue for rear/rear.]