#2443 PR `merged`: Support for systemd and parallel restore with Data Protector (BACKUP=DP)¶

Labels: enhancement, bug, fixed / solved / done, external tool

sebastian-koehler opened issue at 2020-06-30 03:15:¶

Type: Bug Fix and Enhancement

Impact: High
Without those changes the Data Protector 10.70 and later won't be able to use systemd (over xinetd). Restore performance from BACKUP=DP has been increased by doing parallel restores now.

Reference to related issue (URL):
none

How was this pull request tested?
Applied the changes to a SLES15 SP1 system running rear 2.6.1. Created a new ISO, booted the system and ran through various test cases.

Brief description of the changes in this pull request:
Additional changes to Data Protector branding and general script cleanup
Adjustment required to support systemd-only with Data Protector 10.70 and later in addition to xinetd-only deployments
Parallel restore support from BACKUP=DP

jsmeix commented at 2020-06-30 07:44:¶

@gdha
I dared to assign this pull request also to you
because you did https://github.com/rear/rear/pull/2335

jsmeix commented at 2020-06-30 08:26:¶

@sebastian-koehler
thank you for your ongoing contributions and improvements for BACKUP=DP!

It is much appreciated in particular because we at ReaR upstream usually
do not use third-party backup tools (personally I use only BACKUP=NETFS with tar)
and in particular we usually do not have proprietary third-party backup tools
so we totally depend on contributions form users who use third-party backup tools.

sebastian-koehler commented at 2020-06-30 22:23:¶

Thanks for the valuable feedback. I have implemented and tested the recommendations and this makes everything so much easier. Let me know if you find anything else where I can improve the BACKUP=DP scripts from today's point of view.

jsmeix commented at 2020-07-01 13:45:¶

Via https://github.com/rear/rear/commit/daf35e235d0770c663ff8dba866dddec76586a27
I added an explanatory comment in lib/_input-output-functions.sh
that using the ...IfError functions can result unexpected behaviour in certain cases.

sebastian-koehler commented at 2020-07-06 10:44:¶

Thanks for fixing the scripts. I did the same kind of change to 300_create_dp_restore_fs_list.sh, which was the last one missing for DP.

jsmeix commented at 2020-07-06 10:51:¶

@sebastian-koehler
please tell me when things are OK for you (no rush - take your time)
so that I can merge it.

jsmeix commented at 2020-07-07 06:38:¶

@sebastian-koehler
I am sorry, via
https://github.com/rear/rear/pull/2443/commits/33e477ccf30d60263622a456ad28459cd0aeee37
it happened that I did basically a rewrite of 500_restore_ssc.sh

My initial intent was to not error out here at this late state of "rear recover"
after the backup was restored because error out here would make all
what had worked up to this point useless for the user.

I added to skip that script when when the certificate files already exist in the recreated system.
I assume the certificate files in the ReaR recovery system are same as those on the original system
because of default.conf COPY_AS_IS_DP=( ... /etc/opt/omni/client )
so I think when when the certificate files already exist in the recreated system there is no need
to copy them from the recovery system into the recreated system.

I have a general question related to this:
I would expect that all what belongs to the Data Protector client
(in particular /etc/opt/omni/client) is included in the backup
so that during backup restore all what belongs to the Data Protector client
gets restored into the recreated system so that in particular the
Data Protector client certificate files already exist in the recreated system
after the backup was restored.
So I wonder what the reason is why 500_restore_ssc.sh is needed at all?
I mean when the backup is incomplete there is not much what ReaR could do.

sebastian-koehler commented at 2020-07-07 15:46:¶

First of all, thanks for the additional changes to 500_restore_ssc.sh. I have completed testing, found and fixed a typo preventing proper execution and did a few cosmetic changes. This looks pretty good now and we're ready for the merge.

To answer your question, the client certificate is currently excluded from a Data Protector backup due to security concerns. This means we're not able to restore it using BACKUP=DP. This is something we may change within Data Protector, but this will take some time. The current implementation in ReaR will get the required files from the running client, use it during rear recover and place it on the recovered system as a last step (if not found). This allows us to resume backup and restore operation without manual intervention. Also the current implementation will not replace the certificate in case localhost_key.pem is restored using BACKUP=DP (in the future).

jsmeix commented at 2020-07-08 06:53:¶

@sebastian-koehler
thank you four your explanation why 500_restore_ssc.sh is needed.
Now I do perfectly understand the reasoning behind.

A generic note regarding things like "client certificate" and "security concerns":

By default (i.e. unless explicitly configured by the user)
the ReaR recovery system should be free of secrets.

The default behaviour with BACKUP=DP should be improved so that the user knows
that with BACKUP=DP there will be secrets in his ReaR recovery system
so that he knows that his recovery system ISO image and any recovery medium
needs to be protected against unwanted access.

See in default.conf the places that mention secret e.g. the section about
BACKUP_PROG_CRYPT_ENABLED and BACKUP_PROG_CRYPT_KEY
https://github.com/rear/rear/blob/master/usr/share/rear/conf/default.conf#L1041
and in particular the section about
SSH_FILES and SSH_ROOT_PASSWORD and SSH_UNPROTECTED_PRIVATE_KEYS
https://github.com/rear/rear/blob/master/usr/share/rear/conf/default.conf#L1562

For the reasoning why the recovery system should not contain secrets
see https://github.com/rear/rear/pull/1472#issuecomment-328129375
and https://github.com/rear/rear/pull/1472#issuecomment-328459748
and https://github.com/rear/rear/pull/1472#issuecomment-328461809
that read (excerpts)

In general the ReaR recovery system is intended
to not contain possibly private or confidential data
so that in general no files like /etc/shadow will be copied
from the original system into the recovery system.
I.e. it is the intended behaviour to not have /etc/shadow
in the recovery system.
For ReaR only the backup needs to be protected
against unwanted access but the recovery system ISO image
should be more or less considered to be "world readable".

I forgot to tell about the reason behind why the recovery system
ISO image should not contain private or confidential data
(at least not unless the user explicitly configured something):
As far as I remember the reason behind is that one can burn
the recovery system ISO image onto a CD or DVD and then
one can take that recovery system medium anywhere without
the need to protect it (e.g. against it gets lost).
E.g. the admin could misuse a ReaR recovery system CD
after work as beer coaster in the next pub and he could
even forget it there after his Nth beer but still sleep well ;-)
...
In particular when one has both the ReaR recovery system
plus the backup in the ISO, then it is clear that such an ISO
usually needs to be protected against unwanted access.
Accordingly in this case having /etc/shadow also in the ReaR
recovery system won't make things less secure because
usually /etc/shadow is in the backup anyway.
Except one uses backup encryption ...
then /etc/shadow also in the ReaR recovery system
does make things less secure.

The core and basic design principle for ReaR looks like this:

The rescue system contains metadata but not data.
That means that it contains everything required to recreate
the target system but nothing that goes into the target system.
Therefore the rescue media is not a secret.

The backup system or tool contains all the data
that goes into the target system and therefore
it also contains the secrets of the target system
and must be protected accordingly.

jsmeix commented at 2020-07-08 10:27:¶

With
https://github.com/rear/rear/pull/2443/commits/66f7247859638282549e79ee7686c885f435baff
I tried to explain in default.conf the client certificate files handling
(as far as I imagine how things work with Data Protector Secure Communication)
so that the user is informed ( "is" because users are expected to read default.conf ;-)
cf. https://github.com/rear/rear/pull/2443#issuecomment-655325414

jsmeix commented at 2020-07-08 10:47:¶

@sebastian-koehler
if there are no objections from you
I would like to merge it tomorrow afternoon.

jsmeix commented at 2020-07-09 13:40:¶

@sebastian-koehler
thank you for all your work here!

sebastian-koehler commented at 2020-07-09 13:42:¶

@jsmeix
Thanks for your help! I'm waiting for ReaR 2.7 now. :)

jsmeix commented at 2020-07-09 13:52:¶

@sebastian-koehler FYI:
It is possible to have several ReaR versions in parallel
each one in its own separated directory without conflicts between each other
and without conflicts with a normally installed ReaR version via RPM package.
See the section
"Testing current ReaR upstream GitHub master code" in
https://en.opensuse.org/SDB:Disaster_Recovery

[Export of Github issue for rear/rear.]

#2443 PR merged: Support for systemd and parallel restore with Data Protector (BACKUP=DP)¶