#1464 PR merged: Implement new YUM backup method (based on ZYPPER)¶
Labels: enhancement, fixed / solved / done
N3WWN opened issue at 2017-08-31 18:11:¶
Hi!
This is my first major commit to this project, so I hope that I'm doing this correctly.
This PR is to add a YUM backup method which will recreate a RPM-based system that uses the yum package manager in the same manner as the ZYPPER backup method.
I symlinked files where possible and copied and updated files when changes were necessary for the YUM method.
I have also updated a couple of existing files to [hopefully] improve them. The primary changes which could likely be made common to all methods are:
-
usr/share/rear/rescue/default/500_ssh.sh
- improved detection of the ssh user and don't duplicate an existing user
-
usr/share/rear/rescue/default/900_clone_users_and_groups.sh
- improve extraction of users and groups and don't duplicate an existing user
-
usr/share/rear/restore/YUM/default/405_clone_users_and_groups.sh
- clone the users and groups only if the same user or group does not exist. Previously, the entire line had to match, resulting in duplicate entries if, for instance, the UID differed for the same user.
-
usr/share/rear/restore/YUM/default/940_generate_fstab.sh
- improve detection of existing fstab entries to avoid duplication
If you'd prefer that I submit separate PRs for each, please let me know.
Upon acceptance of the YUM method, I will submit another PR which adds a file backup option to the YUM method. This will recreate the system from RPM and restore any non-RPM-originated files or RPM-originated files which may have changed (or been removed).
Let me know if there's a better way for me to submit changes of this type or if there's anything I can do to help with the acceptance of this PR.
Thanks!
-Rich Alloway (Rogue Wave)
jsmeix commented at 2017-09-01 12:00:¶
@N3WWN
wow!
Many thanks for that!
Because I made the current (rather initial and experimental)
support for 'ZYPPER' I assign your pull request to me.
I added the other ReaR upstream members as reviewers
in particular because your pull request also contains
changes in other general files.
Regarding your other PR which adds a file backup option
to the YUM method:
I look forward to seeing it because this is what I
also like to implement for the 'ZYPPER' method.
For me a main purpose of restoring files from a backup
after the initial RPM installation is to provide ready-made
config-files so that afterwards the system is ready-to-use.
This method is also done by the KIWI image installation, cf.
https://en.wikipedia.org/wiki/KIWI_%28openSUSE%29
After the initial KIWI image installation a tar.gz archive
with ready-mde config-files (or whatever other files)
can get installed (optionally).
jsmeix commented at 2017-09-01 12:36:¶
@N3WWN
I noticed a few
Debug "WARNING: ..."
I wonder if a warning as a debug message makes much sense.
I think - if at all - warnings are meant directly for the user?
For background information about warnings in general see
http://blog.schlomo.schapiro.org/2015/04/warning-is-waste-of-my-time.html
jsmeix commented at 2017-09-01 12:49:¶
@N3WWN
in your restore/YUM/default/405_clone_users_and_groups.sh
In general instead of things like
if [[ "$CLONE_ALL_USERS_GROUPS" =~ ^[yY1] ]]; then
use
if is_true "$CLONE_ALL_USERS_GROUPS" ; then
cf. "Relax-and-Recover functions" at
https://github.com/rear/rear/wiki/Coding-Style
For me it looks af if you misunderstood or misuse how
CLONE_USERS CLONE_GROUPS CLONE_ALL_USERS_GROUPS
are meant to be used.
As far as I understand it those are meant to copy
users and groups from the original system
where "rear mkrescue/mkbackup" runs into the
ReaR recovery system but those variables are not meant
to copy users and groups during "rear recover"
into the recreated system.
Or I misunderstand how your
restore/YUM/default/405_clone_users_and_groups.sh
is meant in your use-case.
jsmeix commented at 2017-09-01 12:55:¶
Nice!
This way it became clear that
rescue/default/900_clone_users_and_groups.sh
(last actual code change dated 2013)
also needs some code cleanup.
In particular the "WARNING" messages therein
had been implemented in 2009 by @schlomo himself ;-)
provided
git log -p --follow usr/share/rear/rescue/default/900_clone_users_and_groups.sh
tells the truth.
jsmeix commented at 2017-09-01 13:05:¶
@N3WWN
you wrote
YUM backup method which will recreate a RPM-based system that uses the yum package manager in the same manner as the ZYPPER backup method
The "recreate" therein might be a misunderstanding about the
actual primary idea behind the ZYPPER "backup" method:
See
https://github.com/rear/rear/issues/1085
that reads (excerpt)
the new BACKUP=ZYPPER method is intended to ... "install a system from scratch" . . . also use ReaR generically for all kind of installations . . . ReaR system deployment
and
https://hackweek.suse.com/15/projects/2242
and
https://en.opensuse.org/SDB:Disaster_Recovery#Using_ReaR_as_generic_installer_in_any_installation_system
N3WWN commented at 2017-09-01 15:40:¶
@jsmeix
Looks like I can't respond to each msg, so I'll have to do so all at once :)
Regarding your other PR which adds a file backup option
to the YUM method:
I look forward to seeing it because this is what I
also like to implement for the 'ZYPPER' method.
For me a main purpose of restoring files from a backup
after the initial RPM installation is to provide ready-made
config-files so that afterwards the system is ready-to-use.
This method is also done by the KIWI image installation
Though I wasn't aware of KIWI before now, I could definitely see this as an option using BACKUP_PROG_INCLUDE/BACKUP_PROG_EXCLUDE!
My goal was to be able to have as small of a backup ISO as possible but with a complete system restore. The versions of the packages may change since we're installing from YUM repos, but I haven't seen it as a problem with the CentOS 6.x and CentOS 7.x testing that I've done, purposely using older 6.x and 7.x VMs that were restored with the latest RPMs from the public repos.
I make VMs a LOT for supporting our customers and would like to be able to take a full system backup without storing all of the files that are the same as what is in the RPMs so that I can destroy the VM and have a minimal-size backup. I used to use ReaR to make full backups:
OUTPUT=ISO
BACKUP=NETFS
BACKUP_PROG=tar
BACKUP_URL=iso://backup
OUTPUT_URL=null
That method took 2.7GB of dynamic disk and turned it into about a 1.4GB ISO.
The same 2.7GB of dynamic disk, but with BACKUP=YUM is now a 210MB ISO. :)
Debug "WARNING: ..."
Those were in the original files that I modified, but can certainly strip them out or alter them to be included in the log only.
We can keep the history of those Debug messages between ourselves. ;)
CLONE_ALL_USERS_GROUPS
The if statement there was also originally from the code I cloned out
of the ReaR repo
(usr/share/rear/rescue/default/900_clone_users_and_groups.sh), but
can adjust it to use is_true.
I saw that the users/groups were being cloned into the rescue system and just carried that forward to cloning from the rescue system into the restore system.
My reasoning being that, since the system was installed from scratch via RPM, which causes the user and group IDs sometimes change (they did in my testing), the files will be restored with the wrong UID and/or GID during the file restore (which will be included a later PR). Instead of completely reinventing the wheel, I just painted it a different color. :)
YUM backup method which will recreate a RPM-based system
that uses the yum package manager in the same manner
as the ZYPPER backup method
Would it be better to say
YUM backup method which will recreate a RPM-based system that uses the yum package manager in a _similar_ manner as the ZYPPER backup method?
Or perhaps
YUM backup method will install a system from scratch in a _similar_ manner as the ZYPPER backup method, but utilizing the yum package manager?
As it sits now, with the exception of the cloning of users and groups, I think that the YUM method is behaving the same as the ZYPPER method (but without the repository specification), but I may be missing something as I've not used ZYPPER itself very much.
Thanks for all of the feedback so far! I'm excited to contribute and am thrilled that you seem to be equally excited about my contribution! :)
Just let me know how you'd like me to proceed...
jsmeix commented at 2017-09-04 15:55:¶
@N3WWN
many thanks for your descriptions.
Your use case (get the backup smaller) is very interesting!
As YUM and ZYPPER backup methods work basically same
(o.k. - I need to add backup support to ZYPPER)
your use-case also applies for ZYPPER.
I think we should keep how YUM and ZYPPER work the same
as much as we can because I do not see any reason why
they should behave really different for the user.
Regarding clone_users_and_groups I did right now
https://github.com/rear/rear/pull/1471
which will conflict with your changes in
usr/share/rear/rescue/default/900_clone_users_and_groups.sh
I would appreciate it if you have a look if
https://github.com/rear/rear/pull/1471
works o.k. for you or if I did something wrong.
Regarding using the CLONE_USERS CLONE_GROUPS
CLONE_ALL_USERS_GROUPS also to clone them
from the recovery system into the recreated system:
I do not like that duplicated usage.
I would very much prefer to keep separated things separated
(cf. RFC 1925 item 5:
https://www.ietf.org/rfc/rfc1925.txt
).
In this case I think all what is needed is a new config variable
e.g. named RECREATE_USERS_GROUPS=yes/no
so that the user can specify whether or not he really
wants to have all the users and groups from the
recovery system also in the recreated system.
jsmeix commented at 2017-09-05 09:54:¶
@N3WWN
I think we can move forward a lot
if you could split this pull request into parts:
One main part with all what is strictly additional
(i.e. what cannot change any existing behaviour)
which is - as far as I see - all except your changes in
usr/share/rear/rescue/default/500_ssh.sh
and
usr/share/rear/rescue/default/900_clone_users_and_groups.sh
This main part only adds the YUM backup method
and I can and would like to "just merge" it as is
because it is your code and you tested it and
I assume in case of issues with the YUM backup method
you would fix issues there (i.e. you maintain your own code).
The rest which might possibly change any existing behaviour
should be in separated pull requests i.e. your changes in
usr/share/rear/rescue/default/500_ssh.sh
and
usr/share/rear/rescue/default/900_clone_users_and_groups.sh
because changes in existing scripts can cause
some discussion until all is clear.
Because I assume your changes in
usr/share/rear/rescue/default/500_ssh.sh
are independent of your changes in
usr/share/rear/rescue/default/900_clone_users_and_groups.sh
I would recommend to do two additional and separated
new pull requests for them.
It seems your changes in
usr/share/rear/rescue/default/900_clone_users_and_groups.sh
would be mostly obsoleted when I merge my
https://github.com/rear/rear/pull/1471
and if not you may prefer to do a follow-up pull request later
based on my
https://github.com/rear/rear/pull/1471
jsmeix commented at 2017-09-05 10:16:¶
@N3WWN
mostly out of curiosity:
I do not yet understand the actual reason why you need
this double cloning of users/gropus.
You wrote that your reasoning is "that, since the system
was installed from scratch via RPM, which causes the
user and group IDs sometimes change".
I understand this as if user and group IDs sometimes change
in /etc/passwd and /etc/group after RPM installation
and with your double cloning of users/gropus you fix that.
But I would expect that basically all in /etc/ is in the backup
so that after backup restore you should have the right
/etc/passwd and /etc/group - or what do I misunderstand?
N3WWN commented at 2017-09-05 19:25:¶
@jsmeix
I was wondering if I should have broken up the PRs into much smaller bites, and I guess I should have. ;)
I believe that we should be able to add file backups to ZYPPER by symlinking to the appropriate files in the YUM tree.
IIRC, I kept the file backups and RPM backups segregated completely since I planned on separate PRs for those two segments of the feature.
The changes to usr/share/rear/rescue/default/900_clone_users_and_groups.sh look good and appear to address the issues I saw with duplicate users and groups.
Regarding using the CLONE_USERS CLONE_GROUPS
CLONE_ALL_USERS_GROUPS also to clone them
from the recovery system into the recreated system:
I do not like that duplicated usage.
I would very much prefer to keep separated things separated
For the new configuration item, RECREATE_USERS_GROUPS=yes/no, do you think that would be best set up per backup method? i.e. YUM_RECREATE_USERS_GROUPS and ZYPPER_RECREATE_USERS_GROUPS?
Would it be appropriate to duplicate the code from usr/share/rear/rescue/default/900_clone_users_and_groups.sh in usr/share/rear/restore/YUM/default/405_clone_users_and_groups.sh (basically as it is now) but using the new RECREATE_USERS_GROUPS config item?
Or would it be preferred that this code be extracted as functions in a library?
Duplicating the code will allow us to more easily merge the main YUM feature code as it won't touch any existing code, but, by definition, duplicates code in a project.
I assume in case of issues with the YUM backup method
you would fix issues there (i.e. you maintain your own code).
Yes, not a problem at all. I'm totally on-board with ongoing support of the changes.
Do I need to "unpull" #1464 or anything to break it into 3 distinct PRs (YUM, usr/share/rear/rescue/default/500_ssh.sh and usr/share/rear/rescue/default/900_clone_users_and_groups.sh)?
N3WWN commented at 2017-09-05 20:04:¶
@jsmeix
mostly out of curiosity:
I do not yet understand the actual reason why you need
this double cloning of users/gropus.
You wrote that your reasoning is "that, since the system
was installed from scratch via RPM, which causes the
user and group IDs sometimes change".
I understand this as if user and group IDs sometimes change
in /etc/passwd and /etc/group after RPM installation
and with your double cloning of users/gropus you fix that.
But I would expect that basically all in /etc/ is in the backup
so that after backup restore you should have the right
/etc/passwd and /etc/group - or what do I misunderstand?
The issues derived from how/where user IDs and group IDs are stored and how they differ on the three systems involved: the source system, the rescue/recovery system and the target system.
Let's say that we have 2 system users sshd and postfix (users randomly picked, this issue may not show up with these particular users, especially since these users may have predefined UIDs on many systems).
On the source system, postfix was installed with the OS and ssh was
added later, so let's say that postfix has a UID of 80 and sshd has a
UID of 90. From the tar info pertaining to the --same-owner parameter:
When writing an archive, 'tar' writes the user ID and user name
separately. If it can't find a user name (because the user ID is
not in '/etc/passwd'), then it does not write one. When restoring,
it tries to look the name (if one was written) up in '/etc/passwd'.
If it fails, then it uses the user ID stored in the archive
instead.
This means the archive contains stuff like /etc/ssh/sshd_config with a user of sshd and a UID of 90 and /var/lib/postfix/master.lock with a user of postfix and a UID of 80.
On the target system, we reinstall the RPMs and the UIDs no longer match... let's say sshd is now UID 80 and postfix is now UID 90.
We perform the file restore on the rescue/recovery system, which thinks these users have their original IDs, funny things can happen, even if we chroot into the target system to do the file restore.
I fought with this for days; only a few files would have incorrect ownership and/or SELinux contexts after an otherwise successful restore. Usually, the system would be usable, but I just wasn't content with having the differences.
rear/usr/share/rear/restore/YUM/default/405_clone_users_and_groups.sh actually excludes /etc/passwd, /etc/group and /etc/shadow from being restored. We still include them in the archive so we can see the contents later if we need to reference them, though.
The reason for excluding them from the restore goes along with these other issues... if any UID or GID change or are otherwise different between the source and target systems, the UID and GID on the target system disk won't match the UID and/or GID specified in the files in /etc.
Let me know if my rambling is making sense or if I need to dig up the exact circumstances where the issues arose.
jsmeix commented at 2017-09-06 09:48:¶
To get your YUM backup method merged, just leave out the
usr/share/rear/rescue/default/500_ssh.sh
and
usr/share/rear/rescue/default/900_clone_users_and_groups.sh
files from this pull request.
I think (but not tested) it should work when you copy the current
ones from current ReaR upstream over your changed ones
in your git checkout branch for this pull request
so that there are no longer changes in those files and then
redo a "git push ..." which updates this pull request.
Regarding symlinking YUM and ZYPPER scripts:
I would prefer to wait with symlinking YUM and ZYPPER scripts
until both implementations stabilized because currently
both are unfinished and work-in-progress so that I would prefer
that you can "just change what you like" in your YUM scripts
and I can "just change what I like" in my ZYPPER scripts
(i.e. so that we both are free in what we develop).
I am curious what I can learn from your YUM scripts
that is also useful for my ZYPPER scripts.
Later when we found out which scripts are really same
and work stable we can link them where we must decide
who becomes the main-maintainer of a script (i.e. the
one who maintains the actual script - not the symlink).
Regarding "double-clone-users-and-groups":
Many thanks for your descriptive and explanatory
https://github.com/rear/rear/pull/1464#issuecomment-327287675
where you describe that your "double-clone-users-and-groups"
functionality is a generically useful functionality.
If I understand it correctly wrong numerical user IDs
and/or wrong numerical group IDs might even happen with
any backup restore software. Not only with 'tar' and not limited
to a particular backup method like YUM or ZYPPER because
any backup restore software runs "inside" the ReaR recovery
system with the users and groups there and not with the
users and groups of the original system.
Accordingly - as far as I see probably even by default - the
ReaR recovery system should contain all users and groups
of the original system to ensure any backup restore software
can restore the files with the right user names and IDs
and group names and IDs.
This would mean another general enhancement of the current
rescue/default/900_clone_users_and_groups.sh
functionality.
In particular for the backup methods YUM and ZYPPER
your "double-clone-users-and-groups" functionality
is additionally needed but I do not see why that
"double-clone-users-and-groups" functionality must be
limited to only the backup methods YUM and ZYPPER.
I would implement a generic RECREATE_USERS_GROUPS
functionality that must be by default disabled in default.conf
to avoid behavioural changes (i.e. stay backward compatible).
I would recommend to implement RECREATE_USERS_GROUPS
as a bash array to be prepared for future enhancements,
cf. how the MODULES and FIRMWARE_FILES arrays work.
Instead of restore/YUM/default/405_clone_users_and_groups.sh
use e.g. restore/default/395_recreate_users_and_groups.sh
that starts with something like:
# 395_recreate_users_and_groups.sh
#
# Recreate the same users and groups that exist
# in the ReaR recovery system also in the target system.
# Skip if not needed.
# For the 'test' one must have all array members
# as a single word i.e. "${name[*]}" because
# it should succeed when there is any non-empty
# array member, not necessarily the first one:
test "${RECREATE_USERS_GROUPS[*]}" || return
I think the script number must be less than the script number
of the script that actually restores the backup which is
usually 400_restore_clone.sh because otherwise
"Excluding group, password and shadow files from restore"
cannot work because that must happen before the backup
gets restored.
Check the "rear -s recover" output in which ordering the
scripts will be actually sourced (i.e. run) in your case.
Alternatively if NNN_recreate_users_and_groups.sh
should run after the backup was restored
(e.g. to ensure that the backup restore cannot overwrite
etc/passwd etc/group etc/shadow that was writen by
the NNN_recreate_users_and_groups.sh script)
you can use BACKUP_RESTORE_MOVE_AWAY
to move the restored etc/passwd etc/group etc/shadow
files away, see its description in default.conf
but note that the BACKUP_RESTORE_MOVE_AWAY
functionality is run very late in the 'restore' stage as
restore/default/990_move_away_restored_files.sh
so that the NNN_recreate_users_and_groups.sh script
would have to run after that e.g. as
restore/default/993_recreate_users_and_groups.sh
or perhaps even better it should run in the 'finalize' stage
where all such final clean up tasks are done - preferably
very early in the 'finalize' stage to ensure the right
user names and IDs and group names and IDs
are there for the other 'finalize' scripts.
In any case also the implementation of a generic
RECREATE_USERS_GROUPS functionality
should be submitted as a separated pull request.
Until a generic RECREATE_USERS_GROUPS functionality
gets actually implemented just keep your current
restore/YUM/default/405_clone_users_and_groups.sh
in this pull request - it can be deleted any time later
in the separated pull request where the generic
RECREATE_USERS_GROUPS functionality
is implemented.
schlomo commented at 2017-09-06 10:58:¶
👍 to splitting PRs into different topics.
This discussion made me curious about one detail: Why do our users so far not report lots of problems with restoring files while the users and groups are not defined?
My only guess is that all tools behave like tar: If the user name is not known then simply set the user id (which is the only thing that is stored in the file system in any case) and hope for the best.
As all backup strategies will both restore /etc/passwd /
/etc/group together with the files the system actually works correctly
after it boots up from the restored file system. The crucial part here
is the fact that both the user/group database and the files where stored
together in the backup and recovered together into the blank system.
With both the YUM and ZYPPER recovery methods this strong correlation between user/group database and file content is on purpose broken. The user/group database is recovered via RPM and then some files are recovered from a backup. In my opinion this discrepancy is the root cause for these problems that you are describing here.
ReaR has always prided itself in correctly separating between the rescue system and the recovered target system. The current implementation of YUM and ZYPPER also breaks with this separation and the complexity you describe here is a result of that.
I can imagine a slightly different approach that might be easier to follow:
- Use ReaR to recreate the disk layout and mount the filesystems
- Use RPM to reinstall the base system
- Restore the additional files in a chroot environment within the reinstalled base system
- Use ReaR to adjust the recovered system to the new hardware
The main difference is running the restore of the additional files in the context of the system that was installed via RPM. There of course the tar logic of first looking up users in the database and then falling back to IDs works as intended. If the UIDs will have changed in the recovered system then the restored files will also get the new UIDs.
This for sure works well for all users and groups created in RPM package scripts. If you want to also handle manually created users then I suggest to add the users that are missing in the target system after installing all the RPMs instead of creating all users beforehand.
The whole point of this exercise is maintaining a strict separation of concerns between files (and information) governed by different owners:
- files and users owned by an RPM package
- files and users created manually
To summarize: I have the impression that if YUM and ZYPPER use chroot where applicable that then the users and groups in the rescue system don't matter.
About the actual installation of the chroot via yum, I haven't done this myself before but I remember building chroot environments with steps like the ones mentioned in http://blog.parahard.com/2011/06/five-steps-to-create-fedora-chroot-jail.html, especially the part of initializing the RPM DB, installing some basic RPMs via rpm (and not yum) and putting some files into the chroot to make the scripts happy.
Could you maybe also let us know why you don't use existing tools like https://github.com/dozzie/yumbootstrap or https://github.com/rpm-software-management/mock which already contain a lot of expert knowledge about creating a fresh RPM-based setup in a chroot?
jsmeix commented at 2017-09-06 12:04:¶
@N3WWN @schlomo
have a look at my current 900_clone_users_and_groups.sh
in
https://github.com/rear/rear/pull/1472
regardless if "copy all 'getent' users and gropus" is really needed
or only needed in exceptional cases (like the YUM method)
it should now be at least possible (provided my code is o.k.)
to clone all available users and groups if needed.
There should be still no behavioural changes compared
to the 'old' 900_clone_users_and_groups.sh
(i.e. before I touched it).
N3WWN commented at 2017-09-07 21:50:¶
@jsmeix
Thanks for the info on how to update the PR to exclude the common files. I'm still testing 900_clone_users_and_groups.sh for both "YUM" and "YUM+file backup" but hope to get the PR updated before the weekend.
I would prefer to wait with symlinking YUM and ZYPPER scripts
until both implementations stabilized because currently
both are unfinished and work-in-progress
Right now I'm symlinking to some ZYPPER files, but will remove the symlinks and create copies of the files before I update the PR so they exist in isolation until they stabilize.
@jsmeix and @schlomo
double-clone-users-and-groups
I think the problem stems from having a system that has dynamic UID/GID assignment during package installation. Source system has UIDs/GIDs assigned dynamically when it was built. Those UIDs/GIDs are copied over to the ReaR rescue/restore system. Target system has UIDs/GIDs assigned dynamically when it is built by the ReaR rescue/restore system and there exist files on the target filesystem which adhere to those UIDs/GIDs. We now have two databases with dissimilar assignment of those UIDs/GIDs within one running OS. The tar archive also has a mapping of username and group to UID and GID.
If the ReaR rescue/restore system performs the restore natively (non-chroot), tar will use the ReaR rescue/restore system's UID/GID mappings when writing the files.
If the ReaR rescue/restore system performs the restore via chroot within the target environment, tar will use the target system's UID/GID mappings when writing the files. This is almost perfect, except when the user or group doesn't exist and tar has to fall back on the UID or GID stored in the tar archive. What happens then is that the files are restored with the wrong UID and/or GID.
The double-clone-users-and-groups should take care of this by ensuring that all users and groups that exist on the source system will exist (well, should exist) on the target system when the files are restored.
Why do our users so far not report lots of problems with restoring files while the users and groups are not defined?
I think because it requires a perfect storm of using tar (or other backup method that stores both usernames/groups and UIDs/GIDs and falls back to UID/GID if the username or group are missing) and a system rebuilt from a method that does so from scratch using packages (ZYPPER or YUM).
Since I ran into it in my testing, I worked on a solution to resolve it should anyone else encounter it, regardless of how often it may actually occur. 🎲
yumbootstrap and/or mock
I suppose there may exist other tools that may be a better solution at this time... I haven't looked, honestly, since I have been advocating ReaR within my organization since I first fell in love with it many months ago. When I saw the brand new functionality that ZYPPER provides, I saw an opportunity to contribute a similar YUM feature (based heavily on the ZYPPER code) and thought I'd add a file backup capability, also.
The double-clone-users-and-groups solution is actually only required for the file backup capability, so I could exclude it from the initial YUM PR, also, if that would be better. Then, YUM and ZYPPER will operate almost identically.
RECREATE_USERS_GROUPS and 395_recreate_users_and_groups.sh
I understand where you're coming from and going with this and will attempt to get this done ASAP, also. 😁
Thanks!
jsmeix commented at 2017-09-08 11:28:¶
Regarding
https://github.com/rear/rear/pull/1464#issuecomment-327449099
"Restore the additional files in a chroot environment
within the reinstalled base system":
This is a very interesting new functionality for ReaR
which basically matches what I wrote in the
"Other possible ways to do native disaster recovery with AutoYaST"
section at
https://en.opensuse.org/SDB:Disaster_Recovery
which reads (excerpt):
... let AutoYaST install all RPM software packages of the complete system and afterwards restore a backup that only contains those files that are added or changed compared to the original files in the RPM software packages. In this case the restore script could run as a so called "chroot script" (with "chrooted=false" or "chrooted=true" as needed in the particular case).
Backup restore via chroot within the target system
requires an already configured target system at least
up to the point so that the backup restore software
can successfully run within the target system.
For third-party backup software backup restore
within the target system requires that the third-party
backup restore software and all what it needs to run
is already there "ready-to-use" in the target system.
In contrast backup restore in the ReaR recovery system
already "just works" also for various third-party backup
software.
Therefore from my current point of view I would first and
foremost try to get the usual backup restore in the ReaR
recovery system also working well for YUM and ZYPPER
so that users can use the YUM and ZYPPER methods
in the same way as any other method.
In general regarding backup restore for YUM and ZYPPER
see my current ZYPPER description in default.conf
that reads (excerpts):
it is not an usual file-based backup/restore method ... During "rear mkbackup" it will basically only save a list of installed RPM packages ... during "rear recover" it will basically only install those RPM packages as pristine RPM packages ... Any other files that should be restored (e.g. prepared config files or /home directories) must be done by "Using Multiple Backups for Relax-and-Recover", see doc/user-guide/11-multiple-backups.adoc
Now as I read this again I wonder meanwhile
if I ever need to implement a particular
file-based backup/restore for ZYPPER,
i.e. I wonder if I need to implement something in
backup/ZYPPER/default/500_make_backup.sh
@N3WWN I have a question:
In
https://github.com/rear/rear/pull/1464#issuecomment-327287675
you wrote about 'tar' and
"We perform the file restore on the rescue/recovery system"
but in
https://github.com/rear/rear/pull/1464/files
I don't see any usage of 'tar' or any file-based backup restore.
Now I wonder what you mean with 'tar' and
"We perform the file restore on the rescue/recovery system"?
Do you need a special file-based backup/restore for YUM?
Or do you already use
"Using Multiple Backups for Relax-and-Recover"
for YUM?
jsmeix commented at 2017-09-08 15:23:¶
@N3WWN
In Germany it is evening so that I do not merge it today
and perhaps you like to do some minor adaptions.
If there are no strict objections I would like to merge
your YUM additions on Monday.
Have a nice weekend and many thanks for your
valuable contributions that help to move ReaR
forward to new use-cases!
N3WWN commented at 2017-09-08 16:16:¶
@jsmeix
...current point of view I would first and
foremost try to get the usual backup restore in the ReaR
recovery system also working well for YUM and ZYPPER
so that users can use the YUM and ZYPPER methods
in the same way as any other method.
In my dev branch that contains the yet-to-be-submitted YUM+backup PR, I used the NETFS backup workflow as my starting point, but forced the use of tar. Now that I'm becoming more and more familiar with the inner workings of ReaR, I see that I may be able to alter how the backup is performed to use whatever other file backup methods ReaR supports.
Regarding the lack of any mention of 'tar', that's because that'll be in a future PR 😁
Sadly, the somewhat interconnectedness of usr/share/rear/rescue/default/900_clone_users_and_groups.sh between YUM and YUM+backup have blurred the lines between those two in the discussion.
With the future work that I'll do for RECREATE_USERS_GROUPS and 395_recreate_users_and_groups.sh, which I think will be entirely within the YUM+backup PR, that interconnectedness will cease, at least for the time being.
Merging on Monday sounds great. I think I've finished the changes that I'm going to commit for this PR, so it should be stable between now and Monday.
Thanks and have a great weekend!
jsmeix commented at 2017-09-11 10:50:¶
@N3WWN
in your
restore/YUM/default/970_set_root_password.sh
I noticed
test "$ZYPPER_ROOT_PASSWORD" && root_password="$ZYPPER_ROOT_PASSWORD"
perhaps you meant to use YUM_ROOT_PASSWORD?
jsmeix commented at 2017-09-11 11:50:¶
Regardless of minor issues like
https://github.com/rear/rear/pull/1464#issuecomment-328493711
I will merge it right now as is because further issues
can be fixed via subsequent pull requests as needed.
N3WWN commented at 2017-09-12 14:45:¶
@jsmeix
Thank you sir! 😀
I'll get the <X>_ROOT_PASSWORD thing sorted out and will submit a new PR for it.
jsmeix commented at 2017-09-13 09:29:¶
@N3WWN
you are welcome!
FYI:
I have an idea regarding files backup for YUM and ZYPPER
that is related to what I wrote in
https://github.com/rear/rear/pull/1464#issuecomment-328078995
Because one cannot have two BACKUP=... at the same time
"Using Multiple Backups for Relax-and-Recover"
is the only way to restore a real files backup
(e.g. databases or tons of files in user home directores)
for YUM and ZYPPER.
But the plain YUM and ZYPPER RPM package installation
restults a pristine/raw unconfigured system.
To only get some configuration files coiped into the
target system after the RPM package installation
"Using Multiple Backups for Relax-and-Recover"
looks like a bit too big hammer for that purpose.
Therefore my idea to only get some files coiped into
the target system is to do basically the same as
init/default/030_update_recovery_system.sh
implements currently for the ReaR recovery system.
I.e. my idea is to add something like a
finalize/default/150_update_target_system.sh
script with a TARGET_SYSTEM_UPDATE_URL
(cf. RECOVERY_UPDATE_URL in default.conf)
that points to a download location wherefrom it
downloads a tar.gz archive and extracts that
at $TARGET_FS_ROOT i.e. into the root directory
of the target system.
pcahyna commented at 2021-05-18 22:44:¶
I symlinked files where possible and copied and updated files when changes were necessary for the YUM method.
Hello @N3WWN,
according to the diff, there are actually many files that could have
been symlinked, because they contain no significant changes. Those
are:
usr/share/rear/verify/YUM/default/980_umount_YUM_dir.sh
usr/share/rear/verify/YUM/default/070_set_backup_archive.sh
usr/share/rear/verify/YUM/default/060_mount_YUM_path.sh
usr/share/rear/verify/YUM/default/050_check_YUM_requirements.sh
usr/share/rear/restore/YUM/default/980_umount_YUM_dir.sh
usr/share/rear/restore/YUM/default/100_mount_YUM_path.sh
usr/share/rear/prep/YUM/default/070_set_backup_archive.sh
usr/share/rear/build/YUM/default/600_create_python_symlink.sh
Why have you converted those symlinks to copies? I am asking because I am doing a cleanup and having to change so many identical files is not very productive.
jsmeix commented at 2021-05-19 06:48:¶
@pcahyna
see what I wrote "Regarding symlinking YUM and ZYPPER scripts" in
https://github.com/rear/rear/pull/1464#issuecomment-327433717
and for an example where that separation was useful see
https://github.com/rear/rear/pull/1740#pullrequestreview-99619497
Since a longer time there have been almost no further changes
so it seems things had stabilized meanwhile.
In particular I noticed zero user interest about my ZYPPER method
so the ZYPPER method is not further developed by me.
Things may change if there is user interest in particular
by SUSE customers who in the end pay my salary.
In contrast the YUM method was further developed
in particular by
https://github.com/rear/rear/pull/1740
so currently the YUM code is the leading and active code
while the ZYPPER code rests (at least for now) in peace.
Accordingly when there are duplicated scritps
the YUM scripts should contain the actual code
with matching ZYPPER scripts as symlinks.
jsmeix commented at 2021-05-19 07:38:¶
Oops - I think I misunderstood
https://github.com/rear/rear/pull/1464#issuecomment-843615776
It is not about duplicates of YUM and ZYPPER scripts.
It is about YUM scripts that are basically copies of other scripts.
In this case I think the YUM scripts should be usually symlinks.
pcahyna commented at 2021-05-19 08:00:¶
@jsmeix indeed most of the files I am concerned with are not from ZYPPER
but from NETFS (and were converted to copies in commit
3d9cc245799317102ec5311d67c104494da1cbaf).
The ZYPPER ones are usually a bit different from the YUM ones. It would
be nice to unify them using some parameters, but it would be more work
and I don't attempt to do this here.
jsmeix commented at 2021-05-19 08:06:¶
@pcahyna
don't worry about ZYPPER.
If ZYPPER scripts need adaptions I would do it.
[Export of Github issue for rear/rear.]