#2936 Issue `closed`: "rear mkrescue -C ..." option setting is not automatically used by "rear recover"¶

Labels: enhancement, discuss / RFC, support / question, no-issue-activity

idanbenezra opened issue at 2023-02-16 21:08:¶

ReaR 2.7

When using the rear mkrescue -C /path/to/conf/coutomname.conf the rear recover will use the default.conf file and not the custom config file.

Please note that i can't use the /etc/rear/local.conf as I'm using it in a dynamic system that can execute multiple rear mkrescue at the same time with different configs.

In my custom config I've set the BACKUP=MYINTEGRATION, and when recover is executed it uses BACKUP=REQUESTRESTORE as this is the default in default.conf file.

Is this a normal behavior?

jsmeix commented at 2023-02-17 07:47:¶

usr/share/rear/conf/default.conf
gets sourced by sbin/rear in any case
because ReaR won't work correctly without its default settings
and there is no need to change usr/share/rear/conf/default.conf
because the defaults can be changed as needed by user config files.

What config files get sourced by sbin/rear is same
for all workflows (in particular same for "rear mkbackup"
and "rear recover") because the same code in sbin/rear
is run for all workflows.
The only difference is that during "rear recover" an
additional config file "rescue.conf" is there which
was created by "rear mkrescue/mkbackup".

See "man rear"
https://github.com/rear/rear/blob/rear-2.7/doc/rear.8.adoc
how to use a different directory for user config files
and how to specify additional user config files
(excerpt):

-c DIR
alternative config directory; instead of /etc/rear

-C CONFIG
additional config file; absolute path or relative to config directory

For me it seems to work
(but I did not test "rear recover" with that):

# cat /tmp/foodir/foo.conf

MYVAR=MYVALUE

# usr/sbin/rear -D -C /tmp/foodir/foo.conf mkrescue

Relax-and-Recover 2.7 / Git
...
Using build area: /var/tmp/rear.gHTk4xo2616VVcy
Sourcing additional configuration file '/tmp/foodir/foo.conf'
...

# chroot /var/tmp/rear.gHTk4xo2616VVcy/rootfs/

bash-4.4# cat /tmp/foodir/foo.conf

MYVAR=MYVALUE

bash-4.4# rear -D -C /tmp/foodir/foo.conf help

Sourcing additional configuration file '/tmp/foodir/foo.conf'
...

My blind guess is that perhaps your implementation
of your BACKUP=MYINTEGRATION is not yet right?

Or is the issue perhaps actually that
when you call

# rear -C /path/to/conf/myconfig.conf mkrescue

then calling only plain

# rear recover

will (of course) not use /path/to/conf/myconfig.conf
because you would have to call

# rear -C /path/to/conf/myconfig.conf recover

to tell the program rear to use the additional config file
also for the 'recover' workflow.

schlomo commented at 2023-02-17 08:25:¶

Maybe we should add a line like source /path/to/conf/myconfig.conf to $ROOTFS_DIR/etc/rear/rescue.conf so that the rescue system will automatically use the same extra configuration files that had been used to create it?

I think that this would help a lot to make the rescue → recovery workflow be more consistent with itself.

I wonder what the DRLM perspective on this is, @didacog and @proura what do you think?

jsmeix commented at 2023-02-17 08:41:¶

@schlomo
in general I am very much against all those
"well intended automatisms" all over the place.

What when a user wants to do

rear -C /path/to/mkrescue1.conf mkrescue

rear -C /path/to/mkrescue2.conf mkrescue

rear -C /path/to/mkbackup1.conf mkbackup

rear -C /path/to/mkbackup2.conf mkbackup

rear -C /path/to/mkbackuponly1.conf mkbackuponly

rear -C /path/to/mkbackuponly2.conf mkbackuponly

rear -C /path/to/recover1.conf recover

rear -C /path/to/recover2.conf recover

Most automatisms work against "final power to the user"
unless automatisms would have been implemented
very carefully and very thoroughly
but most are not.

idanbenezra commented at 2023-02-17 09:28:¶

@jsmeix

I do understand you point, However If i created the ISO with BACKUP=MYINTEGRATION, shouldn't it restored automatically with BACKUP=MYINTEGRATION?

schlomo commented at 2023-02-17 09:33:¶

I agree with @jsmeix and with @idanbenezra as you talk IMHO about different scenarios:

on the original system a user might have multiple configurations → need to manually specify it for every command
on the rescue system the main intention for ReaR is that everything is predefined and even an inexperienced user should be able to perform a recovery → the "right" configuration files should be used without specifying them

For the "final power to the user" perspective we need to provide a way how to achieve what @idanbenezra needs.

Personally I'd make this the default behaviour but maybe the following is a suitable compromise:

In https://github.com/rear/rear/blob/46d4dd3fe03595cc7b969db66d6ab1a881fcb6eb/usr/sbin/rear#L614-L624 we start sourcing the additional configuration files after reading the local.conf, site.conf and rescue.conf files. That means that instead of providing -C file on the command line you can configure that behaviour into your configuration via the CONFIG_APPEND_FILES variable.

So we could introduce a new configuration directive like "CONFIG_AUTOLOAD_EXTRA_FILES" or such that simply adds sets CONFIG_APPEND_FILES in /etc/rear/rescue.conf from CONFIG_APPEND_FILES_PATHS which is a new variable that I introduced with #2932 to record the extra configs that we read successfully during ReaR initialisation. Setting that configuration directive would induce the automatic behaviour that @idanbenezra wants.

didacog commented at 2023-02-17 09:42:¶

Maybe we should add a line like source /path/to/conf/myconfig.conf to $ROOTFS_DIR/etc/rear/rescue.conf so that the rescue system will automatically use the same extra configuration files that had been used to create it?

I think that this would help a lot to make the rescue → recovery workflow be more consistent with itself.

I wonder what the DRLM perspective on this is, @didacog and @proura what do you think?

Hello @schlomo,

As far as you simply add source <path_to_configs> before /usr/share/rear/init/default/010_set_drlm_env.sh will not cause any problem to DRLM provided configurations. The only concern is that if the config to source is not present, then rear must continue, not exiting. but if is controlled by CONFIG_AUTOLOAD_EXTRA_FILES=yes|no should be under control.

thanks for taking us into account!
Kind regards,
Didac

jsmeix commented at 2023-02-20 09:55:¶

I agree with what @schlomo wrote in his
https://github.com/rear/rear/issues/2936#issuecomment-1434376825

2. ... an inexperienced user should be able to perform a recovery ...

In sbin/rear the additional config files are
intentionally sourced after the ususal config files
so that the user can overwrite settings in his
ususal config files by different ones in his
additional config files so he can have settings
in his ususal config files as his specific defaults.

I think what @idanbenezra wants can be done without
any changes in ReaR code because sourcing config files
means to run them as bash scripts so the user can run
any commands as he needs for his specific use case, see
https://github.com/rear/rear/blob/master/etc/rear/local.conf

For example to automatically source an additional config file
when no '-C /tmp/foodir/foo.conf' was specified
in etc/rear/local.conf like

if ! [[ "$CONFIG_APPEND_FILES" == */tmp/foodir/foo.conf* ]] ; then
    LogPrint "Sourcing /tmp/foodir/foo.conf"
    test -s /tmp/foodir/foo.conf || Error "/tmp/foodir/foo.conf does not exists or is empty"
    Source /tmp/foodir/foo.conf || Error "Failed to Source /tmp/foodir/foo.conf"
fi

which seems to work for me
(but I didn't test with "rear recover"):

# usr/sbin/rear -D help
Sourcing /tmp/foodir/foo.conf
...

Currently one cannot set CONFIG_APPEND_FILES in etc/rear/local.conf
because CONFIG_APPEND_FILES is set readonly in sbin/rear before
etc/rear/local.conf is sourced.
Perhaps that readonly is superfluous and can be removed?

@idanbenezra
be careful with sourcing config files:
Sourcing the same config file more than once
can lead to trouble when settings are not idempotent,
e.g. when things are appended to variables.

jsmeix commented at 2023-02-20 10:13:¶

Currently ReaR behaves as usual traditional commands do
which means that a subsequent call of a command does not inherit
command line option settings from a former command call.
E.g. nobody expects that after one 'cat -n ...' subsequent
calls of plain 'cat' automatically number output lines.

In contrast to usual traditional commands modern GUI stuff
often behaves with "automated inheritance" of former calls.
E.g. after a GUI tool to play some music was run once
a subsequent run starts with the volume setting of the
former run (which is often rather unexpected volume).

I do not want such "automated inheritance" in ReaR.

On the other hand I understand that
"an inexperienced user should be able to perform a recovery".
In particular because this should help the user to avoid
accidental errors during a real disaster recovery
in case of emergency and time pressure.

As far as I see both conflict so I think
careful and thorough investigation is needed
to find out step by step what a right solution is
(or to show that no right solution exists in this case).

schlomo commented at 2023-02-20 12:05:¶

Good point about GUI apps, @jsmeix

I think ReaR is somewhat a special case, it is neither just a CLI tool nor a GUI app.

On top of that I take the ReaR Rescue System to be a very special tool, which is custom built for a single job and purpose: Recover the original system as easy as we can make it.

For that purpose the rear mkrescue phase takes a large amount of decisions and pre-configures the rescue system in a way that ensures that rear recover works well. And we don't support changing course mid-way, e.g. starting a rescue system and then changing the backup method to something completely different via command line option. At least we don't provide that kind of functionality by default.

With regard to extra configs that thinking makes me lean towards adding a configuration option that automatically includes the extra configs for recovery, to follow the existing approach that a rear mkrescue run pre-configures the rescue system for the intended job.

The result would be that calling rear mkrescue with different -C conf options would yield different rescue systems that also behave different. After all, in that extra config everything can be changed to the extend that calling rear recover without that extra config won't actually work.

As the recovery is intended to be used even by less technically skilled people I would choose "reliably run recovery as configured" over "the admin must choose the recovery extra options if he did so during mkrescue"

Makes any sense?

jsmeix commented at 2023-02-20 12:23:¶

All what you describe in
https://github.com/rear/rear/issues/2936#issuecomment-1436879543
"just works" with current ReaR with
BACKUP=MYINTEGRATION in etc/rear/local.conf
It works because you talk about "the system" (singular).

The issue here is that for obscure reasons @idanbenezra
cannot have BACKUP=MYINTEGRATION in etc/rear/local.conf

I do not at all understand what is meant with

a dynamic system that can execute multiple rear mkrescue
at the same time with different configs

Multiple rear mkrescue at the same time
are normally explicitly forbidden via

SIMULTANEOUS_RUNNABLE_WORKFLOWS=( 'mkbackuponly' 'restoreonly' )

in default.conf.

FYI:
The primarily intended use case behind '-C' is described in
https://github.com/rear/rear/blob/master/doc/user-guide/11-multiple-backups.adoc

schlomo commented at 2023-02-21 12:32:¶

@idanbenezra can you please shed some light on our discussion from your perspective? What - and why - would be your expectation with regard to how rear recover should behave in context of additional configuration files?

idanbenezra commented at 2023-02-21 13:05:¶

@schlomo

I'm integration ReaR into our own product (JetBackup https://www.jetbackup.com/).
Our product can execute multiple backup jobs simultaneously, However after @jsmeix last replay that rear mkrescue can't be executed simultaneously I've disabled it on our end also.

Our software creates custom config for ReaR during the backup process and then provides it to the mkrescue -C.

The end user that will use it won't know that he need to use recover -C on the rescue system, also the "Automatic Recover" will never work.

jsmeix commented at 2023-02-21 13:09:¶

By the way
(sleeping on an issue always helps - if not sleep more ;-)

I found a real issue with our current '-c' and '-C':

The 'automatic/auto_recover' and 'unattended' modes
currently call only

rear $rear_debug_options recover

see
https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/skel/default/etc/scripts/system-setup#L155
and
https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/skel/default/etc/scripts/system-setup#L192

So '-c' and '-C' are ignored / not possible
with our current 'automatic/auto_recover' and 'unattended' modes
which is a bug that needs to be fixed.

jsmeix commented at 2023-02-21 13:13:¶

@idanbenezra
yes,
I also found out that the 'automatic/auto_recover'
and 'unattended' modes currently don't work
with '-c' and/or '-C'
(which tells a lot how often '-c' and/or '-C'
plus 'automatic/auto_recover' or 'unattended'
are actually used by users "out there").

jsmeix commented at 2023-02-21 13:16:¶

My current offhanded idea how to fix that generically
is to store the whole 'rear [options] workflow' command line
of each last called workflow in the ReaR recovery system.

Because only "rear mkrescue" and "rear mkbackup" create
a ReaR recovery system only those workflows can store
their command line in the ReaR recovery system.

So some "rear special_option mkbackuponly" cannot
store its command line in the ReaR recovery system.

jsmeix commented at 2023-02-21 13:20:¶

I did
https://github.com/rear/rear/issues/2942
to have that specific issue separated from this one here.

jsmeix commented at 2023-02-21 13:26:¶

@idanbenezra
I do not understand what you actually mean with

Our product can execute multiple backup jobs simultaneously

How could it make sense to execute multiple backup jobs
simultaneously on one same system?

What exactly is a "backup job"?

I can imagine to run multiple file backups like 'tar'
simultaneously on one same system when each one
backups a different part of the files, cf. my
https://github.com/rear/rear/blob/master/doc/user-guide/11-multiple-backups.adoc

But I cannot imagine what the use case could be to
create multiple times simultaneously on one same system
a rescue/recovery system (e.g. a ReaR recovery system)?

idanbenezra commented at 2023-02-21 13:32:¶

@jsmeix

A "backup job" define a set of configurations that can be executed at a user configurable schedule.

For example, a user can set 2 different accounts backup jobs (will backup the entire account - homedir, databases, Email accounts and etc) that runs at the same time and backs up the data to 2 different offsite destinations.

jsmeix commented at 2023-02-21 13:41:¶

@idanbenezra
so with "backup" you mean making a backup of some files
but not creating a rescue/recovery system.

We use the same meaning of "backup" in ReaR.
But often users confuse the meaning of "backup" in ReaR
with ReaR's native task: "creating a rescue/recovery system"
cf. the section
"Relax-and-Recover versus backup and restore" in
https://en.opensuse.org/SDB:Disaster_Recovery

github-actions commented at 2023-04-24 02:22:¶

Stale issue message

[Export of Github issue for rear/rear.]

#2936 Issue closed: "rear mkrescue -C ..." option setting is not automatically used by "rear recover"¶

idanbenezra opened issue at 2023-02-16 21:08:¶

jsmeix commented at 2023-02-17 07:47:¶

schlomo commented at 2023-02-17 08:25:¶

jsmeix commented at 2023-02-17 08:41:¶

idanbenezra commented at 2023-02-17 09:28:¶

schlomo commented at 2023-02-17 09:33:¶

didacog commented at 2023-02-17 09:42:¶

jsmeix commented at 2023-02-20 09:55:¶

jsmeix commented at 2023-02-20 10:13:¶

schlomo commented at 2023-02-20 12:05:¶

jsmeix commented at 2023-02-20 12:23:¶

schlomo commented at 2023-02-21 12:32:¶

idanbenezra commented at 2023-02-21 13:05:¶

jsmeix commented at 2023-02-21 13:09:¶

jsmeix commented at 2023-02-21 13:13:¶

jsmeix commented at 2023-02-21 13:16:¶

jsmeix commented at 2023-02-21 13:20:¶

jsmeix commented at 2023-02-21 13:26:¶

idanbenezra commented at 2023-02-21 13:32:¶

jsmeix commented at 2023-02-21 13:41:¶

github-actions commented at 2023-04-24 02:22:¶

#2936 Issue `closed`: "rear mkrescue -C ..." option setting is not automatically used by "rear recover"¶