#3069 Issue closed: BACKUP=NSR: "rear checklayout" always results exit code 1

Labels: bug, fixed / solved / done, external tool

jsmeix opened issue at 2023-11-07 10:27:

  • ReaR version:
    2.7

  • OS version:
    SLES15 SP3

  • ReaR configuration file:
    local.conf excerpt for what is needed here

OUTPUT=ISO
BACKUP=NSR

I ( @jsmeix ) would like to report an issue with BACKUP=NSR
that a SUSE customer reported to us.

Regarding BACKUP=NSR:
In general there is nothing what I could do
in case of issues with third-party backup tools
because I do not have such software
so I can neither test nor reproduce anything.

A SUSE customer is using ReaR version 2.7

The customer got the following issue:

With BACKUP=NSR "rear checklayout" always results exit code 1
even if nothing of the disk layout has changed.
So "rear checklayout" would create a new ISO every day
even if this is not necessary.

The customer found
https://github.com/rear/rear/issues/2162
that tells (excerpts):

So IMHO removing the line
NSRSERVER=$(cat $VAR_DIR/recovery/nsr_server )
from the script 650_check_iso_recoverable.sh in the existing code
will not harm the execution of the NSR workflow

and

Log "NSRSERVER ($NSRSERVER) was defined in $CONFIG_DIR/local.conf"

The customer does not want to set a fixed NSRSERVER in local.conf
because NSRSERVER is dynamically determined via "rear mkrescue"
and stored in /var/lib/rear/recovery/nsr_server,
see rear/rescue/NSR/default/460_save_nsr_server_name.sh

In
https://github.com/rear/rear/issues/2162
there were 2 proposed workarounds:

Proposed workaround(s):
In 650_check_iso_recoverable.sh comment in the line as follows
#NSRSERVER=$(cat $VAR_DIR/recovery/nsr_server )
or
use the more sofisticated code snippet taken from 400_verify_nsr.sh instead:
[...]

if [[ ! -z "$NSRSERVER" ]]; then
    Log "NSRSERVER ($NSRSERVER) was defined in $CONFIG_DIR/local.conf"
elif [[ -f $VAR_DIR/recovery/nsr_server ]]; then
    NSRSERVER=$( cat $VAR_DIR/recovery/nsr_server )
else
    Error "Could not retrieve the EMC NetWorker Server name. Define NSRSERVER in $CONFIG_DIR/local.conf file"
fi

[...]

The first workaround was implemented but it causes that now
with BACKUP=NSR "rear checklayout" always results exit code 1
even if nothing of the disk layout has changed.

The customer tried the second workaround which works as expected.
In particular with the second workaround it works again as before
https://github.com/rear/rear/issues/2162

Furthermore the customer just linked it

usr/share/rear/layout/save/NSR/default # ls -l
total 4
lrwxrwxrwx 1 ... 400_verify_nsr.sh -> ../../../../verify/NSR/default/400_verify_nsr.sh
-rw-r--r-- 1 ... 650_check_iso_recoverable.sh

which also works for the customer
so this could be a third possible workaround.

So the questions are:

Is the customer right, that the second workaround should be used?
If yes, could ReaR be fixed this way?
Or would the second workaround cause other unwanted side effects?
Or could the third possible workaround be the best solution?

The customer would prefer the third possible workaround
provided this does not cause unwanted side effects.

  • Workaround:

Create
usr/share/rear/layout/save/NSR/default/400_verify_nsr.sh
as relative symlink that points to
../../../../verify/NSR/default/400_verify_nsr.sh

jsmeix commented at 2023-11-07 10:33:

@hpannenb @gdha @viper1986
could you please have a look here.

In particular I would be interested to know if reverting the fix
https://github.com/rear/rear/commit/420c177d446abec8a64b9683c1a3a3088eae11af
for the issue
https://github.com/rear/rear/issues/2162
and instead creating

usr/share/rear/layout/save/NSR/default/400_verify_nsr.sh

as relative symlink that points to

../../../../verify/NSR/default/400_verify_nsr.sh

would also still fix
https://github.com/rear/rear/issues/2162
plus make "rear checklayout" again result exit code 1
only when the disk layout has changed.

jsmeix commented at 2023-11-07 10:50:

Hmm - wait! - perhaps I misunderstand things:

Perhaps the third possible workaround is
to leave the fix
https://github.com/rear/rear/commit/420c177d446abec8a64b9683c1a3a3088eae11af
for
https://github.com/rear/rear/issues/2162
as is
and only add

usr/share/rear/layout/save/NSR/default/400_verify_nsr.sh

as relative symlink that points to

../../../../verify/NSR/default/400_verify_nsr.sh

to have
https://github.com/rear/rear/issues/2162
still fixed
plus make "rear checklayout" again result exit code 1
only when the disk layout has changed.

schlomo commented at 2023-11-12 19:03:

@jsmeix I guess to verify potential fixes and to evaluate the right approach somebody will need to run some tests on a system with NSR installed.

If a personal and direct interaction with this SUSE customer is possible to do so, then please reach out to @gdha or me as this might yield the fastest results.

Do I understand correctly that upstream ReaR already behaves well and you are now looking for a potential back-port for SUSE?

jsmeix commented at 2023-11-13 08:32:

@schlomo
current upstream ReaR does not behave well
because it contains the change
https://github.com/rear/rear/commit/420c177d446abec8a64b9683c1a3a3088eae11af
of
https://github.com/rear/rear/issues/2162

ReaR from before that change behaved well for what
our customer needs but that does not behave well
for what @viper1986 needs according to
https://github.com/rear/rear/issues/2162#issue-460949301

Our customer has already a workaround for his needs in place.

This issue is about to get a proper ReaR upstream fix
that makes things behave well for both use cases,
the one our customer has and the one @viper1986 has.

In
https://github.com/rear/rear/issues/2162#issuecomment-541343374
@hpannenb proposed two "workarounds"
where the first one was implemented
which causes the issue for our customer
but for the more sophisticated second one
our customer reported that this works for him.

So it seems the more sophisticated second one
makes things behave well for both use cases,
the one our customer has and the one @viper1986 has.

Accordingy I will implement that second workaround now
as a GitHub pull request to somehow move this issue forward...

jsmeix commented at 2023-11-13 08:47:

@hpannenb @viper1986
could you please have a look at
https://github.com/rear/rear/pull/3077

jsmeix commented at 2023-11-13 12:42:

https://github.com/rear/rear/pull/3077
didn't work.

schlomo commented at 2023-11-13 13:00:

The suggestion from the discussion in #3077 is to actually remove the check for NSR backups from the checklayout workflow and instead adding support for easily running custom code as part of the checklayout workflow.

Specifically this would also mean removing the complete 650_check_iso_recoverable.sh
https://github.com/rear/rear/blob/01f503ed05beef78d8397719b9dce25fcdb211c6/usr/share/rear/layout/save/NSR/default/650_check_iso_recoverable.sh#L1-L18

Any thoughts on implementing that? Wouldn't that actually solve the customer problem, too?

I know that this essentially reverts #653 and might force users like @tomglx to adjust the way how they use ReaR. However I hope that this is acceptable if it will prevent troubles for other NSR users.

schlomo commented at 2023-11-13 13:47:

BTW, looking at the non-default code paths in layout/save it seems like only NSR has that type of additional backup monitoring functionality:
image

github-actions commented at 2024-01-13 02:09:

Stale issue message

hpannenb commented at 2024-01-13 11:16:

@hpannenb @viper1986 could you please have a look at #3077

Sorry. I have missed this issue so I am quiet late responding. If required I might need to dig into this if time permits.

As mentioned elsewhere I just added some snippets of code to the existing NSR related backup code to enable our companies/my usecase and to be compatible to the existing one(s). I currently do not get to know where the rear checklayout exits with the wrong error code. Is there any (debug) log available or did I miss that?

hpannenb commented at 2024-01-13 11:45:

#3077 didn't work.

@jsmeix I would have opted for this solution. Is there anything like a (debug) log to find out more about why it did not solve the rear checklayout situation for the customer?

github-actions commented at 2024-03-15 02:02:

Stale issue message

github-actions commented at 2024-05-25 02:08:

Stale issue message

viper1986 commented at 2024-05-25 05:34:

Hello everyone,
Unfortunately, I don't have the opportunity to test REAR with NSR.
I'm very sorry, but our NSR-based environment has been disabled and I am unable to perform any tests.

gdha commented at 2024-09-05 07:31:

@jsmeix milestone is still 'ReaR v2.8' - please adjust as you think it best fits (v3.0 or v3.1). Thanks.

jsmeix commented at 2024-09-05 07:58:

FYI
in general regarding issues with third-party backup tools:

Usually we at ReaR upstream do not have or use third-party backup tools
(in particular not if a third-party backup tool is proprietary software)
so usually we cannot reproduce issues with third-party backup tools.

In case of issues with third-party backup tools and ReaR
we at ReaR upstream can usually do nothing but totally
depend on contributions and help from those specific users
who use and know about each specific third-party backup tool.

schlomo commented at 2024-09-05 09:12:

I'd like to suggest again (https://github.com/rear/rear/issues/3069#issuecomment-1808196982) that we remove this very NSR specific check and reduce the complexity for ReaR. Happy to add a custom code hook instead.

jsmeix commented at 2024-09-05 10:23:

I propose to deprecate
layout/save/NSR/default/650_check_iso_recoverable.sh
https://github.com/rear/rear/pull/3309

jsmeix commented at 2024-09-05 13:27:

With https://github.com/rear/rear/pull/3309 merged
the script
layout/save/NSR/default/650_check_iso_recoverable.sh
is deprecated because it conflicts with "rear checklayout"

[Export of Github issue for rear/rear.]