#1166 Issue closed: Possibly useless backup selection when backup restore from USB

Labels: enhancement, bug, cleanup, fixed / solved / done

jsmeix opened issue at 2017-01-13 14:16:

From my experience with "rear recover" at
https://github.com/rear/rear/pull/1165#issuecomment-272449196

I use

OUTPUT=USB
USB_DEVICE=/dev/disk/by-label/REAR-000
BACKUP=NETFS
BACKUP_URL=usb:///dev/disk/by-label/REAR-000

I did several (5) times "rear mkbackup" and got
(with the default USB_RETAIN_BACKUP_NR=2)
on the REAR-000 medium three backup directories

rear/e205/20170113.1422
rear/e205/20170113.1422/kernel
rear/e205/20170113.1422/rear-e205.log
rear/e205/20170113.1422/syslinux.cfg
rear/e205/20170113.1422/backup.log
rear/e205/20170113.1422/initrd.cgz
rear/e205/20170113.1422/backup.tar.gz
rear/e205/20170113.1429
rear/e205/20170113.1429/kernel
rear/e205/20170113.1429/rear-e205.log
rear/e205/20170113.1429/syslinux.cfg
rear/e205/20170113.1429/backup.log
rear/e205/20170113.1429/initrd.cgz
rear/e205/20170113.1429/backup.tar.gz
rear/e205/20170113.1442
rear/e205/20170113.1442/kernel
rear/e205/20170113.1442/rear-e205.log
rear/e205/20170113.1442/syslinux.cfg
rear/e205/20170113.1442/backup.log
rear/e205/20170113.1442/initrd.cgz
rear/e205/20170113.1442/backup.tar.gz

During boot I selected in the syslinux boot menue
2017-01-13 14:29 backup

Then I run "rear -d -D recover" which results (excerpts):

Running workflow recover within the ReaR rescue/recovery system
Using backup archive 'backup.tar.gz'
...
Select a backup archive.
1) 20170113.1422
2) 20170113.1429
3) 20170113.1442
4) Abort
#? 3
...
2017-01-13 13:52:09.209884020 Backup archive /tmp/rear.9fahNJ3x3Dm5i43/outputfs/rear/e205/20170113.1442/backup.tar.gz chosen.
...
Calculating backup archive size
Backup archive size is 992M        /tmp/rear.9fahNJ3x3Dm5i43/outputfs/rear/e205/20170113.1429/backup.tar.gz (compressed)
...
Disk layout created.
Restoring from '/tmp/rear.9fahNJ3x3Dm5i43/outputfs/rear/e205/20170113.1429/backup.tar.gz'...
Backup restore program 'tar' started in subshell (PID=2148)
Restored 344 MiB [avg. 117665 KiB/sec] 
...
Restored 2465 MiB [avg. 114745 KiB/sec] 
OK
Restored 2506 MiB in 25 seconds [avg. 102656 KiB/sec]
Restoring finished.

In short:
I got the backup restored that matches what
I selected during boot in the syslinux menu
and not the backup that I selected during "rear recover".

I wonder why there is that backup selection
during "rear recover" when the right backup is
already specified during boot in the syslinux menu?

If there is a reason for the backup selection
during "rear recover" it seems it does not work.

@gdha
do you know the reason for the backup selection
during "rear recover" in case of backup on USB?

jsmeix commented at 2017-01-13 14:19:

FYI:
I did not change the backup selection during "rear recover" in
verify/USB/NETFS/default/540_choose_backup_archive.sh
in my recent https://github.com/rear/rear/pull/1165

jsmeix commented at 2017-01-13 14:36:

I think I know why the backup selection during "rear recover" in
verify/USB/NETFS/default/540_choose_backup_archive.sh
does no longer work:

"usr/sbin/rear -s recover" shows (excerpt)

Source verify/NETFS/default/070_set_backup_archive.sh
...
Source verify/USB/NETFS/default/540_choose_backup_archive.sh

The right backup is set as RESTORE_ARCHIVES in
verify/NETFS/default/070_set_backup_archive.sh
but later
verify/USB/NETFS/default/540_choose_backup_archive.sh
does not adapt RESTORE_ARCHIVES so that it has no effect.

This means the issue is a regression since my
https://github.com/rear/rear/pull/1071
https://github.com/rear/rear/issues/1069

I need to implement RESTORE_ARCHIVES in
verify/USB/NETFS/default/540_choose_backup_archive.sh
provided there is a real reason for another backup selection
during "rear recover" regardless that the right backup
should have already specified during boot in the syslinux menu.

Furthermore that backup selection during "rear recover"
makes it impossible to run "rear recover" unattended
in case of backup from USB.

If at all that backup selection during "rear recover" should
only hapen if RESTORE_ARCHIVES is empty or
its content cannot be found on the REAR-000 medium.

gozora commented at 2017-01-13 14:59:

Hello @jsmeix ,
I've used something like this but it is more an workaround....

jsmeix commented at 2017-01-13 15:08:

Yes, that is the basic idea but if I implement
RESTORE_ARCHIVES support in
verify/USB/NETFS/default/540_choose_backup_archive.sh
I would implement support to select multiple archives
because restore of multiple archives just works now
with RESTORE_ARCHIVES.

jsmeix commented at 2017-01-13 15:23:

The code in
verify/USB/NETFS/default/540_choose_backup_archive.sh
looks wrong because it sets "backuparchive"
and not "restorearchive" what it was in the old code
and what I have completely replaced by RESTORE_ARCHIVES.

But that whole old code about "backuparchive" looks so ugly
because in
restore/NETFS/default/400_restore_backup.sh
there is

# The RESTORE_ARCHIVES array contains the restore input files.
# If it is not set, RESTORE_ARCHIVES is only one element which is the backup archive:
test "$RESTORE_ARCHIVES" || RESTORE_ARCHIVES=( "$backuparchive" )

What a mess of badly named and badly used variables!

jsmeix commented at 2017-01-13 15:24:

I guess I need to rework the whole old code about "backuparchive"
and replace it with something that at least I can understand
to avoid more such regressions in the future :-(

jsmeix commented at 2017-01-16 11:43:

In contrast to what I wrote last week above in
https://github.com/rear/rear/issues/1166#issuecomment-272469071
I think now that the "backuparchive" usage and implementation
is actually o.k.- only the variable name should be uppercase, cf.
https://github.com/rear/rear/pull/1165#issuecomment-272814988

jsmeix commented at 2017-01-16 11:51:

I think meanwhile I understand why that backup selection
dialog during "rear recover" is there:

It is not needed when one uses "rear mkbackup"
where kernel, initrd, and backup are in one same directory
but it is needed when one runs separated
"rear mkrescue" and "rear mkbackuponly"
because then by default on USB kernel and initrd
are in one timestamp directory and the backup
is in another timestamp directory.

Because by default on USB there are more than one
backup (via USB_RETAIN_BACKUP_NR) there should
be a dialog during "rear recover" where the user can
select what backup he likes to restore.

But I think that backup selection dialog should behave
more user friendly and preselect the latest backup
that can be found on the USB medium so that the user
could just hit [Enter] to get the latest backup restored.

When only one backup is found on the USB medium
there should not be such a dialog - just that one backup
gets restored - because what else could the user do?

jsmeix commented at 2017-01-16 12:03:

For me it does not work with separated
"rear mkrescue" and "rear mkbackuponly":

I did two times
"rear mkrescue" and "rear mkbackuponly"
so that I have now on my REAR-000 medium
(cf. https://github.com/rear/rear/issues/1166#issue-200640200
what I had initially):

rear/e205/20170116.1230
rear/e205/20170116.1230/backup.log
rear/e205/20170116.1230/backup.tar.gz
rear/e205/20170116.1227
rear/e205/20170116.1227/kernel
rear/e205/20170116.1227/rear-e205.log
rear/e205/20170116.1227/syslinux.cfg
rear/e205/20170116.1227/initrd.cgz
rear/e205/20170113.1442
rear/e205/20170113.1442/kernel
rear/e205/20170113.1442/rear-e205.log
rear/e205/20170113.1442/syslinux.cfg
rear/e205/20170113.1442/backup.log
rear/e205/20170113.1442/initrd.cgz
rear/e205/20170113.1442/backup.tar.gz
rear/e205/20170116.1219
rear/e205/20170116.1219/kernel
rear/e205/20170116.1219/rear-e205.log
rear/e205/20170116.1219/syslinux.cfg
rear/e205/20170116.1219/initrd.cgz
rear/e205/20170116.1225
rear/e205/20170116.1225/backup.log
rear/e205/20170116.1225/backup.tar.gz

Summary what timestamps belong to what action:
...1442 mkbackup
...1219 mkrescue
...1225 mkbackuponly
...1227 mkrescue
...1230 mkbackuponly

The syslinux.cfg files contain those menu labels:

rear/e205/20170113.1442/syslinux.cfg: menu label 2017-01-13 14:42 backup
rear/e205/20170113.1442/syslinux.cfg: menu label 2017-01-13 14:42 backup - AUTOMATIC RECOVER
rear/e205/20170116.1219/syslinux.cfg: menu label 2017-01-16 12:19 rescue image
rear/e205/20170116.1219/syslinux.cfg: menu label 2017-01-16 12:19 rescue image - AUTOMATIC RECOVER
rear/e205/20170116.1227/syslinux.cfg: menu label 2017-01-16 12:27 rescue image
rear/e205/20170116.1227/syslinux.cfg: menu label 2017-01-16 12:27 rescue image - AUTOMATIC RECOVER

For recovery I booted the syslinux menu label
"2017-01-16 12:19 rescue image"

Then in the ReaR recovery system it fails as follows:

Using backup archive 'backup.tar.gz'
NOTICE: Will do driver migration
Select a backup archive.
1) 20170113.1442
2) 20170116.1225
3) 20170116.1230
4) Abort
#? 3
Backup archive /tmp/rear.dMag4O5WJ908yfk/outputfs/rear/e205/20170116.1230/backup.tar.gz chosen.
ERROR: Backup archive 'backup.tar.gz' not found.
Aborting due to an error, check /var/log/rear/rear-e205.log for details
Terminated

The
ERROR: Backup archive 'backup.tar.gz' not found.
message comes from
verify/NETFS/default/550_check_backup_archive.sh

"rear -s recover" shows:

Source verify/NETFS/default/070_set_backup_archive.sh
...
Source verify/USB/NETFS/default/540_choose_backup_archive.sh
Source verify/NETFS/default/550_check_backup_archive.sh

I will make
verify/USB/NETFS/default/540_choose_backup_archive.sh
behave well...

jsmeix commented at 2017-01-16 12:39:

Initial bug is in 070_set_backup_archive.sh that blindly sets

++ backuparchive=/tmp/rear.B5LcS9D1ZfCXfJs/outputfs/rear/e205/20170116.1219/backup.tar.gz
++ RESTORE_ARCHIVES=("$backuparchive")

regardless that there is no such file
(the "1219" directory is from a "mkrescue")
which then leads to 550_check_backup_archive.sh
not setting RESTORE_ARCHIVES to the right backuparchive
that is specified during 540_choose_backup_archive
via

test "$RESTORE_ARCHIVES" || RESTORE_ARCHIVES=( "$backuparchive" )

because RESTORE_ARCHIVES is already set.

Accordingly this small change in 070_set_backup_archive.sh
makes "rear recover" works for me in this case:

@@ -38,8 +38,12 @@ local backup_directory=$BUILD_DIR/outputfs/$NETFS_PREFIX
 if ! test "incremental" = "$BACKUP_TYPE" -o "differential" = "$BACKUP_TYPE" ; then
     backuparchive="$backup_directory/$backup_file_name"
     # In case of normal (i.e. non-incremental) backup there is only one restore archive
-    # and its name is the same as the backup archive (usually 'backup.tar.gz'):
-    RESTORE_ARCHIVES=( "$backuparchive" )
+    # and its name is the same as the backup archive (usually 'backup.tar.gz').
+    # Only set RESTORE_ARCHIVES for "rear recover" when the backup archive is accessible
+    # cf. https://github.com/rear/rear/issues/1166
+    test -r "$backuparchive" && RESTORE_ARCHIVES=( "$backuparchive" )
+    # Do not error out here when backup archive is not accessible because that same code
+    # is also run during "rear mkbackup" where usually the backup archive does not yet exist.
     LogPrint "Using backup archive '$backup_file_name'"
     return
 fi

gdha commented at 2017-01-16 13:04:

@jsmeix USB code was designed and written by @dagwieers and @jhoekx for their special customer. It always used to work as it was designed. I cannot remember I changed lot of code in that area...

jsmeix commented at 2017-01-16 13:59:

@gdha
many thanks for the information that this code was made
as a special case (hack) for a special customer.

I will make it working reasonably well also for other users
because that functionality is now there in ReaR
so that it should work reasonably well in general.

jsmeix commented at 2017-01-17 10:42:

Only a side note (I won't fix that):

It seems the default USB_RETAIN_BACKUP_NR=2
does not strictly work this way.

Continuing from the state what I had
on my REAR-000 medium as shown in
https://github.com/rear/rear/issues/1166#issuecomment-272845326
I did one more "rear mkrescue" (timestamp "...1447")
then a "rear mkbackup" (timestamp "...1057") and
a "rear mkbackuponly" (timestamp "...1106") and
now I have those files on my REAR-000 medium

rear/e205/20170116.1230
rear/e205/20170116.1230/backup.log
rear/e205/20170116.1230/backup.tar.gz
rear/e205/20170116.1447
rear/e205/20170116.1447/kernel
rear/e205/20170116.1447/rear-e205.log
rear/e205/20170116.1447/syslinux.cfg
rear/e205/20170116.1447/initrd.cgz
rear/e205/20170116.1225
rear/e205/20170116.1225/backup.log
rear/e205/20170116.1225/backup.tar.gz
rear/e205/20170117.1106
rear/e205/20170117.1106/backup.log
rear/e205/20170117.1106/backup.tar.gz
rear/e205/20170117.1057
rear/e205/20170117.1057/kernel
rear/e205/20170117.1057/rear-e205.log
rear/e205/20170117.1057/syslinux.cfg
rear/e205/20170117.1057/backup.log
rear/e205/20170117.1057/initrd.cgz
rear/e205/20170117.1057/backup.tar.gz

so that I have now 4 backups and two recovery systems.

jsmeix commented at 2017-01-17 15:45:

With https://github.com/rear/rear/pull/1168
merged, this issue should be sufficiently fixed.


[Export of Github issue for rear/rear.]