#1028 Issue closed: Recover failed - file not found

Labels: support / question, fixed / solved / done

deepesh-agarwal opened issue at 2016-10-06 06:23:

  • rear version (/usr/sbin/rear -V): 1.18-git20160031211 / 2016-10-03
  • OS version (cat /etc/rear/os.conf or lsb_release -a): No OS on SAS RAID-0, booted via REAR USB rescue media
  • rear configuration files (cat /etc/rear/site.conf or cat /etc/rear/local.conf):
### write the rescue initramfs to USB and update the USB bootloader
OUTPUT=USB

### create a backup using the internal NETFS method, using 'tar'
BACKUP=NETFS

### write both rescue image and backup to the device labeled REAR-000
BACKUP_URL=usb:///dev/disk/by-label/REAR-000
  • Brief description of the issue

Made a backup and tried restoring after drive change and I get this :-0 , any ideas what just happened :-(

sshot-6

sshot-6

jsmeix commented at 2016-10-06 12:22:

I never used rear that way (from USB rescue media)
but from what I see on your screenshots it seems
you try to boot a "backup" and not a "rescue image".

Regarding "OS version (cat /etc/rear/os.conf or lsb_release -a)":
We like to know on what Linux system and version
you did run "rear mkbackup".

From my current point of view it looks as if you need
first and foremost a better generic understanding
how rear works.

Or do I perhaps somehow completely misunderstand
what your issue is about?

jsmeix commented at 2016-10-06 13:03:

Meanwhile I think I am wrong.

The naming "backup" versus "rescue image" confused me.

In output/USB/Linux-i386/30_create_extlinux.sh
I found (excerpts):

case "$WORKFLOW" in
    (mkbackup) usb_label_workflow="backup";;
    (mkrescue) usb_label_workflow="rescue image";;
...
syslinux_write <"$USB_REAR_DIR/syslinux.cfg"
label $HOSTNAME-$time
    menu label ${time:0:4}-${time:4:2}-${time:6:2} ${time:9:2}:${time:11:2} $usb_label_workflow

Accordingly the "backup" named thingy should boot a
rear recovery system that resulted from "rear mkbackup".

@deepesh-agarwal
can you inspect what files there actually are
on your USB rescue media?

jsmeix commented at 2016-10-06 13:36:

Did my first try with kind of "USB rescue medium"
and for me it sems to "just work".

I am using a QEMU KVM virtual machine for my testing
with a second virtual harddisk /dev/sdb.

For preparation of /dev/sdb as rear rescue medium I did

# rear format /dev/sdb
USB device /dev/sdb must be formatted with ext2/3/4 or btrfs file system
Please type Yes to format /dev/sdb in ext3 format: Yes
# parted -s /dev/sdb unit GiB print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sdb: 1.00GiB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 
Number  Start    End      Size     Type     File system  Flags
 1      0.00GiB  1.00GiB  1.00GiB  primary  ext3         boot, type=83

I have in etc/rear/local.conf

OUTPUT=USB
USB_DEVICE=/dev/disk/by-label/REAR-000
BACKUP=NETFS
BACKUP_URL=usb:///dev/disk/by-label/REAR-000

Then I did only "rear mkrescue" because my 1GiB /dev/sdb
is too small to store also the backup.tar.gz:

# rear -d -D mkrescue
Relax-and-Recover 1.18 / Git
Using log file: /root/rear/var/log/rear/rear-g136.log
Creating disk layout
Creating root filesystem layout
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Creating initramfs
Writing MBR to /dev/sdb
Copying resulting files to usb location
You should also rm -Rf /tmp/rear.HYOgHYJbzr6leKR

Then I booted that virtual testing machine from /dev/sdb
and got the usual rear boot menue where I can select
an entry to boot the rear recovery system which boots
as usual for me.

What files there are on my /dev/sdb:

# mount /dev/sdb1 /mnt
# find /mnt -ls
     2    4 drwxr-xr-x   6 root     root         4096 Oct  6 15:18 /mnt
 40961    4 drwx------   3 root     root         4096 Oct  6 15:18 /mnt/boot
 40962    4 drwx------   2 root     root         4096 Oct  6 15:18 /mnt/boot/syslinux
 40980  280 -rw-------   1 root     root       280644 Oct  6 15:18 /mnt/boot/syslinux/hdt.c32
 40977    4 -rw-------   1 root     root          985 Oct  6 15:18 /mnt/boot/syslinux/rear.help
 40976  156 -rw-------   1 root     root       153104 Oct  6 15:18 /mnt/boot/syslinux/vesamenu.c32
 40983    4 -rw-------   1 root     root          239 Oct  6 15:18 /mnt/boot/syslinux/poweroff.com
 40972   12 -rw-------   1 root     root         9132 Oct  6 15:18 /mnt/boot/syslinux/ls.c32
 40965    8 -rw-------   1 root     root         5696 Oct  6 15:18 /mnt/boot/syslinux/cat.c32
 40981  992 -rw-------   1 root     root      1011048 Oct  6 15:18 /mnt/boot/syslinux/pci.ids
 40964    4 -rw-------   1 root     root         1638 Oct  6 15:18 /mnt/boot/syslinux/extlinux.conf
 40978   60 -rw-------   1 root     root        55140 Oct  6 15:18 /mnt/boot/syslinux/menu.c32
 40975   40 -rw-------   1 root     root        40464 Oct  6 15:18 /mnt/boot/syslinux/sysdump.c32
 40967    4 -rw-------   1 root     root          800 Oct  6 15:18 /mnt/boot/syslinux/cmd.c32
 40973  244 -rw-------   1 root     root       244124 Oct  6 15:18 /mnt/boot/syslinux/lua.c32
 40970    8 -rw-------   1 root     root         4236 Oct  6 15:18 /mnt/boot/syslinux/host.c32
 40968    8 -rw-------   1 root     root         5388 Oct  6 15:18 /mnt/boot/syslinux/cpuid.c32
 40971    8 -rw-------   1 root     root         5084 Oct  6 15:18 /mnt/boot/syslinux/kbdmap.c32
 40969    8 -rw-------   1 root     root         5132 Oct  6 15:18 /mnt/boot/syslinux/disk.c32
 40966    8 -rw-------   1 root     root         4620 Oct  6 15:18 /mnt/boot/syslinux/config.c32
 40984   32 -r--------   1 root     root        32768 Oct  6 15:18 /mnt/boot/syslinux/ldlinux.sys
 40982    4 -rw-------   1 root     root          800 Oct  6 15:18 /mnt/boot/syslinux/reboot.c32
 40974   24 -rw-------   1 root     root        21000 Oct  6 15:18 /mnt/boot/syslinux/rosh.c32
 40979   20 -rw-------   1 root     root        20192 Oct  6 15:18 /mnt/boot/syslinux/chain.c32
 40963    4 -rw-------   1 root     root          262 Oct  6 15:18 /mnt/boot/syslinux/message
  8193    4 drwxr-x---   2 root     root         4096 Oct  6 15:18 /mnt/g136
  8194    0 -rw-------   1 root     root            0 Oct  6 15:18 /mnt/g136/.lockfile
  8195    4 -rw-------   1 root     root          262 Oct  6 15:18 /mnt/g136/VERSION
  8197 8788 -rw-------   1 root     root      8979944 Oct  6 15:18 /mnt/g136/rear.log
  8196    4 -rw-------   1 root     root           62 Oct  6 15:18 /mnt/g136/README
    11   16 drwx------   2 root     root        16384 Oct  6 15:11 /mnt/lost+found
 32769    4 drwx------   3 root     root         4096 Oct  6 15:18 /mnt/rear
 32773    4 -rw-------   1 root     root          492 Oct  6 15:18 /mnt/rear/syslinux.cfg
 32770    4 drwx------   3 root     root         4096 Oct  6 15:18 /mnt/rear/g136
 32771    4 drwx------   2 root     root         4096 Oct  6 15:18 /mnt/rear/g136/20161006.1518
 32772    4 -rw-------   1 root     root          985 Oct  6 15:18 /mnt/rear/g136/20161006.1518/syslinux.cfg
 32776 8768 -rw-------   1 root     root      8960926 Oct  6 15:18 /mnt/rear/g136/20161006.1518/rear.log
 32774 5620 -rw-r--r--   1 root     root      5741648 Aug 27 02:14 /mnt/rear/g136/20161006.1518/kernel
 32775 146324 -rw-r--r--   1 root     root     149681392 Oct  6 15:18 /mnt/rear/g136/20161006.1518/initrd.cgz

where in particular the syslinux.cfg contains in my case

# cat /mnt/rear/syslinux.cfg
label rear
    say Relax-and-Recover - Recover g136 from 20161006.1518
    menu hide
    kernel g136-20161006.1518
label -
    menu label Recovery images
    menu disable
menu begin g136
    menu label g136
    text help
Recover backup of g136 to this system.
    endtext
    include /rear/g136/20161006.1518/syslinux.cfg
    menu separator
    label -
        menu label ^Back
        menu default
        text help
Return to the main menu
        endtext
        menu exit
menu end

and rear/g136/20161006.1518/syslinux.cfg contains

# cat /mnt/rear/g136/20161006.1518/syslinux.cfg
label g136-20161006.1518
    menu label 2016-10-06 15:18 rescue image
    say g136-20161006.1518 - Recover g136 rescue image (20161006.1518)
    text help
Relax-and-Recover v1.18 - rescue image using kernel 4.4.19-60-default 
BACKUP=NETFS OUTPUT=USB BACKUP_URL=usb:///dev/disk/by-label/REAR-000
    endtext
    kernel /rear/g136/20161006.1518/kernel
    append initrd=/rear/g136/20161006.1518/initrd.cgz root=/dev/ram0 vga=normal rw selinux=0 console=ttyS0,9600 console=tty0
label g136-20161006.1518
    menu label 2016-10-06 15:18 rescue image - AUTOMATIC RECOVER
    say g136-20161006.1518 - Recover g136 rescue image (20161006.1518)
    text help
Relax-and-Recover v1.18 - rescue image using kernel 4.4.19-60-default 
BACKUP=NETFS OUTPUT=USB BACKUP_URL=usb:///dev/disk/by-label/REAR-000
    endtext
    kernel /rear/g136/20161006.1518/kernel
    append initrd=/rear/g136/20161006.1518/initrd.cgz root=/dev/ram0 vga=normal rw selinux=0 console=ttyS0,9600 console=tty0 auto_recover

deepesh-agarwal commented at 2016-10-06 13:36:

I followed the exact procedure shown here - http://relax-and-recover.org/documentation/getting-started on a Debian 8.6 installation to create the rescue media and then created the backup on a 2TB USB disk. I got the drives changed and now unable to restore :(

jsmeix commented at 2016-10-06 13:50:

@deepesh-agarwal

many thanks for your information.
Now I better understand what you did.

As far as I understand your screenshots,
your current problem is that your USB rescue medium
does not boot the rear recovery system on it.

As far as I understand your screenshots, it seems
you have several rear recovery systems and
probably also several backup-tar.gz files on
your USB rescue medium.

I think at the syslinux/extlinux boot prompt "boot:"
it should work to enter the label or file name
of what syslinux/extlinux should load and launch
(but I am really not at all a syslinux/extlinux expert).

When it finally fails to boot a rear recovery system
from your USB rescue medium you can perhaps
(on another computer - provided you have one)
check whether or not there is at least one or more
backup-tar.gz files on your USB rescue medium
so that you still have at least a backup of your files.

In gereral note that there is
"No disaster recovery without testing and continuous validation"
see https://en.opensuse.org/SDB:Disaster_Recovery
excerpts:

You must test in advance that it works in your particular case
to recreate your particular system with your particular recovery
medium and that the recreated system can boot on its own and
that the recreated system with all its system services still work
as you need it in your particular case.
You must have replacement hardware available on which your
system can be recreated and you must try out if it works to
recreate your system with your recovery medium on your
replacement hardware.
You must continuously validate that the recovery still works
on the replacement hardware in particular after each change
of the basic system.

Bottom line:
There is no such thing as a disaster recovery solution
that "just works", see also
https://en.opensuse.org/SDB:Disaster_Recovery

deepesh-agarwal commented at 2016-10-06 14:10:

Will try your suggestion, but why it made multiple systems in the first place? I did the process only once!!!

This is the rescue media creation logfile:

http://pastebin.com/qgGvj3tt

jsmeix commented at 2016-10-07 13:54:

I did a second test like in
https://github.com/rear/rear/issues/1028#issuecomment-251963289
but now with a 10GiB virtual harddisk /dev/sdb.

Preparing /dev/sdb for rear:

# usr/sbin/rear format /dev/sdb
USB device /dev/sdb must be formatted with ext2/3/4 or btrfs file system
Please type Yes to format /dev/sdb in ext3 format: Yes
# parted -s /dev/sdb unit GiB print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sdb: 10.0GiB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 
Number  Start    End      Size     Type     File system  Flags
 1      0.00GiB  10.0GiB  10.0GiB  primary  ext3         boot, type=83

Running "rear mkbackup":

# usr/sbin/rear -d -D mkbackup
Relax-and-Recover 1.18 / Git
Using log file: /root/rear/var/log/rear/rear-g136.log
Creating disk layout
Creating root filesystem layout
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Creating initramfs
Writing MBR to /dev/sdb
Copying resulting files to usb location
Encrypting disabled
Creating tar archive '/tmp/rear.bY5ExGhCCGXg79V/outputfs/rear/g136/20161007.1519/backup.tar.gz'
Archived 1020 MiB [avg 9008 KiB/sec]OK
Archived 1020 MiB in 117 seconds [avg 8931 KiB/sec]

Booting from that virtual harddisk results
usual rear boot screen where I can select
to boot the rear recovery system (labeled "backup")
on that virtual harddisk.

The rear recovery system boots as usual and I can
log in there as root (no password) and run the recovery
(excerpts):

RESCUE g136:~ # rear -d -D recover
Relax-and-Recover 1.18 / Git
Using log file: /var/log/rear/rear-g136.log
NOTICE: Will do driver migration
Select a backup archive.
++ select choice in '"${backup_times[@]}"' '"Abort"'
1) 20161007.1519
2) Abort
#? 1
...
Calculating backup archive size
Backup archive size is 1.1G     /tmp/rear.Z2zC18yvDlp2CUG/outputfs/rear/g136/20161007.1519/backup.tar.gz (compressed)
Comparing disks.
Disk configuration is identical, proceeding with restore.
...
Creating swap on /dev/sda1
Disk layout created.
Decrypting disabled
Restoring from '/tmp/rear.Z2zC18yvDlp2CUG/outputfs/rear/g136/20161007.1519/backup.tar.gz'
Restored 2352 MiB [avg 92642 KiB/sec]OK
Restored 2352 MiB in 27 seconds [avg 89211 KiB/sec]
...
Installing GRUB2 boot loader
...
Finished recovering your system. You can explore it under '/mnt/local'.

I rebooted the rear recovery system and booted the system
from /dev/sda (where the receated system is) and everything
works well for me.

Then I did "rear mkbackup" a second time:

# usr/sbin/rear -d -D mkbackup
Relax-and-Recover 1.18 / Git
Using log file: /root/rear/var/log/rear/rear-g136.log
Creating disk layout
Creating root filesystem layout
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Creating initramfs
Writing MBR to /dev/sdb
Copying resulting files to usb location
Encrypting disabled
Creating tar archive '/tmp/rear.GpgywlrJeFzqlId/outputfs/rear/g136/20161007.1543/backup.tar.gz'
Archived 1027 MiB [avg 8994 KiB/sec]OK
Archived 1027 MiB in 118 seconds [avg 8917 KiB/sec]

I booted again from that /dev/sdb
got again the rear boot menue
where I selected first my system (via its host name)
and then I can now select two recovery systems
both labeled "backup" but with different time stamps.
I selected the last one which again boots as usual.

I can log in there as root and run the recovery
ans now (as expected) I can select between
two backups with different time stamps
where I also selected the last one:
(excerpts):

# rear -d -D recover
Relax-and-Recover 1.18 / Git
Using log file: /var/log/rear/rear-g136.log
NOTICE: Will do driver migration
Select a backup archive.
++ select choice in '"${backup_times[@]}"' '"Abort"'
1) 20161007.1519
2) 20161007.1543
3) Abort
#? 2
...
Calculating backup archive size
Backup archive size is 1.1G     /tmp/rear.sM5cxBPdnSaO03C/outputfs/rear/g136/20161007.1543/backup.tar.gz (compressed)
Comparing disks.
Disk configuration is identical, proceeding with restore.
...
Restoring from '/tmp/rear.sM5cxBPdnSaO03C/outputfs/rear/g136/20161007.1543/backup.tar.gz'
Restored 2528 MiB [avg 95905 KiB/sec]OK
Restored 2528 MiB in 28 seconds [avg 92480 KiB/sec]
...
Finished recovering your system. You can explore it under '/mnt/local'.

Summary:
For me using rear with a kind of "USB rescue medium"
(another disk drive in my case) "just works" even for
more than one backup on the rescue medium.

@deepesh-agarwal
I have no idea what went wrong in your particulat case
why you don't get the usual working rear boot menue.

deepesh-agarwal commented at 2016-10-07 16:12:

It's my bad day, while I asked my host to reinstall the OS and leave the attached USB untouched - they wiped the USB clean. Thankfully, the server had only 15 days of work which I had a 3 day old source-code backup.

Will definitely use and learn more about REAR now to backup as ISO on a different server.

Lesson learned.

[Export of Github issue for rear/rear.]