#1629 Issue closed: Cannot boot from ReaR ISO / Recover from ReaR USB¶
Labels: support / question, fixed / solved / done
AmonBune opened issue at 2017-12-07 17:12:¶
rear version (/usr/sbin/rear -V):¶
Relax-and-Recover 2.2 / Git
OS version (lsb_release -a):¶
Distributor ID: Ubuntu
Description: Ubuntu 17.10
Release: 17.10
Codename: artful
rear configuration files (cat /etc/rear/local.conf):¶
OUTPUT=ISO
# Backup Filesystem backend (Should normally be NETFS)
BACKUP=NETFS
# Backup type
BACKUP_TYPE=incremental
# Location where the backups should be stored
BACKUP_URL="file:///mnt/backupstorage0/rear-backups"
# Add a GRUB entry for the backup
GRUB_RESCUE=1
## For USB rescue media creation
#OUTPUT=USB
#BACKUP_URL="usb:///dev/disk/by-label/REAR-000"
#USB_UEFI_PART_SIZE=400
Are you using legacy BIOS or UEFI boot?¶
UEFI
Brief description of the issue:¶
My server recently crashed and the main filesystem now has some errors. I want to recover from a backup I made some days ago, but I cannot boot into the Relax-and-Recover entry from the UEFI screen anymore after this crash (GRUB2 also doesn't show the entry anymore) so something messed up rear's bootfiles or the partition itself. When I manually select the UEFI entry "Relax-and-Recover", I get an empty GRUB prompt.
First, I thought with a rear mkrescue the ISO would be recreated and
it would boot then. But it did not work. It still doesn't boot.
spitfire@Cloudsdale:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 63G 0 63G 0% /dev
tmpfs 13G 10M 13G 1% /run
/dev/md0p1 30G 4.0G 24G 15% /
tmpfs 63G 0 63G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 63G 0 63G 0% /sys/fs/cgroup
/dev/sde1 1.9G 5.6M 1.9G 1% /boot/efi
/dev/sdf1 3.7T 1.9T 1.8T 52% /mnt/backupstorage0
/dev/md127p1 3.7T 2.1T 1.7T 56% /mnt/mainstorage0
/dev/sdg2 29G 384M 27G 2% /mnt/tmp
tmpfs 13G 0 13G 0% /run/user/1000
The filesystem error occured in the [/] partition. The
[/mnt/mainstorage0] and [/mnt/backupstorage0] which are both
separate disks are still okay. [/mnt/tmp] is an empty ReaR USB
stick.
[/mnt/backupstorage0] is where my backup archive and also the ReaR ISO
is stored.
Work-around, if any:¶
I tried to load the system with an USB stick but I don't know how to
recover my system from there. A simple "rear recover" doesn't work. My
backup files are on the [/mnt/backupstorage0] disk, which are not
mounted anyway in this ReaR environment. How can I tell the
rear recover program which backup it should use? And how does it know
which disk it has to recover? man rear and rear --help don't mention
anything about such options.
gozora commented at 2017-12-07 18:42:¶
@AmonBune I'm still a bit confused about your setup :-).
You've stated that you use OUTPUT=ISO which creates ISO boot file, but
then you've mentioned following as well:
I tried to load the system with an USB stick but I don't know how to recover my system from there.
Should I understand it like you currently have ISO (which is not booting) as well as USB stick which is booting fine, form which are you just not able to recover?
I tried to load the system with an USB stick but I don't know how to recover my system from there. A simple "rear recover" doesn't work. My backup files are on the [/mnt/backupstorage0] disk, which are not mounted anyway in this ReaR environment. How can I tell the rear recover program which backup it should use? And how does it know which disk it has to recover? man rear and rear --help don't mention anything about such options.
This is implication of using file://, before running rear recover
you need to mount your backup source manually, something like:
mkdir /mnt/backupstorage0
# /dev/sdf1 is just a blind guess, you need to find right disk by your self ;-)
mount /dev/sdf1 /mnt/backupstorage0
rear recover
V.
AmonBune commented at 2017-12-07 19:52:¶
Should I understand it like you currently have ISO (which is not booting) as well as USB stick which is booting fine, form which are you just not able to recover?
Yes, this is correct, I have the non-bootable ISO and a bootable USB stick.
How does ReaR know which disk is the target disk? Does it store the unique ID of the hard disk or how does it do that? I will try your solution later and report back to you.
AmonBune commented at 2017-12-07 20:28:¶
Okay, your solution would probably work, but ReaR searches the backup
archive at a wrong location. It looks like a random folder, like so:
/tmp/rear.GasN2HK2jS/rear-backups/Cloudsdale/. The random part also
changes everytime I run the rear recover command. Really annoying
since I cannot mount something to a location I don't know. How can I
change this? I used the options that are commented out in the config for
USB creation.
gozora commented at 2017-12-07 20:36:¶
Okay, your solution would probably work, but ReaR searches the backup archive at a wrong location. It looks like a random folder, like so: /tmp/rear.GasN2HK2jS/rear-backups/Cloudsdale/
I don't think this is valid behavior, backup should be loaded from
location specified in
BACKUP_URL="file:///mnt/backupstorage0/rear-backups", but it is some
time since I've used file:// ...
I'll try to setup ReaR in same way as yours and let you know ...
Can you confirm that /etc/rear/local.conf in your ReaR recovery system (after it boots from USB) looks exactly same as you posted in issue template?
V.
gozora commented at 2017-12-07 21:02:¶
So, I've created rear backup (rear mkbackup) with config as follows:
BACKUP=NETFS
OUTPUT=ISO
BACKUP_URL=file:///mnt/rear
OUTPUT_URL=nfs://backup.virtual.sk/mnt/rear/iso
Then
- I've booted ISO
- manually mounted /mnt/rear
- run
rear recoverwhich outputted to log
2017-12-07 20:53:06.633204005 Decrypting backup archive is disabled
2017-12-07 20:53:06.634746875 Restoring from '/mnt/rear/sp2/backup.tar.gz'...
2017-12-07 20:53:06.637181374 Launched backup restore subshell (PID=4177)
2017-12-07 20:53:06.638837441 dd if=/mnt/rear/sp2/backup.tar.gz | cat | tar --block-number --totals --verbose --anchored --xattrs --xattrs-include=security.capability --xattrs-include=security.selinux --acls --gzip -C /mnt/local/ -x -f -
After this system booted just fine ...
I'll do one more test where I create ReaR rescue system on USB ...
V.
AmonBune commented at 2017-12-07 21:19:¶
I created the USB with the following configuration:
OUTPUT=USB
BACKUP=NETFS
BACKUP_URL=usb:///dev/disk/by-label/REAR-000
USB_UEFI_PART_SIZE=400
When booting from this stick and running rear recover I get above
posted issue with the weird folder name.
I did some testing too. I booted with the USB stick. Then I changed the
/etc/rear/local.conf to:
OUTPUT=ISO
BACKUP=NETFS
BACKUP_TYPE=incremental
GRUB_RESCUE=1
BACKUP_URL=file:///mnt/backupstorage0/rear-backups
I had to mount the backupstorage to some /mnt/tmp and then execute
ln -s /mnt/tmp/rear-backups/Cloudsdale/backup.tar.gz /mnt/backupstorage0/rear-backups/rear/Cloudsdale/20171207.2150/backup.tar.gz
After that I can execute rear recover but it says that it can't
recover the system layout on /dev/sdf. I have no idea now what I
should do. /dev/sdf1 is my backupstorage0, which should be the only
thing that DOESN'T get recreated on recovery. I get a selection from 1-6
and I selected [6-Abort Relax-and-Recover]. I fear for my backup
archive to get corrupted.
AmonBune commented at 2017-12-07 21:45:¶
Okay, my backup drive is now corrupted thanks to the whole process. Seems like I have to setup the server again, sigh.
I don't get why ReaR tries to recover the backup drive itself. Did I forget to set a specific "exclude this-and-this folder" option in the local.config or is there something else I missed?
gozora commented at 2017-12-07 21:54:¶
I'm sorry to hear that :-(, I'm afraid that you've become victim of
https://github.com/rear/rear/issues/1271
.
I don't know how far ReaR went when formatting your USB, but you can try
ddrescue to recover you backup
...
V.
AmonBune commented at 2017-12-07 22:07:¶
I will try to extract all the configuration with ddrescue but it is not such a big deal since I have all my important data somewhere else and I wanted to migrate to something like FreeNAS anyway. It's just time consuming to setup a new OS.
Thanks anyway for your support and I hope I never have to run
rear recover ever again.
jsmeix commented at 2017-12-08 11:08:¶
@AmonBune
I know it does not at all help you right now,
nevertheless for the future I would like to mention
https://en.opensuse.org/SDB:Disaster_Recovery
AmonBune commented at 2017-12-08 17:09:¶
Well, what I learn from this: Make a backup of the backup and disconnect it on a restore process. It's good that I have copied all important data to a separate disk before. Only some configuration files etc. are gone now which is not a crisis. I will certainly look into that suse article you posted @jsmeix, thanks.
gdha commented at 2017-12-11 12:32:¶
@AmonBune The lesson that should be learned is "test on sandbox systems which are not important when something goes wrong" and even more important "test, test and test it again" before going to production.
AmonBune commented at 2017-12-11 12:43:¶
@gdha Good point, I actually did test it in the beginning and I also recovered successfully a few times but since the ReaR installation on the backup disk gave itself the final shot, emergency measurements had to be found. And they failed. Or I failed. Anyway, I have a new system now, so everything's fine... for now...
jsmeix commented at 2018-07-18 10:23:¶
Some final notes:
Since ReaR 2.3 we have an improved autodetection during 'rear recover'
when disks on the replacement hardware seem to not match
compared to what there was on the original system so that ReaR is now
more fail-safe against recreating on a possibly wrong disk, see
https://github.com/rear/rear/issues/1271
and
https://github.com/rear/rear/pull/1593
But that alone is not yet fully sufficient because there is still
a generic problem when a ReaR recovery disk is used, see
https://github.com/rear/rear/issues/1854
[Export of Github issue for rear/rear.]