#1578 Issue closed: Create a recovery RAW disk

Labels: enhancement, documentation, fixed / solved / done

GreenBlood opened issue at 2017-11-16 15:47:

Hi, I use rear as a disaster recovery software in my company ; We do DRP in the cloud through an automation process.
The issue (and enhancement suggestion) I have is that the Cloud infrastructure we use does not support ISO files as boot device, only classic disk images (like raw, vmdk, qcow2 and such), therefore, I cannot boot from the ISO to recover my machine.

I tried to "convert" the ISO file to an disk image but so far I couldn't make it work...

One possible workaround would be to use OUTPUT=USB and send the recovery environment to a mounted disk image which could be then booted on. The other, more practical, would be to add the possibility to create a RAW (or another format) disk image in place of the ISO file.

Does it seems like a feasable feature ?

Regards,
GreenBlood

gozora commented at 2017-11-16 16:26:

Hey @GreenBlood,

In my opinion this is interesting feature indeed (but I'm somehow strange, loving exotic things in ReaR c.f. https://github.com/rear/rear/pull/1172 :-)).
Just in theory (without knowing much about virtual disks) I think this is possible to implement, question is however effort ...
I can add this on my TODO list, but implementation date would be a riddle, as I have quite lot of work nowadays, so If you are willing to implement this, you are more then welcome ;-)

V.

jsmeix commented at 2017-11-17 09:14:

@GreenBlood
When you set in local.conf

KEEP_BUILD_DIR="yes"

and then run "rear mkrescue" you get
all the ReaR recovery system files
kept in /tmp/rear.XXXX/rootfs
(without KEEP_BUILD_DIR="yes" the /tmp/rear.XXXX
directory gets removed at the end of "rear mkrescue")
so that you could do whatever you want with
the ReaR recovery system files to manually create
any kind of "bootable thingy" as you need it
in your particular case.

When you know how to manually create what you need
it should be relatively simple and straightforward to
add support for new OUTPUT types in ReaR like

OUTPUT=RAWDISK
OUTPUT=VMDK
OUTPUT=QCOW2

Don't be afraid to do an initial GitHub pull request with
your enhancements. It does not need to be a "perfect
final solution". Any well documented enhancement is
greately appreciated. Actually the only real requirement
we have for GitHub pull request is that anybody can
fully understand what "great new stuff" really is about cf.
https://github.com/rear/rear/wiki/Coding-Style

For an outstanding example of perfectly documented
"great new stuff" you may have a look at
https://github.com/rear/rear/pull/1574

GreenBlood commented at 2017-11-17 12:17:

@jsmeix Thank you very much for your support, I'm rather new in the "community coding" universe so you make it less scary for me :)

Indeed, I had forgotten about the KEEP_BUILD_DIR option ; I'll try to look into it and to make up some code to implement this.

Edit : The main question i'm asking myself is that, by design, an ISO is not /dev/sd*, but a raw/vmdl/qcow is, so the recovery disk would have to detect and not try to restore on the first hard disk. For example, we backup /dev/sda but restore it as /dev/sdb because sda is the recovery disk.
Is it already handled by the recovery script or is it fixed ?

gdha commented at 2017-11-17 13:22:

@GreenBlood Could you explain in simple wording how you see this new feature to work with ReaR? How is the process working to generate a RAW image? Do you start from the KEEP_BUILD_DIR or ISO image? Would like to understand the process behind it.
And, secondly, how would the recover part work?
Sorry for these (stupid) questions, but I have no experience on that level (yet).

schlomo commented at 2017-11-17 14:14:

@GreenBlood did you try to use isohybrid? If it works then you can extend ReaR to require and run isohybrid after creating to ISO image 😄

GreenBlood commented at 2017-11-17 16:01:

@gdha
I was thinking of maybe using the rescue system in the "initrd" to create a small filesystem on a disk image.
In summary, imho, "rear mkrescue" would go like :

  1. Build the base rear rescue system
  2. Create a block device (with mknod for example) from an empty file and format it
  3. Mount the newly create partition and copy the rescue system onto this partition
  4. Create MBR and other boot-related process accordingly
  5. The result is a raw disk image. It can be dd on a thumbdrive or, in my case, uploaded to a Cloud provider to be started on (and to recover data in the cloud infrastructure)
  6. The image sits next to the backup data in the repository

But with @schlomo's idea it might not be necessary to go through all this

The issue it raised in my mind is that rear would then have to recover to the SECOND block device, and not the first as usual. I'm not really familiar with ReaR code structure so I guess the choice of the destination disk is made to be first disk in => first disk out which is not what we would want right here.

@schlomo I had never heard of this tool. Just tried it and I'm amazed at how fast it is and efficient ! When the MBR was added, I converted it to a qcow2 format and was able to boot from it seamlessly. So this is definetly a step in the right way. The major problem then is the one I talked about just above in this message

schlomo commented at 2017-11-18 16:32:

Are you sure that the ISO shows up as the first block device? I always thought that isohybrid works only on the BIOS level. Truth is, I haven't used isohybrid in many years and don't remember all the details.

jsmeix commented at 2017-11-20 09:30:

@GreenBlood
regarding "first disk in => first disk out" in your
https://github.com/rear/rear/issues/1578#issuecomment-345284227

It depends what exactly you mean with "first disk", "second disk".

During "rear mkrescue/mkbackup" ReaR saves the disk layout
(disks, partitions, filesystems, mountpoints, ...) of the original
system (where "rear mkrescue/mkbackup" runs) into the file
/tmp/rear.XXX/rootfs/root/rear.master/var/lib/rear/layout/disklayout.conf

ReaR saves the disk layout using traditional kernel block
device node names like /dev/sda /dev/sda1 ...
cf. https://github.com/rear/rear/issues/1063
and follow the links to other issues therein.

During "rear recover" ReaR recreates the system using those
device node names like /dev/sda /dev/sda1 ... as present on
the replacement hardware.

There is some basic autodetection during "rear recover" when
disks on the replacement hardware seem to not match what
there was on the original system. If a mismatch is autodetected
then ReaR goes into its "migration mode" (search the scripts
for MIGRATION_MODE) where it asks via user dialogs what to do.

Currently that autodetection is based on comparing disk size
cf. layout/prepare/default/250_compare_disks.sh

Accordingly things can go wrong when there are
several disks with same size, cf.
https://github.com/rear/rear/issues/1057#issuecomment-259390395

Things can even go terribly wrong when your USB disk
where your ReaR recovery system and backup is
has same size as your system harddisk, cf.
https://github.com/rear/rear/issues/1271

OliverO2 commented at 2017-11-21 19:18:

I came across this issue accidentally and it looks like I've already created something very close: Over here, an invocation of rear -v mkopalbpa creates a raw disk image with a minimal ReaR installation plus a combination of EFI+Legacy BIOS bootloaders (syslinux-efi or Grub 2 for EFI, syslinux for legacy BIOS). The result is a TCG Opal pre-boot authentication (PBA) image, which is meant to be stored in a special section on an Opal-compliant self-encrypting disk to support unlocking such disks via DTA sedutil-cli.

Currently I'm using the disk image for testing on VirtualBox (after running something like VBoxManage convertfromraw TCG-Opal-PBA.img TCG-Opal-PBA.vdi --format VDI) where the same image boots via EFI and Legacy BIOS. Copying the disk image via dd onto a USB device and booting from there on real hardware also works. At this time the bootloaders are minimal installations as the PBA area on Opal-compliant disk is of limited capacity (e.g. 128 MB or so on a Samsung 850 Pro SSD).

I've already thought about making this available as a universal output method not only for PBA images but also for complete ReaR rescue media. I should have the PBA stuff ready by next week.

Using a test configuration like this:

OUTPUT=OPALPBA
OPALPBA_URL="file:///mnt/reserve/Transfer/Rear.nobackup/OPALPBA"
OPALPBA_INCLUDE_SYSLINUX_EFI=no

this is what rear -v mkopalbpa output looks like:

Relax-and-Recover 2.2 / Git
Using log file: /home/oliver/Repositories/external/rear/var/log/rear/rear-foxtrot.log
Using UEFI Boot Loader for Linux (USING_UEFI_BOOTLOADER=1)
Creating disk layout
Ignoring non-existing btrfs subvolume listed as mounted: /var/lib/docker/btrfs
Using guessed bootloader 'EFI'
Creating root filesystem layout
Trying to find what to use as UEFI bootloader...
Trying to find a 'well known file' to be used as UEFI bootloader...
Using '/boot/efi/EFI/ubuntu/grubx64.efi' as UEFI bootloader file
Copying logfile /home/oliver/Repositories/external/rear/var/log/rear/rear-foxtrot.log into initramfs as '/tmp/rear-foxtrot-partial-2017-11-21T20:10:18+01:00.log'
Copying files and directories
Copying binaries and libraries
Copying only currently loaded kernel modules (MODULES contains 'loaded_modules')
Omit copying files in /lib*/firmware/ (FIRMWARE_FILES='no')
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (34180551 bytes) in 4 seconds
DISABLED: Using syslinux to create an EFI bootloader for the TCG Opal pre-boot authentication (PBA) image
Using Grub 2 to create an EFI bootloader for the TCG Opal pre-boot authentication (PBA) image
Creating TCG Opal pre-boot authentication (PBA) image (45 MiB)
Using syslinux to install a Legacy BIOS bootloader for the TCG Opal pre-boot authentication (PBA) image
Copying resulting files to file location
Saving /home/oliver/Repositories/external/rear/var/log/rear/rear-foxtrot.log as rear-foxtrot.log to file location

gdha commented at 2017-11-22 06:55:

@OliverO2 Looking forward to the PR - looks very promising 👍

GreenBlood commented at 2017-11-22 09:00:

@schlomo
Yup, but that might be my fault, as my cloud provider right now allows only disk image (like vhd or qcow or antything, hence my original question) so I had to convert it. I just tried again with just isohybrid'ing the file and launching it on a local hypervisor and indeed it displays as a standard disc, it's the convert part that does that.

@OliverO2 Sounds very nice! Out of curiosity, what does the Opal TCG thingy implies on the output disk image ?

OliverO2 commented at 2017-11-22 11:50:

@GreenBlood Regarding TCG Opal requirements and raw disk image output, there is really nothing special with these exceptions:

  • The disk image must be small (maximum 128 MB on a Samsung 850 Pro SSD).
  • For usability reasons a very quiet boot process is required (so there are no bootloader menus or other fancy stuff).

I have actually just adapted my scripts to provide a RAWDISK output method, so you can just configure ReaR like this:

OUTPUT=RAWDISK
OUTPUT_URL="file:///mnt/reserve/Transfer/Rear.nobackup/RAWDISK"

I have verified that I can boot a full ReaR rescue system from that disk image. If you'd like to check it out, I could push this to my ReaR fork, hopefully later today.

The bootable raw disk image doesn't really care about its boot device so if you can configure your cloud infrastructure to address the ReaR image as a second (third, or whatever) disk device, you should be fine.

jsmeix commented at 2017-11-22 12:08:

@OliverO2
I like your meaningful ReaR verbose mode messages as in
https://github.com/rear/rear/issues/1578#issuecomment-346132125
very much!
Regardless that I know nothing at all about
"TCG Opal pre-boot authentication (PBA)"
I get some basic understanding what goes on.
I look forward to your pull request!

I only wonder why you implemented
your new OUTPUT=OPALPBA method
as a complete new workflow "mkopalbpa"?

OliverO2 commented at 2017-11-22 12:42:

@jsmeix Thanks!

Envision a PBA as a small black-box program that

  1. is booted instead of your regular operating system when your boot disk is locked,
  2. asks for your boot disk password,
  3. unlocks your boot disk,
  4. initiates a reboot to start the regular operating system.

The trick is that a locked Opal-compliant disk will present itself as a smaller disk, which just contains the PBA. The firmware will then try to boot that PBA. As soon as the Opal disk is unlocked, it will show its real contents and the PBA will be hidden when rebooting into the regular OS.

In my implementation, there is a complete rear-built system acting as the PBA.

The PBA creation is more than just a new output method: Over here, a complete rescue system is about 190 MiB in size while the PBA is just 45 MiB. Creating a PBA should be possible without changing the ReaR configuration. So you should be able to use rear mkrescue to create the rescue system as ususal and rear mkopalpba to create the PBA, both from a single ReaR configuration.

I actually intend to go further and add another installopalpba workflow. This is the usage information currently provided with my RAWDISK output method:

How to recover your system:
---------------------------

To recover your system, copy the attached raw disk image onto a disk device (a USB stick will do).
Do not use a partition (like /dev/sdc1), use the complete disk device (like /dev/sdc).
Be careful, this will overwrite disk contents.

Example:

   dd if=<image> bs=1M of=<device>

Then boot from the disk device into the rescue system.



How to install a TCG Opal pre-boot authentication (PBA) image
-------------------------------------------------------------


a) To enable TCG Opal support and install the PBA image onto a disk device:

      sudo rear installopalpba --setup [<image>] <device>


b) To replace the PBA image on a disk device where TCG Opal support has been enabled:

      sudo rear installopalpba [<image>] <device>


Note: If the <image> parameter is not provided, ReaR will either
- use the provided image when called on a rescue system, or
- generate a new image.

OliverO2 commented at 2017-11-22 15:29:

@GreenBlood and everyone interested:

At https://github.com/OliverO2/rear/tree/feature/opal-support please find my incarnation of a RAWDISK output method.

Just configure like this and you should be ready to go:

OUTPUT=RAWDISK
OUTPUT_URL="file:///your/preferred/output/location"

This is the first step towards TCG Opal support but the output method works independently.

Tested on Ubuntu 16.04.3 LTS.

GreenBlood commented at 2017-11-24 16:34:

@OliverO2 It's really working great and the resulting image is perfectly working, I only have one issue, some of my servers are not on an EFI bootloader so the script trying to create EFI bootloader(grub-mkimage i think) fails because it cannot find the required files.

2017-11-24 16:29:20.774205693 Including output/RAWDISK/Linux-i386/270_create_grub2_efi_bootloader.sh
2017-11-24 16:29:20.778481003 Using Grub 2 to create an EFI bootloader
mkdir: created directory '/tmp/rear.ucQzU9aAufOZiZ0/EFI'
mkdir: created directory '/tmp/rear.ucQzU9aAufOZiZ0/EFI/BOOT'
grub-mkimage: error: cannot open /usr/lib/grub/x86_64-efi/moddep.lst': No such file or directory.
2017-11-24 16:29:20.787039299 ERROR: Error occurred during grub-mkimage of /tmp/rear.ucQzU9aAufOZiZ0/EFI/BOOT/BOOTX64.efi

I had to install the efi related packages to move on from there
Tomorrow i'll try to remove the efi-related bash scripts (numbered 260 and 270 iirc) and see what's happening, but this whole feature looks promising :)

OliverO2 commented at 2017-11-24 16:56:

@GreenBlood Good to hear that it's working in general, thanks for your feedback. Seems like the autodetection for Grub 2/EFI needs to improve.

This is how I'd diagnose your situation:

  • The scripts for the EFI bootloaders try to auto-detect if their respective bootloaders can actually be built, before trying to create their respective bootloader.
  • The syslinux/EFI detection works as you don't get any error messages from there. 260_create_syslinux_efi_bootloader.sh just determines that the EFI variant of syslinux is not available and proceeds without trying to build a syslinux-based EFI bootloader.
  • The detection for Grub 2 succeeds in checking for grub-mkimage and thinks a complete Grub 2 installation (including EFI) is available. 270_create_grub2_efi_bootloader.sh then tries to build a Grub 2-based EFI bootloader but fails as required files are missing.

Fortunately, you can use configuration variables to prevent EFI bootloaders from being created:

  • For syslinux/EFI you can set RAWDISK_INCLUDE_SYSLINUX_EFI=no
  • For Grub 2/EFI you can set RAWDISK_INCLUDE_GRUB2_EFI=no

So in your case, it should be sufficient to just set RAWDISK_INCLUDE_GRUB2_EFI=no. Then everything should work without any change to scripts.

Hope this helps.

OliverO2 commented at 2017-11-24 17:45:

@GreenBlood A quick fix: Try to change lines 7-8 in 270_create_grub2_efi_bootloader.sh to:

# Run only if no EFI bootloader has been created yet and Grub 2 EFI is available
([[ -n "$RAWDISK_EFI_STAGING_ROOT" ]] || ! has_binary grub-mkimage || ! [[ -d /usr/lib/grub/x86_64-efi ]]) && return 0

GreenBlood commented at 2018-01-08 15:58:

@OliverO2 I just saw your code got merged into the main branch. Thank you very much for your work on this 👍

OliverO2 commented at 2018-01-08 16:11:

@GreenBlood Apparently it's about to get merged, but not done yet. Thanks for your feedback!

jsmeix commented at 2018-01-09 10:10:

With https://github.com/rear/rear/pull/1659 merged
I consider this issue to be fixed.

@OliverO2
many thanks for your huge contribution to ReaR!

@GreenBlood
FYI how to test the current ReaR GitHub master code:

Basically "git clone" it into a directory and then
configure and run ReaR from within that directory like:

# git clone https://github.com/rear/rear.git

# cd rear

# vi etc/rear/local.conf

# usr/sbin/rear -D mkbackup

(note the relative paths "etc/rear/" and "usr/sbin/").

GreenBlood commented at 2018-01-09 14:55:

@jsmeix Yup dont worry about me I'm using a bunch of temporary VMs and building the deb file (to integrate it with my others scripts). Thank you tho !

[Export of Github issue for rear/rear.]