#1850 PR merged: RAWDISK output portability improvements (fix #1846)

Labels: enhancement, fixed / solved / done

OliverO2 opened issue at 2018-07-03 21:05:

Relax-and-Recover (ReaR) Pull Request Template

Please fill in the following items before submitting a new pull request:

Pull Request Details:

  • Type: Enhancement

  • Impact: Normal

  • Reference to related issue (URL): https://github.com/rear/rear/issues/1846

  • How was this pull request tested? Generated RAWDISK output on Ubuntu 16.04.4 LTS and CentOS 6.10

  • Brief description of the changes in this pull request: Enables RAWDISK output for kernels which do not recognize loop device partitions. Should work with any distribution where the kpartx tool is available.

jsmeix commented at 2018-07-04 07:43:

@schlomo @gdha
I assigned this pull request to you because I cannot merge it
because currently it shows

Merging is blocked
The target branch requires all commits to be signed.

cf. https://github.com/rear/rear/pull/1849#issuecomment-402125330
and the "Merge pull request" button is grayed out for me (using Firefox on Linux)
cf. https://github.com/rear/rear/pull/1849#issuecomment-402188400

schlomo commented at 2018-07-04 07:51:

If I understand the documentation correctly the button is greyed out because the commits are not signed (do we have another example of a PR where the commits are actually signed?).

In this case the correct way would be for one of us to merge the PR on a client and use that way to sign that commit.

I guess we have to decide between the convenience of merging unsigned commits in the GitHub GUI and having signed commits in our source code. Classical security vs. convenience decision :-(

OliverO2 commented at 2018-07-04 08:09:

I'd be willing to contribute signed commits in the future. I hope the hurdles won't deter others. The node.js folks already had a detailed discussion on the issue: https://github.com/nodejs/node/issues/15457

I wonder how this could happen apparently without any maintainer of this repo actively enabling it...

jsmeix commented at 2018-07-04 08:39:

@OliverO2
for ReaR this did not happen magically, see
https://github.com/rear/rear/issues/1419#issuecomment-397034110

GreenBlood commented at 2018-07-04 08:48:

Just as a heads up regarding this PR and #1846
Tested by @OliverO2 and myself on CentOS 6,
by myself on old unsupported debian, and on recent openSUSE. This chunk of code seems to make the process more robust on a variety of distros.

OliverO2 commented at 2018-07-04 09:02:

@GreenBlood Excellent! Thanks for reporting. Did you verify that the generated rescue system actually works on CentOS 6 and the others? (I have just tested that it boots.)

@jsmeix I see. Thanks for explaining.

GreenBlood commented at 2018-07-04 09:28:

@OliverO2 Affirmative, I started the debian 7, centos6 and openSUSE rescue system successfully and was able to restore datas onto their disks.

jsmeix commented at 2018-07-04 09:39:

@GreenBlood
and after "rear recover" finished
did the recreated systems even boot on their own
and perhaps even actually work again?
;-)

More seriously:
I wonder if there is a generic way how to find out with reasonable effort
if there are noteworthy differences between the original system
and the recreated system.
I mean during testing when the original system is still available for comparison
(not after an actual disaster when the original system was destroyed).

GreenBlood commented at 2018-07-04 09:49:

@jsmeix Haha, yeah of course the systems boots and from what I see it works as expected :-) The only thing RAWDISK rescue would need at this stage is auto mapping the disks, because in ISO mode source sda is destination sda, as with RAWDISK, sda is the rescue system and sdb the first destination disk. But it's another problem

My guess on checking restoration would be to compare package list, or maybe launched services/open tcp ports. This list could even be included in the saved data, as a mean to validate, at the end of recovery, that the system looks like the original one.

OliverO2 commented at 2018-07-04 10:04:

@GreenBlood Perfect! :-)

@jsmeix In my view it's mostly the backup tool's responsibility to ensure accurate restoration. ReaR does the necessary setup that such restore can actually take place and is responsible for accurately re-creating the drive and partition setup. So the generic test for ReaR would be to
a) verify that restoring a backup actually works, and
b) comparing the drive and partition configuration.

a) is trivial: If restore works, the test ist successful. For b) I could envision some script, written independently, not relying on any ReaR code, generating its own text files describing the configurations (i.e. not using disklayout.conf), which could then be compared. Writing such a script is probably not trivial given all the possible variants.

jsmeix commented at 2018-07-04 10:34:

@GreenBlood
regarding your

RAWDISK rescue would need at this stage is auto mapping the disks,
as in ISO mode source sda is destination sda,
as with RAWDISK, sda is the rescue system and sdb the first destination disk

That sda is the rescue system and sdb the first destination disk
is the same as the known issue when using USB e.g. like

OUTPUT=USB
USB_DEVICE=/dev/disk/by-label/REAR-000
USB_SUFFIX="mybackup"
BACKUP=NETFS
BACKUP_URL=usb:///dev/disk/by-label/REAR-000

cf. https://github.com/rear/rear/issues/1271
and the section about MIGRATION_MODE in usr/share/rear/conf/default.conf

I would expect that the therein mentioned

# There is some basic autodetection during "rear recover" when
# disks on the replacement hardware seem to not match compared to
# what there was stored in disklayout.conf on the original system.

also happens during "rear recover" in case of RAWDISK.

That basic autodetection is implemented in
usr/share/rear/layout/prepare/default/250_compare_disks.sh
and when MIGRATION_MODE is set the mappings are done in
usr/share/rear/layout/prepare/default/300_map_disks.sh

@OliverO2
I wonder if usr/share/rear/layout/prepare/default/250_compare_disks.sh
is perhaps not run in case of RAWDISK or if it is run
but fails to autodetect an "Ambiguous disk layout" in case of RAWDISK?

OliverO2 commented at 2018-07-04 10:44:

@jsmeix I don't see any reason why the system should behave differently just because of RAWDISK.

Whether the boot device appears as sda or something else depends on how the BIOS reports disk devices the device's controller type (SATA, SCSI, IDE, ...) and (in case of multiple controllers of the same type) the order of initialization (which may change between boots).

If you just dd the RAWDISK image onto a USB stick, disk naming and ReaR behavior should be exactly the same as if you created the recovery image directly via OUTPUT=USB. (I always create USB rescue media with RAWDISK to avoid the formatting step.)

Edit: Corrected section on device naming.

jsmeix commented at 2018-07-04 12:20:

@OliverO2
I also don't see a reason why "rear recover"
should behave differently because of RAWDISK.
When I have in etc/rear/local.conf

OUTPUT=RAWDISK
OUTPUT_URL="file://$VAR_DIR/output"

and run usr/sbin/rear -s recover I get (excerpts)

Source layout/prepare/default/250_compare_disks.sh
...
Source layout/prepare/default/300_map_disks.sh

which shows that those scripts are executed (sourced) but
I don't know how the code in those scripts actually behaves
in case of RAWDISK - this is the reason why I asked.
E.g. when MIGRATION_MODE is not set in 250_compare_disks.sh
then 300_map_disks.sh is run but it does actually nothing.

@GreenBlood
can you provide what you get on the terminal plus the debug log
when you run "rear -D recover" with RAWDISK on one of your systems
where sda is the first system disk on the original system but during
"rear recover" sda is the rescue system and sdb is the first destination disk
so that I could understand why the "Ambiguous disk layout" autodetection
does not work in your particular case.
Please do it as a new separated GitHub issue.

OliverO2 commented at 2018-07-04 13:02:

@jsmeix Being just another output method, the RAWDISK stuff is pretty much isolated from everything else:

  • Code files:
    • usr/share/rear/conf/templates/RESULT_usage_RAWDISK.txt
    • usr/share/rear/output/RAWDISK/Linux-i386/*
  • Code changes:
    • RAWDISK_-prefixed variables in usr/share/rear/conf/default.conf

That's it. (There is, of course, the OPALPBA stuff which makes use of RAWDISK but it's a separate workflow, which is not active here).

jsmeix commented at 2018-07-09 09:27:

@OliverO2
many thanks for your enhancement
that fixes https://github.com/rear/rear/issues/1846

[Export of Github issue for rear/rear.]