#1795 Issue closed: Restoring to a bigger HD fails, due to over increasing the 2nd. last (data-)partition, when the following one is a non-data-partion

Labels: support / question, fixed / solved / done

chymian opened issue at 2018-05-03 16:15:

  • ReaR version: Relax-and-Recover 2.3 / Git from debian buster 2.3+dfsg-1
  • OS version: debian stretch
  • System architecture (x86 compatible): amd64 bare metal
  • Are you using BIOS or UEFI: BIOS
  • Brief description of the issue: if the disk to restore to is bigger than the original, diskrestore.sh fails due to overexpanding the last data-partition, which is followed by a non-data partition. It then has no space left.
  • Work-around, if any: manual edit diskrestore.sh

There a two points here:

  1. ReaR grows – without consent – the partitions to max. of disk (100%) so that the last part. – in this case bios_grub with a size of only 860K – does not have any space left.
    There seems to be an bug in the size-calculation/does not handle the case, when a non-data partition is the last one.

  2. May I suggest to set the default behavior to create the partitions in the same size, if the disk is bigger than the original.
    For autoresizing: it could ask whether it should increase all/some partitions by an percental/absolut value. Not every data-partiton needs an increase: i.e. efi/root/boot

I know, there is the possibility to manual edit the diskrestore.sh and work with Bytes-values, which is very uncomfortable.

The point here is, that a simple restore (which can be done by any user), as it is with restoring to a same-size disk is not possible and therefore needs an admin onsite.

Also it would help a lot if an ssh-server (with the same keys) would be started on the recovery-image and even an openvpn connection (and/or the whole Network-Manager Config w/ plugins) if existent on the original host – to make offsite-restore/help possible.

For debugging purposes it would also help if there are more consoles (gettys) availlable.

Original Disklayout:

parted /dev/sdb p free
Model: ATA OCZ-VECTOR150 (scsi)
Disk /dev/sdb: 480GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system     Name       Flags
        17,4kB  1049kB  1031kB  Free Space
 1      1049kB  98,6MB  97,5MB  fat32           efi        boot, esp
 2      98,6MB  50,0GB  49,9GB  btrfs           root
 3      50,0GB  66,0GB  16,0GB  linux-swap(v1)  swap
 4      66,0GB  480GB   414GB                   lvm1       lvm
 5      480GB   480GB   860kB                   bios_grub  bios_grub

Created Disklayout:

Model: ATA Samsung SSD 860 (scsi)
Disk /dev/sdb: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
        17.4kB  2097kB  2080kB  Free Space
 1      2097kB  209MB   207MB                efi   boot, esp
        209MB   209MB   1024B   Free Space
 2      209MB   106GB   106GB                root
        106GB   106GB   1536B   Free Space
 3      106GB   122GB   16.0GB               swap
        122GB   122GB   4096B   Free Space
 4      122GB   1000GB  878GB                lvm1  lvm

the same layout with units set to Bytes

Model: ATA Samsung SSD 860 (scsi)
Disk /dev/sdb: 1000204886016B
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start          End             Size           File system  Name  Flags
        17408B         2097151B        2079744B       Free Space
 1      2097152B       208899071B      206801920B                  efi   boot, esp
        208899072B     208900095B      1024B          Free Space
 2      208900096B     106033576447B   105824676352B               root
        106033576448B  106033577983B   1536B          Free Space
 3      106033577984B  122033799167B   16000221184B                swap
        122033799168B  122033803263B   4096B          Free Space
 4      122033803264B  1000204869119B  878171065856B               lvm1  lvm

rear-hansa.log

...
+++ create_component /dev/sdb disk
+++ local device=/dev/sdb
+++ local type=disk
+++ local touchfile=disk--dev-sdb
+++ '[' -e /tmp/rear.uDRcanrrPOzMLnF/tmp/touch/disk--dev-sdb ']'
+++ return 0
+++ Log 'Stop mdadm'
++++ date '+%Y-%m-%d %H:%M:%S.%N '
+++ local 'timestamp=2018-05-03 15:52:48.374346194 '
+++ test 1 -gt 0
+++ echo '2018-05-03 15:52:48.374346194 Stop mdadm'
2018-05-03 15:52:48.374346194 Stop mdadm
+++ grep -q md /proc/mdstat
+++ Log 'Erasing MBR of disk /dev/sdb'
++++ date '+%Y-%m-%d %H:%M:%S.%N '
+++ local 'timestamp=2018-05-03 15:52:48.378571303 '
+++ test 1 -gt 0
+++ echo '2018-05-03 15:52:48.378571303 Erasing MBR of disk /dev/sdb'
2018-05-03 15:52:48.378571303 Erasing MBR of disk /dev/sdb
+++ dd if=/dev/zero of=/dev/sdb bs=512 count=1
1+0 records in
1+0 records out
512 bytes copied, 0.000307959 s, 1.7 MB/s
+++ sync
+++ LogPrint 'Creating partitions for disk /dev/sdb (gpt)'
+++ Log 'Creating partitions for disk /dev/sdb (gpt)'
++++ date '+%Y-%m-%d %H:%M:%S.%N '
+++ local 'timestamp=2018-05-03 15:52:48.383776293 '
+++ test 1 -gt 0
+++ echo '2018-05-03 15:52:48.383776293 Creating partitions for disk /dev/sdb (gpt)'
2018-05-03 15:52:48.383776293 Creating partitions for disk /dev/sdb (gpt)
+++ Print 'Creating partitions for disk /dev/sdb (gpt)'
+++ test 1
+++ echo -e 'Creating partitions for disk /dev/sdb (gpt)'
+++ my_udevsettle
+++ has_binary udevadm
+++ for bin in $@
+++ type udevadm
+++ return 0
+++ udevadm settle
+++ return 0
+++ parted -s /dev/sdb mklabel gpt
+++ my_udevsettle
+++ has_binary udevadm
+++ for bin in $@
+++ type udevadm
+++ return 0
+++ udevadm settle
+++ return 0
+++ my_udevsettle
+++ has_binary udevadm
+++ for bin in $@
+++ type udevadm
+++ return 0
+++ udevadm settle
+++ return 0
+++ parted -s /dev/sdb mkpart efi 2097152B 208898892B
+++ my_udevsettle
+++ has_binary udevadm
+++ for bin in $@
+++ type udevadm
+++ return 0
+++ udevadm settle
+++ return 0
+++ my_udevsettle
+++ has_binary udevadm
+++ for bin in $@
+++ type udevadm
+++ return 0
+++ udevadm settle
+++ return 0
+++ parted -s /dev/sdb set 1 boot on
+++ my_udevsettle
+++ has_binary udevadm
+++ for bin in $@
+++ type udevadm
+++ return 0
+++ udevadm settle
+++ return 0
+++ my_udevsettle
+++ has_binary udevadm
+++ for bin in $@
+++ type udevadm
+++ return 0
+++ udevadm settle
+++ return 0
+++ parted -s /dev/sdb set 1 esp on
+++ my_udevsettle
+++ has_binary udevadm
+++ for bin in $@
+++ type udevadm
+++ return 0
+++ udevadm settle
+++ return 0
+++ my_udevsettle
+++ has_binary udevadm
+++ for bin in $@
+++ type udevadm
+++ return 0
+++ udevadm settle
+++ return 0
+++ parted -s /dev/sdb name 1 efi
+++ my_udevsettle
+++ has_binary udevadm
+++ for bin in $@
+++ type udevadm
+++ return 0
+++ udevadm settle
+++ return 0
+++ my_udevsettle
+++ has_binary udevadm
+++ for bin in $@
+++ type udevadm
+++ return 0
+++ udevadm settle
+++ return 0
+++ parted -s /dev/sdb mkpart root 208900096B 106033576083B
Warning: The resulting partition is not properly aligned for best performance.
+++ my_udevsettle
+++ has_binary udevadm
+++ for bin in $@
+++ type udevadm
+++ return 0
+++ udevadm settle
+++ return 0
+++ my_udevsettle
+++ has_binary udevadm
+++ for bin in $@
+++ type udevadm
+++ return 0
+++ udevadm settle
+++ return 0
+++ parted -s /dev/sdb name 2 root
+++ my_udevsettle
+++ has_binary udevadm
+++ for bin in $@
+++ type udevadm
+++ return 0
+++ udevadm settle
+++ return 0
+++ my_udevsettle
+++ has_binary udevadm
+++ for bin in $@
+++ type udevadm
+++ return 0
+++ udevadm settle
+++ return 0
+++ parted -s /dev/sdb mkpart swap 106033577984B 122033799167B
Warning: The resulting partition is not properly aligned for best performance.
+++ my_udevsettle
+++ has_binary udevadm
+++ for bin in $@
+++ type udevadm
+++ return 0
+++ udevadm settle
+++ return 0
+++ my_udevsettle
+++ has_binary udevadm
+++ for bin in $@
+++ type udevadm
+++ return 0
+++ udevadm settle
+++ return 0
+++ parted -s /dev/sdb name 3 swap
+++ my_udevsettle
+++ has_binary udevadm
+++ for bin in $@
+++ type udevadm
+++ return 0
+++ udevadm settle
+++ return 0
+++ my_udevsettle
+++ has_binary udevadm
+++ for bin in $@
+++ type udevadm
+++ return 0
+++ udevadm settle
+++ return 0
+++ parted -s /dev/sdb mkpart lvm1 122033803264B 1000204869119B
Warning: The resulting partition is not properly aligned for best performance.
+++ my_udevsettle
+++ has_binary udevadm
+++ for bin in $@
+++ type udevadm
+++ return 0
+++ udevadm settle
+++ return 0
+++ my_udevsettle
+++ has_binary udevadm
+++ for bin in $@
+++ type udevadm
+++ return 0
+++ udevadm settle
+++ return 0
+++ parted -s /dev/sdb set 4 lvm on
+++ my_udevsettle
+++ has_binary udevadm
+++ for bin in $@
+++ type udevadm
+++ return 0
+++ udevadm settle
+++ return 0
+++ my_udevsettle
+++ has_binary udevadm
+++ for bin in $@
+++ type udevadm
+++ return 0
+++ udevadm settle
+++ return 0
+++ parted -s /dev/sdb name 4 lvm1
+++ my_udevsettle
+++ has_binary udevadm
+++ for bin in $@
+++ type udevadm
+++ return 0
+++ udevadm settle
+++ return 0
+++ my_udevsettle
+++ has_binary udevadm
+++ for bin in $@
+++ type udevadm
+++ return 0
+++ udevadm settle
+++ return 0
+++ parted -s /dev/sdb mkpart bios_grub 1000205168640B 1000204869119B
Error: The location 1000205168640B is outside of the device /dev/sdb.

diskrestore.sh

#!/bin/bash

LogPrint "Start system layout restoration."

mkdir -p /mnt/local
if create_component "vgchange" "rear" ; then
    lvm vgchange -a n >/dev/null
    component_created "vgchange" "rear"
fi

set -e
set -x

if create_component "/dev/sdb" "disk" ; then
# Create /dev/sdb (disk)
Log "Stop mdadm"
if grep -q md /proc/mdstat &>/dev/null; then
    mdadm --stop -s >&2 || echo "stop mdadm failed"
    # Prevent udev waking up mdadm later.
    # Reasoning: At least on RHEL6 when parted created a raid partition on disk,
    # udev (via /lib/udev/rules.d/65-md-incremental.rules) wakes up mdadm which locks the disk,
    # so further parted commands with the disk will fail since the disk is busy now.
    # The /lib/udev/rules.d/65-md-incremental.rules detects anaconda (the Red Hat installer),
    # and if it find itself running under anaconda, it will not run.
    # Accordingly also for other installers (in particular the ReaR installer)
    # this rule should not be there (and other Linux distros probably do not have it)
    # which means removing it is the right solution to make ReaR work also for RHEL6:
    if [ -e /lib/udev/rules.d/65-md-incremental.rules ] ; then
        rm -f /lib/udev/rules.d/65-md-incremental.rules || echo "rm 65-md-incremental.rules failed"
    fi
fi
Log "Erasing MBR of disk /dev/sdb"
dd if=/dev/zero of=/dev/sdb bs=512 count=1
sync
LogPrint "Creating partitions for disk /dev/sdb (gpt)"
my_udevsettle
parted -s /dev/sdb mklabel gpt >&2
my_udevsettle
my_udevsettle
parted -s /dev/sdb mkpart 'efi' 2097152B 208898892B >&2
my_udevsettle
my_udevsettle
parted -s /dev/sdb set 1 boot on >&2
my_udevsettle
my_udevsettle
parted -s /dev/sdb set 1 esp on >&2
my_udevsettle
my_udevsettle
parted -s /dev/sdb name 1 'efi' >&2
my_udevsettle
my_udevsettle
parted -s /dev/sdb mkpart 'root' 208900096B 106033576083B >&2
my_udevsettle
my_udevsettle
parted -s /dev/sdb name 2 'root' >&2
my_udevsettle
my_udevsettle
parted -s /dev/sdb mkpart 'swap' 106033577984B 122033799167B >&2
my_udevsettle
my_udevsettle
parted -s /dev/sdb name 3 'swap' >&2
my_udevsettle
my_udevsettle
parted -s /dev/sdb mkpart 'lvm1' 122033803264B 1000204869119B >&2
my_udevsettle
my_udevsettle
parted -s /dev/sdb set 4 lvm on >&2
my_udevsettle
my_udevsettle
parted -s /dev/sdb name 4 'lvm1' >&2
my_udevsettle
my_udevsettle
parted -s /dev/sdb mkpart 'bios_grub' 1000205168640B 1000204869119B >&2
my_udevsettle
my_udevsettle
parted -s /dev/sdb set 5 bios_grub on >&2
my_udevsettle
my_udevsettle
parted -s /dev/sdb name 5 'bios_grub' >&2
my_udevsettle
sleep 1
...

jsmeix commented at 2018-05-04 07:45:

As far as I see on first glance this issue proves that what is called
Relax-and-Recover 2.3 / Git from debian buster 2.3+dfsg-1
is not a recent ReaR upstream GitHub master code
(but probably the 2.3 release plus whatever GitHub stuff),
cf. https://github.com/rear/rear/issues/1793#issuecomment-386309765

Revovery on a bigger HD means you are in MIGRATION_MODE and
when in MIGRATION_MODE any non-last partition (e.g. the 2nd. last partition)
gets changed if you do not use the current ReaR upstream GitHub master code.

The current ReaR upstream GitHub master code changes
by default - if at all - at most the last partition.

Without reading all your detailed information (many thanks for that!)
I assume you are hit by one of the issues that are described at
https://github.com/rear/rear/issues/1731
and
https://github.com/rear/rear/pull/1733
or by a related issue like
https://github.com/rear/rear/issues/102

I recommend to try out our current ReaR upstream GitHub master code
because that is the only place where we at ReaR upstream fix bugs.
Bugs in released ReaR versions are not fixed by us (by ReaR upstream).
Bugs in released ReaR versions that got fixed in current ReaR upstream
GitHub master code might be fixed (if the fix can be backported with
reasonable effort) by the Linux distributor wherefrom you got ReaR.

To use our current ReaR upstream GitHub master code
do the following:

Basically "git clone" it into a separated directory and then
configure and run ReaR from within that directory like:

# git clone https://github.com/rear/rear.git

# mv rear rear.github.master

# cd rear.github.master

# vi etc/rear/local.conf

# usr/sbin/rear -D mkbackup

Note the relative paths "etc/rear/" and "usr/sbin/".

If the issue also happens with current ReaR upstream GitHub master code
please provide us a complete ReaR debug log file of "rear -D recover"
plus your initial/original disklayout.conf file from your original system
so that we can have a look how it behaves in your particular environment
cf. "Debugging issues with Relax-and-Recover" at
https://en.opensuse.org/SDB:Disaster_Recovery

If it works with current ReaR upstream GitHub master code
we would really appreciate an explicit positive feedback.

jsmeix commented at 2018-05-04 08:52:

@chymian
I think my current code in
usr/share/rear/layout/prepare/default/420_autoresize_last_partitions.sh
also handles the case when the last partition is not a mounted partition
on the original system while "rear mkrescue" runs because
I assume also non-mounted partitions are stored during "rear mkrescue"
in disklayout.conf as active (i.e. non-commented) part ... entries and
my current code in 420_autoresize_last_partitions.sh works on
what is stored in disklayout.conf as non-commented part ... entries.

But there is the assumption that the last partition
is the one with biggest partition start(bytes) value
and not the last entry in the partition table on the disk.

E.g. if you have (with MBR partitioning table)
/dev/sda2 from 300MiB to 400MiB and
/dev/sda4 from 100MiB to 200MiB
then /dev/sda2 is "the last partition" (and not /dev/sda4).

jsmeix commented at 2018-05-04 09:12:

@chymian
regarding ssh-server (with the same keys) would be started in the recovery system
please read the description about the SSH_* config variables
SSH_FILES
SSH_ROOT_PASSWORD
SSH_UNPROTECTED_PRIVATE_KEYS
in default.conf (the new SSH_* config variables exist since the ReaR 2.3 release).

jsmeix commented at 2018-07-19 12:20:

Because there are no further comments I assume
it works sufficiently with our ReaR upstream GitHub master code
so that this isssue was sufficiently answered and handled
so that I can close it hereby.

[Export of Github issue for rear/rear.]