#2693 PR merged: Overhauled automapping in 300_map_disks.sh¶
Labels: enhancement, fixed / solved / done, minor bug
jsmeix opened issue at 2021-10-05 15:19:¶
-
Type: Enhancement
-
Impact: Normal
-
Reference to related issue (URL):
https://github.com/rear/rear/issues/2690 -
How was this pull request tested?
"rear mkbackup" and "rear recover"
on two KVM test systems with 4 disks
original system replacement system
sda 2.1 GiB sda 2.2 GiB
sdb 2.4 GiB sdb 2.1 GiB
sdc 2.3 GiB sdc 2.4 GiB
sdd 2.2 GiB sdd 2.3 GiB
"rear -D recover" terminal output (excerpt)
Comparing disks
Device sda has size 2362232832 bytes but 2254858240 bytes is expected (needs manual configuration)
Device sdb has size 2254858240 bytes but 2576980992 bytes is expected (needs manual configuration)
Device sdc has size 2576980992 bytes but 2469606400 bytes is expected (needs manual configuration)
Device sdd has size 2469606400 bytes but 2362232832 bytes is expected (needs manual configuration)
Switching to manual disk layout configuration
Using /dev/sdb (same size) for recreating /dev/sda
Using /dev/sdc (same size) for recreating /dev/sdb
Using /dev/sdd (same size) for recreating /dev/sdc
Using /dev/sda (same size) for recreating /dev/sdd
Current disk mapping table (source => target):
/dev/sda => /dev/sdb
/dev/sdb => /dev/sdc
/dev/sdc => /dev/sdd
/dev/sdd => /dev/sda
UserInput -I LAYOUT_MIGRATION_CONFIRM_MAPPINGS needed in /usr/share/rear/layout/prepare/default/300_map_disks.sh line 291
Confirm or edit the disk mapping
1) Confirm disk mapping and continue 'rear recover'
2) n/a
3) Edit disk mapping (/var/lib/rear/layout/disk_mappings)
4) Use Relax-and-Recover shell and return back to here
5) Abort 'rear recover'
- Brief description of the changes in this pull request:
In layout/prepare/default/300_map_disks.sh overhauled the
automapping of original 'disk' devices and 'multipath' devices
to current block devices in the currently running recovery system
so that now it automatically finds an existing unique disk size
mapping.
For details see the comments in the code that should (hopefully)
explain
what goes on.
jsmeix commented at 2021-10-06 12:46:¶
Another test with latest
https://github.com/rear/rear/pull/2693/commits/62a99159bc90daa86155d4dacb7c6312ec51eac7
"rear mkbackup" and "rear recover"
on two KVM test systems with 3 disks
original system replacement system
sda 3 GiB sda 1 GiB
sdb 2 GiB sdb 2 GiB
sdc 1 GiB sdc 3 GiB
"rear -D recover" terminal output (excerpt)
Comparing disks
Device sda has size 1073741824 bytes but 3221225472 bytes is expected (needs manual configuration)
Device sdb has expected (same) size 2147483648 bytes (will be used for 'recover')
Device sdc has size 3221225472 bytes but 1073741824 bytes is expected (needs manual configuration)
Switching to manual disk layout configuration
Using /dev/sdc (same size 3221225472) for recreating /dev/sda
Using /dev/sdb (same name and same size 2147483648) for recreating /dev/sdb
Using /dev/sda (same size 1073741824) for recreating /dev/sdc
Current disk mapping table (source => target):
/dev/sda => /dev/sdc
/dev/sdb => /dev/sdb
/dev/sdc => /dev/sda
UserInput -I LAYOUT_MIGRATION_CONFIRM_MAPPINGS needed in /usr/share/rear/layout/prepare/default/300_map_disks.sh line 291
Confirm or edit the disk mapping
1) Confirm disk mapping and continue 'rear recover'
2) n/a
3) Edit disk mapping (/var/lib/rear/layout/disk_mappings)
4) Use Relax-and-Recover shell and return back to here
5) Abort 'rear recover'
gdha commented at 2021-10-06 13:28:¶
@jsmeix please also make a test with disk sizes completely different - not the same as the originals. What will happen then?
jsmeix commented at 2021-10-07 11:23:¶
One more test with
"rear mkbackup" and "rear recover"
on two KVM test systems with 3 disks
original system replacement system
sda 3 GiB sda 1 GiB
sdb 2 GiB sdb 4 GiB
sdc 1 GiB sdc 3 GiB
"rear -D recover" terminal output (excerpt)
Comparing disks
Device sda has size 1073741824 bytes but 3221225472 bytes is expected (needs manual configuration)
Device sdb has size 4294967296 bytes but 2147483648 bytes is expected (needs manual configuration)
Device sdc has size 3221225472 bytes but 1073741824 bytes is expected (needs manual configuration)
Switching to manual disk layout configuration
Using /dev/sdc (same size 3221225472) for recreating /dev/sda
Could not automap /dev/sdb (no disk with same size 2147483648 found)
Using /dev/sda (same size 1073741824) for recreating /dev/sdc
Original disk /dev/sdb does not exist (with same size) in the target system
Using /dev/sdb (the only appropriate) for recreating /dev/sdb
Current disk mapping table (source => target):
/dev/sda => /dev/sdc
/dev/sdc => /dev/sda
/dev/sdb => /dev/sdb
UserInput -I LAYOUT_MIGRATION_CONFIRM_MAPPINGS needed in /usr/share/rear/layout/prepare/default/300_map_disks.sh line 291
Confirm or edit the disk mapping
1) Confirm disk mapping and continue 'rear recover'
2) n/a
3) Edit disk mapping (/var/lib/rear/layout/disk_mappings)
4) Use Relax-and-Recover shell and return back to here
5) Abort 'rear recover'
UserInput: No real user input (empty or only spaces) - using default input
UserInput: Valid choice number result 'Confirm disk mapping and continue 'rear recover''
User confirmed disk mapping
Disabling excluded components in /var/lib/rear/layout/disklayout.conf
Applied disk layout mappings to /var/lib/rear/layout/disklayout.conf
Applied disk layout mappings to /var/lib/rear/layout/config/df.txt
Applied disk layout mappings to /etc/rear/rescue.conf
Examining gpt disk /dev/sdc to automatically resize its last active partition
Skipping /dev/sdc (size of new disk same as size of old disk)
Examining gpt disk /dev/sdb to automatically resize its last active partition
Checking /dev/sdb1 if it is the last partition on /dev/sdb
Found 'rear-noname' partition /dev/sdb1 as last partition on /dev/sdb
Determining if last partition /dev/sdb1 is resizeable
Determining new size for last partition /dev/sdb1
Determining if last partition /dev/sdb1 actually needs to be increased or shrinked
New /dev/sdb is 2147483648 bytes bigger than old disk
Increasing last partition /dev/sdb1 up to end of disk (new disk at least 10% bigger)
Changed last partition /dev/sdb1 size from 2146418176 to 4292870144 bytes
Examining gpt disk /dev/sda to automatically resize its last active partition
Skipping /dev/sda (size of new disk same as size of old disk)
UserInput -I LAYOUT_FILE_CONFIRMATION needed in /usr/share/rear/layout/prepare/default/500_confirm_layout_file.sh line 26
Confirm or edit the disk layout file
1) Confirm disk layout and continue 'rear recover'
2) Edit disk layout (/var/lib/rear/layout/disklayout.conf)
3) View disk layout (/var/lib/rear/layout/disklayout.conf)
4) View original disk space usage (/var/lib/rear/layout/config/df.txt)
5) Use Relax-and-Recover shell and return back to here
6) Abort 'rear recover'
Regarding Using ... (the only appropriate) for recreating ...
see the comment in layout/prepare/default/300_map_disks.sh
that explains why that automatism should be OK
# Automatically map when only one appropriate current block device
is found where to it could be mapped.
# At the end the mapping file is shown and the user can edit it
if he does not like an automated mapping:
jsmeix commented at 2021-10-07 11:36:¶
Next test with
"rear mkbackup" and "rear recover"
on two KVM test systems with 3 disks
original system replacement system
sda 3 GiB sda 1 GiB
sdb 2 GiB sdb 3 GiB
sdc 1 GiB sdc 3 GiB
"rear -D recover" terminal output (excerpt)
Comparing disks
Ambiguous possible target disks need manual configuration (more than one with same size found)
Switching to manual disk layout configuration
Using /dev/sdb (same size 3221225472) for recreating /dev/sda
Could not automap /dev/sdb (no disk with same size 2147483648 found)
Using /dev/sda (same size 1073741824) for recreating /dev/sdc
Original disk /dev/sdb does not exist (with same size) in the target system
Using /dev/sdc (the only appropriate) for recreating /dev/sdb
Current disk mapping table (source => target):
/dev/sda => /dev/sdb
/dev/sdc => /dev/sda
/dev/sdb => /dev/sdc
UserInput -I LAYOUT_MIGRATION_CONFIRM_MAPPINGS needed in /usr/share/rear/layout/prepare/default/300_map_disks.sh line 291
Confirm or edit the disk mapping
1) Confirm disk mapping and continue 'rear recover'
2) n/a
3) Edit disk mapping (/var/lib/rear/layout/disk_mappings)
4) Use Relax-and-Recover shell and return back to here
5) Abort 'rear recover'
(default '1' timeout 300 seconds)
UserInput: No real user input (empty or only spaces) - using default input
UserInput: Valid choice number result 'Confirm disk mapping and continue 'rear recover''
User confirmed disk mapping
Disabling excluded components in /var/lib/rear/layout/disklayout.conf
Applied disk layout mappings to /var/lib/rear/layout/disklayout.conf
Applied disk layout mappings to /var/lib/rear/layout/config/df.txt
Applied disk layout mappings to /etc/rear/rescue.conf
Examining gpt disk /dev/sdb to automatically resize its last active partition
Skipping /dev/sdb (size of new disk same as size of old disk)
Examining gpt disk /dev/sdc to automatically resize its last active partition
Checking /dev/sdc1 if it is the last partition on /dev/sdc
Found 'rear-noname' partition /dev/sdc1 as last partition on /dev/sdc
Determining if last partition /dev/sdc1 is resizeable
Determining new size for last partition /dev/sdc1
Determining if last partition /dev/sdc1 actually needs to be increased or shrinked
New /dev/sdc is 1073741824 bytes bigger than old disk
Increasing last partition /dev/sdc1 up to end of disk (new disk at least 10% bigger)
Changed last partition /dev/sdc1 size from 2146418176 to 3219128320 bytes
Examining gpt disk /dev/sda to automatically resize its last active partition
Skipping /dev/sda (size of new disk same as size of old disk)
UserInput -I LAYOUT_FILE_CONFIRMATION needed in /usr/share/rear/layout/prepare/default/500_confirm_layout_file.sh line 26
Confirm or edit the disk layout file
1) Confirm disk layout and continue 'rear recover'
2) Edit disk layout (/var/lib/rear/layout/disklayout.conf)
3) View disk layout (/var/lib/rear/layout/disklayout.conf)
4) View original disk space usage (/var/lib/rear/layout/config/df.txt)
5) Use Relax-and-Recover shell and return back to here
6) Abort 'rear recover'
jsmeix commented at 2021-10-07 11:46:¶
Another test with
"rear mkbackup" and "rear recover"
on two KVM test systems with 3 disks
original system replacement system
sda 3 GiB sda 3 GiB
sdb 2 GiB sdb 3 GiB
sdc 1 GiB sdc 3 GiB
"rear -D recover" terminal output (excerpt)
Comparing disks
Ambiguous possible target disks need manual configuration (more than one with same size found)
Switching to manual disk layout configuration
Using /dev/sda (same name and same size 3221225472) for recreating /dev/sda
Could not automap /dev/sdb (no disk with same size 2147483648 found)
Could not automap /dev/sdc (no disk with same size 1073741824 found)
Original disk /dev/sdb does not exist (with same size) in the target system
UserInput -I LAYOUT_MIGRATION_REPLACEMENT_SDB needed in /usr/share/rear/layout/prepare/default/300_map_disks.sh line 254
Choose an appropriate replacement for /dev/sdb
1) /dev/sdb
2) /dev/sdc
3) Do not map /dev/sdb
4) Use Relax-and-Recover shell and return back to here
(default '1' timeout 300 seconds)
1
UserInput: Valid choice number result '/dev/sdb'
Using /dev/sdb (chosen by user) for recreating /dev/sdb
Original disk /dev/sdc does not exist (with same size) in the target system
Using /dev/sdc (the only appropriate) for recreating /dev/sdc
Current disk mapping table (source => target):
/dev/sda => /dev/sda
/dev/sdb => /dev/sdb
/dev/sdc => /dev/sdc
UserInput -I LAYOUT_MIGRATION_CONFIRM_MAPPINGS needed in /usr/share/rear/layout/prepare/default/300_map_disks.sh line 291
Confirm or edit the disk mapping
1) Confirm disk mapping and continue 'rear recover'
2) Confirm identical disk mapping and proceed without manual configuration
3) Edit disk mapping (/var/lib/rear/layout/disk_mappings)
4) Use Relax-and-Recover shell and return back to here
5) Abort 'rear recover'
(default '1' timeout 300 seconds)
UserInput: No real user input (empty or only spaces) - using default input
UserInput: Valid choice number result 'Confirm disk mapping and continue 'rear recover''
User confirmed disk mapping
Disabling excluded components in /var/lib/rear/layout/disklayout.conf
Applied disk layout mappings to /var/lib/rear/layout/disklayout.conf
Applied disk layout mappings to /var/lib/rear/layout/config/df.txt
Applied disk layout mappings to /etc/rear/rescue.conf
Examining gpt disk /dev/sda to automatically resize its last active partition
Skipping /dev/sda (size of new disk same as size of old disk)
Examining gpt disk /dev/sdb to automatically resize its last active partition
Checking /dev/sdb1 if it is the last partition on /dev/sdb
Found 'rear-noname' partition /dev/sdb1 as last partition on /dev/sdb
Determining if last partition /dev/sdb1 is resizeable
Determining new size for last partition /dev/sdb1
Determining if last partition /dev/sdb1 actually needs to be increased or shrinked
New /dev/sdb is 1073741824 bytes bigger than old disk
Increasing last partition /dev/sdb1 up to end of disk (new disk at least 10% bigger)
Changed last partition /dev/sdb1 size from 2146418176 to 3219128320 bytes
Examining gpt disk /dev/sdc to automatically resize its last active partition
Checking /dev/sdc1 if it is the last partition on /dev/sdc
Found 'rear-noname' partition /dev/sdc1 as last partition on /dev/sdc
Determining if last partition /dev/sdc1 is resizeable
Determining new size for last partition /dev/sdc1
Determining if last partition /dev/sdc1 actually needs to be increased or shrinked
New /dev/sdc is 2147483648 bytes bigger than old disk
Increasing last partition /dev/sdc1 up to end of disk (new disk at least 10% bigger)
Changed last partition /dev/sdc1 size from 1072676352 to 3219128320 bytes
UserInput -I LAYOUT_FILE_CONFIRMATION needed in /usr/share/rear/layout/prepare/default/500_confirm_layout_file.sh line 26
Confirm or edit the disk layout file
1) Confirm disk layout and continue 'rear recover'
2) Edit disk layout (/var/lib/rear/layout/disklayout.conf)
3) View disk layout (/var/lib/rear/layout/disklayout.conf)
4) View original disk space usage (/var/lib/rear/layout/config/df.txt)
5) Use Relax-and-Recover shell and return back to here
6) Abort 'rear recover'
jsmeix commented at 2021-10-07 11:56:¶
One more test with
"rear mkbackup" and "rear recover"
on two KVM test systems with 3 disks
original system replacement system
sda 3 GiB sda 4 GiB
sdb 2 GiB sdb 4 GiB
sdc 1 GiB sdc 4 GiB
"rear -D recover" terminal output (excerpt)
Comparing disks
Device sda has size 4294967296 bytes but 3221225472 bytes is expected (needs manual configuration)
Device sdb has size 4294967296 bytes but 2147483648 bytes is expected (needs manual configuration)
Device sdc has size 4294967296 bytes but 1073741824 bytes is expected (needs manual configuration)
Switching to manual disk layout configuration
Could not automap /dev/sda (no disk with same size 3221225472 found)
Could not automap /dev/sdb (no disk with same size 2147483648 found)
Could not automap /dev/sdc (no disk with same size 1073741824 found)
Original disk /dev/sda does not exist (with same size) in the target system
UserInput -I LAYOUT_MIGRATION_REPLACEMENT_SDA needed in /usr/share/rear/layout/prepare/default/300_map_disks.sh line 254
Choose an appropriate replacement for /dev/sda
1) /dev/sda
2) /dev/sdb
3) /dev/sdc
4) Do not map /dev/sda
5) Use Relax-and-Recover shell and return back to here
(default '1' timeout 300 seconds)
1
UserInput: Valid choice number result '/dev/sda'
Using /dev/sda (chosen by user) for recreating /dev/sda
Original disk /dev/sdb does not exist (with same size) in the target system
UserInput -I LAYOUT_MIGRATION_REPLACEMENT_SDB needed in /usr/share/rear/layout/prepare/default/300_map_disks.sh line 254
Choose an appropriate replacement for /dev/sdb
1) /dev/sdb
2) /dev/sdc
3) Do not map /dev/sdb
4) Use Relax-and-Recover shell and return back to here
(default '1' timeout 300 seconds)
1
UserInput: Valid choice number result '/dev/sdb'
Using /dev/sdb (chosen by user) for recreating /dev/sdb
Original disk /dev/sdc does not exist (with same size) in the target system
Using /dev/sdc (the only appropriate) for recreating /dev/sdc
Current disk mapping table (source => target):
/dev/sda => /dev/sda
/dev/sdb => /dev/sdb
/dev/sdc => /dev/sdc
UserInput -I LAYOUT_MIGRATION_CONFIRM_MAPPINGS needed in /usr/share/rear/layout/prepare/default/300_map_disks.sh line 291
Confirm or edit the disk mapping
1) Confirm disk mapping and continue 'rear recover'
2) Confirm identical disk mapping and proceed without manual configuration
3) Edit disk mapping (/var/lib/rear/layout/disk_mappings)
4) Use Relax-and-Recover shell and return back to here
5) Abort 'rear recover'
(default '1' timeout 300 seconds)
UserInput: No real user input (empty or only spaces) - using default input
UserInput: Valid choice number result 'Confirm disk mapping and continue 'rear recover''
User confirmed disk mapping
Disabling excluded components in /var/lib/rear/layout/disklayout.conf
Applied disk layout mappings to /var/lib/rear/layout/disklayout.conf
Applied disk layout mappings to /var/lib/rear/layout/config/df.txt
Applied disk layout mappings to /etc/rear/rescue.conf
Examining gpt disk /dev/sda to automatically resize its last active partition
Checking /dev/sda1 if it is the last partition on /dev/sda
Checking /dev/sda2 if it is the last partition on /dev/sda
Found 'rear-noname' partition /dev/sda2 as last partition on /dev/sda
Determining if last partition /dev/sda2 is resizeable
Last partition /dev/sda2 not resizeable (used during boot)
Determining new size for last partition /dev/sda2
Determining if last partition /dev/sda2 actually needs to be increased or shrinked
New /dev/sda is 1073741824 bytes bigger than old disk
Skip increasing last partition /dev/sda2 (not resizeable)
Examining gpt disk /dev/sdb to automatically resize its last active partition
Checking /dev/sdb1 if it is the last partition on /dev/sdb
Found 'rear-noname' partition /dev/sdb1 as last partition on /dev/sdb
Determining if last partition /dev/sdb1 is resizeable
Determining new size for last partition /dev/sdb1
Determining if last partition /dev/sdb1 actually needs to be increased or shrinked
New /dev/sdb is 2147483648 bytes bigger than old disk
Increasing last partition /dev/sdb1 up to end of disk (new disk at least 10% bigger)
Changed last partition /dev/sdb1 size from 2146418176 to 4292870144 bytes
Examining gpt disk /dev/sdc to automatically resize its last active partition
Checking /dev/sdc1 if it is the last partition on /dev/sdc
Found 'rear-noname' partition /dev/sdc1 as last partition on /dev/sdc
Determining if last partition /dev/sdc1 is resizeable
Determining new size for last partition /dev/sdc1
Determining if last partition /dev/sdc1 actually needs to be increased or shrinked
New /dev/sdc is 3221225472 bytes bigger than old disk
Increasing last partition /dev/sdc1 up to end of disk (new disk at least 10% bigger)
Changed last partition /dev/sdc1 size from 1072676352 to 4292870144 bytes
UserInput -I LAYOUT_FILE_CONFIRMATION needed in /usr/share/rear/layout/prepare/default/500_confirm_layout_file.sh line 26
Confirm or edit the disk layout file
1) Confirm disk layout and continue 'rear recover'
2) Edit disk layout (/var/lib/rear/layout/disklayout.conf)
3) View disk layout (/var/lib/rear/layout/disklayout.conf)
4) View original disk space usage (/var/lib/rear/layout/config/df.txt)
5) Use Relax-and-Recover shell and return back to here
6) Abort 'rear recover'
jsmeix commented at 2021-10-07 13:08:¶
@gdha
see my above tests - all looks well to me.
Please check whether or not it also looks OK to you.
Do you like to see some more other tests?
pcahyna commented at 2021-10-07 13:35:¶
@jsmeix I will have a look at the change, but it will take me a bit of time because I am not familiar with this particular code (yet).
jsmeix commented at 2021-10-07 13:40:¶
@pcahyna
thank you for having a look!
jsmeix commented at 2021-10-13 12:44:¶
@pcahyna
do you soon have time to have a look?
@gdha
are my tests sufficient for you?
I ask because I would like to merge it if there are no objections.
The changes improve things for me and also for a SUSE customer
so the changes should be at least an improvement for MIGRATION_MODE.
Of course this does not mean current MIGRATION_MODE "just works", cf.
https://github.com/rear/rear/issues/2696
pcahyna commented at 2021-10-13 15:42:¶
@pcahyna do you soon have time to have a look?
sure, looking at it now.
gdha commented at 2021-10-14 11:56:¶
@jsmeix tests are fine to me - ok to merge the PR
jsmeix commented at 2021-10-14 13:08:¶
@gdha @pcahyna
thank you for your reviews.
I will merge it tomorrow afternoon
provided no objections appear until then.
jsmeix commented at 2021-10-15 10:28:¶
Test with two disks according to
https://github.com/rear/rear/issues/2690#issuecomment-942548003
original system replacement system
sda 7 GiB sda 2 GiB
sdb 2 GiB sdb 7 GiB
This works with master code (i.e. without the changes in this pull request)
# rear -D recover
...
Comparing disks
Device sda has size 2147483648 bytes but 7516192768 bytes is expected (needs manual configuration)
Device sdb has size 7516192768 bytes but 2147483648 bytes is expected (needs manual configuration)
Switching to manual disk layout configuration
Using /dev/sdb (same size) for recreating /dev/sda
Original disk /dev/sdb does not exist (with same size) in the target system
Using /dev/sda (the only appropriate) for recreating /dev/sdb
Current disk mapping table (source => target):
/dev/sda => /dev/sdb
/dev/sdb => /dev/sda
UserInput -I LAYOUT_MIGRATION_CONFIRM_MAPPINGS needed in /usr/share/rear/layout/prepare/default/300_map_disks.sh line 275
Confirm or edit the disk mapping
1) Confirm disk mapping and continue 'rear recover'
2) n/a
3) Edit disk mapping (/var/lib/rear/layout/disk_mappings)
4) Use Relax-and-Recover shell and return back to here
5) Abort 'rear recover'
(default '1' timeout 300 seconds)
This also explains why the current code in 300_map_disks.sh is as
is.
Very likely I had only tested it with two disks.
How that works with the changes in this pull request:
# rear -D recover
...
Comparing disks
Device sda has size 2147483648 bytes but 7516192768 bytes is expected (needs manual configuration)
Device sdb has size 7516192768 bytes but 2147483648 bytes is expected (needs manual configuration)
Switching to manual disk layout configuration
Using /dev/sdb (same size 7516192768) for recreating /dev/sda
Using /dev/sda (same size 2147483648) for recreating /dev/sdb
Current disk mapping table (source => target):
/dev/sda => /dev/sdb
/dev/sdb => /dev/sda
UserInput -I LAYOUT_MIGRATION_CONFIRM_MAPPINGS needed in /usr/share/rear/layout/prepare/default/300_map_disks.sh line 291
Confirm or edit the disk mapping
1) Confirm disk mapping and continue 'rear recover'
2) n/a
3) Edit disk mapping (/var/lib/rear/layout/disk_mappings)
4) Use Relax-and-Recover shell and return back to here
5) Abort 'rear recover'
(default '1' timeout 300 seconds)
[Export of Github issue for rear/rear.]