#1307 Issue closed: Migrate: Partition resized to a negative number (new_size=0)¶
Labels: enhancement, bug, cleanup, fixed / solved / done
ProBackup-nl opened issue at 2017-04-16 11:30:¶
- rear version: 2.00 git
- OS version: Arch Linux
- rear configuration files: USB NETFS
- Are you using legacy BIOS or UEFI boot? UEFI
- Brief description of the issue: Recover and migrate from 32G to 128G drive, from /dev/sda to /dev/sda results in Partition /dev/sda resized to a negative number
Source drive:
Disk /dev/sdc: 32.1GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number Start End Size Name File system Flags
1 4194kB 1078MB 1074MB EFIboot fat32 boot, esp
2 1078MB 22.0GB 21.0GB root btrfs
3 22.0GB 32.0GB 9966MB home btrfs
Target drive:
Disk /dev/sda: 126GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number Start End Size Name File system Flags
1 4194kB 104MB 99.6MB EFIboot fat16 boot, esp
2 104MB 30.0GB 29.9GB root btrfs
3 29.9GB 60.0B 30.0GB home btrfs
On screen:
This is the disk mapping table:
/dev/sda /dev/sda
ERROR:
===================
BUG in /usr/share/rear/layout/prepare/default/400_autoresize_disks.sh:
'Partition /dev/sda2 resized to a negative number.'
/var/lib/rear/layout/disklayout.conf:
part /dev/sda 1073741824 4194304 EFIboot boot /dev/sda1
part /dev/sda 20971520000 1077936128 root none /dev/sda2
part /dev/sda 9965666304 22049456128 home none /dev/sda3
Debug log (dots inserted as thousand separator for readability):
difference=93.957.251.072B
...
partitions=()
resizeable_space=0
available_space=126.035.288.064
...
available_space=124.961.546.240
Log 'Will not resize partition /dev/sda1.'
...
.../dev/sda2...
...
resizeable_space=20.971.520.000
...Will resize partition /dev/sda2.'
...
.../dev/sda3...
...
resizeable_space=30.937.186.304
...Will resize partition /dev/sda3.'
...
(( available_space < 0 ))
for data in "${partitions[@]}"
name=/dev/sda2
partition_size=20.971.520.000
new_size=0
(( new_size > 0 ))
BugifError 'Partition dev/sda2 resized to a negative number.'
(( 1 != 0 ))
BugifError 'Partition dev/sda2 resized to a negative number.'
I would guess that
$(( ( $partition_size / $resizeable_space ) * $available_space ))
becomes: $(( (20971520000/30937186304)*124961546240 )) which equals to
0.
ProBackup-nl commented at 2017-04-16 19:14:¶
Isn't line 49 an issue?
if (( available_space < 0 )) ; then
Shouldn't that available_space var not be prefixed with a $?
ProBackup-nl commented at 2017-04-16 19:35:¶
@gozora https://github.com/rear/rear/commit/2fb97e5dd789151516369deec8577d03c0ca14b0#diff-1478c668347b35dc86c26c2fa918dd73 breaks resizing with a Bug (new_size=0).
Bash 4.4.12 seems not be able to handle these large numbers:
$ echo $(( 20971520000*30971520000 ))
3883808530565693440
ProBackup-nl commented at 2017-04-16 20:40:¶
This issue is related to #1269 and #1272.
gdha commented at 2017-04-18 08:46:¶
@ProBackup-nl According bc it should be:
$ echo 20971520000*30971520000|bc
649519851110400000000
@jsmeix @gozora If bash results are not correct, perhaps we should make a function which is using bc instead?
gozora commented at 2017-04-18 09:52:¶
I have a dimm feeling that bash mathematic expansion does not work with
decimal numbers.
looks we will indeed need bc to deal with this... Will check that later
today.
jsmeix commented at 2017-04-18 11:35:¶
I didn't check all the details but when the issue is that in bash
(( ( a / b ) * c ))
evaluates to 0 when b > a as in
# echo $(( ( 2 / 3 ) * 4 )) 0
then it should help to do the multiplication before the division
# echo $(( ( 2 * 4 ) / 3 )) 2
@gozora
bash does not support decimal numbers like 0.666666 ( = 2 / 3).
bash only supports integers so that 2 / 3 evaluates to 0 in bash:
# echo $(( 2 / 3 )) 0
jsmeix commented at 2017-04-18 11:41:¶
I suggest to change in
layout/prepare/default/400_autoresize_disks.sh
the line
new_size=$(( ( $partition_size / $resizeable_space ) * $available_space ))
with
new_size=$(( ( $partition_size * $available_space ) / $resizeable_space ))
jsmeix commented at 2017-04-18 11:49:¶
Regarding
https://github.com/rear/rear/issues/1307#issuecomment-294369824
what the maximum numbers are that bash arithmetic can hande cf.
https://github.com/rear/rear/issues/1269
where it seems bash arithmetic works up to 2^63 - 1
i.e. up to 9223372036854775807
but on my SLES11 32-bit system
and on my SLES12 64-bit system I also get
# echo $(( 20971520000 * 30971520000 )) 3883808530565693440
:-(
jsmeix commented at 2017-04-18 11:57:¶
I got confused what the 20971520000 * 30971520000 means.
I think when the issue is to get the right result for
( 20971520000 / 30937186304 ) * 124961546240
then the multiplication should be
20971520000 * 124961546240
which of course fails even worse in bash.
jsmeix commented at 2017-04-18 12:11:¶
Meanwhile I also think we must use 'bc -l'
basically everywhere in partitioning code
because bash arithmetic "just fails".
Note that plain 'bc' also does only integer calculations:
# echo '( 2 / 3 ) * 4' | bc 0 # echo '( 20971520000 / 30937186304 ) * 124961546240' | bc 0
in contrast to 'bc -l' that results what we need:
# echo '( 2 / 3 ) * 4' | bc -l 2.66666666666666666664 # echo '( 20971520000 / 30937186304 ) * 124961546240' | bc -l 84708206507.59219088829208903680
To get only the final result as integer from 'bc -l' we can use
# echo ' r = ( 2 / 3 ) * 4 ; scale=0 ; r / 1 ' | bc -l 2 # echo -e 'r = ( 20971520000 / 30937186304 ) * 124961546240 ; scale = 0 ; r / 1 ' | bc -l 84708206507
a bit complicated and ugly but it seems to work.
gozora commented at 2017-04-18 18:52:¶
@jsmeix
> To get only the final result as integer from 'bc -l' we can use
>
> # echo ' r = ( 2 / 3 ) * 4 ; scale=0 ; r / 1 ' | bc -l
> 2
>
> # echo -e 'r = ( 20971520000 / 30937186304 ) * 124961546240 ; scale = 0 ; r / 1 ' | bc -l
> 84708206507
This is exactly it, and it is not ugly, I'd call it clever!
As this is primary your idea, would you like to implement it, or should I?
V.
gozora commented at 2017-04-18 19:02:¶
Just for fun of it, I've try to run before listed code on my Arch, and guess what I've got:
[root@arch ~]# bc
bash: bc: command not found
;-)
V.
jsmeix commented at 2017-04-19 11:25:¶
@ProBackup-nl
can you comment about whether or not 'bc'
is usually available on Arch Linux
and if not what we could do here?
I would think ReaR should switch to 'bc'
for all partitioning calculations and error out
if 'bc' is not available.
I think all partitioning calculations should be done
with bytes 'B' as unit
cf.
https://github.com/rear/rear/issues/1270
and
https://github.com/rear/rear/pull/1273
Only the final result may - if needed - be rounded
to whatever MiB unit that is reasonable for a real disk.
Usually only in disk migration mode partitioning values
should be re-calculated and finally rounded to a MiB unit.
By default this MiB unit should be 8 MiB nowadays
cf.
https://github.com/rear/rear/issues/1201
and
https://github.com/rear/rear/pull/1217
ProBackup-nl commented at 2017-04-21 08:31:¶
@jsmeix bc is usually not installed on Arch Linux.
If not what we could do here?
Revert back to the awk solution, and try to fix awk for the failing
cases.
jsmeix commented at 2017-04-21 11:27:¶
ReaR cannot fix awk (or any tool that is calls).
I would rather "fix Arch Linux" when it comes without 'bc'.
In general regarding ReaR on very minimal systems, see
https://github.com/rear/rear/issues/755
schlomo commented at 2017-04-21 13:26:¶
👍 for fixing Arch. Doesn't Arch also have some kind of dependency mechanism which we can use to pull in required software?
Typically users will use the package management to install ReaR, developers should install the dependencies manually.
So, let's add bc to REQUIRED_PROGS and also add it to the package
dependencies.
jsmeix commented at 2017-04-21 14:46:¶
Hopefully I find some time during May
to use 'bc' for the calculations in the partitioning code
but no promises...
@gozora
if you like to do it, I would very much appreciate it.
jsmeix commented at 2017-04-21 14:49:¶
Only a side note: I think a lot more programs
should be added to REQUIRED_PROGS
cf.
https://github.com/rear/rear/issues/892
schlomo commented at 2017-04-21 15:20:¶
👍
On 21 April 2017 at 16:49, Johannes Meixner notifications@github.com
wrote:
Only a side note: I think a lot more programs
should be added to REQUIRED_PROGS
cf. #892 https://github.com/rear/rear/issues/892—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/rear/rear/issues/1307#issuecomment-296212029, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAGMCNtMqtLxuv_ErvFNxiRbTNRawSZSks5ryMHzgaJpZM4M-mss
.
gozora commented at 2017-04-24 07:41:¶
@jsmeix I certainly can take a look on this.
Just to my understanding, we want to replace ALL calculations through
whole ReaR code, right?
V.
schlomo commented at 2017-04-24 08:01:¶
IMHO we should take this in steps and start from cleaning up the
partitioning calculations, after all there we have the current pains.
Maybe here it is worthwhile to render a bc script that does all the
calculations in one go instead of calling bc many times.
I wouldn't go so far as to replace all trivial calculations with bc.
jsmeix commented at 2017-04-24 08:38:¶
@gozora
my understanding is that we need to replace
all calculations with possibly big numbers
through whole ReaR code.
But I also fully agree with @schlomo
to do first things first so that as a first step we should
only replace the partitioning calculations and later
when that "just works perfectly fine" we can replace
also other calculations as needed.
Ideally there should be a new generic function calculate()
in global-functions.sh so that we have this one function
where we could fix things if something doesn't work.
Something (not thoroughly tested) like:
function calculate()
{ # Use 'bc' for calculations because other tools
# fail in various unexpected ways for big numbers,
# e.g. see https://github.com/rear/rear/issues/1307
echo " result = $@ ; scale=0 ; result / 1 " | bc -l
}
According to a very first quick test (on my SLES11 system)
it seems to work for me:
# function calculate() { echo " result = $@ ; scale=0 ; result / 1 " | bc -l ; }
# foo=20971520000
# bar=30937186304
# baz=124961546240
# set -x
# res=$( calculate "( $foo / $bar ) * $baz" )
++ calculate '( 20971520000 / 30937186304 ) * 124961546240'
++ echo ' result = ( 20971520000 / 30937186304 ) * 124961546240 ; scale=0 ; result / 1 '
++ bc -l
+ res=84708206507
# { set +x 2>/dev/null ; }
# echo $res
84708206507
gozora commented at 2017-04-24 08:52:¶
OK, so for the start I'll implement idea of @jsmeix with calculate
function in 100_include_partition_code.sh and
400_autoresize_disks.sh and run couple of tests.
Once it works we can spread it wherever necessary.
V.
schlomo commented at 2017-04-24 09:10:¶
BTW, you can use Bash here strings instead of pipes to speed up stuff:
$ function calculate() { bc -ql <<<"result=$@ ; scale=0 ; result / 1 "; }
$ foo=20971520000 bar=30937186304 baz=124961546240
$ set -x
$ res=$( calculate "( $foo / $bar ) * $baz" )
++ calculate '( 20971520000 / 30937186304 ) * 124961546240'
++ bc -ql
+ res=84708206507
$ set +x ; echo $res
+ set +x
84708206507
As this calculate function actually rounds the result we might call it
appropriately calculate_round or so. And we should beware of rounding
issues...
jsmeix commented at 2017-04-24 09:42:¶
@schlomo
thanks for the interesting information!
It speeds up - but not much - about 14% on my SLES11 system:
# foo=20971520000
# bar=30937186304
# baz=124961546240
# function calculate() { bc -ql <<<"result=$@ ; scale=0 ; result / 1 "; }
# time for i in $( seq 1000 ) ; do res=$( calculate "( $foo / $bar ) * $baz" ) ; done
real 0m4.366s
user 0m0.208s
sys 0m0.372s
# function calculate() { echo " result = $@ ; scale=0 ; result / 1 " | bc -l ; }
# time for i in $( seq 1000 ) ; do res=$( calculate "( $foo / $bar ) * $baz" ) ; done
real 0m5.000s
user 0m0.152s
sys 0m0.408s
schlomo commented at 2017-04-24 09:50:¶
IMHO 14% is really impressive, didn't think that the pipe was that expensive.
jsmeix commented at 2017-04-24 10:24:¶
Yes, the relative speedup is impressive.
My comment was not clear.
I meant the absolute gain in seconds.
On my old slow 32-bit i386 SLES11 computer
I got less than one second for a thousand calculations
which is less than a millisecond for a single calculation.
With usually about tens of calculations in the partitioning code
the overall gain is not so much.
On the other hand using bash here strings instead of pipes
costs us nothing so we can "just use them".
On big iron with thousands of disks (like IBM System z)
it may actually help.
gozora commented at 2017-04-24 10:51:¶
FYI,
I'll use
https://github.com/gozora/rear/tree/issue/1307
while working on this issue ...
V.
jsmeix commented at 2017-04-26 12:00:¶
With
https://github.com/rear/rear/pull/1332
merged
this issue should (hopefully) be solved.
@ProBackup-nl
note that since
https://github.com/rear/rear/pull/1332
merged
the 'bc' tool is mandatory to use ReaR because now 'bc' is
added to the REQUIRED_PROGS array in default.conf
jsmeix commented at 2017-04-26 14:37:¶
An addednum only FYI:
I got an unexpected
Partition primary on /dev/sda: size reduced to fit on disk.
message plus different partitioning (by one MiB)
for "rear recover" on a disk with exact same size
and swap was not at all migrated.
I think this shows that disk migration mode
is not yet working really well but hopefully
it works somewhat o.k. for now.
Details:
I did "rear mkbackup" on a system (virtual KVM/QEMU machine)
with two exactly 20 GiB disks (/dev/sda and /dev/sdb)
with same partitioning on each one.
Only /dev/sda is mounted but swap is used on /dev/sdb1
# parted -s /dev/sda unit B print Model: ATA QEMU HARDDISK (scsi) Disk /dev/sda: 21474836480B Sector size (logical/physical): 512B/512B Partition Table: msdos Disk Flags: Number Start End Size Type File system Flags 1 1048576B 1562378239B 1561329664B primary linux-swap(v1) type=82 2 1562378240B 21474836479B 19912458240B primary ext4 boot, type=83 # parted -s /dev/sda unit MiB print Model: ATA QEMU HARDDISK (scsi) Disk /dev/sda: 20480MiB Sector size (logical/physical): 512B/512B Partition Table: msdos Disk Flags: Number Start End Size Type File system Flags 1 1.00MiB 1490MiB 1489MiB primary linux-swap(v1) type=82 2 1490MiB 20480MiB 18990MiB primary ext4 boot, type=83 # parted -s /dev/sdb unit B print Model: ATA QEMU HARDDISK (scsi) Disk /dev/sdb: 21474836480B Sector size (logical/physical): 512B/512B Partition Table: msdos Disk Flags: Number Start End Size Type File system Flags 1 1048576B 1562378239B 1561329664B primary linux-swap(v1) type=83 2 1562378240B 21474836479B 19912458240B primary ext4 boot, type=83 # mount | grep sd /dev/sda2 on / type ext4 (rw,relatime,data=ordered) # cat /proc/swaps Filename Type Size Used Priority /dev/sdb1 partition 1524732 1084 -1 # rear -d -D mkbackup ... # grep -v '^#' var/lib/rear/layout/disklayout.conf disk /dev/sda 21474836480 msdos part /dev/sda 1561329664 1048576 primary none /dev/sda1 part /dev/sda 19912458240 1562378240 primary boot /dev/sda2 disk /dev/sdb 21474836480 msdos part /dev/sdb 1561329664 1048576 primary none /dev/sdb1 part /dev/sdb 19912458240 1562378240 primary boot /dev/sdb2 fs /dev/sda2 / ext4 uuid=46d7e8be-7812-49d1-8d24-e25ed0589e94 label= blocksize=4096 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=16377 default_mount_options=user_xattr,acl options=rw,relatime,data=ordered swap /dev/sdb1 uuid=28e43119-dac1-4426-a71a-1d70b26d33d7 label=
I did "rear recover" on a new system (same kind of virtual machine)
with one exactly 20 GiB disk (/dev/sda)
and got (excerpts):
# rear -d -D recover
...
Comparing disks.
Device sdb does not exist.
Switching to manual disk layout configuration.
This is the disk mapping table:
/dev/sda /dev/sda
Please confirm that '/var/lib/rear/layout/disklayout.conf' is as you expect.
++ select choice in '"${choices[@]}"'
1) View disk layout (disklayout.conf) 3) View original disk space usage 5) Continue recovery
2) Edit disk layout (disklayout.conf) 4) Go to Relax-and-Recover shell 6) Abort Relax-and-Recover
#? 5
++ case "$REPLY" in
++ break
Partition primary on /dev/sda: size reduced to fit on disk.
Please confirm that '/var/lib/rear/layout/diskrestore.sh' is as you expect.
++ select choice in '"${choices[@]}"'
1) View restore script (diskrestore.sh) 3) View original disk space usage 5) Continue recovery
2) Edit restore script (diskrestore.sh) 4) Go to Relax-and-Recover shell 6) Abort Relax-and-Recover
#? 5
++ case "$REPLY" in
++ break
Start system layout restoration.
Creating partitions for disk /dev/sda (msdos)
Creating filesystem of type ext4 with mount point / on /dev/sda2.
Mounting filesystem /
Disk layout created.
Restoring from '/tmp/rear.8JX5LAaL1d7Y3Hn/outputfs/e205/backup.tar.gz'...
...
Restoring finished.
...
Finished recovering your system. You can explore it under '/mnt/local'.
# reboot
The recreated system works well
but I got a bit different partitioning (by one MiB)
regardless that the disk has exactly same size:
# parted -s /dev/sda unit B print Model: ATA QEMU HARDDISK (scsi) Disk /dev/sda: 21474836480B Sector size (logical/physical): 512B/512B Partition Table: msdos Disk Flags: Number Start End Size Type File system Flags 1 2097152B 1563426815B 1561329664B primary type=83 2 1563430912B 21474836479B 19911405568B primary ext4 boot, type=83 # parted -s /dev/sda unit MiB print Model: ATA QEMU HARDDISK (scsi) Disk /dev/sda: 20480MiB Sector size (logical/physical): 512B/512B Partition Table: msdos Disk Flags: Number Start End Size Type File system Flags 1 2.00MiB 1491MiB 1489MiB primary type=83 2 1491MiB 20480MiB 18989MiB primary ext4 boot, type=83 # grep -v '^#' /var/log/rear/recover/layout/disklayout.conf disk /dev/sda 21474836480 msdos part /dev/sda 1561329664 1048576 primary none /dev/sda1 part /dev/sda 19912458240 1562378240 primary boot /dev/sda2 fs /dev/sda2 / ext4 uuid=46d7e8be-7812-49d1-8d24-e25ed0589e94 label= blocksize=4096 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=16377 default_mount_options=user_xattr,acl options=rw,relatime,data=ordered # cat /proc/swaps Filename Type Size Used Priority [no further output]
This shows that swap was not at all migrated
which is probably a missing enhancement.
But I think it is a bug that there is not any LogPrint message
so that the original swap on /dev/sdb1 seems to have been
silently ignored.
jsmeix commented at 2017-05-11 08:37:¶
FYI regarding calculations with non-bc tools like 'awk' cf.
https://github.com/rear/rear/issues/1269#issuecomment-300719707
[Export of Github issue for rear/rear.]