#1307 Issue closed: Migrate: Partition resized to a negative number (new_size=0)

Labels: enhancement, bug, cleanup, fixed / solved / done

ProBackup-nl opened issue at 2017-04-16 11:30:

  • rear version: 2.00 git
  • OS version: Arch Linux
  • rear configuration files: USB NETFS
  • Are you using legacy BIOS or UEFI boot? UEFI
  • Brief description of the issue: Recover and migrate from 32G to 128G drive, from /dev/sda to /dev/sda results in Partition /dev/sda resized to a negative number

Source drive:

Disk /dev/sdc: 32.1GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start          End        Size       Name     File system  Flags
 1      4194kB         1078MB     1074MB     EFIboot  fat32        boot, esp
 2      1078MB         22.0GB     21.0GB     root     btrfs
 3      22.0GB         32.0GB     9966MB     home     btrfs

Target drive:

Disk /dev/sda: 126GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start          End       Size       Name     File system  Flags
 1      4194kB         104MB     99.6MB     EFIboot  fat16        boot, esp
 2      104MB          30.0GB    29.9GB     root     btrfs
 3      29.9GB         60.0B     30.0GB     home     btrfs

On screen:

This is the disk mapping table:
    /dev/sda /dev/sda
ERROR:
===================
BUG in /usr/share/rear/layout/prepare/default/400_autoresize_disks.sh:
'Partition /dev/sda2 resized to a negative number.'

400_autoresize_disks.sh

/var/lib/rear/layout/disklayout.conf:

part /dev/sda 1073741824 4194304 EFIboot boot /dev/sda1
part /dev/sda 20971520000 1077936128 root none /dev/sda2
part /dev/sda 9965666304 22049456128 home none /dev/sda3

Debug log (dots inserted as thousand separator for readability):

difference=93.957.251.072B
...
partitions=()
resizeable_space=0
available_space=126.035.288.064
...
available_space=124.961.546.240
Log 'Will not resize partition /dev/sda1.'
...
.../dev/sda2...
...
resizeable_space=20.971.520.000
...Will resize partition /dev/sda2.'
...
.../dev/sda3...
...
resizeable_space=30.937.186.304
...Will resize partition /dev/sda3.'
...
(( available_space < 0 ))
for data in "${partitions[@]}"
name=/dev/sda2
partition_size=20.971.520.000
new_size=0
(( new_size > 0 ))
BugifError 'Partition dev/sda2 resized to a negative number.'
(( 1 != 0 ))
BugifError 'Partition dev/sda2 resized to a negative number.'

I would guess that $(( ( $partition_size / $resizeable_space ) * $available_space )) becomes: $(( (20971520000/30937186304)*124961546240 )) which equals to 0.

ProBackup-nl commented at 2017-04-16 19:14:

Isn't line 49 an issue?

if (( available_space < 0 )) ; then

Shouldn't that available_space var not be prefixed with a $?

ProBackup-nl commented at 2017-04-16 19:35:

@gozora https://github.com/rear/rear/commit/2fb97e5dd789151516369deec8577d03c0ca14b0#diff-1478c668347b35dc86c26c2fa918dd73 breaks resizing with a Bug (new_size=0).

Bash 4.4.12 seems not be able to handle these large numbers:

$ echo $(( 20971520000*30971520000 ))
3883808530565693440

ProBackup-nl commented at 2017-04-16 20:40:

This issue is related to #1269 and #1272.

gdha commented at 2017-04-18 08:46:

@ProBackup-nl According bc it should be:

$ echo  20971520000*30971520000|bc
649519851110400000000

@jsmeix @gozora If bash results are not correct, perhaps we should make a function which is using bc instead?

gozora commented at 2017-04-18 09:52:

I have a dimm feeling that bash mathematic expansion does not work with decimal numbers.
looks we will indeed need bc to deal with this... Will check that later today.

jsmeix commented at 2017-04-18 11:35:

I didn't check all the details but when the issue is that in bash

(( ( a / b ) * c ))

evaluates to 0 when b > a as in

# echo $(( ( 2 / 3 ) * 4 ))
0

then it should help to do the multiplication before the division

# echo $(( ( 2 * 4 ) / 3 ))
2

@gozora
bash does not support decimal numbers like 0.666666 ( = 2 / 3).
bash only supports integers so that 2 / 3 evaluates to 0 in bash:

# echo $(( 2 / 3 ))
0

jsmeix commented at 2017-04-18 11:41:

I suggest to change in layout/prepare/default/400_autoresize_disks.sh
the line

new_size=$(( ( $partition_size / $resizeable_space ) * $available_space ))

with

new_size=$(( ( $partition_size * $available_space ) / $resizeable_space ))

jsmeix commented at 2017-04-18 11:49:

Regarding
https://github.com/rear/rear/issues/1307#issuecomment-294369824
what the maximum numbers are that bash arithmetic can hande cf.
https://github.com/rear/rear/issues/1269
where it seems bash arithmetic works up to 2^63 - 1
i.e. up to 9223372036854775807
but on my SLES11 32-bit system
and on my SLES12 64-bit system I also get

# echo $(( 20971520000 * 30971520000 ))
3883808530565693440

:-(

jsmeix commented at 2017-04-18 11:57:

I got confused what the 20971520000 * 30971520000 means.
I think when the issue is to get the right result for
( 20971520000 / 30937186304 ) * 124961546240
then the multiplication should be
20971520000 * 124961546240
which of course fails even worse in bash.

jsmeix commented at 2017-04-18 12:11:

Meanwhile I also think we must use 'bc -l'
basically everywhere in partitioning code
because bash arithmetic "just fails".

Note that plain 'bc' also does only integer calculations:

# echo '( 2 / 3 ) * 4' | bc
0

# echo '( 20971520000 / 30937186304 ) * 124961546240' | bc
0

in contrast to 'bc -l' that results what we need:

# echo '( 2 / 3 ) * 4' | bc -l
2.66666666666666666664

# echo '( 20971520000 / 30937186304 ) * 124961546240' | bc -l
84708206507.59219088829208903680

To get only the final result as integer from 'bc -l' we can use

# echo ' r = ( 2 / 3 ) * 4 ; scale=0 ; r / 1 ' | bc -l
2

# echo -e 'r = ( 20971520000 / 30937186304 ) * 124961546240 ; scale = 0 ; r / 1 ' | bc -l
84708206507

a bit complicated and ugly but it seems to work.

gozora commented at 2017-04-18 18:52:

@jsmeix

> To get only the final result as integer from 'bc -l' we can use
> 
> # echo ' r = ( 2 / 3 ) * 4 ; scale=0 ; r / 1 ' | bc -l
> 2
> 
> # echo -e 'r = ( 20971520000 / 30937186304 ) * 124961546240 ; scale = 0 ; r / 1 ' | bc -l
> 84708206507

This is exactly it, and it is not ugly, I'd call it clever!

As this is primary your idea, would you like to implement it, or should I?

V.

gozora commented at 2017-04-18 19:02:

Just for fun of it, I've try to run before listed code on my Arch, and guess what I've got:

[root@arch ~]# bc
bash: bc: command not found

;-)

V.

jsmeix commented at 2017-04-19 11:25:

@ProBackup-nl
can you comment about whether or not 'bc'
is usually available on Arch Linux
and if not what we could do here?

I would think ReaR should switch to 'bc'
for all partitioning calculations and error out
if 'bc' is not available.

I think all partitioning calculations should be done
with bytes 'B' as unit
cf. https://github.com/rear/rear/issues/1270
and https://github.com/rear/rear/pull/1273

Only the final result may - if needed - be rounded
to whatever MiB unit that is reasonable for a real disk.
Usually only in disk migration mode partitioning values
should be re-calculated and finally rounded to a MiB unit.

By default this MiB unit should be 8 MiB nowadays
cf. https://github.com/rear/rear/issues/1201
and https://github.com/rear/rear/pull/1217

ProBackup-nl commented at 2017-04-21 08:31:

@jsmeix bc is usually not installed on Arch Linux.

If not what we could do here?

Revert back to the awk solution, and try to fix awk for the failing cases.

jsmeix commented at 2017-04-21 11:27:

ReaR cannot fix awk (or any tool that is calls).
I would rather "fix Arch Linux" when it comes without 'bc'.
In general regarding ReaR on very minimal systems, see
https://github.com/rear/rear/issues/755

schlomo commented at 2017-04-21 13:26:

👍 for fixing Arch. Doesn't Arch also have some kind of dependency mechanism which we can use to pull in required software?

Typically users will use the package management to install ReaR, developers should install the dependencies manually.

So, let's add bc to REQUIRED_PROGS and also add it to the package dependencies.

jsmeix commented at 2017-04-21 14:46:

Hopefully I find some time during May
to use 'bc' for the calculations in the partitioning code
but no promises...

@gozora
if you like to do it, I would very much appreciate it.

jsmeix commented at 2017-04-21 14:49:

Only a side note: I think a lot more programs
should be added to REQUIRED_PROGS
cf. https://github.com/rear/rear/issues/892

schlomo commented at 2017-04-21 15:20:

👍

On 21 April 2017 at 16:49, Johannes Meixner notifications@github.com
wrote:

Only a side note: I think a lot more programs
should be added to REQUIRED_PROGS
cf. #892 https://github.com/rear/rear/issues/892

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/rear/rear/issues/1307#issuecomment-296212029, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAGMCNtMqtLxuv_ErvFNxiRbTNRawSZSks5ryMHzgaJpZM4M-mss
.

gozora commented at 2017-04-24 07:41:

@jsmeix I certainly can take a look on this.
Just to my understanding, we want to replace ALL calculations through whole ReaR code, right?

V.

schlomo commented at 2017-04-24 08:01:

IMHO we should take this in steps and start from cleaning up the partitioning calculations, after all there we have the current pains. Maybe here it is worthwhile to render a bc script that does all the calculations in one go instead of calling bc many times.

I wouldn't go so far as to replace all trivial calculations with bc.

jsmeix commented at 2017-04-24 08:38:

@gozora
my understanding is that we need to replace
all calculations with possibly big numbers
through whole ReaR code.

But I also fully agree with @schlomo
to do first things first so that as a first step we should
only replace the partitioning calculations and later
when that "just works perfectly fine" we can replace
also other calculations as needed.

Ideally there should be a new generic function calculate()
in global-functions.sh so that we have this one function
where we could fix things if something doesn't work.

Something (not thoroughly tested) like:

function calculate()
{   # Use 'bc' for calculations because other tools
    # fail in various unexpected ways for big numbers,
    # e.g. see https://github.com/rear/rear/issues/1307
    echo " result = $@ ; scale=0 ; result / 1 " | bc -l
}

According to a very first quick test (on my SLES11 system)
it seems to work for me:

# function calculate() { echo " result = $@ ; scale=0 ; result / 1 " | bc -l ; }

# foo=20971520000

# bar=30937186304

# baz=124961546240

# set -x

# res=$( calculate "( $foo / $bar ) * $baz" )
++ calculate '( 20971520000 / 30937186304 ) * 124961546240'
++ echo ' result = ( 20971520000 / 30937186304 ) * 124961546240 ; scale=0 ; result / 1 '
++ bc -l
+ res=84708206507

# { set +x 2>/dev/null ; }

# echo $res
84708206507

gozora commented at 2017-04-24 08:52:

OK, so for the start I'll implement idea of @jsmeix with calculate function in 100_include_partition_code.sh and 400_autoresize_disks.sh and run couple of tests.
Once it works we can spread it wherever necessary.

V.

schlomo commented at 2017-04-24 09:10:

BTW, you can use Bash here strings instead of pipes to speed up stuff:

$ function calculate() { bc -ql <<<"result=$@ ; scale=0 ; result / 1 "; }
$ foo=20971520000 bar=30937186304 baz=124961546240
$ set -x
$ res=$( calculate "( $foo / $bar ) * $baz" )
++ calculate '( 20971520000 / 30937186304 ) * 124961546240'
++ bc -ql
+ res=84708206507
$ set +x ; echo $res
+ set +x
84708206507

As this calculate function actually rounds the result we might call it appropriately calculate_round or so. And we should beware of rounding issues...

jsmeix commented at 2017-04-24 09:42:

@schlomo
thanks for the interesting information!

It speeds up - but not much - about 14% on my SLES11 system:

# foo=20971520000
# bar=30937186304
# baz=124961546240

# function calculate() { bc -ql <<<"result=$@ ; scale=0 ; result / 1 "; }

# time for i in $( seq 1000 ) ; do res=$( calculate "( $foo / $bar ) * $baz" ) ; done
real    0m4.366s
user    0m0.208s
sys     0m0.372s

# function calculate() { echo " result = $@ ; scale=0 ; result / 1 " | bc -l ; }

# time for i in $( seq 1000 ) ; do res=$( calculate "( $foo / $bar ) * $baz" ) ; done
real    0m5.000s
user    0m0.152s
sys     0m0.408s

schlomo commented at 2017-04-24 09:50:

IMHO 14% is really impressive, didn't think that the pipe was that expensive.

jsmeix commented at 2017-04-24 10:24:

Yes, the relative speedup is impressive.
My comment was not clear.
I meant the absolute gain in seconds.
On my old slow 32-bit i386 SLES11 computer
I got less than one second for a thousand calculations
which is less than a millisecond for a single calculation.
With usually about tens of calculations in the partitioning code
the overall gain is not so much.
On the other hand using bash here strings instead of pipes
costs us nothing so we can "just use them".
On big iron with thousands of disks (like IBM System z)
it may actually help.

gozora commented at 2017-04-24 10:51:

FYI,
I'll use https://github.com/gozora/rear/tree/issue/1307 while working on this issue ...

V.

jsmeix commented at 2017-04-26 12:00:

With https://github.com/rear/rear/pull/1332 merged
this issue should (hopefully) be solved.

@ProBackup-nl
note that since https://github.com/rear/rear/pull/1332 merged
the 'bc' tool is mandatory to use ReaR because now 'bc' is
added to the REQUIRED_PROGS array in default.conf

jsmeix commented at 2017-04-26 14:37:

An addednum only FYI:

I got an unexpected

Partition primary on /dev/sda: size reduced to fit on disk.

message plus different partitioning (by one MiB)
for "rear recover" on a disk with exact same size
and swap was not at all migrated.

I think this shows that disk migration mode
is not yet working really well but hopefully
it works somewhat o.k. for now.

Details:

I did "rear mkbackup" on a system (virtual KVM/QEMU machine)
with two exactly 20 GiB disks (/dev/sda and /dev/sdb)
with same partitioning on each one.

Only /dev/sda is mounted but swap is used on /dev/sdb1

# parted -s /dev/sda unit B print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sda: 21474836480B
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 
Number  Start        End           Size          Type     File system     Flags
 1      1048576B     1562378239B   1561329664B   primary  linux-swap(v1)  type=82
 2      1562378240B  21474836479B  19912458240B  primary  ext4            boot, type=83

# parted -s /dev/sda unit MiB print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sda: 20480MiB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 
Number  Start    End       Size      Type     File system     Flags
 1      1.00MiB  1490MiB   1489MiB   primary  linux-swap(v1)  type=82
 2      1490MiB  20480MiB  18990MiB  primary  ext4            boot, type=83

# parted -s /dev/sdb unit B print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sdb: 21474836480B
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 
Number  Start        End           Size          Type     File system     Flags
 1      1048576B     1562378239B   1561329664B   primary  linux-swap(v1)  type=83
 2      1562378240B  21474836479B  19912458240B  primary  ext4            boot, type=83

# mount | grep sd
/dev/sda2 on / type ext4 (rw,relatime,data=ordered)

# cat /proc/swaps
Filename                                Type            Size    Used    Priority
/dev/sdb1                               partition       1524732 1084    -1

# rear -d -D mkbackup
...

# grep -v '^#' var/lib/rear/layout/disklayout.conf
disk /dev/sda 21474836480 msdos
part /dev/sda 1561329664 1048576 primary none /dev/sda1
part /dev/sda 19912458240 1562378240 primary boot /dev/sda2
disk /dev/sdb 21474836480 msdos
part /dev/sdb 1561329664 1048576 primary none /dev/sdb1
part /dev/sdb 19912458240 1562378240 primary boot /dev/sdb2
fs /dev/sda2 / ext4 uuid=46d7e8be-7812-49d1-8d24-e25ed0589e94 label= blocksize=4096 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=16377 default_mount_options=user_xattr,acl options=rw,relatime,data=ordered
swap /dev/sdb1 uuid=28e43119-dac1-4426-a71a-1d70b26d33d7 label=

I did "rear recover" on a new system (same kind of virtual machine)
with one exactly 20 GiB disk (/dev/sda)
and got (excerpts):

# rear -d -D recover
...
Comparing disks.
Device sdb does not exist.
Switching to manual disk layout configuration.
This is the disk mapping table:
    /dev/sda /dev/sda
Please confirm that '/var/lib/rear/layout/disklayout.conf' is as you expect.

++ select choice in '"${choices[@]}"'
1) View disk layout (disklayout.conf)  3) View original disk space usage      5) Continue recovery
2) Edit disk layout (disklayout.conf)  4) Go to Relax-and-Recover shell       6) Abort Relax-and-Recover
#? 5
++ case "$REPLY" in
++ break
Partition primary on /dev/sda: size reduced to fit on disk.
Please confirm that '/var/lib/rear/layout/diskrestore.sh' is as you expect.

++ select choice in '"${choices[@]}"'
1) View restore script (diskrestore.sh)  3) View original disk space usage        5) Continue recovery
2) Edit restore script (diskrestore.sh)  4) Go to Relax-and-Recover shell         6) Abort Relax-and-Recover
#? 5
++ case "$REPLY" in
++ break
Start system layout restoration.
Creating partitions for disk /dev/sda (msdos)
Creating filesystem of type ext4 with mount point / on /dev/sda2.
Mounting filesystem /
Disk layout created.
Restoring from '/tmp/rear.8JX5LAaL1d7Y3Hn/outputfs/e205/backup.tar.gz'...
...
Restoring finished.
...
Finished recovering your system. You can explore it under '/mnt/local'.
# reboot

The recreated system works well
but I got a bit different partitioning (by one MiB)
regardless that the disk has exactly same size:

# parted -s /dev/sda unit B print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sda: 21474836480B
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 
Number  Start        End           Size          Type     File system  Flags
 1      2097152B     1563426815B   1561329664B   primary               type=83
 2      1563430912B  21474836479B  19911405568B  primary  ext4         boot, type=83

# parted -s /dev/sda unit MiB print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sda: 20480MiB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 
Number  Start    End       Size      Type     File system  Flags
 1      2.00MiB  1491MiB   1489MiB   primary               type=83
 2      1491MiB  20480MiB  18989MiB  primary  ext4         boot, type=83

# grep -v '^#' /var/log/rear/recover/layout/disklayout.conf
disk /dev/sda 21474836480 msdos
part /dev/sda 1561329664 1048576 primary none /dev/sda1
part /dev/sda 19912458240 1562378240 primary boot /dev/sda2
fs /dev/sda2 / ext4 uuid=46d7e8be-7812-49d1-8d24-e25ed0589e94 label= blocksize=4096 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=16377 default_mount_options=user_xattr,acl options=rw,relatime,data=ordered

# cat /proc/swaps 
Filename                                Type            Size    Used    Priority
[no further output]

This shows that swap was not at all migrated
which is probably a missing enhancement.

But I think it is a bug that there is not any LogPrint message
so that the original swap on /dev/sdb1 seems to have been
silently ignored.

jsmeix commented at 2017-05-11 08:37:

FYI regarding calculations with non-bc tools like 'awk' cf.
https://github.com/rear/rear/issues/1269#issuecomment-300719707

[Export of Github issue for rear/rear.]