#2760 Issue closed: ERROR: Not enough disk space available on $RSYNC_HOST:$RSYNC_PATH¶
Labels: enhancement, cleanup, fixed / solved / done
ramadan222 opened issue at 2022-02-14 15:58:¶
Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):
-
ReaR version ("/usr/sbin/rear -V"):
Relax-and-Recover 2.6 / 2020-06-17 -
OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):
Red Hat Enterprise Linux release 8.5 (Ootpa) -
ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
OUTPUT=ISO
OUTPUT_URL=rsync://root@192.168.1.10/backup
BACKUP=RSYNC
BACKUP_URL=rsync://root@192.168.1.10/backup
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" /media' '/var/tmp' '/var/crash' '/proc' '/run' 'run/user/1000' '/backup' '/boot' '/dev' '/dev/shm' '/home/mostafa' '/lib' '/lib64' '/mnt' '/sys')
#ONLY_INCLUDE_VG=( rhel )
#USING_UEFI_BOOTLOADER=1
-
Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR):
Baremetal -
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
X86 -
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
UEFI -
Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
local disk -
Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"):
NAME KNAME PKNAME TRAN TYPE FSTYPE LABEL SIZE MOUNTPOINT
/dev/loop0 /dev/loop0 loop iso9660 RHEL-8-5-0-BaseOS-x86_64 10.2G /mnt/iso
/dev/sda /dev/sda disk 237.5G
|-/dev/sda1 /dev/sda1 /dev/sda part vfat 600M /boot/efi
|-/dev/sda2 /dev/sda2 /dev/sda part xfs 1G /boot
`-/dev/sda3 /dev/sda3 /dev/sda part LVM2_member 235.9G
|-/dev/mapper/rhel-root /dev/dm-0 /dev/sda3 lvm xfs 70G /
|-/dev/mapper/rhel-swap /dev/dm-1 /dev/sda3 lvm swap 4G [SWAP]
`-/dev/mapper/rhel-home /dev/dm-2 /dev/sda3 lvm xfs 161.9G /home
/dev/sdb /dev/sdb disk 136G
/dev/sdc /dev/sdc usb disk 1.9G
|-/dev/sdc1 /dev/sdc1 /dev/sdc part vfat ESXi 4M
|-/dev/sdc5 /dev/sdc5 /dev/sdc part vfat 250M
|-/dev/sdc6 /dev/sdc6 /dev/sdc part vfat 250M
|-/dev/sdc7 /dev/sdc7 /dev/sdc part 110M
`-/dev/sdc8 /dev/sdc8 /dev/sdc part vfat 286M
/dev/sr0 /dev/sr0 ata rom 1024M
- Description of the issue (ideally so that others can reproduce it):
ERROR: Not enough disk space available ...
on RSYNC target server
I need to backup only a few FS with max capacity 15 GB
On target i have 45 GB free space
How can i set the rear to ignore calculate ALL local space ?
so only the desired FS can be copied not all the server
-
Workaround, if any: No
-
Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
To paste verbatim text like command output or file content,
include it between a leading and a closing line of three backticks like
# rear -D mkrescue/mkbackup/recover
Relax-and-Recover 2.6 / 2020-06-17
Running rear mkrescue/mkbackup/recover (PID 3073425)
Using log file: /var/log/rear/rear-hostname.log
Running workflow mkrescue/mkbackup/recover on the normal/original system
ERROR: The specified command 'mkrescue/mkbackup/recover' does not exist!
Exiting rear mkrescue/mkbackup/recover (PID 3073425) and its descendant processes ...
Running exit tasks
You should also rm -Rf --one-file-system /tmp/rear.lsoWqujrvVR5qUR
```
verbatim content
```
jsmeix commented at 2022-02-15 13:27:¶
I am not a RHEL user so I cannot comment on RHEL specific issues.
If you have a support contract with Red Hat you may better ask there.
In general (regardless that this issue is about RHEL) you may have a
look at
https://en.opensuse.org/SDB:Disaster_Recovery
therein in particular the section
"First steps with Relax-and-Recover"
ramadan222 commented at 2022-02-15 14:44:¶
@jsmeix
thank you for your answer ,
my question was not related to space specifically
When I run the backup the exact data I choose to backup is 14 GB
The target server directory have 45 GB ( enough space for backup )
the rear application is calculating all VG space ( used and not used :
14 GB used and 114 GB not used )
Please how can I set the rear config file to check only space used
really and not all the VG ?
Thank you
gdha commented at 2022-02-17 16:49:¶
@ramadan222 See script
/backup/RSYNC/default/450_calculate_req_space.sh where the magic
happens. You could replace StopIfError with LogPrintError and that's
it. Or, introduce a variable in the default.conf file to skip the error
in this script. You may propose a pull request for this and we will
guide you ;-)
ramadan222 commented at 2022-02-18 20:32:¶
@gdha
Thank you for your response
I changed StopIfError to LogPrintError
and run backup and got this error :
# rear -d -v mkbackuponly
Relax-and-Recover 2.6 / 2020-06-17
Running rear mkbackuponly (PID 205043)
Using log file: /var/log/rear/rear-testmachine.205043.log
Running workflow mkbackuponly on the normal/original system
Found EFI system partition /dev/sda1 on /boot/efi type vfat
Using UEFI Boot Loader for Linux (USING_UEFI_BOOTLOADER=1)
Using autodetected kernel '/boot/vmlinuz-4.18.0-348.2.1.el8_5.x86_64' as kernel in the recovery system
Creating disk layout
Using guessed bootloader 'EFI' (found in first bytes on /dev/sda)
Verifying that the entries in /tmp/rear.2OsmPtNlOkLNeJ0/tmp/backuplayout.conf are correct ...
Calculating size of /mnt
Failed to determine size of /mnt
Calculating size of /home
Failed to determine size of /home
Calculating size of /
Failed to determine size of /
Calculating size of /boot/efi
Failed to determine size of /boot/efi
Calculating size of /boot
Failed to determine size of /boot
Estimated size of local file systems is 124981 MB
Calculating size of 192.168.1.10:/backup
Failed to determine size of /backup
Not enough disk space available on 192.168.1.10:/backup (43560160 < 127981372)
Creating rsync backup on '1192.168.1.10:/backup'
Backed up 0 MiB [avg 0 KiB/sec]OK
ERROR:
There was an error (Error in file I/O) during backup creation.
Please check the destination and see '/var/log/rear/rear-testmachine.205043.log' for more information.
If the error is related to files that cannot and should not be saved by
rsync, they should be excluded from the backup.
Some latest log messages since the last called script 500_make_rsync_backup.sh:
2022-02-18 15:20:50.922366468 /lib64
2022-02-18 15:20:50.925547515 /mnt
2022-02-18 15:20:50.928645024 /sys
2022-02-18 15:20:50.931874173 /tmp/rear.2OsmPtNlOkLNeJ0
2022-02-18 15:20:50.935293087 Creating rsync backup on '192.168.1.10:/backup'
2022-02-18 15:20:50.945398848 rsync --sparse --archive --hard-links --numeric-ids --stats --relative --one-file-system --delete --exclude-from=/tmp/rear.2OsmPtNlOkLNeJ0/tmp/backup-exclude.txt --dele
rsync: mkdir "/backup/testmachine/backup" failed: No such file or directory (2)
rsync error: error in file IO (code 11) at main.c(657) [Receiver=3.1.2]
Aborting due to an error, check /var/log/rear/rear-testmachine.205043.log for details
Exiting rear mkbackuponly (PID 205043) and its descendant processes ...
Running exit tasks
You should also rm -Rf --one-file-system /tmp/rear.2OsmPtNlOkLNeJ0
Terminated
thomasmerz commented at 2022-02-25 18:13:¶
@gdha
@ramadan222 See script
/backup/RSYNC/default/450_calculate_req_space.shwhere the magic happens. You could replaceStopIfErrorwithLogPrintErrorand that's it.
I had also the same problem with Relax-and-Recover 2.6 / Git on my
openSUSE 15.3:
Estimated size of local file systems is 1924210 MB
Calculating size of localhost:/automnt/external_2tb/rear/
Not enough disk space available on localhost:/automnt/external_2tb/rear/ (1952558172 < 1970391348)
With this site.conf:
OUTPUT=ISO
OUTPUT_URL=rsync://root@localhost/automnt/external_2tb/iso/
BACKUP=RSYNC
BACKUP_URL=rsync://root@localhost/automnt/external_2tb/rear/
ISO_MKISOFS_BIN=/usr/bin/ebiso
BACKUP_PROG_INCLUDE=( '/' '/boot/efi' '/boot/grub2/i386-pc' '/boot/grub2/x86_64-efi' '/home' '/home/birgit/.VirtualBox' '/home/thomas/.VirtualBox' '/home/thomas/MediathekView' '/home/thomas/Music' '/opt' '/srv' '/usr/local' '/var/cache' '/var/crash' '/var/lib/libvirt/images' '/var/lib/machines' '/var/lib/mailman' '/var/lib/mariadb' '/var/lib/mysql' '/var/lib/named' '/var/lib/pgsql' '/var/log' '/var/opt' '/var/spool' '/var/tmp')
BACKUP_ONLY_INCLUDE="yes"
All my filesystems/partitions have this size in M (not all are included in rear-backup!):
/dev/sda1 | 5
/dev/sda3 | 66665
/dev/sda5 | 195141
/dev/sdb1 | 1156521
/dev/sdc1 | 34
/dev/sdc10 | 39080
/dev/sdc2 | 137
/dev/sdc3 | 1369
/dev/sdc4 | 10
/dev/sdc5 | 10
/dev/sdc6 | 10
/dev/sdc7 | 54096
/dev/sdc8 | 55206
/dev/sdc9 | 314128
765G are available on my backup-medium, my backup-data on backup-medium is currently 1.1T overall. But not all data must be backupped "on top" on every run! So this should really not stop ReaR from working. A warning should be enough in my opinion. Every backup-"admin" should take care to have enough backup-(free-)space available. Or is the calculation misleading because 1.1 T is the maximum amount to be backupped because this are already 1.1 T backupped and 765 G still are free. I really don't have sooooo much changes/updates (100s of G) every day on my home computer 😲
So, what's the problem for ReaR? 🤔
Or, introduce a variable in the default.conf file to skip the error in this script. You may propose a pull request for this and we will guide you ;-)
This would be good enough as a "plan b". But this won't fix the root cause.
Update:
After the "Fix" in 450_calculate_req_space.sh there's still 762G
(before: 765G) available after a successfull backup on my
backup-medium.
thomasmerz commented at 2022-02-25 18:25:¶
Please check if this is more than "support / question". For me it smells like a "bug" 🐛
jsmeix commented at 2022-02-28 11:03:¶
I am not a rsync user so cannot comment on rsync specific issues.
I like to make only a general comment regarding things as in
backup/RSYNC/default/450_calculate_req_space.sh
As far as I see we do not do such tests for other backup methods
(i.e. whether or not there is enough space where the backup will be
stored)
so I wonder why we do it for BACKUP=RSYNC
(but then only when RSYNC_PROTO is 'ssh').
In general regarding ReaR and backup methods see the section
"Relax-and-Recover versus backup and restore" in
https://en.opensuse.org/SDB:Disaster_Recovery
Accordingly I suggest to drop the whole
backup/RSYNC/default/450_calculate_req_space.sh
script and leave it to the user to ensure there is enough space
where the backup will be stored in his particular case
as we do it for all other backup methods.
Again: I am not a rsync user so cannot comment on rsync specific
issues
so things would be different if there is a reason why BACKUP=RSYNC
requires to have such a test in ReaR.
github-actions commented at 2022-04-30 03:06:¶
Stale issue message
thomasmerz commented at 2022-05-01 10:43:¶
Accordingly I suggest to drop the whole
backup/RSYNC/default/450_calculate_req_space.sh
script and leave it to the user to ensure there is enough space
where the backup will be stored in his particular case
as we do it for all other backup methods.
I think we are missing a PR or do you @jsmeix take care of this being taken out of the right scripts? Both would be fine for me, but before making a PR I wait for an answer from you. Thanks a lot so long.
jsmeix commented at 2022-05-02 07:00:¶
According to
# git log --follow -p usr/share/rear/backup/RSYNC/default/450_calculate_req_space.sh
the script backup/RSYNC/default/450_calculate_req_space.sh
originated by @gdha via
https://github.com/rear/rear/commit/2e226ebaec332200ced582f091d025e43b2f84c5
so it is @gdha who should decide here.
gdha commented at 2022-05-02 15:36:¶
The question is shall we drop the code mentioned below in script
/usr/share/rear/backup/RSYNC/default/450_calculate_req_space.sh?
case $RSYNC_PROTO in
(ssh)
LogPrint "Calculating size of $RSYNC_HOST:$RSYNC_PATH"
ssh -l $RSYNC_USER $RSYNC_HOST "df -P $RSYNC_PATH" >$TMP_DIR/rs_size
StopIfError "Failed to determine size of $RSYNC_PATH"
_div=1 # 1024-blocks
grep -q "512-blocks" $TMP_DIR/rs_size && _div=2 # HPUX: divide with 2 to get kB size
_remote_size=$( tail -n 1 $TMP_DIR/rs_size | awk '{print $2}' )
_remote_size=$(( _remote_size / _div ))
[[ $_remote_size -gt $_local_size ]]
StopIfError "Not enough disk space available on $RSYNC_HOST:$RSYNC_PATH ($_remote_size < $_local_size)"
;;
(rsync)
# TODO: how can we calculate the free size on remote system via rsync protocol??
:
;;
esac
Vote for it:
- [x] Yes
- [ ] No
thomasmerz commented at 2022-05-03 06:47:¶
@gdha - I can't click your vote. My vote: Yes.
jsmeix commented at 2022-05-05 10:29:¶
With
https://github.com/rear/rear/pull/2797
merged
this issue should be avoided so I consider this issue to be done.
jsmeix commented at 2022-05-05 10:31:¶
@thomasmerz @ramadan222
if possible for you please test our current ReaR GitHub master code,
see the section
"Testing current ReaR upstream GitHub master code" in
https://en.opensuse.org/SDB:Disaster_Recovery
gdha commented at 2022-05-05 15:35:¶
@thomasmerz @ramadan222 New RPM has the included updates:
http://download.opensuse.org/repositories/Archiving:/Backup:/Rear:/Snapshot/CentOS_8/x86_64/rear-2.6-152.git.4764.58ddf3e.master.changed.el8.x86_64.rpm
thomasmerz commented at 2022-05-06 14:42:¶
@gdha , can you also be make this RPM available for/in https://download.opensuse.org/repositories/Archiving:/Backup:/Rear:/Snapshot/openSUSE_Leap_15.3/ ?
I'm stuck with this version:
🦎🖥 thomas@merz-nimbus:~/temp [0/1492]
16:36 $ zypper se -s rear
…
S | Name | Type | Version | Arch | Repository
---+-----------------------+------------+---------------------------------+--------+---------------------------------------------------------
| libretro-pcsx-rearmed | package | 0~git20200507-bp153.1.14 | x86_64 | openSUSE-Leap-15.3-Oss
i+ | rear | package | 1625557278.1053e0e5-1 | x86_64 | Relax-and-Recover snapshot packages (openSUSE_Leap_15.3)
v | rear | package | 2.6.5-2 | x86_64 | Relax-and-Recover snapshot packages (openSUSE_Leap_15.3)
v | rear | package | 2.6-150.git.4698.a16e840.master | x86_64 | Relax-and-Recover snapshot packages (openSUSE_Leap_15.3)
| rear | srcpackage | 1625557278.1053e0e5-1 | noarch | Relax-and-Recover snapshot packages (openSUSE_Leap_15.3)
| rear | srcpackage | 2.6.5-2 | noarch | Relax-and-Recover snapshot packages (openSUSE_Leap_15.3)
| rear | srcpackage | 2.6-150.git.4698.a16e840.master | noarch | Relax-and-Recover snapshot packages (openSUSE_Leap_15.3)
| rear23a | package | 2.3.a-19.9 | x86_64 | openSUSE-Leap-15.3-Oss
i+ | yast2-rear | package | 4.3.0-1.23 | noarch | openSUSE-Leap-15.3-Oss
And a downloaded version from CentOS repo won't work:
🦎🖥 thomas@merz-nimbus:~/Downloads [0/1497]
16:40 $ zypper in ./rear-2.6-152.git.4764.58ddf3e.master.changed.el8.x86_64.rpm
Loading repository data...
Reading installed packages...
The selected package 'rear-2.6-152.git.4764.58ddf3e.master.changed.el8.x86_64' from repository 'Plain RPM files cache' has lower version than the installed one. Use 'zypper install --oldpackage rear-2.6-152.git.4764.58ddf3e.master.changed.el8.x86_64' to force installation of the package.
Resolving package dependencies...
Nothing to do.
gdha commented at 2022-05-06 15:15:¶
@thomasmerz There is a RPM available under http://download.opensuse.org/repositories/Archiving:/Backup:/Rear:/Snapshot/openSUSE_Leap_15.3/x86_64/ I see and I was able to download it.
thomasmerz commented at 2022-05-06 16:15:¶
TL;DR:
there has been a bogus version in previous version(s) which made me
curios:
🦎🖥 thomas@merz-nimbus:~/Downloads [0/1595]
17:44 $ rpm -qi rear-1625557278.1053e0e5-1.x86_64
Name : rear
Version : 1625557278.1053e0e5
Release : 1
Architecture: x86_64
Install Date: Fri 04 Feb 2022 04:40:58 PM CET
Group : Applications/File
Size : 2338433
License : GPL-3.0
Signature : RSA/SHA256, Tue 06 Jul 2021 11:29:32 AM CEST, Key ID 082caa5eadb2e40a
Source RPM : rear-1625557278.1053e0e5-1.src.rpm
Build Date : Tue 06 Jul 2021 11:29:25 AM CEST
vs. your newly added version:
🦎🖥 thomas@merz-nimbus:~/Downloads [0/1593]
17:43 $ rpm -qi ./rear-2.6-152.git.4764.58ddf3e.master.changed.x86_64.rpm
Name : rear
Version : 2.6
Release : 152.git.4764.58ddf3e.master.changed
Architecture: x86_64
Install Date: (not installed)
Group : Applications/File
Size : 2594094
License : GPL-3.0
Signature : RSA/SHA256, Thu 05 May 2022 05:27:15 PM CEST, Key ID 082caa5eadb2e40a
Source RPM : rear-2.6-152.git.4764.58ddf3e.master.changed.src.rpm
Build Date : Thu 05 May 2022 05:27:08 PM CEST
"Version: 1625557278.1053e0e5" seems to be totally nonsense. I used the "--oldpackage" parameter with "zypper install" and tested:
Currently I'm far away from running out of disk space, so I couldn't provoke this message before (and even after), but rear ran fine without any errors or anomalies and as good as it did before 👍🏻
pcahyna commented at 2022-06-29 15:40:¶
@gdha thanks, I believe that commenting out the code was a good idea, I prefer to avoid such special code that assumes one can ssh to the rsync server, if possible.
[Export of Github issue for rear/rear.]