#3117 Issue open: BACKUP=RSYNC progress not showing with '--progress' in BACKUP_RSYNC_OPTIONS¶
Labels: support / question
DheerajSS2000 opened issue at 2023-12-27 12:31:¶
-
ReaR version ("/usr/sbin/rear -V"):
Relax-and-Recover 2.7 / Git -
If your ReaR version is not the current version, explain why you can't upgrade:
-
OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):
NAME="Fedora Linux"
VERSION="35 (Workstation Edition)"
ID=fedora
VERSION_ID=35
VERSION_CODENAME=""
PLATFORM_ID="platform:f35"
PRETTY_NAME="Fedora Linux 35 (Workstation Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:35"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f35/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=35
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=35
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Workstation Edition"
VARIANT_ID=workstation
- ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
OUTPUT=ISO
OUTPUT_URL="rsync://dheeraj@192.168.8.199::backup/"
BACKUP=RSYNC
BACKUP_PROG=rsync
BACKUP_URL="rsync://dheeraj@192.168.8.199::backup/"
BACKUP_RSYNC_OPTIONS+=(-z -r -v --progress --password-file=/etc/rear/rsync_pass)
-
Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR):
Virtual machine -
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
x86 compatible -
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
BIOS and GRUB -
Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
local disk -
Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"):
[fedora@fedora35 rear]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 512G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 511G 0 part /home
/
sr0 11:0 1 1024M 0 rom
zram0 252:0 0 3.8G 0 disk [SWAP]
- Description of the issue (ideally so that others can reproduce it):
While trying to backup and restore using rsync it is not showing progress , it will be showing only " Backed up 0 MiB [avg 0 KiB/sec]", util entire backup is completed. I tried this with fedora34,35 and 36, the same issue i was facing for all these versions
-
Workaround, if any:
-
Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
rear-fedora35.log
You can drag-drop log files into this editor to create an attachment
or paste verbatim text like command output or file content
by including it between a leading and a closing line of
three backticks like this:
verbatim content
pcahyna commented at 2024-01-02 11:52:¶
I believe this is due to the redirection of stdout from the backup program. The output, including the progress info, can be found in ${TMP_DIR}/${BACKUP_PROG_ARCHIVE}.log :
I understand that this behavior is not very useful in the case of
--progress.
jsmeix commented at 2024-01-05 10:54:¶
There are two different kind of progress things here.
ReaR's own progress subsystem via
usr/share/rear/lib/progresssubsystem.nosh
that shows ProgressInfo "Backed up XXX MiB [avg YYY KiB/sec]"
messages which seems to not work as intended because it is
showing only " Backed up 0 MiB [avg 0 KiB/sec]",
util entire backup is completed
instead of increasing numbers for "XXX MiB"
and current numbers for "YYY KiB/sec".
Additionally in this case here there is
the rsync -v --progress info.
It is questionable whether or not that info
should appear on the user's terminal.
It is also questionable whether or not that info
should appear in the backup log file because:
I am not a BACKUP=RSYNC user but I guess that perhaps
when there is '--progress' or '-v' in BACKUP_RSYNC_OPTIONS
then the code to generate the numbers for the
ProgressInfo message in
backup/RSYNC/default/500_make_rsync_backup.sh
may no longer work, in particular note the line
nfile="$(tail -1 "${TMP_DIR}/${BACKUP_PROG_ARCHIVE}.log")"
I guess because of '--progress' or because of '-v'
the last line in ${TMP_DIR}/${BACKUP_PROG_ARCHIVE}.log
may be no longer what is needed for the code
to generate the numbers for the ProgressInfo messages.
schlomo commented at 2024-01-13 18:20:¶
I'd guess that the root cause of the problem is that you use the
rsync protocol and not the ssh protocol. The way how we "calculate"
the progress of the backup is actually via df and du with those two
functions:
https://github.com/rear/rear/blob/d0496cc751b7f667eb0c8ab0cc5116c1ec05007a/usr/share/rear/backup/RSYNC/default/500_make_rsync_backup.sh#L65-L73
But since you use rsync:// this probably doesn't work so that the
numbers stay 0 all the time. TBH, I cannot think of a clear way how to
provide truthful info on rsync progress without parsing the output.
There are multiple open source tools out there that can do this trick,
and provide a progress bar for rsync. Also, newer rsync versions support
--info=progress2 that provides transfer statistics for the entire job.
I guess that it would be a nice contribution to ReaR to rework the rsync progress info to use that feature instead of checking the free disk space via SSH.
pcahyna commented at 2024-01-15 12:47:¶
Also, newer rsync versions support
--info=progress2that provides transfer statistics for the entire job.
Redirection of output will hide it though, no? Just as in the case of
--progress.
schlomo commented at 2024-01-15 12:49:¶
Yes, but we can pass the rsync output directly to the user via 2>&8 or
something like that.
[Export of Github issue for rear/rear.]