#2369 Issue closed: Set "dd ... bs=1M" blocksize for faster NETFS backup

Labels: enhancement, fixed / solved / done

dsamx opened issue at 2020-04-15 14:42:

Relax-and-Recover (ReaR) Issue Template

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

  • ReaR version ("/usr/sbin/rear -V"):
    Relax-and-Recover 2.4 / Git
    This is actually the SLES15 SP1 package rear23a-2.3.a-9.6.1.x86_64

  • OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"):
    NAME="SLES"
    VERSION="15-SP1"
    VERSION_ID="15.1"
    PRETTY_NAME="SUSE Linux Enterprise Server 15 SP1"
    ID="sles"
    ID_LIKE="suse"
    ANSI_COLOR="0;32"
    CPE_NAME="cpe:/o:suse:sles:15:sp1"

  • ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

AUTOSHRINK_DISK_SIZE_LIMIT_PERCENTAGE=80

OUTPUT=ISO
KEEP_OLD_OUTPUT_COPY=y

BACKUP=NETFS
BACKUP_URL=cifs://....
BACKUP_OPTIONS="cred=/etc/rear/cifs"
BACKUP_INTEGRITY_CHECK=Y
NETFS_KEEP_OLD_BACKUP_COPY=y

PRE_BACKUP_SCRIPT=(
"rm -rf /data/supportconfig/*;"
"/sbin/supportconfig -Q -c -R /data/supportconfig;"
)
POST_BACKUP_SCRIPT=()
PRE_RECOVERY_SCRIPT=()
POST_RECOVERY_SCRIPT=()

BACKUP_PROG_EXCLUDE=(
"${BACKUP_PROG_EXCLUDE[@]}"
/lost+found/*
/mnt/*
/var/cache/*
/var/tmp/*
)

BACKUP_PROG_INCLUDE=(
/boot/grub2/i386-pc/*
/boot/grub2/x86_64-efi/*
/opt/*
/srv/*
/usr/local/*
/var/lib/libvirt/images/*
/var/lib/machines/*
/var/lib/mailman/*
/var/lib/mariadb/*
/var/lib/mysql/*
/var/lib/named/*
/var/lib/pgsql/*
/var/log/*
/var/opt/*
/var/spool/*
)

REQUIRED_PROGS=(
awk
base64
dialog
diff
dos2unix
lsblk
lsof
lspci
md5sum
mkfs.vfat
mkswap
mount
mount.cifs
mount.fuse
mount.nfs
mount.nfs4
parted
ping
ping6
pstree
renice
rsync
screen
sha256sum
tmux
swapon
touch
vim
vimdiff
traceroute
traceroute6
scp
ssh
watch
snapper
chattr
lsattr
clear
infocmp
reset
tabs
toe
tput
tset
"${REQUIRED_PROGS[@]}"
)

COPY_AS_IS=(
/usr/lib/snapper/installation-helper
/etc/snapper/config-templates/default
"${COPY_AS_IS[@]}"
)

MODULES=( 'all_modules' )

  • Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR):
    HPE ProLiant 380 Gen10

  • System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
    x86_64

  • Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
    grub2

  • Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
    local disk, hardware raid

  • Description of the issue (ideally so that others can reproduce it):
    The system was supposed to create a tar backup on a cifs mount. The problem was that the speed was 500KiB/s to a Windows share on a 100MBit line. (The server itself has 1GBit bonding but that does not matter). During tests we never had such slow speed. We discovered that rear basically does the following: tar | dd of=backup.tar
    This is the case if no ISO_MAX_SIZE is set, otherwise dd would be replaced by split.
    We were able to see in the network traffic that only very small packages were sent and ACK'ed by the samba server.
    After adding bs=1M to the dd command the speed was now really fast in comparison: it used the full capacity of the line (about 100MBit/s).

  • Workaround, if any:

We created a one-liner patch that adds bs=1M to the dd command in usr/share/rear/backup/NETFS/default/500_make_backup.sh which worked pretty well for us.

blocksize.patch.txt

jsmeix commented at 2020-04-15 15:19:

@dsamx
thank you for your explanatory report that makes things very clear
and for your analysis what the root cause was and for your fix!

I did https://github.com/rear/rear/pull/2370
so that the other ReaR maintainers could have a look.

I always had a vague feeling that making the backup via "rear mkbackup"
seems to run noticeably slower than restoring it via "rear recover"
but I never analyzed things so perhaps this issue explains it.

jsmeix commented at 2020-04-29 14:03:

I tested it in my usual testing environment where I run "rear mkbackup"
in a virtual machine and the backup.tar.gz goes to a NFS server
and that NFS server is on the host where the virtual machine runs
so in my usual testing environment there is no real outer network used.
Therefore I do not see a big improvement in my testing environment.

Before I had in backup.log from tar and dd

Total bytes written: 3065948160 (2.9GiB, 9.7MiB/s)

2379075+1 records in
2379075+1 records out
1218086742 bytes (1.2 GB, 1.1 GiB) copied, 303.749 s, 4.0 MB/s

and with the added bs=1M I get in backup.log from tar and dd

Total bytes written: 3065036800 (2.9GiB, 11MiB/s)

0+74145 records in
0+74145 records out
1217955985 bytes (1.2 GB, 1.1 GiB) copied, 285.225 s, 4.3 MB/s

The overall time rear mkbackuponly
was before 5 minutes 12 seconds
that decreased a bit to 4 minutes 53 seconds.

Nevertheless bs=1M can make things very much faster:

# time dd if=/boot/vmlinuz of=/dev/null

14320+1 records in
14320+1 records out
7331952 bytes (7.3 MB, 7.0 MiB) copied, 0.0235268 s, 312 MB/s

real    0m0.025s
user    0m0.017s
sys     0m0.008s

# time dd if=/boot/vmlinuz of=/dev/null bs=1M

6+1 records in
6+1 records out
7331952 bytes (7.3 MB, 7.0 MiB) copied, 0.00251888 s, 2.9 GB/s

real    0m0.004s
user    0m0.000s
sys     0m0.004s

# time dd if=/etc/os-release of=/dev/null

0+1 records in
0+1 records out
265 bytes copied, 7.4226e-05 s, 3.6 MB/s

real    0m0.003s
user    0m0.002s
sys     0m0.000s

# time dd if=/etc/os-release of=/dev/null bs=1M

0+1 records in
0+1 records out
265 bytes copied, 5.8819e-05 s, 4.5 MB/s

real    0m0.002s
user    0m0.000s
sys     0m0.002s

(on my Intel Core i3-4000M CPU @ 2.40GHz homeoffice laptop)
so adding bs=1M seems to be an improvement in any case
(it gets a bit faster even if less than 512 bytes need to be copied).

jsmeix commented at 2020-04-30 12:28:

With https://github.com/rear/rear/pull/2370 merged
this issue is done.

@dsamx
thank you for your improvement that makes ReaR working better!

jsmeix commented at 2020-07-21 09:16:

With https://github.com/rear/rear/pull/2459 merged
this issue should now be completely fixed.

[Export of Github issue for rear/rear.]