#2241 Issue `open`: Backup restore fails for BACKUP_PROG_COMPRESS_OPTIONS=("--zstd") due to missing zstd binaries in recovery system (probably also for --lzip --lzma --lzop)¶

Labels: enhancement, documentation, fixed / solved / done

aasami opened issue at 2019-09-27 13:59:¶

Relax-and-Recover (ReaR) Issue Template¶

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

ReaR version ("/usr/sbin/rear -V"):
Relax-and-Recover 2.5 / Git
OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"):

$ lsb_release -a
LSB Version:    n/a
Distributor ID: ManjaroLinux
Description:    Manjaro Linux
Release:        18.1.0
Codename:       Juhraya

ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

$ cat /etc/rear/site.conf
TIMESYNC=NTP

$ cat /etc/rear/local.conf
OUTPUT=RAWDISK
OUTPUT_URL=nfs://nfs.tu/srv/bkp
BACKUP=NETFS
BACKUP_URL=nfs://nfs.tu/srv/bkp
BACKUP_TYPE=differential
FULLBACKUP_OUTDATED_DAYS=92
BACKUP_PROG_COMPRESS_OPTIONS=("--zstd")
BACKUP_PROG_COMPRESS_SUFFIX=".zst"
KERNEL_FILE="/boot/vmlinuz-$( uname -r|cut -d\. -f1-2 )-x86_64"

Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR):

$ dmidecode -t1
\# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 2.8 present.

Handle 0x000F, DMI type 1, 27 bytes
System Information
        Manufacturer: LENOVO
        Product Name: 20ET004BXS
        Version: ThinkPad E460
        Serial Number: PF0ISVA2
        UUID: 37a370cc-2181-11b2-a85c-e2db9b721f93
        Wake-up Type: Power Switch
        SKU Number: LENOVO_MT_20ET_BU_Think_FM_ThinkPad E460
        Family: ThinkPad E460

System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):

$ uname -m
x86_64

Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
UEFI + GRUB
Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
local SSD
Description of the issue (ideally so that others can reproduce it):
When using option BACKUP_PROG_COMPRESS_OPTIONS=("--zstd")
recovery is not possible due to missing zstd binaries in recovery image.
(The same might be true for options --lzip --lzma and --lzop but I haven't tested it)
Workaround, if any:
Possible: copy missing binary from other location when in recovery mode (not tested)
Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):

jsmeix commented at 2019-09-27 14:56:¶

@aasami
there is currently no automatism in ReaR that would automatically add
what is needed by tar for special BACKUP_PROG_COMPRESS_OPTIONS
settings into the ReaR recovery system.

When you know what binaries and libraries and other files
are needed by tar when using zstd compression
you can add them into the ReaR recovery system
via generic ReaR functionality using config variables like
COPY_AS_IS, REQUIRED_PROGS, and LIBS
see the default.conf description
https://github.com/rear/rear/blob/master/usr/share/rear/conf/default.conf

After "rear mkrescue/mkbackup" you can check things inside the
ReaR recovery system by using KEEP_BUILD_DIR="yes",
see the KEEP_BILD_DIR description in default.conf
https://github.com/rear/rear/blob/master/usr/share/rear/conf/default.conf#L128
via chroot $TMPDIR/rear.XXXXXXXXXXXXXXX/rootfs/
and then try to run tar with the options you like
to test things inside the ReaR recovery system
so that you can check more easily and directly
if all what is needed to run tar as you want it
is included in the ReaR recovery system.

Regarding how BACKUP_PROG_COMPRESS_OPTIONS
is actually used for the tar command see the scripts
usr/share/rear/backup/NETFS/default/500_make_backup.sh
https://github.com/rear/rear/blob/master/usr/share/rear/backup/NETFS/default/500_make_backup.sh
and
usr/share/rear/restore/NETFS/default/400_restore_backup.sh
https://github.com/rear/rear/blob/master/usr/share/rear/restore/NETFS/default/400_restore_backup.sh

gdha commented at 2019-10-05 09:55:¶

@aasami Will you make PR when you got it working?

jsmeix commented at 2019-10-16 12:44:¶

Via
https://github.com/rear/rear/commit/9e07846b460976084e2bb2bdec01b901c42ec413
I added explanatory comments to default.conf
about backup restore that may fail for things like
BACKUP_PROG_COMPRESS_OPTIONS=("--zstd")
due to missing zstd binaries in recovery system
(probably also for --lzip --lzma --lzop).

Accordingly this issue is now at least documented
and because I do not plan to implement an automatism in ReaR
that would automatically include additional things
that are needed by tar into the ReaR recovery system
I close this issue hereby and set the 'needs sponsorship' label.

gdha commented at 2023-09-25 11:21:¶

RHEL 9.2 compression tests with zstandard (zstd) and gzip¶

Ref.: https://facebook.github.io/zstd/

1. Using zstd¶

zstd mkbackup: Archived 3522 MiB in 334 seconds [avg 10799 KiB/sec]
Content of /etc/rear/local.conf:

OUTPUT=ISO
BACKUP=NETFS
BACKUP_URL=nfs://nas/volume1/RearSpace
BACKUP_PROG_COMPRESS_OPTIONS=( '--use-compress-program=zstd' )
COPY_AS_IS+=( $(which zstd) )

Size of archive:

-rw-------  1 admin users 3693584343 Sep 25 09:55 backup.tar.gz

Hint: use following variable setting to avoid confusion with gzip:

BACKUP_PROG_COMPRESS_SUFFIX=".zst"

zstd recover: Restored 6040 MiB in 190 seconds [ avg 32554 LiB/sec]

2. Using gzip¶

gzip mkbackup: Archived 6573 MiB in 674 seconds [avg 9987 KiB/sec]
Content of /etc/rear/local.conf:

OUTPUT=ISO
BACKUP=NETFS
BACKUP_URL=nfs://nas/volume1/RearSpace
#BACKUP_PROG_COMPRESS_OPTIONS=( '--use-compress-program=zstd' )
COPY_AS_IS+=( $(which zstd) )

Size of archive:

-rw-------  1 admin users 6892888474 Sep 25 11:28 backup.tar.gz

gzip recover: Restored 9228 MiB in 316 seconds [avg 29905 KiB/sec]

github-actions commented at 2023-11-25 02:04:¶

Stale issue message

gdha commented at 2023-11-27 13:56:¶

@jsmeix @pcahyna @schlomo Would it be an idea to switch with next ReaR release the gzip with zstandard? It is a much better compression algorithm and faster then gzip too?

jsmeix commented at 2023-11-27 14:23:¶

@gdha
I don't know - I have no experience in this area.

I wonder if switching to a newer compression method
might annoy this or that users who are used to deal
with the traditional compression method?

For example when they are used to do "some more stuff"
with their 'backup.tar.gz' files (e.g. move them to some
other locations and there do something with the backups)
and then unexpectedly some of them (i.e. the newer ones)
behave somehow different in this or that cases?

aasami commented at 2023-11-28 08:01:¶

@gdha I don't know - I have no experience in this area.

I wonder if switching to a newer compression method might annoy this or that users who are used to deal with the traditional compression method?

For example when they are used to do "some more stuff" with their 'backup.tar.gz' files (e.g. move them to some other locations and there do something with the backups) and then unexpectedly some of them (i.e. the newer ones) behave somehow different in this or that cases?

Of course you have to mention it in release notes, so one can be prepared. ;-)

jsmeix commented at 2023-11-28 08:45:¶

Users who read release notes are not those who make me worry ;-)

schlomo commented at 2023-11-28 09:07:¶

Since we do so few releases I would actually consider every release a "major" release, and things are just bound to change.

If we would do monthly releases or such, then we could also afford to be more conservative with regard to changes and hold some changes back for a yearly "major" release.

jsmeix commented at 2023-11-28 09:26:¶

Likely such time ranges look rather different
from business/enterprise customer's point of view
who would prefer they could run their systems stable
for longer than a decade.

Because those SUSE customers pay my salary
my personal preference is backward compatibility.

I even think that nobody likes
any change that happens to one
except of course those few features
where one is explicitly interested in
to get things moved forward.

Simply put: "Nothing must change except what I need!" ;-)

schlomo commented at 2023-11-28 09:34:¶

I think reducing the disk space requirement by 50% is a good motivation to change the compression standard, even if it might be perceived as a breaking change for some users.

However, I think that the change is much less "breaking" than it might seem because we do assume that the backup archive and ReaR image used for recovery are always a "matching set". So the actual breaking change could be that after upgrading ReaR it won't automatically overwrite or remove the old backup archive that has a different suffix.

Maybe @jsmeix you will feel better if we more often increase the major version of ReaR to indicate the many significant changes that we make?

jsmeix commented at 2023-11-28 10:00:¶

OK from me to move forward here
(at least to find out if unexpected major issues appear).

@schlomo
thank you for your explanation!

I agree with your reasoning that the backup archive and ReaR image
used for recovery are always a "matching set", cf. my
"Relax-and-Recover versus backup and restore" section in
https://en.opensuse.org/SDB:Disaster_Recovery
what I wrote there about consistency.

For SUSE I keep separated ReaR versions separated
to help our customers to better deal with possible
backward incompatible changes, see my
"Relax-and-Recover (ReaR) RPM packages for disaster recovery"
and "SUSE support for Relax-and-Recover" sections in
https://en.opensuse.org/SDB:Disaster_Recovery

Furthermore I documented it explicitly in the section
"Version upgrades with Relax-and-Recover" in
https://en.opensuse.org/SDB:Disaster_Recovery
what one has to consider before upgrading ReaR.

So I think SUSE is reasonably well prepared
to help customers to deal with possible
backward incompatible changes in ReaR.

I think the version numbering scheme does not mean much
(i.e. whether or not it is a "major" version upgrade)
because whether or not a particular change causes a "major" issue
for a particular user is often rather unrelated.
From my experience it happens often that rather small things
cause a "major" issue for a particular customer because
in his specific environment some tiny thing is crucial
and he cannot "simply change" his environment with
tons of (possibly hidden/unknown) interdependencies.

gdha commented at 2023-11-28 10:04:¶

We could make it a rear optional setting on the command line, eg -gz or -zstd?

aasami commented at 2023-11-28 11:28:¶

As a compromise, would it be too bad to ask you to include other compression binaries in recovery image, so one can simply use BACKUP_PROG_COMPRESS_OPTIONS=("--[zstd|lzip|lzma|lzop]") and be sure, that it will have no issues on restore? How much bigger would the recovery image get if this gets added by default?

schlomo commented at 2023-11-28 12:27:¶

I think we would do good by adding the relevant compression helpers by default as PROGS. This they will come into the rescue system if available. If not, then selecting a missing compression helper will fail during mkbackup

I'm against using command line options for things that should be part of the configuration because that would reduce the consistency of repeat ReaR interactions.

jsmeix commented at 2023-11-28 15:29:¶

For comparison my results
with current BACKUP=NETFS defaults
versus BACKUP=NETFS with zstd
on one same original system (QEMU/KVM virtual machine)
with "rear recover" on a second same virtual machine:

With current BACKUP=NETFS defaults:

Excerpts from "rear mkbackup" log (long line folded here):

2023-11-28 15:13:55.020077714 Including backup/NETFS/default/500_make_backup.sh
...
2023-11-28 15:13:55.141022722 tar --warning=no-xdev --sparse
 --block-number --totals --verbose --no-wildcards-match-slash
 --one-file-system --ignore-failed-read --anchored --xattrs
 --xattrs-include=security.capability
 --xattrs-include=security.selinux
 --acls --gzip
 -X /var/tmp/rear.Q4OcZml0OlbxMyF/tmp/backup-exclude.txt
 -C / -c -f -
 /boot/grub2/x86_64-efi /boot/grub2/i386-pc /opt /srv
 /usr/local /root /tmp /var /home /
 /root/rear.pull3089/var/log/rear/rear-localhost.log
 | dd of=/var/tmp/rear.Q4OcZml0OlbxMyF/outputfs/localhost/backup.tar.gz bs=1M
...
2023-11-28 15:18:37.561304506 Archived 1461 MiB in 281 seconds [avg 5326 KiB/sec]

backup.tar.gz has 1532658271 bytes = 1461.657 MiB = 1.427 GiB

rear-localhost.iso has 74397696 bytes = 70.951 MiB

Excerpts from "rear recover" log (long line folded here):

2023-11-28 15:33:25.451963621 Including restore/NETFS/default/400_restore_backup.sh
...
2023-11-28 15:33:25.481183629 dd
 if=/var/tmp/rear.JrfrCWkfRdN4TeX/outputfs/localhost/backup.tar.gz bs=1M
 | tar --block-number --totals --verbose --anchored --xattrs
   --xattrs-include=security.capability --xattrs-include=security.selinux
   --acls --gzip -C /mnt/local/ -x -f -
2023-11-28 15:35:34.245339277 Restored 3327 MiB in 128 seconds [avg. 26623 KiB/sec]

BACKUP=NETFS with zstd:

Excerpt from etc/rear/local.conf

PROGS+=( zstd )
BACKUP_PROG_COMPRESS_OPTIONS=( '--use-compress-program=zstd' )
BACKUP_PROG_COMPRESS_SUFFIX=".zst"