#2911 Issue closed: BACKUP_PROG_COMPRESS_OPTIONS="" makes 'tar ... -C /mnt/local/' untar in $(pwd) i.e. in /root¶

Labels: support / question, fixed / solved / done, not ReaR / invalid

scprsync opened issue at 2023-01-16 19:05:¶

setting
BACKUP_PROG_COMPRESS_OPTIONS=""
BACKUP_PROG_COMPRESS_SUFFIX=""

makes a nice tar file in the backup, but fails to restore the files. i think its still trying to do a tar xzf which fails because it's not gzipped.

what option can i set to make it untar on recovery ?
thank you

~chris

Relax-and-Recover (ReaR) Issue Template¶

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

• ReaR version ("/usr/sbin/rear -V"):

• OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):

• ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

• Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR):

• System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):

• Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):

• Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):

• Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"):

• Description of the issue (ideally so that others can reproduce it):

• Workaround, if any:

• Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):

To paste verbatim text like command output or file content,
include it between a leading and a closing line of three backticks like

```
verbatim content
```

jsmeix commented at 2023-01-17 12:36:¶

@scprsync
how does it fail for you?

scprsync commented at 2023-01-17 13:54:¶

Using NETFS and ISO mode, storing on a Truenas ZFS NFS server.

When restoring a machine, with gzip disabled,
the file download and install doesn’t restore any files.

It tried to do an nfs mount and untar the backup.tar file
but it doesn’t extract any files

/mnt/local has directories, but no files… so of course
the rest of the restoration fails

When allowing gzip it all works, creation, restoration…

When removing the gzip, it makes backup.tar , but then
when restoring it doesn’t untar the files. It actually
shows them being downloaded, just doesn’t untar them

i was guessing that REaR was still trying to tar xzf
which would fail/error out because its not gzipped.

~chris

a different solution i tried was using pigz on creation.
worked well,
much faster to create a backup like 10x-15x depending on processors.
But restoration fails.
I'm guessing because it saves pigz as the restoration command
and its not in the initrd.
If it just used gzip to decompress, it would have worked,
they are compatible.

jsmeix commented at 2023-01-17 14:47:¶

For me it seems to "just work":

RESCUE localhost:~ # rear -D recover
Relax-and-Recover 2.7 / Git
...
Disk layout created.
Running 'restore' stage ======================
Restoring from '/var/tmp/rear.H1FRjykwUNNKLmO/outputfs/localhost/backup.tar' (restore log in /var/lib/rear/restore/recover.backup.tar.725.restore.log) ...
Backup restore program 'tar' started in subshell (PID=4761)
Restored 519 MiB [avg. 106376 KiB/sec] 
Restored 1051 MiB [avg. 107675 KiB/sec] 
Restored 1512 MiB [avg. 103268 KiB/sec] 
...

scprsync commented at 2023-01-17 15:02:¶

rear version 2.6, debian 11

its restoring in / instead of /mnt/local.
watching recovery says

"creating filesystem of type ext4 with mount point / on /dev/mapper/rootvg-rootlv" and then proceeds to untar in / instead of /mnt/local and that of course breaks much of the rest of the script

i see that bind mount still works /mnt/local/sys /mnt/local/dev /mnt/local/proc are there
but /mnt/local/lib..var..home and the rest were untard in /

uploaded log, not sure why its untaring in /

we have a large number of machines, some of them with a good amount of data. standardly with REaR, im getting 10-15MB/s creation rate, not great. i found 2 ways to increase up to 100-150MB/s,

1. not using gzip, breaks for me on restoration
2. using pigz, breaks for me on restoration

scprsync commented at 2023-01-17 16:55:¶

rear-adcphltappserver46.log

jsmeix commented at 2023-01-18 07:38:¶

@scprsync
I had a first look at your
https://github.com/rear/rear/files/10437505/rear-adcphltappserver46.log
and it shows (excerpts):

+++ Print 'Creating filesystem of type ext4 with mount point / on /dev/mapper/rootvg-rootlv.'
+++ wipefs --all --force /dev/mapper/rootvg-rootlv
/dev/mapper/rootvg-rootlv: 2 bytes were erased at offset 0x00000438 (ext4): 53 ef
+++ mkfs -t ext4 -b 4096 -i 16255 -U 73e56cee-dc12-4209-ae25-c48aba05a423 -F /dev/mapper/rootvg-rootlv
...
+++ Print 'Mounting filesystem /'
+++ mkdir -p /mnt/local/
+++ mount -o rw,relatime,errors=remount-ro /dev/mapper/rootvg-rootlv /mnt/local/
+++ mount -o rw,relatime,errors=remount-ro,remount,user_xattr /dev/mapper/rootvg-rootlv /mnt/local/

so dev/mapper/rootvg-rootlv is mounted at /mnt/local/
while "rear recover" is running.
The 'Print' messages are meant from what it was on the original system
(i.e. what was stored in disklayout.conf by "rear mkrescue/mkbackup")
and what it will be finally on the recreated target system
(i.e. after the reboot when "rear recover" had finished).

2023-01-17 16:40:20.180005623 Restoring from '/tmp/rear.X9RyqxbJOSVg4Lo/outputfs/adcphltappserver46/backup.tar' (restore log in /var/lib/rear/restore/recover.backup.tar.815.restore.log) ...
2023-01-17 16:40:20.188737813 dd if=/tmp/rear.X9RyqxbJOSVg4Lo/outputfs/adcphltappserver46/backup.tar | tar --block-number --totals --verbose --anchored --xattrs --xattrs-include=security.capability --xattrs-include=security.selinux --acls  -C /mnt/local/ -x -f -
2023-01-17 16:40:20.189415958 Launched backup restore subshell (PID=1880)
2023-01-17 16:41:21.648781917 Backup restore 'tar' finished with zero exit code
2023-01-17 16:41:21.652773917 Restoring finished (verify backup restore log messages in /var/lib/rear/restore/recover.backup.tar.815.restore.log)

so ... | tar ... -C /mnt/local/ ... should untar in /mnt/local/
(at least this is what "man tar" tells what should happen
as far as I understand what "man tar" tells).
Did you "verify backup restore log messages in /var/lib/rear/restore/recover.backup.tar.815.restore.log"?

gdha commented at 2023-01-18 08:14:¶

@scprsync Could you paste the /etc/rear/local.conf please?

jsmeix commented at 2023-01-18 10:57:¶

@scprsync
thank you for your description that "its untaring in /".

When I tried to reproduce it yesterday
on a KVM/QEMU test VM with 2 GiB of RAM
I had issues like this one which is from right now today:

Restoring from '/var/tmp/rear.21UbIcsbfyqiwo3/outputfs/localhost/backup.tar' (restore log in /var/lib/rear/restore/recover.backup.tar.1497.restore.log) ...
Backup restore program 'tar' started in subshell (PID=5510)
Restored 241 MiB [avg. 49419 KiB/sec] 
Restored 519 MiB [avg. 53188 KiB/sec] 
Restored 754 MiB [avg. 51496 KiB/sec] 
Restored 975 MiB [avg. 49970 KiB/sec] 
Restored 1137 MiB [avg. 46598 KiB/sec] 
Restored 1404 MiB [avg. 47952 KiB/sec] 
Restored 1551 MiB [avg. 45406 KiB/sec] 
Killed

I noticed 'oom-killer' in the dmesg output like

... tar invoked oom-killer ...
...
... Out of memory: Killed process ... (rear) ...

But I failed to see the reason why it runs out of memory.

Now I see it:
It is because 'tar ... -C /mnt/local/' untars in $(pwd)
i.e. in '/root' in the ReaR recovery system
if 'tar' is called without '--gzip'
like this:

Welcome to Relax-and-Recover. Run "rear recover" to restore your system !

RESCUE localhost:~ # du -xhsc /
167M    /
167M    total

RESCUE localhost:~ # rear -D recover
...
Restoring from '/var/tmp/rear.21UbIcsbfyqiwo3/outputfs/localhost/backup.tar' (restore log in /var/lib/rear/restore/recover.backup.tar.1497.restore.log) ...
Backup restore program 'tar' started in subshell (PID=5510)
Restored 241 MiB [avg. 49419 KiB/sec] 
Restored 519 MiB [avg. 53188 KiB/sec] 
Restored 754 MiB [avg. 51496 KiB/sec] 
Restored 975 MiB [avg. 49970 KiB/sec] 
Restored 1137 MiB [avg. 46598 KiB/sec] 
Restored 1404 MiB [avg. 47952 KiB/sec] 
Restored 1551 MiB [avg. 45406 KiB/sec] 
Killed

RESCUE localhost:~ # ls -lhtr
total 0
drwxr-xr-x   3 root root 0 Jun 29  2021 srv
drwxr-xr-x   3 root root 0 Jun 29  2021 home
drwxr-xr-x   3 root root 0 Jun 29  2021 boot
drwxr-xr-x   3 root root 0 Aug  5  2021 opt
drwxr-xr-x   4 root root 0 Jan 17 12:21 rear.github.master
drwx------  14 root root 0 Jan 17 12:50 root
drwxrwxrwt   2 root root 0 Jan 17 12:50 tmp
drwx------  10 root root 0 Jan 18 11:35 var
drwx------ 101 root root 0 Jan 18 11:35 etc
drwxr-xr-x   4 root root 0 Jan 18 11:35 usr

RESCUE localhost:~ # du -xhsc /
1.8G    /
1.8G    total

RESCUE localhost:~ # du -xhsc /mnt/local
4.0K    /mnt/local
4.0K    total

RESCUE localhost:~ # pwd
/root

Because the ReaR recovery system runs in a ramdisk
and my VM has only 2 GiB memory it runs out of memory.

jsmeix commented at 2023-01-18 11:06:¶

I have no idea why in the ReaR recovery system
'tar ... -C /mnt/local/' untars in $(pwd) i.e. in '/root'
when 'tar' is called without '--gzip'
because on my homeoffice laptop it works:

~> mkdir /tmp/mydir1 /tmp/mydir2 /tmp/mydir3 /tmp/mydir4

~> cd /tmp/mydir1

/tmp/mydir1> echo Hello >hello.txt

/tmp/mydir1> tar -c -f hello.tar hello.txt

/tmp/mydir1> tar -c --gzip -f hello.tar.gz hello.txt

/tmp/mydir1> ls -l
total 20
-rw-r--r-- 1 johannes users 10240 Jan 18 12:02 hello.tar
-rw-r--r-- 1 johannes users   227 Jan 18 12:02 hello.tar.gz
-rw-r--r-- 1 johannes users     6 Jan 18 12:02 hello.txt

/tmp/mydir1> cat hello.tar.gz | tar -v --gzip -C /tmp/mydir2 -x -f -
hello.txt

/tmp/mydir1> ls -l /tmp/mydir2
total 4
-rw-r--r-- 1 johannes users 6 Jan 18 12:02 hello.txt

/tmp/mydir1> cat hello.tar | tar -v -C /tmp/mydir3 -x -f -
hello.txt

/tmp/mydir1> ls -l /tmp/mydir3
total 4
-rw-r--r-- 1 johannes users 6 Jan 18 12:02 hello.txt

/tmp/mydir1> cat hello.tar | tar --block-number --totals --verbose --anchored --xattrs --xattrs-include=security.capability --xattrs-include=security.selinux --acls  -C /tmp/mydir4 -x -f -
block 3: hello.txt
block 4: ** Block of NULs **
Total bytes read: 10240 (10KiB, 20MiB/s)

/tmp/mydir1> ls -l /tmp/mydir4
total 4
-rw-r--r-- 1 johannes users 6 Jan 18 12:02 hello.txt

jsmeix commented at 2023-01-18 11:15:¶

In my above ReaR recovery system the
/var/lib/rear/restore/recover.backup.tar.1497.restore.log
shows nothing that tells me (I am not a 'tar' expert) that
'tar ... -C /mnt/local/' does not untar in /mnt/local/
or even why it seems tar ignores '-C /mnt/local/':

++ case "$BACKUP_PROG" in
++ '[' -s /var/tmp/rear.21UbIcsbfyqiwo3/tmp/restore-exclude-list.txt ']'
++ is_true false
++ case "$1" in
++ return 1
++ Log 'dd if=/var/tmp/rear.21UbIcsbfyqiwo3/outputfs/localhost/backup.tar bs=1M | tar --block-number --totals --verbose --anchored' --xattrs --xattrs-include=security.capability --xattrs-include=security.selinux '--acls  -C /mnt/local/ -x -f -'
++ test -w /var/log/rear/rear-localhost.log
++ echo '2023-01-18 11:34:58.792671397 dd if=/var/tmp/rear.21UbIcsbfyqiwo3/outputfs/localhost/backup.tar bs=1M | tar --block-number --totals --verbose --anchored --xattrs --xattrs-include=security.capability --xattrs-include=security.selinux --acls  -C /mnt/local/ -x -f -'
++ tar --block-number --totals --verbose --anchored --xattrs --xattrs-include=security.capability --xattrs-include=security.selinux --acls '' -C /mnt/local/ -x -f -
++ dd if=/var/tmp/rear.21UbIcsbfyqiwo3/outputfs/localhost/backup.tar bs=1M
block 3: boot/grub2/x86_64-efi/
block 6: boot/grub2/i386-pc/
tar: acl_set_file_at: Cannot set POSIX ACLs for file 'boot/grub2/x86_64-efi': Operation not supported
tar: acl_delete_def_file_at: Cannot drop default POSIX ACLs for file 'boot/grub2/x86_64-efi': Operation not supported
block 9: boot/grub2/i386-pc/load.cfg
...
block 3265634: usr/share/locale/vi/LC_TIME/coreutils.mo
block 3265637: usr/share/locale/wa/
block 3265640: usr/share/locale/wa/LC_MESSAGES/
block 3265643: usr/share/locale/wa/LC_MESSAGES/popt.mo

there it stops logging (because oom-killer killed it).

I had noticed those "POSIX ACLs" messages also yesterday
but I did not see the reason (because it untars into a ramdisk).

I use tar (GNU tar) version 1.34 (on openSUSE Leap 15.4).

Of course the actual target filesystems are mounted
below /mnt/local in my above ReaR recovery system:

RESCUE localhost:~ # lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT

NAME                        KNAME     PKNAME    TRAN TYPE FSTYPE      LABEL     SIZE MOUNTPOINT
/dev/sda                    /dev/sda            ata  disk                        20G 
|-/dev/sda1                 /dev/sda1 /dev/sda       part                         8M 
`-/dev/sda2                 /dev/sda2 /dev/sda       part LVM2_member            20G 
  |-/dev/mapper/system-swap /dev/dm-0 /dev/sda2      lvm  swap                    2G 
  |-/dev/mapper/system-home /dev/dm-1 /dev/sda2      lvm  xfs                   5.4G /mnt/local/home
  `-/dev/mapper/system-root /dev/dm-2 /dev/sda2      lvm  btrfs                12.6G /mnt/local
/dev/sr0                    /dev/sr0            ata  rom  iso9660     REAR-ISO 70.9M

And with '--gzip' it works as expected (of course):

RESCUE localhost:~ # rear -D recover
...
Restoring from '/var/tmp/rear.xHEhrqXKQRUiUTZ/outputfs/localhost/backup.tar.gz' (restore log in /var/lib/rear/restore/recover.backup.tar.gz.710.restore.log) ...
Backup restore program 'tar' started in subshell (PID=4712)
Restored 241 MiB [avg. 49419 KiB/sec] 
Restored 335 MiB [avg. 34342 KiB/sec] 
...
Restored 3666 MiB [avg. 35418 KiB/sec] 
OK
Restored 3693 MiB in 111 seconds [avg. 34075 KiB/sec]
Restoring finished (verify backup restore log messages in /var/lib/rear/restore/recover.backup.tar.gz.710.restore.log)

and /var/lib/rear/restore/recover.backup.tar.gz.710.restore.log
contains (excerpts):

++ case "$BACKUP_PROG" in
++ '[' -s /var/tmp/rear.xHEhrqXKQRUiUTZ/tmp/restore-exclude-list.txt ']'
++ is_true false
++ case "$1" in
++ return 1
++ Log 'dd if=/var/tmp/rear.xHEhrqXKQRUiUTZ/outputfs/localhost/backup.tar.gz bs=1M | tar --block-number --totals --verbose --anchored' --xattrs --xattrs-include=security.capability --xattrs-include=security.selinux '--acls --gzip -C /mnt/local/ -x -f -'
++ test -w /var/log/rear/rear-localhost.log
++ echo '2023-01-18 12:31:07.722522270 dd if=/var/tmp/rear.xHEhrqXKQRUiUTZ/outputfs/localhost/backup.tar.gz bs=1M | tar --block-number --totals --verbose --anchored --xattrs --xattrs-include=security.capability --xattrs-include=security.selinux --acls --gzip -C /mnt/local/ -x -f -'
++ tar --block-number --totals --verbose --anchored --xattrs --xattrs-include=security.capability --xattrs-include=security.selinux --acls --gzip -C /mnt/local/ -x -f -
++ dd if=/var/tmp/rear.xHEhrqXKQRUiUTZ/outputfs/localhost/backup.tar.gz bs=1M
block 3: boot/grub2/x86_64-efi/
block 6: boot/grub2/i386-pc/
block 9: boot/grub2/i386-pc/load.cfg
...
block 7560869: sbin/dhclient
1790+1 records in
1790+1 records out
1877849577 bytes (1.9 GB, 1.7 GiB) copied, 107.868 s, 17.4 MB/s
block 7564757: sbin/dhclient-script
block 7564807: sbin/dhclient6
block 7564810: selinux/
block 7564813: root/rear.github.master/var/log/rear/rear-localhost.log
Total bytes read: 3873187840 (3.7GiB, 35MiB/s)
block 7564813: ** Block of NULs **

jsmeix commented at 2023-01-18 12:53:¶

For me 'tar ... -C /mnt/local/' without --gzip
works manually in the ReaR recovery system.
I used the enforced MIGRATION_MODE
so that I can halt "rear recover"
before the backup is restored
at this user dialog

UserInput -I LAYOUT_MIGRATED_CONFIRMATION needed in /usr/share/rear/layout/recreate/default/200_run_layout_code.sh line 168
Confirm the recreated disk layout or go back one step
1) Confirm recreated disk layout and continue 'rear recover'
2) Go back one step to redo disk layout recreation
3) Use Relax-and-Recover shell and return back to here
4) Abort 'rear recover'
(default '1' timeout 300 seconds)
3
UserInput: Valid choice number result 'Use Relax-and-Recover shell and return back to here'

Welcome to Relax-and-Recover.

rear>

In the "Relax-and-Recover shell" I did the following test:

rear> pwd
/var/lib/rear

rear> ls -l
total 0
drwxr-xr-x 6 root root 0 Jan 18 13:33 layout
drwxr-xr-x 2 root root 0 Jan 17 10:39 output
drwxr-xr-x 2 root root 0 Jan 17 10:38 recovery
drwxr-xr-x 2 root root 0 Jan 17 10:39 sysreqs

rear> echo Hello >hello.txt

rear> tar -c -f hello.tar hello.txt

rear> tar -c --gzip -f hello.tar.gz hello.txt

rear> ls -l hello.*
-rw-r--r-- 1 root root 10240 Jan 18 13:43 hello.tar
-rw-r--r-- 1 root root   210 Jan 18 13:43 hello.tar.gz
-rw-r--r-- 1 root root     6 Jan 18 13:43 hello.txt

rear> cat hello.tar.gz | tar -v --gzip -C /mnt/local -x -f -
hello.txt

rear> ls -l /mnt/local/hello.*
-rw-r--r-- 1 root root 6 Jan 18 13:43 /mnt/local/hello.txt

rear> rm /mnt/local/hello.txt

rear> cat hello.tar | tar -v -C /mnt/local -x -f -
hello.txt

rear> ls -l /mnt/local/hello.*
-rw-r--r-- 1 root root 6 Jan 18 13:43 /mnt/local/hello.txt

rear> rm /mnt/local/hello.txt

rear> cat hello.tar | tar --block-number --totals --verbose --anchored --xattrs --xattrs-include=security.capability --xattrs-include=security.sel
inux --acls  -C /mnt/local -x -f -
block 3: hello.txt
block 4: ** Block of NULs **
Total bytes read: 10240 (10KiB, 10MiB/s)

rear> ls -l /mnt/local/hello.*
-rw-r--r-- 1 root root 6 Jan 18 13:43 /mnt/local/hello.txt

rear> rm /mnt/local/hello.txt

rear> exit
Are you sure you want to exit the Relax-and-Recover shell ? y
exit

Leaving the "Relax-and-Recover shell"
makes "rear recover" continue with this:

...
Restoring from '/var/tmp/rear.x5KbBEeRTtGGXGS/outputfs/localhost/backup.tar' (restore log in /var/lib/rear/restore/recover.backup.tar.715.restore.log) ...
Backup restore program 'tar' started in subshell (PID=5291)
Restored 591 MiB [avg. 121069 KiB/sec] 
Restored 890 MiB [avg. 91230 KiB/sec] 
Restored 1199 MiB [avg. 81891 KiB/sec] 
Killed

Currently I have no idea what is going on here...

jsmeix commented at 2023-01-18 13:06:¶

... I found something:

When I change in
/usr/share/rear/restore/NETFS/default/400_restore_backup.sh
the actual restore command line from

    (   case "$BACKUP_PROG" in
            (tar)
...
                if is_true "$BACKUP_PROG_CRYPT_ENABLED" ; then 
...
                else
...
                    dd if=$restore_input bs=1M | $BACKUP_PROG --block-number --totals --verbose "${BACKUP_PROG_OPTIONS[@]}" "${BACKUP_PROG_COMPRESS_OPTIONS[@]}" -C $TARGET_FS_ROOT/ -x -f -
                fi

to

                    dd if=$restore_input bs=1M | $BACKUP_PROG --block-number --totals --verbose "${BACKUP_PROG_OPTIONS[@]}" -C $TARGET_FS_ROOT/ -x -
f -

i.e. when I remove "${BACKUP_PROG_COMPRESS_OPTIONS[@]}"
then it works for me.

jsmeix commented at 2023-01-18 13:15:¶

Argh!

It "just works" (without code modifications) with

BACKUP_PROG_COMPRESS_OPTIONS=()
BACKUP_PROG_COMPRESS_SUFFIX=""

cf. the BACKUP_PROG_COMPRESS_OPTIONS definition in default.conf
https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/conf/default.conf#L1282

# BACKUP_PROG_COMPRESS_OPTIONS is an array ...
...
BACKUP_PROG_COMPRESS_OPTIONS=( --gzip )

FYI:

How an array that is set to the empty string
(actually when its only/first element is set to an empty string)
evaluates versus how it evaluates when it is set to an empty array:

# arr=( --foo )

# arr=""

# ( set -x ; echo --this "${arr[@]}" --that )
+ echo --this '' --that
--this  --that

# arr=()

# ( set -x ; echo --this "${arr[@]}" --that )
+ echo --this --that
--this --that

Note the one more space character in the first echo output
so on first glance it may look same but there is a difference,
cf. "Beware of the emptiness" in
https://github.com/rear/rear/wiki/Coding-Style

How tar behaves with an additional empty argument:

~> ls -l /tmp/mydir*

/tmp/mydir1:
total 20
-rw-r--r-- 1 johannes users 10240 Jan 18 12:02 hello.tar
-rw-r--r-- 1 johannes users   227 Jan 18 12:02 hello.tar.gz
-rw-r--r-- 1 johannes users     6 Jan 18 12:02 hello.txt

/tmp/mydir2:
total 0

/tmp/mydir3:
total 0

~> cd /tmp/mydir2

/tmp/mydir2> cat /tmp/mydir1/hello.tar | tar -v '' -C /tmp/mydir3 -x -f -
hello.txt

/tmp/mydir2> ls -l /tmp/mydir*

/tmp/mydir1:
total 20
-rw-r--r-- 1 johannes users 10240 Jan 18 12:02 hello.tar
-rw-r--r-- 1 johannes users   227 Jan 18 12:02 hello.tar.gz
-rw-r--r-- 1 johannes users     6 Jan 18 12:02 hello.txt

/tmp/mydir2:
total 4
-rw-r--r-- 1 johannes users 6 Jan 18 12:02 hello.txt

/tmp/mydir3:
total 0

That empty arguments make such a crucial difference
for tar and that tar reports nothing that it does
not do what is requested by a '-C ...' argument
loks like a bug in tar.

But that possible issue in tar is not something
we at ReaR upstream will try to work around
(except there is a generic, simple, fail-safe solution).

From my point of view it is in any case a bug
when a program silently(!) ignores what is
requested by its user (in particular when
the program is run in verbose mode).

scprsync commented at 2023-01-18 14:23:¶

My version is slightly different (bs=1M isn’t in it),
but I dropped the BACKUP_PROGRAM_OPTIONS and it now works

So

Thank you

Does this break anything for others?
Can we get this change into the Debian 11 repository
so that I don’t have to track it on 400 computers
and can just install from repo?

And again, thank you.
It’s a nice product.
One that we used to do ourselves locally,
but your is more robust and I like it.
Honestly, more people should know about it.

~chris

scprsync commented at 2023-01-18 14:36:¶

Correction,
I dropped the BACKUP_PROG_COMPRESS_OPTIONS and it works

jsmeix commented at 2023-01-19 07:41:¶

@scprsync
it seems you missed my last entry here
https://github.com/rear/rear/issues/2911#issuecomment-1387056556

It reads in particular

It "just works" (without code modifications) with

BACKUP_PROG_COMPRESS_OPTIONS=()
BACKUP_PROG_COMPRESS_SUFFIX=""

So use

BACKUP_PROG_COMPRESS_OPTIONS=()

to set it to an actually empty array
instead of

BACKUP_PROG_COMPRESS_OPTIONS=""

which sets the first array element to be an empty string

# arr=( one two )

# declare -p arr
declare -a arr=([0]="one" [1]="two")

# arr=""

# declare -p arr
declare -a arr=([0]="" [1]="two")

# arr=()

# declare -p arr
declare -a arr=()

and I assume then things will "just work" also for you
(without code modifications).

For more information and some details behind how tar behaves
in this particular case see my complete last entry
https://github.com/rear/rear/issues/2911#issuecomment-1387056556

jsmeix commented at 2023-01-19 07:54:¶

A side note regarding the current (since ReaR 2.7)

dd if=$restore_input bs=1M | ...

versus the before (up to ReaR 2.6)

dd if=$restore_input | ...

in usr/share/rear/restore/NETFS/default/400_restore_backup.sh
see https://github.com/rear/rear/issues/2369
and https://github.com/rear/rear/issues/2458
where the former reads in particular (excerpts)

... the speed was 500KiB/s ... on a 100MBit line ...
...
After adding bs=1M to the dd command
the speed was now really fast in comparison:
it used the full capacity of the line (about 100MBit/s).

scprsync commented at 2023-01-19 15:02:¶

I understand.

I added the bs=1M option.
Restoration went from ~85MB/s -> 220MB/s, significant improvement.

How large does /tmp need to be with a NETFS option?

Anyway, great tool, thank you a ton for your effort.

~chris

jsmeix commented at 2023-01-20 13:01:¶

FYI:

Simple reproducers for the inexplicable behaviour
when 'tar' gets an additional empty argument
(I use GNU tar 1.34 on openSUSE Leap 15.4):

$ tar -v -x '' -C /targetdir -f archive.tar

and

$ tar -v '' -x -C /targetdir -f archive.tar

untar into the current working directory
while

$ tar -v -x -C /targetdir '' -f archive.tar

and

$ tar '' -v -x -C /targetdir -f archive.tar

untar into /targetdir

I guess this might be related to the "Old Option Style"
that is described in
https://www.gnu.org/software/tar/manual/tar.html#Old-Options

But I fail to see how the additional empty argument
interferes with the -C option and its value.

I asked a colleague and here his reply:

Suppose you have a tar archive ar.tar
that contains two directories: A and B.
So it might have:
  A/foo
  A/bar
  A/baz
  B/this
  B/that
  B/the_other

If you run

 tar -x -f ar.tar -C /tmp A -C /var/tmp B

Then tar would chdir to /tmp, extract the "A" directory,
then chdir to /var/tmp and extract the B directory.

If you don't have the first "-C /tmp",
then "A" would be extracted into the current directory.

If you have the name "" rather than "A" or "B",
then tar would extract the entire archive
(everything starting with "").

With this understanding, your observations make perfect sense.

---

[Export of Github issue for rear/rear.]