#1175 Issue closed: tar in ReaR didn't recover ACL and capabilities attributes of files

Labels: enhancement, documentation, fixed / solved / done, not ReaR / invalid

gdha opened issue at 2017-01-19 10:27:

  • rear version (/usr/sbin/rear -V): 1.17.2
  • OS version (cat /etc/rear/os.conf or lsb_release -a): RHEL 7.2
  • rear configuration files (cat /etc/rear/site.conf or cat /etc/rear/local.conf):
OUTPUT=ISO
OUTPUT_URL=nfs://xxx.xxx.xxx.xxx/sswp_appshr/backup
BACKUP=NETFS
BACKUP_URL=nfs://xxx.xxx.xxx.xxx/sswp_appshr/backup
BACKUP_PROG=tar
BACKUP_PROG_OPTIONS="--anchored --acls --xattrs"
BACKUP_TYPE=incremental
FULLBACKUPDAY="Mon"
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp' '/var/crash' '/var/mqm' '/applog' '/oraclient' '/appsw' '/infsw' '/infha4' '/appcore' '/appbin' '/appwrk' '/inftmp' '/appshr' '/applog')
  • Are you using legacy BIOS of UEFI boot? BIOS
  • Brief description of the issue:
    Although applying the following backup options so that ACL could be backed up and recovered
    for the certain files when backing filesystem up, ACL could not be applied after recovering.

gdha commented at 2017-01-19 10:28:

  • Description of problem:
    Although applying the following backup options so that ACL could be backed up and recovered
    for the certain files when backing filesystem up, ACL could not be applied after recovering.

  • Version-Release number of selected component (if applicable):
    RHEL 7.2
    rear-1.17.2-1.el7.noarch

  • How reproducible:
    Always

  • Steps to Reproduce:

  • Created VM

  • Installed RHEL 7.2

  • Configured local repository after mount ISO installation image.

  • Install rear package
    $ yum install rear

  • Configure /etc/reare/local.conf

OUTPUT=ISO
OUTPUT_URL=nfs://xxx.xxx.xxx.xxx/sswp_appshr/backup
BACKUP=NETFS
BACKUP_URL=nfs://xxx.xxx.xxx.xxx/sswp_appshr/backup
BACKUP_PROG=tar
BACKUP_PROG_OPTIONS="--anchored --acls --xattrs"
BACKUP_TYPE=incremental
FULLBACKUPDAY="Mon"
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp' '/var/crash' '/var/mqm' '/applog' '/oraclient' '/appsw' '/infsw' '/infha4' '/appcore' '/appbin' '/appwrk' '/inftmp' '/appshr' '/applog')

  1. Run the following command.
    @ setfacl -m u::r /var/log/messages

  2. Create backup
    $ rear -d -D mkbackup

  3. Confirm if backup is created.
    We confirmed if backup data and ISO image are created in local /backup directory and backup directory in NFS Server.

  4. Perform recovery with rear ISO created.

  5. Recover from rear ISO and the reboot.

  6. Confirm if ACL is properly applied with the following command.
    $ getfacl /var/log/messages

  7. You could see that ACL before backup is not applied.

  8. Actual results:
    ACL changed is not applied after recovery via rear ISO.

  9. Expected results:
    ACL changed should be applied after recovery via rear ISO.

  10. Additional info:
    Here are the configuration and log file that we'd seen

/etc/rear/local.conf

OUTPUT=ISO
OUTPUT_URL=nfs://xxx.xxx.xxx.xxx/sswp_appshr/backup
BACKUP=NETFS
BACKUP_URL=nfs://xxx.xxx.xxx.xxx/sswp_appshr/backup
BACKUP_PROG=tar
BACKUP_PROG_OPTIONS="--anchored --acls --xattrs"
BACKUP_TYPE=incremental
FULLBACKUPDAY="Mon"
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp' '/var/crash' '/var/mqm' '/applog' '/oraclient' '/appsw' '/infsw' '/infha4' '/appcore' '/appbin' '/appwrk' '/inftmp' '/appshr' '/applog')

We see the following "Warning" messages when checking log files regarding recover.

Warning: Cannot acl_from_text


2016-11-10 08:58:21 Including restore/NETFS/default/38_prepare_multiple_isos.sh
2016-11-10 08:58:21 Including restore/NETFS/default/40_restore_backup.sh
2016-11-10 08:58:21 Decrypting disabled
2016-11-10 08:58:21 Restoring tar archive '/tmp/rear.yRb6VE9IiDq3wUW/outputfs/skt-sswpbt04/2016-11-10-1743-I.tar.gz'
2016-11-10 08:58:21 dd if=/tmp/rear.yRb6VE9IiDq3wUW/outputfs/skt-sswpbt04/2016-11-10-1736-F.tar.gz | cat | tar --block-number --totals --verbose --anchored --acls --xattrs --gzip -C /mnt/local/ -x -f -
tar: var/log/messages: Warning: Cannot acl_from_text
10003278+1 records in
10003278+1 records out
5121678690 bytes (5.1 GB) copied, 65.728 s, 77.9 MB/s
Total bytes read: 11235604480 (11GiB, 164MiB/s)
2016-11-10 08:59:31 dd if=/tmp/rear.yRb6VE9IiDq3wUW/outputfs/skt-sswpbt04/2016-11-10-1743-I.tar.gz | cat | tar --block-number --totals --verbose --anchored --acls --xattrs --gzip -C /mnt/local/ -x -f -
tar: var/log/messages: Warning: Cannot acl_from_text
10003847+1 records in
10003847+1 records out
5121969805 bytes (5.1 GB) copied, 92.0977 s, 55.6 MB/s

We see this issue when checking if ACL is properly applied or not at the filesystem recovered as running "rear recover" after entering Recovery menu since booting with "Rear ISO" media.
Initial logged as RHBZ# https://bugzilla.redhat.com/show_bug.cgi?id=1398082

andral commented at 2017-01-26 15:24:

I think we are having the same issue, although not with ACLs but with the sticky bit not being recovered.
We discovered this when running ping as a non-root user after a otherwise successful restore.

 $ ping ntp
ping: icmp open socket: Operation not permitted

Further inspection showed that other files are also affected.

Before restore:

 $ sudo ls -al /usr/bin/a*
-rwxr-xr-x  1 root root 107824 Jul  7  2015 /usr/bin/a2p
-rwxr-xr-x  1 root root   1661 Jul  2  2015 /usr/bin/abs2rel
-rwxr-xr-x  1 root root  29032 Oct 13  2015 /usr/bin/addr2line
-rwxr-xr-x  1 root root  66928 Jul  3  2014 /usr/bin/afio
-rwxr-xr-x  1 root root  19616 Mar 18  2016 /usr/bin/agentxtrap
-rwxr-xr-x  1 root root     29 Jul 12  2016 /usr/bin/alias
-rwxr-xr-x  1 root root   2668 Jan 26  2014 /usr/bin/amuFormat.sh
lrwxrwxrwx. 1 root root      6 Sep  7  2015 /usr/bin/apropos -> whatis
-rwxr-xr-x  1 root root  62576 Oct 13  2015 /usr/bin/ar
-rwxr-xr-x  1 root root  33048 Nov 25  2015 /usr/bin/arch
-rwxr-xr-x  1 root root 365208 Oct 13  2015 /usr/bin/as
-rwxr-xr-x. 1 root root  28800 Sep 16  2014 /usr/bin/aserver
-rwsr-xr-x  1 root root  52944 Jun 22  2015 /usr/bin/at
lrwxrwxrwx  1 root root      2 Apr 18  2016 /usr/bin/atq -> at
lrwxrwxrwx  1 root root      2 Apr 18  2016 /usr/bin/atrm -> at
-rwxr-xr-x  1 root root  11416 Jan 29  2014 /usr/bin/attr
-rwxr-xr-x. 1 root root  19872 Jan 14  2015 /usr/bin/aulast
-rwxr-xr-x. 1 root root  11544 Jan 14  2015 /usr/bin/aulastlog
-rwxr-xr-x. 1 root root  11360 Jan 14  2015 /usr/bin/ausyscall
-rwxr-xr-x. 1 root root  32672 Jan 14  2015 /usr/bin/auvirt
lrwxrwxrwx. 1 root root      4 Sep  7  2015 /usr/bin/awk -> gawk

After restore:

 $ sudo ls -al /usr/bin/a*
-rwxr-xr-x 1 root root 107824 Jul  7  2015 /usr/bin/a2p
-rwxr-xr-x 1 root root   1661 Jul  2  2015 /usr/bin/abs2rel
-rwxr-xr-x 1 root root  29032 Oct 13  2015 /usr/bin/addr2line
-rwxr-xr-x 1 root root  66928 Jul  3  2014 /usr/bin/afio
-rwxr-xr-x 1 root root  19616 Mar 18  2016 /usr/bin/agentxtrap
-rwxr-xr-x 1 root root     29 Jul 12  2016 /usr/bin/alias
-rwxr-xr-x 1 root root   2668 Jan 26  2014 /usr/bin/amuFormat.sh
lrwxrwxrwx 1 root root      6 Sep  4  2015 /usr/bin/apropos -> whatis
-rwxr-xr-x 1 root root  62576 Oct 13  2015 /usr/bin/ar
-rwxr-xr-x 1 root root  33048 Nov 25  2015 /usr/bin/arch
-rwxr-xr-x 1 root root 365208 Oct 13  2015 /usr/bin/as
-rwxr-xr-x 1 root root  28800 Sep 16  2014 /usr/bin/aserver
-rwsr-xr-x 1 root root  52944 Jun 22  2015 /usr/bin/at
lrwxrwxrwx 1 root root      2 Apr 18  2016 /usr/bin/atq -> at
lrwxrwxrwx 1 root root      2 Apr 18  2016 /usr/bin/atrm -> at
-rwxr-xr-x 1 root root  11416 Jan 29  2014 /usr/bin/attr
-rwxr-xr-x 1 root root  19872 Jan 14  2015 /usr/bin/aulast
-rwxr-xr-x 1 root root  11544 Jan 14  2015 /usr/bin/aulastlog
-rwxr-xr-x 1 root root  11360 Jan 14  2015 /usr/bin/ausyscall
-rwxr-xr-x 1 root root  32672 Jan 14  2015 /usr/bin/auvirt
lrwxrwxrwx 1 root root      4 Sep  4  2015 /usr/bin/awk -> gawk

But rpm only complains about these files, which have 'capabilities' changed.

 $ sudo rpm -Va
[snip]
........P    /usr/bin/ping
........P    /usr/bin/ping6
........P    /usr/sbin/arping
........P    /usr/sbin/clockdiff
[snip]

Versions:

 $ lsb_release -d
Description:    Red Hat Enterprise Linux Server release 7.2 (Maipo)
 $ rear -V
Relax-and-Recover 1.18 / Git

Rear config:

OUTPUT=ISO
BACKUP_URL=nfs://xxx/rear
BACKUP=NETFS

A reinstall of the iputils package fixes the issue.
Does anyone have an idea what went wrong during the restore?

Cheers

jsmeix commented at 2017-01-26 15:51:

@andral
you need to run rear with debugging and
then inspect the log file, cf.
"Debugging issues with Relax-and-Recover" at
https://en.opensuse.org/SDB:Disaster_Recovery

Excerpt from
https://github.com/rear/rear/issues/1175#issuecomment-273737147

dd if=/tmp/rear.yRb6VE9IiDq3wUW/outputfs/skt-sswpbt04/2016-11-10-1736-F.tar.gz | cat | tar --block-number --totals --verbose --anchored --acls --xattrs --gzip -C /mnt/local/ -x -f -
tar: var/log/messages: Warning: Cannot acl_from_text

At first glance this looks more like an issue in tar
and not like an issue in ReaR itself, cf.
"Relax-and-Recover versus backup and restore" in
https://en.opensuse.org/SDB:Disaster_Recovery

jsmeix commented at 2017-01-26 18:13:

Google finds only ancient issues at Red Hat about
"tar Cannot acl_from_text" like
https://rhn.redhat.com/errata/RHBA-2010-0224.html
that reads (excerpt)

Last updated on: 2010-03-30
Affected Products: Red Hat Enterprise Linux (v. 5 server)
...
This updated tar package provides fixes for the following bugs:
...
extracting a tar archive that had been created
using the "--xattrs" flag, which saves extended
attribute information to the file, resulted in tar displaying
"Warning: Cannot acl_from_text: Invalid argument"
error messages for many extracted files. This was caused
by an off-by-one coding error, and has been fixed in this
update so that extended attributes are restored correctly
from archive files. (BZ#472553)

Regardless that this one is old and that the error message
is a bit different, it indicates that the root cause is more
likely in tar and not in ReaR.

gdha commented at 2017-02-24 14:57:

@andral According your rpm -Va output some executables like ping have a changed (or missing) capabilities? Can you confirm?
The title of this topic is IMHO wrong - it is not ACL, but capabilities.
Capabilities are saved by tar with the --xattrs flag, which was set in your rear configuration file. Therefore, it might be a bug in your tar executable...

gdha commented at 2017-02-24 16:25:

@andral @jsmeix @gozora @schlomo Nice story:

Test 1 on RHEL 7.2 with --xattrs

# tar --version
tar (GNU tar) 1.26
# rpm -qf $(which tar)
tar-1.26-29.el7.x86_64

# ls -l /bin/ping
-rwxr-xr-x. 1 root root 44896 Jun  5  2015 /bin/ping

# getcap /bin/ping
/bin/ping = cap_net_admin,cap_net_raw+p

# tar --xattrs -cpvf /tmp/ping.tar /bin/ping
# tar --xattrs -xpvf /tmp/ping.tar
# getcap bin/ping
# ls -l bin/ping
-rwxr-xr-x 1 root root 44896 Jun  5  2015 bin/ping

Test 2 with --format=pax (according https://www.gnu.org/software/tar/manual/html_node/Extended-File-Attributes.html)

# tar --xattrs --format=pax -cpvf /tmp/ping.tar /bin/ping
# tar --xattrs --format=pax -xpvf /tmp/ping.tar
bin/ping
# getcap bin/ping
#

Test 3 with --selinux

# tar --xattrs --format=pax --selinux -cpvf /tmp/ping.tar /bin/ping
# tar --xattrs --format=pax --selinux -xpvf /tmp/ping.tar
# getcap bin/ping
#

=> https://rhn.redhat.com/errata/RHBA-2016-2681.html
Previously, using the tar command with the --selinux option failed, as the tar
utility freed memory incorrectly. With this update, the memory is freed
correctly, and tar works as expected with the --selinux option. (BZ#1365645)
New version: tar-1.26-30.el7_2.x86_64.rpm

Test 4 with --acls:

# tar --xattrs --format=pax --selinux --acls -cpvf /tmp/ping.tar /bin/ping
# tar --xattrs --format=pax --selinux --acls -xpvf /tmp/ping.tar
# getcap bin/ping
#

According https://www.lesbonscomptes.com/pages/extattrs.html xattrs are visible with:

# getfattr /bin/ping
# getfattr bin/ping

Test 5 with star:

# star -artype=exustar -xattr-linux -c -f archive.star /bin/ping
star: 7 blocks + 0 bytes (total of 71680 bytes = 70.00k).
# star -artype=exustar -xattr-linux -x -f ./archive.star
#

Unfortunately, my knowledge of star is limited...

Conclusions

tar-1.26-29.el7.x86_64 does not save the capabilities (extend attributes) as describe in the man pages. This should be reported upstream (or via RH).

star hits the same bug, or I used the wrong options? To be investigated by ??

praiskup commented at 2017-03-01 12:27:

tar-1.26-29.el7.x86_64 does not save the capabilities

When you do 'tar -c --xattrs', capabilities (== extended attributes in security.capability namespace) is stored in archive, but it is not automatically restored with --xattrs (only 'user.*' namespace is restored). Have a look ate '--xattrs-include' option for more info. But note that security.capability namespace is stored in pax archive in binary form (more info in rhbz 771927)

gdha commented at 2017-03-01 12:28:

@andral I can only suggest to open a bugzilla with redhat to tackle the tar issue

gdha commented at 2017-03-01 12:47:

@praiskup Thank you for your valuable comment and the interesting link to the tar rhbz 771927.

According http://git.savannah.gnu.org/cgit/tar.git/plain/NEWS?id=release_1_27 :

  • Support for POSIX ACLs, extended attributes and SELinux context.

Starting with this version tar is able to store, extract and list
extended file attributes, POSIX.1e ACLs and SELinux context. This is
controlled by the command line options --xattrs, --acls and --selinux,
correspondingly. Each of these options has a `--no-' counterpart
(e.g. --no-xattrs), which disables the corresponding feature.
Additionally, the options --xattrs-include and --xattrs-exclude allow
you to selectively control for which files to store (or extract) the
extended attributes.
==> however, I think rh added this also in the tar rpm, no?

However, no trace yet of adding capabilities (as in rh patch https://bugzilla.redhat.com/attachment.cgi?id=849830)

praiskup commented at 2017-03-01 12:55:

however, I think rh added this also in the tar rpm, no?

I believe --xattrs-include is in RHEL 7 already.

However, no trace yet of adding capabilities

That's to be discussed upstream.

eblazquez commented at 2017-07-12 11:14:

Hi, it seems that the capabilities, same as with the SELinux and ACLs extended attributes, are not included on the xattrs filters used by tar. We've solved this issue using the following BACKUP_PROG_OPTIONS:

BACKUP_PROG_OPTIONS="--anchored --xattrs --xattrs-include='*.*'"

This way you're modifying the xattrs filter of tar, and telling it to store every extended attribute, including ACLs, SELinux context and capabilities.

This seems to be a similar behaviour than when you execute

getfattr -d <file>

because with that command you don't get neither ACLs nor SELinux contexts, but if you change the filter to match everything with

getfattr -d -m '.*' <file>

you see all this information.

jsmeix commented at 2017-07-13 08:07:

@Eblazquez
many thanks for your analysis.
It helps a lot to better understand the issue!

In ReaR we already use --anchored by default,
see usr/share/rear/conf/default.conf that contains

BACKUP_PROG_OPTIONS="--anchored"

Accordingly only additionally the options

--xattrs --xattrs-include='*.*'

are needed to get full xattrs support via 'tar' backup
(if I understand it correctly)
e.g in etc/rear/local.conf via something like

BACKUP_PROG_OPTIONS="$BACKUP_PROG_OPTIONS --xattrs-include='*.*' --xattrs"

FYI:

We cannot just have in default.conf

BACKUP_PROG_OPTIONS="--anchored --xattrs --xattrs-include='*.*'"

because it seems older tar versions do not support
the --xattrs and --xattrs-include options because
at least on my SLES11 system with GNU tar 1.26
"man tar" shows nothing about xattrs or xattrs-include
but on my openSUSE Leap 42.1 system with GNU tar 1.27
"man tar" shows both xattrs and xattrs-include.

Therefore some magic would be needed to let ReaR
use the --xattrs and --xattrs-include options automatically
if the tar program that is currently installed on the system
where ReaR runs provides --xattrs and --xattrs-include.

But I still think any special backup stuff
does not really belong to ReaR itself, cf.
"Relax-and-Recover versus backup and restore" in
https://en.opensuse.org/SDB:Disaster_Recovery

I think BACKUP_PROG_OPTIONS could be
in general better described in ReaR.

But I think the ReaR documentation is the wrong place
how to use special 'tar' options because ReaR is not
meant to document how to use 'tar'.

schlomo commented at 2017-07-13 10:15:

IMHO we should do our best to support standard setups. As many distros now enable xattr and acl by default we should make an effort to support that as well.

To support older and newer tar versions I suggest dynamically check for the capabilities of tar and to add more options based on that. The code for syslinux is already doing something similar.

jsmeix commented at 2017-07-14 10:38:

@schlomo
I agree - but such things are enhancements for the future:
https://github.com/rear/rear/issues/1411

eblazquez commented at 2017-07-19 12:22:

Hi, I've seen that on the script located on /usr/share/rear/restore/NETFS/default/40_restore_backup.sh, the first action taken if the backup program is tar, is to include the --selinux flag in case it's supported:

case "$BACKUP_PROG" in
    # tar compatible programs here
    (tar)
        # Add the --selinux option to be safe with SELinux context restoration
        if [[ ! $BACKUP_SELINUX_DISABLE =~ ^[yY1] ]]; then
            if tar --usage | grep -q selinux;  then
                BACKUP_PROG_OPTIONS="$BACKUP_PROG_OPTIONS --selinux"
            fi
        fi

Maybe the option to store/restore the xattrs can be included on the scripts on a similar way.

Another issue that I've encountered is that the option --xattrs-include='*.*' works creating the backup, but for some reason is ignored on the restore (it has to do with the * symbols), at least on RHEL7 which is the OS we are using. If you want this to work with capabilities you should include the specific attribute:

BACKUP_PROG_OPTIONS="--anchored --xattrs --xattrs-include=security.capability"

On our case we used that options since we don't use SELinux, but if you do you should also include its xattr, since if you add a xattrs-include mask it ignores the rest, even if you use the --selinux option. Therefor if you use SELinux your options should look like this:

BACKUP_PROG_OPTIONS="--anchored --xattrs --xattrs-include=security.capability --xattrs-include=security.selinux"

Since this issue causes ping not to work on RHEL7 systems, I think it's important enough to at least include this information on the documentation.

gdha commented at 2018-02-23 17:27:

I believe issue is fixed in the meantime: https://gist.github.com/fce4dabba6719a3f615cb8dc02a4c913

jsmeix commented at 2018-02-27 09:37:

I close the issue because it is considered to be fixed.

[Export of Github issue for rear/rear.]