#2577 PR `closed`: Fix rsync --fake-super option on remote server¶

Labels: support / question, fixed / solved / done

bwelterl opened issue at 2021-02-25 17:39:¶

With """, in the expected arg for rsync:
--rsync-path="""rsync_ --fake-super""", the arg will remove all """, thus the --fake-super will not be executed on the remote server.
And the ownerships will not be saved and restored.

A workaround is to add
BACKUP_RSYNC_OPTIONS=( "${BACKUP_RSYNC_OPTIONS[@]}" -M--fake-super )

in /etc/rear/local.conf

Thanks

Relax-and-Recover (ReaR) Pull Request Template¶

Please fill in the following items before submitting a new pull request:

Pull Request Details:¶

Type: Bug Fix
Impact: High
Reference to related issue (URL):
How was this pull request tested?

Backup/restore with rsync on RHEL8.2

Brief description of the changes in this pull request:

Only changed """ to "

jsmeix commented at 2021-03-02 10:46:¶

By the way
in prep/RSYNC/default/150_check_rsync_protocol_version.sh

if [ ${BACKUP_RSYNC_OPTIONS[@]/--fake-super/} != ${BACKUP_RSUNC_OPTIONS[@]} ]; then

that was added via
https://github.com/rear/rear/commit/3548430302cef3a8445de970eb763da33d9f6e8e
that condition would be always true because ${BACKUP_RSUNC_OPTIONS[@]} is empty
because BACKUP_RSYNC_OPTIONS is not BACKUP_RSUNC_OPTIONS (typo!) :-(
but actually that condition is always false because its syntax is simply plain wrong:

# unset BACKUP_RSYNC_OPTIONS

# BACKUP_RSYNC_OPTIONS=( foo bar )

# if [ ${BACKUP_RSYNC_OPTIONS[@]/--fake-super/} != ${BACKUP_RSUNC_OPTIONS[@]} ]; then echo Y ; fi
bash: [: bar: binary operator expected

# BACKUP_RSYNC_OPTIONS+=( '--rsync-path="rsync --fake-super"' )

# if [ ${BACKUP_RSYNC_OPTIONS[@]/--fake-super/} != ${BACKUP_RSUNC_OPTIONS[@]} ]; then echo Y ; fi
bash: [: too many arguments

What a shitty piece of shitty code shit - cf.
https://motherfuckingwebsite.com/
("Yes, this is f** satire ...")

I think

if [ "${BACKUP_RSYNC_OPTIONS[*]/--fake-super/}" != "${BACKUP_RSYNC_OPTIONS[*]}" ];

could work.
But I will have to have a closer look at that code.
But only as time permits - no promises - in particular not for such awful code :-(

jsmeix commented at 2021-03-03 12:59:¶

@gdha
I am totally confused about the rsync implementation in ReaR.
I never used ReaR with rsync.

I noticed we have two scripts
usr/share/rear/backup/NETFS/default/500_make_backup.sh
and
usr/share/rear/backup/RSYNC/default/500_make_rsync_backup.sh
which seem to do the same
and I fail to understand how rsync is meant to work in ReaR.

Could you perhaps help me to understand how rsync is meant to work in ReaR?

gdha commented at 2021-03-03 13:09:¶

@jsmeix

if BACKUP=NETFS and BACKUP_PROG=rsync then you will use usr/share/rear/backup/NETFS/default/500_make_backup.sh to make a backup archive with the rsync (over ssh only) executable
if BACKUP=RSYNC (then BACKUP_PROG=rsync is always defined) and then you will use usr/share/rear/backup/RSYNC/default/500_make_rsync_backup.sh (in this case you can use rsync over ssh or use the remote rsync daemon approach - see man page of rsync for the details)
I use most of the times the RSYNC backup method (as you can browse easily though your directories/files on the remote server)

jsmeix commented at 2021-03-18 12:45:¶

I cannot reproduce it on SLES15 that has GNU bash, version 4.4.23

My etc/rear/local.conf

OUTPUT=ISO
BACKUP=RSYNC
BACKUP_URL=rsync://root@192.168.122.1/nfs/rearRSYNC/
BACKUP_ONLY_INCLUDE="yes"
BACKUP_PROG_INCLUDE=( /home )
SSH_ROOT_PASSWORD='rear'
USE_DHCLIENT="yes"
PROGRESS_MODE="plain"
PROGRESS_WAIT_SECONDS="3"
FIRMWARE_FILES=( 'no' )
MODULES=( 'loaded_modules' )

In particular I use

BACKUP_ONLY_INCLUDE="yes"
BACKUP_PROG_INCLUDE=( /home )

to get only a small backup for my test here.

With the unchanged current ReaR upstream master code

BACKUP_RSYNC_OPTIONS+=( --rsync-path="""rsync --fake-super""" )

it "just works" for me and
I get in my rear debug log file (excerpts)

++ BACKUP_RSYNC_OPTIONS+=(--rsync-path="""rsync --fake-super""")
...
++ Log rsync --sparse --archive --hard-links --numeric-ids --stats '--rsync-path=rsync --fake-super' --relative --one-file-system --delete --exclude-from=/tmp/rear.2o92TKFmGHCniuX/tmp/backup-exclude.txt --delete-excluded /home root@192.168.122.1:/nfs/rearRSYNC//localhost/backup
++ echo '2021-03-18 13:23:17.931957271 rsync --sparse --archive --hard-links --numeric-ids --stats --rsync-path=rsync --fake-super --relative --one-file-system --delete --exclude-from=/tmp/rear.2o92TKFmGHCniuX/tmp/backup-exclude.txt --delete-excluded /home root@192.168.122.1:/nfs/rearRSYNC//localhost/backup'
2021-03-18 13:23:17.931957271 rsync --sparse --archive --hard-links --numeric-ids --stats --rsync-path=rsync --fake-super --relative --one-file-system --delete --exclude-from=/tmp/rear.2o92TKFmGHCniuX/tmp/backup-exclude.txt --delete-excluded /home root@192.168.122.1:/nfs/rearRSYNC//localhost/backup
+++ cat /tmp/rear.2o92TKFmGHCniuX/tmp/backup-include.txt
++ rsync --sparse --archive --hard-links --numeric-ids --stats '--rsync-path=rsync --fake-super' --relative --one-file-system --delete --exclude-from=/tmp/rear.2o92TKFmGHCniuX/tmp/backup-exclude.txt --delete-excluded /home root@192.168.122.1:/nfs/rearRSYNC//localhost/backup
++ echo 0

and on the terminal I get (excerpts)

# usr/sbin/rear -D mkbackup
...
Creating rsync archive on '192.168.122.1:/nfs/rearRSYNC/'
Running archive operation
OK
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 13,930 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 471
Total bytes received: 26

sent 471 bytes  received 26 bytes  331.33 bytes/sec
total size is 13,930  speedup is 28.03 in 6 seconds.
Exiting rear mkbackup (PID 17699) and its descendant processes ...

I did that "rear -D mkbackup" basically directly after a previous one
so basicaly no few files got actually 'rsynced'.

@bwelterl
I guess the root cause in your particular case is something rather specific
in your particular environment (perhaps something with bash in RHEL8.2?)
so I would recommend to report this issue to Red Hat so that they
could have a look what the actual root cause is.

I don't like to experiment with this or that changes
of the RSYNC code in ReaR that I never touched before
unless we better understand what the actual root cause is
so that we could fix things properly.

bwelterl commented at 2021-03-18 12:55:¶

Hello Johannes,

Thanks for the test. In fact, with --rsync-path="""rsync --fake-super"""
there is not error, but you don't have the expected result that is
to have ownerships and permissions added in xattr of remote files.
You are using root as rsync user, please try with another user.

With --rsync-path="""rsync --fake-super""" the command become

'--rsync-path=rsync --fake-super'

it should be

--rsync-path="rsync --fake-super"

to have the expected result.

Thank you !

Benoit

jsmeix commented at 2021-03-18 12:56:¶

@pcahyna @rmetrich
could you please have a look here (as time permits)?

It seems something behaves special (and fails) for @bwelterl with RHEL8.2
which I cannot reproduce - neither with SLES15 directly with ReaR
nor with openSUSE Leap 15.2 on the command line.

jsmeix commented at 2021-03-18 12:57:¶

@bwelterl
thank you for your prompt https://github.com/rear/rear/pull/2577#issuecomment-801903899

I didn't see that while I wrote my https://github.com/rear/rear/pull/2577#issuecomment-801904488

So I can reproduce the issue on my system which is good!

jsmeix commented at 2021-03-18 14:26:¶

At least for today I give up lost in quotation hell.

My failed cases:

(1)

BACKUP_RSYNC_OPTIONS+=( --rsync-path="""rsync --fake-super""" )

results the rsync command option (with single quotes as shown here):

rsync ... '--rsync-path=rsync --fake-super' ...

which doesn't result ownerships and permissions added in xattr of remote files

(2)

BACKUP_RSYNC_OPTIONS+=( '--rsync-path="rsync --fake-super"' )

results the rsync command option (with single quotes and double quotes as shown here):

rsync ... '--rsync-path="rsync --fake-super"' ...

which results an error like:

bash: rsync --fake-super: command not found
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [sender=3.1.3]

(3)

BACKUP_RSYNC_OPTIONS+=( --rsync-path=\"rsync --fake-super\" )

results the rsync command option (with single quotes and double quotes as shown here):

rsync ... '--rsync-path="rsync' '--fake-super"' ...

which results an error like:

rsync: --fake-super": unknown option
rsync error: syntax or usage error (code 1) at main.c(1593) [client=3.1.3]

(4)

BACKUP_RSYNC_OPTIONS+=( --rsync-path='rsync --fake-super' )

results the rsync command option (with single quotes as shown here):

rsync ... '--rsync-path=rsync --fake-super' ...

i.e. same as case (1) above

(5)

BACKUP_RSYNC_OPTIONS+=( --rsync-path=\'rsync --fake-super\' )

results the rsync command option (with single quotes as shown here):

rsync ... '--rsync-path='\''rsync' '--fake-super'\''' ...

which results an error as in case (3) above like:

rsync: --fake-super': unknown option
rsync error: syntax or usage error (code 1) at main.c(1593) [client=3.1.3]

jsmeix commented at 2021-03-18 14:34:¶

Tomorrow I will try (basically a blind desperate attempt) something like

BACKUP_RSYNC_OPTIONS+=( $( echo '--rsync-path="rsync --fake-super"' ) )

in the hope that then bash might somehow do its quoting automatically right...

bwelterl commented at 2021-03-18 16:26:¶

Thanks @jsmeix

But no luck: with

BACKUP_RSYNC_OPTIONS+=( $( echo '--rsync-path="rsync --fake-super"' ) )

Same error:

++ rsync --sparse --archive --hard-links --numeric-ids --stats '--rsync-path="rsync' '--fake-super"' --relative --one-file-system --delete --exclude-from=/tmp/rear.Yb9Ly9BUsdSOKsw/tmp/backup-exclude.txt --delete-excluded / /boot/efi /boot user1@192.168.122.220:/home/user1/rear//localhost/backup
rsync: --fake-super": unknown option
rsync error: syntax or usage error (code 1) at main.c(1578) [client=3.1.3]

I think the easiest way to enable this feature is to use:

-M, --remote-option=OPTION  send OPTION to the remote side only

as it's included in recent rsync version

Thanks

jsmeix commented at 2021-03-19 13:54:¶

@bwelterl
thank you for testing it.

Perhaps I have to give up and use the alternative option setting
via something like

BACKUP_RSYNC_OPTIONS+=( --rsync-path=rsync -M--fake-super )

But I like to understand and solve the generic issue that it seems
somehow it is not possible to specify an option of the form

--long-option="some words as value"

via a variable or via an array (element) to a command.

Next Monday I will experiment with the various ways
how an array can get evaluated in bash i.e. things like

COMMAND ${ARRAY_OF_OPTIONS[*]}

COMMAND ${ARRAY_OF_OPTIONS[@]}

COMMAND "${ARRAY_OF_OPTIONS[@]}"

Perhaps the root cause is not how the array is created
but how the array is evaluated at the COMMAND call
which is in this case here

$BACKUP_PROG "${BACKUP_RSYNC_OPTIONS[@]}"

in usr/share/rear/backup/RSYNC/default/500_make_rsync_backup.sh
https://github.com/rear/rear/blob/master/usr/share/rear/backup/RSYNC/default/500_make_rsync_backup.sh#L27

jsmeix commented at 2021-03-22 14:05:¶

It seems I can get things working with

eval $BACKUP_PROG "${BACKUP_RSYNC_OPTIONS[@]}" ...

in usr/share/rear/backup/RSYNC/default/500_make_rsync_backup.sh
but currently I cannot foresee if that additional eval might have unwanted side-effects.

In general I don't like eval because it makes things behave more complicated,
cf. https://mywiki.wooledge.org/BashFAQ/048
and RFC 1925 items 6 and 6a (excerpts)

(6) It is easier to move a problem around ... than it is to solve it.
(6a) It is always possible to add another level of indirection.

So I would prefer plain and simple

BACKUP_RSYNC_OPTIONS+=( --rsync-path=rsync -M--fake-super )

provided -M is supported by sufficiently recent rsync i.e. by those rsync versions
that are included in the Linux distributions that are supported by current ReaR, cf.
https://github.com/rear/rear/blob/master/doc/rear-release-notes.txt#L2513

ReaR-2.6 is supported on the following Linux based operating systems:

  o Fedora 29, 30, 31, and 32
  o RHEL 6, 7, and 8
  o CentOS 6, 7, and 8
  o Scientific Linux 6 and 7
  o SLES 12 and 15
  o openSUSE Leap 15.x
  o Debian 8, and 9
  o Ubuntu 16, 17, and 18

jsmeix commented at 2021-03-23 09:55:¶

Regarding what rsync versions that are included in the Linux distributions
hat are supported by current ReaR support -M / --remote-option:

https://serverfault.com/questions/119846/rsync-and-fake-super-how-to-preserve-attributes
reads (excerpts)

Note that at least as of rsync 3.1.2 (but probably already earlier versions)
there is a --remote-option parameter which might generally be preferable.
There is also the shortcut "-M" for it. So instead of
--rsync-path="rsync --fake-super"
one could also have written
-M--fake-super

I can confirm that --remote-option aka -M is also in rsync 3.1.1

--remote-option/-M is not available in rsync 3.0.9 on CentOS 7

But CentOS 6 and 7 are supported by current ReaR.

--remote-option/-M is not available in rsync-3.0.4 on SLES11
and SLES11 is no longer supported since ReaR 2.6
but SLES11 was supported up to ReaR 2.5 cf.
https://github.com/rear/rear.github.com/blob/master/documentation/release-notes-2-5.md

--remote-option/-M is available in rsync 3.1.3 on SLES12 and SLES15

jsmeix commented at 2021-03-23 09:59:¶

@gdha @rmetrich @pcahyna
I am at my wits' end here because I don't know how I could solve it properly.
Could you please (as time permits) have a look here and help to find a solution together?

gdha commented at 2021-04-06 16:00:¶

@jsmeix RHEL 7.6 and higher has rsync version 3.1.2. RHEL 6.x need to check once I find a system with this version (becoming rare as not supported anymore by RH).

bwelterl commented at 2021-04-06 17:19:¶

In RHEL 6, it's rsync-3.0.6-12.el6.x86_64 and the option -M remote is not available.
But I'm not sure we expect a fix in RHEL6 anyway...

gdha commented at 2021-04-16 10:08:¶

@bwelterl @jsmeix Thanks - RHEL6/CentOS6 support will be dropped for ReaR v2.7 anyhow so we are good to proceed with -M--fake-super IMHO

jsmeix commented at 2021-04-16 10:26:¶

@gdha @bwelterl
I fully agree that proceeding with -M--fake-super and
dropping official support for old distributions in upcoming ReaR 2.7
is much better compared to not proceed and keep things as broken as is.

In particular users of old distributions should already have set up
their disaster recovery procedures as they need it with current ReaR
so users of old distributions should not need to upgrade to ReaR 2.7,
cf. "Version upgrades with Relax-and-Recover" in
https://en.opensuse.org/SDB:Disaster_Recovery

Even if users of old distributions need to upgrade to ReaR 2.7
only those users who use BACKUP=RSYNC could be affected
and then they could manually adapt the ReaR scripts as needed
to make things work for their specific cases,
cf. "Disaster recovery with Relax-and-Recover (ReaR)" in
https://en.opensuse.org/SDB:Disaster_Recovery

pcahyna commented at 2021-05-12 16:23:¶

@jsmeix @bwelterl sorry for the delay. Looking at the issue now.

pcahyna commented at 2021-05-13 18:22:¶

The issue can't be reproduced on 2.6, as I explained in https://github.com/rear/rear/pull/2577#discussion_r631997526 . Please close.
(Not that the rsync code is entirely correct: it does not restore extended attributes: so before restore you have

$ getfattr -d -m- /usr/bin/ping
getfattr: Removing leading '/' from absolute path names
# file: usr/bin/ping
security.capability=0sAAAAAgAwAAAAAAAAAAAAAAAAAAA=
security.selinux="system_u:object_r:ping_exec_t:s0"

while after restore just

getfattr -d -m- /usr/bin/ping
getfattr: Removing leading '/' from absolute path names
# file: usr/bin/ping
security.selinux="system_u:object_r:ping_exec_t:s0"

so the system will be a bit broken in different ways. But that's a separate issue. The problem reported here would be also easier to detect if rsync errors during restore were reported, but that's a separate issue as well, #2612 .)

Since the problem discussed here was how to recognize properly quoted arguments from improperly quoted arguments, here is a trick I have been using. In ps you will not see the difference between rsync --rsync-path=rsync --fake-super and rsync "--rsync-path=rsync --fake-super" (the quotes are not part of the arguments themselves and they should not be, the shell removes them), but you can determine it using procfs, where the individual ARGV entries are separated by zeros. So:
rear 2.6 (fixed) has this

# tr '\0' '\n' < /proc/19605/cmdline
rsync
--sparse
--archive
--hard-links
--numeric-ids
--stats
--rsync-path=rsync --fake-super
--relative
--one-file-system
--delete
--exclude-from=/tmp/rear.fCuNUO7ZP85C4Tg/tmp/backup-exclude.txt
--delete-excluded
/
...

while 2.4 (buggy) has this:

# tr '\0' '\n' < /proc/17970/cmdline
rsync
--sparse
--archive
--hard-links
--numeric-ids
--stats
--rsync-path=rsync
--fake-super
--relative
--one-file-system
--delete
--exclude-from=/tmp/rear.b2Dz5Dj4S9qNHf7/tmp/backup-exclude.txt
--delete-excluded
/
...

pcahyna commented at 2021-05-14 19:32:¶

At least for today I give up lost in quotation hell.

My failed cases:

(1)
BACKUP_RSYNC_OPTIONS+=( --rsync-path="""rsync --fake-super""" )
results the rsync command option (with single quotes as shown here):
rsync ... '--rsync-path=rsync --fake-super' ...
which doesn't result ownerships and permissions added in xattr of remote files

It works for me. How did you check? Does your target filesystem support user extended attributes?

pcahyna commented at 2021-05-15 11:25:¶

The only thing unclear is why was @jsmeix able to reproduce the problem with (presumably) the current development version (was it really the current version, or something older that is shipped with SLES15?), which should have the problem fixed (and I was not able to reproduce it anymore).

jsmeix commented at 2021-05-17 12:32:¶

@pcahyna
thank you so much for your deep analysis
and your comprehensive explanations
that help so much to understand things!

My "doesn't result ownerships and permissions added in xattr of remote files"
is only from what @bwelterl told about '--rsync-path=rsync --fake-super'
(as fas as I understood it - perhaps I misunderstood it).
I did not verify if that was actually true or false.

Unfortunately it is too long ago so I don't remember what I exactly did to "reproduce" it.
Usually I use GitHub master code because that is the only place where we fix things.
I think I vaguely remember that with "I can reproduce" I had only meant how the
'rsync' command is shown by set -x in the 'rear -D mkrescue' debug log file.
So I guess my "I can reproduce" was false alarm because I failed to
recognize properly quoted arguments from improperly quoted arguments
that you described so well in your
https://github.com/rear/rear/pull/2577#issuecomment-840743973

jsmeix commented at 2021-05-17 13:23:¶

Now ARRAY+=( appended elements ) is documented as the only way in
https://github.com/rear/rear/wiki/Coding-Style

Append to array only via ARRAY+=( appended elements )
because ARRAY=( "${ARRAY[@]}" appended elements ) is not fail-safe

pcahyna commented at 2021-05-17 13:59:¶

Now ARRAY+=( appended elements ) is documented as the only way in
https://github.com/rear/rear/wiki/Coding-Style
Append to array only via ARRAY+=( appended elements )
because ARRAY=( "${ARRAY[@]}" appended elements ) is not fail-safe

That's a misunderstanding. I believe that ARRAY=( "${ARRAY[@]}" appended elements ) should be fail-safe. But the code that you have replaced with += lacked the quotes, that was the problem.
+= is fail-safe in the sense of "trottelfest" (fool-safe) because it can be hard to remember to use quotes consistently, while with += the intention is obvious and one doesn't have to worry about such nasty details.

pcahyna commented at 2021-05-17 14:05:¶

My "doesn't result ownerships and permissions added in xattr of remote files"
is only from what @bwelterl told about '--rsync-path=rsync --fake-super'
(as fas as I understood it - perhaps I misunderstood it).
I did not verify if that was actually true or false.

For the record, if you look at the backup created using --fake-super on the backup server, it looks quite weird and you will quickly recognize it. Even symlinks become normal files and the information that they are intended to be symlinks is recorded in the extended attribute.

jsmeix commented at 2021-05-17 14:10:¶

On https://github.com/rear/rear/wiki/Coding-Style
I reverted that ARRAY+=( appended elements ) is the only way
so now it is again only the recommended way.

[Export of Github issue for rear/rear.]

#2577 PR closed: Fix rsync --fake-super option on remote server¶

bwelterl opened issue at 2021-02-25 17:39:¶

Relax-and-Recover (ReaR) Pull Request Template¶

Pull Request Details:¶

jsmeix commented at 2021-03-02 10:46:¶

jsmeix commented at 2021-03-03 12:59:¶

gdha commented at 2021-03-03 13:09:¶

jsmeix commented at 2021-03-18 12:45:¶

bwelterl commented at 2021-03-18 12:55:¶

jsmeix commented at 2021-03-18 12:56:¶

jsmeix commented at 2021-03-18 12:57:¶

jsmeix commented at 2021-03-18 14:26:¶

jsmeix commented at 2021-03-18 14:34:¶

bwelterl commented at 2021-03-18 16:26:¶

jsmeix commented at 2021-03-19 13:54:¶

jsmeix commented at 2021-03-22 14:05:¶

jsmeix commented at 2021-03-23 09:55:¶

jsmeix commented at 2021-03-23 09:59:¶

gdha commented at 2021-04-06 16:00:¶

bwelterl commented at 2021-04-06 17:19:¶

gdha commented at 2021-04-16 10:08:¶

jsmeix commented at 2021-04-16 10:26:¶

pcahyna commented at 2021-05-12 16:23:¶

pcahyna commented at 2021-05-13 18:22:¶

pcahyna commented at 2021-05-14 19:32:¶

pcahyna commented at 2021-05-15 11:25:¶

jsmeix commented at 2021-05-17 12:32:¶

jsmeix commented at 2021-05-17 13:23:¶

pcahyna commented at 2021-05-17 13:59:¶

pcahyna commented at 2021-05-17 14:05:¶

jsmeix commented at 2021-05-17 14:10:¶

#2577 PR `closed`: Fix rsync --fake-super option on remote server¶