#2867 Issue `open`: ReaR connects to rsync server over ssh even if configured to use the rsync protocol without ssh¶

Labels: bug

oldunclez opened issue at 2022-09-22 00:50:¶

I am using the most fresh "Relax-and-Recover 2.7 / Git" to backup my "src vm" , and then recover it to "dst vm".
For non-interactive , when backup "src vm" , I set " --password-file" for rsync.
/etc/rear/local.conf is :

OUTPUT=ISO
BACKUP=RSYNC
BACKUP_URL=rsync://rsync@192.168.1.27::rsync
BACKUP_RSYNC_OPTIONS=( -az  --password-file=/etc/rsync.pass  )

Note 1:¶

No matter what rsync transfer method is using :

BACKUP_URL=rsync://[USER@]HOST[:PORT]/PATH    # using ssh
BACKUP_URL=rsync://[USER@]HOST[:PORT]::/PATH  # using rsync

For displaying the progress of rsync data tranfer , rear will ssh to rsync server to count how much data has been transfered.
( function "check_remote_du" , line 94 in file /usr/share/rear/backup/RSYNC/default/500_make_rsync_backup.sh ).

So if you want to make it works , you have to make sure "non-interactive SSH login for rsync@192.168.1.27" is accessable.
Otherwise, if you do not need the progress of the rsync transfer , maybe you could modify 500_make_rsync_backup.sh to the following :

        while sleep $PROGRESS_WAIT_SECONDS ; kill -0 $BackupPID 2>/dev/null ; do
            ProgressInfo "Waiting  "$(rsync_remote_full $BACKUP_URL)"/backup  to be  finished"
        done

Note 2:¶

Rear will pack "/etc/rear/local.conf" of the "src vm" into the the rescue iso.
When excuting "rear recover" on "dst vm" , rear will copy the backup files from rsync server to "dst vm" via rsync with "BACKUP_RSYNC_OPTIONS=( -az --password-file=/etc/rsync.pass )" .
But the "/etc/rsync.pass " will not be packed into rescue iso , so it does not exist in "dst vm",
You have to create it manually or remove it from BACKUP_RSYNC_OPTIONS .
Otherwise , rear recover will fail because of rsync fail with the non-existent "/etc/rsync.pass "

jsmeix commented at 2022-09-22 09:02:¶

@oldunclez
from ReaR 2.6 to ReaR 2.7 we changed something with rsync,
see what is described about rsync in our release notes
https://github.com/rear/rear/blob/master/doc/rear-release-notes.txt
therein in particular (excerpt)

Rsync OUTPUT_URLs are now properly supported with BACKUP=RSYNC.
Previously the output went to the location specified
by BACKUP_URL and OUTPUT_URL was ignored.
One exception was OUTPUT=PXE, where the output
was uploaded to OUTPUT_URL in addition to BACKUP_URL,
but RSYNC_PREFIX was not respected and the
interpretation of the URL was different:
A URL of the form
rsync://[USER@]HOST[:PORT]/PATH
was interpreted as using the rsync protocol,
while in all other cases such URL would be
interpreted as using rsync over ssh.
This special handling is now removed:
An rsync OUTPUT_URL with OUTPUT=PXE
now creates the RSYNC_PREFIX directory
at the destination and the URL is interpreted
as in all other cases.
...
Refactor rsync URL support, fixes rsync OUTPUT_URL:
The code to parse rsync:// URLs was BACKUP_URL specific.
If one specified BACKUP=RSYNC and an OUTPUT_URL
different from BACKUP_URL, the OUTPUT_URL was ignored
and the output files went to BACKUP_URL.
Fix by introducing generic functions
for rsync URL parsing and use them
for both BACKUP_URL and OUTPUT_URL,
as appropriate.
Replace all uses of global RSYNC_* variables
derived from BACKUP_URL by those functions.
There also was inconsistent special handling
for OUTPUT=PXE which is now removed:
An rsync OUTPUT_URL with OUTPUT=PXE now creates
the RSYNC_PREFIX directory at the destination
and the URL is interpreted as in all other cases.
See https://github.com/rear/rear/pull/2831
and https://github.com/rear/rear/issues/2781

For details and background information
about the reasoning behind those changes
you may have a look the issue
https://github.com/rear/rear/issues/2781
and the pull request
https://github.com/rear/rear/pull/2831

FYI:
I am not a rsync user (and not at all a PXE user)
so I can not much help with rsync issues
(and I can not at all help with PXE issues)
but others here may help (as their time permits).

As a side note:
You may also have a look at
https://github.com/rear/rear/pull/2577#issuecomment-789696867
(which shows why I can not much help with rsync issues)
and
https://github.com/rear/rear/pull/2577#issuecomment-789702926

pcahyna commented at 2022-09-22 09:07:¶

Re Note 1 : that's real and is one of the many oddities I noticed (but not fixed) when working on PR #2831 , see https://github.com/rear/rear/pull/2831#issuecomment-1170105155 (the 4th bullet point).

Re Note 2 : you need to add the file to the rescue system using COPY_AS_IS+=( /etc/rsync.pass ). This is sort of expected; I am afraid it is not feasible to make it so automated that it would detect what files you refer to and add them by itself.

jsmeix commented at 2022-09-22 09:10:¶

@pcahyna
thank you for your prompt help here.
It is much appreciated!

pcahyna commented at 2022-09-22 13:19:¶

@oldunclez generally I have found that the rsync-over-ssh seems to be better supported and tested than rsyncd. See the third bullet point in https://github.com/rear/rear/pull/2831#issuecomment-1170105155 that might affect you. Please check whether the solution for Note 2 works for you and if yes I will change the topic of the issue to cover only Note 1.

oldunclez commented at 2022-09-26 01:26:¶

For note 1 @jsmeix ,thank you for your reply .

For note2
@pcahyna , I agree that people should set COPY_AS_IS+=( /etc/rsync.pass ) manually .
I prefer "rsyncd" instead of "rsync-over-ssh" , because I don't want to place a SSH private key in any hosts.

pcahyna commented at 2022-09-27 09:45:¶

@oldunclez I then consider note 2 solved and focus only on note 1. I think the only solution is to disable the reporting of disk space used and disk full entirely in the case of the rsync protocol. I think that it would not be reported correctly anyway, because the "path" in the case of the rsync protocol is actually a module name and one can't now which actual path on the server the module name refers to (it depends on the server configuration), so there is no way to construct the correct df or du command.

fadamo commented at 2022-09-28 12:25:¶

About the ssh connection we found a w/a adding these lines in local.conf

check_remote_df() { echo 0 ; }
check_remote_du() { echo 0 ; }
readonly -f check_remote_df
readonly -f check_remote_du

pcahyna commented at 2022-09-29 09:35:¶

@fadamo wow, I have not thought about a possibility of such a workaround, thanks for sharing it!

oldunclez commented at 2022-09-30 07:08:¶

Thank you for all your reply.
It is a good news that people can customize his code in local.conf without modifying the "offical" shell script files.

jsmeix commented at 2022-09-30 12:19:¶

@oldunclez

regarding etc/rear/local.conf and other config files
see the comment in
https://github.com/rear/rear/blob/master/etc/rear/local.conf
which reads (excerpts)

# ReaR reads the configuration files via the bash builtin command 'source'
...
# Because 'source' executes the content as bash scripts you can run commands
# within your configuration files ...
...
# ... gets always executed when 'rear' is run
# so ensure nothing can go wrong if you run commands in configuration files.

Regarding "offical" shell script files:

There is no such thing as "offical" scripts in ReaR
that should be considered as "sacrosanct" by users.

Relax-and-Recover is intentionaly written entirely
in the native language for system administration
as shell (bash) scripts because experienced users
and system admins can and should adapt or extend
the ReaR scripts if needed to make things work
for their specific cases.
Additionally - if the worst comes to the worst - even
temporary quick and dirty workarounds are relatively
easily possible - provided one knows ReaR and
one's own system well so that one is prepared
for appropriate manual ad hoc intervention.

Things could be different when you use a ReaR software
package from some vendor like a Linux distributor.
Then in case of issues you should at least tell their
support people if and what you have manually modified.
In particular regarding SUSE see the section
"SUSE support for Relax-and-Recover" in
https://en.opensuse.org/SDB:Disaster_Recovery