#926 Issue `closed`: Local data backup with BACKUP=NETFS fails¶

Labels: enhancement, bug, cleanup, fixed / solved / done

tcerna opened issue at 2016-07-18 08:55:¶

Relax-and-Recover (rear) Issue Template¶

Please fill in the following items before submitting a new issue:

rear version (/usr/sbin/rear -V):
rear-1.17.2-1.el7.noarch
OS version (cat /etc/rear/os.conf or lsb_release -a):
RHEL-7
rear configuration files (cat /etc/rear/site.conf or cat /etc/rear/local.conf):

# cat /etc/rear/local.conf
BACKUP=NETFS
BACKUP_URL=file:///tmp/rearbackup

Brief description of the issue

When running rear -v mkbackuponly with BACKUP=NETFS and specified BACKUP_URL in local.conf it fails with ERROR: Making backup on / is forbidden.
Actual results:
ERROR: Making backup on / is forbidden. Use an external device!
Terminated
Expected results:
Data backup is created in specified directory.

jsmeix commented at 2016-07-18 10:39:¶

Works for me
(with currently newest rear GitHub master code):

f121:~/rear # grep -v '^#' etc/rear/local.conf 
OUTPUT=ISO
BACKUP=NETFS
BACKUP_URL=file:///tmp/rearbackup
...
f121:~/rear # usr/sbin/rear -d -D mkbackup
Relax-and-Recover 1.18 / Git
Using log file: /root/rear/var/log/rear/rear-f121.log
Creating disk layout
Creating root filesystem layout
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Creating initramfs
Making ISO image
Wrote ISO image: /root/rear/var/lib/rear/output/rear-f121.iso (28M)
Copying resulting files to file location
Encrypting disabled
Creating tar archive '/tmp/rearbackup/f121/backup.tar.gz'
Archived 1192 MiB [avg 10009 KiB/sec]OK
Archived 1192 MiB in 123 seconds [avg 9927 KiB/sec]
f121:~/rear # find /tmp/rearbackup
/tmp/rearbackup
/tmp/rearbackup/f121
/tmp/rearbackup/f121/rear-f121.iso
/tmp/rearbackup/f121/VERSION
/tmp/rearbackup/f121/README
/tmp/rearbackup/f121/rear.log
/tmp/rearbackup/f121/backup.tar.gz
/tmp/rearbackup/f121/backup.log

@tcerna
the "Making backup on / is forbidden" error message comes from
usr/share/rear/prep/NETFS/default/40_automatic_exclude_recreate.sh

You need to run it in debug mode:

rear -d -D mkbackup

and then inspect the rear log file what leads to that error
in your particular case.

Inspect your rear log file what happens after the line

+ source /root/rear/usr/share/rear/prep/NETFS/default

jsmeix commented at 2016-07-18 11:20:¶

Something smells here...

Sometimes it also fails for me, sometimes it works for me.

I tested different BACKUP_URL=file://
but that does not matter whether or not it work.

Below it worked with BACKUP_URL=file:///rearbackup

When it works for me I have in the rear log file:

+ source /root/rear/usr/share/rear/prep/NETFS/default/40_automatic_exclude_recreate.sh
+++ url_scheme file:///rearbackup
+++ local url=file:///rearbackup
+++ local scheme=file
+++ echo file
+++ grep -q :
+++ echo file
++ local scheme=file
+++ url_path file:///rearbackup
+++ local url=file:///rearbackup
+++ local url_without_scheme=/rearbackup
+++ echo /rearbackup
++ local path=/rearbackup
++ case $scheme in
+++ df -P /rearbackup
+++ tail -1
+++ awk '{print $6}'
++ _mntpt=
++ [[ '' = \/ ]]

I.e. when it works for me the command

_mntpt=$(df -P ${path} 2>/dev/null | tail -1 | awk '{print $6}')

results an empty _mntpt variable
regardless that on command line I get

f121:~ # df -P /rearbackup                                         
Filesystem     1024-blocks    Used Available Capacity Mounted on
/dev/sda2         19437568 5222120  13775032      28% /
f121:~ # df -P /rearbackup 2>/dev/null | tail -1 | awk '{print $6}'
/

Something is wrong here => somewhere there is a bug.

jsmeix commented at 2016-07-18 11:23:¶

I love git blame!
;-)

git blame usr/share/rear/prep/NETFS/default/40_automatic_exclude_recreate.sh

shows that the "Making backup on / is forbidden" code
was added by @gdha via
git commit 4bfd2e3a235ab6a86ddd403a7cfe542c40d314c3

jsmeix commented at 2016-07-18 11:30:¶

@gdha
can you provide background information about the
reason why "Making backup on / is forbidden"?

Is perhaps meant that for a locally stored backup
the backup directory must be on a different filesystem
than what the '/' directory is to avoid a loop where
the backup directory content gets also backed up?

Do you perhaps even have an idea why sometimes

_mntpt=$(df -P ${path} 2>/dev/null | tail -1 | awk '{print $6}')

results an empty _mntpt variable?

As a side note:
I also do not understand what you meant with
the comment in the line

     (file) # if user added path manually then there is no need to do it again

in prep/NETFS/default/40_automatic_exclude_recreate.sh

tcerna commented at 2016-07-20 14:39:¶

I tried it again and I have to results: good and bad:

BAD ONE:

[root@rearclient ~]# cat /etc/rear/local.conf
BACKUP=NETFS
BACKUP_URL="file:///tmp/rear"

[root@rearclient ~]# rear -v -d -D mkbackup
Relax-and-Recover 1.17.2 / Git
Using log file: /var/log/rear/rear-rearclient.log
ERROR: Making backup on / is forbidden. Use an external device!
Aborting due to an error, check /var/log/rear/rear-rearclient.log for details
You should also rm -Rf /tmp/rear.MqH2f4NMbLXbssk
Terminated

[root@rearclient ~]# echo $?
143

I didn't find any strange in log file, but it does not mean that there is any strangeness.

GOOD ONE:

[root@rearclient ~]# cat /etc/rear/local.conf
BACKUP_URL="file:///tmp/rear"

[root@rearclient ~]# rear -v -d -D mkbackup
Relax-and-Recover 1.17.2 / Git
Using log file: /var/log/rear/rear-rearclient.log
Creating disk layout
Creating root filesystem layout
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Creating initramfs
Making ISO image
Wrote ISO image: /var/lib/rear/output/rear-rearclient.iso (128M)
Copying resulting files to file location
You should also rm -Rf /tmp/rear.QOY6wAAs4K2LPQn

[root@rearclient ~]# echo $?
0

scr4bble commented at 2016-07-20 16:01:¶

rear version (/usr/sbin/rear -V):
Relax-and-Recover 1.17.2 / Git
OS version (cat /etc/rear/os.conf or lsb_release -a):
RedHatEnterpriseServer 7
rear configuration files (cat /etc/rear/site.conf or cat /etc/rear/local.conf):

OUTPUT=ISO
OUTPUT_URL="file:///root/"
BACKUP=NETFS
BACKUP_URL="file:///root/backup/"
GRUB_RESCUE=1
GRUB_RESCUE_PASSWORD="...."

Brief description of the issue

When I run rear mkbackup -v for the first time with above settings it creates backup with image successfully.
If I wants to make backup again (with same settings, but without deleting previously created files), it fails with this error (bellow) - no matter what backup... it fails with all "subcommands": mkbackup, mkbackuponly, mkrescue!!!

Using log file: /var/log/rear/rear-rearserver.log
ERROR: Making backup on / is forbidden. Use an external device!
Aborting due to an error, check /var/log/rear/rear-rearserver.log for details

Questions & suggestions:

Is there a problem with overwriting files ? (if yes, it could print proper message)
Is storing backup on the same filesystem forbidden ?
I think it shouldn't block to make a rescue image even if there is any problem with BACKUP* variables/settings.

jsmeix commented at 2016-07-21 07:15:¶

I think by default "rear mkbackup" does not care about old files
(i.e. it just overwrites them).
Regarding avoiding overwriting files see
NETFS_KEEP_OLD_BACKUP_COPY
cf. "Relax-and-Recover (rear) versus backup and restore"
at https://en.opensuse.org/SDB:Disaster_Recovery

I think "rear mkbackup" should error out whenever there
is an error, cf. https://github.com/rear/rear/wiki/Coding-Style

I think it must error out whenever a specified thing
cannot be done, cf. https://github.com/rear/rear/issues/913#issuecomment-232661112

I assume the root cause here is that rear does not care
sufficiently about possible errors at the actually right place
and blindly proceeds until it errors out somewhere later
at an unrelated place with a weird error message which leads
to false assumptions about the root cause of the issue and then
to false fixes that cure only a symptom but not the root cause.

I will do a deeper analysis what actually goes on here.

I would very much appreciate any help from others with that
analysis so that finally we can fix it properly.

"Given enough eyeballs, all bugs are shallow"
cf. https://en.wikipedia.org/wiki/Linus%27s_Law

jsmeix commented at 2016-07-21 10:08:¶

One must avoid a BACKUP_URL=file:///dir
where /dir is included in the backup.

For me "rear mkbackup" with BACKUP_URL=file:///rearbackup
(where /rearbackup did not already exist before) worked
but results the backup in the backup:

# tar -tzvf /rearbackup/f121/backup.tar.gz | grep rearbackup
drwx------ root/root            0 2016-07-21 11:35 rearbackup/
drwxr-x--- root/root            0 2016-07-21 11:35 rearbackup/f121/
-rw------- root/root            0 2016-07-21 11:35 rearbackup/f121/.lockfile
-rw------- root/root    120426496 2016-07-21 11:35 rearbackup/f121/rear-f121.iso
-rw------- root/root          262 2016-07-21 11:35 rearbackup/f121/VERSION
-rw------- root/root          202 2016-07-21 11:35 rearbackup/f121/README
-rw------- root/root      6747236 2016-07-21 11:35 rearbackup/f121/rear.log
-rw------- root/root   1385283584 2016-07-21 11:37 rearbackup/f121/backup.tar.gz
# ls -l /rearbackup/f121/
total 2823976
-rw------- 1 root root   11080034 Jul 21 11:38 backup.log
-rw------- 1 root root 2753482505 Jul 21 11:38 backup.tar.gz
-rw------- 1 root root        202 Jul 21 11:35 README
-rw------- 1 root root  120426496 Jul 21 11:35 rear-f121.iso
-rw------- 1 root root    6747236 Jul 21 11:35 rear.log
-rw------- 1 root root        262 Jul 21 11:35 VERSION

Restoring that backup results a /rearbackup/$HOSTNAME
directory that contains invalid files (in particular an invalid
backup.tar.gz).

I think "rear recover" that restores /rearbackup/$HOSTNAME
directory that contains invalid files only looks not nice
but it does not lead to an actually broken system
because the other files should be o.k.

What matters more is that having the backup in the backup
increases needlessly the backup size to basically double size
at least in my case where /rearbackup/$HOSTNAME/backup.tar.gz
was added last to the backup as reported in
/rearbackup/$HOSTNAME/backup.log

# tail -n20 /rearbackup/f121/backup.log
block 5531760: /root/rear/var/lib/rear/layout/lvm/
block 5531763: /root/rear/var/lib/rear/sysreqs/
block 5531766: /root/rear/var/lib/rear/sysreqs/Minimal_System_Requirements.txt
block 5531771: /root/.Xauthority-n
block 5531774: /root/.viminfo
block 5531793: /selinux/
block 5531796: /rearbackup/
block 5531799: /rearbackup/f121/
block 5531802: /rearbackup/f121/.lockfile
block 5531805: /rearbackup/f121/rear-f121.iso
block 5767016: /rearbackup/f121/VERSION
block 5767020: /rearbackup/f121/README
block 5767024: /rearbackup/f121/rear.log
block 5780206: /rearbackup/f121/backup.tar.gz
tar: /rearbackup/f121/backup.tar.gz: file changed as we read it
block 8485826: /root/rear/var/log/rear/rear-f121.log
Total bytes written: 4344750080 (4.1GiB, 24MiB/s)
5377895+1 records in
5377895+1 records out
2753482505 bytes (2.8 GB) copied, 174.252 s, 15.8 MB/s

But again a needlessly increased backup size
is not a fatal error.

I wonder if rear should try to be smarter than the user
and implement sophisticated test to avoid such kind
of user faults.

I think it belongs to the user to think about in advance
how to set up rear correctly.

From my current point of view I would close that issue
as an example of improper usage of rear.

The only bug that is see is the misleading error message

ERROR: Making backup on / is forbidden. Use an external device!

when one does a second "rear mkbackup"
with BACKUP_URL=file:///rearbackup
where /rearbackup did already exists.

jsmeix commented at 2016-07-21 10:32:¶

With BACKUP_URL=file:///rearbackup
where /rearbackup is a mountpoint of a separated filesystem
(ex2 on /dev/sdb1in my case) the backup is o.k.:

# ls -l /rearbackup/f121/backup.tar.gz
-rw------- 1 root root 1266451979 Jul 21 12:20 /rearbackup/f121/backup.tar.gz
# tar -tzvf /rearbackup/f121/backup.tar.gz | grep rearbackup
drwxr-xr-x root/root            0 2016-07-21 12:18 rearbackup/
[no further output]
# mount | grep sdb
/dev/sdb1 on /rearbackup type ext2 (rw,relatime)

Now even a second "rear mkbackup" works as intended
regardless that /rearbackup already exists.
I have NETFS_KEEP_OLD_BACKUP_COPY=yes
and accordingly I have after the second "rear mkbackup"

# find /rearbackup
/rearbackup
/rearbackup/f121.old
/rearbackup/f121.old/VERSION
/rearbackup/f121.old/backup.tar.gz
/rearbackup/f121.old/rear-f121.iso
/rearbackup/f121.old/README
/rearbackup/f121.old/rear.log
/rearbackup/f121.old/backup.log
/rearbackup/lost+found
/rearbackup/f121
/rearbackup/f121/VERSION
/rearbackup/f121/backup.tar.gz
/rearbackup/f121/rear-f121.iso
/rearbackup/f121/README
/rearbackup/f121/rear.log
/rearbackup/f121/backup.log
# ls -l /rearbackup/*/backup.tar.gz
-rw------- 1 root root 1266910902 Jul 21 12:30 /rearbackup/f121/backup.tar.gz
-rw------- 1 root root 1266451979 Jul 21 12:20 /rearbackup/f121.old/backup.tar.gz

jsmeix commented at 2016-07-21 11:35:¶

And a third "rear mkbackup" now
without NETFS_KEEP_OLD_BACKUP_COPY
just overwrites the files in /rearbackup/f121/
in particular I have now

# ls -l /rearbackup/*/backup.tar.gz
-rw------- 1 root root 1266897447 Jul 21 13:23 /rearbackup/f121/backup.tar.gz
-rw------- 1 root root 1266451979 Jul 21 12:20 /rearbackup/f121.old/backup.tar.gz

Conclusion from my point of view:

With a proper set up everything works fine for me with
BACKUP_URL=file:///...

The only thing left is to find out how the "_mntpt" test
in prep/NETFS/default/40_automatic_exclude_recreate.sh
is meant:

@gdha
see https://github.com/rear/rear/issues/926#issuecomment-233304673
can you explain how the "_mntpt" test
in prep/NETFS/default/40_automatic_exclude_recreate.sh
is meant to work - i.e. what is the idea behind that test.

jsmeix commented at 2016-07-21 12:02:¶

Stupid me!

The "_mntpt" test
in prep/NETFS/default/40_automatic_exclude_recreate.sh
results an empty _mntpt when the directory in
BACKUP_URL=file:///directory
does not yet exist because

# df -P qqq 2>/dev/null
# df -P qqq
df: 'qqq': No such file or directory

and it correctly finds out that the director is mounted at '/'
when it already exists.

I will enhace the test to be more fail-safe...

jsmeix commented at 2016-07-21 12:57:¶

With
https://github.com/rear/rear/pull/936
I tried to make
prep/NETFS/default/40_automatic_exclude_recreate.sh
more fail safe.

@gdha
please have a look if you like it this way
or if I may have misunderstood something

jsmeix commented at 2016-07-21 13:11:¶

With https://github.com/rear/rear/pull/936 I get:

With BACKUP_URL=file:///rootfsrearbackup I get
when /rootfsrearbackup does not yet exist

ERROR: URL 'file:///rootfsrearbackup' would result the backup directory '/rootfsrearbackup' in the '/' filesystem which is forbidden.

and when /rootfsrearbackup does already exist

ERROR: URL 'file:///rootfsrearbackup' has the backup directory '/rootfsrearbackup' in the '/' filesystem which is forbidden.

With BACKUP_URL=file:///etc/rear/local.conf I get

ERROR: URL 'file:///etc/rear/local.conf' specifies '/etc/rear/local.conf' which is not a directory.

With BACKUP_URL=file:///rearbackup (as above)
where /rearbackup is a mountpoint of a separated filesystem
it still works for me:

# usr/sbin/rear -d -D mkbackup
Relax-and-Recover 1.18 / Git
Using log file: /root/rear/var/log/rear/rear-f121.log
Creating disk layout
Excluding component fs:/rearbackup
...

jsmeix commented at 2016-07-21 13:39:¶

But with BACKUP_URL=file:///rearbackup/subdir
where /rearbackup is a mountpoint of a separated filesystem
but /rearbackup/subdir does not yet exist
it did not work correctly:

ERROR: URL 'file:///rearbackup/subdir' would result the backup directory '/rearbackup/subdir' in the '/' filesystem which is forbidden.

I fixed handling when backup directory does not yet exist in
https://github.com/rear/rear/pull/936
so that now BACKUP_URL=file:///rearbackup/subdir
also works as intended

# usr/sbin/rear -d -D mkbackup
Relax-and-Recover 1.18 / Git
Using log file: /root/rear/var/log/rear/rear-f121.log
Creating disk layout
Excluding component fs:/rearbackup
...

and in rear-f121.log I get now

+ source /root/rear/usr/share/rear/prep/NETFS/default/40_automatic_exclude_recreate.sh
+++ url_scheme file:///rearbackup/subdir
+++ local url=file:///rearbackup/subdir
+++ local scheme=file
+++ grep -q :
+++ echo file
+++ echo file
++ local scheme=file
+++ url_path file:///rearbackup/subdir
+++ local url=file:///rearbackup/subdir
+++ local url_without_scheme=/rearbackup/subdir
+++ echo /rearbackup/subdir
++ local backup_directory=/rearbackup/subdir
++ local backup_directory_mountpoint=
++ case $scheme in
++ test -e /rearbackup/subdir
++ mkdir -v -p /rearbackup/subdir
mkdir: created directory '/rearbackup/subdir'
++ test -d /rearbackup/subdir
+++ tail -1
+++ awk '{print $6}'
+++ df -P /rearbackup/subdir
++ backup_directory_mountpoint=/rearbackup
++ test / = /rearbackup
...

and this is result

# ls -l /rearbackup/*/*/backup.tar.gz
-rw------- 1 root root 1266459574 Jul 21 15:34 /rearbackup/subdir/f121/backup.tar.gz

jsmeix commented at 2016-07-21 14:19:¶

Since https://github.com/rear/rear/issues/926#issuecomment-234255678
it looks hood to me and I like to have it tested by @tcerna
so that I simply merged https://github.com/rear/rear/pull/936 now.

@tcerna
does it now work better for you with current GitHub master code?

jsmeix commented at 2016-07-22 13:02:¶

Of course I also like to have it tested by
@scr4bble
does it now work better for you with current GitHub master code?

gdha commented at 2016-07-23 10:41:¶

@jsmeix thank you for your troubleshooting and fix

jsmeix commented at 2016-07-29 10:18:¶

No feedback from users.
Assuming no news are good news.
Closing it.

[Export of Github issue for rear/rear.]

#926 Issue closed: Local data backup with BACKUP=NETFS fails¶