mountpoint is not skipped during creating backup¶

Labels: support / question, fixed / solved / done

asmkaiser opened issue at 2020-06-03 10:52:¶

Hello experts,

Hope you are doing well.
Its been years since I was really happy with the Rear backup recovery.
Buy recently I have been facing a picualliar issue while creating a backup.

so here is the scenario. I accidentally installed the server into 'nvme' disk instead of 'sda'. So I was planning to take backup of the image. and recover the system under 'sda'.

Disk output

**$ lsblk**
NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda             8:0    0 446.6G  0 disk 
├─sda1          8:1    0     1G  0 part 
├─sda2          8:2    0    16G  0 part 
└─sda3          8:3    0 429.6G  0 part 
sdb             8:16   0  65.5T  0 disk 
**└─sdb1          8:17   0  65.5T  0 part /Backup  <=== [should be skipped]**
nvme0n1       259:0    0 372.6G  0 disk 
├─nvme0n1p1   259:1    0     1G  0 part /boot
├─nvme0n1p2   259:2    0     4G  0 part [SWAP]
└─nvme0n1p3   259:3    0 367.6G  0 part 
  └─rhel-root 253:0    0 367.6G  0 lvm  /
nvme1n1       259:4    0 372.6G  0 disk 
```

My rear conf :

 $ sudo cat /etc/rear/local.conf

OUTPUT=ISO
OUTPUT_URL=file:///Backup/rear/
BACKUP=NETFS
BACKUP_URL=iso:///Backup/rear/
EXCLUDE_MOUNTPOINTS=( '/Backup/*' )
AUTOEXCLUDE_PATH=( /Backup )
USE_RESOLV_CONF="no"
AUTOEXCLUDE_DISKS=( /dev/sda /dev/sda1 /dev/sda2 /dev/sda3 /dev/sdb1 /dev/sdb )
EXCLUDE_COMPONENTS=( "${EXCLUDE_COMPONENTS[@]}" "fs:/sda1" "swap:/dev/sda2" "pv:/dev/sda3" "fs:/dev/sdb1" "fs:/Backup/" )
export TMPDIR="/Backup/rear-tmp/"

So , while I ran the backup it showed that was skipping the '/Backup' directory where the '/dev/sdb1' was mounted. So the total backup should be 450GB maximum. but the backup is still compressing and continue upto 3 TB. that means it is also compressing the data under /Backup directory. Because I have over 6TB of backup under that /backup directory.

output:

# rear -v mkbackup                                       
Relax-and-Recover 2.5 / 2019-05-10
Running rear mkbackup (PID 34528)
Using log file: /var/log/rear/rear-syslog-jessore.log
Using backup archive '/Backup/rear-tmp/rear.KGxKgj8F7REACSE/tmp/isofs/Backup/rear//backup.tar.gz'
Using autodetected kernel '/boot/vmlinuz-3.10.0-693.el7.x86_64' as kernel in the recovery system
Creating disk layout
Excluding component fs:/sda1.
Excluding component swap:/dev/sda2.
Excluding component pv:/dev/sda3.
**Excluding component fs:/dev/sdb1. <== [ Skipped ]
Excluding component fs:/Backup/. <== [ Skipped ]**
Using guessed bootloader 'GRUB' (found in first bytes on /dev/nvme0n1)
Verifying that the entries in /var/lib/rear/layout/disklayout.conf are correct ...
Creating root filesystem layout
Skipping 'virbr0': not bound to any physical interface.
To log into the recovery system via ssh set up /root/.ssh/authorized_keys or specify SSH_ROOT_PASSWORD
Copying logfile /var/log/rear/rear-syslog-jessore.log into initramfs as '/tmp/rear-syslog-jessore-partial-2020-06-01T22:11:06+0600.log'
Copying files and directories
Copying binaries and libraries
Copying all kernel modules in /lib/modules/3.10.0-693.el7.x86_64 (MODULES contains 'all_modules')
Copying all files in /lib*/firmware/
Broken symlink '/usr/lib/modules/3.10.0-693.el7.x86_64/source' in recovery system because 'readlink' cannot determine its link target
Broken symlink '/usr/lib/modules/3.10.0-693.el7.x86_64/build' in recovery system because 'readlink' cannot determine its link target
Testing that the recovery system in /Backup/rear-tmp/rear.KGxKgj8F7REACSE/rootfs contains a usable system
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (175796106 bytes) in 18 seconds
Creating tar archive '/Backup/rear-tmp/rear.KGxKgj8F7REACSE/tmp/isofs/Backup/rear//backup.tar.gz'
****Archived 2575943 [avg 17381 Kib/sec] **<==== [ Still compressing 2.5TB ++]**

Anything in your mind guys ???

jsmeix commented at 2020-06-03 11:37:¶

Ugh, yet another issue with our "include exclude hell" in ReaR ;-)

As a quick offhanded proposal without actual analysis you may have
a look at things like BACKUP_ONLY_INCLUDE in default.conf
https://github.com/rear/rear/blob/master/usr/share/rear/conf/default.conf#L1091

In general including/excluding components like disks/partitions/filesystems
is different from what gets included/excluded regarding the backup.
There are some relationships but in general both are different things.

What gets included/excluded regarding the backup
in case of BACKUP=NETFS happens mainly via
backup/NETFS/default/400_create_include_exclude_files.sh
https://github.com/rear/rear/blob/master/usr/share/rear/backup/NETFS/default/400_create_include_exclude_files.sh

Run "rear -D mkbackup" (-D is debugscript mode)
and inspect the log file what exactly happens while
backup/NETFS/default/400_create_include_exclude_files.sh
is running. The log file should (hopefully) show why something
is automatically included in the backup in your particular case.

Regarding including/excluding components see
https://github.com/rear/rear/issues/2229

asmkaiser commented at 2020-06-04 10:54:¶

@jsmeix , Yes those are as it is, I tried another varification after checking the log , that included the directory,

 $ sudo cat /etc/rear/local.conf

OUTPUT=ISO
OUTPUT_URL=file:///Backup/rear/
BACKUP=NETFS
BACKUP_URL=iso:///Backup/rear/
EXCLUDE_MOUNTPOINTS=( '/Backup/*' ) <========== this is the problem, shoule be /Backup
AUTOEXCLUDE_PATH=( /Backup )
USE_RESOLV_CONF="no"
AUTOEXCLUDE_DISKS=( /dev/sda /dev/sda1 /dev/sda2 /dev/sda3 /dev/sdb1 /dev/sdb )
EXCLUDE_COMPONENTS=( "${EXCLUDE_COMPONENTS[@]}" "fs:/sda1" "swap:/dev/sda2" "pv:/dev/sda3" "fs:/dev/sdb1" "fs:/Backup/" )
export TMPDIR="/Backup/rear-tmp/"

Corrected by removing the asterisk * after /Backup/

 $ sudo cat /etc/rear/local.conf

OUTPUT=ISO
OUTPUT_URL=file:///Backup/rear/
BACKUP=NETFS
BACKUP_URL=iso:///Backup/rear/
EXCLUDE_MOUNTPOINTS=( '/Backup' )  <==== corrected
AUTOEXCLUDE_PATH=( /Backup )
USE_RESOLV_CONF="no"
AUTOEXCLUDE_DISKS=( /dev/sda /dev/sda1 /dev/sda2 /dev/sda3 /dev/sdb1 /dev/sdb )
EXCLUDE_COMPONENTS=( "${EXCLUDE_COMPONENTS[@]}" "fs:/sda1" "swap:/dev/sda2" "pv:/dev/sda3" "fs:/dev/sdb1" "fs:/Backup/" )
export TMPDIR="/Backup/rear-tmp/"

So the problem is solved but question remains . it was not supposed to include the directory with this.

EXCLUDE_MOUNTPOINTS=( '/Backup/*' )

jsmeix commented at 2020-06-04 13:27:¶

@asmkaiser
thank you for your feedback what the actual reason was!

I had missed the wrong syntax of the
EXCLUDE_MOUNTPOINTS array element.

I think the examples in default.conf
https://github.com/rear/rear/blob/master/usr/share/rear/conf/default.conf#L2599

# examples: /tmp
#           /media/bigdisk
EXCLUDE_MOUNTPOINTS=()

indicate that the values in the EXCLUDE_MOUNTPOINTS array
should be directories but in particular no quoted globbing patterns.

The code for EXCLUDE_MOUNTPOINTS in
https://github.com/rear/rear/blob/master/usr/share/rear/backup/NETFS/default/400_create_include_exclude_files.sh#L29

    for excluded_mountpoint in "${EXCLUDE_MOUNTPOINTS[@]}" ; do
        test "$excluded_mountpoint" && echo "$excluded_mountpoint/"
    done >> $TMP_DIR/backup-exclude.txt

proves that quoted globbing patterns actually behave
as quoted values should behave in bash, for example:

# EXCLUDE_MOUNTPOINTS=( '/etc/*' )

# for excluded_mountpoint in "${EXCLUDE_MOUNTPOINTS[@]}" ; do echo "$excluded_mountpoint/" ; done
/etc/*/

versus non-quoted globbing patterns

# EXCLUDE_MOUNTPOINTS=( /etc/* )

# for excluded_mountpoint in "${EXCLUDE_MOUNTPOINTS[@]}" ; do echo "$excluded_mountpoint/" ; done
/etc/adjtime/
/etc/aliases/
/etc/aliases.d/
/etc/aliases.db/
/etc/alsa-pulse.conf/
/etc/alternatives/
...

jsmeix commented at 2020-06-04 14:01:¶

Via
https://github.com/rear/rear/commit/0ea61df57be40833127a30736b9e3f9c5f0cc5a8
I better describe the usage of non-quoted globbing patterns
versus quoted globbing patterns in default.conf

[Export of Github issue for rear/rear.]