#2928 Issue `closed`: NETFS tar backup no btrfs subvolumes by default¶

Labels: enhancement

schlomo opened issue at 2023-02-14 07:59:¶

ReaR 2.7 on Leap 15.4 x86_64

Config:

OUTPUT=ISO
BACKUP=NETFS
BACKUP_URL=nfs://<REDACTED-SERVER-NAME>/srv/scratch
REQUIRED_PROGS+=(chattr)

Disk layout is typical SUSE default

rear-sle15sp4:~ # df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        4,0M     0  4,0M   0% /dev
tmpfs           985M     0  985M   0% /dev/shm
tmpfs           394M   17M  378M   5% /run
tmpfs           4,0M     0  4,0M   0% /sys/fs/cgroup
/dev/sda2       8,0G  6,7G  1,1G  87% /
/dev/sda2       8,0G  6,7G  1,1G  87% /boot/grub2/x86_64-efi
/dev/sda2       8,0G  6,7G  1,1G  87% /home
/dev/sda2       8,0G  6,7G  1,1G  87% /boot/grub2/i386-pc
/dev/sda2       8,0G  6,7G  1,1G  87% /opt
/dev/sda2       8,0G  6,7G  1,1G  87% /root
/dev/sda2       8,0G  6,7G  1,1G  87% /srv
/dev/sda2       8,0G  6,7G  1,1G  87% /tmp
/dev/sda2       8,0G  6,7G  1,1G  87% /usr/local
/dev/sda2       8,0G  6,7G  1,1G  87% /var
tmpfs           197M  4,0K  197M   1% /run/user/0

The problem is that important content from /var is missing from the backup archive (checking on the backup server):

$ tar -tvzf /srv/scratch/rear-sle15sp4/backup.tar.gz | grep " var/"
drwxr-xr-x root/root         0 2023-02-12 21:38 var/
-rw------- root/root     1509845 2023-02-14 08:30 var/log/rear/rear-rear-sle15sp4.log

The root cause for this is that the actual backup tar command looks like this and includes only / and not the other BTRFS subvolumes. But it also has the --one-file-system option set so that tar won't descend into the subvolumes:

++ Print 'Creating tar archive '\''/var/tmp/rear.1fX2nBnGHRya4cJ/outputfs/rear-sle15sp4/backup.tar.gz'\'''
++ ProgressStart 'Preparing archive operation'
++ echo -en '\e[2K\rPreparing archive operation\e7'
++ BackupPID=5297
++ starttime=61
++ sleep 1
2023-02-14 08:28:36.965871429 tar --warning=no-xdev --sparse --block-number --totals --verbose --no-wildcards-match-slash --one-file-system --ignore-failed-read --anchored --xattrs --xattrs-include=security.capability --xattrs-include=security.selinux --acls --gzip -X /var/tmp/rear.1fX2nBnGHRya4cJ/tmp/backup-exclude.txt -C / -c -f - / /var/log/rear/rear-rear-sle15sp4.log | dd of=/var/tmp/rear.1fX2nBnGHRya4cJ/outputfs/rear-sle15sp4/backup.tar.gz bs=1M

And the root cause for that is that the backup-include.txt only contains /

It seems like we don't add the subvolumes to the backup automatically on BTRFS?? I'm wondering how many SUSE users stumble over that?

And yes, the SUSE-related example config actually explains the problem and provides the solution (thanks for including that @jsmeix !)

Maybe we should add some logic to automatically set the following in case BACKUP_PROG_INCLUDE is not defined:

BACKUP_PROG_INCLUDE=( $(findmnt -n -r -o TARGET -t btrfs | grep -v '^/$' | egrep -v 'snapshots|crash') )

Another question: On other systems not using BTRFS, does ReaR behave the same and include only / in the backup or all filesystems? If on non-BTRFS systems we include everything by default unless configured differently then I'd suggest to make ReaR behave the same on BTRFS systems too.

jsmeix commented at 2023-02-14 11:55:¶

Technically it behaves as btrfs behaves.
For btrfs a subvolume means for 'tar' with '--one-file-system'
that the btrfs subvolume is not included.

If we changed that we would make things behave incosistently
in ReaR because then 'tar' with '--one-file-system' would behave
different in ReaR for btrfs subvolumes than what plain 'tar'
would do with '--one-file-system' for btrfs subvolumes and
ReaR would behave different for btrfs subvolumes than for
other things that are not included with '--one-file-system'.

Making the backup is in general external functionality in ReaR
cf. "Relax-and-Recover versus backup and restore" in
https://en.opensuse.org/SDB:Disaster_Recovery

ReaR calls a backup tool but ReaR is not meant to be
backup configuration or management software so it is
the user's task to specify things in ReaR for 'tar'
in the same way as when he makes a backup whith plain 'tar'.

Perhaps I should explain this in more detail in default.conf
what it means that 'tar' is called with '--one-file-system'
so the user is informed to specify things as he needs it.
Because as far as I see we do not tell our users
that we call 'tar' with '--one-file-system'
neither in default.conf nor in a doc/* file.

pcahyna commented at 2023-02-14 11:58:¶

ISTR @antonvoznia saw a similar issue on Fedora, where the default is now also BTRFS and /home got excluded from backup (because it was a subvolume).

pcahyna commented at 2023-02-14 12:00:¶

IMO ReaR should treat subvolumes the same way as it does for separate filesystems: pass their mountpoints to the tar command explicitly. But maybe I misunderstand how it works - I have not yet dug deeper in the internal backup inclusion/exclusion code.

schlomo commented at 2023-02-14 12:08:¶

All correct. My question would be "what does the user expect?"

And my answer would be: "By default the backup should cover everything"

That is why I would like to suggest that by default ReaR NETFS backup should always perform a "full backup" of the system.

pcahyna commented at 2023-02-14 12:18:¶

I would not expect a different behavior if /home is a separate BTRFS subvolume and if it is a separate XFS filesystem.

schlomo commented at 2023-02-14 13:19:¶

So what happens on a not-BTRFS system where different directories like /home or /var are on different file systems? Will ReaR automatically include them all in the backup by default or only after explicit configuration?

pcahyna commented at 2023-02-14 13:24:¶

I am quite sure it is the former - all disk-based filesystems are added to backup by default.

schlomo commented at 2023-02-14 13:31:¶

Then I'd like to see that behaviour for SUSE as well, regardless of the filesystem used

jsmeix commented at 2023-02-14 13:35:¶

Offhandedly I may agree that there should not be a different behavior
if /somedir is a btrfs subvolume or if it is a separate filesystem
but when I think about it I get stuck in the dilemma
that I described above:

'tar' with '--one-file-system' has same behavior
when /somedir is a btrfs subvolume or a filesystem.

But now ReaR should have a differnt behavior
when /somedir is a btrfs subvolume versus a filesystem.

How should ReaR behave consistently and trustworthy?

When the user knows 'tar' is called with '--one-file-system'
then he must be able to trust ReaR to behave accordingly.

FYI:
I will not enter a discussion what users expect.
I had that in a totally different area in the past
more than enough ad nauseam (yes, really AD NAUSEAM).
It only wastes my energy and likely the energy of others too.
I would stop contributing to ReaR (except SUSE forces me)
if I had to deal with user expectation discussions.
I want ReaR to stay as it always was and still is:
A lower level expert tool (like 'parted' and such).
I would appreciate a fully separated user frontend
when someone makes it (e.g DRLM is some kind of that)
but I would not contribute to that.

I leave this issue now and let other ReaR maintainers
find out how to solve it.

jsmeix commented at 2023-02-16 14:36:¶

FYI
for clarification to avoid misunderstandings:

I refuse to think about what is commonly
called "user expectations" or similar.

It is right to ask for what one himself needs.

It is OK to forward truthfully what others asked for
but it would be better when they speak for themselves.

It is false to "impersonate" others and speak for them.

It works best to only focus on hard (technical) facts
and avoid to fade away into wishful thinking and the like.

jsmeix commented at 2023-02-16 14:52:¶

schlomo commented at 2023-02-21 21:51:¶

I guess that this is currently by design. I'm fine with leaving this as it is till somebody is willing to sponsor a better BTRFS support in ReaR.

jsmeix commented at 2023-02-22 08:41:¶

After sleeping on it for some time now
I meanwhile got a vague tentative untested idea
how it might be possible to implement that some
btrfs subvolumes get automatically included in the backup.

Please be patient - I need to try that out.

For me crucial parts are
"final power to the user"
so that the user must not have to work against the automatism
(i.e. an automatism must never enforce its stuff on the user)
and
backward compatibility
i.e. users who have specified

BACKUP_PROG_INCLUDE=( specific btrfs subvolumes )

must not get a changed behaviour because a new automatism
adds some more btrfs subvolumes that are not wanted.

pcahyna commented at 2023-03-14 10:23:¶

Hello @jsmeix ,
I also checked the behavior of tar with --one-file-system and btrfs subvolumes on and it is consistent with its behavior on mounted filesystems: if tar is called on /, the /home filesystem is not included in the backup if it is a separate mounted filesystem, and neither if it is a mounted btrfs subvolume. The problem is that ReaR does not pass the subvolume mount point to tar (while it passes the mount point to tar if it is a regular mounted FS like XFS), as described in #2604 .
I agree that snapshot subvolumes are a problem, even if one wants to include btrfs subvolumes, one should skip snapshots (at least in the first version). Maybe this is not particularly different from LVM snapshots? I don't even know how those would behave.
I am sorry that I can't provide a more concrete advice on btrfs as it seems fairly complicated and I am not familiar with it.
I can provide information on how it is configured by default on Fedora, though (the default install uses btrfs).
Do you want to see a disklayout.conf from a Fedora installation?

pcahyna commented at 2023-03-14 10:24:¶

@jsmeix If you have a proposed fix, I can also test it on Fedora.

jsmeix commented at 2023-03-14 10:45:¶

@pcahyna
could you show me on a Fedora installation the output of

# lsblk -ipo NAME,KNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINTS

if this works on Fedora (i.e. with 'MOUNTPOINTS' plural)
plus the disklayout.conf file.

The current layout/save/GNU/Linux/100_create_layout_file.sh
does not yet use 'MOUNTPOINTS' but only 'MOUNTPOINT'
in contrast to the function lsblk_output() in
layout/recreate/default/200_run_layout_code.sh
(I know - I need to align them - as time permits).

pcahyna commented at 2023-03-14 11:26:¶

Fedora 37:

lsblk -ipo NAME,KNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINTS

NAME       KNAME     TRAN TYPE FSTYPE LABEL                     SIZE MOUNTPOINTS
/dev/sda   /dev/sda  sas  disk                                465.7G 
|-/dev/sda1
|          /dev/sda1      part                                    1M 
|-/dev/sda2
|          /dev/sda2      part ext4                               1G /boot
`-/dev/sda3
           /dev/sda3      part btrfs  fedora_hpe-dl360gen8-01 464.7G /home
                                                                     /
/dev/zram0 /dev/zram0
                          disk                                    8G [SWAP]

disklayout.conf:

# Disk layout dated 20230314053733 (YYYYmmddHHMMSS)
# NAME        KNAME      PKNAME   TRAN TYPE FSTYPE LABEL                     SIZE MOUNTPOINT UUID                                 WWN
# /dev/sda    /dev/sda            sas  disk                                465.7G                                                 0x600508b1001c3b7c2286e4a50ffeb61f
# |-/dev/sda1 /dev/sda1  /dev/sda      part                                    1M                                                 0x600508b1001c3b7c2286e4a50ffeb61f
# |-/dev/sda2 /dev/sda2  /dev/sda      part ext4                               1G /boot      19baee1c-778c-49b8-90d5-a08e45618d1d 0x600508b1001c3b7c2286e4a50ffeb61f
# `-/dev/sda3 /dev/sda3  /dev/sda      part btrfs  fedora_hpe-dl360gen8-01 464.7G /home      12b9da8a-6151-49c1-a9af-fe6896f4d9f8 0x600508b1001c3b7c2286e4a50ffeb61f
# /dev/zram0  /dev/zram0               disk                                    8G [SWAP]                                          
# Disk /dev/sda
# Format: disk <devname> <size(bytes)> <partition label type>
disk /dev/sda 500051402752 gpt
# Partitions on /dev/sda
# Format: part <device> <partition size(bytes)> <partition start(bytes)> <partition type|name> <flags> /dev/<partition>
part /dev/sda 1048576 1048576 rear-noname bios_grub /dev/sda1
part /dev/sda 1073741824 2097152 rear-noname none /dev/sda2
part /dev/sda 498975375360 1075838976 rear-noname none /dev/sda3
# Filesystems (only ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs are supported).
# Format: fs <device> <mountpoint> <fstype> [uuid=<uuid>] [label=<label>] [<attributes>]
fs /dev/sda2 /boot ext4 uuid=19baee1c-778c-49b8-90d5-a08e45618d1d label= blocksize=4096 reserved_blocks=4% max_mounts=-1 check_interval=0d bytes_per_inode=16384 default_mount_options=user_xattr,acl options=rw,relatime,stripe=128
fs /dev/sda3 / btrfs uuid=12b9da8a-6151-49c1-a9af-fe6896f4d9f8 label='fedora_hpe-dl360gen8-01' options=rw,relatime,compress=zstd:1,space_cache=v2,subvolid=257,subvol=/root
# Btrfs default subvolume for /dev/sda3 at /
# Format: btrfsdefaultsubvol <device> <mountpoint> <btrfs_subvolume_ID> <btrfs_subvolume_path>
btrfsdefaultsubvol /dev/sda3 / 5 /
# Btrfs normal subvolumes for /dev/sda3 at /
# Format: btrfsnormalsubvol <device> <mountpoint> <btrfs_subvolume_ID> <btrfs_subvolume_path>
btrfsnormalsubvol /dev/sda3 / 256 home
btrfsnormalsubvol /dev/sda3 / 257 root
btrfsnormalsubvol /dev/sda3 / 258 root/var/lib/portables
# All mounted btrfs subvolumes (including mounted btrfs default subvolumes and mounted btrfs snapshot subvolumes).
# Determined by the findmnt command that shows the mounted btrfs_subvolume_path.
# Format: btrfsmountedsubvol <device> <subvolume_mountpoint> <mount_options> <btrfs_subvolume_path>
btrfsmountedsubvol /dev/sda3 / rw,relatime,seclabel,compress=zstd:1,space_cache=v2,subvolid=257,subvol=/root root
btrfsmountedsubvol /dev/sda3 /home rw,relatime,seclabel,compress=zstd:1,space_cache=v2,subvolid=256,subvol=/home home
# Mounted btrfs subvolumes that have the 'no copy on write' attribute set.
# Format: btrfsnocopyonwrite <btrfs_subvolume_path>
# Swap partitions or swap files
# Format: swap <filename> uuid=<uuid> label=<label>
swap /dev/zram0 uuid=cc5ac272-70ae-4538-b9e6-ace9852465a8 label=zram0