#3175 PR open: Automatically include mounted btrfs subvolumes in NETFS backups

Labels: enhancement

lzaoral opened issue at 2024-03-07 12:00:

Pull Request Details:

  • Type: Enhancement

  • Impact: High

  • Reference to related issue (URL): https://github.com/rear/rear/issues/2928

  • How was this pull request tested? rear savelayout and manual inspection of generated files and backup/restore of a Fedora Rawhide machine

  • Description of the changes in this pull request:

    • automatically include mounted btrfs subvolumes in NETFS backups
    • improve generation of $RESTORE_EXCLUDE_FILE

jsmeix commented at 2024-03-07 13:24:

@lzaoral
thank you for this enhancement!

I will test how it behaves on SLES systems
with their rather complicated default btrfs structure.

Offhandedly I think the main problem is
possibly mounted btrfs snapshot subvolumes, see
"When btrfs is used with snapshots ...
then usual backup and restore cannot work." in
https://en.opensuse.org/SDB:Disaster_Recovery#btrfs

I.e. when there are mounted btrfs "thingies"
listed as 'btrfsmountedsubvol' in disklayout.conf
that are also listed as '#btrfssnapshotsubvol'.

For example a disklayout.conf on SLES15-SP5 (excerpts)

btrfsdefaultsubvol /dev/sda2 / 268 @/.snapshots/1/snapshot
...
#btrfssnapshotsubvol /dev/sda2 / 272 @/.snapshots/2/snapshot
#btrfssnapshotsubvol /dev/sda2 / 273 @/.snapshots/3/snapshot
#btrfssnapshotsubvol /dev/sda2 / 274 @/.snapshots/4/snapshot
...
btrfsmountedsubvol /dev/sda2 / rw,relatime,space_cache,subvolid=268,subvol=/@/.snapshots/1/snapshot @/.snapshots/1/snapshot
btrfsmountedsubvol /dev/sda2 /.snapshots rw,relatime,space_cache,subvolid=267,subvol=/@/.snapshots @/.snapshots
btrfsmountedsubvol /dev/sda2 /boot/grub2/x86_64-efi rw,relatime,space_cache,subvolid=265,subvol=/@/boot/grub2/x86_64-efi @/boot/grub2/x86_64-efi
btrfsmountedsubvol /dev/sda2 /root rw,relatime,space_cache,subvolid=262,subvol=/@/root @/root
btrfsmountedsubvol /dev/sda2 /opt rw,relatime,space_cache,subvolid=263,subvol=/@/opt @/opt
btrfsmountedsubvol /dev/sda2 /home rw,relatime,space_cache,subvolid=264,subvol=/@/home @/home
btrfsmountedsubvol /dev/sda2 /boot/grub2/i386-pc rw,relatime,space_cache,subvolid=266,subvol=/@/boot/grub2/i386-pc @/boot/grub2/i386-pc
btrfsmountedsubvol /dev/sda2 /srv rw,relatime,space_cache,subvolid=261,subvol=/@/srv @/srv
btrfsmountedsubvol /dev/sda2 /tmp rw,relatime,space_cache,subvolid=260,subvol=/@/tmp @/tmp
btrfsmountedsubvol /dev/sda2 /usr/local rw,relatime,space_cache,subvolid=259,subvol=/@/usr/local @/usr/local
btrfsmountedsubvol /dev/sda2 /var rw,relatime,space_cache,subvolid=258,subvol=/@/var @/var

For example assume in addition to @/.snapshots/1/snapshot
that is the current system snapshot which is mounted at /
also
some other snapshots of the system like
@/.snapshots/2/snapshot and @/.snapshots/4/snapshot
are mounted e.g. at /snapshot2 and /snapshot4

Then a 'tar' backup would contain the system files
basically three times:

  1. the files of what is mounted at /
  2. the files of what is mounted at /snapshot2
  3. the files of what is mounted at /snapshot4

So the backup would be basically about three times
as big as if only what is mounted at / was backed up.

But during 'tar' restore there is no deduplication
so the restore would basically need about three times
the disk space as the original system needed.

pcahyna commented at 2024-03-07 13:32:

I think the main problem is
possibly mounted btrfs snapshot subvolumes

I am not a btrfs expert at all, but is it possible to distinguish snapshot subvolumes from "normal" (non-snapshot) subvolumes? Then we could save this subvolume metadata (snapshot yes/no) and do something based on the information when recreating and restoring. First we would probably just skip snapshots, later we could do something more intelligent if possible.
It is definitely possible to distinguish snapshots from regular filesystems (filessytems are equivalent to btrfs subvolumes) in ZFS. It is also possible to recognize snapshots from regular volumes in LVM.

jsmeix commented at 2024-03-07 13:49:

Only an offhanded thought:

I fear btrfs normal subvolumes versus btrfs snapshot subvolumes
is only one example of a very generic problem when by default
every mounted "thingy" is included in the 'tar' backup:

I think it is in general possible that one same
"mountable thingy" can be mounted at the same time
at different mount points e.g. at '/here' and '/there'.

When '/here' and '/there' are included in a 'tar' backup
things may get restored twice as distinct sets of files
and not as one same set of files that is mounted two times
under the mountpoint directories '/here' and '/there'.

Tomorrow I will experiment a bit with that.

jsmeix commented at 2024-03-07 13:52:

Perhaps we can by default have every mounted "thingy"
included in the 'tar' backup
BUT
we may need some check for duplicates in the 'tar' backup
i.e. something that detects when one same "mountable thingy"
will become included in the 'tar' backup more than once.

jsmeix commented at 2024-03-08 08:49:

Currently I am exploring how 'tar' behaves in general
when exact same files are provided as 'tar' arguments
to be archived.

It seems 'tar' behaves well forgiving in this case:

# mkdir test

# cd test

# echo foo >foo

# tar -cvvvf test.tar foo foo
-rw-r--r-- root/root         4 2024-03-08 09:41 foo
hrw-r--r-- root/root         0 2024-03-08 09:41 foo link to foo

# tar -tvvvf test.tar
-rw-r--r-- root/root         4 2024-03-08 09:41 foo
hrw-r--r-- root/root         0 2024-03-08 09:41 foo link to foo

# mkdir untartest

# cd untartest/

# tar -xvvvf ../test.tar
-rw-r--r-- root/root         4 2024-03-08 09:41 foo
hrw-r--r-- root/root         0 2024-03-08 09:41 foo link to foo

# ls -l
total 4
-rw-r--r-- 1 root root 4 Mar  8 09:41 foo

So perhaps only mounted btrfs snapshot subvolumes
added to 'tar' arcives cause real problems in practice.
In this case btrfs snapshot subvolumes should be excluded
by default from being added to what 'tar' should archive.

I think it is OK if a user mounts the same stuff
at different mount points and ReaR includes those
mount points by default in the 'tar' backup
then it is up to the user to manually exclude things
as needed from his backup.

pcahyna commented at 2024-03-08 08:59:

@jsmeix

I fear btrfs normal subvolumes versus btrfs snapshot subvolumes is only one example of a very generic problem when by default every mounted "thingy" is included in the 'tar' backup:

I think it is in general possible that one same "mountable thingy" can be mounted at the same time at different mount points

I disagree that it is an example of this problem. Snapshots are not the same thing mounted at different places. They are different things mounted at different places - snapshots exist because their content is (at least in principle) different.

One filesystem mounted at more places can occur as well, and it will result in an explosion of backup data, but it restoring it twice then should not result in an increase of the size of the restored system, only in a slower restore, because you keep restoring to the same filesystem.

I would not try to solve these two problems in the same way (cf. RFC 1925 item 5).

pcahyna commented at 2024-03-08 09:51:

It seems 'tar' behaves well forgiving in this case:

# mkdir test

# cd test

# echo foo >foo

# tar -cvvvf test.tar foo foo
-rw-r--r-- root/root         4 2024-03-08 09:41 foo
hrw-r--r-- root/root         0 2024-03-08 09:41 foo link to foo

# tar -tvvvf test.tar
-rw-r--r-- root/root         4 2024-03-08 09:41 foo
hrw-r--r-- root/root         0 2024-03-08 09:41 foo link to foo

# mkdir untartest

# cd untartest/

# tar -xvvvf ../test.tar
-rw-r--r-- root/root         4 2024-03-08 09:41 foo
hrw-r--r-- root/root         0 2024-03-08 09:41 foo link to foo

# ls -l
total 4
-rw-r--r-- 1 root root 4 Mar  8 09:41 foo

It thinks that the doubled file names are different names for the same files (i.e. hardlinks), which is not entirely correct - not sure if it can have some unwanted consequences or not.

jsmeix commented at 2024-03-08 12:16:

Yes.

The whole point of my experiments with 'tar' here
is to find out if my "fear" above is true or not and
in general to better understand what we have to deal with.

If it is actually only one root problem
then this root problem should be solved
(instead of solving each of its instances).

If it is actually several separated problems then
each separated problem should be solved separately.

https://github.com/rear/rear/pull/3175#issuecomment-1985290555
indicates that it is several separated problems
(but this is only my very first test in this area).

From my experiments with 'tar' in the past I know that
'tar' behaves deterministically (i.e. as programmed and
documented when reading the whole 'tar' manual carefully)
but that could appear rather often 'unexpectedly'
(i.e. different than what one may expect offhandedly), e.g.
https://github.com/rear/rear/issues/2911#issuecomment-1398346148

pcahyna commented at 2024-03-08 12:45:

If it is actually only one root problem then this root problem should be solved (instead of solving each of its instances).

If it is actually several separated problems then each separated problem should be solved separatedly.

That's an interesting idea. For multiple identical arguments to tar, tar duplicates the backup and considers the secondary copy as a hardlink to the first copy. I checked that the same happens when there is a filesystem mounted multiple times:

# mkdir /mnt/mount1
# mkdir /mnt/mount2
# mount /dev/vdb /mnt/mount1
# mount /dev/vdb /mnt/mount2
# touch /mnt/mount1/foo
# ls -l /mnt/mount2/foo
-rwxr-xr-x. 1 root root 0 Mar  8 07:36 /mnt/mount2/foo
# tar cvvf /dev/null /mnt/mount1 /mnt/mount2
tar: Removing leading `/' from member names
drwxr-xr-x root/root         0 1969-12-31 19:00 /mnt/mount1/
-rwxr-xr-x root/root         0 2024-03-08 07:36 /mnt/mount1/foo
tar: Removing leading `/' from hard link targets
drwxr-xr-x root/root         0 1969-12-31 19:00 /mnt/mount2/
hrwxr-xr-x root/root         0 2024-03-08 07:36 /mnt/mount2/foo link to mnt/mount1/foo

Is the same happening with different btrfs snapshots mounted at different mountpoints? I.e. does tar consider files in different snapshots (originally same, but possibly different when they have been modified since the snapshot was taken) as hardlinks to the same file?

lzaoral commented at 2024-03-08 12:48:

Thank you for the feedback, @jsmeix! I'll amend the code to skip backup of all mounted btrfs snapshot subvolumes.

The duplication of files in backup when a filesystem/btrfs subvolume is mounted more than once is a different (though related) issue, therefore, I suggest to resolve it separately.

jsmeix commented at 2024-03-08 15:35:

I tested it with SLES15-SP5
with the default btrfs structure
on a QEMU/KVM test VM:

# lsblk -ipo NAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINTS
NAME        TRAN TYPE FSTYPE   SIZE MOUNTPOINTS
/dev/sda    ata  disk           15G 
|-/dev/sda1      part            8M 
|-/dev/sda2      part btrfs     13G /var
|                                   /usr/local
|                                   /root
|                                   /tmp
|                                   /srv
|                                   /boot/grub2/i386-pc
|                                   /opt
|                                   /home
|                                   /boot/grub2/x86_64-efi
|                                   /.snapshots
|                                   /
`-/dev/sda3      part swap       2G [SWAP]

I was in particular interested how things behave
with the "well known" (to SLES users) SUSE specific

BACKUP_PROG_INCLUDE=( $( findmnt -n -r -o TARGET -t btrfs | grep -v '^/$' | egrep -v 'snapshots|crash' ) )

manual setting in etc/rear/local.conf
cf. conf/examples/SLE12-SP2-btrfs-example.conf
so I have

OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://192.168.178.66/nfs
REQUIRED_PROGS+=( snapper chattr )
PROGS+=( lsattr )
COPY_AS_IS+=( /usr/lib/snapper/installation-helper /etc/snapper/config-templates/default )
BACKUP_PROG_INCLUDE=( /boot/grub2/i386-pc /boot/grub2/x86_64-efi /home /opt /root /srv /tmp /usr/local /var )

With that I got duplicated things in the backup.tar.gz

To make ReaR behave backward compatible for SLES users
and because it seems to be "the right thing" in general
I implemented
https://github.com/rear/rear/pull/3177

With this additional changes I get no longer
duplicated things in the backup.tar.gz
BUT
I did not yet test "rear recover".
This will be done next week.

@lzaoral @pcahyna @rear/contributors
I wish you a relaxed and recovering weekend!

pcahyna commented at 2024-03-08 16:19:

@jsmeix if you have snapshots, can you please test https://github.com/rear/rear/pull/3175#issuecomment-1985632419 : "does tar consider files in different snapshots (originally same, but possibly different when they have been modified since the snapshot was taken) as hardlinks to the same file?" ?

pcahyna commented at 2024-03-10 13:34:

I was in particular interested how things behave with the "well known" (to SLES users) SUSE specific

BACKUP_PROG_INCLUDE=( $( findmnt -n -r -o TARGET -t btrfs | grep -v '^/$' | egrep -v 'snapshots|crash' ) )

manual setting in etc/rear/local.conf cf. conf/examples/SLE12-SP2-btrfs-example.conf so I have

OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://192.168.178.66/nfs
REQUIRED_PROGS+=( snapper chattr )
PROGS+=( lsattr )
COPY_AS_IS+=( /usr/lib/snapper/installation-helper /etc/snapper/config-templates/default )
BACKUP_PROG_INCLUDE=( /boot/grub2/i386-pc /boot/grub2/x86_64-efi /home /opt /root /srv /tmp /usr/local /var )

With that I got duplicated things in the backup.tar.gz

Is it a regression with this PR, or did you get duplicated entries in backup.tar.gz even before? What are the duplicated entries? Aren't you missing BACKUP_ONLY_INCLUDE="yes"? (but then you should probably add /boot to BACKUP_PROG_INCLUDE unless on SLES you have /boot as part of /).

pcahyna commented at 2024-03-11 13:23:

@jsmeix if you have snapshots, can you please test #3175 (comment) : "does tar consider files in different snapshots (originally same, but possibly different when they have been modified since the snapshot was taken) as hardlinks to the same file?" ?

I tested ZFS and the same files in a snapshot and in the original filesystem do not show up as hardlinks to the same file in the tar output. Of course, although Btrfs is in many ways analogous to ZFS, it can behave differently in details like that.

pcahyna commented at 2024-03-26 16:43:

Hi all, reviewing what needs to be done there.

pcahyna commented at 2024-04-09 10:04:

Hi @jsmeix can you please have a look? I believe the first four items in the checklist above are for you (the second only partially, you implement uniq_unsorted in #3177 and then @lzaoral will use it here). Is the task list ok?

jsmeix commented at 2024-04-09 12:48:

@pcahyna
I will have a look.

First I would like to implement uniq_unsorted
or perhaps even better named unique_unsorted cf.
https://github.com/rear/rear/pull/3177#issuecomment-2045095158
so that @lzaoral could use it here.

[Export of Github issue for rear/rear.]