#821 Issue closed: ReaR + TSM: btrfs subvolumes excluded by default from restore¶
Labels: bug, documentation, fixed / solved / done
pavoldomin opened issue at 2016-04-25 13:06:¶
- rear version (/usr/sbin/rear -V): 1.18
- OS version (cat /etc/rear/os.conf or lsb_release -a): SLES11SP3
- rear configuration files (cat /etc/rear/site.conf or cat /etc/rear/local.conf):
OUTPUT=ISO
BACKUP=TSM
COPY_AS_IS_TSM=( /etc/adsm/TSM.PWD /opt/tivoli/tsm/client /usr/local/ibm/gsk8* )
COPY_AS_IS_EXCLUDE_TSM=( )
PROGS_TSM=( dsmc tput )
# where to copy the resulting files to and save them with TSM
TSM_RESULT_FILE_PATH=/opt/tivoli/tsm/rear
TSM_DSMC_RESTORE_OPTIONS=( )
TSM_RESTORE_PIT_DATE=
TSM_RESTORE_PIT_TIME=
# Should the result from mkrecover/backup saved via TSM
TSM_RESULT_SAVE=y
OUTPUT_PREFIX="backup"
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://nfs_server/rear/tsm/$HOSTNAME
NETFS_KEEP_OLD_BACKUP_COPY=yes
BACKUP_PROG_INCLUDE=( '/var/lib/*' '/var/spool/*' '/var/log/*' '/var/run/*' '/srv/*' '/boot/grub2/x86_64-efi/*' '/opt/*' '/var/*' '/boot/grub2/i386-pc/*' '/boot/* /boot/efi/*' )
EXCLUDE_RECREATE=( "${EXCLUDE_RECREATE[@]}" "fs:/.snapshots" "fs:/var/crash" "fs:/var/run" "fs:/osbackup" "fs:/hana/osbackup" "fs:/origin" "fs:/origin_tmp" "fs:/backup"
...
- Brief description of the issue: TSM is not including automatically for btrfs subvolumes as those are excluded from the recreation, to prevent creating them as normal btrfs volumes (see 548965e).
- Work-around: We need to manually include the auto-excluded subvolumes to the restore list, like
The TSM Server reports the following for this node:
# Last Incr Date Type File Space Name
--------------------------------------------------------------------------------
1 23-04-2016 04:59:41 BTRFS /
2 00-00-0 00:00:00 XFS /backuptest
3 23-04-2016 04:59:42 EXT3 /boot
4 23-04-2016 04:59:42 VFAT /boot/efi
...
17 23-04-2016 04:59:55 BTRFS /var
Please enter the numbers of the filespaces we should restore.
Pay attention to enter the filesystems in the correct order
(like restore / before /var/log) !
(default: 1 3 4 5 6 7 11 12 13 15 16): [30 secs] 1 3 4 5 6 7 10 11 12 13 15 16 17
I do not have any smart suggestion how this can be 'fixed'. I just found
that /var was not restored by default by TSM on btrfs system. This may
well be also a TSM support for BTRFS is not perfect.
gdha commented at 2016-04-26 08:13:¶
@jsmeix Any idea how we can fore-come this (auto-excluding btrfs /var subvolume)
jsmeix commented at 2016-04-26 12:03:¶
There is nothing at all what I could do regarding
issues with third-party backup tools.
When the issue also hapens with 'tar'
I will have a look.
@pavoldomin
does the issue also happen with 'tar'?
As far as I understand the comments in default.conf
and as far as I remember how it works with 'tar'
items in EXCLUDE_RECREATE are added to EXCLUDE_BACKUP
but items in BACKUP_PROG_INCLUDE take precedence over
items in EXCLUDE_BACKUP so that in the end what is specified
in BACKUP_PROG_INCLUDE gets included in the backup
regardless of what is set in EXCLUDE_RECREATE.
jsmeix commented at 2016-04-26 13:50:¶
The issue (or at least a similar issue with same result)
also happens with 'tar'.
With
BACKUP_PROG_INCLUDE=( '/var/spool/*' )
EXCLUDE_RECREATE=( "${EXCLUDE_RECREATE[@]}" 'fs:/var/spool' )
one gets the whole content of /var/spool/* in the backup.tar.gz
(during "rear mkbackup")
but
one does not get the content of /var/spool/* restored
(during "rear recover").
In 61_exclude_from_restore.sh the entries
in EXCLUDE_RECREATE become added
to the restore-exclude-list.txt file
that is used for the 'tar' restore
as --exclude-from=...restore-exclude-list.txt
so that entries in BACKUP_PROG_INCLUDE
do not get restored.
The fix with newer rear versions is easy:
With newer rear versions theer is no longer the need
for entries in /etc/rear/local.conf like
EXCLUDE_RECREATE=( ... 'fs:/var/spool' ... )
to exclude them during component recreation.
Reason:
With newer rear versions btrfs subvolumes are not
listed in disklayout.conf as
fs ...
components (as it was during my initial btrfs support implementation)
but now via well separated components as
btrfsdefaultsubvol ... btrfssnapshotsubvol ... btrfsnormalsubvol ... btrfsmountedsubvol ... btrfsnocopyonwrite ...
so that in /etc/rear/local.conf entries like
EXCLUDE_RECREATE=( ... 'fs:/var/spool' ... )
do not match anything during component recreation
but still match during backup restore.
I tested on SLES12-SP1 with default btrfs subvolumes
that without entries in /etc/rear/local.conf like
EXCLUDE_RECREATE=( ... 'fs:/var/spool' ... )
I do get the whole content of /var/spool/* restored
from the backup.
@pavoldomin
please test if it also works for you this way with TSM.
jsmeix commented at 2016-04-26 13:52:¶
I need to update the btrfs example config files
/usr/share/rear/conf/examples/SLE12-btrfs-example.conf
/usr/share/rear/conf/examples/SLE12-SP1-btrfs-example.conf
jsmeix commented at 2016-04-26 14:06:¶
And this whole automagic transitive excluding
needs to be much better described in default.conf
I will do a pull request after feedback from @pavoldomin
pavoldomin commented at 2016-04-26 14:21:¶
Sorry for late response, our exchange decided to move rear emails to junk folder, how rude.
Thanks @jsmeix. I'll test this tomorrow. For NETFS backup I still
should keep subvols in BACKUP_PROG_INCLUDE, right? Just remove them
from EXCLUDE_RECREATE
jsmeix commented at 2016-04-26 14:30:¶
@pavoldomin
yes,
you must list those btrfs subvolumes in BACKUP_PROG_INCLUDE
that you want to have in backup.tar.gz during "rear mkbackup"
but you do not need any btrfs subvolumes in EXCLUDE_RECREATE
jsmeix commented at 2016-04-27 12:47:¶
Ha!
Now I even know why it had worked
with my initial btrfs support implementation
with entries in /etc/rear/local.conf like
EXCLUDE_RECREATE=( ... 'fs:/var/spool' ... )
to get the whole content of /var/spool/* restored
from the backup.
I was sure that I had tested it at that time
and that it had worked without missing
stuff during backup restore.
My initial btrfs support implementation
was never sent to rear upstream
because it was a bad first-attempt hack
that only works for SLES12-GA.
My initial btrfs support implementation
was only implemented for SLES12-GA as a
patch "adaptions_for_btrfs_for_SLE12.diff" for
rear version 1.16 in our SLES12 RPM package "rear116".
The "rear116" RPM package changelog shows
in particular this entry:
Fri Jul 4 12:42:49 CEST 2014 - jsmeix@suse.de - Enhanced adaptions_for_btrfs_for_SLE12.diff so that it skips filesystem components from being automatically added to restore-exclude-list.txt when the files are explicitly listed to be included in the backup via BACKUP_PROG_INCLUDE to avoid that files in btrfs subvolumes get automatically exclued from being restored from the backup (bnc#885698). - SLE12-btrfs-example.conf is a working example for SLE12 with default btrfs subvolume that gets installed as /usr/share/rear/conf/SLE12-btrfs-example.conf
The adaptions_for_btrfs_for_SLE12.diff changes
in particular 61_exclude_from_restore.sh
to skip filesystem components in EXCLUDE_RECREATE
that have a matching entry in BACKUP_PROG_INCLUDE
from being automatically added to restore-exclude-list.txt
The "automatically added" is the crucial point here.
What the user has specified in EXCLUDE_RESTORE
is still added to restore-exclude-list.txt
so that with e.g. the following in local.conf
BACKUP_PROG_INCLUDE=( '/var/spool/*' '/var/log/*' )
EXCLUDE_RECREATE=( "${EXCLUDE_RECREATE[@]}" 'fs:/var/spool' )
EXCLUDE_RESTORE=( '/var/log/*' )
the whole content in /var/spool/ and /var/log/ will be
saved in backup.tar.gz during "rear mkbackup"
and during "rear recover" /var/spool/* will no longer
be automatically added to restore-exclude-list.txt
but /var/log/* will still be added to restore-exclude-list.txt
so that whole content of /var/spool/ will be restored
but not the content of /var/log/.
In contrast my more generic btrfs support implementation
(that should also work on Red Hat and Fedora) which
was included in rear upstream did no longer have that
change of 61_exclude_from_restore.sh because it does
not need that change because in my rear upstream btrfs
support implementation there is no longer the need
to specify btrfs subvolumes in EXCLUDE_RECREATE.
All what I missed to do was to remove the outdated
and leftover EXCLUDE_RECREATE stuff from
the btrfs example config files
/usr/share/rear/conf/examples/SLE12-btrfs-example.conf
/usr/share/rear/conf/examples/SLE12-SP1-btrfs-example.conf
pavoldomin commented at 2016-04-28 09:42:¶
Hi,
I've tested it and it is not working with TSM.
I also found a reason: in verify/TSM/default/40_verify_tsm.sh the
default filesystems included in TSM restore plan are taken from
disklayout.conf by
# Use the included_mountpoints array derived from the disklayout.conf to determine the default
# TSM filespaces to include in a restore.
included_mountpoints=( $(grep ^fs $VAR_DIR/layout/disklayout.conf | awk '{print $3}') )
This wont match btrfs subvolumes - as those are listed as
...
btrfsnormalsubvol /dev/sda2 / 262 @/var
...
I am afraid decision which BTRFS volumes are included in restore by default requires bit of redesign of the code.
In any case, I am happy with my workaround, just need to place the missing filespaces to the offered list within 30 seconds. We are using workarounds with the TSM backup already - as the used version does not seem to work with BTRFS subvols properly. Without them, the critical filesystems like '/var' are not even backed up.
jsmeix commented at 2016-04-28 12:50:¶
@pavoldomin
have you also tested it with 'tar' (i.e. NETFS backup).
If yes, does it work for you with tar/NETFS ?
Regarding TSM:
First a general question about TSM:
Can TSM operate only on filesystems (i.e. mountpoints)
or can TSM operate also on any directories (like 'tar')?
In other words:
Does TSM save avd restore whole filesystems (like 'dd')
or does TSM save and restore files and directories?
Do I understand it correctly that content of btrfs subvolumes
get saved in the TSM backup during "rear mkbackup"
but does not get restored during "rear recover"?
In general I think backup restore based on 'fs' components
in disklayout.conf at least looks strange from my point of view
(and I would have never ever expected such a dependency).
In general I think what is saved in the backup and
what is restored should be handled well separated from
what storage components are saved and recreated.
For example I would expect that it "just works" to recreate
only the root filesystem component and restore all files
from the backup in there regardless from how many
different filesystems or whatever other kind of storage
components (like btrfs subvolumes) the files in the backup
had been saved.
In general I would prefer to keep separated things separated.
In particular:
I can maintain the plain btrfs support in rear
but I cannot also maintain any kind of "hidden"
interdependent stuff - especially I cannot do anything
for third-party backup tools that I do not have on my
systems (i.e. where I cannot test anything).
In particular see https://github.com/rear/rear/issues/769#issuecomment-183282736 (excerpt):
IMHO we would be better off to extract the backup functionality from ReaR into a separate Open Source project which deals only with backup and restore. That way we can stay true to ReaRs goal of doing the "bare metal" part while also delivering value with a new backup solution that can focus only on that part.
In general I think there are too many interdependant
automatisms that try to make "usual things just work"
but result unexpected obscure failures in other cases
(like this one).
jsmeix commented at 2016-04-28 13:01:¶
In verify/TSM/default/40_verify_tsm.sh
# Use the included_mountpoints array derived from the disklayout.conf to determine the default # TSM filespaces to include in a restore.
is perfectly o.k. to have a reasonable default behaviour.
But I fail to find where the user can specify what he wants to have
if the default behaviour is not the right one for his needs.
From my current (total TSM noob) point of view
the whole TSM issue could be that it is missing
how the user can specify what TSM should restore
(and perhaps also what TSM should save in its backup).
jsmeix commented at 2016-04-28 13:44:¶
Right now I noticed in https://github.com/rear/rear/issues/821#issuecomment-215369140
We are using workarounds with the TSM backup already ... Without them, the critical filesystems like '/var' are not even backed up.
This means TSM does not support things like
BACKUP_PROG_INCLUDE=( '/var/spool/*' '/var/log/*' )
in /etc/rear/local.conf which is no big surprise
because default.conf reads:
# These settings apply to all cases of internal Relax-and-Recover backup ... BACKUP_PROG_INCLUDE=( )
But I wonder what the right way is how the user can specify
what to include in the TSM backup if the default behaviour
is not what he actually needs?
pavoldomin commented at 2016-04-28 13:59:¶
That the case, from what I understand. TSM takes everything it includes/excludes simply from ^fs entry in the disklayout and ignores whatever backup options we define (I defined them, because I wont to have more or less same config everywhere).
Thats why I assumed that TSM code needs some redesign.
NETFS restore is working fine when I cut subvols from
EXCLUDE_RECREATE.
jsmeix commented at 2016-04-28 14:31:¶
@pavoldomin
many thanks for your testing that it works with tar/NETFS.
I will do a pull request that fixes my btrfs example config files.
Regarding the TSM issue I submitted a new separated
issue
https://github.com/rear/rear/issues/823
to
"enhance TSM support to specify what to backup and restore"
jsmeix commented at 2016-04-29 09:09:¶
With
https://github.com/rear/rear/pull/824
this issue here is solved
at least for the tar/NETFS backup method.
Regarding TSM there is https://github.com/rear/rear/issues/823
jsmeix commented at 2016-05-12 11:56:¶
https://github.com/rear/rear/pull/833 mitigates the problem where the btrfs subvlums are not restored by default via TSM.
Now by default both 'fs' and 'btrfsmountedsubvol' entries
in disklayout.conf are used to generate the default TSM
filespaces to include in a restore.
This mitigates this issue here because now a better default is
used but this does not yet implement
https://github.com/rear/rear/issues/823
because the user still cannot explicitly specify what to backup
and restore with TSM when the default behaviour is not
what he actually needs.
@pavoldomin
I would appreciate it very much if you could test
whether or not the new default behaviour
with
https://github.com/rear/rear/pull/833
is already
sufficient for your needs.
pavoldomin commented at 2016-05-12 12:24:¶
Hi Johannes
I should have another opportunity to test soon (within couple of weeks I hope). Of course I'll inform you when done.
Thanks for this commit!
Kind regards
Pavol
jsmeix commented at 2016-05-12 13:34:¶
For clarification:
It was @Joeri-MS who did the work in https://github.com/rear/rear/pull/833
I only merged his pull request.
jsmeix commented at 2016-05-13 08:15:¶
Regarding btrfs snapshot subvolumes:
Note that in
https://github.com/rear/rear/pull/833/commits/0f622cadcf61df6b4f947a237b23751b55e9e5f8
btrfs snapshot subvolumes are excluded
from the default TSM filespaces
in a SUSE-specific way via
... | grep -v "/.snapshots"
to exclude from disklayout.conf entries like
btrfsmountedsubvol /dev/sda2 /.snapshots rw,relatime,space_cache,subvolid=258,subvol=/@/.snapshots @/.snapshots
where "/.snapshots" is the SUSE default mountpoint
for all the SUSE default btrfs snapshot subvolumes
(i.e. what SUSE's "snapper" tool does).
This does not work on systems that have
btrfs snapshot subvolumes elsewhere.
Perhaps how the default TSM filespaces are created
could be further enhanced regarding exclusion of
any btrfs snapshot subvolumes by using the
"btrfssnapshotsubvol" entries in disklayout.conf:
# There is no recovery of btrfs snapshot subvolumes. # Format: btrfssnapshotsubvol <device> <mountpoint> <btrfs_subvolume_ID> <btrfs_subvolume_path> #btrfssnapshotsubvol /dev/sda2 / 259 @/.snapshots/1/snapshot #btrfssnapshotsubvol /dev/sda2 / 281 @/.snapshots/2/snapshot
but therein the <mountpoint> is always '/'
(i.e. not the mountpoint directory as is looks
from within SUSE's mounted btrfs structure)
which is shown in the "btrfsmountedsubvol"
entries in disklayout.conf:
# All mounted btrfs subvolumes (including mounted btrfs default subvolumes and mounted btrfs snapshot subvolumes). # Determined by the findmnt command that shows the mounted btrfs_subvolume_path. # Format: btrfsmountedsubvol <device> <subvolume_mountpoint> <mount_options> <btrfs_subvolume_path> btrfsmountedsubvol /dev/sda2 / rw,relatime,space_cache,subvolid=259,subvol=/@/.snapshots/1/snapshot @/.snapshots/1/snapshot btrfsmountedsubvol /dev/sda2 /.snapshots rw,relatime,space_cache,subvolid=258,subvol=/@/.snapshots @/.snapshots
jsmeix commented at 2016-05-13 08:21:¶
I got confused
(as usual when I deal with SUSE's default btrfs structure):
What is mounted at the directory "/.snapshots"
is not a btrfs snapshot subvolume
but a "normal" btrfs subvolume,
see disklayout.conf
# Btrfs normal subvolumes for /dev/sda2 at / # Format: btrfsnormalsubvol <device> <mountpoint> <btrfs_subvolume_ID> <btrfs_subvolume_path> btrfsnormalsubvol /dev/sda2 / 257 @ btrfsnormalsubvol /dev/sda2 / 258 @/.snapshots
[Export of Github issue for rear/rear.]