#2215 Issue closed: Exclude temporary/external drives and mountpoints from checklayout and mkrescue's disklayout

Labels: support / question, fixed / solved / done

adatum opened issue at 2019-08-14 03:09:

  • Relax-and-Recover 2.5-git.3350.aa82834d.unknown / 2019-05-10

  • Fedora 30

  • ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
    PROGS=( "${PROGS[@]}" /home/test/Downloads/borg-linux64 locale )

  • PC, x86_64

  • UEFI, GRUB

  • local SSD

  • Description of the issue (ideally so that others can reproduce it):

Currently, temporarily mounted external USB drives and HDDs will be detected as changes by checklayout, and incorporated into the disk layout made by mkrescue. Apparently this is by design according to https://github.com/rear/rear/issues/2207.

This is a problem because the recovery media created will be unpredictable and depend on what temporary devices may be present at the moment mkrescue is run. Their inclusion in mkrescue's disk layout will be an unpleasant surprise at recovery and far from "relaxing."

Is it currently possible to exclude such devices that are not part of the permanent system? For example, something like Borg's --one-file-system or ReaR's EXCLUDE_MOUNTPOINTS for backups?

jsmeix commented at 2019-08-14 11:44:

@adatum
I have no idea how to reliably and in a fail-safe way distinguish in an
automated way what is temporarily mounted from what is permanantly
mounted and how to distinguish an external disk from a built-in disk.

I don't know what "temporarily mounted" should mean from a
computer's internal point of view (in contrast to the user who knows it).

I know about server hardware that has a USB disk built in on the motherboard
(which can even get into one's way when that device suddenly becomes /dev/sda
and you cannot just unplug that unwanted device because it is built-in).

Because I am currently and for some more weeks not in the office
I cannot try out and test how current ReaR actually behaves but
what you describe has never been an issue before (as far as I can
remember) so that I think how current ReaR behaves is o.k. for
how it was and is used by our current users.

As far as I see from plain looking at the code in
https://github.com/rear/rear/blob/master/usr/share/rear/lib/checklayout-workflow.sh
and in particular in
https://github.com/rear/rear/blob/master/usr/share/rear/layout/compare/default/500_compare_layout.sh
what happens is that a new temporary disklayout.conf file is created
where its active (i.e. non-commented) entries are compared with
those of the last stored "official" disklayout.conf file.

So all the usual ways how to exclude disk layout components
should work to avoid that unwanted mounted stuff gets included
in disklayout.conf.

See in particular the section "Including/Excluding components" in
https://github.com/rear/rear/blob/master/doc/user-guide/06-layout-configuration.adoc

adatum commented at 2019-08-15 22:11:

@jsmeix

Thank you so much for taking the time to write such detailed explanations.

I can appreciate that it is difficult, maybe even impossible, to tell programmatically what the user considers temporary or undesirable for inclusion in the disklayout.conf.

Thanks for the links. The section on "Manual Excludes" is almost perfectly what I was asking about, specifically something like:

EXCLUDE_RECREATE=( "${EXCLUDE_RECREATE[@]}" "fs:/run/media" )

The problem is that it seems to exclude only exact paths (eg. fs:/run/media/[USER]/[DISK_LABEL]), and not everything inside /run/media. It also does not expand globs like fs:/run/media/*.

Maybe I'm missing something?

jsmeix commented at 2019-08-16 08:45:

Offhandedly I think something like wildcards to exclude things
is currently not supported by ReaR.

But all config files (in particular etc/rear/local.conf) are sourced by the
usr/sbin/rear main script which means the config files are read and
executed as all the other ReaR scripts so that you can do any bash
programming you like also in ReaR config files.
In particular you could do whathever works for your particular use case
to determine automatically the currently right values for what you need
to exclude.
Simply put: Replace the hardcoded 'fs:/run/media/some_user/some_disk_label'
value with code that autodetects it at runtime.

adatum commented at 2019-08-18 06:19:

Thanks for the reminder. That the config files are sourced as bash scripts gives a lot of flexibility.

I am quite wary of adding my own code due to code quality/maintainability and possible side effects. Nevertheless, this seems to give me the behavior I want:

EXCLUDE_RECREATE=( "${EXCLUDE_RECREATE[@]}" $( EXCLUDE_RECREATE_MOUNTPOINTS="/run/media|/mnt"; if [ -n "$EXCLUDE_RECREATE_MOUNTPOINTS" ]; then grep -Eo "(^|\s)($EXCLUDE_RECREATE_MOUNTPOINTS)\S*" /proc/mounts | sed -r 's/\s(.*)/ fs:\1 /' | tr -d '\n'; fi ) )

Breaking it down:

  • EXCLUDE_RECREATE_MOUNTPOINTS

    • Seems to be a unique variable name (searched the GitHub repo for this string).
    • The base mount points to exclude must all be included in one set of quotes, with no spaces, and separated by |.
    • Right now I made everything inline, but for clarity this variable should probably be placed outside on a line of its own.

  • if [ -n "$EXCLUDE_RECREATE_MOUNTPOINTS" ]

    • Run the logic only if the exclusion string is not empty.

  • grep -Eo "(^|\s)($EXCLUDE_RECREATE_MOUNTPOINTS)\S*" /proc/mounts

    • Look at /proc/mounts for the definitive list of mounts.
    • Use extended regular expressions -E to support parentheses.
    • -o: output only the matched portion of the string instead of the entire line.
    • ($EXCLUDE_RECREATE_MOUNTPOINTS)\S*: match from the mount prefix until a whitespace character to capture all full paths.
    • (^|\s): start matching from whitespace as our starting delimiter (to make sure the exclusion prefix is the base and not a subdirectory, i.e. avoid capturing /somedirectory/prefix), or start of the line (to catch potential matches at the very beginning of the line/file without preceding whitespace).

  • sed -r 's/\s(.*)/ fs:\1 /'

    • Use sed to add the surrounding text necessary for EXCLUDE_RECREATE
    • \s(.*): capture everything except the initial whitespace. Could this be a problem if the match was at the start of the line with no preceding whitespace? This is unlikely for this purpose since the desired match is always on the second column of /proc/mounts, but it would be nice to have robust logic nonetheless.
    • / fs:\1 /: prepend fs: and add a space before and after the mount point. Not sure if quotes are also needed, but it seems to work as is.

  • tr -d '\n': delete newlines to have everything on one line to be included in the EXCLUDE_RECREATE array.

Any comments about potential issues or corrections are appreciated.

adatum commented at 2019-09-13 02:41:

As discovered in #2229, the AUTOEXCLUDE_PATH variable addresses the request in this issue. It excludes not just exact paths but everything below the specified paths as well.

However, wildcards still seem not to be supported. The solution in the previous post does exclude partial matches for directories. i.e. specifying /dir will exclude /dir1, /dir2, etc.

jsmeix commented at 2019-09-13 14:47:

@adatum
thanks to mention AUTOEXCLUDE_PATH.
Right now I did https://github.com/rear/rear/issues/2239

In general see also
https://github.com/rear/rear/issues/2229#issuecomment-531264805

Currently - as far as I see - there is no simple clear and consistently working way
how the user could specify what disk layout components he wants to get recreated
and what mountpoints or directories he wants to get included in the backup, cf.
https://github.com/rear/rear/issues/2236#issuecomment-531204474

jsmeix commented at 2020-06-24 10:56:

I think with https://github.com/rear/rear/issues/2239 merged
this issue is fixed to some acceptable extent so that I can close this issue
and we still have https://github.com/rear/rear/issues/2229 open
for the more general case.

[Export of Github issue for rear/rear.]