#3478 Issue `closed`: umount fails because target is in use by realtime antivirus scanner¶

Labels: support / question, fixed / solved / done, external tool, not ReaR / invalid

damm620 opened issue at 2025-06-04 10:10:¶

ReaR version¶

Relax-and-Recover 2.7 / Git

Describe the ReaR bug in detail¶

Hi,

executing this on efi servers resolves in this error:

sudo TMPDIR=/tmp /usr/sbin/rear mkrescue
Failed to copy all contents of /lib/modules/5.14.0-570.el9.x86_64 (dangling symlinks could be a reason)
/usr/lib64/systemd/libsystemd-core-252.so requires additional libraries
        libsystemd-shared-252.so => not found
ReaR recovery system in '/tmp/rear.qkdqeKaa0Ccz6BZ/rootfs' needs additional libraries, check /var/log/rear/rear-exunxlvj19302.log for details
Failed to 'rm -Rf --one-file-system /tmp/rear.qkdqeKaa0Ccz6BZ/tmp'
Could not remove build area /tmp/rear.qkdqeKaa0Ccz6BZ (something still exists therein)
Something is still mounted within the build area
  /tmp/rear.qkdqeKaa0Ccz6BZ/tmp/isofs/boot/efiboot.img (deleted) on /tmp/rear.qkdqeKaa0Ccz6BZ/tmp/efi_virt type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
You must manually umount it, then you could manually remove the build area
To manually remove the build area use (with caution): rm -Rf --one-file-system /tmp/rear.qkdqeKaa0Ccz6BZ

I cannot remove it myself:

rm -Rf --one-file-system /tmp/rear.qkdqeKaa0Ccz6BZ/tmp
rm: cannot remove '/tmp/rear.qkdqeKaa0Ccz6BZ/tmp': Permission denied


sudo rm -Rf --one-file-system /tmp/rear.qkdqeKaa0Ccz6BZ/tmp
rm: skipping '/tmp/rear.qkdqeKaa0Ccz6BZ/tmp/efi_virt', since it's on a different device

But I umount /tmp/rear.qkdqeKaa0Ccz6BZ/tmp/efi_virt.

In my opinion these lines in /usr/share/rear/lib/_input-output-functions.sh are responsable for this:

function cleanup_build_area_and_end_program () {
    # Cleanup build area
    local mounted_in_BUILD_DIR
    Log "Finished $PROGRAM $WORKFLOW in $(( $( date +%s ) - START_SECONDS )) seconds"
    # is_true is in lib/global-functions.sh which is not yet sourced in case of early Error() in usr/sbin/rear
    if has_binary is_true && is_true "$KEEP_BUILD_DIR" ; then
        mounted_in_BUILD_DIR="$( mount | grep "$BUILD_DIR" | sed -e 's/^/  /' )"
        if test "$mounted_in_BUILD_DIR" ; then
            LogPrintError "Caution - there is something mounted within the build area"
            LogPrintError "$mounted_in_BUILD_DIR"
            LogPrintError "You must manually umount that before you may remove the build area"
        fi
        # Show this message also inside the recovery system (e.g. at the end of "rear -D recover")
        # because there may be a reason why manually removing the build area is wanted
        # (e.g. some additional manual things need be done before rebooting).
        # In any case one must be careful if one wants to remove the build area because
        # e.g. the NFS share with the backup.tar.gz may still be erroneously mounted therein.
        LogPrint "To remove the build area you may use (with caution): rm -Rf --one-file-system $BUILD_DIR"
    else
        Log "Removing build area $BUILD_DIR"
        # Use '--one-file-system' to be safe against also deleting by accident
        # all mounted things below mountpoints in TMP_DIR or ROOTFS_DIR
        # (regardless if mountpoints in TMP_DIR or ROOTFS_DIR may happen):
        rm -Rf --one-file-system $TMP_DIR || LogPrintError "Failed to 'rm -Rf --one-file-system $TMP_DIR'"
        rm -Rf --one-file-system $ROOTFS_DIR || LogPrintError "Failed to 'rm -Rf --one-file-system $ROOTFS_DIR'"
        # Before removing BUILD_DIR check that outputfs is gone (i.e. check that nothing is mounted there):
        if mountpoint -q "$BUILD_DIR/outputfs" ; then
            # If still mounted wait a bit (perhaps some ongoing umount needs more time) then try lazy umount:
            sleep 2
            # umount_mountpoint_lazy is in lib/global-functions.sh
            # which is not yet sourced in case of early Error() in usr/sbin/rear
            has_binary umount_mountpoint_lazy && umount_mountpoint_lazy $BUILD_DIR/outputfs
        fi
        # remove_temporary_mountpoint is in lib/global-functions.sh
        # which is not yet sourced in case of early Error() in usr/sbin/rear
        if has_binary remove_temporary_mountpoint ; then
            # It is a bug in ReaR if BUILD_DIR/outputfs was not properly umounted and made empty by the scripts before:
            remove_temporary_mountpoint "$BUILD_DIR/outputfs" || BugError "Directory $BUILD_DIR/outputfs not empty, cannot remove"
        fi
        if ! rmdir $v "$BUILD_DIR" ; then
            LogPrintError "Could not remove build area $BUILD_DIR (something still exists therein)"
            mounted_in_BUILD_DIR="$( mount | grep "$BUILD_DIR" | sed -e 's/^/  /' )"
            if test "$mounted_in_BUILD_DIR" ; then
                LogPrintError "Something is still mounted within the build area"
                LogPrintError "$mounted_in_BUILD_DIR"
                LogPrintError "You must manually umount it, then you could manually remove the build area"
            fi
            LogPrintError "To manually remove the build area use (with caution): rm -Rf --one-file-system $BUILD_DIR"
        fi
    fi
    Log "End of program '$PROGRAM' reached"
}

It seems that the code is trying to remove the mount without trying to umount it first?
Did I miss here something or is this a bug?

Platform¶

Linux x64

OS version¶

SUSE Linux Enterprise Server 15 SP6

Backup¶

TSM

Storage layout¶

No response

What steps will reproduce the bug?¶

No response

Workaround, if any¶

No response

Additional information¶

No response

jsmeix commented at 2025-06-04 11:16:¶

Because of

/tmp/rear.qkdqeKaa0Ccz6BZ/tmp/isofs/boot/efiboot.img (deleted) on /tmp/rear.qkdqeKaa0Ccz6BZ/tmp/efi_virt type vfat

this one is likely a duplicate of
https://github.com/rear/rear/issues/2908
which I fixed first by
https://github.com/rear/rear/pull/2909
only for OUTPUT=ISO
and meanwhile both for OUTPUT=ISO and OUTPUT=USB by
https://github.com/rear/rear/pull/3408

ReaR 2.7 does not contain those fixes.

The issue is no bug in ReaR because nowadays
a simple and straightforward sequence of commands like

# mount /dev/something /mountpoint
# cp /some/file /mountpoint
# umount /mountpoint

does sometimes no longer work reliably because nowadays
umount may fail in such cases with "target is busy", cf.
https://github.com/rear/rear/issues/2908#issuecomment-1378811748

Via
https://github.com/rear/rear/pull/2909
and
https://github.com/rear/rear/pull/3408
and
https://github.com/rear/rear/pull/3447
we at ReaR upstream try to make ReaR somehow deal
with nowadays more and more unpredictable behaviour
of low level basic system stuff, cf.
https://github.com/rear/rear/wiki/Coding-Style#dirty-hacks-welcome

The correct fix would be that basic things like

# mount /dev/something /mountpoint
# cp /some/file /mountpoint
# umount /mountpoint

work reliable.

My blind guess why it nowadays sometimes fails
is something in the kernel area.
Perhaps related to the VFAT filesystem because
as far as I noticed from user reports here at ReaR upstream
umount only fails when a VFAT filesystem is mounted.

damm620 commented at 2025-06-04 11:24:¶

thanks for the references to the fixes!

jsmeix commented at 2025-06-04 12:10:¶

@damm620
I think the simplest "fix" in ReaR 2.7
is when you add in
output/ISO/Linux-i386/700_create_efibootimg.sh
something like sleep 3 or sleep 1
before the 'umount' line so that it looks like

# Copy files from staging directory into efiboot.img
cp $v -r $TMP_DIR/mnt/. $TMP_DIR/efi_virt
sleep 3
umount $v $TMP_DIR/efiboot.img

https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/output/ISO/Linux-i386/700_create_efibootimg.sh#L46

Cf.
https://github.com/rear/rear/issues/3397#issuecomment-2656838638
where sleep helped in the OUTPUT=USB case.

In this issue see also
https://github.com/rear/rear/issues/3397#issuecomment-2685552225

I wrote "fix" above because obviously sleep cannot
actually fix the root cause - but when sleep helps
to avoid the issue sufficiently well in practice,
then such a band-aid is a "fix" to a certain extent.

damm620 commented at 2025-06-04 13:11:¶

Update:
We found out, that our Realtime Antivirus Scanner interfered here with the umount, so that the "target was busy".
We are now thinking about a umount-loop in usr/share/rear/output/ISO/Linux-i386/700_create_efibootimg.sh to try umounting it until it is not busy.

jsmeix commented at 2025-06-05 07:07:¶

@damm620

thank you for your interesting information!

Perhaps there is no bug at all and umount, kernel, VFAT filesystem,...
all work as they should and all the cases where that umount failed
in ReaR behave correctly.

It is important for us at ReaR upstream to better understand
such issues as a whole i.e. including the whole environment.

For example when there is a Realtime Antivirus Scanner running
it is its job to do continuous realtime scanning so mounted
things will get scanned and they have to be scanned because
the admin of the system explicitly wanted it to be this way.

So ReaR has no right for example to kill a non-ReaR process
which got in its way, e.g. some realtime scanning process,
to make ReaR's umount command succeed by brute force,
because then ReaR would work against what the admin
of the system had explicitly set up.
Cf. my
https://github.com/rear/rear/pull/3447#issuecomment-2789658854
and
https://github.com/rear/rear/pull/3447#discussion_r2077472659

See also
https://github.com/rear/rear/issues/3410
where CrowdStrike Falcon (used as AV solution) caused issues in ReaR
where I wrote in
https://github.com/rear/rear/issues/3410#issuecomment-2690634795

we should not even try to work around
or against CrowdStrike Falcon in ReaR

When the admin of a system has set up things which cause issues in ReaR
then it is first and foremost the admin's task to find a solution
but it is not ReaR's task to do "whatever it takes" in the admin's system
to enforce that ReaR succeeds.

jsmeix commented at 2025-06-05 08:02:¶

Regarding a umount-loop to umount it when is no longer busy:

This would delay ReaR until it is no longer busy,
unless the umount-loop is implemented as background process
but then the umount-loop background process would get SIGKILL
when ReaR finishes so it may not get umounted.

I think a lazy umount is better because (as far as I understand things)
a lazy umount is pending as long as needed until is no longer busy.

damm620 commented at 2025-06-05 08:07:¶

Regarding a umount-loop to umount it when is no longer busy:

This would delay ReaR until it is no longer busy, unless the umount-loop is implemented as background process but then the umount-loop background process would get SIGKILL when ReaR finishes so it may not get umounted.

I think a lazy umount is better because (as far as I understand things) a lazy umount is pending as long as needed until is no longer busy.

Well a umount -l would wait endlessly (or a very long time to say the least) as the umount loop would try to umount and wait 1 second and this 10 times. So that will "only" delay 10 seconds max.
Of cause this is not feasable for upstream, but in our environemnt this does make a lot of sence in my opinion.

Thanks for your help here :)

jsmeix commented at 2025-06-05 11:43:¶

Ah! Now I do better understand.

So the "umount-loop ... try umounting it until it is not busy"
which loops unlimited "until it is not busy" changed into a
limited amount of retries to umount it.

But what happens when the last umount retry still fails?

Is the issue then ignored (i.e. it will not get umounted)?

Or will then ReaR (because that loop is run by ReaR)
"get angry" and do "whatever it takes" to get that thingy
(i.e. what keeps the mount target in use) out of the way
by more and more forceful strikes against that thingy,
for example strikes like

SIGTERM to non-ReaR processes
SIGKILL to non-ReaR processes
system reboot ;-)
whatnot... ?

When a program runs as 'root'
those who make that program have basically unlimited power
what happens on the system of the user who runs their program
so for those who make such a program this is mandatory:
"With great power comes great responsibility"
https://en.wikipedia.org/wiki/With_great_power_comes_great_responsibility

Ob the other hand:
A system admin (i.e. a user who can log in as 'root') can do
whatever he thinks is appropriate in his specific case
on his particular system.
Therefore ReaR is intentionally written in the native language
for system administration as shell (bash) scripts because
ReaR is meant to be adapted and/or extended by the admin
as needed to make things work for his specific use case.

damm620 commented at 2025-06-05 12:08:¶

If the umount loop still cannot umount the mount after 10 trys or so it will continue with the "rest of the rear workflow". That will result in the error when ReaR ends, but thats okay. Because when umounting it 10 times does not work, there should be a admin present checking on what going on i think.

jsmeix commented at 2025-06-05 13:00:¶

Yes, this looks very reasonable
(at least for your particular use case).

Perhaps when the umount loop cannot umount
do a lazy umount call as last thing and then
continue with the "rest of the rear workflow"
because this way it could umount "by the way"
while the "rest of the rear workflow" is running
so when ReaR ends it could be already umounted
and the error at ReaR's end could be avoided.

I.e. when in your particular case
your Realtime Antivirus Scanner finished
while the "rest of the rear workflow" is running
then a pending lazy umount should umount it
so it could be umounted before ReaR ends.

[Export of Github issue for rear/rear.]

#3478 Issue closed: umount fails because target is in use by realtime antivirus scanner¶

damm620 opened issue at 2025-06-04 10:10:¶

ReaR version¶

Describe the ReaR bug in detail¶

Platform¶

OS version¶

Backup¶

Storage layout¶

What steps will reproduce the bug?¶

Workaround, if any¶

Additional information¶

jsmeix commented at 2025-06-04 11:16:¶

damm620 commented at 2025-06-04 11:24:¶

jsmeix commented at 2025-06-04 12:10:¶

damm620 commented at 2025-06-04 13:11:¶

jsmeix commented at 2025-06-05 07:07:¶

jsmeix commented at 2025-06-05 08:02:¶

damm620 commented at 2025-06-05 08:07:¶

jsmeix commented at 2025-06-05 11:43:¶

damm620 commented at 2025-06-05 12:08:¶

jsmeix commented at 2025-06-05 13:00:¶

#3478 Issue `closed`: umount fails because target is in use by realtime antivirus scanner¶