#3168 PR merged: Set TMPDIR to ReaR's TMP_DIR=$BUILD_DIR/tmp for programs that are called by ReaR

Labels: enhancement, fixed / solved / done

jsmeix opened issue at 2024-02-28 15:36:

  • Type: Enhancement

  • Impact: High
    Hopefully a high positive impact
    because of improved default confidentiality and
    because of improved careful and complete cleanup of temporary leftovers

  • Reference to related issue (URL):
    https://github.com/rear/rear/issues/3167

  • How was this pull request tested?
    A simple "rear mkbackup"
    and "rear recover" work for me.

  • Description of the changes in this pull request:

In usr/sbin/rear set TMPDIR to TMP_DIR (within BUILD_DIR)
to ensure
that also for programs that are called by ReaR
ReaR will clean up all temporary stuff carefully and completely
via the function cleanup_build_area_and_end_program()
AND
that temporary files from programs that are called by ReaR
are sufficiently safe against unwanted access by non-root users
via 'mktemp -d' where BUILD_DIR and implicitly TMP_DIR=$BUILD_DIR/tmp
has permissions only for root but none for the group or others

jsmeix commented at 2024-02-28 15:45:

My simple test of "rear mkbackup" with

OUTPUT=ISO
BACKUP=NETFS
BACKUP_URL=file:///other/
BACKUP_ONLY_INCLUDE="yes"
BACKUP_PROG_INCLUDE=( /home/johannes )

# mount | grep other
/dev/nvme0n1p4 on /other type ext4 (rw,relatime,stripe=32)

For testing I added at the beginning of
init/default/001_verify_config_arrays.sh

tmpdir="$( mktemp -d )"
echo tmpdata >$tmpdir/tmpfile
DebugPrint "$( find $tmpdir -ls )"

Before the changes in this pull request:

# usr/sbin/rear -D mkrescue
...
Running 'init' stage ======================
 14419482 4 drwx------ 2 root root 4096 Feb 28 13:47 /var/tmp/tmp.FWlUGBmzz8
 14419484 4 -rw-r--r-- 1 root root    8 Feb 28 13:47 /var/tmp/tmp.FWlUGBmzz8/tmpfile
...
To remove the build area you may use (with caution): rm -Rf --one-file-system /var/tmp/rear.Bw6o1RnUaVZCc5M

# rm -Rf --one-file-system /var/tmp/rear.Bw6o1RnUaVZCc5M
# echo $?
0
# find /var/tmp/rear.Bw6o1RnUaVZCc5M -ls
find: ‘/var/tmp/rear.Bw6o1RnUaVZCc5M’: No such file or directory

# find /var/tmp/tmp.FWlUGBmzz8 -ls
 14419482 4 drwx------ 2 root root 4096 Feb 28 13:47 /var/tmp/tmp.FWlUGBmzz8
 14419484 4 -rw-r--r-- 1 root root    8 Feb 28 13:47 /var/tmp/tmp.FWlUGBmzz8/tmpfile

With the changes in this pull request:

# usr/sbin/rear -D mkbackup
...
Running 'init' stage ======================
 14811425 4 drwx------ 2 root root 4096 Feb 28 16:16 /var/tmp/rear.7rxBQsZQKFf6f5e/tmp/tmp.esJeyHDyzM
 14811426 4 -rw-r--r-- 1 root root    8 Feb 28 16:16 /var/tmp/rear.7rxBQsZQKFf6f5e/tmp/tmp.esJeyHDyzM/tmpfile
...
To remove the build area you may use (with caution): rm -Rf --one-file-system /var/tmp/rear.7rxBQsZQKFf6f5e

# rm -Rf --one-file-system /var/tmp/rear.7rxBQsZQKFf6f5e
# echo $?
0
# find /var/tmp/rear.7rxBQsZQKFf6f5e -ls
find: ‘/var/tmp/rear.7rxBQsZQKFf6f5e’: No such file or directory

jsmeix commented at 2024-03-06 15:20:

I did "rear mkbackup" and "rear recover" with

OUTPUT=ISO
BACKUP=NETFS
BACKUP_URL=nfs://192.168.178.66/nfs

which "just worked" normally for me.

I did the same artificial change at the beginning of
init/default/001_verify_config_arrays.sh
as above, i.e.

tmpdir="$( mktemp -d )"
echo tmpdata >$tmpdir/tmpfile
DebugPrint "$( find $tmpdir -ls )"

and got

RESCUE localhost:~ # rear -D recover
...
Running 'init' stage ======================
29774 0 drwx------ 2 root root 0 Mar 6 16:14 /var/tmp/rear.PrD9Rdyf07gHSxT/tmp/tmp.N7hW9sdA75
29775 4 -rw-r--r-- 1 root root 8 Mar 6 16:14 /var/tmp/rear.PrD9Rdyf07gHSxT/tmp/tmp.N7hW9sdA75/tmpfile
...
Running exit tasks
To remove the build area you may use (with caution): rm -Rf --one-file-system /var/tmp/rear.PrD9Rdyf07gHSxT

The recreated system boots normally and
works normally (as far as I see at first glance).

jsmeix commented at 2024-03-06 15:23:

@rear/contributors @pcahyna
could you please review it (as time permits),
perhaps you see some obvious mistake.

If there are no objections
I would like to merge it on Friday afternoon
so users who use our GitHub master code could test it and
report if there are issues in this or that circumstances.

pcahyna commented at 2024-03-07 11:00:

Something is wrong in the CI runs:

2024-02-28 16:53:54.526692213 WARNING:
                              Failed to create initrd for kernel version '5.14.0-419.el9.x86_64'.
                              Check '/var/log/rear/rear-ip-172-31-29-183.log' to see the error messages in detail
                              and decide yourself, whether the system will boot or not.
2024-02-28 16:53:54.555617047 Running dracut...
2024-02-28 16:53:54.585746511 WARNING:
                              Failed to create initrd for kernel version '5.14.0-425.el9.x86_64'.
                              Check '/var/log/rear/rear-ip-172-31-29-183.log' to see the error messages in detail
                              and decide yourself, whether the system will boot or not.

pcahyna commented at 2024-03-07 11:23:

I don't know what's wrong, but I suspect that the TMPDIR setting is used also in the rescue system and exported to programs called in the /mnt/local chroot, and the /var/tmp/rearXXXXXX directory does not exist in the chroot (unlike the former /var/tmp value that had the advantage of existing both in the recovery ramdisk and in the recovered system), and some programs called in the chroot do not like an invalid TMPDIR value.
This is purely my speculation. Now confirmed by error messages in @jsmeix's rear -D recover run.

pcahyna commented at 2024-03-07 11:25:

@jsmeix do you know where to find the error messages from dracut, if any? I don't see them in the recovery log: https://artifacts.dev.testing-farm.io/659057b7-cf20-4bf1-9519-3768cad95c92/work-backup-and-restoreom9c6wvr/tests/plans/backup-and-restore/execute/data/guest/default-0/make-backup-and-restore-iso-1/data/rear-recover.log

pcahyna commented at 2024-03-07 11:27:

The recreated system boots normally and
works normally (as far as I see at first glance).

Do you see the dracut warnings in the recovery log? Or maybe the distribution you are testing on does not use dracut?

gdha commented at 2024-03-07 12:01:

@pcahyna When inspecting https://artifacts.dev.testing-farm.io/5f65a14d-eba9-467a-b03e-24c667b17f9c/work-backup-and-restoreu9mahx67/tests/plans/backup-and-restore/execute/data/guest/default-0/make-backup-and-restore-iso-1/output.txt
I do see the following:

:: [ 16:48:38 ] :: [   PASS   ] :: Command 'export TMPDIR='/var/tmp'' (Expected 0, got 0)
...
2024-02-28 16:51:39.691572435 Running dracut...
2024-02-28 16:51:39.711368631 WARNING:
                              Failed to create initrd for kernel version '6.8.0-0.rc5.41.fc41.x86_64'.
                              Check '/var/log/rear/rear-ip-172-31-30-152.log' to see the error messages in detail
                              and decide yourself, whether the system will boot or not.

Therefore, IMHO TMPDIR was set to /var/tmp
Perhaps, add a step to copy the /var/log/rear/rear-ip-172-31-30-152.log as well?

jsmeix commented at 2024-03-07 12:09:

Sigh!
I have that dracut error too.
Somehow I must have overlooked that yesterday.
Today I see it clearly:

RESCUE localhost:~ # rear -D recover
...
Using build area: /var/tmp/rear.bClgGxIS12HNrGx
...
Warning:
Failed to recreate initrd with /usr/bin/dracut.
Check '/var/log/rear/rear-localhost.log' why /usr/bin/dracut failed
and decide if the recreated system will boot
with the initrd 'as is' from the backup restore.

In ReaR debug mode my /var/log/rear/rear-localhost.log
shows why /usr/bin/dracut failed

++ chroot /mnt/local /bin/bash -c 'PATH=/sbin:/usr/sbin:/usr/bin:/bin /usr/bin/dracut --force'
realpath: /var/tmp/rear.bClgGxIS12HNrGx/tmp: No such file or directory
dracut: Invalid tmpdir '/var/tmp/rear.bClgGxIS12HNrGx/tmp'.
++ LogPrintError 'Warning:
Failed to recreate initrd with /usr/bin/dracut.
Check '\''/var/log/rear/rear-localhost.log'\'' why /usr/bin/dracut failed
and decide if the recreated system will boot
with the initrd '\''as is'\'' from the backup restore.
'

jsmeix commented at 2024-03-07 12:11:

@pcahyna
thank you so much for your careful review.
So often you detect things.
I do appreciate that so very much!

pcahyna commented at 2024-03-07 12:19:

@jsmeix Thank mainly Anton and Lukas for implementing the CI, especially Lukas for grepping for WARNING and ERROR in the log and reporting a test failure even if the recovery is successful! Please do not ignore test failures especially if they occur on multiple distros.

By the way:
the verification steps at the end of the test script contain:
grep -C 10 -P "$log_prefix WARNING:" $path
so it will stop working if the letter case of the messages is changed.

@gdha

Perhaps, add a step to copy the /var/log/rear/rear-ip-172-31-30-152.log as well?

The steps has been in ReaR by default for a long time! See wrapup/default/990_copy_logfile.sh , the definition of copy_log_file_exit_task. Unfortunately the log does not contain anything more (the message I quoted is from that very same log file), so the message Check '/var/log/rear/rear-ip-172-31-29-183.log' to see the error messages is unhelpful and misleading.

pcahyna commented at 2024-03-07 12:20:

I do see the following:
:: [ 16:48:38 ] :: [ PASS ] :: Command 'export TMPDIR='/var/tmp'' (Expected 0, got 0)
Therefore, IMHO TMPDIR was set to /var/tmp

The script sets it before calling ReaR, but then ReaR changes TMPDIR internally (which is the cause of the problem).

And setting TMPDIR before a rear mkrescue run will not affect rear recover anyway, and it is rear recover that has the problem, not rear mkrescue. The test script does not alter the rear recover run in any way (except for setting some PRE_ / POST_ stuff).

jsmeix commented at 2024-03-07 12:25:

I will have to think some time about it
whether or not there could be a good solution,
i.e. a solution that is reasonably simple
so that we can safely assume it works reasonably fail-safe
and that it will be reasonably future-proof.

What I see now and what I do not like is that
all our chroot /mnt/local COMMANDs
could leave arbitrary temporary stuff behind
in the recreated system.

I wished ReaR could provide some generic means
that all ReaR's temporary files get cleaned up properly
and that ReaR's temporary files are sufficiently safe
against unwanted access.

jsmeix commented at 2024-03-07 12:30:

By the way
regarding WARNING: versus Warning: see
https://github.com/rear/rear/pull/3153#issuecomment-1952301713

jsmeix commented at 2024-03-07 12:38:

Regarding
"Please do not ignore test failures":
I am sorry for that.
Obviously I did not have a good day yesterday.
Yesterday there were various different things in parallel
so apparently my brain had some kind of "overflow blackout".

jsmeix commented at 2024-03-07 12:50:

There is no dracut error message in ReaR's log file
unless ReaR runs in debug mode, cf.
https://github.com/rear/rear/issues/2416 and
https://github.com/rear/rear/pull/2633

jsmeix commented at 2024-03-11 17:06:

Via
https://github.com/rear/rear/pull/3168/commits/4e78ab50f9805a9f35935783646726f283b9a9d3
I set TMPDIR to TMP_DIR only when we are not in RECOVERY_MODE
as a currently not yet testet attempt to avoid
https://github.com/rear/rear/pull/3168#issuecomment-1983377528

My reasoning:

For programs that run within the ReaR recovery system
it should not matter where TMPDIR is or whether or not
some temporary files of called programs are left behind
because the ReaR recovery system is on a ramdisk that
gets completely removed by 'reboot' or 'poweroff'
so nothing of the ReaR recovery system is left behind.

For programs that run within the recreated/restored system via

chroot /mnt/local COMMAND

exporting TMPDIR to ReaR's TMP_DIR cannot work because
there cannot be a TMP_DIR in the recreated/restored system
i.e. there is no /mnt/local/var/tmp/rear.XXXXXXXXXXXXXXX/tmp
because mktemp -d -t rear.XXXXXXXXXXXXXXX is unpredictable
(i.e. unpredictable with sufficiently high probability).

The alternative to create (and remove it at the end)
(e.g. via some initial 'finalize' stage script)
/mnt/local/var/tmp/rear.XXXXXXXXXXXXXXX/tmp
with same /var/tmp/rear.XXXXXXXXXXXXXXX/tmp
as in the ReaR recovery system looks a too dirty hack to me
at least according to my current gut feeling.

pcahyna commented at 2024-03-12 09:58:

The alternative to create (and remove it at the end)
(e.g. via some initial 'finalize' stage script)
/mnt/local/var/tmp/rear.XXXXXXXXXXXXXXX/tmp
with same /var/tmp/rear.XXXXXXXXXXXXXXX/tmp
as in the ReaR recovery system looks a too dirty hack to me
at least according to my current gut feeling.

I actually wanted to suggest this solution, to me it does not look that bad, and helps (the 'remove it at the end' part) with this problem as well:

What I see now and what I do not like is that
all our chroot /mnt/local COMMANDs
could leave arbitrary temporary stuff behind
in the recreated system.

But your proposed solution works as well of course. CI likes it, too.

jsmeix commented at 2024-03-12 10:47:

@pcahyna
after sleeping on it I also think that creating
/mnt/local/var/tmp/rear.XXXXXXXXXXXXXXX/tmp
and removing it at the end is not so bad
as it looked to me yesterday.

But for now I prefer it as it is currently done because
creating /mnt/local/var/tmp/rear.XXXXXXXXXXXXXXX/tmp
could become more tricky than expected:

One reason is that not all workflows that can
run in the recovery system source the 'finalize' stage
so creating /mnt/local/var/tmp/rear.XXXXXXXXXXXXXXX/tmp
via some initial 'finalize' stage script
would be only partially working.

jsmeix commented at 2024-03-12 10:52:

And the nightmare continues:

# find usr/sbin/rear usr/share/rear/ -type f | xargs grep -l 'chroot '

shows in particular
usr/share/rear/build/default/490_fix_broken_links.sh
and
usr/share/rear/build/default/990_verify_rootfs.sh
which run

chroot $ROOTFS_DIR COMMAND

where COMMAND fails when it uses TMPDIR
because there is no

/var/tmp/rear.XXXXXXXXXXXXXXX/tmp

in

/var/tmp/rear.XXXXXXXXXXXXXXX/rootfs

i.e. there is no

/var/tmp/rear.XXXXXXXXXXXXXXX/rootfs/var/tmp/rear.XXXXXXXXXXXXXXX/tmp

As an artificial test I added at the beginning of
usr/share/rear/build/default/990_verify_rootfs.sh

chroot $ROOTFS_DIR /bin/bash -c 'tmpdir="$( mktemp -d )"'

and did

# usr/sbin/rear -D mkrescue
...
Using build area: /var/tmp/rear.ljfRf4UOfVJktrT
...

and got in var/log/rear/rear-localhost.log

+ source /root/rear.github.master.TMPDIR/usr/share/rear/build/default/990_verify_rootfs.sh
++ chroot /var/tmp/rear.ljfRf4UOfVJktrT/rootfs /bin/bash -c 'tmpdir="$( mktemp -d )"'
mktemp: failed to create directory via template '/var/tmp/rear.ljfRf4UOfVJktrT/tmp/tmp.XXXXXXXXXX': No such file or directory

:-(

jsmeix commented at 2024-03-12 12:08:

With
https://github.com/rear/rear/pull/3168/commits/9b4efb2469b8cf3585206dbb10700960b480008e
I create TMPDIR inside ROOTFS_DIR
i.e. I create $ROOTFS_DIR$TMP_DIR which is
/var/tmp/rear.XXX/rootfs/var/tmp/rear.XXX/tmp
so that "chroot $ROOTFS_DIR COMMAND"
will not fail when COMMAND uses TMPDIR.

jsmeix commented at 2024-03-12 12:11:

With latest changes and
as an artificial test added at the beginning of
usr/share/rear/build/default/990_verify_rootfs.sh

chroot $ROOTFS_DIR /bin/bash -c 'tmpdir="$( mktemp -d )" ; echo tmpdata >$tmpdir/tmpfile ; find $tmpdir -ls'

I get

# usr/sbin/rear -D mkrescue
...
Using build area: /var/tmp/rear.vSlVJGxFx9U5HVY
...
Exiting rear mkrescue (PID 8329) and its descendant processes ...

and

# find /var/tmp/rear.vSlVJGxFx9U5HVY/rootfs/var/tmp/rear.vSlVJGxFx9U5HVY/tmp/
/var/tmp/rear.vSlVJGxFx9U5HVY/rootfs/var/tmp/rear.vSlVJGxFx9U5HVY/tmp/
/var/tmp/rear.vSlVJGxFx9U5HVY/rootfs/var/tmp/rear.vSlVJGxFx9U5HVY/tmp/tmp.QgqgDBsfmZ
/var/tmp/rear.vSlVJGxFx9U5HVY/rootfs/var/tmp/rear.vSlVJGxFx9U5HVY/tmp/tmp.QgqgDBsfmZ/tmpfile

and in var/log/rear/rear-localhost.log there is

+ source /root/rear.github.master.TMPDIR/usr/share/rear/build/default/990_verify_rootfs.sh
++ chroot /var/tmp/rear.vSlVJGxFx9U5HVY/rootfs /bin/bash -c 'tmpdir="$( mktemp -d )" ; echo tmpdata >$tmpdir/tmpfile ; find $tmpdir -ls'
 15341688      4 drwx------   2 root     root         4096 Mar 12 12:58 /var/tmp/rear.vSlVJGxFx9U5HVY/tmp/tmp.QgqgDBsfmZ
 15342002      4 -rw-r--r--   1 root     root            8 Mar 12 12:58 /var/tmp/rear.vSlVJGxFx9U5HVY/tmp/tmp.QgqgDBsfmZ/tmpfile

jsmeix commented at 2024-03-12 12:40:

FYI
whereto we chroot in the ReaR scripts

# find usr/sbin/rear usr/share/rear/ -type f -name '*.sh' \
 | xargs grep -h 'chroot [^ ]*' \
 | grep -v '^ *#' \
 | grep -o 'chroot [^ ]*' \
 | sort -u

chroot $ROOTFS_DIR
chroot $TARGET_FS_ROOT
chroot $TARGET_FS_ROOT'
chroot $TARGET_FS_ROOT/
chroot $TARGET_FS_ROOT\ncd

where chroot $TARGET_FS_ROOT'
and chroot $TARGET_FS_ROOT\ncd
are false-positives:

You should 'chroot $TARGET_FS_ROOT' and try to fix this.

in finalize/Debian/i386/550_rebuild_initramfs.sh
and

rear_shell_history="$( echo -e "chroot $TARGET_FS_ROOT\ncd $TARGET_FS_ROOT/etc/\nvi $restored_fstab\nless $restored_fstab" )"

in finalize/default/520_confirm_finalize.sh

jsmeix commented at 2024-03-12 13:56:

On the one hand I like such details problems
because I find it interesting to experience
the truth behind RFC 1925 items 4 and 8 and the
WYSIATI (What you see is all there is) fallacy, cf.
https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow

On the other had I fear such details problems
because it proves that in practice
no matter how hard you try,
you can't make a program that works
(i.e. RFC 1925 item 1 is impossible).

There is always something unrecognized left.

pcahyna commented at 2024-03-13 10:24:

usr/share/rear/build/default/490_fix_broken_links.sh and usr/share/rear/build/default/990_verify_rootfs.sh which run

chroot $ROOTFS_DIR COMMAND

where COMMAND fails when it uses TMPDIR because there is no

/var/tmp/rear.XXXXXXXXXXXXXXX/tmp

in

/var/tmp/rear.XXXXXXXXXXXXXXX/rootfs

i.e. there is no

/var/tmp/rear.XXXXXXXXXXXXXXX/rootfs/var/tmp/rear.XXXXXXXXXXXXXXX/tmp

I would expect this to be also a problem when an user sets and exports TMPDIR to some value like /a/big/and/fast/filesystem before calling ReaR, because /a/big/and/fast/filesystem will not exist either in rootfs, and this problem exists regardless of your changes in this PR. I suspect that to fix all this consistently one needs to add $TMPDIR to COPY_AS_IS and $TMPDIR/* to COPY_AS_IS_EXCLUDE. One needs to do it after TMPDIR is set, though.

pcahyna commented at 2024-03-13 10:56:

With mkdir /run/user/111240/rear; TMPDIR=/run/user/111240/rear usr/sbin/rear -D mkrescue and

an artificial test added at the beginning of usr/share/rear/build/default/990_verify_rootfs.sh

chroot $ROOTFS_DIR /bin/bash -c 'tmpdir="$( mktemp -d )" ; echo tmpdata >$tmpdir/tmpfile ; find $tmpdir -ls'

I get this in the log:

+ source /home/pcahyna/rear/rear/usr/share/rear/build/default/990_verify_rootfs.sh
++ LogPrint 'Testing that the recovery system in /run/user/111240/rear/rear.0K0i19A75LZJHIF/rootfs contains a usable system'
2024-03-13 11:37:29.479931931 Testing that the recovery system in /run/user/111240/rear/rear.0K0i19A75LZJHIF/rootfs contains a
 usable system
++ chroot /run/user/111240/rear/rear.0K0i19A75LZJHIF/rootfs /bin/bash -c 'tmpdir="$( mktemp -d )" ; echo tmpdata >$tmpdir/tmpf
ile ; find $tmpdir -ls'
mktemp: failed to create directory via template '/run/user/111240/rear/tmp.XXXXXXXXXX': No such file or directory

so indeed setting TMPDIR in the environment to a directory not included in rootfs can be a problem even without this PR.

jsmeix commented at 2024-03-13 11:54:

@pcahyna
with the code changes of this pull request
and the artificial test added at the beginning
of usr/share/rear/build/default/990_verify_rootfs.sh

chroot $ROOTFS_DIR /bin/bash -c 'tmpdir="$( mktemp -d )" ; echo tmpdata >$tmpdir/tmpfile ; find $tmpdir -ls'

I get

# mkdir /var/tmp/mytmp

# export TMPDIR=/var/tmp/mytmp

# usr/sbin/rear -D mkrescue
...
Using build area: /var/tmp/mytmp/rear.BMiD0hqGkOXxPOt
...
Exiting rear mkrescue (PID 5086) and its descendant processes ...

# less var/log/rear/rear-localhost.log
...
+ source /root/rear.github.master.TMPDIR/usr/share/rear/build/default/990_verify_rootfs.sh
++ chroot /var/tmp/mytmp/rear.BMiD0hqGkOXxPOt/rootfs /bin/bash -c 'tmpdir="$( mktemp -d )" ; echo tmpdata >$tmpdir/tmpfile ; find $tmpdir -ls'
 14811269      4 drwx------   2 root     root         4096 Mar 13 12:48 /var/tmp/mytmp/rear.BMiD0hqGkOXxPOt/tmp/tmp.MIQ2WXDfHP
 14811270      4 -rw-r--r--   1 root     root            8 Mar 13 12:48 /var/tmp/mytmp/rear.BMiD0hqGkOXxPOt/tmp/tmp.MIQ2WXDfHP/tmpfile

So - as far as I see - the code changes of this pull request
fix by the way that setting TMPDIR to a directory
not included in 'rootfs' ($ROOTFS_DIR) is a problem.
Or do I misunderstand something?

schlomo commented at 2024-03-13 12:00:

I suspect that to fix all this consistently one needs to add $TMPDIR to COPY_AS_IS and $TMPDIR/* to COPY_AS_IS_EXCLUDE. One needs to do it after TMPDIR is set, though.

I really like this idea. It seems to me to resolve this problem more on a root-cause level by ensuring that "whatever" is going on outside the chroot will also work inside.

jsmeix commented at 2024-03-13 13:23:

I think adding $TMPDIR to COPY_AS_IS
and $TMPDIR/* to COPY_AS_IS_EXCLUDE
looks needlessly complicated and obscured
(I think the intent is to add an empty $TMPDIR)
compared to mkdir -p $ROOTFS_DIR$TMP_DIR
that is simple and straightforward
at least as far as I see.
Or do I misunderstand something?

I think adding an empty $TMPDIR to COPY_AS_IS
won't help for the chroot /mnt/local case and
it also won't help for whatever other directories
so "whatever" is going on outside the chroot
will not necessarily also work inside
at least as far as I see.
Or do I misunderstand something?

pcahyna commented at 2024-03-14 07:48:

@jsmeix I think I had not realized that your change helps also with the preexisting problem, so it is better than I thought (but see my latest comment).

jsmeix commented at 2024-03-14 08:25:

Via
https://github.com/rear/rear/pull/3168/commits/4b772e0fe0612eca257fa4eaeff405e19f1ab6ea
I check now in prep/default/990_verify_empty_rootfs.sh
only for regular files in ROOTFS_DIR so in particular
empty directories like ROOTFS_DIR/TMP_DIR are OK, cf.
https://github.com/rear/rear/commit/9b4efb2469b8cf3585206dbb10700960b480008e#r139765325

FYI
how long it takes to run find $ROOTFS_DIR
on my homeoffice system with a AMD Ryzen 3 4300G
and a Samsung NVMe SSD 980

2024-03-14 09:14:28.331693134 Entering debugscript mode via 'set -x'.
+ source /root/rear.github.master.TMPDIR/usr/share/rear/prep/default/990_verify_empty_rootfs.sh
...
2024-03-14 09:14:28.363963485 Source function: 'source /root/rear.github.master.TMPDIR/usr/share/rear/prep/default/990_verify_empty_rootfs.sh' returns 1

i.e. about a third of a second
(28.363963485 - 28.331693134 = .032270351).
I think this is acceptable.

schlomo commented at 2024-03-14 08:31:

Another way to solve this would be to actually create the ROOTFS_DIR/TMP_DIR after this check, e.g. at the very beginning of the rescue stage.

TBH I'd appreciate that approach because:

  1. it reduces the clutter in the rear main script
  2. it will allow us to catch other empty directories or special files created during prep
  3. One way to look at the temp dir problem is to see it as a required step to create a "good" ROOTFS_DIR that also supports running chroot mktemp inside. Therefore the problem belongs to rescue or build and not to the rear main script.

WDYT @jsmeix @pcahyna ?

jsmeix commented at 2024-03-14 08:40:

@schlomo
I understand but on the other hand
I do not like to spread things that belong together
so I would prefer to do all what belongs to setting TMPDIR
at one code place.
For the same reason I don't mind if sbin/rear becomes
longer as long as things belong to sbin/rear.
So we may move out the whole ROOTFS_DIR related code
out of sbin/rear - but not within this pull request.

On the third hand ("tertium datur" - always in practice)
my recent
https://github.com/rear/rear/commit/4b772e0fe0612eca257fa4eaeff405e19f1ab6ea
already spreads things that belong to setting TMPDIR
now over two code places :-(

I think spreading things that belong together
over several code places is worse 'clutter'
than having longer code parts that do one thing
and do it (reasonably) well (as far as possible
with reasonable effort).

By the way:
What is the English expression for the German
"es ist zum Mäusemelken"?

schlomo commented at 2024-03-14 09:01:

I'm all in favour to set ROOTFS_DIR=/dev/null (or something like that) in rear and then create it at the beginning of rescuewhere it is actually needed.

jsmeix commented at 2024-03-14 09:06:

@schlomo
OK - please implement it.
If you like I could first merge this pull request
or you could implement setting TMPDIR by the way.

jsmeix commented at 2024-03-14 09:11:

By the way regarding what there is in ROOTFS_DIR
at the end of the 'prep' stage:
With my recent
https://github.com/rear/rear/pull/3168/commits/05feecf0ab6db086d43d8f961e8a390c8f4a86d0
I get

# usr/sbin/rear -D mkrescue
...
Using build area: /var/tmp/rear.xQVhflLY7FL95z9
...
Modified ReaR recovery system area after 'prep' stage (/var/tmp/rear.xQVhflLY7FL95z9/rootfs contains regular files)
...
Exiting rear mkrescue (PID 5705) and its descendant processes ...

# less var/log/rear/rear-localhost.log
...
+ source /root/rear.github.master.TMPDIR/usr/share/rear/prep/default/990_verify_empty_rootfs.sh
+++ find /var/tmp/rear.xQVhflLY7FL95z9/rootfs -type f
++ test /var/tmp/rear.xQVhflLY7FL95z9/rootfs/etc/rear/rescue.conf
++ DebugPrint 'Modified ReaR recovery system area after '\''prep'\'' stage (/var/tmp/rear.xQVhflLY7FL95z9/rootfs contains regular files)'
2024-03-14 10:00:41.761946593 Modified ReaR recovery system area after 'prep' stage (/var/tmp/rear.xQVhflLY7FL95z9/rootfs contains regular files)
+++ find /var/tmp/rear.xQVhflLY7FL95z9/rootfs -ls
++ Debug ' 14949008      4 drwxr-xr-x   4 root     root         4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs
 14949016      4 drwxr-xr-x   3 root     root         4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/etc
 14949017      4 drwxr-xr-x   2 root     root         4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/etc/rear
 14949037      4 -rw-r--r--   1 root     root          642 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/etc/rear/rescue.conf
 14949010      4 drwxr-xr-x   4 root     root         4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var
 14949011      4 drwxr-xr-x   3 root     root         4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/tmp
 14949012      4 drwxr-xr-x   3 root     root         4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/tmp/rear.xQVhflLY7FL95z9
 14949013      4 drwxr-xr-x   2 root     root         4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/tmp/rear.xQVhflLY7FL95z9/tmp
 14949019      4 drwxr-xr-x   3 root     root         4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib
 14949020      4 drwxr-xr-x   7 root     root         4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs
 14949021      4 drwxr-xr-x   2 root     root         4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/v4recovery
 14949022      4 drwxr-xr-x   2 root     root         4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/sm.bak
 14949023      4 drwxr-xr-x   2 root     root         4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/sm
 14949024      4 drwxr-xr-x  11 root     root         4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/rpc_pipefs
 14949027      4 drwxr-xr-x   2 root     root         4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/rpc_pipefs/nfsd
 14949032      4 drwxr-xr-x   2 root     root         4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/rpc_pipefs/nfs
 14949034      4 drwxr-xr-x   2 root     root         4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/rpc_pipefs/lockd
 14949031      4 drwxr-xr-x   2 root     root         4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/rpc_pipefs/portmap
 14949025      4 drwxr-xr-x   3 root     root         4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/rpc_pipefs/gssd
 14949026      4 drwxr-xr-x   2 root     root         4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/rpc_pipefs/gssd/clntXX
 14949030      4 drwxr-xr-x   2 root     root         4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/rpc_pipefs/statd
 14949033      4 drwxr-xr-x   2 root     root         4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/rpc_pipefs/mount
 14949028      4 drwxr-xr-x   2 root     root         4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/rpc_pipefs/cache
 14949029      4 drwxr-xr-x   2 root     root         4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/rpc_pipefs/nfsd4_cb
 14949035      4 drwxr-xr-x   2 root     root         4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/nfsdcltrack'

so at least for my test case
there are only empty directories in ROOTFS_DIR
except one regular file

/var/tmp/rear.xQVhflLY7FL95z9/rootfs/etc/rear/rescue.conf

which contains

# initialize our /etc/rear/rescue.conf file sourced by the rear command in recover mode
# also the configuration is sourced by system-setup script during booting our recovery image

SHARE_DIR="/usr/share/rear"
CONFIG_DIR="/etc/rear"
VAR_DIR="/var/lib/rear"
LOG_DIR="/var/log/rear"

BACKUP_PROG_OPTIONS=( --anchored --xattrs --xattrs-include=security.capability --xattrs-include=security.selinux --acls )
# The following 2 lines were added by 210_include_dhclient.sh
USE_DHCLIENT=yes
DHCLIENT_BIN=dhclient

# The following lines were added by 490_store_write_protect_settings.sh
WRITE_PROTECTED_IDS=(  )
WRITE_PROTECTED_FS_LABEL_PATTERNS=( )

# All set NETFS_* variables (cf. rescue/NETFS/default/600_store_NETFS_variables.sh):
NETFS_KEEP_OLD_BACKUP_COPY=
NETFS_PREFIX=localhost
NETFS_RESTORE_CAPABILITIES=([0]="No")

USING_UEFI_BOOTLOADER=1
UEFI_BOOTLOADER="/boot/efi/EFI/opensuse/grubx64.efi"

schlomo commented at 2024-03-14 09:15:

OK, I think you should go ahead @jsmeix so that you can stop milking mice. Please merge when you feel confident that the solution is an improvement over the status quo and we'll take it from there.

jsmeix commented at 2024-03-14 09:30:

@pcahyna
when you think it is OK, in particular when you think
it is an improvement over the status quo,
then please approve it.

Afterwards I will merge it and then we wait
and see how it behaves "out there in the wild"
for (venturous) users who try out our GitHub master code.

If we learn that it causes more trouble than it helps
I will revert it and then we know at least that
ReaR should not change an already set TMPDIR.

I think "interesting" issues may appear in particular
with certain third-party backup tools that may need their
specific TMPDIR and not something that is set by ReaR.

For an example where a third-party backup tool needs
a specific TMPDIR see
restore/DUPLICITY/default/200_prompt_user_to_start_restore.sh
https://github.com/rear/rear/blob/master/usr/share/rear/restore/DUPLICITY/default/200_prompt_user_to_start_restore.sh#L23
Here it happens during restore i.e. inside the recovery system
where /sbin/rear does not set TMPDIR=$BUILD_DIR/tmp
so this pull request is not affected by it
but it shows that in particular third-party
backup tools may need a specific TMPDIR.

jsmeix commented at 2024-03-14 09:53:

Hmm...
Regarding "ReaR should not change an already set TMPDIR".

I think I must implement that too for this pull request.

So /sbin/rear would set TMPDIR=$BUILD_DIR/tmp
only if TMPDIR was not specified by the user via
export TMPDIR="/path/to/my/tmpdir"
before he called 'rear'.

Because conf/default.conf does

export TMPDIR="${TMPDIR-/var/tmp}"

it sets TMPDIR to '/var/tmp' only if TMPDIR is unset
so when TMPDIR is '/var/tmp' in ReaR it means
either TMPDIR was unset
or the user had specified TMPDIR to be '/var/tmp'.

So testing if TMPDIR is '/var/tmp' does not tell
if TMPDIR was unset before the user called 'rear'.

Therefore I would need to add something like

test -v TMPDIR || TMPDIR_WAS_UNSET="yes"

directly before TMPDIR is set in default.conf
to be able to test if TMPDIR was unset
before the user called 'rear'.

jsmeix commented at 2024-03-14 12:00:

With my recent changes in /sbin/rear via
https://github.com/rear/rear/pull/3168/commits/7e278f3329fe52edaa9527dfde1816e74ea5cd5a
"rear mkrescue" works for me for both cases
with unset TMPDIR and with specified TMPDIR
when I use the same artificial test at the beginnming of
usr/share/rear/build/default/990_verify_rootfs.sh

chroot $ROOTFS_DIR /bin/bash -c 'tmpdir="$( mktemp -d )" ; echo tmpdata >$tmpdir/tmpfile ; find $tmpdir -ls'

With unset TMPDIR after "rear -D mkrescue":

# find /var/tmp/rear.a7njJ6TRIl5fDf4/rootfs/var/tmp/rear.a7njJ6TRIl5fDf4/tmp

/var/tmp/rear.a7njJ6TRIl5fDf4/rootfs/var/tmp/rear.a7njJ6TRIl5fDf4/tmp
/var/tmp/rear.a7njJ6TRIl5fDf4/rootfs/var/tmp/rear.a7njJ6TRIl5fDf4/tmp/tmp.SxpqUcZrE8
/var/tmp/rear.a7njJ6TRIl5fDf4/rootfs/var/tmp/rear.a7njJ6TRIl5fDf4/tmp/tmp.SxpqUcZrE8/tmpfile

With "export TMPDIR=/var/tmp/mytmp" after "rear -D mkrescue":

# find /var/tmp/mytmp/rear.2i697tfj2TI0hnF/rootfs/var/tmp/mytmp

/var/tmp/mytmp/rear.2i697tfj2TI0hnF/rootfs/var/tmp/mytmp
/var/tmp/mytmp/rear.2i697tfj2TI0hnF/rootfs/var/tmp/mytmp/tmp.Nsttoa3prg
/var/tmp/mytmp/rear.2i697tfj2TI0hnF/rootfs/var/tmp/mytmp/tmp.Nsttoa3prg/tmpfile

I wait now that the CI "rear recover" tests pass
and if yes I will also test myself "rear recover"
with unset TMPDIR and with specified TMPDIR.

jsmeix commented at 2024-03-15 16:14:

Only a side note for the fun of it:

Out of curiosity I tested how current master code
(i.e. without the changes in this pull request)
behaves when TMPDIR is relative to the current working dir
wherefrom 'rear' is called:

# mkdir QQQtmp

# export TMPDIR=QQQtmp

# usr/sbin/rear -D mkrescue
Relax-and-Recover 2.7 / Git
Running rear mkrescue (PID 27033 date 2024-03-15 17:08:20)
Command line options: usr/sbin/rear -D mkrescue
Using log file: /root/rear.github.master/var/log/rear/rear-localhost.log
Using build area: QQQtmp/rear.UsuNYa5CmjjrIPN
...
Running 'rescue' stage ======================
Creating recovery system root filesystem skeleton layout
ERROR: Failed to copy '/root/rear.github.master/usr/share/rear/skel/default' contents to QQQtmp/rear.UsuNYa5CmjjrIPN/rootfs
Some latest log messages since the last called script 010_merge_skeletons.sh:
  2024-03-15 17:08:26.075362134 Entering debugscript mode via 'set -x'.
  2024-03-15 17:08:26.086398529 Creating recovery system root filesystem skeleton layout
  2024-03-15 17:08:26.092087642 Copying '/root/rear.github.master/usr/share/rear/skel/default' contents to QQQtmp/rear.UsuNYa5CmjjrIPN/rootfs
  tar: QQQtmp/rear.UsuNYa5CmjjrIPN/rootfs: Cannot open: No such file or directory
  tar: Error is not recoverable: exiting now
  tar: -: Cannot write: Broken pipe
  tar: Error is not recoverable: exiting now
Aborting due to an error, check /root/rear.github.master/var/log/rear/rear-localhost.log for details

# less var/log/rear/rear-localhost.log
...
++ LogPrint 'Creating recovery system root filesystem skeleton layout'
2024-03-15 17:08:26.086398529 Creating recovery system root filesystem skeleton layout
++ pushd /root/rear.github.master/usr/share/rear/skel
++ for skel_dir in default "$ARCH" "$OS" "$OS_MASTER_VENDOR/default" "$OS_MASTER_VENDOR_ARCH" "$OS_MASTER_VENDOR_VERSION" "$OS_VENDOR/default" "$OS_VENDOR_ARCH" "$OS_VENDOR_VERSION" "$BA
CKUP" "$OUTPUT"
++ test default
++ test -d default -o -s default.tar.gz
++ test -d default
++ Log 'Copying '\''/root/rear.github.master/usr/share/rear/skel/default'\'' contents to QQQtmp/rear.UsuNYa5CmjjrIPN/rootfs'
2024-03-15 17:08:26.092087642 Copying '/root/rear.github.master/usr/share/rear/skel/default' contents to QQQtmp/rear.UsuNYa5CmjjrIPN/rootfs
++ tar -C default -c .
++ tar -C QQQtmp/rear.UsuNYa5CmjjrIPN/rootfs -x
tar: QQQtmp/rear.UsuNYa5CmjjrIPN/rootfs: Cannot open: No such file or directory

@rear/contributors
do you perhaps know if a relative TMPDIR
is forbidden by some standard?

Regardless if a relative TMPDIR is forbidden or not:
We should simply error out in this case
at least as long as ReaR fails with relative TMPDIR.
And because nobody reported an issue with that
it seems that ReaR works with relative TMPDIR
is not needed in practice by our users.

jsmeix commented at 2024-03-18 15:51:

Via my recent
https://github.com/rear/rear/pull/3168/commits/b741883d1c724f14e3cbe7fd7d295411c3aef4e9
I implemented:

Again set TMPDIR to ReaR's TMP_DIR
in any case unless in RECOVERY_MODE,
see https://github.com/rear/rear/issues/3167#issuecomment-1999460501

Set TMPDIR to its resolved absolute path
so a specified relative TMPDIR gets changed
into what works for ReaR and
verify that TMPDIR is a directory.

Remember what TMPDIR was originally set when ReaR was launched
and provide meaningful debug info
when ReaR uses a different TMPDIR.

Added the changes from
https://github.com/rear/rear/pull/3181
to fix https://github.com/rear/rear/issues/3180
via this pull request (to aviod a merge conflict).

jsmeix commented at 2024-03-18 16:32:

So far things look good to me for "rear mkrescue":

I use this artificial test at the beginning of
build/default/990_verify_rootfs.sh

DebugPrint "TMPDIR='$TMPDIR'"

tmpdir="$( mktemp -d -t normal_tmp.XXXX )"
echo tmpdata.normal >$tmpdir/tmpfile.normal
DebugPrint "$( find $tmpdir -ls )"

chroot $ROOTFS_DIR /bin/bash -c 'tmpdir="$( mktemp -d -t chroot_tmp.XXXX )" ; echo tmpdata.chroot >$tmpdir/tmpfile.chroot ; find $tmpdir -ls'
DebugPrint "$( find $ROOTFS_DIR$TMPDIR/chroot_tmp* -ls )"

Some test results:

# export TMPDIR=" "

# usr/sbin/rear -d mkrescue
tac: failed to create temporary file in ' ': No such file or directory
tac: failed to create temporary file in ' ': No such file or directory
ERROR: TMPDIR '' is no directory
Exiting rear mkrescue (PID 18543) and its descendant processes ...
/root/rear.github.master.TMPDIR/usr/share/rear/lib/_input-output-functions.sh: line 151: kill: (18594) - No such process
Running exit tasks
rear mkrescue failed, check /root/rear.github.master.TMPDIR/var/log/rear/rear-localhost.log for details

# export TMPDIR="QQQ"

# ls -l QQQ
ls: cannot access 'QQQ': No such file or directory

# usr/sbin/rear -d mkrescue
tac: failed to create temporary file in 'QQQ': No such file or directory
tac: failed to create temporary file in 'QQQ': No such file or directory
ERROR: TMPDIR '' is no directory
Exiting rear mkrescue (PID 18641) and its descendant processes ...
/root/rear.github.master.TMPDIR/usr/share/rear/lib/_input-output-functions.sh: line 151: kill: (18692) - No such process
Running exit tasks
rear mkrescue failed, check /root/rear.github.master.TMPDIR/var/log/rear/rear-localhost.log for details

# mkdir QQQ

# export TMPDIR="QQQ"

# pwd
/root/rear.github.master.TMPDIR

# usr/sbin/rear -d mkrescue
Relax-and-Recover 2.7 / Git
Running rear mkrescue (PID 18758 date 2024-03-18 17:23:36)
Command line options: usr/sbin/rear -d mkrescue
Using log file: /root/rear.github.master.TMPDIR/var/log/rear/rear-localhost.log
Using build area: /root/rear.github.master.TMPDIR/QQQ/rear.gU6EFIFvSEjnu6s
Changing TMPDIR to '/root/rear.github.master.TMPDIR/QQQ/rear.gU6EFIFvSEjnu6s/tmp' (was 'QQQ' when ReaR was launched)
...
TMPDIR='/root/rear.github.master.TMPDIR/QQQ/rear.gU6EFIFvSEjnu6s/tmp'
  5769278 4 drwx------ 2 root root 4096 Mar 18 17:24 /root/rear.github.master.TMPDIR/QQQ/rear.gU6EFIFvSEjnu6s/tmp/normal_tmp.BTLe
  5769358 4 -rw-r--r-- 1 root root   15 Mar 18 17:24 /root/rear.github.master.TMPDIR/QQQ/rear.gU6EFIFvSEjnu6s/tmp/normal_tmp.BTLe/tmpfile.normal
  5904418 4 drwx------ 2 root root 4096 Mar 18 17:24 /root/rear.github.master.TMPDIR/QQQ/rear.gU6EFIFvSEjnu6s/rootfs/root/rear.github.master.TMPDIR/QQQ/rear.gU6EFIFvSEjnu6s/tmp/chroot_tmp.QmIb
  5904727 4 -rw-r--r-- 1 root root   15 Mar 18 17:24 /root/rear.github.master.TMPDIR/QQQ/rear.gU6EFIFvSEjnu6s/rootfs/root/rear.github.master.TMPDIR/QQQ/rear.gU6EFIFvSEjnu6s/tmp/chroot_tmp.QmIb/tmpfile.chroot
...
To remove the build area you may use (with caution): rm -Rf --one-file-system /root/rear.github.master.TMPDIR/QQQ/rear.gU6EFIFvSEjnu6s

# rm -Rf --one-file-system /root/rear.github.master.TMPDIR/QQQ/rear.gU6EFIFvSEjnu6s

# ls -l QQQ
total 0

# unset TMPDIR

# usr/sbin/rear -d mkrescue
Relax-and-Recover 2.7 / Git
Running rear mkrescue (PID 9862 date 2024-03-18 17:26:22)
Command line options: usr/sbin/rear -d mkrescue
Using log file: /root/rear.github.master.TMPDIR/var/log/rear/rear-localhost.log
Using build area: /var/tmp/rear.TvfORdqueenqWjK
Setting TMPDIR to '/var/tmp/rear.TvfORdqueenqWjK/tmp' (was unset when ReaR was launched)
...
TMPDIR='/var/tmp/rear.TvfORdqueenqWjK/tmp'
 14425255 4 drwx------ 2 root root 4096 Mar 18 17:26 /var/tmp/rear.TvfORdqueenqWjK/tmp/normal_tmp.hsc8
 14425256 4 -rw-r--r-- 1 root root   15 Mar 18 17:26 /var/tmp/rear.TvfORdqueenqWjK/tmp/normal_tmp.hsc8/tmpfile.normal
 14549201 4 drwx------ 2 root root 4096 Mar 18 17:26 /var/tmp/rear.TvfORdqueenqWjK/rootfs/var/tmp/rear.TvfORdqueenqWjK/tmp/chroot_tmp.eB82
 14549211 4 -rw-r--r-- 1 root root   15 Mar 18 17:26 /var/tmp/rear.TvfORdqueenqWjK/rootfs/var/tmp/rear.TvfORdqueenqWjK/tmp/chroot_tmp.eB82/tmpfile.chroot
...
To remove the build area you may use (with caution): rm -Rf --one-file-system /var/tmp/rear.TvfORdqueenqWjK

# rm -Rf --one-file-system /var/tmp/rear.TvfORdqueenqWjK

# ls -l /var/tmp/rear.*
ls: cannot access '/var/tmp/rear.*': No such file or directory

Tomorrow I will test "rear recover".

jsmeix commented at 2024-03-19 12:48:

So far things look also good to me for "rear recover".

Some test results:

The standard case with unset TMPDIR:

RESCUE localhost:~ # rear -D recover
Relax-and-Recover 2.7 / Git
Running rear recover (PID 751 date 2024-03-19 13:04:10)
Command line options: /bin/rear -D recover
Using log file: /var/log/rear/rear-localhost.log
Using build area: /var/tmp/rear.7dsqzj6rDbeRHOf
Setting TMPDIR to '/var/tmp' (was unset when ReaR was launched)
...
Recreating initrd with /usr/bin/dracut...
Recreated initrd with /usr/bin/dracut
...
Running exit tasks
To remove the build area you may use (with caution): rm -Rf --one-file-system /var/tmp/rear.7dsqzj6rDbeRHOf

I also did another "rear mkbackup" with

# mkdir -p /path/to/mytmpdir

# export TMPDIR="/path/to/mytmpdir"

# usr/sbin/rear -D mkbackup
Relax-and-Recover 2.7 / Git
Running rear mkbackup (PID 25479 date 2024-03-19 12:43:30)
Command line options: usr/sbin/rear -D mkbackup
Using log file: /root/rear.jsmeix-TMPDIR/var/log/rear/rear-localhost.log
Using build area: /path/to/mytmpdir/rear.H9NYpL0mtHhEjXb
Changing TMPDIR to '/path/to/mytmpdir/rear.H9NYpL0mtHhEjXb/tmp' (was '/path/to/mytmpdir' when ReaR was launched)
...

and with that "rear recover"
worked with unset TMPDIR as above
and also with

RESCUE localhost:~ # export TMPDIR="/path/to/mytmpdir"

RESCUE localhost:~ # rear -D recover
Relax-and-Recover 2.7 / Git
Running rear recover (PID 749 date 2024-03-19 13:06:41)
Command line options: /bin/rear -D recover
Using log file: /var/log/rear/rear-localhost.log
Using build area: /path/to/mytmpdir/rear.jZOvb8sOBMfog8u
Using TMPDIR '/path/to/mytmpdir' (was '/path/to/mytmpdir' when ReaR was launched)
...
Recreating initrd with /usr/bin/dracut...
Recreated initrd with /usr/bin/dracut
...
Running exit tasks
To remove the build area you may use (with caution): rm -Rf --one-file-system /path/to/mytmpdir/rear.jZOvb8sOBMfog8u

RESCUE localhost:~ # find /path/to/mytmpdir                                  
/path/to/mytmpdir
/path/to/mytmpdir/rear.jZOvb8sOBMfog8u
/path/to/mytmpdir/rear.jZOvb8sOBMfog8u/tmp
/path/to/mytmpdir/rear.jZOvb8sOBMfog8u/tmp/mappings
/path/to/mytmpdir/rear.jZOvb8sOBMfog8u/tmp/mappings/routes
/path/to/mytmpdir/rear.jZOvb8sOBMfog8u/tmp/mappings/ip_addresses
/path/to/mytmpdir/rear.jZOvb8sOBMfog8u/tmp/mappings/mac
/path/to/mytmpdir/rear.jZOvb8sOBMfog8u/tmp/by-id_change
/path/to/mytmpdir/rear.jZOvb8sOBMfog8u/tmp/diskbyid_mappings
/path/to/mytmpdir/rear.jZOvb8sOBMfog8u/tmp/restore-exclude-list.txt
/path/to/mytmpdir/rear.jZOvb8sOBMfog8u/tmp/touch
/path/to/mytmpdir/rear.jZOvb8sOBMfog8u/tmp/touch/swap-swap:-dev-sda3
...
/path/to/mytmpdir/rear.jZOvb8sOBMfog8u/tmp/touch/rear-vgchange
/path/to/mytmpdir/rear.jZOvb8sOBMfog8u/tmp/backuparchive_size
/path/to/mytmpdir/rear.jZOvb8sOBMfog8u/tmp/storage_drivers
/path/to/mytmpdir/rear.jZOvb8sOBMfog8u/rootfs
/path/to/mytmpdir/rear.jZOvb8sOBMfog8u/rootfs/path
/path/to/mytmpdir/rear.jZOvb8sOBMfog8u/rootfs/path/to
/path/to/mytmpdir/rear.jZOvb8sOBMfog8u/rootfs/path/to/mytmpdir
/path/to/mytmpdir/rear.H9NYpL0mtHhEjXb
/path/to/mytmpdir/rear.H9NYpL0mtHhEjXb/tmp

This works because for the "rear mkbackup"
with export TMPDIR="/path/to/mytmpdir"
there is

# find /path/to/mytmpdir/rear.H9NYpL0mtHhEjXb/rootfs/path/to/mytmpdir/rear.H9NYpL0mtHhEjXb

/path/to/mytmpdir/rear.H9NYpL0mtHhEjXb/rootfs/path/to/mytmpdir/rear.H9NYpL0mtHhEjXb
/path/to/mytmpdir/rear.H9NYpL0mtHhEjXb/rootfs/path/to/mytmpdir/rear.H9NYpL0mtHhEjXb/tmp

so /path/to/mytmpdir/ exists in the ReaR recovery system
and after backup restore
even /mnt/local/path/to/mytmpdir exists because
/path/to/mytmpdir/ is included in the backup

# tar -tvf backup.tar.gz | grep mytmpdir
drwxr-xr-x root/root           0 2024-03-19 12:43 path/to/mytmpdir/

Only BUILD_DIR (here /path/to/mytmpdir/rear.H9NYpL0mtHhEjXb)
is excluded from the backup in sbin/rear
so /path/to/mytmpdir exists in chroot /mnt/local

What does not work (as expected) is

RESCUE localhost:~ # mkdir mytmpdir

RESCUE localhost:~ # export TMPDIR="mytmpdir"

RESCUE localhost:~ # pwd            
/root

RESCUE localhost:~ # rear -D recover
Relax-and-Recover 2.7 / Git
Running rear recover (PID 751 date 2024-03-19 13:44:22)
Command line options: /bin/rear -D recover
Using log file: /var/log/rear/rear-localhost.log
Using build area: /root/mytmpdir/rear.8AaBPltEp78LZfU
Using TMPDIR '/root/mytmpdir' (was 'mytmpdir' when ReaR was launched)
...
Recreating initrd with /usr/bin/dracut...
Warning:
Failed to recreate initrd with /usr/bin/dracut.
Check '/var/log/rear/rear-localhost.log' why /usr/bin/dracut failed
and decide if the recreated system will boot
with the initrd 'as is' from the backup restore.
...
Running exit tasks
To remove the build area you may use (with caution): rm -Rf --one-file-system /root/mytmpdir/rear.8AaBPltEp78LZfU

RESCUE localhost:~ # grep -B5 '^Failed to recreate initrd with /usr/bin/dracut' /var/log/rear/rear-localhost.log

2024-03-19 13:45:13.817251648 Recreating initrd with /usr/bin/dracut...
++ chroot /mnt/local /bin/bash -c 'PATH=/sbin:/usr/sbin:/usr/bin:/bin /usr/bin/dracut --force'
realpath: /root/mytmpdir: No such file or directory
dracut: Invalid tmpdir '/root/mytmpdir'.
++ LogPrintError 'Warning:
Failed to recreate initrd with /usr/bin/dracut.

jsmeix commented at 2024-03-19 13:08:

Perhaps unset TMPDIR in chroot /mnt/local
is a reasonable way to ensure temporary things
work inside chroot /mnt/local?

I did this change in
finalize/SUSE_LINUX/i386/550_rebuild_initramfs.sh

    if chroot $TARGET_FS_ROOT /bin/bash -c "unset TMPDIR ; PATH=/sbin:/usr/sbin:/usr/bin:/bin $dracut_binary --force" ; then

i.e. I added unset TMPDIR ;
and now "rear recover" also works where it had failed before:

RESCUE localhost:~ # mkdir mytmpdir

RESCUE localhost:~ # export TMPDIR="mytmpdir"

RESCUE localhost:~ # rear -D recover
Relax-and-Recover 2.7 / Git
Running rear recover (PID 753 date 2024-03-19 14:03:35)
Command line options: /bin/rear -D recover
Using log file: /var/log/rear/rear-localhost.log
Using build area: /root/mytmpdir/rear.I4VnrDeRyPJImUM
Using TMPDIR '/root/mytmpdir' (was 'mytmpdir' when ReaR was launched)
...
Recreating initrd with /usr/bin/dracut...
Recreated initrd with /usr/bin/dracut
...

RESCUE localhost:~ # less /var/log/rear/rear-localhost.log
...
2024-03-19 14:04:34.385490756 Recreating initrd with /usr/bin/dracut...
++ chroot /mnt/local /bin/bash -c 'unset TMPDIR ; PATH=/sbin:/usr/sbin:/usr/bin:/bin /usr/bin/dracut --force'
dracut: Executing: /usr/bin/dracut --force
...
dracut: *** Creating initramfs image file '/boot/initrd-5.14.21-150500.55.28-default' done ***
++ LogPrint 'Recreated initrd with /usr/bin/dracut'

jsmeix commented at 2024-03-20 08:00:

@pcahyna
I think it works now reasonably well (at least for me)
so I would much appreciate it if you could have a look
and approve it if it looks OK to you.
When it is merged and issues appear I will of course
try to fix them or revert the whole stuff if needed.

pcahyna commented at 2024-03-20 13:25:

Perhaps unset TMPDIR in chroot /mnt/local
is a reasonable way to ensure temporary things
work inside chroot /mnt/local?

@jsmeix I hope we are not going to clutter the code with explicit unset TMPDIR commands for each chroot command ...

jsmeix commented at 2024-03-20 13:31:

@pcahyna
currently I don't have a good idea how to generically
ensure chroot /mnt/local will work even with

RESCUE localhost:~ # mkdir mytmpdir
RESCUE localhost:~ # export TMPDIR="mytmpdir"

I would like to postpone that part for a subsequent
separated pull request because this pull request
was already much more complicated than I had expected.

schlomo commented at 2024-03-20 13:36:

@jsmeix IMHO we don't need to deal with TMPDIR being a relative path.

jsmeix commented at 2024-03-20 14:48:

The changes in this pull request make ReaR working
even when TMPDIR is a relative path, see
https://github.com/rear/rear/pull/3168#issuecomment-2007134820

RESCUE localhost:~ # mkdir mytmpdir

RESCUE localhost:~ # export TMPDIR="mytmpdir"

RESCUE localhost:~ # rear -D recover
...
Using TMPDIR '/root/mytmpdir' (was 'mytmpdir' when ReaR was launched)

so chroot /mnt/local COMMAND will only fail
when TMPDIR is used by COMMAND and there is
no /root/mytmpdir in /mnt/local.

But it will work when there is /root/mytmpdir
in /mnt/local for whatever reason, e.g. by luck
or when also "rear mkbackup" was run before with
the same export TMPDIR="mytmpdir"

# mkdir mytmpdir

# export TMPDIR="mytmpdir"

# rear.jsmeix-TMPDIR/usr/sbin/rear -D mkbackup
...
Command line options: rear.jsmeix-TMPDIR/usr/sbin/rear -D mkbackup
Using log file: /root/rear.jsmeix-TMPDIR/var/log/rear/rear-localhost.log
Using build area: /root/mytmpdir/rear.c1FApNCh9uvUzpr
Changing TMPDIR to '/root/mytmpdir/rear.c1FApNCh9uvUzpr/tmp' (was 'mytmpdir' when ReaR was launched)
...
Running exit tasks
To remove the build area you may use (with caution): rm -Rf --one-file-system /root/mytmpdir/rear.c1FApNCh9uvUzpr

which includes the directory /root/mytmpdir/ in the backup

# tar -tvf backup.tar.gz | grep mytmpdir
drwxr-xr-x root/root         0 2024-03-20 15:27 root/mytmpdir/

So after backup restore there is
/root/mytmpdir in /mnt/local
which makes chroot /mnt/local COMMAND work

RESCUE localhost:~ # find mytmpdir
mytmpdir
mytmpdir/rear.c1FApNCh9uvUzpr
mytmpdir/rear.c1FApNCh9uvUzpr/tmp

RESCUE localhost:~ # export TMPDIR="mytmpdir"

RESCUE localhost:~ # rear -D recover
...
Using build area: /root/mytmpdir/rear.PTDCYBWhedi1WjN
Using TMPDIR '/root/mytmpdir' (was 'mytmpdir' when ReaR was launched)
...
Recreating initrd with /usr/bin/dracut...
Recreated initrd with /usr/bin/dracut
...
Running exit tasks
To remove the build area you may use (with caution): rm -Rf --one-file-system /root/mytmpdir/rear.PTDCYBWhedi1WjN

Cf. my similar export TMPDIR="/path/to/mytmpdir" example in
https://github.com/rear/rear/pull/3168#issuecomment-2007088986

pcahyna commented at 2024-03-22 13:59:

@jsmeix sorry for the delay, looking...

jsmeix commented at 2024-03-22 14:01:

@rear/contributors
I would like to merge it until Thursday (28 March) next week
(Friday 29 March and Monday 01 April are public holidays)
unless there are objections and
provided no new severe problems appear
i.e. when it is an improvement over the status quo.

pcahyna commented at 2024-03-22 16:02:

Another way to solve this would be to actually create the ROOTFS_DIR/TMP_DIR after this check, e.g. at the very beginning of the rescue stage.

TBH I'd appreciate that approach because:

1. it reduces the clutter in the `rear` main script

2. it will allow us to catch other empty directories or special files created during `prep`

3. One way to look at the temp dir problem is to see it as a required step to create a "good" `ROOTFS_DIR` that also supports running `chroot mktemp` inside. Therefore the problem belongs to `rescue` or `build` and not to the `rear` main script.

WDYT @jsmeix @pcahyna ?

I like the suggestion but it certainly does not belong in this PR and also I think that there are worse offenders that touch ROOTFS_DIR in prep that would need to be cleaned up first.

pcahyna commented at 2024-03-25 10:52:

@jsmeix thank you for your changes and sorry, I had yet one comment after sleeping on it, hopefully this is the last...

jsmeix commented at 2024-03-25 12:09:

@pcahyna
thank you for your review again!

Never be sorry for late comments
like comments after sleeping on an issue
(sleeping on an issue always helps - at least for me)
because in particular all deliberate comments are helpful
(regardless if they are actually right or mistaken).

jsmeix commented at 2024-03-26 06:54:

@rear/contributors
I will merge it today afternoon
unless objections appear.

pcahyna commented at 2024-03-26 16:45:

Thank you for the improvement @jsmeix !

jsmeix commented at 2024-03-27 08:14:

Let's hope it actually improves things,
i.e. let's hope it does not cause a severe regression
for an unforeseen use case.

[Export of Github issue for rear/rear.]