#2924 Issue closed: OUTPUT=USB: REAL_USB_DEVICE mounted both below /tmp/rear... and /var/tmp/rear...

Labels: waiting for info, support / question, no-issue-activity

frilogg opened issue at 2023-02-13 09:23:

Relax-and-Recover (ReaR) Issue Template

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

  • ReaR version ("/usr/sbin/rear -V"):
    Relax-and-Recover 2.4.16 / Git

  • OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):
    NAME="Red Hat Enterprise Linux Server"
    VERSION="7.9 (Maipo)"
    ID="rhel"
    ID_LIKE="fedora"
    VARIANT="Server"
    VARIANT_ID="server"
    VERSION_ID="7.9"
    PRETTY_NAME="Red Hat Enterprise Linux Server 7.9 (Maipo)"
    ANSI_COLOR="0;31"
    CPE_NAME="cpe:/o:redhat:enterprise_linux:7.9:GA:server"
    HOME_URL="https://www.redhat.com/"
    BUG_REPORT_URL="https://bugzilla.redhat.com/"

  • ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
    OUTPUT=USB
    BACKUP=NETFS
    BACKUP_URL="usb:///dev/disk/by-label/REAR-000"
    BACKUP_PROG_EXCLUDE=('/home/rear' '/var/log/audit')
    USB_UEFI_PART_SIZE="1648"
    HOSTNAME="sp7-ui"
    USB_DEVICE_FILESYSTEM="ext4"
    export TMPDIR="/home/rear"

  • Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR):
    PC Dell Precision 3450

  • System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
    uname -a
    Linux solx8ht 3.10.0-1160.66.1.el7.x86_64 #1 SMP Wed Apr 27 20:34:34 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

  • Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
    Grub2

  • Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):

  • Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"):

NAME                       KNAME          PKNAME         TRAN TYPE FSTYPE      LABEL      SIZE MOUNTPOINT
/dev/sda                   /dev/sda                      usb  disk                       57.8G
|-/dev/sda1                /dev/sda1      /dev/sda            part vfat        REAR-EFI   1.6G
`-/dev/sda2                /dev/sda2      /dev/sda            part ext4        REAR-000  56.1G
/dev/nvme0n1               /dev/nvme0n1                       disk                      238.5G
|-/dev/nvme0n1p1           /dev/nvme0n1p1 /dev/nvme0n1        part vfat                   500M /boot/efi
|-/dev/nvme0n1p2           /dev/nvme0n1p2 /dev/nvme0n1        part ext4                     1G /boot
`-/dev/nvme0n1p3           /dev/nvme0n1p3 /dev/nvme0n1        part LVM2_member            237G
  |-/dev/mapper/rhel_-root /dev/dm-0      /dev/nvme0n1p3      lvm  ext4                    50G /
  |-/dev/mapper/rhel_-swap /dev/dm-1      /dev/nvme0n1p3      lvm  swap                   7.6G [SWAP]
  `-/dev/mapper/rhel_-home /dev/dm-2      /dev/nvme0n1p3      lvm  ext4                   150G /home

  • Description of the issue (ideally so that others can reproduce it):
    We cant restore backup. After "enter" on relax and recover option it doesnt "fire up" loading files.

  • Workaround, if any:
    Haven't found any

  • Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):

    rear-solx8ht.log

schlomo commented at 2023-02-13 18:41:

Dear @frilogg, please install ReaR version 2.7, e.g. from http://download.opensuse.org/repositories/Archiving:/Backup:/Rear/RHEL_7/ and try again

In the rescue system you'll have to type rear recover, of course:
image

frilogg commented at 2023-02-13 21:07:

Hi @schlomo , thank you for feedback, but .. had hoped to figure it out with this version. I have used rear for many of my customers, and some of the problem is that they "only" want to use the version that is with the official redhat repo. I also noticed that with redhat 9.1 version 2.6 is default also.. So for this cause no good quick tips or you notice anything from the logs provided ?

schlomo commented at 2023-02-13 22:10:

Well, line 11 of your log says:

/etc/rear/local.conf: line 17: export:=/home/rear': not a valid identifier`

(don't understand why you would export something in a ReaR config anyway as everything is one large shell script)

Besides that you didn't provide anything about the actual problem which is during recovery, right?

frilogg commented at 2023-02-14 06:38:

Hi and good morning @schlomo , this was corrected but I still had the problem.

I followed your advice and installed Rear 2.7., so thats the version Im on now.
When trying to create a backup now it exit after few seconds. When I do lsblk -p this is not mounted. Please advice.

+ source /usr/share/rear/prep/USB/Linux-i386/350_check_usb_disk.sh
++ test /dev/disk/by-label/REAR-000
+++ readlink -f /dev/disk/by-label/REAR-000
++ REAL_USB_DEVICE=/dev/sda2
++ test -b /dev/sda2
++ res=($( grep -v $BUILD_DIR /proc/mounts | grep "^$REAL_USB_DEVICE" ))
+++ grep -v /var/tmp/rear.upjUYcgOpdZ2LjO /proc/mounts
+++ grep '^/dev/sda2'
++ Error 'USB or disk device '\''/dev/sda2'\'' is already mounted on '\''/tmp/rear.upjUYcgOpdZ2LjO/outputfs'\'''
++ test -s /var/log/rear/rear-solx8ht.log
++ PrintError 'ERROR: USB or disk device '\''/dev/sda2'\'' is already mounted on '\''/tmp/rear.upjUYcgOpdZ2LjO/outputfs'\'''

frilogg commented at 2023-02-14 06:44:

The reason we want to export and use another tmpdir or else it would not be enough space on device. I ran now export TMPDIR=/home/rear - and now the backup is created. In rear 2.4 we could have this string in the local.conf but it seems that its not working anymore?!

frilogg commented at 2023-02-14 07:22:

I have now the same problem as https://github.com/rear/rear/issues/2920

schlomo commented at 2023-02-14 07:38:

Sorry @frilogg, we need to somewhat change the mode of our conversation here:

  1. please use correct markdown to make it simpler to read
  2. please provide much more context, log files, screenshot etc. and not just a fragment.

With the information you shared I don't even know what to look at.

If you do this for a customer then maybe a commercial support option might be better suited for you where one of us will join you in a remote screen sharing session to look at your situation.

About your TMPDIR question, in 2.7 we changed that to be more standards compliant and the best place to find out about ReaR variabled is always default.conf, where you can find everything about TMPDIR:

# TMPDIR
#
# Relax-and-Recover needs a (temporary) working area where it builds in particular
# the rescue/recovery system ISO image (and perhaps even stores the backup archive).
# The directory name of the working area is created in /usr/sbin/rear by calling
#   mktemp -d -t rear.XXXXXXXXXXXXXXX
# which usually results /tmp/rear.XXXXXXXXXXXXXXX or $TMPDIR/rear.XXXXXXXXXXXXXXX
# the latter when the canonical Linux/Unix environment variable TMPDIR
# is set in the environment where /usr/sbin/rear is called.
# To have a specific working area directory prefix for Relax-and-Recover call
#   export TMPDIR="/prefix/for/rear/working/directory"
# before calling 'rear' (/prefix/for/rear/working/directory must already exist).
# This is useful for example when there is not sufficient free space
# in /var/tmp or $TMPDIR for the ISO image or even the backup archive.
# TMPDIR cannot be set to a default value here unconditionally but only
# if it is not set before calling the program, otherwise /usr/sbin/rear
# would not work in compliance with the Linux/Unix standards regarding TMPDIR
# see https://github.com/rear/rear/issues/968
# The default is /var/tmp instead of the more usual /tmp (the system default),
# because /tmp is not intended for such large amounts of data that ReaR usually
# produces when creating the image (see file-hierarchy(7)). In particular,
# /tmp can be a tmpfs, and thus restricted by the available RAM/swap.
export TMPDIR="${TMPDIR-/var/tmp}"
####

As it is exported there you don't need to (and shouldn't) export it again

gdha commented at 2023-02-14 09:07:

@frilogg Concerning your customers who want to use the official rear version delivered with their disribution - they have the right to submit an incident at RH and RH support will pick it up and try to help them or provide a fix (which will be merged in our master repo). RH and Suse are activily committing enhancemants and bug fixes in our ReaR master repo.

frilogg commented at 2023-02-16 10:46:

Thank you for quick feedback, and sorry for late response from my side.
Here you can see how far I got, and also its the attached logs with Rear 2.7.

218050959-db94f21b-e786-47b7-b8c0-b710f81339a3

218050953-5843952c-a33b-4ea5-9a10-7fcc265a6635

rear27-solx8ht.log

jsmeix commented at 2023-02-16 11:15:

Only as a side note FYI
(I know it does not actually help):

Sigh!
Just the usual user experience when booting via UEFI fails.
In general, not limited to ReaR, it always behaves is way:
Not any (error) message, neither from the UEFI firmware
nor from the bootloader - so there is nothing helpful.
I have no idea how to debug such UEFI booting issues.

jsmeix commented at 2023-02-16 12:10:

@frilogg
in your
https://github.com/rear/rear/issues/2924#issuecomment-1432885861
the Grub menu text "Boot original system"
does not appear in ReaR 2.7 code
because that was changed via
https://github.com/rear/rear/commit/25f86d4f9cb320db637e65550830cb19a7fd2b02
and https://github.com/rear/rear/pull/2662

Your
https://github.com/rear/rear/files/10755054/rear27-solx8ht.log
does not contain any

+ source /usr/share/rear/output/USB/...

because it ends with

+ source /usr/share/rear/prep/USB/Linux-i386/350_check_usb_disk.sh

where it still errors out with

++ Error 'USB or disk device '\''/dev/sda2'\'' is already mounted on '\''/tmp/rear.upjUYcgOpdZ2LjO/outputfs'\'''

as in your
https://github.com/rear/rear/issues/2924#issuecomment-1429199422

jsmeix commented at 2023-02-16 12:17:

@frilogg
your
https://github.com/rear/rear/files/10755054/rear27-solx8ht.log
contains (excerpts)

mount: /dev/sda2 mounted on /var/tmp/rear.upjUYcgOpdZ2LjO/outputfs.
...
++ Error 'USB or disk device '\''/dev/sda2'\'' is already mounted on '\''/tmp/rear.upjUYcgOpdZ2LjO/outputfs'\'''
...
umount: /var/tmp/rear.upjUYcgOpdZ2LjO/outputfs (/dev/sda2) unmounted

so it seems some 'umount' is somehow somewhere missing
before prep/USB/Linux-i386/350_check_usb_disk.sh
is run or the test in that script is wrong...

jsmeix commented at 2023-02-16 12:30:

Now I see it:

/dev/sda2 mounted on /var/tmp/rear.upjUYcgOpdZ2LjO/outputfs

versus

/dev/sda2 already mounted on /tmp/rear.upjUYcgOpdZ2LjO/outputfs

i.e. /var/tmp/ versus /tmp/

jsmeix commented at 2023-02-16 12:39:

@frilogg
in your
usr/share/rear/prep/USB/Linux-i386/350_check_usb_disk.sh
add at its beginning a line

cat /proc/mounts

and run "rear -D mkbackup" again and
attach its whole debug log file here.

frilogg commented at 2023-02-16 13:15:

The reason we want to export and use another tmpdir or else it would not be enough space on device. I ran now export TMPDIR=/home/rear - and now the backup is created. In rear 2.4 we could have this string in the local.conf but it seems that its not working anymore?!

@jsmeix this error was gone when I exported the TMPDIR, so the backup ran. When I did not have this I could not make a complete backup at all. And it is this backup that I am trying to restore. You still want me to add this?

cat /proc/mounts

jsmeix commented at 2023-02-16 13:46:

@frilogg
but then your
https://github.com/rear/rear/files/10755054/rear27-solx8ht.log
does not match your screenshots in
https://github.com/rear/rear/issues/2924#issuecomment-1432885861

To get a clean state without possibly broken
leftovers from what you did with ReaR 2.4
I suggest to completely redo things
with ReaR 2.7 from scratch i.e.:

First redo "rear -D format" for your USB disk
and save its debug log file - perhaps we need it.
Note that since ReaR 2.7 "rear format" behaves different,
see our release notes about "rear format":
https://github.com/rear/rear/blob/rear-2.7/doc/rear-release-notes.txt#L297

Then run "rear -D mkbackup" again and also
save its debug log file - perhaps we need it.
This will create the ReaR recovery system plus
its bootloader things and the backup.tar.gz
anew from scratch on your USB disk.

Then try to boot this new ReaR recovery system
from your USB disk and report here whether or not
that works for you with ReaR 2.7 - i.e. only
whether or not the ReaR recovery system boots and
if you can log in there as 'root' (no password needed).

Provided this works you can try to recreate your system
by calling "rear -D recover" from inside the running
ReaR recovery system.

When you run "rear recover" on some hardware or VM
it will completely overwrite what there already is
on the harddisk(s) or virtual disk(s) on that hardware or VM
so never try out "rear recover" on your original system
where you had run "rear mkbackup" before,
see the initial sections in
https://en.opensuse.org/SDB:Disaster_Recovery

frilogg commented at 2023-02-16 16:02:

Hi @jsmeix , I have now created the usb again with
rear format -D

But I still get the same error when running
rear -D mkbackup

Relax-and-Recover 2.7 / 2022-07-13
Running rear mkbackup (PID 23959 date 2023-02-16 16:48:13)
Command line options: /sbin/rear -D mkbackup
Using log file: /var/log/rear/rear-solx8ht.log
Using build area: /var/tmp/rear.oVfd2oO0gXvooTT
Running 'init' stage ======================
Running workflow mkbackup on the normal/original system
Running 'prep' stage ======================
Using backup archive '/var/tmp/rear.oVfd2oO0gXvooTT/outputfs/rear/sp7-ui/20230216.1648/backup.tar.gz'
Found EFI system partition /dev/nvme0n1p1 on /boot/efi type vfat
Using UEFI Boot Loader for Linux (USING_UEFI_BOOTLOADER=1)
ERROR: USB or disk device '/dev/sda3' is already mounted on '/tmp/rear.oVfd2oO0gXvooTT/outputfs'
Some latest log messages since the last called script 350_check_usb_disk.sh:

rear-solx8ht.log

frilogg commented at 2023-02-18 07:34:

Highly appreciate some more feedback @jsmeix or @schlomo as the backup media itself is not creating with Rear 2.7.

frilogg commented at 2023-02-23 11:53:

Done some more research and with a help from a colleague we made following changes to

/usr/share/rear/prep/USB/Linux-i386/350_check_usb_disk.sh

# Check if the ReaR USB device is not accidentally mounted on other than $BUILD_DIR location: vBUILD_DIR=$BUILD_DIR BUILD_DIR=$(echo $BUILD_DIR | sed -r 's/\/var//g') if res=( $( grep -v $BUILD_DIR /proc/mounts | grep "^$REAL_USB_DEVICE" ) ) ; then     Error "USB or disk device '$REAL_USB_DEVICE' is already mounted on '${res[1]}'" fi BUILD_DIR=$vBUILD_DIR

This made the script go pass the error

ERROR: USB or disk device '/dev/sda3' is already mounted on

The backup starting to create but stops and "freeze" when trying to finish.

> 2023-02-23 12:39:36.047271490 tar --warning=no-xdev --sparse --block-number --totals --verbose --no-wildcards-match-slash --one-file-system --ignore-failed-read --anchored --xattrs --xattrs-include=security.capability --xattrs-include=security.selinux --acls --gzip -X /home/rear/rear.whvq6zfr2uszZYN/tmp/backup-exclude.txt -C / -c -f - /home / /boot/efi /boot /var/log/rear/rear-solx8ht.log | dd of=/home/rear/rear.whvq6zfr2uszZYN/outputfs/rear/sp7-ui/20230223.1237/backup.tar.gz bs=1M
++ test 1
++ case "$( basename $BACKUP_PROG )" in
+++ basename tar
++ sleep 1
++ kill -0 30009
+++ awk '{s+=$1} END {print s}'
+++ stat -c %s /home/rear/rear.whvq6zfr2uszZYN/outputfs/rear/sp7-ui/20230223.1237/backup.tar.gz
++ size=16515072
++ ProgressInfo 'Archived 15 MiB [avg 8064 KiB/sec] '
++ echo -en '\e[2K\rArchived 15 MiB [avg 8064 KiB/sec] \e7'
++ sleep 1

Any suggestion to this error when its freezing?

rear-solx8ht.log

jsmeix commented at 2023-02-23 12:22:

@frilogg
sorry - no idea - and currently no time for it (too many other things).
I will have to test things myself (but I use openSUSE Leap 15.4)
to get a better understanding what actually is going on
before I can make some reasonable comment here.

frilogg commented at 2023-02-23 14:14:

@schlomo and @jsmeix

I have now got it to work but I had to do the following

  1. Make the changes to the script 350_check_usb_disk.sh , and hopefully this can be something you can use.

  2. Install kernel-devel

  3. Use another TMPDIR location as this got to small for /var/tmp - and created exclude for this folder.

  4. Define use of theese seeings
    BOOTLOADER="GRUB2-EFI"
    USING_UEFI_BOOTLOADER=1

  5. Then use secure boot at startup. No secure is not working. I have also tried checking bios settings if there are any settings that could have impact here but secure boot is actually off, and also settings regarding uefi paths is off.

Its working for now, but if I get the time I will try doing more research but the workaround in the 350 script worked.

jsmeix commented at 2023-02-23 14:38:

@frilogg
thank you for your feedback that
for now you could make it work for your case
and what you did to make it work for your case.

I will have a look when time permits.

github-actions commented at 2023-04-25 02:21:

Stale issue message

[Export of Github issue for rear/rear.]