#2792 Issue `closed`: 'find /usr' several hours delay in 300_create_isolinux.sh with RHEL 8 on some servers¶

Labels: enhancement, support / question, fixed / solved / done

exfarmer opened issue at 2022-04-15 13:02:¶

Relax-and-Recover (ReaR) Issue Template¶

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

ReaR version ("/usr/sbin/rear -V"): Relax-and-Recover 2.6 / 2020-06-17
OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):

LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: RedHatEnterprise
Description: Red Hat Enterprise Linux release 8.5 (Ootpa)
Release: 8.5
Codename: Ootpa

ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

OUTPUT=ISO
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP=NETFS
BACKUP_URL=nfs://edlusxo001.oneabbott.com/data/col1/usxo_linux_bu_image
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp' '/var/crash' '/var/log' '/srv' '/usr/infra' )
ONLY_INCLUDE_VG=( 'VolGroup00' 'vgos' )
NETFS_KEEP_OLD_BACKUP_COPY=yes

Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR):

Dell Inc. PowerEdge R650

System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):

x86

Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):

4.18.0-348.12.2.el8_5.x86_64

Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):

SAN

Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"):

NAME                        KNAME      PKNAME    TRAN TYPE  FSTYPE       LABEL   SIZE MOUNTPOINT
/dev/sda                    /dev/sda                  disk                     446.6G
|-/dev/sda1                 /dev/sda1  /dev/sda       part  xfs                  512M /boot
`-/dev/sda2                 /dev/sda2  /dev/sda       part  LVM2_member        446.1G
  |-/dev/mapper/vgos-lvroot /dev/dm-0  /dev/sda2      lvm   xfs                    4G /
  |-/dev/mapper/vgos-lvswap /dev/dm-1  /dev/sda2      lvm   swap                  20G [SWAP]
  |-/dev/mapper/vgos-lvusr  /dev/dm-2  /dev/sda2      lvm   xfs                   10G /usr
  |-/dev/mapper/vgos-lvopt  /dev/dm-3  /dev/sda2      lvm   xfs                   10G /opt
  |-/dev/mapper/vgos-lvtmp  /dev/dm-4  /dev/sda2      lvm   xfs                    2G /tmp
  |-/dev/mapper/vgos-lvvar  /dev/dm-5  /dev/sda2      lvm   xfs                    6G /var
  `-/dev/mapper/vgos-lvhome /dev/dm-6  /dev/sda2      lvm   xfs                    2G /home
/dev/sdb                    /dev/sdb             fc   disk  mpath_member         350G
`-/dev/mapper/mpatha        /dev/dm-7  /dev/sdb       mpath LVM2_member          350G
  |-/dev/mapper/vgora-lvu01 /dev/dm-9  /dev/dm-7      lvm   ext4                 100G /u01
  |-/dev/mapper/vgora-lvu02 /dev/dm-10 /dev/dm-7      lvm   ext4                 100G /u02
  `-/dev/mapper/vgora-lvu03 /dev/dm-11 /dev/dm-7      lvm   ext4                 100G /u03
/dev/sdc                    /dev/sdc             fc   disk  mpath_member         500G
`-/dev/mapper/PU_ORADATA01  /dev/dm-8  /dev/sdc       mpath                      500G
/dev/sdd                    /dev/sdd             fc   disk  mpath_member         500G
`-/dev/mapper/PU_ORADATA02  /dev/dm-13 /dev/sdd       mpath                      500G
/dev/sde                    /dev/sde             fc   disk  mpath_member         100G
`-/dev/mapper/PU_ORAARC01   /dev/dm-14 /dev/sde       mpath                      100G
/dev/sdf                    /dev/sdf             fc   disk  mpath_member         100G
`-/dev/mapper/PU_ORAARC02   /dev/dm-15 /dev/sdf       mpath                      100G
/dev/sdg                    /dev/sdg             fc   disk  mpath_member         100G
`-/dev/mapper/PU_ORAFRA01   /dev/dm-16 /dev/sdg       mpath                      100G
/dev/sdh                    /dev/sdh             fc   disk  mpath_member         100G
`-/dev/mapper/PU_ORAFRA02   /dev/dm-17 /dev/sdh       mpath                      100G
/dev/sdi                    /dev/sdi             fc   disk  mpath_member          70G
`-/dev/mapper/PU_OCR_VOTE01 /dev/dm-18 /dev/sdi       mpath                       70G
/dev/sdj                    /dev/sdj             fc   disk  mpath_member          70G
`-/dev/mapper/PU_OCR_VOTE02 /dev/dm-19 /dev/sdj       mpath                       70G
/dev/sdk                    /dev/sdk             fc   disk  mpath_member          70G
`-/dev/mapper/PU_OCR_VOTE03 /dev/dm-20 /dev/sdk       mpath                       70G
/dev/sdl                    /dev/sdl             fc   disk  mpath_member          20G
`-/dev/mapper/mpathk        /dev/dm-12 /dev/sdl       mpath                       20G
/dev/sdm                    /dev/sdm             fc   disk  mpath_member         350G
`-/dev/mapper/mpatha        /dev/dm-7  /dev/sdm       mpath LVM2_member          350G
  |-/dev/mapper/vgora-lvu01 /dev/dm-9  /dev/dm-7      lvm   ext4                 100G /u01
  |-/dev/mapper/vgora-lvu02 /dev/dm-10 /dev/dm-7      lvm   ext4                 100G /u02
  `-/dev/mapper/vgora-lvu03 /dev/dm-11 /dev/dm-7      lvm   ext4                 100G /u03
/dev/sdn                    /dev/sdn             fc   disk  mpath_member         500G
`-/dev/mapper/PU_ORADATA01  /dev/dm-8  /dev/sdn       mpath                      500G
/dev/sdo                    /dev/sdo             fc   disk  mpath_member         500G
`-/dev/mapper/PU_ORADATA02  /dev/dm-13 /dev/sdo       mpath                      500G
/dev/sdp                    /dev/sdp             fc   disk  mpath_member         100G
`-/dev/mapper/PU_ORAARC01   /dev/dm-14 /dev/sdp       mpath                      100G
/dev/sdq                    /dev/sdq             fc   disk  mpath_member         100G
`-/dev/mapper/PU_ORAARC02   /dev/dm-15 /dev/sdq       mpath                      100G
/dev/sdr                    /dev/sdr             fc   disk  mpath_member         100G
`-/dev/mapper/PU_ORAFRA01   /dev/dm-16 /dev/sdr       mpath                      100G
/dev/sds                    /dev/sds             fc   disk  mpath_member         100G
`-/dev/mapper/PU_ORAFRA02   /dev/dm-17 /dev/sds       mpath                      100G
/dev/sdt                    /dev/sdt             fc   disk  mpath_member          70G
`-/dev/mapper/PU_OCR_VOTE01 /dev/dm-18 /dev/sdt       mpath                       70G
/dev/sdu                    /dev/sdu             fc   disk  mpath_member          70G
`-/dev/mapper/PU_OCR_VOTE02 /dev/dm-19 /dev/sdu       mpath                       70G
/dev/sdv                    /dev/sdv             fc   disk  mpath_member          70G
`-/dev/mapper/PU_OCR_VOTE03 /dev/dm-20 /dev/sdv       mpath                       70G
/dev/sdw                    /dev/sdw             fc   disk  mpath_member          20G
`-/dev/mapper/mpathk        /dev/dm-12 /dev/sdw       mpath                       20G

Description of the issue (ideally so that others can reproduce it):

gzip: stdout: No space left on device
2022-04-14 17:00:40.394769657 ERROR: Failed to create recovery/rescue system initrd.cgz

Workaround, if any: None
Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
rear-uq00264q.log

To paste verbatim text like command output or file content,
include it between a leading and a closing line of three backticks like

```
verbatim content
```

exfarmer commented at 2022-04-15 13:32:¶

I see in the log file snapshots are included in the backup, those are on an NFS mount. Why wouldn't that be excluded?

[root@uq00264q ~]# df /usr/infra/.snapshot/daily_02.2022-04-06_0200
Filesystem                                                       1K-blocks   Used Available Use% Mounted on
usspfs12:/unix_infraNFS/linux/.snapshot/daily_02.2022-04-06_0200 403726944 533600 403193344   1% /usr/infra/.snapshot/daily_02.2022-04-06_0200

usspfs12:/unix_infraNFS/linux/.snapshot/daily_02.2022-04-06_0200 on /usr/infra/.snapshot/daily_02.2022-04-06_0200 type nfs (ro,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,local_lock=none,addr=10.1.5.67)

pcahyna commented at 2022-04-19 18:19:¶

gzip: stdout: No space left on device
2022-04-14 17:00:40.394769657 ERROR: Failed to create recovery/rescue system initrd.cgz

I don't see this error message in the log rear-uq00264q.log . Is the log complete?

I see in the log file snapshots are included in the backup, those are on an NFS mount. Why wouldn't that be excluded?

[root@uq00264q ~]# df /usr/infra/.snapshot/daily_02.2022-04-06_0200 Filesystem 1K-blocks Used Available Use% Mounted on usspfs12:/unix_infraNFS/linux/.snapshot/daily_02.2022-04-06_0200 403726944 533600 403193344 1% /usr/infra/.snapshot/daily_02.2022-04-06_0200

usspfs12:/unix_infraNFS/linux/.snapshot/daily_02.2022-04-06_0200 on /usr/infra/.snapshot/daily_02.2022-04-06_0200 type nfs (ro,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,local_lock=none,addr=10.1.5.67)

is /usr/infra/.snapshot itself on the local disk? If so, I suppose only the mount points are included in the backup, not the actual file systems mounted at them (i.e they should show up as empty directories).

jsmeix commented at 2022-04-20 12:22:¶

The error message "Failed to create recovery/rescue system ..."
is in pack/GNU/Linux/900_create_initramfs.sh
there at this code place (excerpts)

case "$REAR_INITRD_COMPRESSION" in
    ...
    (*)
        ...
            REAR_INITRD_FILENAME="initrd.cgz"
        ...
        if find . ! -name "*~" | cpio -H newc --create --quiet | gzip > "$TMP_DIR/$REAR_INITRD_FILENAME" ; then
            ...
        else
            ...
            Error "Failed to create recovery/rescue system $REAR_INITRD_FILENAME"
        fi

so the gzip: stdout: No space left on device
appears for "$TMP_DIR/$REAR_INITRD_FILENAME"
so it seems $TMP_DIR runs out of space here.

@exfarmer
see the info about TMP_DIR in your usr/share/rear/conf/default.conf

For the ReaR release 2.6 it is online at
https://github.com/rear/rear/blob/rear-2.6/usr/share/rear/conf/default.conf#L44

For our current GitHub master code it is currently online at
https://github.com/rear/rear/blob/master/usr/share/rear/conf/default.conf#L44

For info what needs most space in the ReaR recovery system initrd see
https://github.com/rear/rear/discussions/2640#discussioncomment-908335

exfarmer commented at 2022-04-20 17:29:¶

The size issue with /tmp is fixed.
Now the issue is the time it takes to run. I’ve attached the last /var/log/rear/rear*.log.

exfarmer commented at 2022-04-21 15:42:¶

Here’s the latest failure:

Error 04/21/2022 08:43:35 SSL_write
Error 04/21/2022 08:43:35 SSL_write (Error on socket: 'Broken pipe')
Error 04/21/2022 08:43:34 SSL_write
Error 04/21/2022 08:43:34 SSL_write (Error on socket: 'Bad file descriptor')
Error 04/21/2022 02:41:45 It will not be copied by default. You can include '/usr/src/kernels/4.18.0-348.12.2.el8_5.x86_64' via the 'COPY_AS_IS' configuration variable.
Error 04/21/2022 02:41:45 Symlink '/usr/lib/modules/4.18.0-348.12.2.el8_5.x86_64/source' -> '/usr/src/kernels/4.18.0-348.12.2.el8_5.x86_64' refers to a non-existing directory on the recovery system.
Error 04/21/2022 02:41:45 It will not be copied by default. You can include '/usr/src/kernels/4.18.0-348.12.2.el8_5.x86_64' via the 'COPY_AS_IS' configuration variable.
Error 04/21/2022 02:41:45 Symlink '/usr/lib/modules/4.18.0-348.12.2.el8_5.x86_64/build' -> '/usr/src/kernels/4.18.0-348.12.2.el8_5.x86_64' refers to a non-existing directory on the recovery system.
Info 04/21/2022 08:43:35 Exit Code 10
Info 04/21/2022 08:43:34 socket_OK : ioctl error: 9
Info 04/21/2022 08:43:34 nexec: ioctl error: 0
Info 04/21/2022 02:41:45 Skip copying broken symlink '/etc/mtab' target '/proc/1251256/mounts' on /proc/ /sys/ /dev/ or /run/
Info 04/21/2022 02:41:44 Copying all files in /lib*/firmware/
Info 04/21/2022 02:41:41 Copying all kernel modules in /lib/modules/4.18.0-348.12.2.el8_5.x86_64 (MODULES contains 'all_modules')
Info 04/21/2022 02:41:36 Copying binaries and libraries
Info 04/21/2022 02:41:33 Copying files and directories
Info 04/21/2022 02:41:33 Copying logfile /var/log/rear/rear-uq00263q.log into initramfs as '/tmp/rear-uq00263q-partial-2022-04-21T06:41:33+00:00.log'
Info 04/21/2022 02:41:32 Handled network interface 'bond0'
Info 04/21/2022 02:41:32 eno12409np1 is a physical device
Info 04/21/2022 02:41:32 bond0 has lower interface eno12409np1
Info 04/21/2022 02:41:32 eno12399np0 is a physical device
Info 04/21/2022 02:41:32 bond0 has lower interface eno12399np0
Info 04/21/2022 02:41:32 bond0 is a bond
Info 04/21/2022 02:41:32 Handling network interface 'bond0'
Info 04/21/2022 02:41:31 Creating recovery system root filesystem skeleton layout
Info 04/21/2022 02:41:31 Verifying that the entries in /var/lib/rear/layout/disklayout.conf are correct ...
Info 04/21/2022 02:41:31 Using guessed bootloader 'GRUB' (found in first bytes on /dev/sda)
Info 04/21/2022 02:41:31 Excluding Volume Group vgora
Info 04/21/2022 02:41:29 Overwriting existing disk layout file /var/lib/rear/layout/disklayout.conf
Info 04/21/2022 02:41:29 Creating disk layout
Info 04/21/2022 02:41:28 Using autodetected kernel '/boot/vmlinuz-4.18.0-348.12.2.el8_5.x86_64' as kernel in the recovery system
Info 04/21/2022 02:41:28 Using backup archive '/tmp/rear.h1KoBAk8YNOvblH/outputfs/uq00263q/backup.tar.gz'
Info 04/21/2022 02:41:28 Running workflow mkbackup on the normal/original system
Info 04/21/2022 02:41:28 Using log file: /var/log/rear/rear-uq00263q.log
Info 04/21/2022 02:41:28 Running rear mkbackup (PID 1236788)
Info 04/21/2022 02:41:28 Relax-and-Recover 2.6 / 2020-06-17
Info 04/21/2022 02:42:25 Created initrd.cgz with gzip default compression (435417328 bytes) in 31 seconds
Info 04/21/2022 02:41:54 Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Info 04/21/2022 02:41:45 Testing that the recovery system in /tmp/rear.h1KoBAk8YNOvblH/rootfs contains a usable system

exfarmer commented at 2022-05-13 14:17:¶

I've been running the backups for about a month now, sometimes they finish in 10 minutes, sometimes they finish in 5-6 hours. The amount of data on the servers is the same. The storage being written to is the same.
The step that is the longest:

2022-05-13 01:30:04.559225663 Finished running 'layout/compare' stage in 0 seconds
2022-05-13 01:30:04.560109139 Finished running checklayout workflow
2022-05-13 01:30:04.561032316 Saving /var/log/rear/rear-uq00547p.log.lockless as /var/log/rear/rear-uq00547p.log
2022-05-13 04:57:58.552253386 Created isolinux configuration
2022-05-13 04:57:58.557895463 Including output/default/400_copy_disk_struct_files.sh

From 01:30:04 to 04:57:58 there are no messages, is there a way to see what is going on at this time?

pcahyna commented at 2022-05-13 15:13:¶

@exfarmer have you tried increasing the log verbosity (-d or even -D)? Maybe it would show something happening between 01:30:04 - 04:57:58, -D should even show the exact command that takes so much time.

exfarmer commented at 2022-05-13 15:16:¶

Hi,

I use: rear -Dv mkbackup - for the backups

pcahyna commented at 2022-05-13 15:28:¶

@exfarmer the log that you provided an hour ago is created by rear with -D ? I don't see any messages about running commands there.

exfarmer commented at 2022-05-13 18:47:¶

I've got a good log file just now
rear-uq00550p.log

exfarmer commented at 2022-05-13 18:49:¶

I’ve uploaded a log file.
The steps that run between 2022-05-13 04:49: 54.455875382 and 2022-05-13 11:02:50.240385155 Created isolinux configuration
Seems longer on a few servers we have, and not all the time.

jsmeix commented at 2022-05-16 07:13:¶

@exfarmer
in your
https://github.com/rear/rear/files/8690141/rear-uq00550p.log
I dont's see further timestamps between
2022-05-13 04:49:54.455875382 and
2022-05-13 11:02:50.240385155
neither are there messages from called programs
that indicate something gets delayed.

To find out what takes so much time in
usr/share/rear/output/ISO/Linux-i386/300_create_isolinux.sh
add as very first line in that script

( while Log 'TICK' ; do sleep 1 ; done ) & tickPID=$!

and add as very last line to that script

kill $tickPID

which will log TICK with a timestamp each second
while that script is running.

Then redo a "rear -D mkbackup" or "rear -D mkrescue"
the latter is sufficient because the delay happens in
usr/share/rear/output/ISO/Linux-i386/300_create_isolinux.sh
so the delay does not happen while making the backup
but while creating the isolinux configuration
which belongs to making the rescue/recovery system.

exfarmer commented at 2022-05-16 12:11:¶

I added the lines to
usr/share/rear/output/ISO/Linux-i386/300_create_isolinux.sh
and started the mkrescue a few minutes ago,
the attached is what I see so far
rear-uq00264q.log

jsmeix commented at 2022-05-16 12:46:¶

@exfarmer
your https://github.com/rear/rear/files/8699870/rear-uq00264q.log
contains (excerpts):

+ source /usr/share/rear/output/ISO/Linux-i386/300_create_isolinux.sh
.
.
.
++++ find /usr -name menu.c32
++++ tail -1
++ Log TICK
...
++ Log TICK
...
++ Log TICK
...

So the delay happens at find /usr

As far as I see in the code this is
the function find_syslinux_modules_dir
which is called by the function make_syslinux_config as

syslinux_modules_dir=$( find_syslinux_modules_dir menu.c32 )

which is called by 300_create_isolinux.sh

The function find_syslinux_modules_dir
in lib/bootloader-functions.sh does (excerpts):

    if [[ -n "$SYSLINUX_MODULES_DIR" ]]; then
        [[ -d "$SYSLINUX_MODULES_DIR" ]] && echo "$SYSLINUX_MODULES_DIR"
        return
    fi
    ...
        else
            # not default location? try to find it
            # file=/usr/lib/syslinux/modules/efi32/menu.c32
            # f23: file=/usr/share/syslinux/menu.c32
            file=$( find /usr -name "$1" 2>/dev/null | tail -1 )
    ...
    echo "$syslinux_modules_dir"

So the find /usr is fallback code when
SYSLINUX_MODULES_DIR is not set.

To speed up things it should work
to set SYSLINUX_MODULES_DIR in etc/rear/local.conf
to the directory that contains the SYSLINUX modules
in particular the SYSLINUX module menu.c32

For example on my openSUSE Leap 15.3 system I get

# rpm -ql syslinux | grep menu.c32
/usr/share/syslinux/menu.c32

What is strange is that SYSLINUX_MODULES_DIR
is not mentioned in usr/share/rear/conf/default.conf

I don't know the reason why it is not in default.conf

Perhaps @gdha may know more because it originated at
https://github.com/rear/rear/commit/7cde528498784a48490feb7cd1fadc4b396db620
which points to
https://github.com/rear/rear/issues/624

jsmeix commented at 2022-05-16 13:04:¶

@exfarmer
in your older
https://github.com/rear/rear/files/8690141/rear-uq00550p.log
there is (excerpt)

++++ find /usr -name menu.c32
++++ tail -1
+++ file=/usr/share/syslinux/menu.c32
++++ dirname /usr/share/syslinux/menu.c32
+++ syslinux_modules_dir=/usr/share/syslinux
+++ syslinux_modules_dir=/usr/share
+++ is_true
+++ case "$1" in
+++ return 1
+++ syslinux_modules_dir=/usr/share/bios
+++ [[ ! -d /usr/share/bios ]]
++++ dirname /usr/share/syslinux/menu.c32
+++ syslinux_modules_dir=/usr/share/syslinux
+++ [[ -d /usr/share/syslinux ]]
+++ BugIfError 'Define SYSLINUX_MODULES_DIR in local.conf as syslinux modules were not found'
+++ ((  0 != 0  ))
+++ echo /usr/share/syslinux
++ syslinux_modules_dir=/usr/share/syslinux
++ SYSLINUX_DIR=/usr/share/syslinux

so in this case on this server in etc/rear/local.conf

SYSLINUX_MODULES_DIR="/usr/share/syslinux"

should be the right value to speed up things.

exfarmer commented at 2022-05-16 13:15:¶

Wow, the mkrescure ran in 1 minute and 4 seconds.
Thank you for now. I'll run a mkbackup and test this on a few other systems.
I'll get back to you later today.

jsmeix commented at 2022-05-16 13:22:¶

Via
https://github.com/rear/rear/commit/339cff93232460fc5098b06c666c925d88459360
the function find_syslinux_modules_dir
now tells the user in debug mode what is going on
(plus a hint one "may specify SYSLINUX_MODULES_DIR")
when the fallback 'find /usr' is run
to find the SYSLINUX modules directory.

This is likely not a final solution but at least the user is now
informed (in debug mode) what is going on and what could help.

exfarmer commented at 2022-05-16 13:45:¶

Hi,

The backups are now running as expected, about 5 minutes.
The command that was taking so long was $( find /usr )
That command didn’t have an issue on any of the other servers we have, only the new RHEL 8 servers. Why would that be?
I suspect it was looking in all the /usr file systems, they are basically the same on all our servers.

exfarmer commented at 2022-05-18 15:13:¶

You can close the ticket, the backups are running as expected.
Thank you

jsmeix commented at 2022-05-19 09:25:¶

@exfarmer
thank you for your explicit feedback that the issue is solved for you.

exfarmer commented at 2022-10-11 09:04:¶

Hi,

Thank you for getting back to me.
The no space left on disk was the /tmp file system, took me a while to figure it out, …
With all the testing and no deleting the old /tmp/rear.* file systems it got filled up.

My issue now is the time it takes to run one of these backups, some are 12 hours.
I running them with rear -Dv mkbackup, but I’m not seeing what happens after:
…
…
It will not be copied by default. You can include '/usr/src/kernels/4.18.0-348.12.2.el8_5.x86_64' via the 'COPY_AS_IS' configuration variable.
Testing that the recovery system in /tmp/rear.7V34l6Nxs8lrHPg/rootfs contains a usable system
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (435439504 bytes) in 31 seconds

I changed my local.conf file a little:
OUTPUT=ISO
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP=NETFS
BACKUP_URL=nfs://edlusxo001.oneabbott.com/data/col1/usxo_linux_bu_image
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp' '/var/crash' '/var/log' '/usr/openv' '/srv')
EXCLUDE_MOUNTPOINTS=( /tmp )
ONLY_INCLUDE_VG=( 'vgos' )
NETFS_KEEP_OLD_BACKUP_COPY=yes

I added the EXCLUDE_MOUNTPOINTS line, thinking it would be better, but I think that’s already part of the BACKUP_PROG_EXCLUDE from the default.conf file.
I will get with the storage tram, but is there anything you can think of to speed this up?

Thank you, again
Abbott
[image001]
Gary Hess
Administrator, Sr - Unix
Abbott

253 Financial Blvd.
Liberty, SC 29657 USA

O:
+1 864-843-8352
M:
+1 864-546-8921
E:
***@***.******@***.***>

From: pcahyna ***@***.***>
Sent: Tuesday, April 19, 2022 2:19 PM
To: rear/rear ***@***.***>
Cc: Hess, Gary ***@***.***>; Author ***@***.***>
Subject: Re: [rear/rear] RHEL 8 backups not finishing (Issue #2792)

EXTERNAL EMAIL: Only click links or open attachments if you recognize the sender and know the content is safe.

gzip: stdout: No space left on device

2022-04-14 17:00:40.394769657 ERROR: Failed to create recovery/rescue system initrd.cgz

I don't see this error message in the log rear-uq00264q.log . Is the log complete?

I see in the log file snapshots are included in the backup, those are on an NFS mount. Why wouldn't that be excluded?

***@***.*** ~]# df /usr/infra/.snapshot/daily_02.2022-04-06_0200 Filesystem 1K-blocks Used Available Use% Mounted on usspfs12:/unix_infraNFS/linux/.snapshot/daily_02.2022-04-06_0200 403726944 533600 403193344 1% /usr/infra/.snapshot/daily_02.2022-04-06_0200

usspfs12:/unix_infraNFS/linux/.snapshot/daily_02.2022-04-06_0200 on /usr/infra/.snapshot/daily_02.2022-04-06_0200 type nfs (ro,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,local_lock=none,addr=10.1.5.67)

is /usr/infra/.snapshot itself on the local disk, if so, I suppose only the mount points are included in the backup, not the actual file systems mounted at them (i.e they should show up as empty directories).

—
Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/rear/rear/issues/2792*issuecomment-1102949875__;Iw!!BBM_p3AAtQ!dTMM1l0FRpbCt5ywuZruTkYHB52DR5qFQn0Gj1Uj_9KW6pEA9V7DWOT3FdxSfvL4$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/ASRDKRL6LPG33FRFBE5EC43VF32LDANCNFSM5TQM3H2Q__;!!BBM_p3AAtQ!dTMM1l0FRpbCt5ywuZruTkYHB52DR5qFQn0Gj1Uj_9KW6pEA9V7DWOT3FQrvGm3P$.
You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>

exfarmer commented at 2022-10-11 09:04:¶

I didn’t include the whole log, but it was -d not -D.
I will include a new log tomorrow. We can discuss on Monday.

Thank you

Abbott
[image001]
Gary Hess
Administrator, Sr - Unix
Abbott

253 Financial Blvd.
Liberty, SC 29657 USA

O:
+1 864-843-8352
M:
+1 864-546-8921
E:
***@***.******@***.***>

From: pcahyna ***@***.***>
Sent: Friday, May 13, 2022 11:29 AM
To: rear/rear ***@***.***>
Cc: Hess, Gary ***@***.***>; Mention ***@***.***>
Subject: Re: [rear/rear] RHEL 8 backups not finishing (Issue #2792)

EXTERNAL EMAIL: Only click links or open attachments if you recognize the sender and know the content is safe.

@exfarmerhttps://urldefense.com/v3/__https:/github.com/exfarmer__;!!BBM_p3AAtQ!NaNicZOuQH1c5w4pz0eHbax_tsxFBUjAe7j8fp0PeYQlLre3NcKNxIhacI7zzN3RHPAXaWSAyJjCcrQ6EtV6NaCe$ the log that you provided an hour ago is created by rear with -D ? I don't see any messages about running commands there.

—
Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/rear/rear/issues/2792*issuecomment-1126179166__;Iw!!BBM_p3AAtQ!NaNicZOuQH1c5w4pz0eHbax_tsxFBUjAe7j8fp0PeYQlLre3NcKNxIhacI7zzN3RHPAXaWSAyJjCcrQ6EgDZ-UbZ$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/ASRDKRJRKELTHZ3L5M2DXULVJZYKXANCNFSM5TQM3H2Q__;!!BBM_p3AAtQ!NaNicZOuQH1c5w4pz0eHbax_tsxFBUjAe7j8fp0PeYQlLre3NcKNxIhacI7zzN3RHPAXaWSAyJjCcrQ6Epl4ebAu$.
You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>>

[Export of Github issue for rear/rear.]

#2792 Issue closed: 'find /usr' several hours delay in 300_create_isolinux.sh with RHEL 8 on some servers¶

exfarmer opened issue at 2022-04-15 13:02:¶

Relax-and-Recover (ReaR) Issue Template¶

exfarmer commented at 2022-04-15 13:32:¶

pcahyna commented at 2022-04-19 18:19:¶

jsmeix commented at 2022-04-20 12:22:¶

exfarmer commented at 2022-04-20 17:29:¶

exfarmer commented at 2022-04-21 15:42:¶

exfarmer commented at 2022-05-13 14:17:¶

pcahyna commented at 2022-05-13 15:13:¶

exfarmer commented at 2022-05-13 15:16:¶

pcahyna commented at 2022-05-13 15:28:¶

exfarmer commented at 2022-05-13 18:47:¶

exfarmer commented at 2022-05-13 18:49:¶

jsmeix commented at 2022-05-16 07:13:¶

exfarmer commented at 2022-05-16 12:11:¶

jsmeix commented at 2022-05-16 12:46:¶

jsmeix commented at 2022-05-16 13:04:¶

exfarmer commented at 2022-05-16 13:15:¶

jsmeix commented at 2022-05-16 13:22:¶

exfarmer commented at 2022-05-16 13:45:¶

exfarmer commented at 2022-05-18 15:13:¶

jsmeix commented at 2022-05-19 09:25:¶

exfarmer commented at 2022-10-11 09:04:¶

exfarmer commented at 2022-10-11 09:04:¶

#2792 Issue `closed`: 'find /usr' several hours delay in 300_create_isolinux.sh with RHEL 8 on some servers¶