#1968 Issue `closed`: Rear mkbackup is hang : Creating disk layout¶

Labels: support / question, fixed / solved / done, not ReaR / invalid

kodiss opened issue at 2018-11-19 05:35:¶

Relax-and-Recover (ReaR) Issue Template¶

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

ReaR version ("/usr/sbin/rear -V"):
Relax-and-Recover 2.2 / 2017-07-20
OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"):
Redhat 7.4
ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

OUTPUT=ISO
BACKUP=NETFS
BACKUP_URL=file:///hana/data/backup/os/
NETFS_KEEP_OLD_BACKUP_COPY=no
ONLY_INCLUDE_VG=("rootvg")
BACKUP_PROG_ARCHIVE="osbackup_`hostname`"
EXCLUDE_MOUNTPOINTS=('/hana/log' '/hana/shared2' '/hana/log2' '/hana/data2' '/hana/shared' '/hana/data')
EXCLUDE_VG=('vgHANA' 'vgHANA2')
GRUB_RESCUE_USER="unrestricted"
BACKUP_PROG_EXCLUDE=( '/dev/shm/*' '/var/lib/rear/output/*')

Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR):
Baremetal ( HPE/DL980 G7)
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device): x86-64
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot): UEFI / GRUB
Storage (lokal disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe): Local Disk
Description of the issue (ideally so that others can reproduce it):
When i try to backup by using rear, the backup status is hang
1. the backup is stopped in Creating disk layout status
2. There is 3 rear process during issue status and never stopped or die.

rear process

ps -ef | grep -i rear
root      76975  74496  0 14:04 pts/1    00:00:00 /bin/bash /usr/sbin/rear -v mkbackup
root      78136  76975  0 14:04 pts/1    00:00:00 /bin/bash /usr/sbin/rear -v mkbackup
root      78280  78136  0 14:04 pts/1    00:00:00 /bin/bash /usr/sbin/rear -v mkbackup
root      79000  75878  0 14:11 pts/3    00:00:00 grep --color=auto -i rear

backup log (rear -v mkbackup)

Relax-and-Recover 2.2 / 2017-07-20
Using log file: /var/log/rear/rear-nfdsdb01.log
Using backup archive '/hana/data/backup/os//nfdsdb01/osbackup_nfdsdb01.tar.gz'
Creating disk layout

kodiss commented at 2018-11-19 05:35:¶

# rear dump

Relax-and-Recover 2.2 / 2017-07-20
Using log file: /var/log/rear/rear-nfdsdb01.log.lockless
Dumping out configuration and system information
This is a 'Linux-x86_64' system, compatible with 'Linux-i386'.
System definition:
                                    ARCH = Linux-i386
                                      OS = GNU/Linux
                        OS_MASTER_VENDOR = Fedora
                       OS_MASTER_VERSION = 7
                   OS_MASTER_VENDOR_ARCH = Fedora/i386
                OS_MASTER_VENDOR_VERSION = Fedora/7
           OS_MASTER_VENDOR_VERSION_ARCH = Fedora/7/i386
                               OS_VENDOR = RedHatEnterpriseServer
                              OS_VERSION = 7
                          OS_VENDOR_ARCH = RedHatEnterpriseServer/i386
                       OS_VENDOR_VERSION = RedHatEnterpriseServer/7
                  OS_VENDOR_VERSION_ARCH = RedHatEnterpriseServer/7/i386
Configuration tree:
                         Linux-i386.conf : OK
                          GNU/Linux.conf : OK
                             Fedora.conf : missing/empty
                        Fedora/i386.conf : missing/empty
                           Fedora/7.conf : missing/empty
                      Fedora/7/i386.conf : missing/empty
             RedHatEnterpriseServer.conf : missing/empty
        RedHatEnterpriseServer/i386.conf : missing/empty
           RedHatEnterpriseServer/7.conf : missing/empty
      RedHatEnterpriseServer/7/i386.conf : missing/empty
                               site.conf : missing/empty
                              local.conf : OK
Backup with NETFS
              NETFS_KEEP_OLD_BACKUP_COPY = no
                            NETFS_PREFIX = nfdsdb01
              NETFS_RESTORE_CAPABILITIES = No
                  BACKUP_INTEGRITY_CHECK =
                         BACKUP_MOUNTCMD =
                     BACKUP_ONLY_EXCLUDE = no
                     BACKUP_ONLY_INCLUDE = no
                          BACKUP_OPTIONS =
      BACKUP_RESTORE_MOVE_AWAY_DIRECTORY = /var/lib/rear/moved_away_after_backup_restore/
          BACKUP_RESTORE_MOVE_AWAY_FILES =
                    BACKUP_RSYNC_OPTIONS = --sparse --archive --hard-links --numeric-ids --stats
                  BACKUP_SELINUX_DISABLE = 1
                             BACKUP_TYPE =
                        BACKUP_UMOUNTCMD =
                              BACKUP_URL = file:///hana/data/backup/os/
Backup program is 'tar':
                             BACKUP_PROG = tar
                     BACKUP_PROG_ARCHIVE = osbackup_nfdsdb01
            BACKUP_PROG_COMPRESS_OPTIONS = --gzip
             BACKUP_PROG_COMPRESS_SUFFIX = .gz
               BACKUP_PROG_CRYPT_ENABLED = 0
                   BACKUP_PROG_CRYPT_KEY =
               BACKUP_PROG_CRYPT_OPTIONS = /usr/bin/openssl des3 -salt -k
             BACKUP_PROG_DECRYPT_OPTIONS = /usr/bin/openssl des3 -d -k
                     BACKUP_PROG_EXCLUDE = /dev/shm/* /var/lib/rear/output/*
                     BACKUP_PROG_INCLUDE =
                     BACKUP_PROG_OPTIONS = --anchored
      BACKUP_PROG_OPTIONS_CREATE_ARCHIVE =
     BACKUP_PROG_OPTIONS_RESTORE_ARCHIVE =
                      BACKUP_PROG_SUFFIX = .tar
       BACKUP_PROG_WARN_PARTIAL_TRANSFER = 1
Output to ISO
                             ISO_DEFAULT = boothd
                                 ISO_DIR = /var/lib/rear/output
                               ISO_FILES =
                        ISO_ISOLINUX_BIN =
                            ISO_MAX_SIZE =
                         ISO_MKISOFS_BIN = /bin/mkisofs
                              ISO_PREFIX = rear-nfdsdb01
                        ISO_RECOVER_MODE =
                               ISO_VOLID = RELAXRECOVER
                         OUTPUT_MOUNTCMD =
                          OUTPUT_OPTIONS =
                           OUTPUT_PREFIX = nfdsdb01
                       OUTPUT_PREFIX_PXE =
                        OUTPUT_UMOUNTCMD =
                              OUTPUT_URL =
                           RESULT_MAILTO =

/usr/share/rear/lib/validated/RedHatEnterpriseServer/7/i386.txt
Your system is validated with the following details:
Version:     Relax-and-Recover 1.17.2 / Git
Validation:  RedHatEnterpriseServer/7/i386
Submitted:   JP jason@thescreaminggoat.net, Screaming Goat Networks LLC, US
Date:        2016-11-12
Features:    LVM, NETFS, ISO
Comment:     Needed to install genisoimage for CentOS/RHEL 7

Version:     Relax-and-Recover 1.19 / 2016-10-03
Validation:  RedHatEnterpriseServer/7/i386
Submitted:   Gratien Dhaese
Date:        2016-10-03
Features:    LVM, RSYNC, ISO, NETFS, BAREOS
Comment:     No issues encountered
Saving /var/log/rear/rear-nfdsdb01.log.lockless as /var/log/rear/rear-nfdsdb01.log

kodiss commented at 2018-11-19 05:36:¶

rear log

2018-11-19 14:04:47.562771480 Relax-and-Recover 2.2 / 2017-07-20
2018-11-19 14:04:47.564890930 Command line options: /usr/sbin/rear -v mkbackup
2018-11-19 14:04:47.566586277 Using log file: /var/log/rear/rear-nfdsdb01.log
2018-11-19 14:04:47.568639894 Including /etc/rear/os.conf
2018-11-19 14:04:47.570581864 Including conf/Linux-i386.conf
2018-11-19 14:04:47.572445226 Including conf/GNU/Linux.conf
2018-11-19 14:04:47.582924785 Including /etc/rear/local.conf
2018-11-19 14:04:47.588424041 ======================
2018-11-19 14:04:47.589984362 Running 'init' stage
2018-11-19 14:04:47.591492882 ======================
2018-11-19 14:04:47.598766717 Including init/default/010_set_drlm_env.sh
2018-11-19 14:04:47.601283613 Including init/default/030_update_recovery_system.sh
2018-11-19 14:04:47.603318320 Including init/default/050_check_rear_recover_mode.sh
2018-11-19 14:04:47.605333709 Finished running 'init' stage in 0 seconds
2018-11-19 14:04:47.615626350 Using build area '/tmp/rear.MahK0Br9is3pmug'
mkdir: created directory '/tmp/rear.MahK0Br9is3pmug/rootfs'
mkdir: created directory '/tmp/rear.MahK0Br9is3pmug/tmp'
2018-11-19 14:04:47.620886313 Running mkbackup workflow
2018-11-19 14:04:47.626099217 ======================
2018-11-19 14:04:47.627758796 Running 'prep' stage
2018-11-19 14:04:47.629447577 ======================
2018-11-19 14:04:47.638853429 Including prep/default/005_remove_workflow_conf.sh
mkdir: created directory '/tmp/rear.MahK0Br9is3pmug/rootfs/etc'
mkdir: created directory '/tmp/rear.MahK0Br9is3pmug/rootfs/etc/rear'
2018-11-19 14:04:47.644734590 Including prep/default/020_translate_url.sh
2018-11-19 14:04:47.647057600 Including prep/default/030_translate_tape.sh
2018-11-19 14:04:47.651954871 Including prep/default/040_check_output_scheme.sh
2018-11-19 14:04:47.657749505 Including prep/NETFS/default/050_check_NETFS_requirements.sh
2018-11-19 14:04:47.665038851 Skipping 'ping' test for host '' in BACKUP_URL 'file:///hana/data/backup/os/'
2018-11-19 14:04:47.680901472 Including prep/default/050_check_keep_old_output_copy_var.sh
2018-11-19 14:04:47.682934760 Including prep/NETFS/default/060_mount_NETFS_path.sh
mkdir: created directory '/tmp/rear.MahK0Br9is3pmug/outputfs'
2018-11-19 14:04:47.694102763 Including prep/NETFS/default/070_set_backup_archive.sh
2018-11-19 14:04:47.701489463 Using backup archive '/hana/data/backup/os//nfdsdb01/osbackup_nfdsdb01.tar.gz'
2018-11-19 14:04:47.703633717 Including prep/NETFS/default/090_check_encrypted_backup.sh
2018-11-19 14:04:47.705538136 Including prep/NETFS/default/100_check_nfs_version.sh
mkdir: created directory '/tmp/rear.MahK0Br9is3pmug/rootfs/var'
mkdir: created directory '/tmp/rear.MahK0Br9is3pmug/rootfs/var/lib'
mkdir: created directory '/tmp/rear.MahK0Br9is3pmug/rootfs/var/lib/nfs'
mkdir: created directory '/tmp/rear.MahK0Br9is3pmug/rootfs/var/lib/nfs/rpc_pipefs'
mkdir: created directory '/tmp/rear.MahK0Br9is3pmug/rootfs/var/lib/nfs/rpc_pipefs/gssd'
mkdir: created directory '/tmp/rear.MahK0Br9is3pmug/rootfs/var/lib/nfs/rpc_pipefs/gssd/clntXX'
mkdir: created directory '/tmp/rear.MahK0Br9is3pmug/rootfs/var/lib/nfs/rpc_pipefs/nfsd'
mkdir: created directory '/tmp/rear.MahK0Br9is3pmug/rootfs/var/lib/nfs/rpc_pipefs/cache'
mkdir: created directory '/tmp/rear.MahK0Br9is3pmug/rootfs/var/lib/nfs/rpc_pipefs/nfsd4_cb'
mkdir: created directory '/tmp/rear.MahK0Br9is3pmug/rootfs/var/lib/nfs/rpc_pipefs/statd'
mkdir: created directory '/tmp/rear.MahK0Br9is3pmug/rootfs/var/lib/nfs/rpc_pipefs/portmap'
mkdir: created directory '/tmp/rear.MahK0Br9is3pmug/rootfs/var/lib/nfs/rpc_pipefs/nfs'
mkdir: created directory '/tmp/rear.MahK0Br9is3pmug/rootfs/var/lib/nfs/rpc_pipefs/mount'
mkdir: created directory '/tmp/rear.MahK0Br9is3pmug/rootfs/var/lib/nfs/rpc_pipefs/lockd'
mkdir: created directory '/tmp/rear.MahK0Br9is3pmug/rootfs/var/lib/nfs/statd'
mkdir: created directory '/tmp/rear.MahK0Br9is3pmug/rootfs/var/lib/nfs/statd/sm'
mkdir: created directory '/tmp/rear.MahK0Br9is3pmug/rootfs/var/lib/nfs/statd/sm.bak'
mkdir: created directory '/tmp/rear.MahK0Br9is3pmug/rootfs/var/lib/nfs/v4recovery'
2018-11-19 14:04:47.749375930 Including prep/default/100_init_workflow_conf.sh
2018-11-19 14:04:47.753442645 Including prep/NETFS/default/150_save_rsync_version.sh
2018-11-19 14:04:47.757259416 Including prep/GNU/Linux/200_include_getty.sh
2018-11-19 14:04:47.941512205 Including prep/GNU/Linux/200_include_serial_console.sh
/usr/share/rear/lib/_input-output-functions.sh: line 199: type: getty: not found
2018-11-19 14:04:47.954257520 Including prep/NETFS/GNU/Linux/200_selinux_in_use.sh
2018-11-19 14:04:47.956799558 Including prep/GNU/Linux/210_include_dhclient.sh
/usr/share/rear/lib/_input-output-functions.sh: line 199: type: dhcpcd: not found
/usr/share/rear/lib/_input-output-functions.sh: line 199: type: dhcp6c: not found
/usr/share/rear/lib/_input-output-functions.sh: line 199: type: dhclient6: not found
2018-11-19 14:04:48.059139669 Including prep/GNU/Linux/220_include_lvm_tools.sh
2018-11-19 14:04:48.062347310 Device mapper found enabled. Including LVM tools.
2018-11-19 14:04:48.065178520 Including prep/GNU/Linux/230_include_md_tools.sh
2018-11-19 14:04:48.068727470 Software RAID detected. Including mdadm tools.
2018-11-19 14:04:48.071472615 Including prep/GNU/Linux/240_include_multipath_tools.sh
2018-11-19 14:04:48.073404056 Including prep/GNU/Linux/280_include_systemd.sh
2018-11-19 14:04:48.105333730 Including systemd (init replacement) tool-set to bootstrap Relax-and-Recover
2018-11-19 14:04:48.107535343 Including prep/GNU/Linux/280_include_vmware_tools.sh
2018-11-19 14:04:48.116719143 Including prep/GNU/Linux/290_include_drbd.sh
2018-11-19 14:04:48.125026483 Including prep/GNU/Linux/300_check_backup_and_output_url.sh
2018-11-19 14:04:48.138028514 Including prep/ISO/default/300_check_iso_dir.sh
2018-11-19 14:04:48.140057048 Including prep/GNU/Linux/300_include_grub_tools.sh
/usr/share/rear/lib/_input-output-functions.sh: line 199: type: grub-probe: not found
2018-11-19 14:04:48.152539533 Including prep/GNU/Linux/310_include_cap_utils.sh
2018-11-19 14:04:48.155027614 Including prep/default/310_include_uefi_tools.sh
2018-11-19 14:04:48.158594320 Including prep/ISO/default/320_check_cdrom_size.sh
2018-11-19 14:04:48.163405958 ISO Directory '/var/lib/rear/output' [/dev/mapper/rootvg-lvroot] has 185234 MB free space
2018-11-19 14:04:48.166005672 Including prep/default/320_include_uefi_env.sh
2018-11-19 14:04:48.171799835 Including prep/ISO/GNU/Linux/320_verify_mkisofs.sh
2018-11-19 14:04:48.173999884 Using '/bin/mkisofs' to create ISO images
2018-11-19 14:04:48.176448828 Including prep/ISO/Linux-i386/330_find_isolinux.sh
2018-11-19 14:04:48.182766109 Including prep/ISO/GNU/Linux/340_add_isofs_module.sh
2018-11-19 14:04:48.187929494 Including prep/NETFS/default/400_automatic_exclude_recreate.sh
2018-11-19 14:04:48.201701170 Including prep/default/400_save_mountpoint_details.sh
2018-11-19 14:04:48.261917018 Including prep/default/950_check_missing_programs.sh
2018-11-19 14:04:48.267261197 Including prep/NETFS/default/980_umount_NETFS_dir.sh
rmdir: removing directory, '/tmp/rear.MahK0Br9is3pmug/outputfs'
2018-11-19 14:04:48.273689478 Finished running 'prep' stage in 1 seconds
2018-11-19 14:04:48.275561962 ======================
2018-11-19 14:04:48.277182619 Running 'layout/save' stage
2018-11-19 14:04:48.278799119 ======================
2018-11-19 14:04:48.286041504 Including layout/save/GNU/Linux/100_create_layout_file.sh
2018-11-19 14:04:48.288111290 Creating disk layout
2018-11-19 14:04:48.290676958 Preparing layout directory.
2018-11-19 14:04:48.301977284 Removing old layout file.
2018-11-19 14:04:48.304404720 Including layout/save/GNU/Linux/150_save_diskbyid_mappings.sh
2018-11-19 14:04:48.430178794 Saved diskbyid_mappings
2018-11-19 14:04:48.432732445 Including layout/save/GNU/Linux/200_partition_layout.sh
2018-11-19 14:04:48.447174172 Saving disk partitions.
Error: /dev/sda: unrecognised disk label
Error: /dev/sdb: unrecognised disk label
Error: /dev/sdc: unrecognised disk label
Error: /dev/sde: unrecognised disk label
2018-11-19 14:04:48.826990017 Including layout/save/GNU/Linux/210_raid_layout.sh
2018-11-19 14:04:48.831137367 Saving Software RAID configuration.
2018-11-19 14:04:49.155183168 Including layout/save/GNU/Linux/220_lvm_layout.sh
2018-11-19 14:04:49.157342669 Saving LVM layout.
  WARNING: Not using lvmetad because config setting use_lvmetad=0.
  WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).
  WARNING: Not using lvmetad because config setting use_lvmetad=0.
  WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).
  WARNING: Not using lvmetad because config setting use_lvmetad=0.
  WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).
2018-11-19 14:04:49.586215527 Including layout/save/GNU/Linux/230_filesystem_layout.sh
2018-11-19 14:04:49.588489112 Begin saving filesystem layout
2018-11-19 14:04:49.592292734 Saving filesystem layout (using the findmnt command).

kodiss commented at 2018-11-19 05:36:¶

rear_deug.txt

gozora commented at 2018-11-19 07:45:¶

Hello @kodiss

According your rear_deug.txt, ReaR stops execution at xfs_admin -u /dev/md0.
Normally xfs_admin should return immediately:

If md0 has XFS:

node1:/ # xfs_admin -u /dev/md0
UUID = 47747f57-ca3e-47ce-a0da-6ff16c4e7349

If md0 does not have XFS:

node1:/ # xfs_admin -u /dev/md0
xfs_admin: /dev/md0 is not a valid XFS filesystem (unexpected SB magic number 0x00000000)
Use -F to force a read attempt.

Can you try to run xfs_admin -u /dev/md0 on your OS and check if it ends its execution or keeps hanging?

V.