#3230 Issue open: ReaR confuses Device Mapper Multipathing (DM) devices with Non-volatile Memory Express™ (NVMe™) multipathing devices.

Labels: support / question

StefanFromMunich opened issue at 2024-05-22 08:22:

  • ReaR version ("/usr/sbin/rear -V"):
    Relax-and-Recover 2.6 / 2020-06-17

  • OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):

NAME="Oracle Linux Server"
VERSION="8.5"
ID="ol"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="8.5"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Oracle Linux Server 8.5"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:8:5:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://bugzilla.oracle.com/"

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 8"
ORACLE_BUGZILLA_PRODUCT_VERSION=8.5
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=8.5
  • ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
OUTPUT=ISO
OUTPUT_URL=nfs://193.14.236.134/volume1/Rear
BACKUP=NETFS
BACKUP_URL=nfs://193.14.236.134/volume1/Rear
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp' '/var/crash' '/u02' '/dev/nvme0n1'  '/dev/nvme1n1'  '/dev/nvme2n1'  '/dev/nvme3n1'  '/dev/nvme4n1'  '/dev/nvme5n1' )
NETFS_KEEP_OLD_BACKUP_COPY=
UEFI_BOOTLOADER=/boot/efi/EFI/redhat/grubx64.efi
SECURE_BOOT_BOOTLOADER=/boot/efi/EFI/redhat/shimx64.efi
EXCLUDE_MOUNTPOINTS=( '/u03/flashdata/ZVK_MUC' '/u02/oradata/ZVK_MUC' )
AUTOEXCLUDE_MULTIPATH=n
BOOT_OVER_SAN=y
AUTORESIZE_PARTITIONS=true
AUTOSHRINK_DISK_SIZE_LIMIT_PERCENTAGE=100

  • Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR):
    PC

  • System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
    x86

  • Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
    UEFI and bootloader GRUB

  • Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
    SSD (DM and NVMe)

  • Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"):

NAME                                      KNAME          PKNAME         TRAN   TYPE  FSTYPE          LABEL   SIZE MOUNTPOINT
/dev/sda                                  /dev/sda                      sata   disk  isw_raid_member       447,1G
|-/dev/sda1                               /dev/sda1      /dev/sda              part  vfat                    600M
|-/dev/sda2                               /dev/sda2      /dev/sda              part  xfs                       1G
|-/dev/sda3                               /dev/sda3      /dev/sda              part  LVM2_member             154G
|-/dev/sda4                               /dev/sda4      /dev/sda              part  LVM2_member           269,2G
`-/dev/md126                              /dev/md126     /dev/sda              raid1                       424,8G
  |-/dev/md126p1                          /dev/md126p1   /dev/md126            md    vfat                    600M /boot/efi
  |-/dev/md126p2                          /dev/md126p2   /dev/md126            md    xfs                       1G /boot
  |-/dev/md126p3                          /dev/md126p3   /dev/md126            md    LVM2_member             154G
  | |-/dev/mapper/ol-root                 /dev/dm-0      /dev/md126p3          lvm   xfs                      30G /
  | |-/dev/mapper/ol-swap                 /dev/dm-1      /dev/md126p3          lvm   swap                     24G [SWAP]
  | `-/dev/mapper/ol-u01                  /dev/dm-11     /dev/md126p3          lvm   xfs                     100G /u01
  `-/dev/md126p4                          /dev/md126p4   /dev/md126            md    LVM2_member           269,2G
/dev/sdb                                  /dev/sdb                      sata   disk  isw_raid_member       447,1G
|-/dev/sdb1                               /dev/sdb1      /dev/sdb              part  vfat                    600M
|-/dev/sdb2                               /dev/sdb2      /dev/sdb              part  xfs                       1G
|-/dev/sdb3                               /dev/sdb3      /dev/sdb              part  LVM2_member             154G
|-/dev/sdb4                               /dev/sdb4      /dev/sdb              part  LVM2_member           269,2G
`-/dev/md126                              /dev/md126     /dev/sdb              raid1                       424,8G
  |-/dev/md126p1                          /dev/md126p1   /dev/md126            md    vfat                    600M /boot/efi
  |-/dev/md126p2                          /dev/md126p2   /dev/md126            md    xfs                       1G /boot
  |-/dev/md126p3                          /dev/md126p3   /dev/md126            md    LVM2_member             154G
  | |-/dev/mapper/ol-root                 /dev/dm-0      /dev/md126p3          lvm   xfs                      30G /
  | |-/dev/mapper/ol-swap                 /dev/dm-1      /dev/md126p3          lvm   swap                     24G [SWAP]
  | `-/dev/mapper/ol-u01                  /dev/dm-11     /dev/md126p3          lvm   xfs                     100G /u01
  `-/dev/md126p4                          /dev/md126p4   /dev/md126            md    LVM2_member           269,2G
/dev/nvme0n1                              /dev/nvme0n1                  nvme   disk                          5,8T
`-/dev/nvme0n1p1                          /dev/nvme0n1p1 /dev/nvme0n1   nvme   part  LVM2_member             5,8T
  |-/dev/mapper/vg_oracle-u02_zvk_muc     /dev/dm-2      /dev/nvme0n1p1        lvm   ext4                    2,5T /u02/oradata/ZVK_MUC
  |-/dev/mapper/vg_oracle-u02_zvk_da      /dev/dm-3      /dev/nvme0n1p1        lvm   ext4                   1000G /u02/oradata/ZVK_DA
  |-/dev/mapper/vg_oracle-u02_bsv         /dev/dm-4      /dev/nvme0n1p1        lvm   ext4                   1000G /u02/oradata/BSV
  |-/dev/mapper/vg_oracle-u02_kvv         /dev/dm-5      /dev/nvme0n1p1        lvm   ext4                    500G /u02/oradata/KVV
  `-/dev/mapper/vg_oracle-u02_daisy       /dev/dm-6      /dev/nvme0n1p1        lvm   ext4                    500G /u02/oradata/DAISY
/dev/nvme3n1                              /dev/nvme3n1                  nvme   disk                          2,9T
/dev/nvme1n1                              /dev/nvme1n1                  nvme   disk                          5,8T
`-/dev/nvme1n1p1                          /dev/nvme1n1p1 /dev/nvme1n1   nvme   part  LVM2_member             5,8T
  |-/dev/mapper/vg_oracle-u02_vip         /dev/dm-7      /dev/nvme1n1p1        lvm   ext4                    500G /u02/oradata/VIP
  |-/dev/mapper/vg_oracle-u02_vbo--portal /dev/dm-8      /dev/nvme1n1p1        lvm   ext4                    500G /u02/oradata/VBO-Portal
  |-/dev/mapper/vg_oracle-u02_bsv--portal /dev/dm-9      /dev/nvme1n1p1        lvm   ext4                    500G /u02/oradata/BSV-Portal
  `-/dev/mapper/vg_oracle-u02_vbo         /dev/dm-10     /dev/nvme1n1p1        lvm   ext4                    500G /u02/oradata/VBO
/dev/nvme4n1                              /dev/nvme4n1                  nvme   disk                          2,9T
/dev/nvme2n1                              /dev/nvme2n1                  nvme   disk                          2,9T
`-/dev/nvme2n1p1                          /dev/nvme2n1p1 /dev/nvme2n1   nvme   part  ext4                    2,9T /u03/flashdata/ZVK_MUC
/dev/nvme5n1                              /dev/nvme5n1                  nvme   disk                          2,9T
  • Description of the issue (ideally so that others can reproduce it):

My analysis of the attached files revealed the following facts:

ReaR confuses DM (Device Mapper Multipathing) multipathing devices with NVMe™ (Non-volatile Memory Express™) multipathing devices.

In the original disklayout.conf file, ReaR correctly detects the following DM multipath devices (Sata devices):

/dev/sda (sata)
/dev/sdb (sata)

ReaR also correctly detects the following (NVMe™) multipathing devices:

/dev/nvme0n1 (nvme)
/dev/nvme1n1 (nvme)
/dev/nvme2n1 (nvme)
/dev/nvme3n1 (nvme)
/dev/nvme4n1 (nvme)
/dev/nvme5n1 (nvme)

When testing disaster recovery, ReaR can no longer correctly reconstruct the multipathing devices.

/dev/sdb correctly recognizes ReaR as a SATA device.
/dev/sda does not correctly recognize ReaR as a SATA, but incorrectly as an NVME device
ReaR incorrectly links /dev/sda (sata) to /dev/nvme0n1 (nvme)

ReaR then fails with the error message during disaster recovery
ERROR: No file system mounted on mnt/local

AUTOEXCLUDE_MULTIPATH=n does not solve the problem.

What measures are required for proper disaster recovery?

jsmeix commented at 2024-05-22 12:33:

I am not at all a multipath expert and
I know nothing at all about "NVMe multipathing"
so only as an offhanded side note regarding

ReaR confuses DM (Device Mapper Multipathing) multipathing devices
with NVMe™ (Non-volatile Memory Express™) multipathing devices.

cf. starting at
https://github.com/rear/rear/issues/3085#issuecomment-1890616502
which - unfortunately - did not further proceed there.

So current ReaR may not work with "NVMe multipathing"
because "NVMe multipathing" may require some special
treatment in ReaR which is currently not implemented.

[Export of Github issue for rear/rear.]