sda" falsely recognized as multipath device (DM_MULTIPATH_DEVICE_PATH="1")¶

Labels: enhancement, bug, fixed / solved / done

jsmeix opened issue at 2019-12-11 14:34:¶

ReaR version ("/usr/sbin/rear -V"):
Current GitHub master code
OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"):
openSUSE Leap 15.1
ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

SSH_ROOT_PASSWORD="rear"
USE_DHCLIENT="yes"
PROGRESS_MODE="plain"
PROGRESS_WAIT_SECONDS=3
KEEP_BUILD_DIR="yes"
GRUB_RESCUE=y
OUTPUT=USB
BACKUP=NETFS
BACKUP_URL=usb:///dev/disk/by-label/REAR-000
USB_SUFFIX="ReaRbackup"
USB_DEVICE_FILESYSTEM_PERCENTAGE=10

Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR):
PC (my laptop)
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
x86_64
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
UEFI capable firmware in legacy BIOS mode
Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
local built-in rotating disk /dev/sda plus rotating USB disk /dev/sdb
Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT" or "lsblk -io NAME,KNAME,FSTYPE,SIZE,MOUNTPOINT"):

NAME        KNAME     PKNAME   TRAN   TYPE FSTYPE   SIZE MOUNTPOINT
/dev/sda    /dev/sda           sata   disk        931.5G 
|-/dev/sda1 /dev/sda1 /dev/sda        part vfat     500M 
|-/dev/sda2 /dev/sda2 /dev/sda        part          100M 
|-/dev/sda3 /dev/sda3 /dev/sda        part swap      20G [SWAP]
|-/dev/sda4 /dev/sda4 /dev/sda        part ext4     400G /
`-/dev/sda5 /dev/sda5 /dev/sda        part ext4     400G /other
/dev/sdb    /dev/sdb           usb    disk        465.8G 
`-/dev/sdb1 /dev/sdb1 /dev/sdb        part ext3    46.6G 
/dev/sr0    /dev/sr0           sata   rom          1024M

Description of the issue (ideally so that others can reproduce it):

During "rear -D mkrescue" I get in the log

+ source /root/rear.github.master/usr/share/rear/layout/save/GNU/Linux/200_partition_layout.sh
...
++ echo '2019-12-11 14:32:21.844992593 Saving disk partitions.'
2019-12-11 14:32:21.844992593 Saving disk partitions.
++ for disk in /sys/block/*
++ blockd=sda
++ [[ sda = hd* ]]
++ [[ sda = sd* ]]
++ is_multipath_path sda
++ '[' sda ']'
++ type multipath
++ multipath -c /dev/sda
++ Log 'Ignoring sda: it is a path of a multipath device'
++ echo '2019-12-11 14:32:21.851233460 Ignoring sda: it is a path of a multipath device'
2019-12-11 14:32:21.851233460 Ignoring sda: it is a path of a multipath device

so that my built-in system disk /dev/sda is not in disklayout.conf:

# Disk /dev/sdb
# Format: disk <devname> <size(bytes)> <partition label type>
#disk /dev/sdb 500107859968 msdos
# Partitions on /dev/sdb
# Format: part <device> <partition size(bytes)> <partition start(bytes)> <partition type|name> <flags> /dev/<partition>
#part /dev/sdb 50002395136 8388608 primary boot /dev/sdb1
# Filesystems (only ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs are supported).
# Format: fs <device> <mountpoint> <fstype> [uuid=<uuid>] [label=<label>] [<attributes>]
fs /dev/sda4 / ext4 uuid=cff8eaf4-2369-439b-8ef2-620dd515d767 label= blocksize=4096 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=16384 default_mount_options=user_xattr,acl options=rw,relatime,data=ordered
fs /dev/sda5 /other ext4 uuid=2aa61137-5fc3-4fca-8b8e-959bc4c3676d label= blocksize=4096 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=16384 default_mount_options=user_xattr,acl options=rw,relatime,data=ordered
# Swap partitions or swap files
# Format: swap <filename> uuid=<uuid> label=<label>
swap /dev/sda3 uuid=6d4a07be-f9d7-4e56-a141-75758754e822 label=

Interestingly it seems multipath -c /dev/sda thinks
/dev/sda should be a path in a multipath device

# multipath -v 3 -c /dev/sda ; echo $?

Dec 11 15:16:31 | set open fds limit to 1048576/1048576
Dec 11 15:16:31 | loading /lib64/multipath/libchecktur.so checker
Dec 11 15:16:31 | checker tur: message table size = 3
Dec 11 15:16:31 | loading /lib64/multipath/libprioconst.so prioritizer
Dec 11 15:16:31 | _init_foreign: foreign library "nvme" is not enabled
Dec 11 15:16:31 | sda: size = 1953525168
Dec 11 15:16:31 | sda: vendor = ATA
Dec 11 15:16:31 | sda: product = HGST HTS541010A9
Dec 11 15:16:31 | sda: rev = A590
Dec 11 15:16:31 | sda: h:b:t:l = 4:0:0:0
Dec 11 15:16:31 | sda: tgt_node_name = ata-5.00
Dec 11 15:16:31 | sda: uid_attribute = ID_SERIAL (setting: multipath internal)
Dec 11 15:16:31 | sda: uid = HGST_HTS541010A9E680_JB10001MKBPXNP (udev)
Dec 11 15:16:31 | sda: udev property ID_WWN whitelisted
Dec 11 15:16:31 | scope limited to HGST_HTS541010A9E680_JB10001MKBPXNP
Dec 11 15:16:31 | Initialized new file [/dev/shm/multipath/failed_wwids/.lock]
DM_MULTIPATH_DEVICE_PATH="1"
Dec 11 15:16:31 | unloading const prioritizer
Dec 11 15:16:31 | unloading tur checker
0

in contrast to /dev/sdb

# multipath -v 3 -c /dev/sdb ; echo $?

Dec 11 15:18:15 | set open fds limit to 1048576/1048576
Dec 11 15:18:15 | loading /lib64/multipath/libchecktur.so checker
Dec 11 15:18:15 | checker tur: message table size = 3
Dec 11 15:18:15 | loading /lib64/multipath/libprioconst.so prioritizer
Dec 11 15:18:15 | _init_foreign: foreign library "nvme" is not enabled
DM_MULTIPATH_DEVICE_PATH="0"
Dec 11 15:18:15 | unloading const prioritizer
Dec 11 15:18:15 | unloading tur checker
1

Workaround, if any:

Because I do not use multipath on my laptop

# multipath -v 3 -ll

Dec 11 15:21:04 | set open fds limit to 1048576/1048576
Dec 11 15:21:04 | loading /lib64/multipath/libchecktur.so checker
Dec 11 15:21:04 | checker tur: message table size = 3
Dec 11 15:21:04 | loading /lib64/multipath/libprioconst.so prioritizer
Dec 11 15:21:04 | _init_foreign: foreign library "nvme" is not enabled
Dec 11 15:21:04 | sda: size = 1953525168
Dec 11 15:21:04 | sda: vendor = ATA
Dec 11 15:21:04 | sda: product = HGST HTS541010A9
Dec 11 15:21:04 | sda: rev = A590
Dec 11 15:21:04 | sda: h:b:t:l = 4:0:0:0
Dec 11 15:21:04 | sda: tgt_node_name = ata-5.00
Dec 11 15:21:04 | sda: 56065 cyl, 255 heads, 63 sectors/track, start at 0
Dec 11 15:21:04 | sda: serial =       JB10001MKBPXNP
Dec 11 15:21:04 | sda: detect_checker = yes (setting: multipath internal)
Dec 11 15:21:04 | sda: path_checker = tur (setting: multipath internal)
Dec 11 15:21:04 | sda: checker timeout = 30 s (setting: kernel sysfs)
Dec 11 15:21:04 | sda: tur state = up
===== paths list =====
uuid hcil    dev dev_t pri dm_st chk_st vend/prod/rev        dev_st 
     4:0:0:0 sda 8:0   -1  undef undef  ATA,HGST HTS541010A9 unknown
Dec 11 15:21:04 | libdevmapper version 1.03.01 (2018-07-19)
Dec 11 15:21:04 | DM multipath kernel driver v1.13.0
Dec 11 15:21:04 | unloading const prioritizer
Dec 11 15:21:04 | unloading tur checker

the following hack helps me

--- a/usr/share/rear/lib/layout-functions.sh
+++ b/usr/share/rear/lib/layout-functions.sh

 function is_multipath_path {
-    [ "$1" ] && type multipath &>/dev/null && multipath -c /dev/$1 &>/dev/null
+    return 1
+    # [ "$1" ] && type multipath &>/dev/null && multipath -c /dev/$1 &>/dev/null
 }

and then "rear mkrescue" produces disklayout.conf as expected

# Disk /dev/sda
# Format: disk <devname> <size(bytes)> <partition label type>
disk /dev/sda 1000204886016 gpt
# Partitions on /dev/sda
# Format: part <device> <partition size(bytes)> <partition start(bytes)> <partition type|name> <flags> /dev/<partition>
part /dev/sda 524288000 1048576 rear-noname boot,legacy_boot,esp /dev/sda1
part /dev/sda 104857600 525336576 rear-noname bios_grub /dev/sda2
part /dev/sda 21474836480 630194176 rear-noname swap /dev/sda3
part /dev/sda 429496729600 22105030656 rear-noname none /dev/sda4
part /dev/sda 429496729600 451601760256 rear-noname none /dev/sda5
# Disk /dev/sdb
# Format: disk <devname> <size(bytes)> <partition label type>
#disk /dev/sdb 500107859968 msdos
# Partitions on /dev/sdb
# Format: part <device> <partition size(bytes)> <partition start(bytes)> <partition type|name> <flags> /dev/<partition>
#part /dev/sdb 50002395136 8388608 primary boot /dev/sdb1
# Filesystems (only ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs are supported).
# Format: fs <device> <mountpoint> <fstype> [uuid=<uuid>] [label=<label>] [<attributes>]
fs /dev/sda4 / ext4 uuid=cff8eaf4-2369-439b-8ef2-620dd515d767 label= blocksize=4096 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=16384 default_mount_options=user_xattr,acl options=rw,relatime,data=ordered
fs /dev/sda5 /other ext4 uuid=2aa61137-5fc3-4fca-8b8e-959bc4c3676d label= blocksize=4096 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=16384 default_mount_options=user_xattr,acl options=rw,relatime,data=ordered
# Swap partitions or swap files
# Format: swap <filename> uuid=<uuid> label=<label>
swap /dev/sda3 uuid=6d4a07be-f9d7-4e56-a141-75758754e822 label=

It seems the function is_multipath_path needs to be enhanced
to work more fail safe (or the multipath command or whatever
underlying things it uses are broken on openSUSE Leap 15.1).

jsmeix commented at 2019-12-11 15:14:¶

Proposal for a better workaround that works for my case
and that hopefully still works when multipath is used:

function is_multipath_path {
    # Return 'false' if there is no device as argument:
    test "$1" || return 1
    # Return 'false' if there is no multipath command:
    type multipath &>/dev/null || return 1
    # Return 'false' if multipath is not used.
    # Because "multipath -l" always returns zero exit code we check if it has real output
    # and we do this via "grep -q" so that no "multipath -l" output appears in the log:
    multipath -l | grep -q '[[:alnum:]]' || return 1
    # Check if a block device should be a path in a multipath device:
    multipath -c /dev/$1 &>/dev/null
}

with that I get during "rear -D mkrescue" in the log

++ Log 'Saving disk partitions.'
++ echo '2019-12-11 16:07:03.851031248 Saving disk partitions.'
2019-12-11 16:07:03.851031248 Saving disk partitions.
++ for disk in /sys/block/*
++ blockd=sda
++ [[ sda = hd* ]]
++ [[ sda = sd* ]]
++ is_multipath_path sda
++ test sda
++ type multipath
++ multipath -l
++ grep -q '[[:alnum:]]'
++ return 1

because multipath -l shows no output in my case.

jsmeix commented at 2019-12-13 14:33:¶

With https://github.com/rear/rear/pull/2299 merged
this issue should be avoided so I close this issue.

@schabrolles
if you find some time for ReaR I would appreciate it
if you could have a look here.

Perhaps you could imagine what the actual root cause could be
why on my laptop with a single built-in disk plus one USB disk
multipath -c /dev/sda falsely recognized it as multipath device.

jsmeix commented at 2022-10-06 06:22:¶

A colleague at SUSE got the following contradicting results
for DM_MULTIPATH_DEVICE_PATH from multipath -c /dev/sda
versus udevadm info --query=property --name /dev/sda
where /dev/sda is a SATA SSD drive
(excerpt what he wrote):

... multipath issue, for example on my KVM server

# multipath  -c /dev/sda
DM_MULTIPATH_DEVICE_PATH="1"

While

# udevadm info --query=property --name /dev/sda | grep MULTI
DM_MULTIPATH_DEVICE_PATH=0

/dev/sda is just a sata ssd drive

jsmeix commented at 2022-10-12 09:18:¶

A message from another colleague at SUSE:

multipath-tools is in "greedy" mode by default under SUSE.
Thus every non-blacklisted device is considered a multipath path.
This explains DM_MULTIPATH_DEVICE_PATH="1".
There's a subtle difference in how "-c" and "-u" operate;
"-u" checks whether multipathd is enabled
and sets DM_MULTIPATH_DEVICE_PATH="0" if not.
"-c" doesn't do this check.
Arguably "-c" and "-u" should behave equally
but they don't; this is historical.

In general, "-c" shouldn't be used any more.
All relevant tests use "-u".
rear should look at the udev properties
rather than running "multipath -c"

jsmeix commented at 2022-10-12 09:50:¶

The problem with the current "fix" in
https://github.com/rear/rear/pull/2299
is that it is not a fix of the actual problem of this issue
but it only avoids this issue when multipath is not used.

But this issue can still happen when multipath is used.
When multipath is used the is_multipath_path function
in usr/share/rear/lib/layout-functions.sh
still calls e.g. multipath -c /dev/sda which still
falsely reports normal disks as multipath devices
(because of its "greedy" mode - at least in SUSE).

So ReaR should look at the udev properties
rather than running "multipath -c"
for example like

udevadm info -q property -n /dev/sda | grep 'DM_MULTIPATH_DEVICE_PATH=1'

to check if a disk is a multipath device.

The question is how far that works backward compatible
i.e. for how old Linux systems that works.

[Export of Github issue for rear/rear.]

#2298 Issue closed: Normal /dev/sda by "multipath -c /dev/sda" falsely recognized as multipath device (DM_MULTIPATH_DEVICE_PATH="1")¶