#1883 Issue `closed`: How to find latest Rear version available from official SUSE repository¶

Labels: support / question, fixed / solved / done

ccjung opened issue at 2018-07-30 20:25:¶

Relax-and-Recover (ReaR) Issue Template¶

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

ReaR version ("/usr/sbin/rear -V"):
Relax-and-Recover 2.1-git.2345.30099a98.xorrisofschrpboot / 2017-07-21
OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"):
cat /etc/os-release

NAME="SLES"
VERSION="12-SP3"
VERSION_ID="12.3"
PRETTY_NAME="SUSE Linux Enterprise Server 12 SP3"
ID="sles"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:12:sp3"

ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

# Default is to create Relax-and-Recover rescue media as ISO image
# set OUTPUT to change that
# set BACKUP to activate an automated (backup and) restore of your data
# Possible configuration values can be found in /usr/share/rear/conf/default.conf
#
# This file (local.conf) is intended for manual configuration. For configuration
# through packages and other automated means we recommend creating a new
# file named site.conf next to this file and to leave the local.conf as it is.
# Our packages will never ship with a site.conf.
AUTOEXCLUDE_MULTIPATH=n
BOOT_OVER_SAN=y
REAR_INITRD_COMPRESSION=lzma
OUTPUT=ISO
ISO_MAX_SIZE=4000
BACKUP=NETFS
BACKUP_URL=iso:///iso_fs/REAR_BACKUP
ISO_DIR=/iso_fs/REAR_ISO
TMPDIR=/iso_fs/REAR_TEMP
OUTPUT_URL=null
BOOT_FROM_SAN=y
EXCLUDE_MOUNTPOINTS=( /iso_fs /opt/IBM/ITM /opt/commvault /opt/splunkforwarder /opt/teamquest /var/opt/BESClient /hana/backup /hana/data /hana/log /hana/shared /usr/sap /usr/sap/basis /usr/sap/srm /PA_backup )
EXCLUDE_COMPONENTS=( /dev/mapper/20017380034880209 /dev/mapper/2001738003488020a /dev/mapper/2001738003488020b /dev/mapper/2001738003488020c /dev/mapper/2001738003488020d /dev/mapper/2001738003488020e /dev/mapper/2001738003488020f /dev/mapper/20017380034880210 /dev/mapper/20017380034880211 /dev/mapper/20017380034880212 /dev/mapper/20017380034880213 /dev/mapper/20017380034880214 /dev/mapper/20017380034880215 /dev/mapper/20017380034880216 )

## SLES12-SP2

REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" snapper chattr lsattr )

COPY_AS_IS=( "${COPY_AS_IS[@]}" /usr/lib/snapper/installation-helper /etc/snapper/config-templates/default )

for subvol in $(findmnt -n -r -t btrfs | cut -d ' ' -f 1 | grep -v '^/$' | egrep -v 'snapshots|crash') ; do
        BACKUP_PROG_INCLUDE=( "${BACKUP_PROG_INCLUDE[@]}" "$subvol" )
done

POST_RECOVERY_SCRIPT=( 'if snapper --no-dbus -r $TARGET_FS_ROOT get-config | grep -q "^QGROUP.*[0-9]/[0-9]" ; then snapper --no-dbus -r $TARGET_FS_ROOT set-config QGROUP= ; snapper --no-dbus -r $TARGET_FS_ROOT setup-quota && echo snapper setup-quota done || echo snapper setup-quota failed ; else echo snapper setup-quota not used ; fi' )

Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR): PowerVM LPAR
System architecture (x86 compatible or PPC64/PPC64LE or what excat ARM device): PPC64LE
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):bootloader GRUB
Description of the issue (ideally so that others can reproduce it):
FYI. I am an IBM AIX system administrator so not too familiar with SUSE Linux.

This is from our SMT server:

# zypper search rear

Refreshing service 'IBM_DLPAR_Utils_for_SLE_12_ppc64le'.
Refreshing service 'IBM_DLPAR_sdk_for_SLE_12_ppc64le'.
Refreshing service 'SUSE_Linux_Enterprise_Server_12_SP2_ppc64le'.
Loading repository data...
Reading installed packages...

S  | Name | Summary                                                    | Type
---+------+------------------------------------------------------------+--------
i+ | rear | Relax-and-Recover is a Linux disaster recovery and syste-> | package

We would like to know if there is a later version of Rear from an official SUSE repository (newer than the 2.1 version which we are using)..

How do I find latest Rear version available from official SUSE repository?

Is there a repository which I should add to get the Rear software?

Thank you.

Work-around, if any:

jsmeix commented at 2018-07-31 09:17:¶

@suseusr168
for questions about what RPM software packages are officially available from SUSE
for SUSE Linux Enterprise products (like SLES-HA) you need to get in contact
with your official SUSE suppport or customer help contact because such things
may depend on what contract you have with SUSE.

Here at ReaR upstream we cannot answer such questions.

See in particular the related isssue
https://github.com/rear/rear/issues/1506

Our ReaR upstream RPM packages are available via the openSUSE build service
cf. http://relax-and-recover.org/download/
but the openSUSE build service is not an official repository for RPM packages
that are officially supported by SUSE for SUSE Linux Enterprise products.

See also the section
"RPM packages for disaster recovery" in
https://en.opensuse.org/SDB:Disaster_Recovery
what rear* RPM packages are available for SUSE Linux Enterprise 12.

jsmeix commented at 2018-07-31 09:23:¶

@suseusr168
our currently latest official SLEHA12 RPM package is rear118a
which is - simply put - too old for the POWER architecture
because since rear118a was made @schabrolles did tons of
enhancements and fixes in particular for the POWER architecture
so that one should use the current up-to-date ReaR on POWER.

ccjung commented at 2018-07-31 12:21:¶

@jsmeix,
Thank you for your reply.

We have been using the version 2.1 which @schabrolles has provided to us.
We are trying to find out if there is a newer and officially supported version from SUSE.

We don't know what command to use to find out what version of Rear is available from SUSE.

When I run the command "zypper search rear" on our SMT server , it seems to show the version we have installed.

Is there another repository name to use to check what version of Rear is currently available to install from the SUSE repositories?

Thanks again.

jsmeix commented at 2018-07-31 13:43:¶

@suseusr168
the latest officially supported version from SUSE
is in the rear118a RPM package for SLEHA12
and in the rear23a RPM package for SLEHA15
but in particular rear118a is nowadays in practice
often too old for the POWER architecture.

The official rear118a RPM package should somewhat work on POWER
but compared to what @schabrolles enhanced and fixed since that time
the official rear118a RPM package behaves poor on POWER.

Accordingly it is good to use a newer ReaR version on POWER
but nowadays even ReaR version 2.1 is meanwhile a bit old
because the latest ReaR GitHub master code contains a few
but possibly critical general fixes (not only on POWER and
not only for SUSE) so that I would recommend to use our latest
ReaR GitHub master code because that one should also become
an official supported version via the updated rear23a RPM package
from SUSE, cf.
https://github.com/rear/rear/issues/1506#issuecomment-409153043

Simply put I think you have to wait until the updated rear23a RPM package
became officially provided for SLEHA15 SLEHA12 and SLEHA11 to get
an up-to-date ReaR version that is officially supported by SUSE.

I know basically noting about the SUSE Subscription Management Tool (SMT)
and its SMT server setup so that I cannot help you in this area.

jsmeix commented at 2018-07-31 14:59:¶

@suseusr168
if you like to try out my above mentioned updated rear23a RPM package
on your SLES12-SP3 PPC64LE system:

I let that package build right now on the public openSUSE build service
wherefrom you could download the current built RPM package
rear23a-2.3.a-1.1.ppc64le.rpm (the RPM release number 1.1 varies) from
http://download.opensuse.org/repositories/home:/jsmeix/SLE_12_SP3/ppc64le/

The openSUSE build service is not an official repository for RPM packages
that are officially supported by SUSE for SUSE Linux Enterprise products
so that in particular any RPM packages from
http://download.opensuse.org/repositories/home:/jsmeix/
are not officially supported by SUSE, cf.
https://build.opensuse.org/project/show/home:jsmeix

But this way you could right now try out what is intended to become
the officially provided one for SLEHA15 SLEHA12 and SLEHA11
provided no serious issues appear with it.

And now my price for that service ;-)
I would very much appreciate it if you try out and test and verify
whether or not that updated rear23a RPM package actually works
for you on your particular system in your particular environment
and provide feedback to us here at ReaR upstream.
Many thanks in advance for testing it!

ccjung commented at 2018-07-31 17:12:¶

I can help with testing.

I tried to install it on our SLES12 SP3 LPAR and it failed with the following messages:

rpm -ihv rear23a-2.3.a-1.1.ppc64le.rpm

warning: rear23a-2.3.a-1.1.ppc64le.rpm: Header V3 DSA/SHA1 Signature, key ID 250f3907: NOKEY
error: Failed dependencies:
        lsb-release is needed by rear23a-2.3.a-1.1.ppc64le

jsmeix commented at 2018-08-01 06:57:¶

@suseusr168
the message tells it:
The RPM capability lsb-release is needed/required
by the RPM package rear23a-2.3.a-1.1.ppc64le.
I.e. you need an RPM package that provides lsb-release
and this is provided by the RPM package lsb-release.
This is no tautology:
A needed RPM capability (usually called RPM requirement) named foo
can be provided by a totally different named RPM package bar,
e.g.: /usr/bin/cat is provided by the RPM package named coreutils

# rpm -q --whatprovides /usr/bin/cat
coreutils-8.25-12.8.x86_64

Simply put:
Install the RPM package lsb-release before you install rear23a.

jsmeix commented at 2018-08-01 10:01:¶

I think https://github.com/rear/rear/issues/1884
could be a serious issue so that I would have to
update again my updated rear23a RPM package for SUSE
because I assume disks with 4K block size (both for logical and physical blocks)
which happens at least with some IBM harddisks
are an important use case where ReaR must "just work".

ccjung commented at 2018-08-01 22:38:¶

I installed the pre-req rpm lsb-release and now able install your Rear version: rear23a-2.3.a-1.1.ppc64le.rpm

rear -V
Relax-and-Recover 2.4 / Git
ps1dca0u:~ # cat /etc/os-release
NAME="SLES"
VERSION="12-SP3"
VERSION_ID="12.3"
PRETTY_NAME="SUSE Linux Enterprise Server 12 SP3"
ID="sles"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:12:sp3"

I was able to create the Rear ISO backup file and restored the backup to another LPAR.
I have attached the restore log.
Rear23a.Restore.log

I noticed there were a few "error: unrecognized number." during the restore menu. I just pressed enter.

What are the differences between the Rear Version (A,B,C) below?
A. Your version rear23a-2.3.a-1.1.ppc64le.rpm.
I noticed it is 2.4 when I run a rear -V
rpm -qa|grep rear
rear23a-2.3.a-1.1.ppc64le

rear -V
Relax-and-Recover 2.4 / Git

B. Version provided by SUSE support
https://software.opensuse.org/download.html?project=Archiving&package=rear#manualSLE
rear-2.3-74.2.ppc64le.rpm

C. The version from http://download.opensuse.org/repositories/Archiving:/Backup:/Rear:/Snapshot/openSUSE_Factory_PowerPC/ppc64le/

When I installed this version : rear-2.4-1.git.0.aa7b197.unknown.ppc64le.rpm, I didn't see any pre-req rpm lsb-release.

When I tried to install your 2.3.a-1.1 and the SUSE 2.3-74.2, it required the lsb-release pre-req.

Is the lsb-release pre-req required now?

Thanks.

jsmeix commented at 2018-08-02 09:36:¶

Regarding error: unrecognized number:

The

Welcome to GRUB!

error: unrecognized number.

                             GNU GRUB  version 2.02

error messages are from GRUB2 as far as it looks.

I don't know how booting with GRUB2 happens on PPC64LE PowerVM LPAR
(in general I am really not at all a bootloader expert) so that I cannot
really help here.

I had a similar (but different error in GRUB2) while I worked on
https://github.com/rear/rear/issues/1828
therein see the entries that contain error: invalid environment block.
Accordingly I have now in usr/share/rear/conf/default.conf

# Move away outdated information from previous boot via GRUB2
# to avoid a possible GRUB2 "error: invalid environment block."
# cf. https://github.com/rear/rear/issues/1828
# GRUB2 can use /boot/grub/grubenv or /boot/grub2/grubenv
# to remember a small amount of information from one boot to the next.
# But on a by "rear recover" recreated system there is no such thing
# as a meaningful previous boot (because "rear recover" recreates a system
# by reinstalling it completely from scratch) so that there is no meaningful
# use-case to remember any information from a possible previous boot,
# cf. https://github.com/rear/rear/issues/1828#issuecomment-399050741
# Nevertheless that information from the last boot on the original system
# could be still of interest for the user so that it is not deleted:
BACKUP_RESTORE_MOVE_AWAY_FILES=( /boot/grub/grubenv /boot/grub2/grubenv )

But I have no idea what the root cause of an error: unrecognized number
GRUB2 error message could be - in particular because that error message
is so very explicit and helpful - GRRR! - what number is unrecognized where?
I really hate useless messages that tell nothing about the current actual
environment where a particular issue happened - e.g. pointlessness like
error: file not found - what file was not found in this particular case?

Regarding different RPM packages that provide ReaR:

My version rear23a-2.3.a-1.1.ppc64le.rpm contains the currently latest GitHub
master code - i.e. it provides you the most up-to-date ReaR.

As far as I know any RPM package from any 'opensuse.org' site is never
officially supported by SUSE support for SUSE Linux Enterprise products.
As far as I know RPM packages that are officially supported by SUSE
for SUSE Linux Enterprise products should come from a 'suse.com' site.

Whether or nor lsb-release is required by a RPM package depends on
what is defined as requirement for the particular RPM package in the
so called RPM 'spec' file.

Different rear RPM packages have (slightly) different RPM 'spec' files,
cf. https://github.com/rear/rear/issues/1855#issuecomment-404100990

My rear RPM packages require lsb-release in their 'spec' files
because on older SLES and openSUSE versions it is perhaps still needed
while on newer SLES and openSUSE versions it should be no longer needed
but - to be on the safe side - I simply keep the RPM requirement for lsb-release.
It does not hurt to have /usr/bin/lsb_release installed (it is a generally useful tool)
and in some cases it could be still needed in some cases and then I think
it is better to have it installed than to let things possibly go wrong in ReaR.

FYI:
/usr/bin/lsb_release is called as fallback in the SetOSVendorAndVersion
function in usr/share/rear/lib/config-functions.sh and also in
usr/share/rear/rescue/GNU/Linux/990_sysreqs.sh
When the SetOSVendorAndVersion function results wrong values
things could go really wrong in ReaR at certain but possibly critical places.
As far as I remember @schabrolles had "some special kind of fun"
with code that depends on the values of variables that are set by
the SetOSVendorAndVersion function like OS_MASTER_VENDOR
where code in usr/share/rear/lib/layout-functions.sh depends on.

Regarding https://github.com/rear/rear/issues/1884 (disk with 4K block size):

I would like to know what on your PPC64LE PowerVM LPAR
both the logical and physical block size of your system disk is
i.e. what the output of the following commands on your system is:

# cat /sys/block/sda/queue/logical_block_size

# cat /sys/block/sda/queue/physical_block_size

Replace sda with what your system disk device basename actually is,
cf. the 'ReaR/PPC Disk size issue' mails on the rear-devel mailing list like
http://lists.relax-and-recover.org/pipermail/rear-devel/2018-July/001643.html

jsmeix commented at 2018-08-02 10:04:¶

@suseusr168
in your Rear23a.Restore.log I found an interesting example where
Verifying md5sums of the files in the Relax-and-Recover rescue system
failed with

./etc/udev/rules.d/70-persistent-net.rules: FAILED
md5sum: ./dev/.SRC-Semaphore: No such file or directory
./dev/.SRC-Semaphore: FAILED open or read
md5sum: WARNING: 1 listed file could not be read
md5sum: WARNING: 1 computed checksum did NOT match
Possibly corrupted Relax-and-Recover rescue system
Proceeding 'bona fide' nevertheless...

I will investigate what modifies /etc/udev/rules.d/70-persistent-net.rules
but I assume this is all o.k. so that I can exclude that by default
It also happened one single time to me that I got a failure
with an unusual file in /dev/ so that I think we may better exclude
all files in /dev/ to avoid such false alarm.

jsmeix commented at 2018-08-02 10:19:¶

@suseusr168
FYI:
When you run rear in debugscript mode '-D' you get at the end the message

You should also rm -Rf /tmp/rear.6CbGwlbsbG0uTrM

because in debugscript mode rear does not automatically remove
its temporary working area.
But after "rear -D recover" you do not need to remove it because
"rear -D recover" runs within the ReaR recovery system
and that runs only inside the main memory of your system
so that for "rear -D recover" rear's working area in /tmp/
is only inside the main memory of your system
where all goes away during reboot.
Within the ReaR recovery system the files of the recreated system
are under "/mnt/local" where the tree of filesystems on the disks is mounted.
In contrast when you run "rear -D mkbackup/mkrescue"
rear's working area in /tmp/ stays there until you remove it maunally
or some automated /tmp/ cleanup stuff removes it - if you have such.

jsmeix commented at 2018-08-02 10:44:¶

@schabrolles
I would like that you have a look at the Rear23a.Restore.log from @suseusr168
cf. https://github.com/rear/rear/issues/1883#issuecomment-409747779

For me things look o.k. but I just like a confirmation from you.

In particular I would like to know whether or not the migration mode
behaves sufficiently well with multipath:

Setting up multipathing
Activating multipath
multipath activated
Starting multipath daemon
multipathd started
Listing multipath device found
 size=125G
 size=125G
 size=50G
 size=11G
Comparing disks
Ambiguous possible target disks need manual configuration (more than one with same size found)
Switching to manual disk layout configuration
Using /dev/mapper/360050764008100b5a8000000000000ef (same size) for recreating /dev/mapper/360050764008100513800000000000038
Current disk mapping table (source => target):
  /dev/mapper/360050764008100513800000000000038 => /dev/mapper/360050764008100b5a8000000000000ef

UserInput -I LAYOUT_MIGRATION_CONFIRM_MAPPINGS needed in /usr/share/rear/layout/prepare/default/300_map_disks.sh line 275
Confirm or edit the disk mapping
1) Confirm disk mapping and continue 'rear recover'
2) n/a
3) Edit disk mapping (/var/lib/rear/layout/disk_mappings)
4) Use Relax-and-Recover shell and return back to here
5) Abort 'rear recover'
(default '1' timeout 300 seconds)

Item 2) n/a is because in layout/prepare/default/300_map_disks.sh there is

    is_completely_identical_layout_mapping && choices[1]="Confirm identical disk mapping and proceed without manual configuration" || choices[1]="n/a"

cf. https://github.com/rear/rear/issues/1857

ccjung commented at 2018-08-02 14:06:¶

Here are our logical and physical block size:
cat /sys/block/sda/queue/logical_block_size
512
cat /sys/block/sda/queue/physical_block_size
512

schabrolles commented at 2018-08-03 13:09:¶

@jsmeix I got a look at the log file sent by @suseusr168.

Only a small Bug here... I think it is also present on master (small bug ... not blocking)

When activating multipath, disk name are not shown (only size). I think it is because multipath firendly name is not use. I have to look how to deal with it.

Listing multipath device found
 size=125G
 size=125G
 size=50G
 size=11G

For the REAL stuff ... it looks good to me, migration is OK as we use a different system to restore. Grub2 was reinstalled correctly, ramdisk regenerated and bootlist setup.... Works perfectly +1:

Patching /etc/default/grub_installdevice: Replacing [/dev/disk/by-id/dm-name-360050764008100513800000000000038-part1] by [/dev/disk/by-id/dm-name-360050764008100b5a8000000000000ef-part1]

Found PPC PReP boot partition /dev/mapper/360050764008100b5a8000000000000ef-part1 - installing GRUB2 there

Running mkinitrd...
Recreated initrd (/sbin/mkinitrd).

Set LPAR bootlist to /dev/sdm /dev/sde /dev/sdi /dev/sda

ccjung commented at 2018-08-03 14:46:¶

@schabrolles : Thanks for verifying this.

@jsmeix ; When will your rear23a-2.3.a-1.1.ppcle.rpm be included with the official SUSE release of SLES12 SP3?

schabrolles commented at 2018-08-03 15:24:¶

@suseusr168, could you please send me the output of multipath -l
I just want to check why device name does not appear during multipath activation.

ccjung commented at 2018-08-03 15:41:¶

This is from the system restored TO:

multipath -l

360050764008100b5a8000000000000f0 dm-1 IBM,2145
size=125G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=active
| |- 2:0:1:1 sdf 8:80  active undef unknown
| `- 3:0:1:1 sdn 8:208 active undef unknown
`-+- policy='service-time 0' prio=0 status=enabled
  |- 2:0:0:1 sdb 8:16  active undef unknown
  `- 3:0:0:1 sdj 8:144 active undef unknown
360050764008100b5a8000000000000ef dm-0 IBM,2145
size=125G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=active
| |- 2:0:0:0 sda 8:0   active undef unknown
| `- 3:0:0:0 sdi 8:128 active undef unknown
`-+- policy='service-time 0' prio=0 status=enabled
  |- 2:0:1:0 sde 8:64  active undef unknown
  `- 3:0:1:0 sdm 8:192 active undef unknown
360050764008100b5a8000000000000f2 dm-3 IBM,2145
size=50G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=active
| |- 2:0:1:3 sdh 8:112 active undef unknown
| `- 3:0:1:3 sdp 8:240 active undef unknown
`-+- policy='service-time 0' prio=0 status=enabled
  |- 2:0:0:3 sdd 8:48  active undef unknown
  `- 3:0:0:3 sdl 8:176 active undef unknown
360050764008100b5a8000000000000f1 dm-2 IBM,2145
size=11G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=active
| |- 2:0:0:2 sdc 8:32  active undef unknown
| `- 3:0:0:2 sdk 8:160 active undef unknown
`-+- policy='service-time 0' prio=0 status=enabled
  |- 2:0:1:2 sdg 8:96  active undef unknown
  `- 3:0:1:2 sdo 8:224 active undef unknown

This is from the system which the backup was created FROM:

multipath -l

360050764008100513800000000000080 dm-7 IBM,2145
size=48G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=active
| |- 2:0:0:7 sdh  8:112  active undef unknown
| `- 3:0:0:7 sdz  65:144 active undef unknown
`-+- policy='service-time 0' prio=0 status=enabled
  |- 2:0:1:7 sds  65:32  active undef unknown
  `- 3:0:1:7 sdai 66:32  active undef unknown
36005076400810051380000000000006d dm-1 IBM,2145
size=48G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=active
| |- 2:0:1:1 sdk  8:160  active undef unknown
| `- 3:0:1:1 sdac 65:192 active undef unknown
`-+- policy='service-time 0' prio=0 status=enabled
  |- 2:0:0:1 sdb  8:16   active undef unknown
  `- 3:0:0:1 sdr  65:16  active undef unknown
360050764008100513800000000000088 dm-8 IBM,2145
size=1.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=active
| |- 2:0:0:8 sdi  8:128  active undef unknown
| `- 3:0:0:8 sdaa 65:160 active undef unknown
`-+- policy='service-time 0' prio=0 status=enabled
  |- 2:0:1:8 sdu  65:64  active undef unknown
  `- 3:0:1:8 sdaj 66:48  active undef unknown
36005076400810051380000000000007f dm-6 IBM,2145
size=48G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=active
| |- 2:0:1:6 sdq  65:0   active undef unknown
| `- 3:0:1:6 sdah 66:16  active undef unknown
`-+- policy='service-time 0' prio=0 status=enabled
  |- 2:0:0:6 sdg  8:96   active undef unknown
  `- 3:0:0:6 sdy  65:128 active undef unknown
360050764008100513800000000000038 dm-0 IBM,2145
size=125G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=active
| |- 2:0:0:0 sda  8:0    active undef unknown
| `- 3:0:0:0 sdp  8:240  active undef unknown
`-+- policy='service-time 0' prio=0 status=enabled
  |- 2:0:1:0 sdj  8:144  active undef unknown
  `- 3:0:1:0 sdab 65:176 active undef unknown
36005076400810051380000000000007e dm-5 IBM,2145
size=25G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=active
| |- 2:0:0:5 sdf  8:80   active undef unknown
| `- 3:0:0:5 sdx  65:112 active undef unknown
`-+- policy='service-time 0' prio=0 status=enabled
  |- 2:0:1:5 sdo  8:224  active undef unknown
  `- 3:0:1:5 sdag 66:0   active undef unknown
36005076400810051380000000000007d dm-4 IBM,2145
size=25G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=active
| |- 2:0:1:4 sdn  8:208  active undef unknown
| `- 3:0:1:4 sdaf 65:240 active undef unknown
`-+- policy='service-time 0' prio=0 status=enabled
  |- 2:0:0:4 sde  8:64   active undef unknown
  `- 3:0:0:4 sdw  65:96  active undef unknown
36005076400810051380000000000007c dm-3 IBM,2145
size=25G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=active
| |- 2:0:0:3 sdd  8:48   active undef unknown
| `- 3:0:0:3 sdv  65:80  active undef unknown
`-+- policy='service-time 0' prio=0 status=enabled
  |- 2:0:1:3 sdm  8:192  active undef unknown
  `- 3:0:1:3 sdae 65:224 active undef unknown
36005076400810051380000000000007b dm-2 IBM,2145
size=25G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=active
| |- 2:0:1:2 sdl  8:176  active undef unknown
| `- 3:0:1:2 sdad 65:208 active undef unknown
`-+- policy='service-time 0' prio=0 status=enabled
  |- 2:0:0:2 sdc  8:32   active undef unknown
  `- 3:0:0:2 sdt  65:48  active undef unknown