#1540 Issue `closed`: ReaR does not support RAID 1 mdraid Intel IMSM/RST based firmware RAID containers¶

Labels: enhancement, needs sponsorship, special hardware or VM, no-issue-activity

v-vidr opened issue at 2017-10-23 02:48:¶

Relax-and-Recover (ReaR) Issue Template¶

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

rear version (/usr/sbin/rear -V):
1.17, 2.0, 2.2
OS version (cat /etc/rear/os.conf or lsb_release -a):
RHEL and SUSE SAP 12 SP1
rear configuration files (cat /etc/rear/site.conf or cat /etc/rear/local.conf):

OUTPUT=ISO
BACKUP=NETFS
BACKUP_URL=cifs://1.1.1.1/backup
NETFS_KEEP_OLD_BACKUP_COPY=yes
EXCLUDE_VG=( vgHANA-data-HC2 vgHANA-data-HC3 vgHANA-log-HC2 vgHANA-log-HC3 vgHANA-shared-HC2 vgHANA-shared-HC3 )
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp/*' '/var/crash' '/hana' '/usr/sap' '/proc')
BACKUP_OPTIONS="cred=/etc/rear/cifs,vers=2.0"

Are you using legacy BIOS or UEFI boot?
UEFI
Brief description of the issue:
Restored failed with device or resource busy
Work-around, if any:
Earlier we received error invalid number of devices. I have modified diskrestore.sh and defined 2 devices.

We have configured two Physical Hard drives with software Raid and raid level is 1.

I am able successfully backed up data and rescue image. we tried to restore the complete os using rescue image.

We received error invalid number of devices and I have changed to 2 devices in diskrestore.sh

later we received device or resource busy error.

All screenshots are attached
device
busy
invalid_logs
invalid_number_devices

Is I am missing anything in my configuration or restore procedure.

jsmeix commented at 2017-10-27 10:05:¶

@vsrdigi
that nobody replied up to now indicates that your particular issue
is a "new and somewhat inexplicable special case" where
currently nobody (including myself) has a good idea
what the root cause could be.

When during "rear recover" a "Device resource busy" error
appears for a valid device node like "/dev/sda" that should
not be already in use - but actually it is already in use -
my blind guess is that you run "rear recover" not on
a machine with a perfectly clean harddisk "/dev/sda"
but on a machine where "/dev/sda" was already used before.

For example when you do "rear recover" on the same machine
where you did "rear mkbackup" before, then "rear recover" runs
on a machine where "/dev/sda" still contails all the data of the
original system, in particular same partitioning data and possibly
same "special data" of whatever higher-level block devices.

When you run "rear recover" with an already used harddisk,
see for general information what issues that may cause
https://github.com/rear/rear/issues/799
and also follow the links therein.

In general how to debug when "rear recover" fails
see for some very basic information the section
"Debugging issues with Relax-and-Recover" in
https://en.opensuse.org/SDB:Disaster_Recovery

Some general background information how things work
that should help to better understand how one can
debug an issue when "rear recover" fails:

In general how to debug when a particular ReaR script fails, see
https://github.com/rear/rear/issues/1532#issuecomment-336383117
i.e. add a "rear_shell" call directly before the command that fails
(this even works inside the diskrestore.sh script, see below).

But when "rear recover" fails things are more complicated:

During "rear mkbackup" or "rear mkrescue" certain ReaR scripts
create the disklayout.conf file and the ReaR recovery system
where that system-specific disklayout.conf file gets included.

During "rear recover" certain ReaR scripts generate
the system-specific diskrestore.sh script according
to the data in the disklayout.conf file in the recovery system
and finally that generated diskrestore.sh script is run
which does the actual work.

Therefore when "rear recover" fails the root cause could be:

1.)
In the scripts that run during "rear mkbackup" or "rear mkrescue"
to create the disklayout.conf file when that file contains wrong data
or when the scripts have insufficient code to create right data
for this or that special case.

2.)
In the scripts that run during "rear recover" that generate
the diskrestore.sh script when there are wrong commands
or when there is insufficient code in the diskrestore.sh script
to recreate the disk layout for this or that special case.

jsmeix commented at 2017-10-27 10:21:¶

Again only a blind guess:
In particular regarding "disk is busy"
I found "mdadm ... locks the disk" in
layout/prepare/GNU/Linux/100_include_partition_code.sh

if grep -q md /proc/mdstat &>/dev/null; then
    mdadm --stop -s >&2 || echo "stop mdadm failed"
    # Prevent udev waking up mdadm later.
    # Reasoning: At least on RHEL6 when parted created a raid partition on disk,
    # udev (via /lib/udev/rules.d/65-md-incremental.rules) wakes up mdadm which locks the disk,
    # so further parted commands with the disk will fail since the disk is busy now.
    # The /lib/udev/rules.d/65-md-incremental.rules detects anaconda (the Red Hat installer),
    # and if it find itself running under anaconda, it will not run.
    # Accordingly also for other installers (in particular the ReaR installer)
    # this rule should not be there (and other Linux distros probably do not have it)
    # which means removing it is the right solution to make ReaR work also for RHEL6:
    if [ -e /lib/udev/rules.d/65-md-incremental.rules ] ; then
        rm -f /lib/udev/rules.d/65-md-incremental.rules || echo "rm 65-md-incremental.rules failed"
    fi
fi

v-vidr commented at 2017-10-28 12:51:¶

I tried below options:

Option 1:
a. wipefs -a -f /dev/sad{a,b}

b. zero-superblock using mdadm

c. Performed recover and result is failed with /dev/sda device or resource busy.

Option2:
a. Reboot machine

b. wipefs -a -f /dev/sad{a,b}

c. zero-superblock using mdadm

before starting recover performed step 1 and step 2

c. Performed recover and result is failed with /dev/sda device or resource busy.

Option3:

a. reboot machine

b. tried creating manual raid and result is failed with /dev/sda device or resource busy.

jsmeix commented at 2017-11-09 11:37:¶

Meanwhile I got some background information
via a SUSE-internal issue that we have for this.

I try to explain (as far as I - as a RAID noob - understand it)
what actually goes on here:

The used hardware has two SSD disks
using Intel Rapid Storage Technology
that is used as mdraid IMSM based containers.

The Intel Matrix Storage Manager (IMSM) has been
superseded by the Intel Rapid Storage Technology (RST).
It is a firmware based RAID located in the Intel chip set
and mdadm controls that firmware.

See
https://en.wikipedia.org/wiki/Intel_Matrix_RAID
which reads (excerpt)

Matrix RAID is a computer storage
technology marketed by Intel.
It is a firmware, rather than hardware
or software, RAID system.

so that it is not "software RAID" but "firmware RAID"
that is called "hardware-assisted software RAID"
or "hybrid model RAID" or even "fake RAID"
cf. the section "Firmware- and driver-based" in
https://en.wikipedia.org/wiki/RAID#Implementations

Accordingly this issue is in the end the same as
https://github.com/rear/rear/issues/1094
and
https://github.com/rear/rear/issues/1460

Currently ReaR has no support at all for that.

jsmeix commented at 2017-11-10 09:10:¶

For example how one could check whether or not there is
Intel Matrix RAID hardware (excerpts):

# lspci | grep RAID
00:1f.2 RAID bus controller: Intel Corporation C600/X79 series chipset SATA RAID Controller ...

and whether or not the Intel Matrix RAID hardware
is actually used or usable for RAID (i.e. when more than
one disk is attached to the Intel Matrix RAID bus controller):

# mdadm --detail-platform
...
Platform : Intel(R) Rapid Storage Technology enterprise
...
RAID Levels : raid0 raid1 raid10 raid5
...
I/O Controller : /sys/devices/pci0000:00/0000:00:1f.2 ...
Port0 : /dev/sda ...
Port1 : - no device attached -
Port2 : /dev/sdb ...

Cf.
https://www.intel.com.au/content/dam/www/public/us/en/documents/white-papers/rst-linux-paper.pdf
and
https://unix.stackexchange.com/questions/273819/how-do-i-rebuild-create-assemble-an-imsm-raid-0-array-from-disk-images-instead

jsmeix commented at 2018-04-05 10:23:¶

An addedum only FYI about "AMD UEFI fake RAID":

Recently I noticed that it seems also
AMD provides some kind of "BIOS fake RAID" which is
probably better called "AMD UEFI fake RAID" in this case
because I noticed in a SUSE internal mail (excerpts):

... a community person who has been trying with partial success
to get AMD BIOS "fake RAID" running under Ubuntu.
This RAID chip is included in several ASUS boards.

Here is the guy's experience:
https://kwikr.de/Howto_Windows_Ubuntu_AMD-RAID.html

The AMD driver package under
https://support.amd.com/en-us/download/chipset?os=Linux+x86_64
is pretty pathetic and supports Ubuntu only.

As far as I understand
https://kwikr.de/Howto_Windows_Ubuntu_AMD-RAID.html
and
https://community.amd.com/message/2833733
on a first glance it seems such an
"AMD UEFI fake RAID" basically works by

enabling and creating an AMD-RAID in UEFI
using the AMD rcraid kernel module (also in initrd for booting)

Accordingly I hope for "rear recover" on replacement hardware we could assume
that enabling AMD-RAID in UEFI and creating an AMD-RAID in UEFI
is already done on the replacement hardware where "rear recover" runs
(i.e. the replacement hardware must have been already prepared
to use such an "AMD UEFI fake RAID").

Additionally I hope that the AMD rcraid kernel module (also in initrd for booting)
gets automatically inherited from the original system where it is loaded so that
the ReaR recovery system kernel also uses the AMD rcraid kernel module.

In particular it seems that with such an "AMD UEFI fake RAID"
each filesystem gets installed into a single normal looking
partition device node like /dev/sdb5, cf.
assume ... you installed Ubuntu on /dev/sdb5
in https://kwikr.de/Howto_Windows_Ubuntu_AMD-RAID.html
so that (hopefully) nothing special needs to be done or set up by ReaR
when such an "AMD UEFI fake RAID" is used.

jsmeix commented at 2018-06-28 07:29:¶

I need to get first and foremost a basic understanding about
MD software RAID setup to be able to implement support for
"RAID 1 mdraid Intel IMSM/RST based firmware RAID containers"
in ReaR.

This precondition will be my SUSE Hack Week 17 project:
https://hackweek.suse.com/17/projects/get-a-basic-understanding-about-md-software-raid-setup

pcahyna commented at 2018-07-14 23:39:¶

I think it is not Intel IMSM/RST hardware specific really. You can use the ddf container format for testing (-e ddf), it should behave in a similar way to imsm and you don't need hardware support.
(actually even IMSM containers can be created on hardware which does not support them by setting IMSM_NO_PLATFORM=1 in the environment. See the mdadm man page.)
FYI how I created the array with a single disk for testing:
(you probably want to remove all partitions on /dev/sdb first)

mdadm -C --force /dev/md/imsm /dev/sdb -n 1 -e imsm
mdadm -C /dev/md/vol0 --force --level=0 --raid-devices=1  /dev/md/imsm

After reboot the names changed to /dev/md/imsm0 and /dev/md/vol0_0. Anyway, /dev/md/vol0_0 is a RAID device now, one can create a filesystem on it and it will autoassemble after reboot.

mdadm  --detail /dev/md/imsm0

/dev/md/imsm0:
           Version : imsm
        Raid Level : container
     Total Devices : 1

   Working Devices : 1


              UUID : e12cfa7f:fd172137:1cca35b0:a9fbb2ce
     Member Arrays : /dev/md/vol0_0

    Number   Major   Minor   RaidDevice

       -       8       16        -        /dev/sdb

mdadm  --detail /dev/md/vol0_0

/dev/md/vol0_0:
         Container : /dev/md/imsm0, member 0
        Raid Level : raid0
        Array Size : 78147584 (74.53 GiB 80.02 GB)
      Raid Devices : 1
     Total Devices : 1

             State : clean 
    Active Devices : 1
   Working Devices : 1
    Failed Devices : 0
     Spare Devices : 0

        Chunk Size : 128K

Consistency Policy : none


              UUID : 8f34580e:83d09b9e:6c835f5b:93ca4125
    Number   Major   Minor   RaidDevice State
       0       8       16        0      active sync   /dev/sdb

this command has the easiest format for machine parsing and clearly shows what devices are inside a container:

mdadm --detail --scan --config=partitions

ARRAY /dev/md/imsm0 metadata=imsm UUID=e12cfa7f:fd172137:1cca35b0:a9fbb2ce
ARRAY /dev/md/vol0_0 container=/dev/md/imsm0 member=0 UUID=8f34580e:83d09b9e:6c835f5b:93ca4125

And you can replace imsm by ddf above for a hardware-independent solution. (Although I have not gotten it to autoassemble this way.)

jsmeix commented at 2018-07-16 09:21:¶

@pcahyna
WOW!
Many thanks for your descriptive information.
I will try it as soon as time permits.

It would help me so much if I could sufficiently well simulate that stuff
on QEMU/KVM virtual machines with two virtual harddisks for a "real" RAID1
because then I can keep the original system on one virtual machine
and test "rear recover" on another same virtual machine
which speeds up development (also because virtual machines
that run on one same powerful host system are really fast
because then all happens within the host's main memory).

gdha commented at 2018-10-05 09:14:¶

@jsmeix is the Dedicated Priority Support label still required as the case has been open for a long time? Personally I have the feeling this is more a consultancy request than a support call. We cannot help this user without his special HW...

v-vidr commented at 2018-10-05 09:29:¶

Hi,

If required we will arrange hardware for testing for remote session

schlomo commented at 2018-10-05 09:43:¶

@v-vidr I guess that this case is one where you should consider paid ReaR consulting to either come on-site or remotely work on your hardware to add support for this use case to ReaR.

Just out of curiosity: If this is in fact a software RAID, what exactly is the benefit of the IMSM format over the regular mdadm formats? Is is about BIOS/UEFI support for booting off the 2nd disk if the first disk fails?

I am asking because it might be easier to use standard software RAID instead. With UEFI you also don't need to mess with boot sectors any more as we had to with BIOS.

jsmeix commented at 2018-10-05 12:36:¶

@gdha
I had set the Dedicated Priority Support label because we at SUSE
have an internal feature request to get that implemented in ReaR
which is currently pending at certain levels of management
evaluation and approval steps that need to be done as
preconditions until someone will get the needed hardware
so that it could be implemented.

@v-vidr
I cannot imagine how remote hardware could be of real help
to get such a feature implemented with reasonable effort
because at least I need direct testing of "rear recover"
on directly available hardware (where for me that "hardware"
is usually a virtual machine where I also completely own its
host system (i.e. I am root on the virtualization host) so that
I can do directly whatever I need and want).

@schlomo
since I learned about "firmware RAID" I was wondering
what its actual benefit is compared to kernel software RAID,
it particular because the Linux Raid wiki
https://raid.wiki.kernel.org/index.php/Linux_Raid
reads (excerpt):

BIOS / firmware RAID aka fake raid cards:

* offer a few performance benefits (like CPU, bus and
  RAM offloading), but may often be much slower
  than SW raid (link?)

* if the 'raid' card or motherboard dies then you often
  have to find an exact replacement and this can be
  tricky for older cards

* if drives move to other machines the data can't easily
  be read

* there is usually no monitoring or reporting on the
  array - if a problem occurs then it may not show up
  unless the machine is rebooted *and* someone is
  actually watching the BIOS boot screen (or until
  multiple errors occur and your data is lost)

* you are entrusting your data to unpatchable software
  written into a BIOS that has probably not been tested,
  has no support mechanism and almost no community.

* having seen how many bugs the kernel works around
  in various BIOSes it would be optimistic to think that
  the BIOS RAID has no bugs. 

Given the point of RAID is usually to reduce risk
it is fair to say that using fakeraid is a terrible idea
and it's better to focus energy on either true HW raid
or in-kernel SW raid .... but there is nothing stopping you :)

@v-vidr
only out of curiosity could you explain to us what the reason is
(in particular what the actual benefit is in your particular use-case)
why you use firmware RAID instead of kernel software RAID
which would - by the way - also "just work" with current ReaR
(at least when you use a usual kernel software RAID setup).

In general when someone likes to use ReaR for disaster recovery
he must set up his original system so that ReaR can recreate it
but not the other way round because that might only work by luck,
cf. sections like "Disaster recovery does not just work" and
"Let's face it: Deployment via the recovery installer is a must" and
"The limitation is what the special ReaR recovery system can do" in
https://en.opensuse.org/SDB:Disaster_Recovery

pcahyna commented at 2018-10-26 18:39:¶

@v-vidr can you provide hardware with remote access to console, including BIOS setup (e.g. serial port)?

jsmeix commented at 2018-11-21 14:54:¶

For the fun of it:
There is IRST as in Intel Rapid Storage Technology - e.g. see
https://pcsupport.lenovo.com/de/en/products/laptops-and-netbooks/lenovo-v-series-laptops/v110-15isk/downloads/ds113245

Intel Rapid Storage Technology (IRST) Driver ...

and IRST as in Intel Rapid Start Technology - e.g. see
https://www.dell.com/support/article/de/de/debsdt1/sln265682/how-to-setup-intel-rapid-start-technology-in-the-uefi-mode?lang=en

What is Intel Rapid Start Technology (IRST) ...

Perhaps the next one will be IRST as in Intel Rapid Something Technology
or whatever you may imagine what the S therein could stand for... ;-)

github-actions commented at 2020-07-01 01:33:¶

Stale issue message

[Export of Github issue for rear/rear.]

#1540 Issue closed: ReaR does not support RAID 1 mdraid Intel IMSM/RST based firmware RAID containers¶