sdb disk without partition type¶

Labels: enhancement

mdbr000 opened issue at 2022-05-23 12:37:¶

ReaR version ("/usr/sbin/rear -V"): Relax-and-Recover 2.6 / Git
OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):

LSB Version:    n/a
Distributor ID: ManjaroLinux
Description:    Manjaro Linux
Release:    21.2.6
Codename:   Qonos

ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

KERNEL_FILE=/boot/vmlinuz-5.15-x86_64

REQUIRED_PROGS+=( borg locale localectl localedef locale-gen mount.nfs mount.nfs4 mount.fuse mount.fuse3 tmux )
LIBS+=( /usr/lib/libcrypt*.so* /usr/lib/libutil*.so* )
MODULES+=( fuse )

OUTPUT=RAWDISK
BACKUP_URL=nfs://arch-pi/srv/nfs/backups

Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR): Dell Precision 7820
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device): x86_64
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot): UEFI / GRUB
Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe): SSD
Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"):

NAME                                                      KNAME          PKNAME         TRAN   TYPE  FSTYPE      LABEL               SIZE MOUNTPOINT
/dev/sda                                                  /dev/sda                      sata   disk                                  1.8T 
`-/dev/sda1                                               /dev/sda1      /dev/sda              part  ext4                            1.8T 
/dev/sdb                                                  /dev/sdb                      usb    disk                                    0B 
/dev/sdc                                                  /dev/sdc                      usb    disk  iso9660     MANJARO_XFCE_2126   7.5G 
|-/dev/sdc1                                               /dev/sdc1      /dev/sdc              part  iso9660     MANJARO_XFCE_2126   3.3G 
`-/dev/sdc2                                               /dev/sdc2      /dev/sdc              part  vfat        MISO_EFI              4M 
/dev/nvme0n1                                              /dev/nvme0n1                  nvme   disk                                953.9G 
|-/dev/nvme0n1p1                                          /dev/nvme0n1p1 /dev/nvme0n1   nvme   part  vfat        NO_LABEL            300M /boot/efi
|-/dev/nvme0n1p2                                          /dev/nvme0n1p2 /dev/nvme0n1   nvme   part  crypto_LUKS                   884.8G 
| `-/dev/mapper/luks-1912fa48-dcca-4001-8fa1-2cd45b2f5a6f /dev/dm-0      /dev/nvme0n1p2        crypt ext4                          884.8G /
`-/dev/nvme0n1p3                                          /dev/nvme0n1p3 /dev/nvme0n1   nvme   part  crypto_LUKS                    68.8G 
  `-/dev/mapper/luks-559f7632-17f5-465e-b714-bb3fb4be1799 /dev/dm-1      /dev/nvme0n1p3        crypt swap        swap               68.8G [SWAP]

Description of the issue (ideally so that others can reproduce it):
rear -d -v mkbackup fails with the following messages due to /dev/sdb

ERROR: Invalid 'disk /dev/sdb' entry (no partition table type for '/dev/sdb')
Some latest log messages since the last called script 200_partition_layout.sh:
  2022-05-22 19:23:20.559444144 Saving disks and their partitions
  blockdev: cannot open /dev/sdb: No medium found
  Error: Error opening /dev/sdb: No medium found

Workaround, if any:

Insert an SD card into /dev/sdb such that it appears populated with a block device. Not a sustainable workaround for scheduled backups. Potential issue for when trying to restore the machine later.

jsmeix commented at 2022-05-23 14:07:¶

@mdbr000
did you test and verify with a previous ReaR version
that "rear recover" actually does work for you
when you have no SD card in /dev/sdb ?

This error exit is new since my recent
https://github.com/rear/rear/pull/2804
therein this new tests

# Ensure syntactically correct 'disk' entries:
# Each value must exist and each value must be a single non-blank word so we 'test' without quoting the value:
test $devname || Error "Invalid 'disk' entry (no disk device name for '$disk')"
test $devsize || Error "Invalid 'disk $devname' entry (no device size for '$devname')"
test $disktype || Error "Invalid 'disk $devname' entry (no partition table type for '$devname')"

in particular the last test which is currently online at
https://github.com/rear/rear/blob/master/usr/share/rear/layout/save/GNU/Linux/200_partition_layout.sh#L409

It tests based on the "Disk layout file syntax" description in
https://github.com/rear/rear/blob/master/doc/user-guide/06-layout-configuration.adoc
which reads (excerpts)

Angle brackets "<" and ">" delimit a value ...
...
Square brackets "[" and "]" indicate an optional parameter.
...
Disks

  disk <name> <size(B)> <partition label>

None of the values is specified as optional there
so - in theory - it should be right to error out
but - in practice - perhaps <partition label> is optional?

I will have to thoroughly inspect the code
that consumes those values to find out
if <partition label> is mandatory or optional.

When there are missing mandatory values in disklayout.conf
things go arbitrarily and badly wrong during "rear recover".

If "rear recover" actually did work for you
even when you have no SD card in /dev/sdb
with a previous ReaR version
you may as a workaround for now remove the line

test $disktype || Error "Invalid 'disk $devname' entry (no partition table type for '$devname')"

from your
usr/share/rear/layout/save/GNU/Linux/200_partition_layout.sh
script.

mdbr000 commented at 2022-05-27 16:37:¶

@jsmeix
Thanks much for the response and helpful information. I was indeed able to run rear mkbackup successfully after reverting to the commit just prior to your #2804 PR when no memory card was present in /dev/sdb.

It sounds like this might be a change/fix that you'll want to make in mainline. For now, I will compile a version that comments out the test line you call out above.

I am still having issues with recovery, but that is a separate topic.

Again, thank you.

jsmeix commented at 2022-05-30 12:50:¶

@mdbr000
please attach your recent var/lib/rear/layout/disklayout.conf
because I need to know how all entries related to sdb
look therein in your particular case.

jsmeix commented at 2022-05-30 13:07:¶

I did a test with a manually modified
var/lib/rear/layout/disklayout.conf
as follows (excerpt):

disk /dev/sda 12884901888 gpt
disk /dev/sdb 5368709120

i.e. I removed the partition type label gpt from sdb
and I also removed any other entry for sdb so

disk /dev/sdb 5368709120

is the only entry related to sdb in
var/lib/rear/layout/disklayout.conf

Then I did a "rear -D recover" which worked.

Except from var/lib/rear/layout/diskrestore.sh
regarding sdb:

if create_component "/dev/sdb" "disk" ; then
# Create /dev/sdb (disk)

#
# Code handling disk '/dev/sdb'
#

### Disks should be block devices.
[ -b "/dev/sdb" ] || BugError "Disk /dev/sdb is not a block device."

Log "Stop mdadm"
if grep -q md /proc/mdstat 2>/dev/null; then
    mdadm --stop -s >&2 || echo "stop mdadm failed"
    # Prevent udev waking up mdadm later.
    # Reasoning: At least on RHEL6 when parted created a raid partition on disk,
    # udev (via /lib/udev/rules.d/65-md-incremental.rules) wakes up mdadm which locks the disk,
    # so further parted commands with the disk will fail since the disk is busy now.
    # The /lib/udev/rules.d/65-md-incremental.rules detects anaconda (the Red Hat installer),
    # and if it find itself running under anaconda, it will not run.
    # Accordingly also for other installers (in particular the ReaR installer)
    # this rule should not be there (and other Linux distros probably do not have it)
    # which means removing it is the right solution to make ReaR work also for RHEL6:
    if [ -e /lib/udev/rules.d/65-md-incremental.rules ] ; then
        rm -f /lib/udev/rules.d/65-md-incremental.rules || echo "rm 65-md-incremental.rules failed"
    fi
fi
Log "Erasing MBR of disk /dev/sdb"
dd if=/dev/zero of=/dev/sdb bs=512 count=1
sync

# Make sure device nodes are visible (eg. in RHEL4)
my_udevtrigger
my_udevsettle

# Clean up transient partitions and resize shrinked ones
delete_dummy_partitions_and_resize_real_ones

#
# End of code handling disk '/dev/sdb'
#

component_created "/dev/sdb" "disk"
else
    LogPrint "Skipping /dev/sdb (disk) as it has already been created."
fi

So it works in the special case when there is
only a 'disk' entry without partition type label value
but nothing else for this disk exists in disklayout.conf

jsmeix commented at 2022-05-30 13:28:¶

Via
https://github.com/rear/rear/commit/0a1d634ed15500bb21f37ac1bbb11c8a4bb11545
we do no longer error out when there is no partition label type value
for a 'disk' entry in disklayout.conf because "rear recover" works
in a special case without partition label type value when there is
only a 'disk' entry but nothing else for this disk exists in disklayout.conf
which can happen when /dev/sdX is an empty SD card slot without medium.

jsmeix commented at 2022-05-30 13:42:¶

@mdbr000
I would appreciate it if you could test our current
ReaR upstream GitHub master code, see the section
"Testing current ReaR upstream GitHub master code" in
https://en.opensuse.org/SDB:Disaster_Recovery

With our current ReaR upstream GitHub master code
you should get the message

No partition label type for 'disk /dev/sdb' (may cause 'rear recover' failure)

because usually a missing partition label type value
will cause a 'rear recover' failure.

Here "'rear recover' failure" also means when a disk gets
partitioned with a fallback partition type via the code in
layout/prepare/GNU/Linux/100_include_partition_code.sh
when the original disk partition type was different
but somehow not detected/stored in disklayout.conf.

pcahyna commented at 2022-05-30 14:55:¶

Interesting - when testing #2804 & #2803 I tested cases with drives without partition tables, but not drives with missing media. I wonder how to test this without a physical drive that supports removable media?

jsmeix commented at 2022-05-31 07:22:¶

I was also wondering how I could somehow reproduce
the behaviour that is described in this issue here.

I am not a SD card user (I don't have a SD card medium).
My homeoffice laptop has an (empty) SD card slot
but I see no block device node for it.

So I guess the SD card reader in this issue here
is some unusual hardware that shows up as block device
even when there is no medium so its block device node
is basically useless and misleading.

I played around with an artificial "dangling" block device node

# mknod /dev/sdq b 8 99

# file /dev/sdq
/dev/sdq: block special (8/99)

# find /sys/block/
/sys/block/
/sys/block/dm-1
/sys/block/sr0
/sys/block/dm-0
/sys/block/sda

# lsblk /dev/sdq
lsblk: /dev/sdq: failed to get sysfs name: No such file or directory

# blockdev --report /dev/sdq
RO    RA   SSZ   BSZ        StartSec            Size   Device
blockdev: cannot open /dev/sdq: No such device or address

and /dev/sdq was just ignored by "rear mkrescue" (as it should).

So I manually modified a 'disk' entry in disklayout.conf
in the ReaR recovery system before "rear recover"
to be able to at least partially reproduce this issue here.

pcahyna commented at 2022-05-31 10:01:¶

The difference with a "dangling" device node is that opening it will report ENXIO "No such device or address", while an existing but empty device reports ENOMEDIUM "No medium found". I don't think the device is necessarily "useless and misleading" - the physical device (card reader) is here, only its slot is empty. This is the usual behavior with removable media devices like floppies or optical media (not USB flash "disks" though, because in their case you remove the entire device, not only the medium).

jsmeix commented at 2022-05-31 11:55:¶

With "basically useless and misleading" I meant how it behaves
in practice in particular compared to "normal" disks.

With a "normal" disk when e.g. /dev/sda is there
one can access "the whole disk data" via /dev/sda

In contrast removable media devices behave different because
when e.g. /dev/sdb is there it does not necessarily mean
one can access "the whole disk data" via /dev/sdb
but only when a medium is inserted.

I wonder for what /dev/sdb could be useful without medium?
My offhanded guess is that a removable media device
without medium is as useful as a "normal" disk device where
its built-in medium was mechanically removed by a technician
or as useful as a USB mass storage device that is disconnected.
But I am not at all an expert here so I only do some
offhanded thinking from an ignorant end-user point of view.

In the end for ReaR this issue means:
We will have to implement special case handling
for removable media devices in ReaR.

jsmeix commented at 2022-05-31 12:04:¶

FYI
how my CD/DVD device behaves on my homeoffice laptop
without medium

# cat /sys/block/sr0/removable
1

# lsblk /dev/sr0
NAME MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0   11:0    1 1024M  0 rom

# blockdev --report /dev/sr0
RO    RA   SSZ   BSZ        StartSec            Size   Device
rw  1024   512   512               0      1073741312   /dev/sr0

# parted -s /dev/sr0 print
Error: Error opening /dev/sr0: No medium found

and with medium (which got automatically mounted)

# lsblk /dev/sr0
NAME MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0   11:0    1  3.8G  0 rom  /run/media/johannes/openSUSE-Leap...

# blockdev --report /dev/sr0
RO    RA   SSZ   BSZ        StartSec            Size   Device
rw  1024  2048  2048               0      4056940544   /dev/sr0

# parted -s /dev/sr0 print
Error: Can't have a partition outside the disk!
Model: hp DVDRAM GU70N (scsi)
Disk /dev/sr0: 4057MB
Sector size (logical/physical): 2048B/2048B
Partition Table: unknown
Disk Flags:

pcahyna commented at 2022-05-31 12:32:¶

I wonder for what /dev/sdb could be useful without medium?

You can ask for its manufacturer and capabilities, for example. With the device node, it is easier to map what devices you have. If the device has a loadable tray, you can also issue commands like open/close the tray and load the medium (this is the case for optical and similar media), which would be difficult without a device node.

It seems that the empty DVD drive behaves similarly to the empty SD card reader, but that's probably not so useful for ReaR testing, because I suppose that ReaR ignores CD/DVD drives (would it help to create a symlink /dev/sdb -> /dev/sr0?)

jsmeix commented at 2022-05-31 12:46:¶

With a /dev/sdq -> /dev/sr0 symlink:

# ln -s /dev/sr0 /dev/sdq

# ls -l  /dev/sr0 /dev/sdq
lrwxrwxrwx  1 root root      8 May 31 14:38 /dev/sdq -> /dev/sr0
brw-rw----+ 1 root cdrom 11, 0 May 31 14:10 /dev/sr0

# usr/sbin/rear -D mkrescue
...
Creating disk layout
Overwriting existing disk layout file /root/rear.github.master/var/lib/rear/layout/disklayout.conf
Excluding component fs:/other in EXCLUDE_RECREATE
Marking component 'fs:/other' as done in /root/rear.github.master/var/lib/rear/layout/disktodo.conf
Disabling excluded components in /root/rear.github.master/var/lib/rear/layout/disklayout.conf
Disabling component 'fs ... /other' in /root/rear.github.master/var/lib/rear/layout/disklayout.conf
Using sysconfig bootloader 'grub2' for 'rear recover'
Verifying that the entries in /root/rear.github.master/var/lib/rear/layout/disklayout.conf are correct
Created disk layout (check the results in /root/rear.github.master/var/lib/rear/layout/disklayout.conf)
...

# grep sdq var/lib/rear/layout/disklayout.conf
[no output]

# grep sdq var/log/rear/rear-linux-h9wr.log
[no output]

# grep sr0 var/lib/rear/layout/disklayout.conf
# /dev/sr0            /dev/sr0       sata   rom          3.8G
[this is only the 'lsblk' comment line in disklayout.conf]

# grep sr0 var/log/rear/rear-linux-h9wr.log
++ ID_NEW=sr0
++ [[ sr0 =~ ^dm- ]]
++ echo ata-hp_DVDRAM_GU70N_M49CBQG2454 /dev/sr0
++ ID_NEW=sr0
++ [[ sr0 =~ ^dm- ]]
++ echo wwn-0x5001480000000000 /dev/sr0
++ blockd=sr0
++ [[ sr0 = hd* ]]
++ [[ sr0 = sd* ]]
++ [[ sr0 = cciss* ]]
++ [[ sr0 = vd* ]]
++ [[ sr0 = xvd* ]]
++ [[ sr0 = dasd* ]]
++ [[ sr0 = nvme* ]]
++ [[ sr0 = mmcblk* ]]

Without /dev/sdq -> /dev/sr0 symlink:

# grep sr0 var/lib/rear/layout/disklayout.conf
# /dev/sr0            /dev/sr0       sata   rom          3.8G
[this is only the 'lsblk' comment line in disklayout.conf]

# grep sr0 var/log/rear/rear-linux-h9wr.log
++ ID_NEW=sr0
++ [[ sr0 =~ ^dm- ]]
++ echo ata-hp_DVDRAM_GU70N_M49CBQG2454 /dev/sr0
++ ID_NEW=sr0
++ [[ sr0 =~ ^dm- ]]
++ echo wwn-0x5001480000000000 /dev/sr0
++ blockd=sr0
++ [[ sr0 = hd* ]]
++ [[ sr0 = sd* ]]
++ [[ sr0 = cciss* ]]
++ [[ sr0 = vd* ]]
++ [[ sr0 = xvd* ]]
++ [[ sr0 = dasd* ]]
++ [[ sr0 = nvme* ]]
++ [[ sr0 = mmcblk* ]]

jsmeix commented at 2022-05-31 13:03:¶

@pcahyna
thank you for your explanation
for what /dev/sdX is useful without medium
e.g. to open/close a tray and things like that
i.e. what is needed to load or unload a medium.
Now I understand.

pcahyna commented at 2022-05-31 13:41:¶

I am trying to create /dev/sdq with the same device major/minor numbers as /dev/sr0 (cp -a /dev/sr0 /dev/sdq does the job) and trick ReaR into believing that it is a disk, but so far I have not been successful.

pcahyna commented at 2022-05-31 13:46:¶

looking at the output here, /dev/sdb is a USB mass storage device. I remember having seen (external) USB card readers, but it was a long time ago. Some laptops apparently have built-in memory card readers on USB, so one might be successful with some of those, but I currently do not have one. @mdbr000 what device is it? (I think a lsusb output would help.)

jsmeix commented at 2022-06-01 06:46:¶

@pcahyna
thanks for trying with an artificial /dev/sdq device node
with same major/minor numbers as /dev/sr0.
That would have been my next attempt for today.

FYI:
My built-in SD card reader is:

# lspci -k
...
05:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd.
 RTS5227 PCI Express Card Reader (rev 01)
        Subsystem: Hewlett-Packard Company Device 1940
        Kernel driver in use: rtsx_pci
        Kernel modules: rtsx_pci

I have nothing about 'rtsx' in dmesg - there is only

# dmesg | egrep -i '05:00\.0|mmc'

[    0.036434] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000)
[    0.036434] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in E820
...
[    2.514760] pci 0000:05:00.0: [10ec:5227] type 00 class 0xff0000
[    2.514861] pci 0000:05:00.0: reg 0x10: [mem 0xb0500000-0xb0500fff]
[    2.515180] pci 0000:05:00.0: supports D1 D2
[    2.515182] pci 0000:05:00.0: PME# supported from D1 D2 D3hot D3cold
...
[    7.722075] sr 5:0:0:0: [sr0] scsi3-mmc drive: 24x/24x writer dvd-ram cd/rw xa/form2 cdda tray

There is no device node for it (neither /dev/sd* nor /dev/mmc*)

# udisksctl status

MODEL                     REVISION  SERIAL               DEVICE
---------------------------------------------------------------
TOSHIBA MQ01ABF050        AM002C    Y2PLP02CT            sda     
hp       DVDRAM GU70N     U703      M49CBQG2454          sr0

I assume this is expected because I do not have a SD card.

pcahyna commented at 2022-06-01 11:49:¶

@jsmeix I have a similar device and it behaves the same way. I decided I will not try further to emulate an empty sd drive with an empty sr drive, because it seems that the code is smart enough to distinguish between CD-ROM drives and usual SCSI disks. I now prefer reproducing the issue with the actual problematic hardware.

@mdbr000 can you please send the lsusb output identifying your card reader?

jsmeix commented at 2022-06-02 07:25:¶

I postponed this issue to ReaR 2.8 because
it does not need to be solved for ReaR 2.7
because since
https://github.com/rear/rear/commit/0a1d634ed15500bb21f37ac1bbb11c8a4bb11545
"rear mkrescue" works again even when there is
no partition label type value for a 'disk' entry in disklayout.conf