#2321 Issue closed: During the restore, the message "No code has been generated to recreate pv:/dev/sda2 (lmdev)"

Labels: support / question, fixed / solved / done

shaunsJM opened issue at 2020-01-22 21:55:

Relax-and-Recover (ReaR) Issue Template

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

  • ReaR version ("/usr/sbin/rear -V"): Relax-and-Recover 2.5 / 2019-05-10

  • OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"): SUSE Linux Enterprise Server 12 SP5

  • ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

# Begin example setup for SLE11 with default ext3 filesystem.
# You must adapt "your.NFS.server.IP/path/to/your/rear/backup" at BACKUP_URL.
# You may activate SSH_ROOT_PASSWORD and adapt the "password_on_the_rear_recovery_system".
# For basic information see the SLE11 manuals.
# Also see the support database article "SDB:Disaster Recovery"
# at http://en.opensuse.org/SDB:Disaster_Recovery
# In particular note:
# There is no such thing as a disaster recovery solution that "just works".
HOSTNAME="$( uname -n | cut -d. -f1 )"
# Create rear recovery system as ISO image:
OUTPUT=ISO
# Store the backup file via NFS on a NFS server:
BACKUP=NETFS
# BACKUP_OPTIONS variable contains the NFS mount options and
# with 'mount -o nolock' no rpc.statd (plus rpcbind) are needed:
BACKUP_OPTIONS="nolock,credentials=/etc/rear/cifs-rear"
#BACKUP_OPTIONS="nfsvers=3,nolock"
# If the NFS server is not an IP address but a hostname,
# DNS must work in the rear recovery system when the backup is restored.
BACKUP_URL=cifs://den-vmutility/Linux-Rear-Migrations
# Keep an older copy of the backup in a HOSTNAME.old directory
# provided there is no '.lockfile' in the HOSTNAME directory:
NETFS_KEEP_OLD_BACKUP_COPY=yes
COPY_AS_IS=( "${COPY_AS_IS[@]}" /etc/rear/cifs-rear )
# This option defines a root password to allow SSH connection
# whithout a public/private key pair
# To avoid a plain text password in the etc/rear/local.conf config file
# generate a hashed password with the command
#   echo "my_recovery_system_root_password" | openssl passwd -1 -stdin
# and use the output of openssl to set SSH_ROOT_PASSWORD='output_of_openssl'
# (single quotes avoid issues with the special bash character $ in the openssl output).
#  NOTE:  the root password for recovery is in Thycotic as 'ReaR boot disk' under servers -> Linux.
SSH_ROOT_PASSWORD='XXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
# Let the rear recovery system run dhclient to get an IP address
# instead of using the same IP address as the original system:
#USE_DHCLIENT="yes"
# End example setup for SLE11 with default ext3 filesystem.
ONLY_INCLUDE_VG=( "system" )
#ISO_MKISOFS_BIN="$( type -p xorrisofs || type -p mkisofs || type -p genisoimage )"
ISO_MKISOFS_BIN=/usr/bin/ebiso
EXCLUDE_VG=( "vgapp" "vgarchlog" "vgu00-04" "vgu05-09" )
EXCLUDE_MOUNTPOINTS=('/app' '/archlog' '/u00' '/u01' '/u02' '/u03' '/u04' '/u05' '/u06' '/u07' '/u08' '/u09')
CLONE_ALL_USERS_GROUPS="true"
KEEP_BUILD_DIR="yes"

  • Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR): Product Name: ThinkSystem SR630

  • System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device): x86_64

  • Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot): UEFI and GRUB

  • Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe): local disk (SAS)

  • Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT" or "lsblk" as makeshift):

NAME                        KNAME     PKNAME    TRAN TYPE FSTYPE        SIZE MOUNTPOINT
/dev/sda                    /dev/sda                 disk             278.5G
|-/dev/sda1                 /dev/sda1 /dev/sda       part vfat          156M /boot/efi
`-/dev/sda2                 /dev/sda2 /dev/sda       part LVM2_member 278.3G
  |-/dev/mapper/system-root /dev/dm-0 /dev/sda2      lvm  ext4           10G /
  |-/dev/mapper/system-swap /dev/dm-1 /dev/sda2      lvm  swap            2G [SWAP]
  `-/dev/mapper/system-home /dev/dm-2 /dev/sda2      lvm  xfs            25G /home
/dev/sdc                    /dev/sdc            usb  disk               177M
`-/dev/sdc1                 /dev/sdc1 /dev/sdc       part vfat          176M
/dev/sr0                    /dev/sr0            usb  rom               1024M

  • Description of the issue (ideally so that others can reproduce it):
    During the restore, the message "No code has been generated to recreate pv:/dev/sda2 (lmdev)" and in the log I see "Ignoring sda: it is a path of a multipath device"

  • Workaround, if any: none yet.

  • Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):

jmt-lxtest1:/etc/rear # rear -D -v mkbackup
Relax-and-Recover 2.5 / 2019-05-10
Running rear mkbackup (PID 8042)
Using log file: /var/log/rear/rear-jmt-lxtest1.log
Using backup archive '/tmp/rear.GcQIcAlWX51FZQI/outputfs/jmt-lxtest1/backup.tar.gz'
Found EFI system partition /dev/sda1 on /boot/efi type vfat
Using UEFI Boot Loader for Linux (USING_UEFI_BOOTLOADER=1)
Using autodetected kernel '/boot/vmlinuz-4.12.14-120-default' as kernel in the recovery system
Creating disk layout
Excluding Volume Group vgapp.
Excluding Volume Group vgarchlog.
Excluding Volume Group vgu00-04.
Excluding Volume Group vgu05-09.
Using sysconfig bootloader 'grub2-efi'
Verifying that the entries in /var/lib/rear/layout/disklayout.conf are correct ...
Creating root filesystem layout
Handling network interface 'eth0'
eth0 is a physical device
Handled network interface 'eth0'
Trying to find what to use as UEFI bootloader...
Trying to find a 'well known file' to be used as UEFI bootloader...
Using '/boot/efi/EFI/sles/grubx64.efi' as UEFI bootloader file
Copying logfile /var/log/rear/rear-jmt-lxtest1.log into initramfs as '/tmp/rear-jmt-lxtest1-partial-2020-01-22T14:05:18-07:00.log'
Copying files and directories
Copying binaries and libraries
Copying all kernel modules in /lib/modules/4.12.14-120-default (MODULES contains 'all_modules')
Copying all files in /lib*/firmware/
Skip copying broken symlink '/etc/mtab' target '/proc/17372/mounts' on /proc/ /sys/ /dev/ or /run/
Testing that the recovery system in /tmp/rear.GcQIcAlWX51FZQI/rootfs contains a usable system
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (303636040 bytes) in 42 seconds
Making ISO image
Wrote ISO image: /var/lib/rear/output/rear-jmt-lxtest1.iso (336M)
Copying resulting files to cifs location
Saving /var/log/rear/rear-jmt-lxtest1.log as rear-jmt-lxtest1.log to cifs location
Copying result files '/var/lib/rear/output/rear-jmt-lxtest1.iso /tmp/rear.GcQIcAlWX51FZQI/tmp/VERSION /tmp/rear.GcQIcAlWX51FZQI/tmp/README /tmp/rear.GcQIcAlWX51FZQI/tmp/rear-jmt-lxtest1.log' to /tmp/rear.GcQIcAlWX51FZQI/outputfs/jmt-lxtest1 at cifs location
Creating tar archive '/tmp/rear.GcQIcAlWX51FZQI/outputfs/jmt-lxtest1/backup.tar.gz'
Archived 1453 MiB [avg 6252 KiB/sec] OK
WARNING: tar ended with return code 1 and below output:
  ---snip---
  tar: /var/spool/postfix/private/error: socket ignored
  tar: /var/spool/postfix/private/retry: socket ignored
  tar: /sys: file changed as we read it
  ----------
This means that files have been modified during the archiving
process. As a result the backup may not be completely consistent
or may not be a perfect copy of the system. Relax-and-Recover
will continue, however it is highly advisable to verify the
backup in order to be sure to safely recover this system.

Archived 1453 MiB in 239 seconds [avg 6226 KiB/sec]
Exiting rear mkbackup (PID 8042) and its descendant processes ...
Running exit tasks
You should also rm -Rf /tmp/rear.GcQIcAlWX51FZQI

The recovery steps:

RESCUE jmt-lxtest1:~ # rear -D -v recover
Relax-and-Recover 2.5 / 2019-05-10
Running rear recover (PID 1279)
Using log file: /var/log/rear/rear-jmt-lxtest1.log
Running workflow recover within the ReaR rescue/recovery system
Using backup archive '/tmp/rear.ubMCqNzTljU5BD6/outputfs/jmt-lxtest1/backup.tar.gz'
Will do driver migration (recreating initramfs/initrd)
Calculating backup archive size
Backup archive size is 1.5G     /tmp/rear.ubMCqNzTljU5BD6/outputfs/jmt-lxtest1/backup.tar.gz (compressed)
Comparing disks
Disk configuration looks identical
UserInput -I DISK_LAYOUT_PROCEED_RECOVERY needed in /usr/share/rear/layout/prepare/default/250_compare_disks.sh line 148
Proceed with recovery (yes) otherwise manual disk layout configuration is enforced
(default 'yes' timeout 30 seconds)
UserInput: No real user input (empty or only spaces) - using default input
UserInput: No choices - result is 'yes'
Proceeding with recovery by default
No code has been generated to recreate pv:/dev/sda2 (lvmdev).
    To recreate it manually add code to /var/lib/rear/layout/diskrestore.sh or abort.
UserInput -I ADD_CODE_TO_RECREATE_MISSING_PVDEVSDA2LVMDEV needed in /usr/share/rear/layout/prepare/default/600_show_unprocessed.sh line 33
Manually add code that recreates pv:/dev/sda2 (lvmdev)
1) View /var/lib/rear/layout/diskrestore.sh
2) Edit /var/lib/rear/layout/diskrestore.sh
3) Go to Relax-and-Recover shell
4) Continue 'rear recover'
5) Abort 'rear recover'
(default '4' timeout 300 seconds)
1
UserInput: Valid choice number result 'View /var/lib/rear/layout/diskrestore.sh'
#!/bin/bash

LogPrint "Start system layout restoration."

mkdir -p /mnt/local
if create_component "vgchange" "rear" ; then
    lvm vgchange -a n >/dev/null
    component_created "vgchange" "rear"
fi

set -e
set -x


set +x
set +e

LogPrint "Disk layout created."

UserInput -I ADD_CODE_TO_RECREATE_MISSING_PVDEVSDA2LVMDEV needed in /usr/share/rear/layout/prepare/default/600_show_unprocessed.sh line 33
Manually add code that recreates pv:/dev/sda2 (lvmdev)
1) View /var/lib/rear/layout/diskrestore.sh
2) Edit /var/lib/rear/layout/diskrestore.sh
3) Go to Relax-and-Recover shell
4) Continue 'rear recover'
5) Abort 'rear recover'
(default '4' timeout 300 seconds)
5
UserInput: Valid choice number result 'Abort 'rear recover''
ERROR: User chose to abort 'rear recover' in /usr/share/rear/layout/prepare/default/600_show_unprocessed.sh
Some latest log messages since the last called script 600_show_unprocessed.sh:
  2020-01-22 14:37:53.173081088 3) Go to Relax-and-Recover shell
  2020-01-22 14:37:53.177680435 4) Continue 'rear recover'
  2020-01-22 14:37:53.182230065 5) Abort 'rear recover'
  2020-01-22 14:37:53.186757936 (default '4' timeout 300 seconds)
  2020-01-22 14:38:02.984850390 UserInput: 'read' got as user input '5'
  2020-01-22 14:38:02.994150838 UserInput: Valid choice number result 'Abort 'rear recover''
  2020-01-22 14:38:02.999914676 Error detected during restore.
  2020-01-22 14:38:03.004135919 Restoring saved original /var/lib/rear/layout/disklayout.conf
Aborting due to an error, check /var/log/rear/rear-jmt-lxtest1.log for details
Exiting rear recover (PID 1279) and its descendant processes ...
Running exit tasks
You should also rm -Rf /tmp/rear.ubMCqNzTljU5BD6
Terminated

THE DISKLAUOUT.CONF FILE

RESCUE jmt-lxtest1:/var/lib/rear/layout # cat disklayout.conf
# Disk /dev/sdb
# Format: disk <devname> <size(bytes)> <partition label type>
#disk /dev/sdb 0
# Partitions on /dev/sdb
# Format: part <device> <partition size(bytes)> <partition start(bytes)> <partition type|name> <flags> /dev/<partition>
# Disk /dev/sdc
# Format: disk <devname> <size(bytes)> <partition label type>
#disk /dev/sdc 185597952 msdos
# Partitions on /dev/sdc
# Format: part <device> <partition size(bytes)> <partition start(bytes)> <partition type|name> <flags> /dev/<partition>
#part /dev/sdc 184549376 1048576 primary none /dev/sdc1
# Disk /dev/sdd
# Format: disk <devname> <size(bytes)> <partition label type>
#disk /dev/sdd 0
# Partitions on /dev/sdd
# Format: part <device> <partition size(bytes)> <partition start(bytes)> <partition type|name> <flags> /dev/<partition>
# Format for LVM PVs
# lvmdev <volume_group> <device> [<uuid>] [<size(bytes)>]
lvmdev /dev/system /dev/sda2 26mo4z-hKFm-bMq2-5Ufr-1nyj-f2ej-Z67cI1 583659520
# Format for LVM VGs
# lvmgrp <volume_group> <extentsize> [<size(extents)>] [<size(bytes)>]
lvmgrp /dev/system 4096 71247 291827712
# Format for LVM LVs
# lvmvol <volume_group> <name> <size(bytes)> <layout> [key:value ...]
lvmvol /dev/system home 26843545600b linear
lvmvol /dev/system root 10737418240b linear
lvmvol /dev/system swap 2147483648b linear
# Filesystems (only ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs are supported).
# Format: fs <device> <mountpoint> <fstype> [uuid=<uuid>] [label=<label>] [<attributes>]
fs /dev/mapper/system-home /home xfs uuid=18040777-6e86-44f9-b5fe-e443341211ff label=  options=rw,relatime,attr2,inode64,sunit=128,swidth=128,noquota
fs /dev/mapper/system-root / ext4 uuid=2c2bcc03-a40c-42f0-ba0b-01592b0198e6 label= blocksize=4096 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=16384 default_mount_options=user_xattr,acl options=rw,relatime,stripe=16,data=ordered
fs /dev/sda1 /boot/efi vfat uuid=B1F4-03B7 label= options=rw,relatime,fmask=0002,dmask=0002,allow_utime=0020,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro
# Swap partitions or swap files
# Format: swap <filename> uuid=<uuid> label=<label>
swap /dev/mapper/system-swap uuid=742b6328-bfe5-4d13-b19f-314368390f58 label=

jmt-lxtest1.tar.gz

jsmeix commented at 2020-01-23 08:19:

@shaunsJM
as far as I see you do not use multipath so the message

Ignoring sda: it is a path of a multipath device

is a false positive of the is_multipath_path function call

if is_multipath_path ${blockd} ; then
    Log "Ignoring $blockd: it is a path of a multipath device"

in layout/save/GNU/Linux/200_partition_layout.sh
https://github.com/rear/rear/blob/master/usr/share/rear/layout/save/GNU/Linux/200_partition_layout.sh#L384

By luck I experienced the same issue recently, see
https://github.com/rear/rear/issues/2298
where such false positives of the is_multipath_path function
should be avoided in our current ReaR upstream master code via
https://github.com/rear/rear/commit/d8946bcc1cc61086b392e5d85ce956fdcd7d69cf

So I would recommend
"Testing current ReaR upstream GitHub master code"
as described in the section with that title in
https://en.opensuse.org/SDB:Disaster_Recovery
and provide feedback if the improved 'is_multipath_path' function
also helps to avoid that issue in your particular case.

shaunsJM commented at 2020-01-23 15:10:

@jsmeix thanks so much! the current ReaR upstream GitHub master code solved this issue for me.
I apologize for not finding your solution during my searches prior to making a posting.
Thanks again!
Shaun

shaunsJM commented at 2020-01-23 15:10:

Closing.

jsmeix commented at 2020-01-28 10:04:

@shaunsJM
no worries - I am glad I could help you so easily.

Furthermore your issue report proved that
https://github.com/rear/rear/issues/2298
was not a singular weird exceptional case and that
https://github.com/rear/rear/pull/2299
is actually a right fix (that works at least for you and me).

[Export of Github issue for rear/rear.]