#2554 Issue closed: SLES 15 SP2 grub2-install error (missing /dev/nvram) on PowerVM pSeries

Labels: enhancement, fixed / solved / done, special hardware or VM

viper1986 opened issue at 2021-01-15 12:49:

Relax-and-Recover (ReaR) Issue Template

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

  • ReaR version ("/usr/sbin/rear -V"): Relax-and-Recover 2.6 / 2020-06-17

  • OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):

RESCUE ite-itesletest15sp2:/etc/rear # cat /etc/rear/os.conf
OS_VENDOR=SUSE_LINUX
OS_VERSION=15.2
# The following information was added automatically by the mkbackup workflow:
ARCH='Linux-ppc64le'
OS='GNU/Linux'
OS_VERSION='15.2'
OS_VENDOR='SUSE_LINUX'
OS_VENDOR_VERSION='SUSE_LINUX/15.2'
OS_VENDOR_ARCH='SUSE_LINUX/ppc64le'
OS_MASTER_VENDOR='SUSE'
# End of what was added automatically by the mkbackup workflow.
  • ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
BACKUP=NETFS
BACKUP_URL=nfs://169.254.252.15/data/col1/rear
BACKUP_OPTIONS="nfsvers=3,nolock"
OUTPUT=ISO
OUTPUT_URL=nfs://169.254.251.200/rear
OUTPUT_OPTIONS="nfsvers=3,nolock"
KEEP_OLD_OUTPUT_COPY=y
REAR_INITRD_COMPRESSION=lzma
AUTOEXCLUDE_MULTIPATH=n
BOOT_OVER_SAN=y
REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" snapper chattr lsattr )
COPY_AS_IS=( "${COPY_AS_IS[@]}" /usr/lib/snapper/installation-helper /etc/snapper/config-templates/default )
POST_RECOVERY_SCRIPT=( 'if snapper --no-dbus -r $TARGET_FS_ROOT get-config | grep -q "^QGROUP.*[0-9]/[0-9]" ; then snapper --no-dbus -r $TARGET_FS_ROOT set-config QGROUP= ; snapper --no-dbus -r $TARGET_FS_ROOT setup-quota && echo snapper setup-quota done || echo snapper setup-quota failed ; else echo snapper setup-quota not used ; fi' )

  • Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR): PowerVM ,platform : pSeries

  • System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device): PPC64LE

  • Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot): GRUB2

  • Description of the issue (ideally so that others can reproduce it):
    I want to recover serwer to another LPAR.
    But GRUB2 fail to install.

  • Workaround, if any:

  • Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
    recover.log

  • Description

Running mkinitrd...
Recreated initrd (/sbin/mkinitrd).
Installing GRUB2 boot loader on PPC64/PPC64LE...
Determining where to install GRUB2 (no GRUB2_INSTALL_DEVICES specified)
Found PPC PReP boot partition /dev/mapper/mpathb-part1 - installing GRUB2 there
Failed to install GRUB2 on PPC PReP boot partition /dev/mapper/mpathb-part1
Failed to install GRUB2 - you may have to manually install it
PPC PReP Boot partition found: /dev/mapper/mpathb-part1
Boot device disk is /dev/mapper/mpathb.
Limiting bootlist to 5 entries as a maximum...
Set LPAR bootlist to '/dev/sdh /dev/sdf /dev/sdd /dev/sdb /dev/sdg'
WARNING:
For this system
SUSE_LINUX/15.2 on Linux-ppc64le (based on SUSE/15/ppc64le)
there is no code to install a boot loader on the recovered system
or the code that we have failed to install the boot loader correctly.
Please contribute appropriate code to the Relax-and-Recover project,
see http://relax-and-recover.org/development/
Take a look at the scripts in /usr/share/rear/finalize - for example
for PC architectures like x86 and x86_64 see the script
/usr/share/rear/finalize/Linux-i386/660_install_grub2.sh
and for POWER architectures like ppc64le see the script
/usr/share/rear/finalize/Linux-ppc64le/660_install_grub2.sh

|  IF YOU DO NOT INSTALL A BOOT LOADER MANUALLY,  |
|  THEN YOUR SYSTEM WILL NOT BE ABLE TO BOOT.     |

You can use 'chroot /mnt/local bash --login'
to change into the recovered system and
manually install a boot loader therein.

jsmeix commented at 2021-01-15 14:30:

@viper1986
your https://github.com/rear/rear/files/5820527/recover.log
contains (excerpts):

+ source /usr/share/rear/finalize/Linux-ppc64le/660_install_grub2.sh
...
++ chroot /mnt/local /bin/bash --login -c 'grub2-install  /dev/mapper/mpathb-part1'
Installing for powerpc-ieee1275 platform.
nvram: ERROR: cannot open "/dev/nvram": No such file or directory
nvram: ERROR: cannot open "/dev/misc/nvram": No such file or directory
grub2-install: error: `nvsetenv' failed. 
You will have to set `boot-device' variable manually.  At the IEEE1275 prompt, type:
  setenv boot-device /vdevice/vfc-client@30000003/disk@500507680c354c27
.
++ LogPrintError 'Failed to install GRUB2 on PPC PReP boot partition /dev/mapper/mpathb-part1'

A few days ago I received a SUSE internal bug report with same symptoms.

I am not a PPC user and I am not a booting expert
in particular I know nothing about booting specific things on PPC
but from what I and a colleague at SUSE found out in that SUSE internal bug report
the issue seems to be as follows:

I guess /proc/cpuinfo shows in your case something like

platform        : pSeries

but the code in
usr/share/rear/finalize/Linux-ppc64le/620_install_grub2.sh
that is (excerpt)

if [[ $(awk '/platform/ {print $NF}' < /proc/cpuinfo) == PowerNV ]] ; then
    grub2_install_option="--no-nvram"
fi

only tests for

platform    : PowerNV

so grub2-install gets called without the --no-nvram option
and because there is no /dev/nvram in the ReaR recovery system and
then also not in the recreated system inside 'chroot /mnt/local'
grub2-install that is run inside 'chroot /mnt/local' fails with

nvram: ERROR: cannot open "/dev/nvram": No such file or directory
nvram: ERROR: cannot open "/dev/misc/nvram": No such file or directory

As far as I was told /dev/nvram exists on the original system
but currently I have no idea why it is missing in the ReaR recovery system.
On a KVM/QEMU x86_64 virtual machine with legacy BIOS
(running on my x86_64 homeoffice laptop)
I have /dev/nvram both in the ReaR recovery system and
also inside 'chroot /mnt/local' because ReaR bind-mounts it.
So the missing /dev/nvram in the ReaR recovery system on PPC
seems to be something that is specific on PPC architecture.

Therefore it seems calling grub2-install with the --no-nvram option
could be more a workaround for the missing /dev/nvram
in the ReaR recovery system on PPC than the real solution
which would be to get /dev/nvram in the ReaR recovery system
also on PPC if it exists on the original system.

To get grub2-install called with the --no-nvram option
it should be sufficient to adapt the code part

if [[ $(awk '/platform/ {print $NF}' < /proc/cpuinfo) == PowerNV ]]
then grub2_install_option="--no-nvram"
fi

in usr/share/rear/finalize/Linux-ppc64le/660_install_grub2.sh
according to what platform vaule is shown in /proc/cpuinfo
on your particular system - e.g. replace PowerNV
by the platform vaule of your particular system.

A side note FYI:
At ReaR upstream @schabrolles (from IBM France)
who contributed basically all latest ReaR enhancements for PPC
got other tasks by IBM so he does no longer actively work for ReaR.
Currently there is nobody at ReaR upstream
who could properly maintain the PPC related code in ReaR
to keep it continuously up to date with newest things
so ReaR on PPC basically rots at ReaR upstream :-(

abbbi commented at 2021-01-15 17:56:

Maybe as a note: it seems the existence of the /dev/nvram device is probably dependend wether if the system is booted from OpenFirmware or SMS:

Install GRUB to /boot/grub (the default path). Option --no-nvram prevents GRUB from setting the Open Firmware boot-device nvram variable.

Was the powerpc lpar/system in question booted via SMS or openfirmware?
Qemu allways uses OpenFirmware. Maybe backup was created while system was booted via OpenFirmware and recovery ISO was booted from SMS? Or vice versa.

viper1986 commented at 2021-01-18 18:28:

@jsmeix thank you for reply.
Yes, pSeries was from /proc/cpuinfo info.

Change some configuration in site.conf and add

BACKUP_PROG_INCLUDE=( /usr/local /home /srv /var /opt /crash /tmp /boot/grub2/powerpc-ieee1275 /dev)

Now in the tar file I have all filesystems ( I noticed that I do not have e.g /opt ).

I booted system from ISO and change
/usr/share/rear/finalize/Linux-ppc64le/660_install_grub2.sh
replace PowerNV to pSeries:

User confirmed restored files
Running mkinitrd...
Recreated initrd (/sbin/mkinitrd).
Installing GRUB2 boot loader on PPC64/PPC64LE...
Determining where to install GRUB2 (no GRUB2_INSTALL_DEVICES specified)
Found PPC PReP boot partition /dev/mapper/mpathb-part1 - installing GRUB2 there
PPC PReP Boot partition found: /dev/mapper/mpathb-part1
Boot device disk is /dev/mapper/mpathb.
Limiting bootlist to 5 entries as a maximum...
Set LPAR bootlist to '/dev/sdh /dev/sdf /dev/sdd /dev/sdb /dev/sdg'
Finished 'recover'. The target system is mounted at '/mnt/local'.
Exiting rear recover (PID 1232) and its descendant processes ...

GRUB was install and all looks fine but ...
When I booted system, it boot from cd-rom.
Then I unmount cd from LPAR, system was properly boot from hard drive.
But bootlist looks like:

sles15sp2:~ # bootlist -o -m normal
cdrom
sda
eth0
    speed = auto
    duplex = auto
    BOOTP Server: 000.000.000.000
    Client: 000.000.000.000
    Gateway: 000.000.000.000

So I'm not sure that the

Set LPAR bootlist to '/dev/sdh /dev/sdf /dev/sdd /dev/sdb /dev/sdg'

is working correctly.

@abbbi
I boot iso via SMS mode.
In tar file I have /dev/nvram file, but when ReaR install Grub the file is gone.
I will talk about nvram with my team because I don't know anything about it.

hramrach commented at 2021-01-21 11:54:

Maybe as a note: it seems the existence of the /dev/nvram device is probably dependend wether if the system is booted from OpenFirmware or SMS:

Install GRUB to /boot/grub (the default path). Option --no-nvram prevents GRUB from setting the Open Firmware boot-device nvram variable.

Was the powerpc lpar/system in question booted via SMS or openfirmware?
Qemu allways uses OpenFirmware. Maybe backup was created while system was booted via OpenFirmware and recovery ISO was booted from SMS? Or vice versa.

I doubt this. SMS is an extension on top of ofw, and uses ofw to boot. qemu does not have SMS yet it has nvram (it has completely different ofw implementtaion, though). In fact I have not seen a pSeries platform that does not support nvram on using POWER7 and later hardware.

jsmeix commented at 2021-01-21 13:59:

@viper1986
you have now

BACKUP_PROG_INCLUDE=( ... /dev)

which is not right in general
because you should not have device nodes in the backup
because you should not restore device nodes from a backup
because that could overwrite the current device nodes of the system
where the backup is restored but one should not overwrite device nodes
because the kernel (and udev) generate them right
so overwriting device nodes may cause any kind of weird arbitrary failures.

viper1986 commented at 2021-01-21 15:17:

@jsmeix
Thank you for information. I will delete /dev from BACKUP_PROG_INCLUDE and test recover again.

jsmeix commented at 2021-01-21 15:25:

@viper1986
please try out an additional setting in etc/rear/local.conf

Add to your etc/rear/local.conf

MODULES_LOAD+=( nvram )

which should result an automated "modprobe nvram" during
ReaR recovery system startup via its startup script
/etc/scripts/system-setup.d/40-start-udev-or-load-modules.sh

The source code file of that startup script is
usr/share/rear/skel/default/etc/scripts/system-setup.d/40-start-udev-or-load-modules.sh

The reason behind is that - as far as I know - on POWER architecture
the nvram kernel driver is not or no longer built into the kernel
but nowadays it is built as a kernel module.

When the kernel module nvram is loaded its associated device node /dev/nvram
should be created by the kernel itself and then it should work to call
grub2-install without the additional --no-nvram option.

I.e. with MODULES_LOAD+=( nvram ) things should "just work"
so that you should no longer need your change in
usr/share/rear/finalize/Linux-ppc64le/660_install_grub2.sh
where you had replaced 'PowerNV' with 'pSeries'.

viper1986 commented at 2021-01-21 18:31:

@jsmeix
Thank will try and back with the update.

jsmeix commented at 2021-02-01 15:10:

I got a confirmation from another SLES15 user on PPC that with

MODULES=( 'all_modules' )
MODULES_LOAD+=( nvram )

things work for him with ReaR version 2.4

Also

MODULES+=( nvram )
MODULES_LOAD+=( nvram )

should be sufficient

Since ReaR version 2.5 there is MODULES=( 'all_modules' ) by default so only

MODULES_LOAD+=( nvram )

should be sufficient since ReaR version 2.5.

jsmeix commented at 2021-03-04 12:26:

The needed adaptions to make nvram kernel module loading automated
should be in https://github.com/rear/rear/pull/2580

jsmeix commented at 2021-03-09 13:42:

With https://github.com/rear/rear/pull/2580 merged
this issue should be completely fixed.

[Export of Github issue for rear/rear.]