#2003 Issue closed: ReaR does not preserve Grub install location

Labels: bug, discuss / RFC, no-issue-activity

gozora opened issue at 2018-12-13 12:40:

  • ReaR version ("/usr/sbin/rear -V"):
    Relax-and-Recover 2.4 / Git

  • OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"):
    SUSE Linux Enterprise Server for SAP Applications 12 SP2

  • ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

# cat /etc/rear/local.conf
UTPUT=ISO
BACKUP=NETFS
BACKUP_URL=nfs://backup/mnt/rear
BACKUP_OPTIONS="nfsvers=3,nolock"

SSH_FILES='yes'

# Network
USE_STATIC_NETWORKING="yes"
USE_DHCLIENT="no"
NETWORKING_PREPARATION_COMMANDS=( 'ip addr add 192.168.56.200/24 dev eth1' 'ip link set dev eth0 up' 'return' )

  • Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR):
    VirtualBox VM

  • System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
    x86 compatible

  • Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
    BIOS/GRUB

  • Storage (lokal disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
    Local (no multipath)

  • Description of the issue (ideally so that others can reproduce it):
    GRUB that was originally installed on /boot (/dev/sda1) is reinstalled on rear recover to MBR (/dev/sda) which caused unbootable system after upgrade from SLE12-SP2->SLE2-SP3:

post_upgrade_fail

This behavior should not apply to installations where GRUB is installed to MBR during OS installation.

  • Workaround, if any:
    yast bootloader->OK to reinstall bootloader to its original location (/boot) or update /etc/default/grub_installdevice
- /dev/sda1
+ /dev/sda
activate
generic_mbr

To reflect new location of boot loader installed by rear recover

This is issue is reproducible as follows:

  1. Install boot loader to /boot partition during OS installation
  2. backup / restore OS with ReaR

  3. do SP upgrade (e.g. from SP2->SP3) either offline (by booting from SLE ISO) or by zypper dist-upgrade

  4. Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):

gozora commented at 2018-12-13 12:41:

@jsmeix I've dared to assign you since we are dealing here with SuSE ;-), hope you don't mind ;-).

V.

gozora commented at 2018-12-13 13:23:

IMHO one sufficient thing would be to patch /mnt/local/etc/default/grub_installdevice with information from finalize/Linux-i386/620_install_grub2.sh is similar way like done in finalize/GNU/Linux/280_migrate_uuid_tags.sh

V.

jsmeix commented at 2018-12-14 10:02:

I know about this kind of issue since a longer time
but I never did a thorough analysis.

As far as I know "rear recover" basically installs GRUB2 into MBR
(or more precisely on the boot disk device node like /dev/sda
but not into a boot partition device node like /dev/sda1)
regardless how GRUB2 was installed on the original system
(I never cared about the GRUB legacy scripts in ReaR, they seem
to "somehow work good enough" because there are no bug reports).

What I implemented as a generic way to give the user final power
where GRUB2 gets installed is GRUB2_INSTALL_DEVICES, cf.
https://github.com/rear/rear/blob/master/usr/share/rear/conf/default.conf#L2450

@gozora
does `GRUB2_INSTALL_DEVICES="/dev/sa1" help in your case?

Did you re-run "rear mkbackup" after you did the
upgrade from SLE12-SP2 to SLE2-SP3 ?

You need to re-do "rear mkbackup" and re-test "rear recover"
after any change of the basic system, cf.
https://en.opensuse.org/SDB:Disaster_Recovery
because you need in particular the new config files
of the upgraded system in your backup because in particular
GRUB2 is installed from within a 'chroot' into the up to that point
recreated system so that GRUB2 gets installed within the
environment of the up to that point recreated system which means
GRUB2 gets installed with the config files from the restored backup.

But I think this does not affect the target location whereto
GRUB2 will be installed (e.g. /dev/sda versus /dev/sda1)
because that is provided as command line argument
to the grub2-install command, cf.
finalize/Linux-i386/620_install_grub2.sh
and that command line argument can be specified
via GRUB2_INSTALL_DEVICES.

jsmeix commented at 2018-12-14 10:15:

@gozora
perhaps I understand now what you meant with

1. Install boot loader to /boot partition during OS installation
2. backup / restore OS with ReaR
3. do SP upgrade (e.g. from SP2->SP3)

During 2. backup / restore OS with ReaR
the system will be changed by "rear recover"
from GRUB2 on /boot (i.e. on /dev/sda1)
to GRUB2 in MBR (i.e. on /dev/sda)
but the matching GRUB2 config file /etc/default/grub_installdevice
is not updated accordingly so that the actual GRUB2 installation
does no longer match the GRUB2 config files.

Then 3. do SP upgrade (e.g. from SP2->SP3) results
a unbootable system because of what is likely a bug in this step.
I think there is a bug in that step because I assume the system
was still bootable after 2. backup / restore OS with ReaR
(i.e. the new GRUB2 installation by "rear recover" worked in practice)
so that 3. do SP upgrade (e.g. from SP2->SP3) made a bootable system
no longer bootable.

Nevertheless it seems there is also a generic bug in ReaR
because it does not autodetect the actual GRUB2 install target
(e.g. whether it is /dev/sda or /dev/sda1) but by default
(i.e. unless GRUB2_INSTALL_DEVICES was specified)
"rear recover" may change the actual GRUB2 install target
which it should not do.

I think what is not right is to let "rear recover" silently change
the actual GRUB2 install target and then even more change things
and adapt GRUB2 config files on the recreated system because this
is not what the admin expects to happen.

jsmeix commented at 2018-12-14 10:50:

By the way:
On my SLES15-like openSUSE Leap 15.0
x86_64 system with GPT and UEFI boot
I do not have /etc/default/grub_installdevice
I guess
https://bugzilla.opensuse.org/show_bug.cgi?id=994322
is related.

gozora commented at 2018-12-14 10:50:

I don't see any easy way how to completely and correctly fix this problem.
I theory we would need similar approach as bootinfoscript, which scans all devices that can contain boot loader, store them during rear mkbackup and restore it afterwards.

V.

gozora commented at 2018-12-14 11:03:

By the way:
On my SLES15-like openSUSE Leap 15.0
x86_64 system with GPT and UEFI boot
I do not have /etc/default/grub_installdevice
I guess
https://bugzilla.opensuse.org/show_bug.cgi?id=994322
is related.

Sweet, this makes my idea in https://github.com/rear/rear/issues/2003#issuecomment-446966462 obsolete ...

jsmeix commented at 2018-12-14 12:21:

I assume the root cause is what is described in the section
"Disaster recovery does not just work" in
https://en.opensuse.org/SDB:Disaster_Recovery
(excerpt):

For example there is the general problem
that it is impossible to determine in a reliable way
how a running system was actually booted.

Perhaps it is possible to find something that indicates
when GRUB2 is not installed in MBR
(e.g. when it is not on /dev/sda but elsewhere)
and in this case error out during "rear mkrescue"
unless GRUB2_INSTALL_DEVICES was specified
to enforce the user when things are not clear to
explicitly tell whereto GRUB2 needs to be installed?

jsmeix commented at 2018-12-14 12:30:

@gozora
during "rear recover" you should have got LogPrint messages
from finalize/Linux-i386/620_install_grub2.sh like

Installing GRUB2 boot loader...
Determining where to install GRUB2 (no GRUB2_INSTALL_DEVICES specified)
Found possible boot disk /dev/sda - installing GRUB2 there

so that you should (theoretically) have been informed
about the changed location whereto GRUB2 was installed.

Based on that another idea how to deal with such issues is
to let the user actively confirm whereto GRUB2 gets installed
by a UserInput dialog.

@gozora
if you got the above LogPrint messages I will do a pull request (next week)
with an added UserInput dialog to let the user actively confirm
whereto GRUB2 gets installed so that you could try out if that
would make things behave more fail-safe for you.

jsmeix commented at 2018-12-14 13:12:

Third offhanded idea how to reasonably guess
where GRUB2 is acually installed on the original system:

In layout/save/default/445_guess_bootloader.sh the code part

# Finally guess the used bootloader by inspecting the first bytes on all disks
# and use the first one that matches a known bootloader string:
for block_device in /sys/block/* ; do
    ...
done

guesses what bootloader is used by inspecting certain disk devices
(currently as far as I see only e.g. /dev/sda but not yet /dev/sda1)
so that it is perhaps possible to re-use and enhance that kind of code
to remember on what disk device the guessed bootloader was found
and use by default that disk device whereto GRUB2 will get re-installed
during "rear recover" unless GRUB2_INSTALL_DEVICES was specified.

If it finds signs of GRUB2 being installed on more than one disk device
we might blindly re-install GRUB2 on all of them (which would be right
in case of RAID1 consisting of /dev/sda and /dev/sdb)
or we could let ReaR behave better safe than sorry by default and
error out with "ambiguous GRUB2 location" during "rear mkrescue" as in
https://github.com/rear/rear/issues/2003#issuecomment-447309268

gozora commented at 2018-12-16 17:12:

@jsmeix don't waste your time too much on this one. It can be rather easy to fix in running system.

I'll take a look on Grub installation code and see if there is some way, how it can be enhanced.

V.

github-actions commented at 2020-06-28 01:33:

Stale issue message

[Export of Github issue for rear/rear.]