#1828 Issue closed: on SLES15 with btrfs after "rear recover" GRUB2 fails to boot (manual GRUB2 install helps)¶
Labels: enhancement, bug, cleanup, fixed / solved / done
jsmeix opened issue at 2018-06-13 10:15:¶
-
ReaR version ("/usr/sbin/rear -V"):
Current ReaR GitHub master code (i.e. what will become ReaR 2.4) -
OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"):
# cat /etc/os-release NAME="SLES" VERSION="15" VERSION_ID="15" PRETTY_NAME="SUSE Linux Enterprise Server 15" ID="sles" ID_LIKE="suse" ANSI_COLOR="0;32" CPE_NAME="cpe:/o:suse:sles:15"
- ReaR configuration files ("cat /etc/rear/site.conf" or "cat /etc/rear/local.conf"):
OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://10.160.4.244/nfs
REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" snapper chattr lsattr )
COPY_AS_IS=( "${COPY_AS_IS[@]}" /usr/lib/snapper/installation-helper /etc/snapper/config-templates/default )
BACKUP_PROG_INCLUDE=( /opt /tmp /usr/local /boot/grub2/i386-pc /boot/grub2/x86_64-efi /srv /var )
-
System architecture (x86 compatible or POWER and/or what kind of virtual machine):
Virtual KVM/QEMU x86_64 machine with a single 20GiB virtual harddisk -
Are you using BIOS or UEFI or another way to boot?
BIOS -
Brief description of the issue:
Reboot after "rear recover" fails in GRUB2 (got into grub rescue mode)
because grub2 could not loadnormal.mod(and other modules)
because - as far as I found out bylsin grub rescue mode - there
are no grub2 modules accessible by grub2.
It seems the contents of the btrfs subvolume/@/boot/grub2/i386-pc
that is mounted at/boot/grub2/i386-pcon the original system
are not accessible during boot time by grub2 as it was installed
by "rear recover" on the replacement system. -
Work-around, if any:
After "rear recover" when still in the running ReaR recovery system
manually re-install GRUB2 on the replacement system as follows:
RESCUE e220:~ # rear -D recover ... Running mkinitrd... Recreated initrd (/sbin/mkinitrd). Installing GRUB2 boot loader Finished recovering your system. You can explore it under '/mnt/local'. RESCUE e220:~ # target_system_filesystem_root=/mnt/local RESCUE e220:~ # mount -t proc none $target_system_filesystem_root/proc RESCUE e220:~ # mount -t sysfs sys $target_system_filesystem_root/sys RESCUE e220:~ # mount -o bind /dev $target_system_filesystem_root/dev RESCUE e220:~ # chroot $target_system_filesystem_root e220:/ # /sbin/mkinitrd Creating initrd: /boot/initrd-4.12.14-15-default ... dracut: Stored kernel commandline: dracut: root=UUID=f4361595-0a29-4b1c-9d71-1560cdfc5de3 rootfstype=btrfs rootflags=rw,relatime,space_cache,subvolid=267,subvol=/@/.snapshots/1/snapshot,subvol=@/.snapshots/1/snapshot dracut: *** Creating image file '/boot/initrd-4.12.14-15-default' *** dracut: *** Creating initramfs image file '/boot/initrd-4.12.14-15-default' done *** e220:/ # /usr/sbin/grub2-mkconfig -o /boot/grub2/grub.cfg Generating grub configuration file ... Found theme: /boot/grub2/themes/SLE/theme.txt Found linux image: /boot/vmlinuz-4.12.14-15-default Found initrd image: /boot/initrd-4.12.14-15-default done e220:/ # /usr/sbin/grub2-install --force /dev/sda Installing for i386-pc platform. Installation finished. No error reported. e220:/ # exit exit RESCUE e220:~ # reboot
Then the replacement system boots
but I still notice that there is a possibly grub error message
which is too fast flashing away so that I cannot read it.
I need to investigate what the right way is how to correctly
install GRUB2 on SLES15 with btrfs.
On SLES15 with e.g. ext4 things still work, cf.
https://github.com/rear/rear/wiki/Test-Matrix-rear-2.4
This issue is no blocker to release ReaR 2.4
because SUSE customers can get a fix
via a SUSE specific maintenance update
and non SUSE customers who try out SLES 15
can use our GitHub master code when it is fixed there.
FYI:
As usual for each and every new SLES version
its default btrfs stuff changes in incompatible ways
which breaks ReaR and I need to fix things - just business as usual.
I could have sworn it had worked with some earlier SLES 15 beta
but I do no longer have SLES 15 beta ISOs to re-test that and
in the end it does not matter up to what version things had worked.
jsmeix commented at 2018-06-13 10:18:¶
Some more details from within the ReaR recovery system:
RESCUE e220:~ # rear -D recover
Using log file: /var/log/rear/rear-e220.log
...
Restoring finished.
Recreating directories (with permissions) from /var/lib/rear/recovery/directories_permissions_owner_group
Updating udev configuration (70-persistent-net.rules)
Running mkinitrd...
Recreated initrd (/sbin/mkinitrd).
Installing GRUB2 boot loader
Finished recovering your system. You can explore it under '/mnt/local'.
RESCUE e220:~ # parted -s /dev/sda unit MiB print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sda: 20480MiB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1.00MiB 9.00MiB 8.00MiB sda1 bios_grub
2 9.00MiB 12906MiB 12897MiB btrfs sda2
4 12906MiB 18431MiB 5525MiB xfs sda4 legacy_boot
3 18431MiB 20480MiB 2049MiB linux-swap(v1) sda3 swap
RESCUE e220:~ # findmnt -a | grep sda
`-/mnt/local /dev/sda2[/@/.snapshots/1/snapshot] btrfs rw,relatime,space_cache,subvolid=267,subvol=/@/.snapshots/1/snapshot
|-/mnt/local/.snapshots /dev/sda2[/@/.snapshots] btrfs rw,relatime,space_cache,subvolid=266,subvol=/@/.snapshots
|-/mnt/local/opt /dev/sda2[/@/opt] btrfs rw,relatime,space_cache,subvolid=262,subvol=/@/opt
|-/mnt/local/tmp /dev/sda2[/@/tmp] btrfs rw,relatime,space_cache,subvolid=260,subvol=/@/tmp
|-/mnt/local/usr/local /dev/sda2[/@/usr/local] btrfs rw,relatime,space_cache,subvolid=259,subvol=/@/usr/local
|-/mnt/local/boot/grub2/i386-pc /dev/sda2[/@/boot/grub2/i386-pc] btrfs rw,relatime,space_cache,subvolid=264,subvol=/@/boot/grub2/i386-pc
|-/mnt/local/boot/grub2/x86_64-efi /dev/sda2[/@/boot/grub2/x86_64-efi] btrfs rw,relatime,space_cache,subvolid=263,subvol=/@/boot/grub2/x86_64-efi
|-/mnt/local/srv /dev/sda2[/@/srv] btrfs rw,relatime,space_cache,subvolid=261,subvol=/@/srv
|-/mnt/local/var /dev/sda2[/@/var] btrfs rw,relatime,space_cache,subvolid=258,subvol=/@/var
|-/mnt/local/home /dev/sda4 xfs rw,relatime,attr2,inode64,noquota
RESCUE e220:~ # less /var/log/rear/rear-e220.log
...
++ echo -e 'Installing GRUB2 boot loader'
...
++ grub2-install --root-directory=/mnt/local /dev/sda
Installing for i386-pc platform.
grub2-install: warning: cannot open directory `/usr/share/locale': No such file or directory.
Installation finished. No error reported.
...
RESCUE e220:~ # find /mnt/local -ls 2>/dev/null | grep 'normal\.mod'
47542 116 -rw-r--r-- 1 root root 116504 Mar 18 13:38 /mnt/local/usr/lib/grub2/i386-pc/normal.mod
47543 792 -rw-r--r-- 1 root root 810048 Mar 18 13:38 /mnt/local/usr/lib/grub2/i386-pc/normal.module
48096 172 -rw-r--r-- 1 root root 174104 Mar 18 13:38 /mnt/local/usr/lib/grub2/x86_64-efi/normal.mod
48097 1208 -rw-r--r-- 1 root root 1236288 Mar 18 13:38 /mnt/local/usr/lib/grub2/x86_64-efi/normal.module
966 116 -rw-r--r-- 1 root root 116504 Jun 13 09:17 /mnt/local/boot/grub2/i386-pc/normal.mod
RESCUE e220:~ # mkdir /tmp/btrfsroot
RESCUE e220:~ # mount -o subvolid=0 /dev/sda2 /tmp/btrfsroot
RESCUE e220:~ # find /tmp/btrfsroot -ls 2>/dev/null | grep 'normal\.mod'
966 116 -rw-r--r-- 1 root root 116504 Jun 13 09:17 /tmp/btrfsroot/@/boot/grub2/i386-pc/normal.mod
47542 116 -rw-r--r-- 1 root root 116504 Mar 18 13:38 /tmp/btrfsroot/@/.snapshots/1/snapshot/usr/lib/grub2/i386-pc/normal.mod
47543 792 -rw-r--r-- 1 root root 810048 Mar 18 13:38 /tmp/btrfsroot/@/.snapshots/1/snapshot/usr/lib/grub2/i386-pc/normal.module
48096 172 -rw-r--r-- 1 root root 174104 Mar 18 13:38 /tmp/btrfsroot/@/.snapshots/1/snapshot/usr/lib/grub2/x86_64-efi/normal.mod
48097 1208 -rw-r--r-- 1 root root 1236288 Mar 18 13:38 /tmp/btrfsroot/@/.snapshots/1/snapshot/usr/lib/grub2/x86_64-efi/normal.module
RESCUE e220:~ # umount /tmp/btrfsroot
RESCUE e220:~ # target_system_filesystem_root=/mnt/local
RESCUE e220:~ # mount -t proc none $target_system_filesystem_root/proc
RESCUE e220:~ # mount -t sysfs sys $target_system_filesystem_root/sys
RESCUE e220:~ # mount -o bind /dev $target_system_filesystem_root/dev
RESCUE e220:~ # chroot $target_system_filesystem_root
e220:/ # /sbin/mkinitrd
Creating initrd: /boot/initrd-4.12.14-15-default
dracut: Executing: /usr/bin/dracut --logfile /var/log/YaST2/mkinitrd.log --force /boot/initrd-4.12.14-15-default 4.12.14-15-default
dracut: dracut module 'dmraid' will not be installed, because command 'dmraid' could not be found!
dracut: dracut module 'dmraid' will not be installed, because command 'dmraid' could not be found!
dracut: *** Including module: bash ***
dracut: *** Including module: systemd ***
dracut: *** Including module: warpclock ***
dracut: *** Including module: systemd-initrd ***
dracut: *** Including module: i18n ***
dracut: Could not find FONT_MAP none!
dracut: *** Including module: drm ***
dracut: *** Including module: plymouth ***
dracut: *** Including module: btrfs ***
dracut: *** Including module: kernel-modules ***
dracut: *** Including module: qemu ***
dracut: *** Including module: rootfs-block ***
dracut: *** Including module: suse-xfs ***
dracut: *** Including module: terminfo ***
dracut: *** Including module: udev-rules ***
dracut: Skipping udev rule: 40-redhat.rules
dracut: Skipping udev rule: 50-firmware.rules
dracut: Skipping udev rule: 50-udev.rules
dracut: Skipping udev rule: 91-permissions.rules
dracut: Skipping udev rule: 80-drivers-modprobe.rules
dracut: *** Including module: dracut-systemd ***
dracut: *** Including module: haveged ***
dracut: *** Including module: usrmount ***
dracut: *** Including module: base ***
dracut: *** Including module: fs-lib ***
dracut: *** Including module: shutdown ***
dracut: *** Including module: suse ***
dracut: *** Including modules done ***
dracut: *** Installing kernel module dependencies and firmware ***
dracut: *** Installing kernel module dependencies and firmware done ***
dracut: *** Resolving executable dependencies ***
dracut: *** Resolving executable dependencies done***
dracut: *** Hardlinking files ***
dracut: *** Hardlinking files done ***
dracut: *** Stripping files ***
dracut: *** Stripping files done ***
dracut: *** Generating early-microcode cpio image ***
dracut: *** Store current command line parameters ***
dracut: Stored kernel commandline:
dracut: root=UUID=f4361595-0a29-4b1c-9d71-1560cdfc5de3 rootfstype=btrfs rootflags=rw,relatime,space_cache,subvolid=267,subvol=/@/.snapshots/1/snapshot,subvol=@/.snapshots/1/snapshot
dracut: *** Creating image file '/boot/initrd-4.12.14-15-default' ***
dracut: *** Creating initramfs image file '/boot/initrd-4.12.14-15-default' done ***
e220:/ # /usr/sbin/grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found theme: /boot/grub2/themes/SLE/theme.txt
Found linux image: /boot/vmlinuz-4.12.14-15-default
Found initrd image: /boot/initrd-4.12.14-15-default
done
e220:/ # /usr/sbin/grub2-install --force /dev/sda
Installing for i386-pc platform.
Installation finished. No error reported.
e220:/ # exit
exit
RESCUE e220:~ # reboot
jsmeix commented at 2018-06-13 12:02:¶
As expected things work when I have the contents of
boot/grub2/i386-pc/
not in a separated btrfs subvolume but in a normal directory in the
btrfs default subvolume
(i.e. in the btrfs subvolume /@/.snapshots/1/snapshot that is mounted
at /).
How I did that (on the original system):
# mkdir -p /boot/grub2/i386-pc-sv # vi /etc/fstab [replace the mount point /boot/grub2/i386-pc by /boot/grub2/i386-pc-sv] # reboot [afterwards the btrfs subvolume /@/boot/grub2/i386-pc is mounted at /boot/grub2/i386-pc-sv] # cp -r /boot/grub2/i386-pc-sv/* /boot/grub2/i386-pc [now the contents are also in the normal directory boot/grub2/i386-pc] # df --output=source,target,file /boot/grub2/i386-pc/normal.mod Filesystem Mounted on File /dev/sda2 / /boot/grub2/i386-pc/normal.mod # findmnt -o SOURCE,TARGET,FSTYPE / SOURCE TARGET FSTYPE /dev/sda2[/@/.snapshots/1/snapshot] / btrfs # rear -D mkbackup
Of course this is not a solution.
It is only to prove that it is the btrfs subvolume stuff that makes the
difference.
It seems when GRUB2 was installed by "rear recover" on the replacement
system
then during boot grub2 can access files that are in the btrfs default
subvolume
/@/.snapshots/1/snapshot but not files that are in other separated
btrfs subvolumes like /@/boot/grub2/i386-pc.
In contrast when GRUB2 is installed by the SUSE installer on the
original system
then during boot grub2 can access files that are in other separated
btrfs subvolumes like /@/boot/grub2/i386-pc.
jsmeix commented at 2018-06-15 15:23:¶
It does not help to modify
usr/share/rear/finalize/Linux-i386/620_install_grub2.sh
@@ -83,7 +83,7 @@
partnr=$((partnr - 1))
if [[ "$bootdisk" == "$disk" ]]; then
- #chroot $TARGET_FS_ROOT $grub_name-mkconfig -o /boot/$grub_name/grub.cfg
+ chroot $TARGET_FS_ROOT $grub_name-mkconfig -o /boot/$grub_name/grub.cfg
#chroot $TARGET_FS_ROOT $grub_name-install "$bootdisk"
$grub_name-install --root-directory=$TARGET_FS_ROOT $bootdisk
else
to let grub2-mkconfig be run before grub2-install is run
as in my workaround to manually re-install GRUB2 in
https://github.com/rear/rear/issues/1828#issue-331930433
With that modified 620_install_grub2.sh I get during rear -D recover
in the log
+++ echo /dev/sda ++ bootdisk=/dev/sda ++ partnr=2 ++ partnr=2 ++ partnr=1 ++ [[ /dev/sda == \/\d\e\v\/\s\d\a ]] ++ chroot /mnt/local grub2-mkconfig -o /boot/grub2/grub.cfg Generating grub configuration file ... Found theme: /boot/grub2/themes/SLE/theme.txt Found linux image: /boot/vmlinuz-4.12.14-23-default Found initrd image: /boot/initrd-4.12.14-23-default done ++ grub2-install --root-directory=/mnt/local /dev/sda Installing for i386-pc platform. grub2-install: warning: cannot open directory `/usr/share/locale': No such file or directory. Installation finished. No error reported.
But when rebooting grub still goes into its grub rescue mode as
follows:
Booting from Hard Disk... GRUB loading.. Welcome to GRUB! error: file '/boot/grub2/i386-pc/normal.mod' not found. Entering rescue mode... grub rescue> ls (hd0,gpt2)/boot/grub2/i386-pc [no output] grub rescue>
i.e. the contents of the btrfs subvolume /@/boot/grub2/i386-pc
that is mounted at /boot/grub2/i386-pc on the original system
are not accessible during boot time by grub2.
In contrast when SLES15 with its default btrfs structure
was installed with the SLES15 YaST installer and
I go to the grub> prompt then it can access the contents
of the btrfs subvolume /@/boot/grub2/i386-pc:
grub rescue> ls (hd0,gpt2)/boot/grub2/i386-pc [tons of files, in particular the GRUB2 modules] grub rescue>
jsmeix commented at 2018-06-20 11:33:¶
I asked our GRUB2 maintainer what the right way is
how to set up GRUB2 and he explained (excerpts)
I would say using --root-directory is not a good idea, that's why it is hidden and the reason it did not get removed is for compatibility (I think it used to provide the option but obselelted). The bootloader installation has to be done in "bootstrape" (aka chroot) to ensure the result is on behalf of the target system. The --root-direcoty did not bootstrape a chroot environment to install bootloader. Here would be the difference between --root-directory and bootstrape that would lead to unwanted side effect. 1. The host's grub2-install vs the grub2-install in chroot may be different. 2. The host's /etc/default on host is used than the one in chroot 3. The host's /usr/share/grub2 is used .. 4. The host's /usr/lib/grub2 is used .. 5. The mount table /proc/self/mount would be different than bootstrape env. So that I would suggest to replace the process with mount --bind /dev /mnt/local/dev mount --bind /proc /mnt/local/proc mount --bind /sys /mnt/local/sys chroot /mnt/local grub2-mkconfig -o /boot/grub2/grub2.cfg chroot /mnt/local grub2-install /dev/sda
Accordingly I modified
usr/share/rear/finalize/Linux-i386/620_install_grub2.sh
as follows (with some findmnt output for my debugging):
# Find boot-disk:
bootdisk=$( find_disk_and_multipath "$part" )
findmnt -a
chroot $TARGET_FS_ROOT findmnt -a
for mount_device in dev proc sys ; do
umount $TARGET_FS_ROOT/$mount_device && sleep 1
mount --bind /$mount_device $TARGET_FS_ROOT/$mount_device || Error "Failed to bind mount /$mount_device at $TARGET_FS_ROOT/$mount_device"
done
chroot $TARGET_FS_ROOT findmnt -a
chroot $TARGET_FS_ROOT $grub_name-mkconfig -o /boot/$grub_name/grub.cfg && chroot $TARGET_FS_ROOT $grub_name-install "$bootdisk" && NOBOOTLOADER=''
Now the recreated system boots.
What I get in the log during "rear recover" for that part of the code is
++ bootdisk=/dev/sda ++ findmnt -a TARGET SOURCE FSTYPE OPTIONS / rootfs rootfs rw |-/sys sysfs sysfs rw,nosuid,nodev,noexec,relatime | |-/sys/kernel/security securityfs securityfs rw,nosuid,nodev,noexec,relatime | |-/sys/fs/cgroup tmpfs tmpfs ro,nosuid,nodev,noexec,mode=755 | | |-/sys/fs/cgroup/unified cgroup cgroup2 rw,nosuid,nodev,noexec,relatime | | |-/sys/fs/cgroup/systemd cgroup cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd | | |-/sys/fs/cgroup/cpu,cpuacct cgroup cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct | | |-/sys/fs/cgroup/cpuset cgroup cgroup rw,nosuid,nodev,noexec,relatime,cpuset | | |-/sys/fs/cgroup/rdma cgroup cgroup rw,nosuid,nodev,noexec,relatime,rdma | | |-/sys/fs/cgroup/net_cls,net_prio cgroup cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio | | |-/sys/fs/cgroup/perf_event cgroup cgroup rw,nosuid,nodev,noexec,relatime,perf_event | | |-/sys/fs/cgroup/memory cgroup cgroup rw,nosuid,nodev,noexec,relatime,memory | | |-/sys/fs/cgroup/freezer cgroup cgroup rw,nosuid,nodev,noexec,relatime,freezer | | |-/sys/fs/cgroup/pids cgroup cgroup rw,nosuid,nodev,noexec,relatime,pids | | |-/sys/fs/cgroup/devices cgroup cgroup rw,nosuid,nodev,noexec,relatime,devices | | |-/sys/fs/cgroup/blkio cgroup cgroup rw,nosuid,nodev,noexec,relatime,blkio | | `-/sys/fs/cgroup/hugetlb cgroup cgroup rw,nosuid,nodev,noexec,relatime,hugetlb | `-/sys/fs/pstore pstore pstore rw,nosuid,nodev,noexec,relatime |-/proc proc proc rw,nosuid,nodev,noexec,relatime |-/dev devtmpfs devtmpfs rw,nosuid,size=456976k,nr_inodes=114244,mode=755 | |-/dev/shm tmpfs tmpfs rw,nosuid,nodev | `-/dev/pts devpts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 | `-/dev/pts none devpts rw,relatime,gid=5,mode=620,ptmxmode=000 |-/run tmpfs tmpfs rw,nosuid,nodev,mode=755 `-/mnt/local /dev/sda2[/@/.snapshots/1/snapshot] btrfs rw,relatime,space_cache,subvolid=267,subvol=/@/.snapshots/1/snapshot |-/mnt/local/opt /dev/sda2[/@/opt] btrfs rw,relatime,space_cache,subvolid=263,subvol=/@/opt |-/mnt/local/tmp /dev/sda2[/@/tmp] btrfs rw,relatime,space_cache,subvolid=260,subvol=/@/tmp |-/mnt/local/srv /dev/sda2[/@/srv] btrfs rw,relatime,space_cache,subvolid=261,subvol=/@/srv |-/mnt/local/root /dev/sda2[/@/root] btrfs rw,relatime,space_cache,subvolid=262,subvol=/@/root |-/mnt/local/.snapshots /dev/sda2[/@/.snapshots] btrfs rw,relatime,space_cache,subvolid=266,subvol=/@/.snapshots |-/mnt/local/var /dev/sda2[/@/var] btrfs rw,relatime,space_cache,subvolid=258,subvol=/@/var |-/mnt/local/boot/grub2/i386-pc /dev/sda2[/@/boot/grub2/i386-pc] btrfs rw,relatime,space_cache,subvolid=265,subvol=/@/boot/grub2/i386-pc |-/mnt/local/boot/grub2/x86_64-efi /dev/sda2[/@/boot/grub2/x86_64-efi] btrfs rw,relatime,space_cache,subvolid=264,subvol=/@/boot/grub2/x86_64-efi |-/mnt/local/usr/local /dev/sda2[/@/usr/local] btrfs rw,relatime,space_cache,subvolid=259,subvol=/@/usr/local |-/mnt/local/home /dev/sda4 xfs rw,relatime,attr2,inode64,noquota `-/mnt/local/proc none proc rw,relatime ++ chroot /mnt/local findmnt -a TARGET SOURCE FSTYPE OPTIONS / /dev/sda2[/@/.snapshots/1/snapshot] btrfs rw,relatime,space_cache,subvolid=267,subvol=/@/.snapshots/1/snapshot |-/opt /dev/sda2[/@/opt] btrfs rw,relatime,space_cache,subvolid=263,subvol=/@/opt |-/tmp /dev/sda2[/@/tmp] btrfs rw,relatime,space_cache,subvolid=260,subvol=/@/tmp |-/srv /dev/sda2[/@/srv] btrfs rw,relatime,space_cache,subvolid=261,subvol=/@/srv |-/root /dev/sda2[/@/root] btrfs rw,relatime,space_cache,subvolid=262,subvol=/@/root |-/.snapshots /dev/sda2[/@/.snapshots] btrfs rw,relatime,space_cache,subvolid=266,subvol=/@/.snapshots |-/var /dev/sda2[/@/var] btrfs rw,relatime,space_cache,subvolid=258,subvol=/@/var |-/boot/grub2/i386-pc /dev/sda2[/@/boot/grub2/i386-pc] btrfs rw,relatime,space_cache,subvolid=265,subvol=/@/boot/grub2/i386-pc |-/boot/grub2/x86_64-efi /dev/sda2[/@/boot/grub2/x86_64-efi] btrfs rw,relatime,space_cache,subvolid=264,subvol=/@/boot/grub2/x86_64-efi |-/usr/local /dev/sda2[/@/usr/local] btrfs rw,relatime,space_cache,subvolid=259,subvol=/@/usr/local |-/home /dev/sda4 xfs rw,relatime,attr2,inode64,noquota `-/proc none proc rw,relatime ++ for mount_device in dev proc sys ++ umount /mnt/local/dev umount: /mnt/local/dev: not mounted. ++ mount --bind /dev /mnt/local/dev ++ for mount_device in dev proc sys ++ umount /mnt/local/proc ++ sleep 1 ++ mount --bind /proc /mnt/local/proc ++ for mount_device in dev proc sys ++ umount /mnt/local/sys umount: /mnt/local/sys: not mounted. ++ mount --bind /sys /mnt/local/sys ++ chroot /mnt/local findmnt -a TARGET SOURCE FSTYPE OPTIONS / /dev/sda2[/@/.snapshots/1/snapshot] btrfs rw,relatime,space_cache,subvolid=267,subvol=/@/.snapshots/1/snapshot |-/opt /dev/sda2[/@/opt] btrfs rw,relatime,space_cache,subvolid=263,subvol=/@/opt |-/tmp /dev/sda2[/@/tmp] btrfs rw,relatime,space_cache,subvolid=260,subvol=/@/tmp |-/srv /dev/sda2[/@/srv] btrfs rw,relatime,space_cache,subvolid=261,subvol=/@/srv |-/root /dev/sda2[/@/root] btrfs rw,relatime,space_cache,subvolid=262,subvol=/@/root |-/.snapshots /dev/sda2[/@/.snapshots] btrfs rw,relatime,space_cache,subvolid=266,subvol=/@/.snapshots |-/var /dev/sda2[/@/var] btrfs rw,relatime,space_cache,subvolid=258,subvol=/@/var |-/boot/grub2/i386-pc /dev/sda2[/@/boot/grub2/i386-pc] btrfs rw,relatime,space_cache,subvolid=265,subvol=/@/boot/grub2/i386-pc |-/boot/grub2/x86_64-efi /dev/sda2[/@/boot/grub2/x86_64-efi] btrfs rw,relatime,space_cache,subvolid=264,subvol=/@/boot/grub2/x86_64-efi |-/usr/local /dev/sda2[/@/usr/local] btrfs rw,relatime,space_cache,subvolid=259,subvol=/@/usr/local |-/home /dev/sda4 xfs rw,relatime,attr2,inode64,noquota |-/proc proc proc rw,nosuid,nodev,noexec,relatime |-/dev devtmpfs devtmpfs rw,nosuid,size=456976k,nr_inodes=114244,mode=755 `-/sys sysfs sysfs rw,nosuid,nodev,noexec,relatime ++ chroot /mnt/local grub2-mkconfig -o /boot/grub2/grub.cfg Generating grub configuration file ... Found theme: /boot/grub2/themes/SLE/theme.txt Found linux image: /boot/vmlinuz-4.12.14-23-default Found initrd image: /boot/initrd-4.12.14-23-default done ++ chroot /mnt/local grub2-install /dev/sda Installing for i386-pc platform. Installation finished. No error reported. ++ NOBOOTLOADER=
Nevertheless something is still not fully correct
because during boot there is still that grub error message
which I managed now to catch before it is too fast flashing away
by pausing the virtual machine in the right moment
so that I can read it now and it shows this:
Booting from Hard Disk... GRUB loading.. Welcome to GRUB! error: invalid environment block.
but it boots somehow regardless of that error.
jsmeix commented at 2018-06-21 09:07:¶
Regarding the GRUB2 error: invalid environment block.
In the rebooted recreated system I re-installed GRUB2 again
(for the log here also the recreated system's btrfs structure)
and rebooted
# findmnt -a -t btrfs -o TARGET,SOURCE,OPTIONS TARGET SOURCE OPTIONS / /dev/sda2[/@/.snapshots/1/snapshot] rw,relatime,space_cache,subvolid=267,subvol=/@/.snapshots/1/snapshot |-/srv /dev/sda2[/@/srv] rw,relatime,space_cache,subvolid=261,subvol=/@/srv |-/root /dev/sda2[/@/root] rw,relatime,space_cache,subvolid=262,subvol=/@/root |-/.snapshots /dev/sda2[/@/.snapshots] rw,relatime,space_cache,subvolid=266,subvol=/@/.snapshots |-/opt /dev/sda2[/@/opt] rw,relatime,space_cache,subvolid=263,subvol=/@/opt |-/var /dev/sda2[/@/var] rw,relatime,space_cache,subvolid=258,subvol=/@/var |-/usr/local /dev/sda2[/@/usr/local] rw,relatime,space_cache,subvolid=259,subvol=/@/usr/local |-/boot/grub2/i386-pc /dev/sda2[/@/boot/grub2/i386-pc] rw,relatime,space_cache,subvolid=265,subvol=/@/boot/grub2/i386-pc |-/boot/grub2/x86_64-efi /dev/sda2[/@/boot/grub2/x86_64-efi] rw,relatime,space_cache,subvolid=264,subvol=/@/boot/grub2/x86_64-efi `-/tmp /dev/sda2[/@/tmp] rw,relatime,space_cache,subvolid=260,subvol=/@/tmp # grub2-mkconfig -o /boot/grub2/grub2.cfg Generating grub configuration file ... Found theme: /boot/grub2/themes/SLE/theme.txt Found linux image: /boot/vmlinuz-4.12.14-23-default Found initrd image: /boot/initrd-4.12.14-23-default done # grub2-install /dev/sda Installing for i386-pc platform. Installation finished. No error reported. # reboot
I had expected that this helps to make the
GRUB2 error: invalid environment block. go away.
But it did not help. During booting I still get the GRUB2
error: invalid environment block.
Then on the original system I also re-installed GRUB2 again
(for the log here also the original system's btrfs structure)
and rebooted in the same way
# findmnt -a -t btrfs -o TARGET,SOURCE,OPTIONS TARGET SOURCE OPTIONS / /dev/sda2[/@/.snapshots/1/snapshot] rw,relatime,space_cache,subvolid=267,subvol=/@/.snapshots/1/snapshot |-/opt /dev/sda2[/@/opt] rw,relatime,space_cache,subvolid=263,subvol=/@/opt |-/root /dev/sda2[/@/root] rw,relatime,space_cache,subvolid=262,subvol=/@/root |-/var /dev/sda2[/@/var] rw,relatime,space_cache,subvolid=258,subvol=/@/var |-/srv /dev/sda2[/@/srv] rw,relatime,space_cache,subvolid=261,subvol=/@/srv |-/.snapshots /dev/sda2[/@/.snapshots] rw,relatime,space_cache,subvolid=266,subvol=/@/.snapshots |-/usr/local /dev/sda2[/@/usr/local] rw,relatime,space_cache,subvolid=259,subvol=/@/usr/local |-/boot/grub2/i386-pc /dev/sda2[/@/boot/grub2/i386-pc] rw,relatime,space_cache,subvolid=265,subvol=/@/boot/grub2/i386-pc |-/boot/grub2/x86_64-efi /dev/sda2[/@/boot/grub2/x86_64-efi] rw,relatime,space_cache,subvolid=264,subvol=/@/boot/grub2/x86_64-efi `-/tmp /dev/sda2[/@/tmp] rw,relatime,space_cache,subvolid=260,subvol=/@/tmp # grub2-mkconfig -o /boot/grub2/grub2.cfg Generating grub configuration file ... Found theme: /boot/grub2/themes/SLE/theme.txt Found linux image: /boot/vmlinuz-4.12.14-23-default Found initrd image: /boot/initrd-4.12.14-23-default done # grub2-install /dev/sda Installing for i386-pc platform. Installation finished. No error reported. # reboot
The original system still boots without any GRUB2 error message.
So I conclude that in the recreated system
something is wrong with the GRUB2 config files.
Investigating...
jsmeix commented at 2018-06-21 09:44:¶
The GRUB2 config file /boot/grub2/grub2.cfg
is exactly the same on the recreated system
as it is on the original system (same md5sum)
so that from my point of view that excludes that
something is wrong with the GRUB2 config files.
FYI:
Both sytems show same output for /dev/disk/by-uuid/
because "rear recover" recreates the filesystem UUIDs
so that it is o.k. to have same UUIDs in /boot/grub2/grub2.cfg
on the recreated system.
jsmeix commented at 2018-06-21 10:00:¶
Also /boot/grub2/grubenv is identical on the recreated system
and on the original system.
Simply moving away that file helps on the recreated system:
# mv /boot/grub2/grubenv /boot/grub2/grubenv.old # grub2-mkconfig -o /boot/grub2/grub2.cfg Generating grub configuration file ... Found theme: /boot/grub2/themes/SLE/theme.txt Found linux image: /boot/vmlinuz-4.12.14-23-default Found initrd image: /boot/initrd-4.12.14-23-default done # grub2-install /dev/sda Installing for i386-pc platform. Installation finished. No error reported. # reboot
Now also the recreated system boots
without any GRUB2 error message.
I got an automatically new created /boot/grub2/grubenv
on the recreated system.
FYI:
That is the content of /boot/grub2/grubenv on the original system
# GRUB Environment Block # WARNING: Do not edit this file other than by grub2-editenv env_block=512+1 saved_entry=SLES 15 ################################################################### ...
and that is the content of the new /boot/grub2/grubenv on the recreated system
# GRUB Environment Block # WARNING: Do not edit this file other than by grub2-editenv ################################################################## ...
jsmeix commented at 2018-06-21 10:13:¶
According to what
https://www.gnu.org/software/grub/manual/grub/html_node/Environment-block.html
reads (excerpts)
It is often useful to be able to remember a small amount of information from one boot to the next. ... GRUB provides an “environment block” which can be used to save a small amount of state. ... The environment block is a preallocated 1024-byte file, which normally lives in /boot/grub/grubenv (although you should not assume this). At boot time, the load_env command (see load_env) loads environment variables from it, and the save_env (see save_env) command saves environment variables to it. From a running system, the grub-editenv utility can be used to edit the environment block.
Because on a by "rear recover" recreated system
there is no such thing as a meaningful previous boot
(because "rear recover" recreates a system by reinstalling
it completely from scratch) so that there is no meaningful use-case
to remember any information from a possible previous boot.
Accordingly it is correct to delete all information from a previous
boot
during "rear recover" to get a clean state on the recreated system
so that "rear recover" should remove boot/grub2/grubenv
on the recreated system i.e. "rear recover" should do
rm /mnt/local/boot/grub2/grubenv
according to man grub2-editenv on SLES15:
There is no 'delete' command; if you want to delete the whole environment block, use 'rm /boot/grub2/grubenv'.
jsmeix commented at 2018-06-22 11:41:¶
To get boot/grub2/grubenv that was restored from the backup
moved away directly after the backup was restored
(i.e. before GRUB2 is reinstalled)
I use now in etc/rear/local.conf
BACKUP_RESTORE_MOVE_AWAY_FILES=( /boot/grub2/grubenv )
The boot/grub2/grubenv that was restored from the backup has this content:
# GRUB Environment Block # WARNING: Do not edit this file other than by grub2-editenv env_block=512+1 saved_entry=SLES 15 #####################################################...
jsmeix commented at 2018-06-26 09:35:¶
For the log:
Only BIOS systems are affected because there GRUB2 is installed via
usr/share/rear/finalize/Linux-i386/620_install_grub2.sh
while on UEFI systems the EFI Boot Manager is manipulated via
usr/share/rear/finalize/Linux-i386/630_run_efibootmgr.sh
jsmeix commented at 2018-06-26 14:51:¶
My current
https://github.com/rear/rear/pull/1843
is a first step to fix this issue properly.
jsmeix commented at 2018-06-27 12:16:¶
With the current
https://github.com/rear/rear/pull/1843
i.e. up to
https://github.com/rear/rear/pull/1843/commits/0ce177d1019064ec4f74110ed003579430b99d34
things do "just work" for me also on SLES15.
jsmeix commented at 2018-06-29 13:10:¶
With
https://github.com/rear/rear/pull/1843
merged
this issue should be fixed on x86 architecture.
Fixing it in the same way for the ppc64le architecture is planned for
next week,
cf.
https://github.com/rear/rear/pull/1843#issuecomment-401326156
jsmeix commented at 2018-07-02 14:50:¶
My attempt to fix it in the same way for PPC64/PPC64LE is
https://github.com/rear/rear/pull/1848
jsmeix commented at 2018-07-12 09:24:¶
I asked our GRUB2 maintainer what the right way is
how to set up GRUB2 on POWER architecture
when GRUB2 is used there (e.g. on PPC64 or PPC64LE)
and whether there are differences (excerpts from the mail):
> > > I would like to know what the "officially right" way is > > > how to install GRUB2 on a SLES15 system when one is > > > not "inside" a running SLES15 system but "outside" > > > e.g. in a rescue system. > > > > ... I would suggest ... > > > > mount --bind /dev /mnt/local/dev > > mount --bind /proc /mnt/local/proc > > mount --bind /sys /mnt/local/sys > > chroot /mnt/local grub2-mkconfig -o /boot/grub2/grub2.cfg > > chroot /mnt/local grub2-install /dev/sda > > is it exactly the same procedure also on POWER architecture > when GRUB2 is used e.g. on PPC64 and/or PPC64LE > or are there differences? Yes the procedure is the same, but the install device ... has to be PPC PReP partition. (The openfirmware is configured to read the elf image embedded in PReP partition, like how the MBR is used by pc-bios to embed boot code).
jsmeix commented at 2018-07-16 11:43:¶
With
https://github.com/rear/rear/pull/1848
merged
this issue should be fixed.
[Export of Github issue for rear/rear.]