#2528 Issue `closed`: Unable to boot the system post recover. System is moving to emergency mode¶

Labels: support / question, no-issue-activity

cvijayvinoth opened issue at 2020-11-26 13:06:¶

Relax-and-Recover (ReaR) Issue Template¶

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

ReaR version ("/usr/sbin/rear -V"):
Relax-and-Recover 2.5 / Git
OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):

NAME="CentOS Linux"
VERSION="7 (AltArch)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (AltArch)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

OUTPUT=ISO
BACKUP=RSYNC
RSYNC_PREFIX="$HOSTNAME"
BACKUP_PROG="/var/www/html/imageBackup/rsync"
OUTPUT_URL=rsync://diskimage@192.168.1.123::rsync_backup
BACKUP_URL=rsync://diskimage@192.168.1.123::rsync_backup
ISO_DIR="/var/www/html/imageBackup/iso/$HOSTNAME"
MESSAGE_PREFIX="$$: "
BACKUP_RSYNC_OPTIONS+=(-z --progress --password-file=/etc/rear/rsync_pass)
BACKUP_PROG_EXCLUDE+=( "$(****/imageBackup/iso/*" "$(****/imageBackup/Log/*" "/etc/rear/rsync_pass" )
PROGRESS_MODE="plain"
AUTOEXCLUDE_PATH=( /tmp )
PROGRESS_WAIT_SECONDS="1"
export TMPDIR="/var/www/html/imageBackup/iso/"
PXE_RECOVER_MODE=automatic
ISO_FILES=("/var/www/html/imageBackup/rsync")
ISO_PREFIX="${HOSTNAME}"
ISO_DEFAULT="automatic"
USE_DHCLIENT="Yes"

Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR):
virtual machine
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
x86 compatible
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
BIOS and GRUB
Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
local disk
Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT" or "lsblk" as makeshift):

NAME        KNAME     PKNAME   TRAN   TYPE FSTYPE  SIZE MOUNTPOINT
/dev/sda    /dev/sda           sata   disk         500G
|-/dev/sda1 /dev/sda1 /dev/sda        part xfs       1G /boot
|-/dev/sda2 /dev/sda2 /dev/sda        part xfs     236G /
|-/dev/sda3 /dev/sda3 /dev/sda        part swap      9G [SWAP]
|-/dev/sda4 /dev/sda4 /dev/sda        part           1K
`-/dev/sda5 /dev/sda5 /dev/sda        part xfs     254G /home
/dev/sdb    /dev/sdb           sata   disk          25G
`-/dev/sdb1 /dev/sdb1 /dev/sdb        part xfs      25G /home/osboxes/apps/list
/dev/sdc    /dev/sdc           sata   disk          20G
`-/dev/sdc1 /dev/sdc1 /dev/sdc        part ext4     20G /home/osboxes/apps/listnew
/dev/sdd    /dev/sdd           sata   disk          15G
`-/dev/sdd1 /dev/sdd1 /dev/sdd        part ext4     15G /home/osboxes/apps/a list
/dev/sr0    /dev/sr0           ata    rom         1024M

Description of the issue (ideally so that others can reproduce it):
Post recover system is going to emergency mode. Unable to boot
Workaround, if any:
Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
sourceCentOs.txt

jsmeix commented at 2020-11-26 13:31:¶

@cvijayvinoth
your https://github.com/rear/rear/files/5603149/sourceCentOs.txt
contains only (excerpts of the finalize stage):

393: 2020-11-26 06:18:47.995326306 Including finalize/default/520_confirm_finalize.sh
393: 2020-11-26 06:18:47.998173003 Including finalize/Linux-i386/610_EFISTUB_run_efibootmgr.sh
393: 2020-11-26 06:18:48.001334395 Including finalize/Linux-i386/630_install_grub.sh
/sbin/grub2-probe
393: 2020-11-26 06:18:48.002691943 Skip installing GRUB Legacy boot loader because GRUB 2 is installed (grub-probe or grub2-probe exist).
393: 2020-11-26 06:18:48.006071564 Including finalize/Linux-i386/640_install_lilo.sh
393: 2020-11-26 06:18:48.008518191 Including finalize/Linux-i386/650_install_elilo.sh
393: 2020-11-26 06:18:48.010859334 Including finalize/Linux-i386/660_install_grub2.sh
/sbin/grub2-probe
393: 2020-11-26 06:18:48.012047898 Installing GRUB2 boot loader...
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.10.0-1160.6.1.el7.centos.plus.i686
Found initrd image: /boot/initramfs-3.10.0-1160.6.1.el7.centos.plus.i686.img
Found linux image: /boot/vmlinuz-3.10.0-1127.el7.centos.plus.i686
Found initrd image: /boot/initramfs-3.10.0-1127.el7.centos.plus.i686.img
Found linux image: /boot/vmlinuz-0-rescue-f9e557d41f5f4381ac6b8171d4e901c8
Found initrd image: /boot/initramfs-0-rescue-f9e557d41f5f4381ac6b8171d4e901c8.img
done
393: 2020-11-26 06:18:51.286096126 Determining where to install GRUB2 (no GRUB2_INSTALL_DEVICES specified)
393: 2020-11-26 06:18:51.309948254 Found possible boot disk /dev/sda - installing GRUB2 there
Installing for i386-pc platform.
Installation finished. No error reported.

so the GRUB2 bootloader was installed without errors.

As far as I see (I am not a booting expert) this matches that your system gets booted
i.e. GRUB2 does its job: it loads kernel and the initrd of your recreated system
and the kernel gets started and things continue to run in the initrd of your recreated system
where then during systemd startup things go wrong.

This indicates that someting could be wrong in the initrd of your recreated system.

What I am missing during the "rear recover" finalize stage in your case
is that the initrd of your recreated system gets recreated
so that the initrd of your recreated system matches your recreated system.

E.g. on my SUSE systems during "rear recover"
usr/share/rear/finalize/SUSE_LINUX/i386/550_rebuild_initramfs.sh
will be run according to

# rear -s recover | grep 'finalize.*rebuild_initramfs'
Source finalize/SUSE_LINUX/i386/550_rebuild_initramfs.sh

The following scripts to recreate the initrd of the recreated system exist

usr/share/rear/finalize/Fedora/i386/550_rebuild_initramfs.sh
usr/share/rear/finalize/Fedora/ppc64le/550_rebuild_initramfs.sh
usr/share/rear/finalize/Fedora/ppc64/550_rebuild_initramfs.sh
usr/share/rear/finalize/SUSE_LINUX/i386/550_rebuild_initramfs.sh
usr/share/rear/finalize/SUSE_LINUX/ppc64le/550_rebuild_initramfs.sh
usr/share/rear/finalize/SUSE_LINUX/ppc64/550_rebuild_initramfs.sh
usr/share/rear/finalize/Debian/i386/550_rebuild_initramfs.sh
usr/share/rear/finalize/Debian/ppc64le/550_rebuild_initramfs.sh

I guess none of those is called because none of those matches
your CentOS Linux 7 system.

To test this copy one of the existing scripts that you think
matches best your CentOS Linux 7 system to
usr/share/rear/finalize/default/

I.e. when you are logged in as root inside your booted ReaR recovery system
do something like (e.g. assume CentOS Linux 7 is sufficiently like Fedora)

# cp /usr/share/rear/finalize/Fedora/i386/550_rebuild_initramfs.sh /usr/share/rear/finalize/default

and check with

# rear -s recover | grep 'finalize.*rebuild_initramfs'

that now finalize/default/550_rebuild_initramfs.sh will be run (sourced)
and afterwards run "rear -D recover".

cvijayvinoth commented at 2020-11-27 09:44:¶

updatedCentOsScript.txt
Post update the code still facing the same issue.

jsmeix commented at 2020-11-27 10:19:¶

https://github.com/rear/rear/files/5607095/updatedCentOsScript.txt
shows

+ source /usr/share/rear/finalize/default/520_confirm_finalize.sh
+ source /usr/share/rear/finalize/Linux-i386/610_EFISTUB_run_efibootmgr.sh

so recreating the initrd of the recreated system is still missing.

The expeced ReaR debug log file entries would be something like

+ source /usr/share/rear/finalize/default/520_confirm_finalize.sh
+ source /usr/share/rear/finalize/default/550_rebuild_initramfs.sh
+ source /usr/share/rear/finalize/Linux-i386/610_EFISTUB_run_efibootmgr.sh

cvijayvinoth commented at 2020-11-30 08:45:¶

rear-snigdhacentos32.log
Here I have attached the updated log..

Now the debug Log has below mentioned file details.

source /usr/share/rear/finalize/default/520_confirm_finalize.sh
source /usr/share/rear/finalize/default/550_rebuild_initramfs.sh
source /usr/share/rear/finalize/Linux-i386/610_EFISTUB_run_efibootmgr.sh

Still no luck. Facing the same issue...

jsmeix commented at 2020-12-02 13:19:¶

I am afraid, I don't see something obviously suspicious or wrong in your
https://github.com/rear/rear/files/5614707/rear-snigdhacentos32.log

At least for now I am at my wits' end - I am no CentOS user.

Someone who uses CentOS and knows how to setup a bootloader for it
should have a look here.

gdha commented at 2020-12-02 17:10:¶

@jsmeix weird message in above log file:
/usr/share/rear/wrapup/default/980_good_bye.sh: line 6: Setting: command not found - line 6 as there are only 4 lines in script https://github.com/rear/rear/blob/master/usr/share/rear/wrapup/default/980_good_bye.sh ?

As far I can see dracut ran successfully and grub2 was successful as well otherwise the emergency mode would not be possible. For the rest I saw no errors - no idea why it halted?

github-actions commented at 2021-02-01 02:17:¶

Stale issue message

[Export of Github issue for rear/rear.]

#2528 Issue closed: Unable to boot the system post recover. System is moving to emergency mode¶