#2968 Issue closed: HP ProLiant XL420 Gen10: Recovery stuck at "Loading intial ramdisk"¶
Labels: support / question, fixed / solved / done
drakkainenn opened issue at 2023-04-05 10:28:¶
-
ReaR version ("/usr/sbin/rear -V"):
Relax-and-Recover 2.7-git.0.0.unknown / 2023-03-24 -
OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):
OS_VENDOR=RedHatEnterpriseServer
OS_VERSION=8.4
- ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
cat /etc/rear/site.conf
OUTPUT=ISO
BACKUP=CDM
TIMESYNC=NTP
ONLY_INCLUDE_VG=("vg_system")
GRUB_RESCUE=n
COPY_AS_IS_EXCLUDE=( "${COPY_AS_IS_EXCLUDE[@]}" "/dev/shm" "/dev/.udev" "/var/log/*" "/tmp/*" )
cat /etc/rear/local.conf
-
Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR):
HP ProLiant XL420 Gen10 -
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
x86_64 -
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
UEFI, GRUB2 -
Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
Local disk -
Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"):
NAME KNAME PKNAME TRAN TYPE FSTYPE LABEL SIZE MOUNTPOINT
/dev/sda /dev/sda sas disk 10.9T
/dev/sdb /dev/sdb sas disk 10.9T
/dev/sdc /dev/sdc sas disk 10.9T
/dev/sdd /dev/sdd sas disk 10.9T
/dev/sde /dev/sde sas disk 10.9T
/dev/sdf /dev/sdf sas disk 10.9T
/dev/sdg /dev/sdg sas disk 10.9T
/dev/sdh /dev/sdh sas disk 10.9T
/dev/sdi /dev/sdi sas disk 10.9T
/dev/sdj /dev/sdj sas disk 10.9T
/dev/sdk /dev/sdk sas disk 10.9T
/dev/sdl /dev/sdl sas disk 10.9T
/dev/sdm /dev/sdm sas disk 10.9T
/dev/sdn /dev/sdn sas disk 10.9T
/dev/sdo /dev/sdo sas disk 10.9T
/dev/sdp /dev/sdp sas disk 10.9T
/dev/sdq /dev/sdq sas disk 10.9T
/dev/sdr /dev/sdr sas disk 10.9T
/dev/sds /dev/sds sas disk 10.9T
/dev/sdt /dev/sdt sas disk 10.9T
/dev/sdu /dev/sdu sas disk 10.9T
/dev/sdv /dev/sdv sas disk 10.9T
/dev/sdw /dev/sdw sas disk 10.9T
/dev/sdx /dev/sdx sas disk 10.9T
/dev/sdy /dev/sdy sas disk 10.9T
/dev/sdz /dev/sdz sas disk 10.9T
/dev/sdaa /dev/sdaa sas disk 447.1G
|-/dev/sdaa1 /dev/sdaa1 /dev/sdaa part vfat 512M /boot/efi
|-/dev/sdaa2 /dev/sdaa2 /dev/sdaa part xfs 1G /boot
`-/dev/sdaa3 /dev/sdaa3 /dev/sdaa part LVM2_member 445.6G
|-/dev/mapper/vg_system-lv_root /dev/dm-0 /dev/sdaa3 lvm xfs 20G /
|-/dev/mapper/vg_system-lv_swap /dev/dm-1 /dev/sdaa3 lvm swap 20G [SWAP]
|-/dev/mapper/vg_system-lv_tmp /dev/dm-2 /dev/sdaa3 lvm xfs 40G /tmp
|-/dev/mapper/vg_system-lv_var_log /dev/dm-3 /dev/sdaa3 lvm xfs 10G /var/log
|-/dev/mapper/vg_system-lv_var_crash /dev/dm-4 /dev/sdaa3 lvm xfs 10G /var/crash
|-/dev/mapper/vg_system-lv_var /dev/dm-5 /dev/sdaa3 lvm xfs 30G /var
|-/dev/mapper/vg_system-lv_opt /dev/dm-6 /dev/sdaa3 lvm xfs 40G /opt
`-/dev/mapper/vg_system-lv_home /dev/dm-7 /dev/sdaa3 lvm xfs 10G /home
- Description of the issue (ideally so that others can reproduce
it):
Recovery stuck at Loading initial ramdisk.
Using: FIRMWARE_FILES=( 'no' ) also not working.
-
Workaround, if any:
No -
Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
jsmeix commented at 2023-04-05 10:39:¶
@drakkainenn
only a guess:
Does it perhaps work for you with
USE_SERIAL_CONSOLE='no'
in etc/rear/local.conf or in etc/rear/site.conf
provided you do not use a serial console.
If you use a serial console specify the config variables
USE_SERIAL_CONSOLE
SERIAL_CONSOLE_DEVICES
SERIAL_CONSOLE_DEVICES_KERNEL
SERIAL_CONSOLE_DEVICE_SYSLINUX
SERIAL_CONSOLE_DEVICE_GRUB
according to what you need in your particular case.
All config variables are described in
usr/share/rear/conf/default.conf
e.g. online for ReaR 2.7 at
https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/conf/default.conf
drakkainenn commented at 2023-04-05 13:33:¶
Sorry for this question, but how to check if and which console am I
using ?
It this correct ?
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root tty1 - 13:24 4.00s 0.00s 0.00s -bash
So is my console tty1?
Moreover i enter "set debug=all" and I can see a long, long, long loading something.
jsmeix commented at 2023-04-06 07:04:¶
I am talking about only the serial console.
By default current ReaR automatically sets up serial console output
by adding certain 'console=...' options to the kernel command line
for booting the ReaR recovery system kernel.
But when you do not have a serial console device connected,
then all output may get falsely redirected to a serial port
where no serial console device is connected
so you may no longer get any output
on your normal (i.e. non-serial) console, see
https://github.com/rear/rear/issues/2843
Usually your normal console is the VGA (video) console,
e.g. for Linux kernel version 5.3 see
https://www.kernel.org/doc/html/v5.3/admin-guide/serial-console.html
In the ReaR recovery system boot menue select
the topmost enty of the form "Recover HOSTNAME" and
press the [Tab] key to see the kernel command line
which you can edit there as you need (e.g. remove or change
possibly wrong 'console=...' kernel command line options), cf.
https://github.com/rear/rear/issues/2843#issuecomment-1196337765
With "set debug=all" you enable all GRUB2 debug messages.
I did this only once some longer time ago and with that
GRUB2 never finished for me in practice because it took
way too long when GRUB2 outputs all debug messages.
I waited for almost an hour with the same kind of
endlessly appearing GRUB2 debug messages as in your
https://user-images.githubusercontent.com/111309896/230096025-fdd31158-852c-4644-9dd0-d993d2f88924.png
Perhaps it works better to enable only specific GRUB2 debug messages
via set debug=this,that where 'this' and 'that' are specific
GRUB2 tags for only certain debug output that should be enabled.
I am not at all a GRUB2 expert so I can
neither list what possible GRUB2 debug tags exist
nor what GRUB2 debug tags are useful in practice
(quick "googling" for it did not show me something useful).
drakkainenn commented at 2023-04-06 08:20:¶
Thank you for your support.
After 10-20 minuites of waiting I finally have something new:
And that is all. Any idea what is wrong here ?
jsmeix commented at 2023-04-06 09:35:¶
No idea.
In particular not because I
neither have your server hardware
nor do I use your Linux distribution.
drakkainenn commented at 2023-04-12 11:57:¶
Hi.
I wasted week for troubleshooting it, using version 2.5,2.6 and 2.7.
Finally - don't laugh - I was using mkbackup instead of mkrescue
command.
Now I see recovery options.
It failed now at
So I need to check it and I will back to you.
drakkainenn commented at 2023-04-13 12:12:¶
2.7 still not working as expected so Im working with 2.6 so far.
I don't know if I should create new issue or someone can check where but I have problem with FS restore:
Somehow /var/log can't be recreate. Any idea what is wrong ? /var/crash is almost the same but without any issue.
//Manually is working fine
jsmeix commented at 2023-04-17 10:07:¶
In ReaR 2.6
layout/prepare/GNU/Linux/110_include_lvm_code.sh
contains
lvm lvcreate $lvopts $vg <<<y
which only inputs a single 'y' into lvcreate
that causes
https://github.com/rear/rear/issues/2820
cf.
https://github.com/rear/rear/issues/2820#issuecomment-1153681719
and
https://github.com/rear/rear/issues/2820#issuecomment-1161942491
Therefore in ReaR 2.7
layout/prepare/GNU/Linux/110_include_lvm_code.sh
contains a fix that is backward compatible with old 'lvcreate'
cf.
https://github.com/rear/rear/issues/2820#issuecomment-1153750220
... ( yes 2>/dev/null || true ) | lvm lvcreate ...
to pipe as many 'y' as asked for into "lvm lvcreate"
cf.
https://github.com/rear/rear/pull/2827
The current ReaR upstream master code
it is no longer backward compatible with old 'lvcreate'
and contains now
... lvm lvcreate -y ...
cf. https://github.com/rear/rear/pull/2839
drakkainenn commented at 2023-04-17 12:26:¶
2.6 is working fine, I am able to restore system,
Unfortunately due to some issue or I don't know what 2.7 is not working
at all.
2.6:
2.7:
I don't know what has been changed 2.6 vs 2.7 but I as you can see on screenshot I cannot boot rear menu.
drakkainenn commented at 2023-04-19 10:59:¶
grub menu 2.6 vs 2.7
Even if I add console=tty0 to 2.7 iso image still not working.
Something changed in 2.7 and I cannot boot rear to recovery menu
jsmeix commented at 2023-04-20 09:38:¶
From what I see in the screenshots in your
https://github.com/rear/rear/issues/2968#issuecomment-1514535513
it looks very much like the 'console=...' kernel options issue
that we have since ReaR 2.7 - see
https://github.com/rear/rear/issues/2843
See my above initial comment
https://github.com/rear/rear/issues/2968#issuecomment-1497276627
and see the related issue
https://github.com/rear/rear/issues/2843
that I mentioned in my above
https://github.com/rear/rear/issues/2968#issuecomment-1498590464
in that issue in particular see
https://github.com/rear/rear/issues/2843#issuecomment-1196337765
I.e. if you inserted 'console=tty0' before 'console=ttyS...'
only the kernel startup messages may get shown
but not the ReaR recovery system messages.
So you need to append 'console=tty0' at the end
as the last kernel option to make the kernel command line
look same in ReaR 2.7 as it looks in ReaR 2.6.
When you do not have a serial console it is usually
best to not specify any 'console=...' kernel option,
i.e. when you do not have a serial console it usually
"just works" to rely on the kernel default behaviour.
Did you actually try out how ReaR 2.7 behaves with
USE_SERIAL_CONSOLE='no'
in etc/rear/local.conf or in etc/rear/site.conf ?
You should try it out on a separated test machine.
The worst thing that I can imagine what could go wrong
is that you will see nothing at all on your screen
when the ReaR recovery system boots and runs.
In this case you would have to shut down the system blindly,
probably you may have to power off the machine without a
clean shutdown which should not matter on a test machine.
drakkainenn commented at 2023-04-24 14:15:¶
Finally I found what what wrong :)
My colleague prepared rpm packages from source files and I had an issue with this rpm.
Today I just typed "make install" from source and everything is fine!
So something is not correct with make rpm then or our environment which we used for building rpm package.
Finally:
"make install" -> working. I am able to start recovery process
"make rpm" -> rpm is not working properly. All looks fine but
recovery process is not loading.
Next step:
Check why "make rpm" is creating damaged rpm
[Export of Github issue for rear/rear.]