#2843 Issue `closed`: No recovery system startup messages (inappropriate kernel 'console' setting)¶

Labels: fixed / solved / done, minor bug

jsmeix opened issue at 2022-07-26 10:33:¶

ReaR version ("/usr/sbin/rear -V"):
2.7 but I noticed it already earlier since some time during 2.6
(it had "just worked" by default everywhere before for me)
OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):
SLES15 SP3 and SP4
ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://192.168.122.1/nfs
FIRMWARE_FILES=( 'no' )

Intentionally I have all kernel modules included by default.

Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR):
virtual QEMU KVM machines on an openSUSE Leap 15.3 host system
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
x86
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
BIOS
Description of the issue (ideally so that others can reproduce it):

When I boot the ReaR recovery system on my KVM VMs I get

...
Loading kernel.....................
Loading initrd.cgz..........................ready.
Probing EDD (edd=off to disable)... ok

and then no further messages are shown
so in particular no recovery system startup messages are shown
but after a short time (only a few seconds)
the video mode changes (which keeps the above messages shown)
(but there are no recovery system startup messages shown)
and after another short time (some more seconds)
the screen blanks and shows the recovery system login screen
where then all works normal.

I tried the kernel command line parameters

edd=off but likely this issue has nothing to do with BIOS Enhanced Disk Drive Services (EDD)
nomodeset makes no difference
removing vga=normal makes no difference
I tried some modes with vga=ask but found none that helps
replacing vga=normal with nomodeset makes no real difference (i.e. still no recovery system startup messages are shown) but then there is no video mode change

I tried all available VGA settings in "Virtual Machine Manager 3.2.0"
QXL (the default), Bochs, Ramfb, VGA, Virtio
but found none that helps

In contrast when I boot another ReaR recovery system made with

OUTPUT=USB
USB_DEVICE_PARTED_LABEL="gpt"
USB_BOOT_PART_SIZE=1024
USB_DEVICE_BOOT_LABEL="MY-BOOT"
USB_BOOTLOADER="grub"
OUTPUT_URL=usb:///dev/disk/by-label/MY-BOOT
USB_DEVICE_FILESYSTEM_PERCENTAGE=10
USB_DEVICE_FILESYSTEM_LABEL="MY-DATA"
BACKUP=NETFS
BACKUP_URL=usb:///dev/disk/by-label/MY-DATA
FIRMWARE_FILES=( 'no' )
MODULES=( 'loaded_modules' )

from my USB disk on my older BIOS laptop
all the usual messages are shown
including the recovery system startup messages.

jsmeix commented at 2022-07-26 13:03:¶

@rear/contributors
do you perhaps have some ideas what I might try out
to determine the root cause or even solve the issue?

jsmeix commented at 2022-07-27 06:59:¶

Got it!

It has nothing to do with video mode.

The reason is the kernel 'console' setting which is
by default console=ttyS0,9600 (at least for OUTPUT=ISO).

Removing console=ttyS0,9600 makes it work:
All recovery system startup messages are shown.
The kernel startup messages, the init (systemd) startup messages,
and the ReaR recovery system startup scripts messages are shown.

Also console=ttyS0,9600 console=tty0 works same for me.

In contrast with console=tty0 console=ttyS0,9600
only the kernel startup messages are shown in my case.

jsmeix commented at 2022-07-27 07:02:¶

This helped me to see the actual root cause:
https://serverfault.com/questions/708447/probing-edd-boot-message-stays-for-ten-minutes-on-centos-6-6

Therein this answer:

I had the same symptoms but the problem was
that the vmware console wasn't showing the boot output
(after the EDD line and a few minutes,
 the login prompt suddelny appeared)
I added

console=tty0 console=ttyS0,115200

..to the grub and next boot all was fine

jsmeix commented at 2022-07-27 07:10:¶

It seems the "Overhauled serial console support code"
cf. https://github.com/rear/rear/pull/2699
in ReaR 2.7 caused some regression in some cases.

At least there is (at least usually) an easy workaround:
Adjust the kernel 'console' parameter setting in the
ReaR recovery system bootloader menu.
And normally (i.e. when there are no failures)
the recovery system startup messages are not needed.
So it is only a "minor bug".

pcahyna commented at 2022-07-27 08:23:¶

it is also related to the change done by @lzaoral recently. If a working serial console is detected on your machine, it is used as /dev/console.

pcahyna commented at 2022-07-27 08:24:¶

https://github.com/rear/rear/blob/edbe2ee9de54ed4e98424094c71bf4a7ba6bd69a/usr/share/rear/lib/serial-functions.sh#L47-L54

hpannenb commented at 2022-07-27 09:41:¶

@jsmeix You were faster than I could write my comment :-)

I recognised this during my short VM testing with CentOS7, AlmaLinux 8 and 9. I thought it was a change by decision/intention I could not follow. It is different compared to at least ReaR 2.6.
IMHO this leads to cumbersome situations especially in migration mode where one cannot find the dialogues on the console but on the serial port only.
Unfortunately those dialogues only appear on the primary used device (e.g. the console) even when there is a serial port attached.

Edit: In my case "migration mode" means a V2V, P2V etc. migration not setting this mode manually.

jsmeix commented at 2022-07-27 10:24:¶

@hpannenb
could you describe more percisely what exactly you experience?

For me with ReaR 2.7 defaults I only do not see
the recovery system startup messages
but all works normally as soon as the
recovery system login screen shows up.

Then I can log in normally and enter

# export MIGRATION_MODE='true'

# rear -D recover

and all migration mode dialogs work normally for me.
I use export MIGRATION_MODE='true' here only
to simulate a migration so I could see
how it behaves in migration mode.

I do not use a serial console.

hpannenb commented at 2022-07-27 10:34:¶

The minimal site.confI use is

# ReaR - site.conf
export TMPDIR="/var/tmp"

USE_STATIC_NETWORKING=yes
USE_RESOLV_CONF=no

ISO_PREFIX="rear-testing-$HOSTNAME"

USER_INPUT_TIMEOUT=15

SSH_ROOT_PASSWORD='root'

I create a rescue image on VM1 and boot this image in VM2. During the startup I will be asked in 55-migrate-network-devices.sh about to provide the proper network mapping. With ReaR 2.7 this dialog is present on e.g. ttyS0 (serial) only; with ReaR 2.6 this dialog was shown on tty0 (console).

Edit: The login prompt thereafter is present on (virtual) console AND on (virtual) serial port.

jsmeix commented at 2022-07-27 11:03:¶

@hpannenb
thank you!
Now I understand what dialog(s) you mean.

jsmeix commented at 2022-07-27 11:08:¶

For me (as not a serial console user)
all works well with explicit

USE_SERIAL_CONSOLE='no'

in etc/rear/local.conf (during "rear mkrescue/mkbackup)
because then there is no longer any 'console' setting
for the recovery system kernel
(at least no longer for OUTPUT=ISO).

jsmeix commented at 2022-07-27 11:12:¶

@hpannenb
when you use a serial console you may experiment with the

USE_SERIAL_CONSOLE
SERIAL_CONSOLE_DEVICES
SERIAL_CONSOLE_DEVICES_KERNEL
SERIAL_CONSOLE_DEVICE_SYSLINUX
SERIAL_CONSOLE_DEVICE_GRUB

config variables what works best for your case.

hpannenb commented at 2022-07-27 11:13:¶

@jsmeix As already mentioned by You: Due to "power to the user" I determined many new settings for CONSOLE parameters in the default.conf of ReaR 2.7 compared to 2.6. In my test setup the described situation is the more obvious change.

Edit: The reason why I test with virtual serial console on my QEMU/KVM is a) it is configured by default and b) the HPE servers I use in the company are all setup with a virtual serial port (VSP) to be used as a means to login via iLO.