#1999 Issue closed: Serial console does not show automatic recovery scripts¶
Labels: waiting for info, support / question,
no-issue-activity
Signum opened issue at 2018-12-06 15:55:¶
-
ReaR version ("/usr/sbin/rear -V"): Relax-and-Recover 2.4 / 2018-06-21
-
OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"): Debian 9
-
ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
OUTPUT=ISO GRUB_RESCUE=y BACKUP=BORG … -
Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR): Occurs on a Virtualbox VM and on an HP DL 380 G7.
-
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device): AMD64
-
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot): BIOS/GRUB
-
Storage (lokal disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe): Local, virtualized, /dev/sda.
-
Description of the issue (ideally so that others can reproduce it):
- Attach a serial console.
- Create a recovery boot medium as ISO.
- Boot from the ISO.
- Choose "Automatic recovery" from the GRUB menu.
- The VGA console shows the beginning of the automatic recovery. However the serial console stops the output right after the kernel messages.
Yes, this sounds a bit like https://github.com/rear/rear/issues/878 but either it's something different or a regression.
And this only happens when booting from the ISO image. It does not occur when using the GRUB menu entry "Relax-and-Recover" that can be added by GRUB_RESCUE=y.
/etc/default/grub has these options set:
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX="console=tty1 console=ttyS0,115200"
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"
Please let me know how I can further track that down.
-
Workaround, if any: None known. Only the VGA console can be used.
-
Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
Output on the serial console:
Output on the VGA console:
gozora commented at 2018-12-06 18:53:¶
Hello @Signum,
When you are in boot menu of your ISO (ReaR recovery system), assuming
your are booting with isolinux (syslinux) press F2 for edit boot
entry you wish to boot and try to manually add
console=tty1 console=ttyS0,115200 maybe it will help ...
V.
jsmeix commented at 2018-12-07 09:06:¶
@Signum
I do not use serial console myself but I guess you need to play around
with
the USE_SERIAL_CONSOLE and KERNEL_CMDLINE config variables
to get it work in your particular case.
Every now and then we get issue reports related to the serial console
in the recovery system (which usually "just works") but as far as I
noticed
things could get really weird if the serial console does not "just
work".
I think often the root cause is not actually inside the ReaR recovery
system
but the hardware or virtual machine setup that is related to serial
console.
For details what really happens you need to inspect the scripts
that deal with the USE_SERIAL_CONSOLE config variable:
usr/share/rear/rescue/GNU/Linux/400_use_serial_console.sh
usr/share/rear/prep/GNU/Linux/200_include_serial_console.sh
usr/share/rear/output/USB/Linux-i386/300_create_extlinux.sh
usr/share/rear/lib/bootloader-functions.sh
Run rear -s mkrescue to see which of those scripts are run
in your particular case.
Signum commented at 2018-12-07 13:20:¶
@gozora
When you are in boot menu of your ISO (ReaR recovery system), assuming your are booting with isolinux (syslinux) press F2 for edit boot entry you wish to boot and try to manually add
console=tty1 console=ttyS0,115200maybe it will help ...
It seems to be "Tab" rather than "F2". But, no, that doesn't work. The default grub commandline is:
┌──────────────────────────────────────────────────────────┐
│ Relax-and-Recover v2.4 │
├──────────────────────────────────────────────────────────┤
│ Recover reartest │
│ Automatic Recover reartest │
│ │
│ Other actions │
│ Help for Relax-and-Recover │
│ Boot First Local disk (hd0) │
│ Boot Second Local disk (hd1) │
│ Boot Next device │
│ Hardware Detection Tool │
│ ReBoot system │
│ │
│ │
└──────────────────────────────────────────────────────────┘
> kernel initrd=initrd.cgz root=/dev/ram0 vga=normal rw selinux=0 console=ttyS0,115200 console=tty0 auto_recover
gozora commented at 2018-12-07 13:24:¶
can you try to change to onsole=ttyS1,115200 console=tty1 virtual
console console on your Proliant might be configured on COM1 ...
V.
Signum commented at 2018-12-07 13:42:¶
@gozora
Thanks for the suggestions. But COM1/ttyS0 is the correct port. After
all I get the GRUB menu and all kernel boot messages. Just on automatic
recovery I don't see any output from the REAR scripts that is sent to
tty0 only.
jsmeix commented at 2018-12-07 14:15:¶
@Signum
do you mean that with non-automatic recovery (i.e. when you
select Recover reartest instead of Automatic Recover reartest)
then you see the output from the ReaR scripts on your serial console?
Signum commented at 2018-12-07 14:31:¶
@jsmeix
Yes, exactly. As long as I login as root through the manual recovery
all is fine. So I guess the grub/kernel start line makes login shells
work. But the automatic recovery doesn't require a login and seems to
start in a different way (that I have not understood yet).
Please take a look what appears on my serial console in case of manual recovery:
ISOLINUX 6.03 20171018 Copyright (C) 1994-2014 H. Peter Anvin et al
┌──────────────────────────────────────────────────────────┐
│ Relax-and-Recover v2.4 │
├──────────────────────────────────────────────────────────┤
│ Recover reartest │
│ Automatic Recover reartest │
│ │
│ Other actions │
│ Help for Relax-and-Recover │
│ Boot First Local disk (hd0) │
│ Boot Second Local disk (hd1) │
│ Boot Next device │
│ Hardware Detection Tool │
│ ReBoot system │
│ │
│ │
└──────────────────────────────────────────────────────────┘
Press [Tab] to edit, [F2] for help, [F1] for version info
Rescue image kernel 4.9.0-8-amd64 Fri, 07 Dec 2018 14:16:02 +0100
BACKUP=REQUESTRESTORE OUTPUT=ISO
Loading kernel... ok
Loading initrd.cgz...ok
[ 000000000] Linux version 4.9.0-8-amd64((eeiinn-krree@@iists.debian.org) (gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1) ) #1 SMP Debian 4.9.130-2 (2018-10-27)
[ 0.000000] Command line: BOOT_IMAGE=kernel initrd=initrd.cgz root=/dev/ram0 vga=normal rw selinux=0 console=ttyS0,115200ccnnsole=tty0
[ 0.000000] x8//pp::SSupporting XSAVE feature 0x001: 'x87 floating point registers'
[...]
[ 16.598586] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[ 16.728924] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 16.805222] random: systemd: uninitialized urandom read (16 bytes read)
[ 16.835884] random: systemd: uninitialized urandom read (16 bytes read)
[ 17.062305] systemd[1]: systemd 232 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
[ 17.265860] random: systemd: uninitialized urandom read (16 bytes read)
Relax-and-Recover 2.4 / 2018-06-21
Relax-and-Recover comes with ABSOLUTELY NO WARRANTY; for details see
the GNU General Public License at: http://www.gnu.org/licenses/gpl.html
Host reartest using Backup REQUESTRESTORE and Output ISO
Build date: Fri, 07 Dec 2018 14:15:35 +0100
Debian GNU/Linux 9 reartest ttyS0
reartest login: root
root
Welcome to Relax-and-Recover. Run "rear recover" to restore your system !
RESCUE reartest:~ #
In automatic recovery mode the output stops right before "Relax-and-Recover 2.4 / 2018-06-21".
jsmeix commented at 2018-12-07 14:49:¶
@Signum
I have absolutely no idea what the root cause could be here.
The recovery system startup script that implements
automatic (and unattended) recovery mode is
usr/share/rear/skel/default/etc/scripts/system-setup
e.g. the current one online at
https://raw.githubusercontent.com/rear/rear/master/usr/share/rear/skel/default/etc/scripts/system-setup
Note my comment therein before the sleep 1
i.e. I already had inexplicable issues with missing output.
Likely a useless attempt:
Add the kernel command line option debug as in
https://github.com/rear/rear/issues/1999#issuecomment-444985949
which let's the recovery system startup scripts run with 'set -x'
but I assume you won't get any output on your serial console
because you get none after the last kernel message.
The difference between non-automatic recovery (the default case) and
automatic recovery is that in case of automatic (and unattended)
recovery
the script usr/share/rear/skel/default/etc/scripts/system-setup
is not finished - i.e. automatic (and unattended) recovery are run
as "rear recover" calls within that script's environment and
I guess that within this environment there is no output on your
serial console - but I have no idea why.
Your
https://github.com/rear/rear/issues/1999#issuecomment-445249831
indicates that there is no output on your serial console while
usr/share/rear/skel/default/etc/scripts/system-setup runs
(it runs as /etc/scripts/system-setup in the recovery system)
because there are none of the messages from that script like
Configuring Relax-and-Recover rescue system
(see the echo commands in that script).
jsmeix commented at 2018-12-07 15:35:¶
Only a blind guess:
Perhaps something is missing to get serial console output
in the systemd unit files in the recovery system
that run /etc/scripts/system-setup like
usr/share/rear/skel/default/usr/lib/systemd/system/sysinit.service
usr/share/rear/skel/default/usr/lib/systemd/system/run-system-setup.service
or whatever script it actually is that executes
/etc/scripts/system-setup
in a particular recovery system?
Signum commented at 2018-12-09 10:53:¶
@jsmeix
I will look into that later today. But it appears that the systemd
service is just sending output to the console (tty1) by default. systemd
services can be told to use other devices, but not both tty1 and ttyS0
at the same time. Maybe if I understand how the kernel bot option
"console=ttyS0" leads to also getting a getty on ttyS0 then I can copy
that.
jsmeix commented at 2018-12-14 12:55:¶
@Signum
are no news good news?
If you somehow solved it (or found a usable workaround)
we would appreciate feedback how you did it in your case
because that helps us to deal with similar issues in the future.
If you found a solution please provide it to us as GitHub pull request
even if your solution is perhaps not yet the "last word in wisdom"
we would appreciate to learn about any intermediate step forward.
Signum commented at 2018-12-14 14:27:¶
@jsmeix Sorry for the delay. I had to shift my focus because we were
facing massive troubles at work on a different topic. I hope I can come
up with a proposal beginning of next week.
I was thinking about the difference of login shells (that are started on
ttyS0 correctly) and the systemd-induced automatic recovery job… Would
it be an option to always make the user login first and then run either
"rear recover" as before or "rear autorecover" for a fully automated
experience?
jsmeix commented at 2018-12-14 15:11:¶
@Signum
I don't know how to automatically login as 'root' during startup
of the recovery system and how to let the auto-logged-in 'root'
automatically type rear recover and afterwards reboot
(depending whether or not if 'rear recover' was successful)
to keep the same automated experience as we have now.
I fear any scripted call of rear recover and afterwards reboot
might behave different compared to a real interactive shell
but I am not at all an expert in this area - in particular not
when systemd is involved...
gdha commented at 2018-12-14 15:14:¶
@Signum perhaps have a look at https://github.com/gdha/rear-automated-testing ?
github-actions commented at 2020-06-28 01:33:¶
Stale issue message
[Export of Github issue for rear/rear.]