#937 Issue `closed`: Problem recovering SLES 11 SP3¶

Labels: support / question, fixed / solved / done

jottschi opened issue at 2016-07-21 14:09:¶

ReaR 1.18 / Git 15.07.2016
SLES 11 SP3 64 Bit + OES 111 on top

content of /etc/rear:
local.conf:
OUTPUT=ISO
BACKUP=NETFS
BACKUP_URL=sshfs://sambasave@10.2.3.4/home/sambasave/Backup
MODULES_LOAD=( vmxnet vmxnet3 e1000e e1000 fuse hpsa )
-# append special kernel parameters on rescue media
KERNEL_CMDLINE="cgroup_disable=memory "
USE_DHCLIENT=Yes
.# line below was automatically added by 21_include_dhclient.sh
DHCLIENT_BIN=dhclient

os.conf
OS_VENDOR=SUSE_LINUX
OS_VERSION=11

Brief description of the issue
The created recover-Image (ISO) contains a wrong isolinux.cfg:
it appends console=ttyS0,9600 console=ttyS1,9600 to the append line
which prevents the kernel from booting.

Work-around, if any
If I manually remove these entries it works and I can recover the Server.

jsmeix commented at 2016-07-21 14:29:¶

SUSE issues are mine regardless how much I know
about the particular case.

@jottschi
please provide a rear log file of the matching
"rear -d -D mkbackup"
run because I hope that helps me to understand
what goes on so that that those stuff is added
to the kernel command line.

Perhaps you can even alrready extract the relevant pieces
of the rear log file that lead to those stuff is added
to the kernel command line.

jsmeix commented at 2016-07-21 14:37:¶

By grepping for 'console=' in the rear sources
the most suspicious file where that happens is
usr/share/rear/rescue/GNU/Linux/40_use_serial_console.sh
in particular that code therein

for devnode in $(ls /dev/ttyS[0-9]* /dev/hvsi[0-9]* | sort); do
    speed=$(stty -F $devnode 2>&8 | awk '/^speed / { print $2 }')
    if [ "$speed" ]; then
        cmdline="${cmdline}console=${devnode##/dev/},$speed "
    fi
done

@jottschi
in the "rear -d -D mkbackup" log file look for

source /root/rear/usr/share/rear/rescue/GNU/Linux/40_use_serial_console.sh

and post what that does in your case.

jottschi commented at 2016-07-22 08:37:¶

2016-07-22 09:33:01.552002517 Including rescue/GNU/Linux/40_use_serial_console.sh
2016-07-22 09:33:01.555055433 Entering debugscripts mode via 'set -x'.
+ source /usr/share/rear/rescue/GNU/Linux/40_use_serial_console.sh
++ [[ ! y =~ ^[yY1] ]]
++ cmdline=
++ for param in '$KERNEL_CMDLINE'
++ case "$param" in
++ cmdline='cgroup_disable=memory '
+++ sort
+++ ls /dev/ttyS0 /dev/ttyS1 /dev/ttyS2 /dev/ttyS3 /dev/ttyS4 /dev/ttyS5 /dev/ttyS6 /dev/ttyS7
++ for devnode in '$(ls /dev/ttyS[0-9]* /dev/hvsi[0-9]* | sort)'
+++ stty -F /dev/ttyS0
+++ awk '/^speed / { print $2 }'
++ speed=9600
++ '[' 9600 ']'
++ cmdline='cgroup_disable=memory console=ttyS0,9600 '
++ for devnode in '$(ls /dev/ttyS[0-9]* /dev/hvsi[0-9]* | sort)'
+++ stty -F /dev/ttyS1
+++ awk '/^speed / { print $2 }'
++ speed=9600
++ '[' 9600 ']'
++ cmdline='cgroup_disable=memory console=ttyS0,9600 console=ttyS1,9600 '
++ for devnode in '$(ls /dev/ttyS[0-9]* /dev/hvsi[0-9]* | sort)'
+++ stty -F /dev/ttyS2
+++ awk '/^speed / { print $2 }'
++ speed=
++ '[' '' ']'
++ for devnode in '$(ls /dev/ttyS[0-9]* /dev/hvsi[0-9]* | sort)'
+++ stty -F /dev/ttyS3
+++ awk '/^speed / { print $2 }'
++ speed=
++ '[' '' ']'
++ for devnode in '$(ls /dev/ttyS[0-9]* /dev/hvsi[0-9]* | sort)'
+++ stty -F /dev/ttyS4
+++ awk '/^speed / { print $2 }'
++ speed=
++ '[' '' ']'
++ for devnode in '$(ls /dev/ttyS[0-9]* /dev/hvsi[0-9]* | sort)'
+++ stty -F /dev/ttyS5
+++ awk '/^speed / { print $2 }'
++ speed=
++ '[' '' ']'
++ for devnode in '$(ls /dev/ttyS[0-9]* /dev/hvsi[0-9]* | sort)'
+++ stty -F /dev/ttyS6
+++ awk '/^speed / { print $2 }'
++ speed=
++ '[' '' ']'
++ for devnode in '$(ls /dev/ttyS[0-9]* /dev/hvsi[0-9]* | sort)'
+++ stty -F /dev/ttyS7
+++ awk '/^speed / { print $2 }'
++ speed=
++ '[' '' ']'
++ [[  cgroup_disable=memory console=ttyS0,9600 console=ttyS1,9600  != \c\g\r\o\u\p\_\d\i\s\a\b\l\e\=\m\e\m\o\r\y\ \  ]]
++ KERNEL_CMDLINE='cgroup_disable=memory console=ttyS0,9600 console=ttyS1,9600 console=tty0'
++ Log 'Modified kernel commandline to: '\''cgroup_disable=memory console=ttyS0,9600 console=ttyS1,9600 console=tty0'\'''
++ test 1 -gt 0
+++ Stamp
+++ date '+%Y-%m-%d %H:%M:%S.%N '
++ echo '2016-07-22 09:33:01.596428937 Modified kernel commandline to: '\''cgroup_disable=memory console=ttyS0,9600 console=ttyS1,9600 console=tty0'\'''
2016-07-22 09:33:01.596428937 Modified kernel commandline to: 'cgroup_disable=memory console=ttyS0,9600 console=ttyS1,9600 console=tty0'

rear-s20.zip

jsmeix commented at 2016-07-22 09:05:¶

@jottschi
you wrote that a wrong 'console=...' kernel command line parameter
setting prevents the kernel from booting.

I cannot reproduce it and I do not understand it
(but I am not at all a kernel expert).

When I boot a rear recovery system and then select on its
boot screen "Recover HOSTNAME" and then press [tab]
to get its kernel command line, I can set any nonsense
for the 'console=...' parameter or remove it completely
and still the kernel boots for me.

Of course with a wrong 'console=...' parameter value
I don't see the kernel boot messages but it boots.

Interestingly when I set e.g. wrongly 'console=ttyS1'
I don't see the kernel boot messages and it shows
the login screen very fast where I can log in as root
but then it seems the usual system startup scripts
were nor run because I have in particular no IP address
(I use USE_DHCLIENT="yes" in /etc/rear/local.conf).

This means with wrong 'console=...' parameter settings
the rear recovery system may not be fully usable.

But at least for me the kernel alwasy booted and
I could always log in as root and run commands like
/etc/scripts/system-setup which runs the system startup scripts
and afterwards I have an IP address so that I can log in even
from remote (I also have SSH_ROOT_PASSWORD="rear")
and run "rear recover" from remote which works well for me.

From my current point of view wrong 'console=...' parameter
settings result only an annoyance but do not hinder to do
a successful "rear recover" with a little more manual
effort (i.e. running /etc/scripts/system-setup manually).

To be able to improve the current way how rear automatically
sets the 'console=...' parameter value I need to get a better
understanding in this area.

Therefore I like to understand how wrong 'console=...' parameter
settings prevent the kernel from booting in your particular case.

Do you perhaps have special hardware?
E.g. not usual x86 (32 or 64 bit) but another architecture?

jsmeix commented at 2016-07-22 10:11:¶

Those automated functionality was implemeted
in git commit https://github.com/rear/rear/commit/0fbc34b5ed4cf5190c7cbab2bea532d287b8e4ed

What looks suspicious therein is that in all files the
automatic serial console support is conditionally done
depending on the USE_SERIAL_CONSOLE value
except in rescue/GNU/Linux/40_use_serial_console.sh
where 'console=...' parameter values are unconditionally
added to the kernel command line.

I think at least with USE_SERIAL_CONSOLE="no"
there should be no 'console=...' parameter values
added to the kernel command line - at least no
'console=ttyS...' parameter values - perhaps the
default 'console=tty0' could be added in any case?

jsmeix commented at 2016-07-22 10:16:¶

Hmpf!
I should read the whole code.

Also in rescue/GNU/Linux/40_use_serial_console.sh
the automatic serial console support is conditionally done
depending on the USE_SERIAL_CONSOLE value:

# Enable serial console, unless explicitly disabled
if [[ ! "$USE_SERIAL_CONSOLE" =~ ^[yY1] ]]; then
    return
fi

jsmeix commented at 2016-07-22 10:23:¶

Something runs wrong here because in
usr/share/rear/conf/default.conf there is

USE_SERIAL_CONSOLE=

but then rescue/GNU/Linux/40_use_serial_console.sh
should not do automatic serial console support:

# USE_SERIAL_CONSOLE=
# if [[ ! "$USE_SERIAL_CONSOLE" =~ ^[yY1] ]]; then echo return ; fi
return

Now we need to find out why USE_SERIAL_CONSOLE
gets automatically a "yes" value and in my rear log file I found:

+ source /usr/share/rear/prep/GNU/Linux/20_include_serial_console.sh
...
++ USE_SERIAL_CONSOLE=y

which makes prep/GNU/Linux/20_include_serial_console.sh
the next most suspicious file that causes this misbehaviour...

jsmeix commented at 2016-07-22 10:30:¶

The code in prep/GNU/Linux/20_include_serial_console.sh
matches what usr/share/rear/conf/default.conf reads:

# Serial Console support is enabled if serial devices are found on the system.
# IA64 platforms do require it, and sometimes people still use serial console
# e.g. when no VGA console is available. (say y, n or leave empty to autodetect)
USE_SERIAL_CONSOLE=

@jottschi
I think all you need to do to make it work for your particular
special case is explicitly setting in /etc/rear/local.conf

USE_SERIAL_CONSOLE="n"

to enforce to have no automatic serial console support.

Because the automatic serial console support works for me
and because I cannot remember other issue reports
regarding this, I consider this particular issue no longer
to be a bug but a special case that needs special
explicit settings in /etc/rear/local.conf.

jsmeix commented at 2016-07-22 10:42:¶

With

USE_SERIAL_CONSOLE="n"

and then "rear mkbackup" I get a recovery system
where the kernel command line does no longer
contain any 'console' parameter setting.

From my point of view I consider the issue to be 'fixed'
which means not fixed in the code but solved by
explanation about a special setup.

jsmeix commented at 2016-07-22 10:46:¶

By the way:

I am now thinking about to add support so that the user can
specify a specific 'console' kernel parameter setting
when he needs something very special on his special
hardware like

SERIAL_CONSOLE="console=xYz1,666666,warp2"

But that is something for the future when needed...

jsmeix commented at 2016-07-29 10:13:¶

Support for specific 'console' kernel parameter setting
via SERIAL_CONSOLE is not needed because
any kernel command line settings are already possible
directly via KERNEL_CMDLINE so that special 'console'
kernel parameter settings can be specified for examle like

# disable automatic serial console support:
USE_SERIAL_CONSOLE="n"
# specify kernel parameter and special 'console' settings:
KERNEL_CMDLINE="cgroup_disable=memory console=xYz1,warp2"

[Export of Github issue for rear/rear.]

#937 Issue closed: Problem recovering SLES 11 SP3¶