#953 Issue `closed`: In recovery system "reboot" no longer works with newest systemd (e.g. SLE12 and FC24)¶

Labels: enhancement, fixed / solved / done, minor bug

jsmeix opened issue at 2016-08-04 09:59:¶

Current rear GitHub master
with latest Git commit 01f35154b35de0d4829cfe2a5502d4f839c27d45
(i.e. with https://github.com/rear/rear/issues/944 fixed)
on SLES12-SP2-beta5.

In the recovery system after successful "rear recover"
"reboot" does not work:

RESCUE g62:~ # rear -d -D recover
...
Finished recovering your system. You can explore it under '/mnt/local'.

RESCUE g62:~ # reboot
Failed to start reboot.target: Transaction contains conflicting jobs 'stop' and 'start' for reboot.service. Probably contradicting requirement dependencies configured.
Failed to talk to init daemon.

Easy workaround:

RESCUE g62:~ # umount -a
umount: /sys/fs/cgroup/systemd: target is busy
        (In some cases useful info about processes that
         use the device is found by lsof(8) or fuser(1).)
umount: /sys/fs/cgroup: target is busy
        (In some cases useful info about processes that
         use the device is found by lsof(8) or fuser(1).)
umount: /run: target is busy
        (In some cases useful info about processes that
         use the device is found by lsof(8) or fuser(1).)
umount: /dev: target is busy
        (In some cases useful info about processes that
         use the device is found by lsof(8) or fuser(1).)
umount: /: not mounted

RESCUE g62:~ # reboot -f
Rebooting.

I guess something related to systemd needs to be enhanced
in the rear recovery system to match newest fancy systemd stuff.

Because there is an easy workaround there is no time pressure
to get this done for Rear version 1.19 and therefore I set
the milestone to "Rear future".

jsmeix commented at 2016-08-23 13:49:¶

Happens also on SLES12-SP2-RC1
=> probably happens on all SLE12-SP2

GreasyMonkee commented at 2016-09-23 07:16:¶

Hi,

Identical problem in Fedora FC24, was OK in FC23 & 22.

Cheers,
GM.

jsmeix commented at 2016-09-23 12:37:¶

@GreasyMonkee
thanks for your posting because it shows that this issue is not
because of SUSE-specific fancy systemd changes but
likely a generic isssue with newer systemd versions.

@gdha
I added you as assignee here mainly FYI.

gdha commented at 2016-09-23 13:53:¶

@GreasyMonkee @jsmeix Just did a test with a fresh Fedora24 without any recover issues. Reboot went fine without any systemd error as well. Sorry - cannot reproduce your issue.

CORRECTION: I do see it - it helps when you reboot the correct window
Typing systemctl --force reboot does work!

jsmeix commented at 2016-09-23 14:38:¶

@gdha
no need to be sorry when it works for you!

I guess that it sometimes works and sometimes fails
is because with systemd booting and shutdown does
no longer happen in a basically hardcoded way
(as it was with SysVinit).
In particular systemd has no longer basically hardcoded
dependencies.

The following is a nice workaround for me
in the rear recovery system:

RESCUE g136:~ # alias reboot='umount -a ; sleep 10 ; reboot -f'
RESCUE g136:~ # type -a reboot
reboot is aliased to `umount -a ; sleep 10 ; reboot -f'
reboot is /bin/reboot
RESCUE g136:~ # reboot
...
Rebooting.

FYI:

On my SLES12-SP2-RC2 system:

g136:~ # type -a reboot
reboot is /sbin/reboot
g136:~ # ls -l /sbin/reboot
lrwxrwxrwx 1 root root ... /sbin/reboot -> /usr/bin/systemctl
g136:~ # ls -l /usr/bin/systemctl
-rwxr-xr-x 1 root root 692208 ... /usr/bin/systemctl
# rpm -qf /usr/bin/systemctl
systemd-228-114.4.x86_64

In the rear recovery system that was generated
on that SLES12-SP2-RC2 system by "rear mkbackup":

RESCUE g136:~ # type -a reboot
reboot is /bin/reboot
RESCUE g136:~ # type -a systemctl
systemctl is /bin/systemctl
RESCUE g136:~ # ls -li /bin/systemctl
6638 -rwxr-xr-x 1 root root 692208 Aug 26 20:33 /bin/systemctl
RESCUE g136:~ # ls -li /bin/reboot   
6529 -rwxr-xr-x 1 root root 692208 Aug 26 20:33 /bin/reboot
RESCUE g136:~ # diff -s /bin/reboot /bin/systemctl
Files /bin/reboot and /bin/systemctl are identical

By the way:
We could save 692208 bytes in the rear recovery system
by linking those indentical files.

gdha commented at 2016-09-23 15:54:¶

@jsmeix perhaps we could replace the /bin/reboot with a script containing systemctl --force reboot when systemd is in place?

jsmeix commented at 2016-09-26 10:47:¶

@gdha before doing such a hardcoded behavioural change
of how "reboot" normally works, I would wait some more time
until hopefully more information could be provided
why "reboot" sometimes fails with

Failed to start reboot.target: Transaction contains conflicting
jobs 'stop' and 'start' for reboot.service. Probably contradicting
requirement dependencies configured.
Failed to talk to init daemon.

so that we hopefully could solve the actual root cause.
I think in the meantime using the workaround

# umount -a ; reboot -f

is self-evident and easy to do.

jsmeix commented at 2016-09-26 10:59:¶

Stupid me!
I noticed right now that both the SLES12-SP2
/usr/lib/systemd/system/systemd-reboot.service
and the rear
usr/share/rear/skel/default/usr/lib/systemd/system/reboot.service
contain

ExecStart=...systemctl --force reboot

i.e. the default behaviour is already an enforced reboot.

jsmeix commented at 2016-09-26 11:09:¶

Plain "reboot" works for me on SLE12-SP2 when I remove
in /usr/lib/systemd/system/reboot.service
the whole "[Unit]" section so that it only is

[Service]
Type=oneshot
ExecStart=/bin/systemctl --force reboot

But I think there is a reason for the "[Unit]" section

[Unit]
Description=Reboot
DefaultDependencies=no
Requires=shutdown.target umount.target final.target
After=shutdown.target umount.target final.target

but I am not at all a sufficient systemd expert
to know the reasons behind.

But e.g. "shutdown" also does not work for me on SLE12-SP2:

RESCUE g136:~ # shutdown
Failed to connect to bus: No such file or directory
Failed to connect to bus: No such file or directory
RESCUE g136:~ # shutdown -r now
Failed to start reboot.target: Transaction contains conflicting
jobs 'stop' and 'start' for reboot.target. Probably contradicting
requirement dependencies configured.
Failed to talk to init daemon.
RESCUE g136:~ # shutdown -P now
Failed to start poweroff.target: Transaction contains conflicting
jobs 'stop' and 'start' for shutdown.target. Probably contradicting
requirement dependencies configured.
Failed to talk to init daemon.

From my current point of view I like most

# umount -a ; reboot -f

jsmeix commented at 2016-09-26 12:06:¶

One of our (i.e. SUSE's) systemd maintainers told me
that the root cause is probably that dbus is not usable
in the rear recovery system and that with
"systemctl --force reboot/poweroff/halt" no dbus is used
so that it works this way.

jsmeix commented at 2016-09-26 12:20:¶

I do not want to fiddle about dbus so that
I start to like the idea to replace in the rear recovery system
/bin/reboot /bin/halt /bin/poweroff /bin/shutdown
by scripts that basically do

umount -a
systemctl --force reboot

umount -a
systemctl --force halt

umount -a
systemctl --force poweroff

For /bin/shutdown there is no "systemctl --force shutdown"
so that probably a more elaborated script is need
that calls by default /bin/poweroff but when
'-r --reboot' is specified it calls /bin/reboot and when
'-H --halt' is specified it calls /bin/halt.

jsmeix commented at 2016-09-26 14:11:¶

https://github.com/rear/rear/pull/1011is a first attempt
to make reboot halt poweroff and shutdown work simpler
and more fail-safe.

jsmeix commented at 2016-09-27 07:09:¶

@GreasyMonkee
could you test my new
63_simplify_systemd_reboot_halt_poweroff_shutdown.sh
script whether or not it also makes it work for you?

To add it to your currently installed rear, download it from
https://raw.githubusercontent.com/jsmeix/rear/simplify_systemd_reboot_halt_poweroff_shutdown_issue953/usr/share/rear/build/GNU/Linux/63_simplify_systemd_reboot_halt_poweroff_shutdown.sh

Then install it as
/usr/share/rear/build/GNU/Linux/63_simplify_systemd_reboot_halt_poweroff_shutdown.sh

Then run "rear mkbackup" or "rear mkrescue", boot the new
created rear recovery system and try out if in the rear recovery
system reboot halt poweroff shutdown also "just work" for you
(preferably after a "rear recover" when things are mounted).

GreasyMonkee commented at 2016-09-27 07:26:¶

Hi, I am not able to test it as I am overseas for the next month, will be
late October at earliest before I have access to my server.

Sorry that I cannot assist at this time, but shall test on my return.