#3573 Issue closed: rsyslog.service fails to start¶
Labels: enhancement, fixed / solved / done
gdha opened issue at 2026-02-26 12:28:¶
ReaR version¶
v2.9
Describe the ReaR bug in detail¶
Seems that since quite a while the rsyslogd cannot start anymore on more recent Ubuntu (maybe also Debian) distros.
RESCUE ubuntu24:/var/lib/rear/layout # systemctl status rsyslog
× rsyslog.service - Relax-and-Recover run-syslog script
Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; static)
Active: failed (Result: exit-code) since Thu 2026-02-26 10:17:44 UTC; 1h 59min ago
Duration: 5ms
TriggeredBy: × syslog.socket
Process: 907 ExecStart=/etc/scripts/run-syslog (code=exited, status=1/FAILURE)
Main PID: 907 (code=exited, status=1/FAILURE)
CPU: 5ms
RESCUE ubuntu24:~ # bash -x /etc/scripts/run-syslog
+ type -p rsyslogd
+ exec rsyslogd -n -i /var/run/rsyslogd.pid -f /etc/rsyslog.conf
rsyslog internal message (3,-2066): could not load module 'lmnet', errors: trying to load module /usr/lib/x86_64-linux-gnu/rsyslog/lmnet.so: /usr/lib/x86_64-linux-gnu/rsyslog/lmnet.so: cannot open shared object file: No such file or directory [v8.2312.0 try https://www.rsyslog.com/e/2066 ]
Error during class init for object 'conf' - failing...
rsyslogd initialization failed - global classes could not be initialized.
Did you do a "make install"?
Suggested action: run rsyslogd with -d -n options to see what exactly fails.
rsyslogd: run failed with error -2066 (see rsyslog.h or try https://www.rsyslog.com/e/2066 to learn what that number means)
- Seems on Ubuntu 24 the path is /lib/x86_64-linux-gnu/rsyslog/ (we
should add this to the
LIBSarray I guess):
$ ls /lib/x86_64-linux-gnu/rsyslog/
fmhash.so imklog.so impstats.so imudp.so lmnetstrms.so lmtcpclt.so lmzstdw.so mmfields.so mmrm1stspace.so omjournal.so omuxsock.so pmcisconames.so
imfile.so imkmsg.so imptcp.so imuxsock.so lmnsd_ptcp.so lmtcpsrv.so mmanon.so mmjsonparse.so mmsequence.so ommail.so pmaixforwardedfrom.so pmlastmsg.so
imjournal.so immark.so imtcp.so lmnet.so lmregexp.so lmzlibw.so mmexternal.so mmpstrucdata.so mmutf8fix.so omprog.so pmciscoios.so pmsnare.so
Platform¶
Linux x64
OS version¶
Ubuntu 24
Backup¶
BORG
Storage layout¶
$ lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT
NAME KNAME PKNAME TRAN TYPE FSTYPE LABEL SIZE MOUNT
/dev/sr0 /dev/sr0 sata rom iso9660 REAR-ISO 770.4M
/dev/vda /dev/vda virtio disk 10G
|-/dev/vda1 /dev/vda1 /dev/vda virtio part 1M
|-/dev/vda2 /dev/vda2 /dev/vda virtio part ext4 1G /tmp
|-/dev/vda3 /dev/vda3 /dev/vda virtio part ext4 1G /boot
`-/dev/vda4 /dev/vda4 /dev/vda virtio part ext4 8G /
/dev/vdb /dev/vdb virtio disk 2G
|-/dev/vdb1 /dev/vdb1 /dev/vdb virtio part zfs_member datapool 2G
`-/dev/vdb9 /dev/vdb9 /dev/vdb virtio part 8M
/dev/vdc /dev/vdc virtio disk 2G
|-/dev/vdc1 /dev/vdc1 /dev/vdc virtio part zfs_member datapool 2G
`-/dev/vdc9 /dev/vdc9 /dev/vdc virtio part 8M
/dev/vdd /dev/vdd virtio disk 2G
|-/dev/vdd1 /dev/vdd1 /dev/vdd virtio part zfs_member datapool 2G
`-/dev/vdd9 /dev/vdd9 /dev/vdd virtio part 8M
/dev/vde /dev/vde virtio disk 2G
|-/dev/vde1 /dev/vde1 /dev/vde virtio part zfs_member datapool 2G
`-/dev/vde9 /dev/vde9 /dev/vde virtio part 8M
/dev/vdf /dev/vdf virtio disk 2G
|-/dev/vdf1 /dev/vdf1 /dev/vdf virtio part zfs_member datapool 2G
`-/dev/vdf9 /dev/vdf9 /dev/vdf virtio part 8M
/dev/vdg /dev/vdg virtio disk 2G
|-/dev/vdg1 /dev/vdg1 /dev/vdg virtio part zfs_member datapool 2G
`-/dev/vdg9 /dev/vdg9 /dev/vdg virtio part 8M
What steps will reproduce the bug?¶
Just boot into RESCUE mode
Workaround, if any¶
Add /lib/x86_64-linux-gnu/rsyslog/ to the LIBS array
Additional information¶
No response
pcahyna commented at 2026-02-26 12:40:¶
I haven't seen this on EL9 nor EL10. I wonder whether it is a "more recent distros" issue or something special to your setup?
gdha commented at 2026-02-26 14:25:¶
@pcahyna
I haven't seen this on EL9 nor EL10. I wonder whether it is a "more recent distros" issue or something special to your setup?
LIBS+=(/lib*/libnss_dns* /lib*/libnss_files* /lib/*/libnss_dns* /lib/*/libnss_files* /lib*/libgcc_s* /lib*/libresolv* /usr/lib*/rsyslog/*so
The library modules seem to be loaded, but the rsyslog service was dead. I did recheck just now and it was running this time:
RESCUE alma:~ # systemctl status rsyslog
● rsyslog.service - Relax-and-Recover run-syslog script
Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; static)
Active: active (running) since Thu 2026-02-26 15:23:18 CET; 1min 2s ago
TriggeredBy: ● syslog.socket
Main PID: 575 (rsyslogd)
Tasks: 4 (limit: 19661)
Memory: 1.6M
CPU: 18ms
CGroup: /system.slice/rsyslog.service
└─575 rsyslogd -n -i /var/run/rsyslogd.pid -f /etc/rsyslog.conf
Strange - it fooled me this morning
gdha commented at 2026-02-27 10:49:¶
I've added:
# On Ubuntu the rsyslog modules are located at /usr/lib/x86_64-linux-gnu/rsyslog/ (issue #3573)
LIBS+=( /usr/lib/x86_64-linux-gnu/rsyslog/*.so )
to configuration file /usr/share/rear/conf/Ubuntu.conf
and now we have:
RESCUE ubuntu24:~ # systemctl status rsyslog
● rsyslog.service - Relax-and-Recover run-syslog script
Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; static)
Active: active (running) since Fri 2026-02-27 10:41:17 UTC; 1s ago
TriggeredBy: ● syslog.socket
Main PID: 908 (rsyslogd)
Tasks: 3 (limit: 3783)
Memory: 912.0K (peak: 1.1M)
CPU: 8ms
CGroup: /system.slice/rsyslog.service
└─908 rsyslogd -n -i /var/run/rsyslogd.pid -f /etc/rsyslog.conf
@rear/contributors Question: is Ubuntu.conf the proper file, or should
we better use Debian.conf or even GNU/Linux.conf instead?
gdha commented at 2026-03-06 08:20:¶
As this issue is isolated to Debian-alike OSes we will use Debian.conf
to add the missing LIBS.
jsmeix commented at 2026-03-06 09:21:¶
Only FYI a general note regarding
whether or not a distribution specific file
or a general file (e.g. Linux.conf) should be used:
See the question
https://github.com/rear/rear/pull/3575#issuecomment-3973688452
and my reasoning
https://github.com/rear/rear/pull/3575#issuecomment-3982901817
In particular regarding LIBS:
I tested how rear mkrescue behaves with
LIBS+=( /usr/lib/x86_64-linux-gnu/rsyslog/*.so )
on my openSUSE Leap 15.6 system where no such file exists:
It gets silently ignored.
I assume this is because of shopt -s nullglob ... in sbin/rear
because in contrast with
LIBS+=( /usr/lib/QQQ.so )
I get during rear mkrescue
Failed to copy '/usr/lib/QQQ.so'
So it should be reasonably fail-safe to have
LIBS+=( /usr/lib/x86_64-linux-gnu/rsyslog/*.so )
in Linux.conf (this does not mean that I recommend
to have it there - it only means it should be OK there).
If distribution specific stuff is in a general file
a comment should explain the reason behind, cf.
https://github.com/rear/rear/wiki/Coding-Style#code-must-be-easy-to-understand-answer-the-why
e.g. see
https://github.com/rear/rear/pull/3575/changes
jsmeix commented at 2026-03-06 09:39:¶
Another side note regarding
"what is a bug":
I think it is no bug in ReaR when ReaR fails
on new systems where things are different, cf.
https://en.opensuse.org/SDB:Disaster_Recovery
(excerpt)
For any kind of innovation that belongs to the basic system
(e.g. kernel, storage, bootloader, init)
the new kind (e.g. ...) will be there first
and afterwards ReaR can adapt step by step to support it.
see also
https://github.com/rear/rear/issues/3566#issuecomment-3972039991
In
https://github.com/rear/rear/labels
I described a "bug" as
The code does not do what it is meant to do
From "what it is meant to do" follows
that existing code cannot be meant
"to work on new systems where things are different"
because noone can predict how things become different
so existing code that fails when things are different
cannot be a bug in the existing code.
I personally consider it as important to distinguish
between
a "bug" where the reason is our fault
so others can (to some reasonable extent)
expect from us that we fix our fault
and
an "enhancement" where the reason is not our fault
so others can not "simply expect" from us
that we "just enhance" things for their needs.
pcahyna commented at 2026-03-06 10:43:¶
@jsmeix I generally agree with what you wrote, but let's take a step
back and examine why the problem occurs in the first place. Leaving
aside the (admittedly valid) question whether rsyslog is still needed at
all (perhaps journald would be enough?), the problem is that rsyslog
libraries are missing. But why they are missing? In
usr/share/rear/conf/GNU/Linux.conf we have this:
LIBS+=(
(...)
/usr/lib*/rsyslog/*so
/lib*/rsyslog/*so
so obviously some thought has been given to include the necessary
libraries, but they are not at the place where we expect them (either
/usr/lib*/rsyslog or /lib*/rsyslog), since we need
LIBS+=( /usr/lib/x86_64-linux-gnu/rsyslog/*.so ) - the prefix is a bit
different. Adding it to ReaR as a fix is obviously not entirely correct,
because we support other architectures beyond x86_64, right? Let's
have a look at this entire section of Linux.conf:
# the lib* serves to cover both 32bit and 64bit libraries!
#
LIBS+=(
### needed for username lookups
/lib*/libnss_dns*
/lib*/libnss_files*
### support multiarch
/lib/*/libnss_dns*
/lib/*/libnss_files*
/lib*/libgcc_s*
/lib*/libresolv*
/usr/lib*/rsyslog/*so
/lib*/rsyslog/*so
# Only copy *.so files in /usr/lib*/syslog-ng/ and skip /usr/lib*/syslog-ng/logg
en/
# because the loggen program is not included in the recovery system
# see https://github.com/rear/rear/issues/2743
/usr/lib*/syslog-ng/*so
### needed for curl HTTPS
/lib*/libnsspem.so*
/usr/lib*/libnsspem.so*
/lib*/libfreebl*.so*
/usr/lib*/libfreebl*.so*
/lib*/libnss3.so*
/usr/lib*/libnss3.so*
/lib*/libnssutil3.so*
/usr/lib*/libnssutil3.so*
/lib*/libsoftokn3.so*
/usr/lib*/libsoftokn3.so*
/lib*/libsqlite3.so*
/usr/lib*/libsqlite3.so*
/lib*/libfreeblpriv3.so*
/usr/lib*/libfreeblpriv3.so*
/lib*/libssl.so*
/usr/lib*/libssl.so*
/lib*/libnssdbm3.so*
/usr/lib*/libnssdbm3.so*
)
So, there is a general pattern of expecting something under either
/lib* or /usr/lib* and Debian clearly breaks this assumption. Adding
a new path for rsyslog modules helps only with one particular case and
we will continue to see the problem again and again for different types
of libraries/modules.
I believe that in order to solve all this properly we need to know what
are the valid library prefixes on a given distro and prepend them to all
library entries consistently. I am not yet sure how to do this. Perhaps
the paths could be given in LIBS without the prefix entirely and the
code that handles LIBS could prepend the relevant prefix(es) itself?
That would also eliminate the already existing duplication. Note that
PROGS also does not need absolute paths.
Not sure though how to get the prefixes in the first place though, the
list is maintained in /etc/ld.so.conf, but the format is not well
documented. /sbin/ldconfig -v -X -N prints the information, but it is
more intended to be human-readable rather than machine-readable.
pcahyna commented at 2026-03-06 11:07:¶
Actually, the part
### support multiarch
/lib/*/(...)
was clearly written with the situation like we have on Debian in mind,
and thus lookups work there. But the same concept was not applied to
other entries of the LIBS array. What we need is to make it consistent
somehow.
jsmeix commented at 2026-03-06 11:30:¶
@pcahyna
I fully agree with your observation that with current LIBS
we will see this kind of problem again and again
and that we must improve and clean up our current mess of
"ad hoc adding this and that missing bits and pieces"
and develop a generic consistently working solution.
But this is a bigger long term task which should
not be done via this issue but as a separated one,
cf. https://github.com/rear/rear/issues/2743
therein see in particular
https://github.com/rear/rear/issues/2743#issuecomment-1018280042
and
https://github.com/rear/rear/issues/2743#issuecomment-1128675464
But from that issue I am not too optimistic if and
until when such a generic cleanup might have been doable.
My comment was only meant for this issue with our
current method of "ad hoc adding missing pieces".
pcahyna commented at 2026-03-06 12:44:¶
@jsmeix thank you for the reference, will have a look.
If the goal is to simply fix the issue at hand, I would say that the
proposed fix is still too dirty. I think that a simple and a reasonably
clean solution is to apply the pattern above consistently everywhere,
i.e. add copies of existing entries with the /lib/*/ and /usr/lib/*/
prefixes.
It may seem dangerous to expand all the directories /lib and
/usr/lib this way, but that's what the current /lib/*/... entries
have been doing for ages without issues.
I would certainly not use Debian.conf for this, let's keep it in one
place, just how it is done now.
jsmeix commented at 2026-03-06 12:49:¶
For the log regarding rsyslogd:
On my openSUSE Leap 15.6 system I get
# ldd /usr/sbin/rsyslogd
linux-vdso.so.1 (0x00007ffe71f58000)
libz.so.1 => /usr/lib64/libz.so.1 (0x00007f88b192e000)
libm.so.6 => /lib64/libm.so.6 (0x00007f88b1844000)
libestr.so.0 => /usr/lib64/libestr.so.0 (0x00007f88b1600000)
libfastjson.so.4 => /usr/lib64/libfastjson.so.4 (0x00007f88b1836000)
libsystemd.so.0 => /usr/lib64/libsystemd.so.0 (0x00007f88b14ff000)
libuuid.so.1 => /usr/lib64/libuuid.so.1 (0x00007f88b182e000)
libc.so.6 => /lib64/libc.so.6 (0x00007f88b1200000)
/lib64/ld-linux-x86-64.so.2 (0x00007f88b1a17000)
libcap.so.2 => /usr/lib64/libcap.so.2 (0x00007f88b1823000)
libgcrypt.so.20 => /usr/lib64/libgcrypt.so.20 (0x00007f88b10bc000)
liblz4.so.1 => /usr/lib64/liblz4.so.1 (0x00007f88b14db000)
liblzma.so.5 => /usr/lib64/liblzma.so.5 (0x00007f88b1494000)
libzstd.so.1 => /usr/lib64/libzstd.so.1 (0x00007f88b0ffd000)
libjitterentropy.so.3 => /usr/lib64/libjitterentropy.so.3 (0x00007f88b1819000)
libgpg-error.so.0 => /usr/lib64/libgpg-error.so.0 (0x00007f88b146e000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f88b1814000)
so /usr/sbin/rsyslogd is not "normally linked"
to any of its own rsyslog specific libraries which are
in openSUSE Leap 15.6 located in /usr/lib64/rsyslog/
The message in the initial description
https://github.com/rear/rear/issues/3573#issue-3995250428
(long line shown wrapped)
rsyslogd: run failed with error -2066 (see rsyslog.h or try https://www.rsyslog.com/e/2066 to learn what that number means)
points to
https://www.rsyslog.com/?s=error+2066
which reads (excerpts)
rsyslog error 2066
...
Module could not be loaded – problem in dlopen().
A problem occurred during dlopen()ing
a loadable module (e.g. an input or output plugin).
Most common causes for this are
that the module ... does not exist ...
So the rsyslog specific libraries (i.e. rsyslog loadable modules)
must be specified via LIBS.
So simply adding yet another path to rsyslog loadable modules
in Linux.conf should fix it for now.
pcahyna commented at 2026-03-06 12:52:¶
So the rsyslog specific libraries (i.e. rsyslog loadable modules) must be specified via LIBS.
So simply adding yet another path to rsyslog loadable modules in Linux.conf should fix it for now.
Sure. But please do not add there a path that has x86_64 in it.
pcahyna commented at 2026-03-06 13:46:¶
See https://github.com/rear/rear/blob/master/usr/share/rear/prep/GNU/Linux/240_include_multipath_tools.sh https://github.com/rear/rear/blob/9360982209b62c7c0696e5b7f81462b04c71dc8f/usr/share/rear/prep/GNU/Linux/240_include_multipath_tools.sh#L11
(and a duplicate in https://github.com/rear/rear/blob/master/usr/share/rear/layout/save/GNU/Linux/280_multipath_layout.sh ) for an example of an ugly-but-generic way to handle library paths :-)
jsmeix commented at 2026-03-06 13:53:¶
I did a quick
https://github.com/rear/rear/pull/3579
for now so you could have a look.
I wish you all a relaxed and recovering weekend!
jsmeix commented at 2026-03-17 13:58:¶
With
https://github.com/rear/rear/pull/3579
merged this issue should be fixed.
[Export of Github issue for rear/rear.]