#2015 Issue `closed`: No DNS Lookups possible in rescue system when systemd-resolved is used¶

Labels: enhancement, support / question, fixed / solved / done

nniehoff opened issue at 2019-01-12 18:13:¶

Relax-and-Recover (ReaR) Issue Template¶

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

ReaR version ("/usr/sbin/rear -V"): 2.4 / 2018-06-21
OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"): Ubuntu 18.04
ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"): NA
Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR): KVM guest
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device): amd64
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot): NA
Storage (lokal disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe): local disk
Description of the issue (ideally so that others can reproduce it): Ubuntu 18.04 uses systemd-resolved for DNS resolution therefore the entry in /etc/resolv.conf is 127.0.0.53. When the rescue media is created this fails to resolve DNS and therefore also fails to resolve hostnames for remote backups. Current DNS servers can be retrieved by systemd-resolve --status, there are probably other methods to get DNS.
Workaround, if any: after booting into recovery media set DNS servers by hand
Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):

jsmeix commented at 2019-01-14 11:58:¶

I do not use systemd-resolved and I don't know
whether or not there is specific support needed for it in ReaR.

I simply use USE_DHCLIENT="yes" because I have a DHCP server
in my network so that this alone is sufficient in my use case to get
networking set up in the ReaR recovery system.

@nniehoff
if you do not have a DHCP server in your network
you may use NETWORKING_PREPARATION_COMMANDS
to specify the needed commands to set up networking
in the ReaR recovery system, see its description in
default.conf e.g. online at
https://raw.githubusercontent.com/rear/rear/master/usr/share/rear/conf/default.conf

Because you did not provide us your ReaR configuration files
(why did you decide your ReaR configuration is not needed?)
I don't know why you need networking in the ReaR recovery system.
Usually networking in the ReaR recovery system is only needed
to access the backup on a remote location to be able to restore it
so that usually you only need a minimal networking setup
that is sufficient to get the backup from another host.
E.g. when you specify the IP address as in

BACKUP_URL=nfs://your.NFS.server.IP/path/to/your/rear/backup

no DNS is needed to access the backup on the NFS server, cf.
https://raw.githubusercontent.com/rear/rear/master/usr/share/rear/conf/examples/SLE11-ext3-example.conf

If you must use systemd-resolved to set up networking
in the ReaR recovery system check first of all that all needed
files to run systemd-resolved within the ReaR recovery system
are included in the ReaR recovery system. You may use
KEEP_BUILD_DIR="yes" (see its description in default.conf)
to check more easily what is included in the ReaR recovery system.
Then boot the ReaR recovery system and try to manually launch
systemd-resolved from within the ReaR recovery system
if it is not already somehow magically running.

nniehoff commented at 2019-01-14 13:51:¶

@jsmeix
Sorry I didn't leave my config:

OUTPUT=RAWDISK
OUTPUT_URL=nfs://fs1.somedomain.com/volume1/rear
BACKUP=BORG

BORGBACKUP_HOST="freenas.somedomain.com"
BORGBACKUP_USERNAME="backup.svc"
BORGBACKUP_REPO="/mnt/data/backups/borg/$(hostname -s)"
BORGBACKUP_REMOTE_PATH="/usr/local/bin/borg"

BORGBACKUP_PRUNE_DAILY=7
BORGBACKUP_PRUNE_WEEKLY=4
BORGBACKUP_PRUNE_MONTHLY=6
BORGBACKUP_PRUNE_YEARLY=2

BORGBACKUP_COMPRESSION="lzma,9"     # Slowest backup, best compression

BORGBACKUP_ENC_TYPE="keyfile"
export BORG_PASSPHRASE="XXXXX"
COPY_AS_IS_BORG=( "/root/.config/borg/keys/" )

export BORG_RELOCATED_REPO_ACCESS_IS_OK="yes"

I will look into using the NETWORKING_PREPARATION_COMMANDS to see if I can come up with a workaround. You are correct a potential workaround would be to use an IP however, I typically avoid using IPs as DNS is usually a better practice (an argument could be made in a disaster recovery situation it may not be a good idea to rely on DNS either).

I reported this bug as the default in Ubuntu Bionic (18.04) and later is to use systemd-resolved.

gdha commented at 2019-01-14 17:06:¶

# cat /lib/systemd/resolv.conf 
# This file belongs to man:systemd-resolved(8). Do not edit.
#
# This is a static resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists no search
# domains.
#
# Run "systemd-resolve --status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53

If I understand it well - if the /etc/resolv.conf symlink is replaced with a real file then DNS lookup should work as well. Could you try this as it would make it easier for us then supported yet another systemd sub-service

jsmeix commented at 2019-01-15 08:54:¶

@nniehoff
I forgot to say that I am not a Ubuntu user (I am from SUSE)
and in openSUSE Leap we do not use systemd-resolved
and I have no personal experience with systemd-resolved
so that I cannot provide informed help in this area.

I guess an alternative (untested) workaround could be
to provide an /etc/hosts entry for your NFS server name
e.g. via something like

NETWORKING_PREPARATION_COMMANDS=( "echo 'NFS.server.IP.address fs1.somedomain.com' >>/etc/hosts" )

or via

PRE_RECOVERY_SCRIPT="echo 'NFS.server.IP.address fs1.somedomain.com' >>/etc/hosts"

(see default.conf for PRE_RECOVERY_SCRIPT).
This changes only the /etc/hosts file in the ReaR recovery system.
Your actual system gets its /etc/hosts file restored from your backup.

jsmeix commented at 2019-01-15 09:21:¶

@OliverO2
this is an issue about ReaR and systemd-resolved in Ubuntu.

As far as I know you use ReaR on Ubuntu
and according to your code in
https://github.com/rear/rear/blob/master/usr/share/rear/build/OPALPBA/Linux-i386/095_exclude_non_essential_files.sh#L10

COPY_AS_IS_EXCLUDE+=( /lib/systemd/systemd-{cryptsetup,logind,networkd*,resolved} )

it seems you may have even used ReaR on Ubuntu with systemd-resolved
(because you excluded systemd-resolved in case of OPALPBA).

My question is now if you could provide us information
if and how ReaR works on Ubuntu with systemd-resolved?

OliverO2 commented at 2019-01-15 10:51:¶

It is true, I am using Ubuntu, 16.04, 18.04 and 18.10. Unfortunately, I have not yet tested the 18.x versions with ReaR yet. But this is scheduled to change soon (in a matter of weeks).

What I can say is that Ubuntu 16.04.x DNS clients actually run systemd-resolved.service (with /etc/resolv.conf being a symlink to ../run/resolvconf/resolv.conf). However, that does not mean systemd-resolved is actually used for DNS lookups (see below). On 16.04, everything always worked out of the box on the original system, on the rescue system and on ReaR-recovered systems. At some point (not ReaR-related) I had issues with unwanted DNS client caching, which I was unable to resolve, because I found the DNS client documentation on Ubuntu really lacking. I did not invest enough time to find out which DNS client service is actually used on 16.04.

Now this issue is about 18.04, and from what I've found in this article everything changes with that version. The old configuration files are still around (huh!) but the relevant stuff seems to sit in /etc/netplan. Maybe that helps. If not, I might have more information in the weeks to come.

jsmeix commented at 2019-01-15 15:17:¶

@OliverO2
thank you for this valuable information!

It seems that thingy is so red hot exciting new
that Wikipedia does not yet know about it
but Wikipedia knows at least about
https://en.wikipedia.org/wiki/Rantanplan
;-)

For more info about it see
https://netplan.io/
and
https://github.com/CanonicalLtd/netplan

schlomo commented at 2019-01-15 20:50:¶

@OliverO2 thanks for pointing out this fundamental change. I guess somebody will have to provide or sponsor some coding to support Ubuntu 18.04 properly.

jsmeix commented at 2019-01-16 11:43:¶

Offhandedly (without any look at the DNS related code in ReaR):

I think we do not need to replicate in the ReaR recovery system
whatever complicated DNS setup there is on the original system.

I think in the ReaR recovery system a plain traditional
/etc/resolv.conf file with real content should be sufficient.

As far as I see (at least on my openSUSE Leap 15.0 system)
the /etc/resolv.conf in the ReaR recovery system is an exact
copy of the /etc/resolv.conf on the original system:

# md5sum /tmp/rear.NqjFe7FOfLJuJ6O/rootfs/etc/resolv.conf
105142156fd4d6b0d755c2c6aa6de6c0  /tmp/rear.NqjFe7FOfLJuJ6O/rootfs/etc/resolv.conf

# md5sum /etc/resolv.conf
105142156fd4d6b0d755c2c6aa6de6c0  /etc/resolv.conf

Because I like it so much when the user has the final power
where he can specify all what ReaR should do if needed
I suggest to provide a config variable where the user can
specify the nameserver(s) that should be used within the
ReaR recovery system like

USE_RESOLV_CONF=( "search my.domain" "nameserver 192.168.100.53" "nameserver 8.8.8.8" )

and then it results /etc/resolv.conf in the recovery system
with the specified content

search my.domain
nameserver 192.168.100.53
nameserver 8.8.8.8

jsmeix commented at 2019-01-16 13:35:¶

@nniehoff
you wrote that your workaround is:
after booting into recovery media set DNS servers by hand

How exactly did you do it?
Was it sufficient to manually enter a DNS server into
the /etc/resolv.conf file in the running recovery system
or did you do something more and/or something different?

nniehoff commented at 2019-01-16 13:48:¶

@jsmeix
Yes, adding a nameserver to /etc/resolv.conf in the recovery system worked.

jsmeix commented at 2019-01-16 13:59:¶

Interestingly that Ubuntu way of DNS setup
had hit us already several times in the past, cf.
https://github.com/rear/rear/issues/520
https://github.com/rear/rear/issues/1200
https://github.com/rear/rear/commit/5390105de046ef4cf861265c10955d803999e008

I guess I can further enhance build/GNU/Linux/630_verify_resolv_conf_file.sh
to make ReaR behave more clearly for the user on Ubuntu
(note that I carefully avoided to write about a "fix for Ubuntu" ;-)
...

gdha commented at 2019-01-16 14:00:¶

@jsmeix from text in https://github.com/rear/rear/issues/2015#issuecomment-454082087 I believe a resolv.conf file is available (albeit via a symlink). Would it not be enough to detect this and make sure we copy the real file (and not just the symlink). I believe we already have such code in place?

jsmeix commented at 2019-01-16 15:52:¶

@gdha
it is not sufficient to copy the content of the the real file because it seems
on Ubuntu 18.x versions /etc/resol.conf is linked to /lib/systemd/resolv.conf
where its actual content is only the following single line

nameserver 127.0.0.53

as far as I understand your
https://github.com/rear/rear/issues/2015#issuecomment-454082087

But a loopback IP address for the DNS nameserver cannot work within
the recovery system because there is no DNS server listening at 127.0.0.53
because systemd-resolved is not running within the recovery system.

Accordingly I did
https://github.com/rear/rear/pull/2018
which works fine for me on my openSUSE Leap 15.0 system
but testing on Ubuntu systems is needed.

jsmeix commented at 2019-01-16 15:57:¶

@nniehoff @OliverO2
I would very much appreciate it if you could test on your Ubuntu systems
how my enhanced usr/share/rear/build/GNU/Linux/630_verify_resolv_conf_file.sh
actually works on your particular Ubuntu systems.

To test it copy my new one e.g. from

https://raw.githubusercontent.com/rear/rear/f7120cea293ba7712be7560ad38cb6f25f79fb3e/usr/share/rear/build/GNU/Linux/630_verify_resolv_conf_file.sh

onto your usr/share/rear/build/GNU/Linux/630_verify_resolv_conf_file.sh

jsmeix commented at 2019-01-21 12:32:¶

No news is good news so that I merged https://github.com/rear/rear/pull/2018

By default /etc/resolv.conf in the recovery system is a copy
of /etc/resolv.conf (or its symlink target) on the original system.
It cannot work when /etc/resolv.conf contains only loopback IP addresses
(which happens when the stub resolver systemd-resolved is used)
or when there is no nameserver entry so that "rear mkrescue/mkbackup"
errors out in this case with an error message like

No nameserver or only loopback addresses in
/tmp/rear.8OotHqKXNrolR0K/rootfs/etc/resolv.conf,
specify a real nameserver via USE_RESOLV_CONF

For non-default cases the user must specify what he wants via the
new USE_RESOLV_CONF variable that is described in default.conf.

Adding support in ReaR when systemd-resolved is used needs
code contribution from ReaR users who use systemd-resolved.

OliverO2 commented at 2019-01-22 15:41:¶

No news is good news

Not necessarily. ;)

I'll test it anyway on Ubuntu 18.04 once I'll be using ReaR on that version. As I've said, in a matter of weeks. If any issues come up, I'll report, of course.

OliverO2 commented at 2019-03-08 12:07:¶

For initial test results, see https://github.com/rear/rear/pull/2018#issuecomment-470906450.

jsmeix commented at 2019-03-11 15:26:¶

With https://github.com/rear/rear/pull/2076 merged
build/GNU/Linux/630_verify_resolv_conf_file.sh does no longer error out
when etc/resolv.conf has no nameserver or only loopback addresses
and USE_DHCLIENT is true (and USE_STATIC_NETWORKING is not true)
because then etc/resolv.conf in the recovery system is generated anew
by /bin/dhclient-script so that its content before does not matter.

This way it should in particular no longer falsely error out on systems
that use systemd-resolved (like Ubuntu 18.04) and do their networking
setup via DHCP (probably pretty standard on usual desktop systems),
cf. https://github.com/rear/rear/pull/2018#issuecomment-470906450

[Export of Github issue for rear/rear.]

#2015 Issue closed: No DNS Lookups possible in rescue system when systemd-resolved is used¶