#2902 Issue closed: Rescue/recovery networking does not work with IPv6 only NICs.

Labels: enhancement, bug, fixed / solved / done

hpannenb opened issue at 2022-12-30 12:35:

Relax-and-Recover (ReaR) Issue Template

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

  • ReaR version ("/usr/sbin/rear -V"):
    2.7 (and 2.4 )

  • OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):

OS_VENDOR=RedHatEnterpriseServer
OS_VERSION=7.9
  • ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
    Slightly adjusted AVA workflow; but this is working as such. Just added for the sake of completeness
## Avamar agent installation for your system
#
export TMPDIR="/var/tmp"

AVAMAR_ROOTDIR=/opt/avamar
AVAMAR_VARDIR=/var/avamar
AVAMAR_ETCDIR=/etc/avamar

# rear staff
ISO_DIR=/var/ReaR
OUTPUT=ISO
BACKUP=AVA00

# Static IP (no DHCP!)
USE_DHCLIENT=
USE_STATIC_NETWORKING="y"
USE_RESOLV_CONF=n

# No additional modules
#MODULES=( 'loaded_modules' )

# Set ROOT Password (optional)
SSH_ROOT_PASSWORD='root'

# Clone all current users
CLONE_ALL_USERS_GROUPS=yes

###
# Avamar specific extensions
COPY_AS_IS_AVA00=( "$AVAMAR_ROOTDIR" "$AVAMAR_VARDIR" "$AVAMAR_ETCDIR" )
COPY_AS_IS_EXCLUDE_AVA00=( "$AVAMAR_VARDIR/*.log" "$AVAMAR_VARDIR/*.dat" "$AVAMAR_VARDIR/clientlogs/*" ) 
PROGS_AVA00=( avtar avregister avagent.bin avscc avupdate detect_firewall avtar.bin )
# End Avamar specific extensions
  • Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR):
    VM
  • System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
    x86
  • Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
    GRUB
  • Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
    Local disks
  • Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"):
NAME                                               KNAME     PKNAME    TRAN TYPE FSTYPE      LABEL          SIZE MOUNTPOINT
/dev/fd0                                           /dev/fd0                 disk                              4K 
/dev/sda                                           /dev/sda                 disk                            300G 
|-/dev/sda1                                        /dev/sda1 /dev/sda       part xfs                        953M /boot
`-/dev/sda2                                        /dev/sda2 /dev/sda       part LVM2_member                299G 
  |-/dev/mapper/rhel_ts4reporting01cbcsv003-root00 /dev/dm-0 /dev/sda2      lvm  xfs                       44.7G /
  |-/dev/mapper/rhel_ts4reporting01cbcsv003-swap   /dev/dm-1 /dev/sda2      lvm  swap                        16G [SWAP]
  |-/dev/mapper/rhel_ts4reporting01cbcsv003-tmp    /dev/dm-2 /dev/sda2      lvm  xfs                        7.5G /tmp
  `-/dev/mapper/rhel_ts4reporting01cbcsv003-data   /dev/dm-3 /dev/sda2      lvm  xfs                      227.2G /data
  • Description of the issue (ideally so that others can reproduce it):

Creating a rescue/recovery environment does not include IPv6 only NICs.
In our case there is an IPv6 only NIC to access the AVAMAR backup service.
Without this NIC no recovery can take place.
Due to the following line

for network_interface in $( ip r | awk '$2 == "dev" && $8 == "src" { print $3 }' | sort -u ) ; do

currently online at
https://github.com/rear/rear/blob/973e102cc14db0126e1a4cb029feb339bacae49f/usr/share/rear/rescue/GNU/Linux/310_network_devices.sh#L961
just IPv4 routing NICs are chosen
which restricts preparing the rescue/recovery networking.
The issue is reproducible with ReaR 2.4 and 2.7.

  • Workaround, if any:

To get all NICs included I changed the above line and simplified it to

for network_interface in $( ls /sys/class/net/ | sort -u ) ; do

Since the network interface setup will checked on if they are UP,
have an IP and are linked to a pyhsical device I selected to go
with the "raw" output of network devices first.

  • Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):

gdha commented at 2022-12-31 11:14:

@hpannenb ReaR has knowledge of AVA (see https://github.com/rear/rear/blob/966d10e9b862f7f005983b6dc81fc0905915edba/usr/share/rear/conf/default.conf#L2194), however, you seem to use AVA00 -> what is the difference? Do we need an update of the complete AVA code? If yes, why not prepare of PR for it?
Have a nice 2023.

hpannenb commented at 2023-01-01 10:19:

@gdha I know there is an AVA implementation. The Avamar usage is quiet new to me in the company I am working for; AVA00 seems to be a slightly modified workflow I currently have to use. I first need to drill into the new setup to decide on PRs in the AVA area. :-)

@jsmeix In this issue I would rather concentrate on the IPv6 only NIC problem/bug which I could work around for my environment. I also tried in a CentOS7 VM at home and could reproduce the situation; it is all about the line of code I already point to which excludes IPv6 only NICs: https://github.com/rear/rear/blob/973e102cc14db0126e1a4cb029feb339bacae49f/usr/share/rear/rescue/GNU/Linux/310_network_devices.sh#L961

Best wishes for 2023 to all of You.

jsmeix commented at 2023-01-02 13:56:

Happy New Year!

@hpannenb
I am wondering why you need sort -u because
I think ls /sys/class/net/ cannot show duplicates
because duplicate filenames should not happen here [*]
and ls output should be sorted alphabetically.

The code

# Use output of 'ip r' to keep interfaces with IP addresses and routing only
for network_interface in $( ip r | awk '$2 == "dev" && $8 == "src" { print $3 }' | sort -u ); do

was added by @rmetrich in
https://github.com/rear/rear/commit/15567ede425401b008e5b1680db36a2c62752b8f
via
https://github.com/rear/rear/pull/1574
so I would like to let him have a look at your proposed changes here.

In particular because I fail to see his reason behind
why he only keeps interfaces with routing.
I guess it is meant to exclude interfaces like lo
(i.e. to only keep "normal" interfaces for real outer networks)
but I am not a sufficient networking expert
to fully imagine the actual reason behind.

For example on my homeoffice laptop I have
(the laptop has WLAN hardware but intentionally I do not use it
because for homeoffice security reasons I use wired ethernet
and the virbr0 is for KVM/QEMU virtual machine networking):

# ip -4 r
default via 192.168.178.1 dev eth0 proto dhcp metric 100 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown 
192.168.178.0/24 dev eth0 proto kernel scope link src 192.168.178.45 metric 100

# ip -4 r | awk '$2 == "dev" && $8 == "src" { print $3 }' | sort -u
eth0
virbr0

# ls -l /sys/class/net/
total 0
lrwxrwxrwx 1 root root 0 Jan  2 08:59 eth0 -> ../../devices/pci0000:00/0000:00:1c.5/0000:03:00.0/net/eth0
lrwxrwxrwx 1 root root 0 Jan  2 08:59 lo -> ../../devices/virtual/net/lo
lrwxrwxrwx 1 root root 0 Jan  2 09:00 virbr0 -> ../../devices/virtual/net/virbr0
lrwxrwxrwx 1 root root 0 Jan  2 08:59 wlan0 -> ../../devices/pci0000:00/0000:00:1c.6/0000:04:00.0/net/wlan0

# ip -6 r
::1 dev lo proto kernel metric 256 pref medium
2001:a61:2a96:1801::/64 dev eth0 proto ra metric 100 pref medium
2001:a61:2a96:1800::/56 via fe80::7eff:4dff:fe5d:42b1 dev eth0 proto ra metric 100 pref medium
fd00::/64 via fe80::7eff:4dff:fe5d:42b1 dev eth0 proto ra metric 100 pref medium
fe80::/64 dev eth0 proto kernel metric 100 pref medium
default via fe80::7eff:4dff:fe5d:42b1 dev eth0 proto ra metric 100 pref medium

Perhaps it might be better to call both ip -4 r and ip -6 r?

In particular because I don't know how far ls /sys/class/net/
works backward compatible for all those old and older
Linux distributions that are still supported by ReaR, cf.
"ReaR 2.7 is supported on the following Linux based operating systems"
at
https://github.com/rear/rear/blob/rear-2.7/doc/rear-release-notes.txt#L3830

On the other hand when ls /sys/class/net/ works sufficiently
backward compatible I would prefer it because it is simpler and
more straightforward than ip ... | awk ... | sort ...
cf. "Code must be easy to read"
and "Code must be easy to understand" in
https://github.com/rear/rear/wiki/Coding-Style

[*]
Filenames that look like duplicates can happen, see
"Use whitespace characters in file names to fool others" in
https://en.opensuse.org/SDB:Plain_Text_versus_Locale

jsmeix commented at 2023-01-02 14:05:

@rmetrich
could you have a look here - as far as time permits?

@hpannenb
proposes to change in
usr/share/rear/rescue/GNU/Linux/310_network_devices.sh
the lines

# Use output of 'ip r' to keep interfaces with IP addresses and routing only
for network_interface in $( ip r | awk '$2 == "dev" && $8 == "src" { print $3 }' | sort -u ) ; do

to something like

# Use the "raw" output of 'ls /sys/class/net/'
# since the network interfaces are checked
# if they are UP, have an IP,
# and are linked to a pyhsical device:
for network_interface in $( ls /sys/class/net/ ) ; do

because ip r lists only IPv4 network interfaces
but @hpannenb has IPv6 only NICs which get ignored.

jsmeix commented at 2023-01-02 14:07:

@pcahyna
in particular because @hpannenb uses RHEL
I would like when you could have a look here.

hpannenb commented at 2023-01-03 09:48:

  1. I left sort -u in my workaround due not not knowing if there would be any further implications on using this approach.
  2. E.g. lo interface is anyway kept out since there is no real device connected to it; see https://github.com/rear/rear/blob/973e102cc14db0126e1a4cb029feb339bacae49f/usr/share/rear/rescue/GNU/Linux/310_network_devices.sh#L153
  3. To me the usage of ls /sys/class/net/ looks "compatible enough" since it is in use within ReaR for quiet some time (ref. to https://github.com/rear/rear/blob/07c0385e931ffad1a79d7ea841a6d3f978f0bea9/usr/share/rear/rescue/GNU/Linux/31_network_devices.sh).
  4. IMHO there are new user options available for NICs (i.e. EXCLUDE_NETWORK_INTERFACES=()) giving the user enough power to excluded unwanted NICs instead of implicitly filtering them elsewhere.

hpannenb commented at 2023-01-08 18:53:

Created the PR #2907 with the required changes.

hpannenb commented at 2023-02-15 08:32:

I could test my PR on several RHEL7 VMs in lab and production VMWare based environment. This showed no flaws or constraints having hybrid IPv4/6 and IPv6 only NICs. Also on my CentOS 7 based VM at home I did not find an issue.

It would be appreciated if others could test this PR in their environments as well.

@jsmeix @pcahyna : I do not see any further issues due to the PR's minimal invasive change to the current code.

jsmeix commented at 2023-03-27 11:58:

With https://github.com/rear/rear/pull/2907 merged
this issue should be solved.

@hpannenb
thank you for your issue report,
your analysis what the root cause is,
and for your fix that improves ReaR
to be usable also with IPv6 only NICs!

[Export of Github issue for rear/rear.]