#655 Issue `closed`: Recovery with teaming-interface not possible¶

Labels: enhancement, support / question, fixed / solved / done

v1p3r0u5 opened issue at 2015-09-17 09:12:¶

Hi together,

on a RHEL 7.1 system with TSM-Backup and a team-interface as default interface I tried to do a recovery with no success. The team-interfaces will not be online while recovery process starts. As I can see, the kernel modules for teaming are available and as I know you support bonding. Do you also support teaming or is there a special option I have to activate for teaming support? It's a regular activebackup configuration. The only unusual part we configured is the name of device team0 (name is net0 and not team0).

Information:¶

Rear-Version: 1.17.2
OS: RHEL 7.1 latest
SELinux: targeted/enforcing
Backup-System: TSM

Networking:¶

# cat /etc/sysconfig/network-scripts/ifcfg-team0
DEVICE=team0
TEAM_CONFIG="{\"runner\":{\"name\":\"activebackup\"}}"
DEVICETYPE=Team
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
NAME=net0 <----------------------- Special!!!!
UUID=8efe140d-80ca-4120-b41e-d4926b8dcd39
ONBOOT=yes
IPADDR=XY.XY.XYZ.30
PREFIX=21
GATEWAY=XY.XY.XYZ.1
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes

# cat /etc/sysconfig/network-scripts/ifcfg-team0-eth0
NAME=team0-eth0
UUID=8d00dd78-eeec-4579-9a7c-9af944023b4c
DEVICE=eth0
ONBOOT=yes
TEAM_MASTER=team0
DEVICETYPE=TeamPort

# cat /etc/sysconfig/network-scripts/ifcfg-team0-eth1
NAME=team0-eth1
UUID=ba42b067-a33f-44c8-b18e-5683f0003e54
DEVICE=eth1
ONBOOT=yes
TEAM_MASTER=team0
DEVICETYPE=TeamPort

Custom site.conf:¶

# cat /etc/rear/site.conf 
OUTPUT=ISO
BACKUP=TSM
GRUB_RESCUE=n
KERNEL_CMDLINE="net.ifnames=0 biosdevname=0"
COPY_AS_IS=( $SHARE_DIR $VAR_DIR /etc/tsm/inclexcl/system.inclexcl )
TMPDIR="/var/tmp/"
export TMPDIR
TSM_RESULT_FILE_PATH=/var/tmp/rear/output

Teaming Modules in initram¶

# find ./usr/lib/modules/3.10.0-229.11.1.el7.x86_64/kernel/drivers/net/team
./usr/lib/modules/3.10.0-229.11.1.el7.x86_64/kernel/drivers/net/team
./usr/lib/modules/3.10.0-229.11.1.el7.x86_64/kernel/drivers/net/team/team_mode_random.ko
./usr/lib/modules/3.10.0-229.11.1.el7.x86_64/kernel/drivers/net/team/team_mode_loadbalance.ko
./usr/lib/modules/3.10.0-229.11.1.el7.x86_64/kernel/drivers/net/team/team_mode_roundrobin.ko
./usr/lib/modules/3.10.0-229.11.1.el7.x86_64/kernel/drivers/net/team/team.ko
./usr/lib/modules/3.10.0-229.11.1.el7.x86_64/kernel/drivers/net/team/team_mode_activebackup.ko
./usr/lib/modules/3.10.0-229.11.1.el7.x86_64/kernel/drivers/net/team/team_mode_broadcast.ko

If you need further information, please let me know.

Thanks for your support 👍

Ralf

schlomo commented at 2015-09-21 16:59:¶

@rg-fine can you please take a look at #530 if this helps you? It removes the dependency on the naming scheme of network devices. Maybe you can extend this code to also cover your use case?

gdha commented at 2015-09-21 18:41:¶

@rg-fine Be aware the pull request is not yet in rear - testing is more then desired...

v1p3r0u5 commented at 2015-09-23 13:55:¶

@schlomo and @gdha: Many thanks for the hint with this pull-request. Unfortunatelly this will not solve the teaming issue as there is no setup function for teaming (like bond_setup()). Teaming interfaces are also declared as virtual devices and because of this you need a function which handle this fact.

# ls /sys/devices/virtual/net
lo  team0  team1

Maybe I can spend some time in writing a function which is compatible to this pull request. But this will need some time, because I'm on customer side and other topics are more important at the moment. So if you have a quicker solution this would be very appreciated.

BTW: I tested this pull request also with "normal" interfaces, and for this case it works perfect...

Cheers, Ralf

v1p3r0u5 commented at 2015-09-25 08:21:¶

@schlomo and @gdha: I've written a workaround which worked for me. Because teaming runs in user space it is not possible to take this interfaces online without the daemons and tools. So my solution adds the teaming ip-addresses to the first attached teaming-member to simplify it.

I used /usr/share/rear/rescue/GNU/Linux/31_network_devices.sh from https://github.com/rear/rear/pull/530 1:1 without any changes.

I changed /usr/share/rear/rescue/GNU/Linux/35_routing.sh. So it will not write down routing for teaming devices.

Why?: I handle everything for teaming in a different script and I will prevent adding the default route to teaming devices because in my solution we don't need the teaming-interface itself but its first member.

The changes in 35_routing.sh:
Lines 86 and 96: from else to the following:

# be sure that it is not a teaming-interface
elif ! ethtool -i $device | grep -w "driver:" | grep -qw team ; then

Then I created /usr/share/rear/rescue/GNU/Linux/36_teaming.sh which handels everything for the teaming devices:

# 36_teaming.sh
#
# record teaming information (network and routing) for Relax-and-Recover
#
#    Relax-and-Recover is free software; you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation; either version 2 of the License, or
#    (at your option) any later version.

#    Relax-and-Recover is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU General Public License for more details.

#    You should have received a copy of the GNU General Public License
#    along with Relax-and-Recover; if not, write to the Free Software
#    Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA

# BUG: Supports Ethernet only (so far)

# where to build networking configuration
netscript=$ROOTFS_DIR/etc/scripts/system-setup.d/63-teaming.sh

### Skip netscript if noip is configured on the command line
cat <<EOT >> ${netscript}
if [[ -e /proc/cmdline ]] ; then
    if grep -q 'noip' /proc/cmdline ; then
        return
    fi
fi
EOT

# add a line at the top of netscript to skip if dhclient will be used
cat - <<EOT > ${netscript}
# if USE_DHCLIENT=y then use DHCP instead and skip 60-network-devices.sh
[[ ! -z "\$USE_DHCLIENT" && -z "\$USE_STATIC_NETWORKING" ]] && return
# if IPADDR=1.2.3.4 has been defined at boot time via ip=1.2.3.4 then configure
if [[ "\$IPADDR" ]] && [[ "\$NETMASK" ]] ; then
    device=\${NETDEV:-eth0}
    ip link set dev "\$device" up
    ip addr add "\$IPADDR"/"\$NETMASK" dev "\$device"
    if [[ "\$GATEWAY" ]] ; then
        ip route add default via "\$GATEWAY"
    fi
    return
fi
EOT

# store virtual devices, because teaming interfaces are declared as virtual
VIRTUAL_DEVICES=$(ls /sys/devices/virtual/net)

TEAMINGS=()

# check if virtual interface is a teaming interface
for VIRT_DEV in ${VIRTUAL_DEVICES}
do
    if ethtool -i ${VIRT_DEV} | grep -w "driver:" | grep -qw team
    then
        TEAMINGS+=($VIRT_DEV)
    fi

done

for TEAM in "${TEAMINGS[@]}"
do
    # catch all ip-addresses for the teaming interface
    ADDR=()
    for x in $(ip ad show dev ${TEAM} scope global | grep "inet.*\ " | tr -s " " | cut -d " " -f 3)
    do
        ADDR+=($x)
    done

    # create netscript only when the interface has at least one configured ip
    # to simplify we attach the configured ip-addresses to the first teaming-member
    if [[ ${ADDR[*]} ]]
    then
        # find out one member interface. Greping for "active port:" will not work for all possible teaming configs (e.g. roundrobin, loadbalance, ...)
        FIRST_PORT=$(teamdctl ${TEAM} state | grep -A1 -w ports: | tail -1  | sed 's/[[:blank:]]*//g')

        for TEAM_IP in ${ADDR[@]}
        do
            echo "ip addr add ${TEAM_IP} dev ${FIRST_PORT}" >>${netscript}
        done

        echo "ip link set dev ${FIRST_PORT} up" >>${netscript}

        PORT_MTU="$(cat /sys/class/net/${FIRST_PORT}/mtu)"
        echo "ip link set dev ${FIRST_PORT} mtu ${PORT_MTU}" >>${netscript}
    fi

    # catch the routing for the teaming interface as we disabled it in 35_routing.sh
    for table in $( { echo "254     main" ; cat /etc/iproute2/rt_tables ; } |\
            grep -E '^[0-9]+' |\
                tr -s " \t" " " |\
                cut -d " " -f 2 | sort -u | grep -Ev '(local|default|unspec)' ) ;
        do
            ip route list table $table |\
                grep -Ev 'scope (link|host)' |\
                while read destination via gateway dev device junk;
                do
            if [[ "${device}" == "${TEAM}" ]]
            then
                echo "ip route add ${destination} ${via} ${gateway} ${dev} ${FIRST_PORT} table ${table}" >>${netscript}
            fi
        done
    done
done

Maybe there is a better way to do it in the rear structure, but I hope this will help you to implement the teaming support into it.

BTW: I've testet this solution including the pull request https://github.com/rear/rear/pull/530 with the following environments:

RHEL 7.1 Latest, physical server, single interfaces, TSM ✅
RHEL 7.1 Latest, physical server, bonding interfaces, TSM ✅
RHEL 7.1 Latest, physical server, teaming interfaces, TSM ✅
RHEL 7.1 Latest, VM, single interfaces, TSM ✅

I could do a successfull recovery in all this scenarios.

I will try to spent some time also for RHEL 5 and RHEL 6 tests (with TSM) next week.

Hope this helps a little bit.

Cheers, Ralf 🍺

gdha commented at 2015-09-25 11:46:¶

@rg-fine Would it help if we accept the pull request of the BONDING/VLAN re-design so you could apply your addition into your trunk?

v1p3r0u5 commented at 2015-09-25 13:09:¶

@gdha: From my point of view it would help for sure, but I can only share my testings with RHEL 7 at the moment. So I don't know if the new Bonding and VLAN script will work correctly on other derivates...

Would you like me to do a pull request afterwards for the teaming enhancement or are you planning without teaming support at the moment? I think teaming will become more popular and important in the future and because of this it would be great to support this kind of network aggregation.

gdha commented at 2015-09-25 13:41:¶

@rg-fine yes - we have to move on and fix what needs to be fixed. And, yes afterwards you can create your pull request on top of it. Thanks Ralf for the hard work

v1p3r0u5 commented at 2015-09-25 14:56:¶

@gdha: Nothing to thank for. I have to thank YOU for all the appreciated work and for this great tool...

gdha commented at 2015-10-21 06:01:¶

@rg-fine Any progress with a pull request from your side?

v1p3r0u5 commented at 2015-10-21 06:12:¶

@gdha Pull request was done with https://github.com/rear/rear/issues/662. Do you need any further information or are there some problems with the request? You commited it about one month ago and I tested it successfull on our environment.

gdha commented at 2015-10-21 06:42:¶

@rg-fine Sorry - I should write better comments as I was confused by my own words... Thanks for the clarification.

[Export of Github issue for rear/rear.]

#655 Issue closed: Recovery with teaming-interface not possible¶