How to Configure Network Interface Teaming in CentOS/RHEL 7 and 8
Network interface teaming was introduced from CentOS/RHEL 7 as a more extensible and scalable alternative to network bonding. This post describes how to configure network teaming on CentOS/RHEL 7/8. The examples provided are based on CentOS/RHEL 8.2 system (Oracle VirtualBox 6.1 guest virtual machine) with two network interfaces using NetworkManager. Specifically, the Network Manager Command Line Interface (nmcli) is primarily used with options specified in abbreviated/shortened form.
1. Original Pre-Team Configuration
The following denotes the original, pre-team network configuration:
# lspci | grep -i eth
00:03.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 02)
00:08.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 02)
# nmcli dev
DEVICE TYPE STATE CONNECTION
enp0s3 ethernet connected enp0s3
enp0s8 ethernet connected enp0s8
lo loopback unmanaged --
# nmcli con
NAME UUID TYPE DEVICE
enp0s3 XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX ethernet enp0s3
enp0s8 XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX ethernet enp0s8
# ls -l /etc/sysconfig/network-scripts/*
-rw-r--r-- 1 root root 334 Aug 21 13:29 /etc/sysconfig/network-scripts/ifcfg-enp0s3
-rw-r--r-- 1 root root 334 Aug 21 13:30 /etc/sysconfig/network-scripts/ifcfg-enp0s8
--/etc/sysconfig/network-scripts/ifcfg-enp0s3
1 MACADDR=[MAC_ADDR1]
2 MTU=1500
3 TYPE=Ethernet
4 PROXY_METHOD=none
5 BROWSER_ONLY=no
6 BOOTPROTO=dhcp
7 DEFROUTE=yes
8 IPV4_FAILURE_FATAL=no
9 IPV6INIT=yes
10 IPV6_AUTOCONF=yes
11 IPV6_DEFROUTE=yes
12 IPV6_FAILURE_FATAL=no
13 IPV6_ADDR_GEN_MODE=stable-privacy
14 NAME="enp0s3"
15 UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
16 DEVICE=enp0s3
17 ONBOOT=yes
--/etc/sysconfig/network-scripts/ifcfg-enp0s8
1 MACADDR=[MAC_ADDR2]
2 MTU=1500
3 TYPE=Ethernet
4 PROXY_METHOD=none
5 BROWSER_ONLY=no
6 BOOTPROTO=dhcp
7 DEFROUTE=yes
8 IPV4_FAILURE_FATAL=no
9 IPV6INIT=yes
10 IPV6_AUTOCONF=yes
11 IPV6_DEFROUTE=yes
12 IPV6_FAILURE_FATAL=no
13 IPV6_ADDR_GEN_MODE=stable-privacy
14 NAME="enp0s8"
15 UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
16 DEVICE=enp0s8
17 ONBOOT=yes
# ip addr
...
2: enp0s3: [BROADCAST,MULTICAST,UP,LOWER_UP] mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether [MAC_ADDR1] brd ff:ff:ff:ff:ff:ff
inet [IP1]/24 brd [IP6] scope global dynamic noprefixroute enp0s3
valid_lft 86059sec preferred_lft 86059sec
inet6 fe80::ca99:46d3:1765:f02b/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: enp0s8: [BROADCAST,MULTICAST,UP,LOWER_UP] mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether [MAC_ADDR2] brd ff:ff:ff:ff:ff:ff
inet [IP2]/24 brd [IP6] scope global dynamic noprefixroute enp0s8
valid_lft 86121sec preferred_lft 86121sec
inet6 fe80::36d0:6bd3:5152:83dc/64 scope link noprefixroute
valid_lft forever preferred_lft forever
2. Delete existing network connections
Delete the existing enp0s3 and enp0s8 connections as follows. These are recreated as team slaves in the following steps.
# nmcli con show
NAME UUID TYPE DEVICE
enp0s3 XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX ethernet enp0s3
enp0s8 XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX ethernet enp0s8
# nmcli con del XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
Connection 'enp0s3' (XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX) successfully deleted.
# nmcli con del XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
Connection 'enp0s8' (XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX) successfully deleted.
# nmcli dev
DEVICE TYPE STATE CONNECTION
enp0s3 ethernet disconnected --
enp0s8 ethernet disconnected --
lo loopback unmanaged --
# ls -l /etc/sysconfig/network-scripts/
total 0
#
3. Create master team connection
Create the master team connection e.g.:
# nmcli con add type team con-name team0 ifname team0 config '{"runner": {"name": "activebackup"}, "link_watch": {"name": "ethtool"}}'
Connection 'team0' (XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX) successfully added.
Above, the team running mode (runner) is activebackup and the team link monitor/watcher (link_watch) is ethtool.
Alternative runner and link_watch values include:
- runner: loadbalance, roundrobin, lacp, broadcast, random
- link_watch: arp_ping, nsna_ping
NetworkManager creates the following interface configuration file:
--/etc/sysconfig/network-scripts/ifcfg-team0
1 TEAM_CONFIG="{\"runner\": {\"name\": \"activebackup\"}, \"link_watch\": {\"name\": \"ethtool\"}}"
2 PROXY_METHOD=none
3 BROWSER_ONLY=no
4 BOOTPROTO=dhcp
5 DEFROUTE=yes
6 IPV4_FAILURE_FATAL=no
7 IPV6INIT=yes
8 IPV6_AUTOCONF=yes
9 IPV6_DEFROUTE=yes
10 IPV6_FAILURE_FATAL=no
11 IPV6_ADDR_GEN_MODE=stable-privacy
12 NAME=team0
13 UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
14 DEVICE=team0
15 ONBOOT=yes
16 DEVICETYPE=Team
# nmcli con
NAME UUID TYPE DEVICE
team0 XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX team team0
# nmcli dev
DEVICE TYPE STATE CONNECTION
team0 team connecting (getting IP configuration) team0
enp0s3 ethernet disconnected --
enp0s8 ethernet disconnected --
lo loopback unmanaged
4. Optionally assign a static IP address to the team
Optionally assign a static IP address, gateway, DNS, etc. to the team connection e.g.:
# nmcli con mod team0 ipv4.addresses <ip3>/24
# nmcli con mod team0 ipv4.gateway <ip4>
# nmcli con mod team0 ipv4.dns <ip5>
# nmcli con mod team0 ipv4.method manual
# nmcli con mod team0 connection.autoconnect yes</ip5></ip4></ip3>
NetworkManager modifies the following team interface configuration file:
--/etc/sysconfig/network-scripts/ifcfg-team0
1 TEAM_CONFIG="{\"runner\": {\"name\": \"activebackup\"}, \"link_watch\": {\"name\": \"ethtool\"}}"
2 PROXY_METHOD=none
3 BROWSER_ONLY=no
4 BOOTPROTO=none
5 DEFROUTE=yes
6 IPV4_FAILURE_FATAL=no
7 IPV6INIT=yes
8 IPV6_AUTOCONF=yes
9 IPV6_DEFROUTE=yes
10 IPV6_FAILURE_FATAL=no
11 IPV6_ADDR_GEN_MODE=stable-privacy
12 NAME=team0
13 UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
14 DEVICE=team0
15 ONBOOT=yes
16 DEVICETYPE=Team
17 IPADDR=<ip3>
18 PREFIX=24
19 GATEWAY=[IP4]
20 DNS1=[IP5]</ip3>
The team will utilise DHCP if no static IP address is assigned.
5. Configure and add slaves to the team
Configure and add slaves to team e.g.:
# nmcli con add type team-slave con-name team0-slave0 ifname enp0s3 master team0
Connection 'team0-slave0' (XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX) successfully added.
# nmcli con add type team-slave con-name team0-slave1 ifname enp0s8 master team0
Connection 'team0-slave1' (XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX) successfully added.
# nmcli conn
NAME UUID TYPE DEVICE
team0 XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX team team0
team0-slave0 XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX ethernet enp0s3
team0-slave1 XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX ethernet enp0s8
# nmcli dev
DEVICE TYPE STATE CONNECTION
team0 team connected team0
enp0s3 ethernet connected team0-slave0
enp0s8 ethernet connected team0-slave1
lo loopback unmanaged --
NetworkManager creates the following team slave interface configuration files:
--/etc/sysconfig/network-scripts/ifcfg-team0-slave0
1 NAME=team0-slave0
2 UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
3 DEVICE=enp0s3
4 ONBOOT=yes
5 TEAM_MASTER=team0
6 DEVICETYPE=TeamPort
--/etc/sysconfig/network-scripts/ifcfg-team0-slave1
1 NAME=team0-slave1
2 UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
3 DEVICE=enp0s8
4 ONBOOT=yes
5 TEAM_MASTER=team0
6 DEVICETYPE=TeamPort
When at least one slave is added to the team, the interface is brought up and becomes accessible e.g.:
# ip addr
...
2: enp0s3: [BROADCAST,MULTICAST,UP,LOWER_UP] mtu 1500 qdisc fq_codel master team0 state UP group default qlen 1000
link/ether [MAC1] brd ff:ff:ff:ff:ff:ff
3: enp0s8: [BROADCAST,MULTICAST,UP,LOWER_UP] mtu 1500 qdisc fq_codel master team0 state UP group default qlen 1000
link/ether [MAC2] brd ff:ff:ff:ff:ff:ff
6: team0: [BROADCAST,MULTICAST,UP,LOWER_UP] mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether [MAC1] brd ff:ff:ff:ff:ff:ff
inet [IP3]/24 brd [IP6] scope global dynamic noprefixroute team0
valid_lft 86045sec preferred_lft 86045sec
inet6 fe80::5b1f:554a:1928:8575/64 scope link noprefixroute
valid_lft forever preferred_lft forever
6. Restart the team
Restart the team for the static IP address, etc. to take effect e.g.:
# nmcli con down team0 && nmcli con up team0
7. Identify the current active/inactive slave interfaces
Identify the current active/inactive slave interface using teamdctl(8) e.g.:
# teamdctl team0 state
setup:
runner: activebackup
ports:
enp0s3
link watches:
link summary: up
instance[link_watch_0]:
name: ethtool
link: up
down count: 0
enp0s8
link watches:
link summary: up
instance[link_watch_0]:
name: ethtool
link: up
down count: 0
runner:
active port: enp0s3
8. Enable promiscuous mode for network interfaces
Some virtualisation technologies, such as Oracle VM VirtualBox, require promiscuous mode to be enabled on network interfaces assigned to guests as well as within guests in order for slave failover/failback to behave correctly. Similarly, enabling promiscuous mode on network interfaces for physical systems may also be required.
For Oracle VM VirtualBox, promiscuous mode can be enabled for guest interfaces as follows: - Oracle VM VirtualBox Manager > [GUEST] > Settings > Network > Adapter 1|… > Advanced > Promiscuous Mode: Allow All
Promiscuous mode can be enabled within CentOS/RHEL 7/8 systems dynamicaly and statically using a custom service as follows:
Dynamic, non-persistent:
# ip link set enp0s3 promisc on
# ip link set enp0s8 promisc on
Static, persistent: Create a custom systemd unit file with the following contents e.g.:
--/usr/lib/systemd/system/promiscuous.service
1 [Unit]
2 Description=Bring up network interfaces in promiscuous mode upon boot
3 After=network.target
4
5 [Service]
6 Type=oneshot
7 ExecStart=/usr/sbin/ip link set dev enp0s3 promisc on
8 ExecStart=/usr/sbin/ip link set dev enp0s8 promisc on
9 ExecStop=/usr/sbin/ip link set dev enp0s3 promisc off
10 ExecStop=/usr/sbin/ip link set dev enp0s8 promisc off
11 TimeoutStartSec=0
12 RemainAfterExit=yes
13
14 [Install]
15 WantedBy=default.target
Inform systemd of the new service e.g.
# systemctl daemon-reload
Enable and start the new service e.g.:
# systemctl enable promiscuous
Created symlink /etc/systemd/system/default.target.wants/promiscuous.service → /usr/lib/systemd/system/promiscuous.service.
# systemctl start promiscuous
# systemctl status promiscuous
● promiscuous.service - Bring up network interfaces in promiscuous mode upon boot
Loaded: loaded (/usr/lib/systemd/system/promiscuous.service; enabled; vendor preset: disabled)
Active: active (exited) since Fri 2020-08-21 16:14:53 AEST; 17s ago
Process: 8088 ExecStart=/usr/sbin/ip link set dev enp0s8 promisc on (code=exited, status=0/SUCCESS)
Process: 8086 ExecStart=/usr/sbin/ip link set dev enp0s3 promisc on (code=exited, status=0/SUCCESS)
Main PID: 8088 (code=exited, status=0/SUCCESS)
Aug 21 16:14:53 [HOST] systemd[1]: Starting Bring up network interfaces in promiscuous mode upon boot...
Aug 21 16:14:53 [HOST] systemd[1]: Started Bring up network interfaces in promiscuous mode upon boot.
Verify promiscuous mode is enabled on all slave interfaces e.g.
# ip addr | grep enp
2: enp0s3: [BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP] mtu 1500 qdisc fq_codel master team0 state UP group default qlen 1000
3: enp0s8: [BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP] mtu 1500 qdisc fq_codel master team0 state UP group default qlen 1000
9. Testing Team network connectivity resilience - slave failover/failback
A. From a remote client, initiate continuous ping(8) of the server for which network teaming has been configured e.g.:
[client]$ ping [SERVER]
PING [IP3] ([IP3]) 56(84) bytes of data.
64 bytes from [IP3]: icmp_seq=1 ttl=64 time=0.025 ms
64 bytes from [IP3]: icmp_seq=2 ttl=64 time=0.034 ms
64 bytes from [IP3]: icmp_seq=3 ttl=64 time=0.039 ms
...
The following methods may be used to disconnect network connectivity:
# ip link set [connection] down
# nmcli dev dis [device]
- disconnect virtual network cable/link via the virtualisation host, for example - Oracle VM VirtualBox Manager > [guest] > Settings > Network > Adapter 1|… > Advanced > Cable Connected (deselect)
- programatically disconnect the physical/virtual switch port to the relevant system interface
B. Disconnect the currently active team slave (currently enp0s3 per Step 7. above) e.g.:
# ip link set enp0s3 down
teamdctl confirms former inactive slave enp0s8 has become the current active slave. Despite slave failover, ping from the remote client continues uninterrupted.
# teamdctl team0 state
setup:
runner: activebackup
ports:
enp0s3
link watches:
link summary: down
instance[link_watch_0]:
name: ethtool
link: down
down count: 1
enp0s8
link watches:
link summary: up
instance[link_watch_0]:
name: ethtool
link: up
down count: 0
runner:
active port: enp0s8
Running the ip command further confirms all network traffic to the team occurs via new active slave enp0s8 i.e.:
# ip -s link
...
2: enp0s3: [BROADCAST,MULTICAST,PROMISC] mtu 1500 qdisc fq_codel master team0 state DOWN mode DEFAULT group default qlen 1000
link/ether [MAC1] brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
3007606561 5014092 0 27008 0 91263
TX: bytes packets errors dropped carrier collsns
787749 7568 0 0 0 0
3: enp0s8: [BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP] mtu 1500 qdisc fq_codel master team0 state UP mode DEFAULT group default qlen 1000
link/ether [MAC1] brd ff:ff:ff:ff:ff:ff
span style="color: red;">RX: bytes packets errors dropped overrun mcast
3829373093 6116137 0 26680 0 111948
TX: bytes packets errors dropped carrier collsns
179163 1337 0 0 0 0
7: team0: [BROADCAST,MULTICAST,UP,LOWER_UP] mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether [MAC1] brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
2106463511 1969187 0 49615 0 60559
TX: bytes packets errors dropped carrier collsns
176341 1378 0 0 0 0
C. Disconnect current active team slave enp0s3 e.g.:
# ip link set enp0s8 down
# teamdctl team0 state
setup:
runner: activebackup
ports:
enp0s3
link watches:
link summary: down
instance[link_watch_0]:
name: ethtool
link: down
down count: 1
enp0s8
link watches:
link summary: down
instance[link_watch_0]:
name: ethtool
link: up
down count: 1
runner:
active port:
At this point, with both team slaves disconnected, ping from the remote client ceases e.g.
...
64 bytes from [IP3]: icmp_seq=1253 ttl=64 time=0.207 ms
64 bytes from [IP3]: icmp_seq=1254 ttl=64 time=0.131 ms
64 bytes from [IP3]: icmp_seq=1255 ttl=64 time=0.227 ms
64 bytes from [IP3]: icmp_seq=1256 ttl=64 time=0.218 ms
64 bytes from [IP3]: icmp_seq=1257 ttl=64 time=0.198 ms
From [IP3] icmp_seq=1258 Destination Host Unreachable
From [IP3] icmp_seq=1259 Destination Host Unreachable
...
D. Reconnect disconnected slave enp0s3 e.g.:
# ip link set enp0s3 up
# teamdctl team0 state
setup:
runner: activebackup
ports:
enp0s3
link watches:
link summary: up
instance[link_watch_0]:
name: ethtool
link: up
down count: 1
enp0s8
link watches:
link summary: down
instance[link_watch_0]:
name: ethtool
link: down
down count: 1
runner:
active port: enp0s3
At this point, with enp0s3 the current active slave, ping from the remote client resumes e.g.
...
From [IP3] icmp_seq=1392 Destination Host Unreachable
From [IP3] icmp_seq=1393 Destination Host Unreachable
From [IP3] icmp_seq=1394 Destination Host Unreachable
From [IP3] icmp_seq=1395 Destination Host Unreachable
64 bytes from [IP3]: icmp_seq=1396 ttl=64 time=1258180 ms
64 bytes from [IP3]: icmp_seq=1397 ttl=64 time=1257180 ms
64 bytes from [IP3]: icmp_seq=1398 ttl=64 time=1256181 ms
64 bytes from [IP3]: icmp_seq=1399 ttl=64 time=1255181 ms
...
E. Finally, reconnect disconnected slave enp0s8 so both team slaves interfaces are reinstated to the team e.g.:
# ip link set enp0s8 up
# teamdctl team0 state
setup:
runner: activebackup
ports:
enp0s3
link watches:
link summary: up
instance[link_watch_0]:
name: ethtool
link: up
down count: 1
enp0s8
link watches:
link summary: up
instance[link_watch_0]:
name: ethtool
link: up
down count: 1
runner:
active port: enp0s3