CONTAINERlab

CONTAINERlab is a Docker orchestration tool for creating virtual network topologies. This article describes how to build and monitor the leaf and spine topology shown above.

Note: Docker testbed describes a simple testbed for experimenting with sFlow analytics using Docker Desktop, but it doesn't have the ability to construct complex topologies. 

multipass launch --cpus 2 --mem 4G --name containerlab
multipass shell containerlab

The above commands use the multipass command line tool to create an Ubuntu virtual machine and open shell access.

sudo apt update
sudo apt -y install docker.io
bash -c "$(curl -sL https://get-clab.srlinux.dev)"

Type the above commands into the shell to install CONTAINERlab.

Note: Multipass describes how to build a Mininet network emulator to experiment with software defined networking.

name: test
topology:
  nodes:
    leaf1:
      kind: linux
      image: sflow/frr
    leaf2:
      kind: linux
      image: sflow/frr
    spine1:
      kind: linux
      image: sflow/frr
    spine2:
      kind: linux
      image: sflow/frr
    h1:
      kind: linux
      image: alpine:latest
    h2:
      kind: linux
      image: alpine:latest
  links: 
    - endpoints: ["leaf1:eth1","spine1:eth1"]
    - endpoints: ["leaf1:eth2","spine2:eth1"]
    - endpoints: ["leaf2:eth1","spine1:eth2"]
    - endpoints: ["leaf2:eth2","spine2:eth2"]
    - endpoints: ["h1:eth1","leaf1:eth3"]
    - endpoints: ["h2:eth1","leaf2:eth3"]

The test.yml file shown above specifies the topology. In this case we are using FRRouting (FRR) containers for the leaf and spine switches and Alpine Linux containers for the two hosts.

sudo containerlab deploy --topo test.yml

The above command creates the virtual network and starts containers for each of the network nodes.

sudo containerlab inspect --topo test.yml

Type the command above to list the container instances in the topology.

The table shows each of the containers and the assigned IP addresses.

sudo docker exec -it clab-test-leaf1 vtysh

Type the command above to run the FRR VTY shell so that the switch can be configured.

leaf1# show running-config 
Building configuration...

Current configuration:
!
frr version 7.5_git
frr defaults datacenter
hostname leaf1
log stdout
!
interface eth3
 ip address 172.16.1.1/24
!
router bgp 65006
 bgp router-id 172.20.20.6
 bgp bestpath as-path multipath-relax
 bgp bestpath compare-routerid
 neighbor fabric peer-group
 neighbor fabric remote-as external
 neighbor fabric description Internal Fabric Network
 neighbor fabric capability extended-nexthop
 neighbor eth1 interface peer-group fabric
 neighbor eth2 interface peer-group fabric
 !
 address-family ipv4 unicast
  network 172.16.1.0/24
 exit-address-family
!
route-map ALLOW-ALL permit 100
!
ip nht resolve-via-default
!
line vty
!
end

The BGP configuration for leaf1 is shown above.

Note: We are using BGP unnumbered to simplify the configuration so peers are automatically discovered.

The other switches, leaf2, spine1, and spine2 have similar configurations.

Next we need to configure the hosts.

sudo docker exec -it clab-test-h1 sh

Open a shell on h1

ip addr add 172.16.1.2/24 dev eth1
ip route add 172.16.2.0/24 via 172.16.1.1

Configure networking on h1. The other host, h2, has a similar configuration.

sudo docker exec -it clab-test-h1 ping 172.16.2.2
PING 172.16.2.2 (172.16.2.2): 56 data bytes
64 bytes from 172.16.2.2: seq=0 ttl=61 time=0.928 ms
64 bytes from 172.16.2.2: seq=1 ttl=61 time=0.160 ms
64 bytes from 172.16.2.2: seq=2 ttl=61 time=0.201 ms

Use ping to verify that there is connectivity between h1 and h2.

apk add iperf3

Install iperf3 on h1 and h2

iperf3 -s --bind 172.16.2.2

Run an iperf3 server on h2

iperf3 -c 172.16.2.2
Connecting to host 172.16.2.2, port 5201
[  5] local 172.16.1.2 port 52066 connected to 172.16.2.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.41 GBytes  12.1 Gbits/sec    0   1.36 MBytes       
[  5]   1.00-2.00   sec  1.41 GBytes  12.1 Gbits/sec    0   1.55 MBytes       
[  5]   2.00-3.00   sec  1.44 GBytes  12.4 Gbits/sec    0   1.55 MBytes       
[  5]   3.00-4.00   sec  1.44 GBytes  12.3 Gbits/sec    0   2.42 MBytes       
[  5]   4.00-5.00   sec  1.46 GBytes  12.6 Gbits/sec    0   3.28 MBytes       
[  5]   5.00-6.00   sec  1.42 GBytes  12.2 Gbits/sec    0   3.28 MBytes       
[  5]   6.00-7.00   sec  1.44 GBytes  12.4 Gbits/sec    0   3.28 MBytes       
[  5]   7.00-8.00   sec  1.28 GBytes  11.0 Gbits/sec    0   3.28 MBytes       
[  5]   8.00-9.00   sec  1.40 GBytes  12.0 Gbits/sec    0   3.28 MBytes       
[  5]   9.00-10.00  sec  1.25 GBytes  10.7 Gbits/sec    0   3.28 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  13.9 GBytes  12.0 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  13.9 GBytes  12.0 Gbits/sec                  receiver

Run an iperf3 test on h1

Now that we have a working test network, it's time to add some monitoring.

We will be installing sFlow agents on the switches and hosts that will stream telemetry to sFlow-RT analytics software which will provide a real-time network-wide view of performance.

sudo docker exec -it clab-test-leaf1 sh

Open a shell on leaf1

apk --update add libpcap-dev build-base linux-headers gcc git
git clone https://github.com/sflow/host-sflow.git
cd host-sflow/
make FEATURES="PCAP"
make install

Install Host sFlow agent on leaf1.

Note: The steps above could be included in a Dockerfile in order to create an image with built-in instrumentation.

vi /etc/hsflowd.conf

Edit the Host sFlow configuration file.

sflow {
  polling = 30
  sampling = 400
  collector { ip = 172.20.20.1 }
  pcap { dev = eth1 }
  pcap { dev = eth2 }
  pcap { dev = eth3 }
}

The above settings enable packet sampling on interfaces eth1, eth2 and eth3

sudo docker exec -d clab-test-leaf1 /usr/sbin/hsflowd -d

Start the Host sFlow agent on leaf1.

Install and run Host sFlow agents on the remaining switches and hosts, leaf2, spine1, spine2, h1, and h2.

sudo docker run --rm -d -p 6343:6343/udp -p 8008:8008 --name sflow-rt sflow/prometheus

Use the pre-built sflow/prometheus container to start an instance of sFlow-RT to collect and analyze the telemetry.

multipass list

List the multipass virtual machines.

containerlab            Running           192.168.64.7     Ubuntu 20.04 LTS
                                          172.17.0.1
                                          172.20.20.1

Use a web browser to connect to the sFlow-RT web interface. In this case at http://192.168.64.7:8008 

The sFlow-RT dashboard verifies that telemetry is being received from 6 agents (the four switches and two hosts).

The screen capture shows a real-time view of traffic flowing across the network during an iperf3 test. 

The chart shows that the traffic flows via spine2. Repeated tests showed that traffic traffic was never taking the path via spine1, indicating that the ECMP hash function was not taking into account the TCP ports.

sudo docker exec clab-test-leaf1 sysctl -w net.ipv4.fib_multipath_hash_policy=1
sudo docker exec clab-test-leaf2 sysctl -w net.ipv4.fib_multipath_hash_policy=1

We are using a newer Linux kernel, so running the above commands changes the hashing algorithm to include the layer 4 headers, see Celebrating ECMP in Linux — part one and Celebrating ECMP in Linux — part two.

Topology describes how knowledge of network topology can be used to enhance the analytics capabilities of sFlow-RT.

{
  "links": {
    "link1": { "node1":"leaf1","port1":"eth1","node2":"spine1","port2":"eth1"},
    "link2": { "node1":"leaf1","port1":"eth2","node2":"spine2","port2":"eth1"},
    "link3": { "node1":"leaf2","port1":"eth1","node2":"spine1","port2":"eth2"},
    "link4": { "node1":"leaf2","port1":"eth2","node2":"spine2","port2":"eth2"}
  }
}

The links specification in the test.yml file can easily be converted into sFlow-RT's JSON format.

CONTAINERlab is a very promising tool for efficiently emulating complex networks. CONTAINERlab supports NokiaSR-Linux, Juniper vMX, Cisco IOS XRv9k and Arista vEOS, as well as Linux containers. Many of the proprietary network operating systems are only delivered as virtual machines and Vrnetlab integration makes it possible for CONTAINERlab to run these virtual machines. However, virtual machine nodes require considerably more resources than simple containers.

Linux with open source routing software (FRRouting) is an accessible alternative to vendor routing stacks (no registration / license required, no restriction on copying means you can share images on Docker Hub, no need for virtual machines).  FRRouting is popular in production network operating systems (e.g. Cumulus Linux, SONiC, DENT, etc.) and the VTY shell provides an industry standard CLI for configuration, so labs built around FRR allow realistic network configurations to be explored.