Leaf and spine network emulation on Mac OS M1/M2 systems

The GitHub sflow-rt/containerlab project contains example network topologies for the Containerlab network emulation tool that demonstrate real-time streaming telemetry in realistic data center topologies and network configurations. The examples use the same FRRouting (FRR) engine that is part of SONiC, NVIDIA Cumulus Linux, and DENT network operating systems. Containerlab can be used to experiment before deploying solutions into production. Examples include: tracing ECMP flows in leaf and spine topologies, EVPN visibility, and automated DDoS mitigation using BGP Flowspec and RTBH controls.

The Containerlab project currently has limited support for Mac OS, stating "ARM-based Macs (M1/2) are not supported, and no binaries are generated for this platform. This is mainly due to the lack of network images built for arm64 architecture as of now." However, this argument doesn't apply to the Linux based images used in these examples.

First install Docker Desktop on your Apple silicon based Mac (select the Apple Chip option).

mkdir clab
cd clab
docker run --rm -it --privileged \
  --network host --pid="host" \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /run/netns:/run/netns \
  -v $(pwd):$(pwd) -w $(pwd) \
  sflow/clab bash

Run Containerlab by typing the above commands in a terminal. This command uses a pre-built multi-architecture sflow/clab image. If you are running on an x86 platform, follow the official Containerlab Installation instructions.

git clone https://github.com/sflow-rt/containerlab.git

Download the Containerlab topologies from the sflow-rt/containerlab project.

containerlab deploy -t containerlab/clos5.yml

Start the 5 stage leaf and spine topology shown at the top of this page. The initial launch may take a couple of minutes as the container images are downloaded for the first time. Once the images are downloaded, the topology deploys in around 10 seconds.

An instance of the sFlow-RT real-time analytics engine receives industry standard sFlow telemetry from all the switches in the network. All of the switches in the topology are configured to send sFlow to the sFlow-RT instance. In this case, Containerlab is running the pre-built sflow/prometheus image which packages sFlow-RT with useful applications for exploring the data.

Connect to the web interface, http://localhost:8008. The sFlow-RT dashboard verifies that telemetry is being received from 10 agents (the 10 switches in the Clos fabric). See the sFlow-RT Quickstart guide for more information.

The screen capture shows a real-time view of traffic flowing across the network during a series iperf3 tests. Click on the sFlow-RT Apps menu and select the browse-flows application, or click here for a direct link to a chart with the settings shown above.

docker exec -it clab-clos5-h1 iperf3 -c 172.16.4.2

Each of the hosts in the network has an iperf3 server, so running the above command will test bandwidth between h1 and h4.

docker exec -it clab-clos5-leaf1 vtysh

Linux with open source routing software (FRRouting) is an accessible alternative to vendor routing stacks (no registration / license required, no restriction on copying means you can share images on Docker Hub, no need for virtual machines). FRRouting is popular in production network operating systems (e.g. Cumulus Linux, SONiC, DENT, etc.) and the VTY shell provides an industry standard CLI for configuration, so labs built around FRR allow realistic network configurations to be explored.

containerlab destroy -t containerlab/clos5.yml

When you are finished, run the above command to stop the containers and free the resources associated with the emulation. Try out other topologies from the project to explore topics such as DDoS mitigation, BGP Flowspec, and EVPN.

Moving the monitoring solution from Containerlab to production is straightforward since sFlow is widely implemented in datacenter equipment from vendors including: A10, Arista, Aruba, Cisco, Edge-Core, Extreme, Huawei, Juniper, NEC, Netgear, Nokia, NVIDIA, Quanta, and ZTE. In addition, the open source Host sFlow agent makes it easy to extend visibility beyond the physical network into the compute infrastructure.

VyOS DDoS mitigation

Real-time flow analytics on VyOS describes how to install real-time analytics based on sFlow and the sFlow-RT analytics engine. This article extends the example to show how to automatically mitigate DDoS attacks using flow analytics combined with BGP Remotely Triggered Black Hole (RTBH) / Flowspec.

vyos@vyos:~$ add container image sflow/ddos-protect

First, download the sflow/ddos-protect image.

vyos@vyos:~$ mkdir -m 777 /config/sflow-rt

Create a directory to store persistent container state.

set container network sflowrt prefix 192.168.1.0/24

Define an internal network to connect to container. Currently VyOS BGP does not allow direct connections to local addresses (e.g. 127.0.0.1), so we need to put controller on its own network so the router can connect and receive DDoS mitigation BGP RTBH / Flowspec controls.

set container name sflow-rt image sflow/ddos-protect
set container name sflow-rt host-name sflow-rt
set container name sflow-rt arguments '-Dddos_protect.router=192.168.1.1 -Dddos_protect.enable.flowspec=yes'
set container name sflow-rt environment RTMEM value 200M
set container name sflow-rt memory 0
set container name sflow-rt volume store source /config/sflow-rt
set container name sflow-rt volume store destination /sflow-rt/store
set container name sflow-rt network sflowrt address 192.168.1.2

Configure a container to run the image. The -Dddos_protect.router argument sets the BGP neighbor address, 192.168.1.1.

vyos@vyos:~$ ifconfig podman-sflowrt
podman-sflowrt: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.1  netmask 255.255.255.0  broadcast 192.168.1.255
        ether be:9e:69:f4:d0:4e  txqueuelen 1000  (Ethernet)
        RX packets 28  bytes 2662 (2.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 27  bytes 8032 (7.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Connections to containers on sflowrt container network appear to originate from 192.168.1.1, the address assigned to VyOS interface podman-sflowrt.

set system sflow interface eth0
set system sflow interface eth1
set system sflow interface eth2
set system sflow polling 30
set system sflow sampling-rate 1000
set system sflow drop-monitor-limit 50
set system sflow server 192.168.1.2

Configure sFlow and send to sflow-rt container address 192.168.1.2.

set protocols bgp system-as 64500
set protocols bgp neighbor 192.168.1.2 port 1179
set protocols bgp neighbor 192.168.1.2 remote-as 65000
set protocols bgp neighbor 192.168.1.2 address-family ipv4-unicast
set protocols bgp neighbor 192.168.1.2 address-family ipv4-flowspec

Configure sflow-rt as BGP neighbor. Documentation ASN 64500 should be replaced by your ASN. The private ASN 65000 is a DDoS Protect default and can be changed with the -Dddos_protect.as argument.

ssh -L 8008:192.168.1.2:8008 vyos@router.example

Use ssh tunnel to connect to the container network and access web interface at http://localhost:8008.

Real-time DDoS mitigation using BGP RTBH and FlowSpec describes how to configure the DDoS protect application. The screen capture above shows the Charts page after a couple of simulated DDoS attacks on an address, 198.51.100.129, protected by the VyOS router. The charts show two ip_flood and a single udp_amplification attack - see DDoS attacks and BGP Flowspec responses for information on simulating different types of DDoS attack to test mitigation responses.

The Controls page shows three active controls. The table shows the targeted address, administrative address group, attack type, protocol, detection time, mitigation action and status of each active DDoS attack.

vyos@vyos:~$ show bgp ipv4
BGP table version is 0, local router ID is 192.168.1.1, vrf id 0
Default local pref 100, local AS 64500
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

    Network          Next Hop            Metric LocPrf Weight Path
    198.51.100.129/32
                    192.0.2.1                              0 65000 i

Displayed  1 routes and 1 total paths

The show command verifies that a Remotely Triggered Black Hole (RTBH) rule has been received for the drop mitigation actions. Advertising a black hole route risks collateral damage since it drops all traffic to the targetted host in order to protect network bandwidth and services provided by other hosts. 

vyos@vyos:~$ show bgp ipv4 flowspec detail 
BGP flowspec entry: (flags 0x418)
        Destination Address 198.51.100.129/32
        IP Protocol = 17 
        Source Port = 53 
        FS:rate 0.000000
        received for 00:00:12
        not installed in PBR

The show command verifies that a Flowspec rule has been received for the filter mitigation action. Using Flowspec to filter traffic is more targetted than using black hole routes. In this case only UDP traffic (IP Protocol 17) with Source Port 53 (DNS) is dropped, all other services provided by the targetted host are still accessible.

vyos@vyos:~$ show log container sflow-rt 
2023-04-08T00:24:14Z INFO: Starting sFlow-RT 3.0-1681
2023-04-08T00:24:16Z INFO: Version check, running latest
2023-04-08T00:24:17Z INFO: Listening, BGP port 1179
2023-04-08T00:24:18Z INFO: Listening, sFlow port 6343
2023-04-08T00:24:19Z INFO: Listening, HTTP port 8008
2023-04-08T00:24:19Z INFO: DNS server 1.1.1.1
2023-04-08T00:24:19Z INFO: app/ddos-protect/scripts/ddos.js started
2023-04-08T00:24:19Z INFO: app/prometheus/scripts/export.js started
2023-04-08T00:24:19Z INFO: app/browse-drops/scripts/top.js started
2023-04-08T00:24:19Z INFO: app/browse-flows/scripts/top.js started
2023-04-08T00:26:11Z INFO: BGP open 192.168.1.1 51252
2023-04-08T14:37:36Z INFO: DDoS drop ip_flood 198.51.100.129 local 47
2023-04-08T14:38:19Z INFO: DDoS filter udp_amplification 198.51.100.129 local 53
2023-04-08T14:38:19Z INFO: DDoS drop ip_flood 198.51.100.129 local 17

Attacks are recorded in the container log. Monitoring DDoS mitigation describes how to use Prometheus / Elasticsearch / Grafana to monitor DDoS activity and build dashboards.

This is only a partial configuration. Peering sessions with upstream routers need to be configured to propagate controls so that DDoS attack traffic can be blocked before it saturates the upstream link. The limited scrubbing capacity of the VyOS software router isn't a factor since traffic will be dropped in hardware upstream. The flexibility of the VyOS router is an advantage in providing visibility and analytics to quickly trigger mitigation actions.

Real-time flow analytics on VyOS

VyOS with Host sFlow agent describes support for streaming sFlow telemetry added to the open source VyOS router operating system. This article describes how to install analytics software on a VyOS router by configuring a container.

vyos@vyos:~$ add container image sflow/ddos-protect

First, download the sflow/ddos-protect image.

vyos@vyos:~$ mkdir -m 777 /config/sflow-rt

Create a directory to store persistent container state.

set container name sflow-rt image sflow/ddos-protect
set container name sflow-rt allow-host-networks
set container name sflow-rt arguments '-Dhttp.hostname=10.0.0.240'
set container name sflow-rt environment RTMEM value 200M
set container name sflow-rt memory 0
set container name sflow-rt volume store source /config/sflow-rt
set container name sflow-rt volume store destination /sflow-rt/store

Configure a container to run the image. The RMEM environment variable setting limits the amount of memory that the container will use to 200M bytes. The -Dhttp.hostname argument sets the internal web server to listen on management address, 10.0.0.240, assigned to eth0 on this router. The container has is no built-in authentication, so access needs to be limited using an ACL or through a reverse proxy - see Download and install.

set system sflow interface eth0
set system sflow interface eth1
set system sflow interface eth2
set system sflow polling 30
set system sflow sampling-rate 1000
set system sflow drop-monitor-limit 50
set system sflow server 127.0.0.1

Next, configure sFlow agent to send to localhost (127.0.0.1).

Finally connect to the web interface on the router at port 8008. The status page verifies that the sFlow-RT analytics engine is receiving sFlow from 1 sFlow Agent (the VyOS router). See Getting started for more information.

The included Flow Browser application provides an up to the second view traffic flows. Defining Flows describes the fields that can be used to break out traffic.

VyOS dropped packet notifications describes how to configure and monitor sFlow dropped packet notifications. The included Discard Browser provides an up to the second view of dropped packets.

The included Metric Browser application lets you explore the metrics that are being streamed. The chart updates in real-time as data arrives and in this case shows CPU utilization on the VyOS router. The standard set of metrics exported by the Host sFlow agent include interface counters as well as host cpu, memory, network and disk performance metrics. Metrics lists the set of available metrics.

Flow metrics with Prometheus and Grafana describes how integrate flow analytics into operational dashboards. The included Prometheus application exposes flow analytics in the standard Prometheus scrape format so that they can be logged in time series databases.

DDoS protection quickstart guide describes how to use real-time sFlow analytics with BGP Flowspec / RTBH to automatically mitigate DDoS attacks. The included DDoS Protect application detects common volumetric attacks and can apply automated responses. The screen capture shows traffic associated with a series of simulated DDoS attacks against hosts behind the VyOS router, see DDoS attacks and BGP Flowspec responses.

The embedded sFlow-RT analytics engine exposes a REST API that can be used to program flow analytics, set thresholds, monitor events, and gather statistics. In addition, the applications shown in this article were all written using sFlow-RT's embedded scripting API. See Writing Applications for more information.

Dropped packet reason codes in VyOS

The article VyOS with Host sFlow agent describes how to use industry standard sFlow telemetry to monitor network traffic flows and statistics in the latest VyOS rolling releases. VyOS dropped packet notifications describes how sFlow also provides visibility into network packet drops and Dropped packet reason codes in Linux 6+ kernels describes how newer kernels are able to provide specific reasons for dropping packets. 

vyos@vyos:~$ uname -r
6.1.22-amd64-vyos

The latest VyOS rolling release runs on a Linux 6.1 kernel and the latest release of VyOS now provides enhanced visibility into dropped packets using kernel reason codes.

vyos@vyos:~$ show version
Version:          VyOS 1.4-rolling-202303310716
Release train:    current

Built by:         autobuild@vyos.net
Built on:         Fri 31 Mar 2023 07:16 UTC
Build UUID:       1a7448d9-d53c-48a0-8644-ed1970c1abb8
Build commit ID:  75c9311fba375e

Architecture:     x86_64
Boot via:         installed image
System type:       guest

Hardware vendor:  innotek GmbH
Hardware model:   VirtualBox
Hardware S/N:     0
Hardware UUID:    da75808d-ff60-1d4c-babd-84a7fa341053

Copyright:        VyOS maintainers and contributors

Verify that the version of of VyOS is VyOS 1.4-rolling-202303310716 or later.

In the previous article, VyOS dropped packet notifications,  two tests were performed, the first a failed attempt to connect to the VyOS router using telnet (telnet has been disabled in the router config), and the second a traceroute test between two hosts connected to the router. The sFlow drop reason codes reported for these two tests were unknown_l4 and unknown_l3 respectively. The Linux kernel functional names weren't much more specific, tcp_v4_rcv and ip_forward respectively. However, in this case, the Linux 6.1 kernel instrumentation allows more specific sFlow drop reasons to be reported, as shown in the chart at the top of this article.

• port_unreachable This sFlow drop reason code is defined by reference to RFC 1812 section 5.2.7.1 and is defined as "Port Unreachable - generated if the designated transport protocol (e.g., UDP) is unable to demultiplex the datagram in the transport layer of the final destination but has no protocol mechanism to inform the sender"
• ip_1_parsing This sFlow drop reason code is defined by reference to Devlink Trap and is defined as "Traps packets dropped due to an error in the first IP header parsing. This packet trap could include packets which do not pass an IP checksum check, a header length check (a minimum of 20 bytes), which might suffer from packet truncation thus the total length field exceeds the received packet length etc."

The detailed reasons make it easier to identify the root causes of packet drops, particularly when combined with information from the dropped packet's header that is also included in the sFlow Dropped Packet Notification messages.

Dropped packet reason codes in Linux 6+ kernels

Using sFlow to monitor dropped packets describes support for standard sFlow Dropped Packet Notications in the open source Host sFlow agent. This article describes additional capabilities in Linux 6+ kernels that clarify reasons why packets are dropped in the kernel.

The recent addition of dropreason.h in Linux 6+ kernels provides detailed reasons for packet drops. The netlink drop_monitor API has been extended to include the NET_DM_ATTR_REASON attribute to report the drop reason, see net_dropmon.h.

The following example illustrates the value of the reason code in explaining Linux packet drops.

tcp_v4_rcv+0x7c/0xef0

The value of NET_DM_ATTR_SYMBOL shown above indicates that the packet was dropped in the tcp_v4_rcv function in Linux kernel at memory location 0x7c/0xef0. While this information is helpful, there are many reasons why a TCP packet may be dropped.

NO_SOCKET

In this case, the value of NET_DM_ATTR_REASON shown above indicates that the TCP packet was dropped because no application had opened a socket and so there was nowhere to deliver the packet.

In the case of Linux-based hardware switches or smart network adapters, where packet processing is offloaded to hardware, the netlink drop_monitor events include NET_DM_ATTR_HW_TRAP_GROUP_NAME and NET_DM_ATTR_HW_TRAP_NAME attributes and packet header information supplied by the hardware driver, see Devlink Trap.

The latest version of the open source Host sFlow agent includes adds support for the NET_DM_ATTR_REASON attribute to improve the accuracy of the sFlow drop_reason.

port_unreachable

In our example, the Host sFlow is now able to report port_unreachable as the reason for the dropped packet, rather than a generic unknown_l4 reason reported for older kernels.

The screen capture at the top of this article shows dropped packet information displayed in real-time using the Discard Browser application running on the sFlow-RT analytics engine. The chart demonstrates how the combination of information from the header of the dropped packet along with the reason for dropping the packet quickly gets to the root cause of the packet drop. In this case an attempt has been made from 172.16.1.174 to connect to 172.16.1.1 via telnet (tcp port 23) and telnet has not been enabled on the server so the packet was dropped - as it should be since telnet is not a secure method of connecting.

docker run --name sflow-rt -p 8008:8008 -p 6343:6343/udp -d sflow/prometheus

A quick way to experiment with sFlow is to run the pre-built sflow/prometheus image using Docker. The bundled Discard Browser with the settings shown in the screen capture can be launched by clicking here.

VyOS dropped packet notifications

VyOS with Host sFlow agent describes how to configure and analyze industry standard sFlow telemetry recently added to the VyOS open source router platform. This article discusses sFlow dropped packet notifications support added to the latest release.

Dropped packets have a profound impact on network performance and availability. Packet discards due to congestion can significantly impact application performance. Dropped packets due to black hole routes, expired TTLs, MTU mismatches, etc. can result in insidious connection failures that are time consuming and difficult to diagnose. Visibility into dropped packets offers significant benefits for network troubleshooting, providing real-time network-wide visibility into the specific packets that were dropped as well the reason the packet was dropped. This visibility instantly reveals the root cause of drops and the impacted connections.

vyos@vyos:~$ show version
Version:          VyOS 1.4-rolling-202303260914
Release train:    current

Built by:         autobuild@vyos.net
Built on:         Sun 26 Mar 2023 09:14 UTC
Build UUID:       72b34f74-bfcd-4b51-9b95-544319c2dac5
Build commit ID:  d68bda6a295ba9

Architecture:     x86_64
Boot via:         installed image
System type:       guest

Hardware vendor:  innotek GmbH
Hardware model:   VirtualBox
Hardware S/N:     0
Hardware UUID:    df0a2b79-b8c4-8342-a27f-76aa3e52ad6d

Copyright:        VyOS maintainers and contributors

Verify that the version of of VyOS is VyOS 1.4-rolling-202303260914 or later.

On VyOS dropped packet monitoring relies on instrumentation built into recent Linux kernels and exposed through the netlink drop_monitor API. Enabling drop_monitor in VyOS kernel configuration allows the Host sFlow agent to capture and export information on dropped packets.

set system sflow interface eth0
set system sflow interface eth1
set system sflow interface eth2
set system sflow polling 30
set system sflow sampling-rate 1000
set system sflow drop-monitor-limit 50
set system sflow server 10.0.0.30 port 6343

The drop-monitor-limit configuration entry enables dropped packet monitoring and sets a rate limit of 50 dropped packets notifications per second.

docker run --name sflow-rt -p 8008:8008 -p 6343:6343/udp -d sflow/prometheus

A quick way to experiment with sFlow is to run the pre-built sflow/prometheus image using Docker on the sFlow server (in this case on 10.0.0.30). The chart at the top of the page uses the Discard Browser application to display an up to the second view of packets dropped by the VyOS router, click on this link to open the application with the settings shown.

The chart shows the results of two tests, the first a failed attempt to connect to the VyOS router using telnet (telnet has been disabled in the router config), and the second a traceroute test between two hosts connected to the router. The reason field reports the sFlow drop reason code and the function reports the linux kernel function that dropped the packet. With the telnet test, the packet was dropped in the tcp_v4_rcv function and is reported as an unknown_l4 sFlow reason. In the case of the traceroute test, 3 packets were dropped in the ip_forward function and are reported as unknown_l3 reason.

Enabling sFlow dropped packet notifications on all switches, routers, and hosts provides end-to-end visibility into dropped packets, rapidly identifying the location and reason for packet drops as well as identifying the impacted services.

Dropped packet monitoring complements sFlow's existing counter polling and packet sampling mechanisms and shares a common data model so that all three sources of data can be correlated. For example, if packets are being discarded because of buffer exhaustion, the discard records don't necessarily tell the whole story. The discarded packets may represent mice flows that are victims of an elephant flow. Packet samples will reveal the traffic that isn't being dropped and provide a more complete picture. Counter data adds additional information such as CPU load, interface speed, link utilization, packet and discard rates that further completes the picture.

VyOS with Host sFlow agent

VyOS described deficiencies with the embedded sFlow implementation in the open source VyOS router operating system and suggested that the open source Host sFlow agent be installed as an alternative. The VyOS developer community embraced the suggestion and has been incredibly responsive,  integrating, and releasing a version of VyOS with Host sFlow support within a week.

vyos@vyos:~$ show version
Version:          VyOS 1.4-rolling-202303170317
Release train:    current

Built by:         autobuild@vyos.net
Built on:         Fri 17 Mar 2023 03:17 UTC
Build UUID:       45391302-1240-4cc7-95a8-da8ee6390765
Build commit ID:  e887f582cfd7de

Architecture:     x86_64
Boot via:         installed image
System type:       guest

Hardware vendor:  innotek GmbH
Hardware model:   VirtualBox
Hardware S/N:     0
Hardware UUID:    871dd0f0-c4ec-f147-b1a7-ed536511f141

Copyright:        VyOS maintainers and contributors

Verify that the version of of VyOS is VyOS 1.4-rolling-202303170317 or later

set system sflow interface eth0
set system sflow interface eth1
set system sflow interface eth2
set system sflow polling 30
set system sflow sampling-rate 1000
set system sflow server 10.0.0.30 port 6343

The above commands configure sFlow export in the VyOS CLI using the embedded Host sFlow agent.

docker run --name sflow-rt -p 8008:8008 -p 6343:6343/udp -d sflow/prometheus

A quick way to experiment with sFlow is to run the pre-built sflow/prometheus image using Docker on the sFlow server (in this case on 10.0.0.30). The chart at the top of the page uses the Flow Browser application to display an up to the second view of the largest tcp flows through the VyOS router, click on this link to open the application with the settings shown.

Flow metrics with Prometheus and Grafana describes how integrate flow analytics into operational dashboards.

DDoS protection quickstart guide describes how to use real-time sFlow analytics with BGP Flowspec / RTBH to automatically mitigate DDoS attacks.