Fixing Ghost Drops: How eBPF Rescued IPv6 Telemetry

A customer complains that they aren't getting IPFIX flow data from a router.

Use socat to check that IPFIX is being received (IANA assigned port for IPFIX is 4739):

socat -b 0 -dd -u UDP6-RECV:4739 - 2>&1

Output demonstrates that at least some IPFIX messages can be received when listening on port 4739.

2026/05/15 22:46:32 socat[108419] N using stdout for writing
2026/05/15 22:46:32 socat[108419] N starting data transfer loop with FDs [5,5] and [1,1]
2026/05/15 22:46:33 socat[108419] N received packet with 0 bytes from AF=10 [fec0:0000:0000:0000:0001:000c:2744:69f1]:50978
2026/05/15 22:46:33 socat[108419] N received packet with 0 bytes from AF=10 [fec0:0000:0000:0000:0001:000c:2744:69f1]:50978

Use tcpdump to check for IPFIX packets. This gives visibility into packets before the host network stack, so you can see packets before they are dropped by host network stack or host firewall

tcpdump -i enp0s3 -n udp port 4739

The output shows that IPFIX datagrams are being received from a second source, fec0::1:c:2744:69f0, but they aren't showing up in the socat output, so the Linux kernel must be dropping them for some reason.

dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp0s3, link-type EN10MB (Ethernet), snapshot length 262144 bytes
21:09:57.217821 IP6 fec0::1:c:2744:69f1.50978 > fec0::27ff:fe8d:4f0b.ipfix: UDP, length 1432
21:09:57.217921 IP6 fec0::1:c:2744:69f0.50978 > fec0::27ff:fe8d:4f0b.ipfix: UDP, length 1428

A check of the host firewall and reverse path filtering settings don't explain the drops, so take a more detailed look with tcpdump with the -v (verbose) option.

tcpdump -i enp0s3 -nv udp port 4739

This time we see that packets from fec0::1:c:2744:69f0 have a bad UDP checksum.

dropped privs to tcpdump
tcpdump: listening on enp0s3, link-type EN10MB (Ethernet), snapshot length 262144 bytes
22:07:36.443823 IP6 (flowlabel 0x1991c, hlim 64, next-header UDP (17) payload length: 1230) fec0::1:c:2744:69f1.50978 > fec0::27ff:fe8d:4f0b.ipfix: [udp sum ok] UDP, length 1222
22:07:37.356528 IP6 (flowlabel 0x1991c, hlim 64, next-header UDP (17) payload length: 1008) fec0::1:c:2744:69f0.50978 > fec0::27ff:fe8d:4f0b.ipfix: [bad udp cksum 0xfc60 -> 0xfb60!] UDP, length 1000

Verify by looking at the UDP error counter:

nstat -az Udp6InErrors

Results show an increasing error counter, confirming that the packets are being dropped by the Linux kernel.

#kernel
Udp6InErrors                    5106               0.0

Unfortunately, simple fixes that might work for IPv4 (disabling/ignoring UDP checksum) are not available for IPv6, see RFC 8200 Internet Protocol, Version 6 (IPv6) Specification

Unlike IPv4, the default behavior when UDP packets are originated by an IPv6 node is that the UDP checksum is not optional. That is, whenever originating a UDP packet, an IPv6 node must compute a UDP checksum over the packet and the pseudo-header, and, if that computation yields a result of zero, it must be changed to hex FFFF for placement in the UDP header. IPv6 receivers must discard UDP packets containing a zero checksum and should log the error.

Ideally the router would send IPFIX with correctly computed UDP checksum and on closer examination it looks that some IFPIX messages from the router do have a correct UDP checksum while others do not. The messages with the correct checksum originate from the routers management CPU, while the messages with incorrect checksum are being directly generated in hardware by the routing chip.

This isn't too surprising - if this were IPv4 export, the hardware could set the UDP checksum to zero (since it is optional) and there would be no issue, However, with IPv6 the mandatory checksum must be computed - a complex calculation that is likely to produce errors.

Computation of the UDP checksum involves creating an IPv6 pseudo header (shown above) and then calculating the checksum over the IPv6 pseudo header, the UDP header, and the UDP payload.

Ideally, the router vendor will fix the issue, but it may not be possible if there are hardware limitations, or it may take time if a fix isn't seen as a priority, so a workaround is needed.

This is where eBPF comes to the rescue! The GitHub fix-udp6-checksum project uses an eBPF program, fix_checksum.c, to compute and rewrite the UDP checksum before the IPFIX packet is handed to the Linux network stack.

Dowload and compile the fix_checksum.c program on a system running Docker:

git clone https://github.com/inmoncorp/fix-udp6-checksum.git
cd fix-udp6-checksum
./build.sh

Copy the resulting fix_checksum.o file to the IPFIX collector.

Install the eBPF program on interface enp0s3

sudo tc qdisc add dev enp0s3 clsact
sudo tc filter add dev enp0s3 ingress bpf da obj fix_checksum.o sec tc/ingress

Check to see that the filter has been installed

sudo tc -s filter show dev enp0s3 ingress

The output shows that the filter is installed and that the eBPF Just-In-Time (JIT) compiler has run for maximum performance.

filter protocol all pref 49151 bpf chain 0 
filter protocol all pref 49151 bpf chain 0 handle 0x1 fix_checksum.o:[tc/ingress] direct-action not_in_hw id 230 name fix_ipfix_check tag c6a96524b9f80adb jited

Finally, using socat to verify that the missing IPFIX data is being received:

2026/05/18 14:06:02 socat[945320] N using stdout for writing
2026/05/18 14:06:02 socat[945320] N starting data transfer loop with FDs [5,5] and [1,1]
2026/05/18 14:06:03 socat[945320] N received packet with 0 bytes from AF=10 [fec0:0000:0000:0000:0001:000c:2744:69f1]:50978
2026/05/18 14:06:03 socat[945320] N received packet with 0 bytes from AF=10 [fec0:0000:0000:0000:0001:000c:2744:69f0]:50978

eBPF is a game changer, providing the ability to change how Linux processes packets without having to build a new kernel or even reboot a production system. What would otherwise have been a major issue is transformed into a relatively straightforward fix and a happy customer.

Vector Packet Processor (VPP) dropped packet notifications

Vector Packet Processor (VPP) release 25.10 extends the sFlow implementation to include support for dropped packet notifications, providing detailed, low overhead, visibility into traffic flowing through a VPP router, see Vector Packet Processor (VPP) for performance information.

sflow sampling-rate 10000
sflow polling-interval 20
sflow header-bytes 128
sflow direction both
sflow drop-monitoring enable
sflow enable GigabitEthernet0/8/0
sflow enable GigabitEthernet0/9/0
sflow enable GigabitEthernet0/a/0

The above VPP configuration commands enable sFlow monitoring of the VPP dataplane, randomly sampling packets, periodically polling counters, and capturing dropped packets and reason codes. The measurements are send via Linux netlink messages to an instance of the open source Host sFlow agent (hsflowd) which combines the measurements and streams standard sFlow telemetry to a remote collector.

sflow {
  collector { ip=192.0.2.1 udpport=6343 }
  psample { group=1 egress=on }
  dropmon { start=on limit=50 }
  vpp { }
}

The /etc/hsflowd.conf file above enables the modules needed to receive netlink messages from VPP and send the resulting sFlow telemetry to a collector at 192.0.2.1. See vpp-sflow for detailed instructions.

docker run -p 6343:6343/udp sflow/sflowtool

Run sflowtool on the sFlow collector host to verify verify that the data is being received and to see the information available in the sFlow telemetry stream. The pre-built sflow/sflowtool Docker image is the fastest way to run sflowtool.

Docker also makes it easy to try out other sFlow analytics tools, for example, Deploy real-time network dashboards using Docker compose, describes how to quickly set up measurement stack consisting of the sFlow-RT real-time analytics engine, Prometheus time series database, and Grafana dashboard builder.

The sFlow VPP module delivers the same industry standard network performance monitoring available in switches and routers from leading vendors, including Arista, Cisco, Dell, Juniper, NVIDIA, VyOS, etc. to provide comprehensive, network-wide, visibility.

Packet trimming

The latest version of the AI Metrics dashboard uses industry standard sFlow telemetry from network switches to monitor the number of trimmed packets to use as a congestion metric.

Ultra Ethernet Specification Update describes how the Ultra Ethernet Transport (UET) Protocol has the ability to leverage optional “packet trimming” in network switches, which allows packets to be truncated rather than dropped in the fabric during congestion events. As packet spraying causes reordering, it becomes more complicated to detect loss. Packet trimming gives the receiver and the sender an early explicit indication of congestion, allowing immediate loss recovery in spite of reordering, and is a critical feature in the low-RTT environments where UET is designed to operate.

cumulus@switch:~$ nv set system forwarding packet-trim profile packet-trim-default
cumulus@switch:~$ nv config apply

NVIDIA Cumulus Linux release 5.14 for NVIDA Spectrum Ethernet Switches includes support for Packet Trimming. The above command enables packet trimming, sets the DSCP remark value to 11, sets the truncation size to 256 bytes, sets the switch priority to 4, and sets the eligibility to all ports on the switch with traffic class 1, 2, and 3. NVIDA BlueField host adapters respond to trimmed packets to ensure fast congestion recovery.

Instructions for deploying the Grafana AI Metrics dashboard are provided in the articles: AI Metrics, AI Metrics with Prometheus and Grafana and AI Metrics with Grafana Cloud.

AI Metrics with InfluxDB Cloud

The InfluxDB AI Metrics dashboard shown above tracks performance metrics for AI/ML RoCEv2 network traffic, for example, large scale CUDA compute tasks using NVIDIA Collective Communication Library (NCCL) operations for inter-GPU communications: AllReduce, Broadcast, Reduce, AllGather, and ReduceScatter.

The metrics include:

• Total Traffic Total traffic entering fabric
• Operations Total RoCEv2 operations broken out by type
• Core Link Traffic Histogram of load on fabric links
• Edge Link Traffic Histogram of load on access ports
• RDMA Operations Total RDMA operations
• RDMA Bytes Average RDMA operation size
• Credits Average number of credits in RoCEv2 acknowledgements
• Period Detected period of compute / exchange activity on fabric (in this case just over 0.5 seconds)
• Congestion Total ECN / CNP congestion messages
• Errors Total ingress / egress errors
• Discards Total ingress / egress discards
• Drop Reasons Packet drop reasons

This article shows how to integrate with InfluxDB Cloud instead of running the services locally.

Note: InfluxDB Cloud has a free service tier that can be used to test this example.

Save the following compose.yml file on a system running Docker.

configs:
  config.telegraf:
    content: |
      [agent]
        interval = '15s'
        round_interval = true
        omit_hostname = true
      [[outputs.influxdb_v2]]
        urls = ['https://<INFLUXDB_CLOUD_INSTANCE>.cloud2.influxdata.com']
        token = '<INFLUXDB_CLOUD_TOKEN>'
        organization = '<INFLUXDB_CLOUD_USER>'
        bucket = 'sflow'
      [[inputs.prometheus]]
        urls = ['http://sflow-rt:8008/app/ai-metrics/scripts/metrics.js/prometheus/txt']
        metric_version = 1

networks:

  monitoring:
    driver: bridge

services:

  sflow-rt:
    image: sflow/ai-metrics
    container_name: sflow-rt
    restart: unless-stopped
    ports:
      - '6343:6343/udp'
      - '8008:8008'
    networks:
      - monitoring

  telegraf:
    image: telegraf:alpine
    container_name: telegraf
    restart: unless-stopped
    configs:
      - source: config.telegraf
        target: /etc/telegraf/telegraf.conf
    depends_on:
      - sflow-rt
    networks:
      - monitoring

Use the Load Data menu to create an sflow bucket, create an API TOKEN to upload data, and find TELEGRAF INFLUXDB OUTPUT PLUGIN settings. Navigate to the Dashboards menu and create a new dashboard by importing ai_metrics.json

Edit the highlighted outputs.influxdb_v2 telegraf settings (INFLUXDB_CLOUD_INSTANCE, INFLUXDB_CLOUD_TOKEN and INFLUXDB_CLOUD_USER) to match those provided by InfluxDB Cloud.

docker compose up -d

Run the command above to start streaming metrics to InfluxDB Cloud.

Enable sFlow on all switches in the cluster (leaf and spine) using the recommeded settings. Enable sFlow dropped packet notifications to populate the drop reasons metric, see Dropped packet notifications with Arista Networks, NVIDIA Cumulus Linux 5.11 for AI / ML and Dropped packet notifications with Cisco 8000 Series Routers for examples.

Note: Tuning Performance describes how to optimize settings for very large clusters.

Industry standard sFlow telemetry is uniquely suited monitoring AI workloads. The sFlow agents leverage instrumentation built into switch ASICs to stream randomly sampled packet headers and metadata in real-time. Sampling provides a scaleable method of monitoring the large numbers of 400G/800G links found in AI fabrics. Export of packet headers allows the sFlow collector to decode the InfiniBand Base Transport headers to extract operations and RDMA metrics. The Dropped Packet extension uses Mirror-on-Drop (MoD) / What Just Happened (WJH) capabilities in the ASIC to include packet header, location, and reason for EVERY dropped packet in the fabric.

Talk to your switch vendor about their plans to support the Transit delay and queueing extension. This extension provides visibility into queue depth and switch transit delay using instrumentation built into the ASIC.

A network topology is required to generate the analytics, see Topology for a description of the JSON file and instructions for generating topologies from Graphvis DOT format, NVIDIA NetQ, Arista eAPI, and NetBox.

Use the Topology Status dashboard to verify that the topology is consistent with the sFlow telemetry and fully monitored. The Locate tab can be used to locate network addresses to access switch ports.

Note: If any gauges indicate an error, click on the gauge to get specific details.

Congratulations! The configuration is now complete and you should see charts above in AI Metric application Traffic tab. In addition, the AI Metrics dashboard at the top of this page should start to populate with data.

Getting Started provides an introductions to sFlow-RT, describes how to browse metrics and traffic flows using tools included in the Docker image, and links to information on creating applications using sFlow-RT APIs.

AI network performance monitoring using containerlab

AI Metrics is available on GitHub. The application provides performance metrics for AI/ML RoCEv2 network traffic, for example, large scale CUDA compute tasks using NVIDIA Collective Communication Library (NCCL) operations for inter-GPU communications: AllReduce, Broadcast, Reduce, AllGather, and ReduceScatter.

The screen capture is from a containerlab topology that emulates a AI compute cluster connected by a leaf and spine network. The metrics include:

• Total Traffic Total traffic entering fabric
• Operations Total RoCEv2 operations broken out by type
• Core Link Traffic Histogram of load on fabric links
• Edge Link Traffic Histogram of load on access ports
• RDMA Operations Total RDMA operations
• RDMA Bytes Average RDMA operation size
• Credits Average number of credits in RoCEv2 acknowledgements
• Period Detected period of compute / exchange activity on fabric (in this case just over 0.5 seconds)
• Congestion Total ECN / CNP congestion messages
• Errors Total ingress / egress errors
• Discards Total ingress / egress discards
• Drop Reasons Packet drop reasons

Note: Clicking on peaks in the charts shows values at that time.

This article gives step-by-step instructions to run the demonstration.

git clone https://github.com/sflow-rt/containerlab.git

Download the sflow-rt/containerlab project from GitHub.

git clone https://github.com/sflow-rt/containerlab.git
cd containerlab
./run-clab

Run the above commands to download the sflow-rt/containerlab GitHub project and run Containerlab on a system with Docker installed. Docker Desktop is a conventient way to run the labs on a laptop.

containerlab deploy -t rocev2.yml

Start the 3 stage leaf and spine emulation.

The initial launch may take a couple of minutes as the container images are downloaded for the first time. Once the images are downloaded, the topology deploys in a few seconds.

./topo.py clab-rocev2

Run the command above to send the topology to the AI Metrics application and connect to http://localhost:8008/app/ai-metrics/html/ to access the dashboard shown at the top of this article.

docker exec -it clab-rocev2-h1 hping3 exec rocev2.tcl 172.16.2.2 10000 500 100

Run the command above to simulate RoCEv2 traffic between h1 and h2 during a training run. The run consists of 100 cycles, each cycle involves exchange of 10000 packets followed by a 500 millisecond pause (similuating GPU compute time). You should immediately see charts updating in the dashboard.

Clicking on the wave symbol in the top right of the Period chart pulls up a fast updating Periodicity chart that shows how sub-second variability in the traffic due to the exchange / compute cycles is used to compute the period shown in the dashboard, in this case a period of just under 0.7 seconds.

Connect to the included sflow-rt/trace-flow application, http://localhost:8008/app/trace-flow/html/

suffix:stack:.:1=ibbt

Enter the filter above and press Submit. The instant a RoCEv2 flow is generated, its path should be shown. See Defining Flows for information on traffic filters.

docker exec -it clab-rocev2-leaf1 vtysh

The routers in this topology run the open source FRRouting daemon used in data center switches with NVIDIA Cumulus Linux or SONiC network operating systems. Type the command above to access the CLI on the leaf1 switch.

leaf1# show running-config

For example, type the above command in the leaf1 CLI to see the running configuration.

Building configuration...

Current configuration:
!
frr version 10.2.3_git
frr defaults datacenter
hostname leaf1
log stdout
!
interface eth3
 ip address 172.16.1.1/24
 ipv6 address 2001:172:16:1::1/64
exit
!
router bgp 65001
 bgp bestpath as-path multipath-relax
 bgp bestpath compare-routerid
 neighbor fabric peer-group
 neighbor fabric remote-as external
 neighbor fabric description Internal Fabric Network
 neighbor fabric capability extended-nexthop
 neighbor eth1 interface peer-group fabric
 neighbor eth2 interface peer-group fabric
 !
 address-family ipv4 unicast
  redistribute connected route-map HOST_ROUTES
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected route-map HOST_ROUTES
  neighbor fabric activate
 exit-address-family
exit
!
route-map HOST_ROUTES permit 10
 match interface eth3
exit
!
ip nht resolve-via-default
!
end

The switch is using BGP to establish equal cost multi-path (ECMP) routes across the fabric - this is very similar to a configuration you would expect to find in a production network. Containerlab provides a great environment to experiment with topologies and configuration options before putting them in production.

The results from this containerlab project are very similar to those observed in production networks, see Comparing AI / ML activity from two production networks. Follow the instruction in AI Metrics with Prometheus and Grafana to deploy this solution to monitor production GPU cluster traffic.

AI Metrics with Grafana Cloud

The Grafana AI Metrics dashboard shown above tracks performance metrics for AI/ML RoCEv2 network traffic, for example, large scale CUDA compute tasks using NVIDIA Collective Communication Library (NCCL) operations for inter-GPU communications: AllReduce, Broadcast, Reduce, AllGather, and ReduceScatter.

The metrics include:

• Total Traffic Total traffic entering fabric
• Operations Total RoCEv2 operations broken out by type
• Core Link Traffic Histogram of load on fabric links
• Edge Link Traffic Histogram of load on access ports
• RDMA Operations Total RDMA operations
• RDMA Bytes Average RDMA operation size
• Credits Average number of credits in RoCEv2 acknowledgements
• Period Detected period of compute / exchange activity on fabric (in this case just over 0.5 seconds)
• Congestion Total ECN / CNP congestion messages
• Errors Total ingress / egress errors
• Discards Total ingress / egress discards
• Drop Reasons Packet drop reasons

AI Metrics with Prometheus and Grafana describes how to stand up an analytics stack with Prometheus and Grafana to track performance metrics for an AI/ML GPU cluster. This article shows how to integrate with Prometheus and Grafana hosted in the cloud, Grafana Cloud, instead of running the services locally.

Note: Grafana Cloud has a free service tier that can be used to test this example.

Save the following compose.yml file on a system running Docker.

configs:
  config.alloy:
    content: |
      prometheus.scrape "prometheus" {
        targets = [{
          __address__ = "sflow-rt:8008",
        }]
        forward_to   = [prometheus.remote_write.grafanacloud.receiver]
        metrics_path = "/app/ai-metrics/scripts/metrics.js/prometheus/txt"
        scrape_interval = "10s"
      }

      prometheus.remote_write "grafanacloud" {
        endpoint {
          url  = "https://<Your Grafana Cloud Prometheus Instance>/api/prom/push"
          basic_auth {
            username = "<Your Grafana.com User ID>"
            password = "<Your Grafana.com API Token>"
          }
        }
      }

networks:

  monitoring:
    driver: bridge

services:

  sflow-rt:
    image: sflow/ai-metrics
    container_name: sflow-rt
    restart: unless-stopped
    ports:
      - '6343:6343/udp'
      - '8008:8008'
    networks:
      - monitoring

  alloy:
    image: grafana/alloy
    container_name: alloy
    restart: unless-stopped
    configs:
      - source: config.alloy
        target: /etc/alloy/config.alloy
    depends_on:
      - sflow-rt
    networks:
      - monitoring

Find the settings needed to upload metrics to Prometheus by clicking on the Send Metrics button in your Grafana Cloud account.

Edit the highlighted prometheus.remote_write endpoint settings (url, username and password) to match those provided by Grafana Cloud.

docker compose up -d

Run the command above to start streaming metrics to Grafana Cloud. Click on the Grafana Launch button to access Grafana and add the AI Metrics dashboard (ID: 23255).

Enable sFlow on all switches in the cluster (leaf and spine) using the recommeded settings. Enable sFlow dropped packet notifications to populate the drop reasons metric, see Dropped packet notifications with Arista Networks, NVIDIA Cumulus Linux 5.11 for AI / ML and Dropped packet notifications with Cisco 8000 Series Routers for examples.

Note: Tuning Performance describes how to optimize settings for very large clusters.

Industry standard sFlow telemetry is uniquely suited monitoring AI workloads. The sFlow agents leverage instrumentation built into switch ASICs to stream randomly sampled packet headers and metadata in real-time. Sampling provides a scaleable method of monitoring the large numbers of 400G/800G links found in AI fabrics. Export of packet headers allows the sFlow collector to decode the InfiniBand Base Transport headers to extract operations and RDMA metrics. The Dropped Packet extension uses Mirror-on-Drop (MoD) / What Just Happened (WJH) capabilities in the ASIC to include packet header, location, and reason for EVERY dropped packet in the fabric.

Talk to your switch vendor about their plans to support the Transit delay and queueing extension. This extension provides visibility into queue depth and switch transit delay using instrumentation built into the ASIC.

A network topology is required to generate the analytics, see Topology for a description of the JSON file and instructions for generating topologies from Graphvis DOT format, NVIDIA NetQ, Arista eAPI, and NetBox.

Use the Topology Status dashboard to verify that the topology is consistent with the sFlow telemetry and fully monitored. The Locate tab can be used to locate network addresses to access switch ports.

Note: If any gauges indicate an error, click on the gauge to get specific details.

Congratulations! The configuration is now complete and you should see charts above in AI Metric application Traffic tab. In addition, the AI Metrics Grafana dashboard at the top of this page should start to populate with data.

Flow metrics with Prometheus and Grafana and Dropped packet metrics with Prometheus and Grafana describe how to define additional flow-based metrics to incorporate in Grafana dashboards.

Getting Started provides an introductions to sFlow-RT, describes how to browse metrics and traffic flows using tools included in the Docker image, and links to information on creating applications using sFlow-RT APIs.

AI Metrics with Prometheus and Grafana

The Grafana AI Metrics dashboard shown above tracks performance metrics for AI/ML RoCEv2 network traffic, for example, large scale CUDA compute tasks using NVIDIA Collective Communication Library (NCCL) operations for inter-GPU communications: AllReduce, Broadcast, Reduce, AllGather, and ReduceScatter.

The metrics include:

• Total Traffic Total traffic entering fabric
• Operations Total RoCEv2 operations broken out by type
• Core Link Traffic Histogram of load on fabric links
• Edge Link Traffic Histogram of load on access ports
• RDMA Operations Total RDMA operations
• RDMA Bytes Average RDMA operation size
• Credits Average number of credits in RoCEv2 acknowledgements
• Period Detected period of compute / exchange activity on fabric (in this case just over 0.5 seconds)
• Congestion Total ECN / CNP congestion messages
• Errors Total ingress / egress errors
• Discards Total ingress / egress discards
• Drop Reasons Packet drop reasons

This article gives step-by-step instructions to set up the dashboard in a production environment.

git clone https://github.com/sflow-rt/prometheus-grafana.git
sed -i -e 's/prometheus/ai-metrics/g' prometheus-grafana/env_vars
./prometheus-grafana/start.sh

The easiest way to get started is to use Docker, see Deploy real-time network dashboards using Docker compose, and deploy the sflow/ai-metrics image bundling the AI Metrics application to generate metrics.

scrape_configs:
  - job_name: 'sflow-rt-ai-metrics'
    metrics_path: /app/ai-metrics/scripts/metrics.js/prometheus/txt
    scheme: http
    static_configs:
      - targets: [ 'sflow-rt:8008' ]

Follow the directions in Deploy real-time network dashboards using Docker compose to add the above Prometheus scrape task to retrieve the metrics and add the Grafana AI Metrics dashboard (ID: 23255).

Enable sFlow on all switches in the cluster (leaf and spine) using the recommeded settings. Enable sFlow dropped packet notifications to populate the drop reasons metric, see Dropped packet notifications with Arista Networks, NVIDIA Cumulus Linux 5.11 for AI / ML and Dropped packet notifications with Cisco 8000 Series Routers for examples.

Note: Tuning Performance describes how to optimize settings for very large clusters.

Industry standard sFlow telemetry is uniquely suited monitoring AI workloads. The sFlow agents leverage instrumentation built into switch ASICs to stream randomly sampled packet headers and metadata in real-time. Sampling provides a scaleable method of monitoring the large numbers of 400G/800G links found in AI fabrics. Export of packet headers allows the sFlow collector to decode the InfiniBand Base Transport headers to extract operations and RDMA metrics. The Dropped Packet extension uses Mirror-on-Drop (MoD) / What Just Happened (WJH) capabilities in the ASIC to include packet header, location, and reason for EVERY dropped packet in the fabric.

Talk to your switch vendor about their plans to support the Transit delay and queueing extension. This extension provides visibility into queue depth and switch transit delay using instrumentation built into the ASIC.

A network topology is required to generate the analytics, see Topology for a description of the JSON file and instructions for generating topologies from Graphvis DOT format, NVIDIA NetQ, Arista eAPI, and NetBox.

Use the Topology Status dashboard to verify that the topology is consistent with the sFlow telemetry and fully monitored. The Locate tab can be used to locate network addresses to access switch ports.

Note: If any gauges indicate an error, click on the gauge to get specific details.

Congratulations! The configuration is now complete and you should see charts above in AI Metric application Traffic tab. In addition, the AI Metrics Grafana dashboard at the top of this page should start to populate with data.

Flow metrics with Prometheus and Grafana and Dropped packet metrics with Prometheus and Grafana describe how to define additional flow-based metrics to incorporate in Grafana dashboards.

Getting Started provides an introductions to sFlow-RT, describes how to browse metrics and traffic flows using tools included in the Docker image, and links to information on creating applications using sFlow-RT APIs.

Emulating congestion with Containerlab

The Containerlab dashboard above shows variation in throughput in a leaf and spine network due to large "Elephant" flow collisions in an emulated network, see Leaf and spine traffic engineering using segment routing and SDN for a demonstration of the issue using physical switches.

This article describes the steps needed to emulate realistic network performance problems using Containerlab. First, using the FRRouting (FRR) open source router to build the topology provides a lightweight, high performance, routing implementation that can be used to efficiently emulate large numbers of routers using the native Linux dataplane for packet forwarding. Second, the containerlab tools netem set command can be used to introduce packet loss, delay, jitter, or restrict bandwidth of ports.

The netem tool makes use of the Linux tc (traffic control) module. Unfortunately, if you are using Docker desktop, the minimal virtual machine used to run containers does not include the tc module.

multipass launch docker

Instead, use Multipass as a convenient way to create and start an Ubuntu virtual machine with Docker support on your laptop. If you are already on a Linux system with Docker installed, skip forward to the git clone step.

multipass ls

List the multipass virtual machines.

Name                    State             IPv4             Image
docker                  Running           192.168.65.3     Ubuntu 22.04 LTS
                                          172.17.0.1

Make a note of the IP address(es) of the docker virtual machine.

multipass shell docker

Run a shell inside the docker virtual machine.

git clone https://github.com/sflow-rt/containerlab.git

Install sflow-rt/containerlab project.

cd containerlab
./run-clab

Run Containerlab using Docker.

In this example we will be using the 3 Stage Close Topology shown above.

env SAMPLING=10 containerlab deploy -t clos3.yml

Start a leaf and spine topology emulation, but use a sampling rate of 1-in-10 rather than the default of 1-in-1000. See Large flow detection for a discussion of scaling sampling rates with link speed to get consistent results between the emulation and a physical network.

./bw.py clab-clos3

Rate limit the links in the topology to 10Mbps.

./topo.py clab-clos3

Post the topology to the sFlow-RT real-time analytics container. Access the Containerlab Dashboard shown at the top of this page using a web browser to connect to http://192.168.65.3:8008/ (where 192.168.65.3 is the IP address of the docker container noted earlier).

docker exec -it clab-clos3-h1 iperf3 -c 172.16.2.2 --parallel 2

Run a series of iperf3 tests to create pairs of large flows between h1 and h2. When the flows take different paths across the fabric the total available bandwidth is 20mbps. If the flows hash onto the same path, then they share 10mbps bandwidth and the throughput is halved.

This example demonstrates that Containerlab is not restricted to emulating and validating configurations, but can also be used to emulate performance issues. In this example, the effects of large flow collisions are relevant to the performance of data center fabrics handling AI/ML workloads where large flow collisions can significantly limit performance, see RoCE networks for distributed AI training at scale.

Dropped packet metrics with Prometheus and Grafana

Dropped packets due to black hole routes, buffer exhaustion, expired TTLs, MTU mismatches, etc. can result in insidious connection failures that are time consuming and difficult to diagnose. Dropped packet notifications with Arista Networks, VyOS dropped packet notifications and Using sFlow to monitor dropped packets describe implementations of the sFlow Dropped Packet Notification Structures extension for Arista Networks switches, VyOS routers, and Linux servers respectively, providing end to end visibility into packet drop events (including switch port, drop reason and packet header for each dropped packet).

Flow metrics with Prometheus and Grafana describes how define flow metrics and create dashboards to trend the flow metrics over time. This article describes how the same setup can be used to define and trend metrics based on dropped packet notifications.

  - job_name: sflow-rt-drops
    metrics_path: /app/prometheus/scripts/export.js/flows/ALL/txt
    static_configs:
      - targets: ['sflow-rt:8008']
    params:
      metric: ['dropped_packets']
      key:
        - 'node:inputifindex'
        - 'ifname:inputifindex'
        - 'reason'
        - 'stack'
        - 'macsource'
        - 'macdestination'
        - 'null:vlan:untagged'
        - 'null:[or:ipsource:ip6source]:none'
        - 'null:[or:ipdestination:ip6destination]:none'
        - 'null:[or:icmptype:icmp6type:ipprotocol:ip6nexthdr]:none'
      label:
        - 'switch'
        - 'port'
        - 'reason'
        - 'stack'
        - 'macsource'
        - 'macdestination'
        - 'vlan'
        - 'src'
        - 'dst'
        - 'protocol'
      value: ['frames']
      dropped: ['true']
      maxFlows: ['20']
      minValue: ['0.001']

The Prometheus scrape configuration above is used to keep track of drop notifications. The highlighed dropped setting is used to select drop notifications for the metric (the default dropped:['false'] creates flow metrics based packet samples and is used to trend normal traffic).

Deploy real-time network dashboards using Docker compose is the simplest way to deploy an sFlow-RT, Prometheus, and Grafana stack with some basic dashboards. Install sFlow-RT Dropped Packets dashboard, code 21721, in Grafana to see the dashboard shown at the top of this page, displaying Drop Locations, Drop Reasons and Dropped Packet Details.

Raspberry Pi 5 network emulation with Containerlab

The GitHub sflow-rt/containerlab project contains example network topologies for the Containerlab network emulation tool that demonstrate real-time streaming telemetry in realistic data center topologies and network configurations. The examples use the same FRRouting (FRR) engine that is part of SONiC, NVIDIA Cumulus Linux, and DENT network operating systems. Containerlab can be used to experiment before deploying solutions into production. Examples include: tracing ECMP flows in leaf and spine topologies, EVPN visibility, and automated DDoS mitigation using BGP Flowspec and RTBH controls.

Raspberry Pi 5 real-time network analytics describes how to install Docker on a Raspberry Pi 5.

docker run hello-world

Run the hello-world container to verify that Docker in properly installed and running before proceeding.

git clone https://github.com/sflow-rt/containerlab.git

Download the sflow-rt/containerlab project from GitHub.

cd containerlab
./run-clab

Start Containerlab.

containerlab deploy -t clos5.yml

Start the 5 stage leaf and spine topology shown at the top of this page. The initial launch may take a couple of minutes as the container images are downloaded for the first time. Once the images are downloaded, the topology deploys in around 10 seconds.

./topo.py clab-clos5

Push the topology to the sFlow-RT analytics software.

An instance of the sFlow-RT real-time analytics engine receives industry standard sFlow telemetry from all the switches in the network. All of the switches in the topology are configured to send sFlow to the sFlow-RT instance. In this case, Containerlab is running the pre-built sflow/clab-sflow-rt image which packages sFlow-RT with useful applications for exploring the data.

Connect to the web interface on port 8008. The sFlow-RT dashboard verifies that telemetry is being received from 10 agents (the 10 switches in the Clos fabric). See the sFlow-RT Quickstart guide for more information.

The Containerlab Dashboard (click on sFlow-RT Apps tab and containerlab-dashboard button) shows real-time dashboard displaying up to the second traffic.

docker exec -it clab-clos5-h1 iperf3 -c 172.16.4.2

Each of the hosts in the network has an iperf3 server, so running the above command will test bandwidth between h1 and h4.

docker exec -it clab-clos5-h1 iperf3 -c 2001:172:16:4::2

Generate a large IPv6 flow between h1 and h4. The traffic flows should immediately appear in the Top Flows chart. You can check the accuracy by comparing the values reported by iperf3 with those shown in the chart.

Click on the Topology tab to see a real-time weathermap of traffic flowing over the topology. See how repeated iperf3 tests take different ECMP (equal-cost multi-path) routes across the network.

docker exec -it clab-clos5-leaf1 vtysh

Linux with open source routing software (FRRouting) is an accessible alternative to vendor routing stacks (no registration / license required, no restriction on copying means you can share images on Docker Hub, no need for virtual machines). FRRouting is popular in production network operating systems (e.g. Cumulus Linux, SONiC, DENT, etc.) and the VTY shell provides an industry standard CLI for configuration, so labs built around FRR allow realistic network configurations to be explored.

docker exec -it clab-clos5-leaf1 vtysh -c "show running-config"

Use vtysh to show the running configuration on leaf1.

containerlab destroy -t clos5.yml

When you are finished, run the above command to stop the containers and free the resources associated with the emulation. Try out other topologies from the project to explore topics such as DDoS mitigation, BGP Flowspec, and EVPN.

Note: If you are building your own topologies, the Raspberry Pi 5 8G can comfortably handle topologies with up to 50 FRR/Alpine Linux nodes.

Getting Started provides an introduction to sFlow-RT analytics and APIs. Containerlab provides a useful environment for developing and testing monitoring applications for sFlow-RT before moving them into production.

Moving monitoring solutions from Containerlab to production is straightforward since sFlow is widely implemented in datacenter equipment from vendors including: A10, Arista, Aruba, Cisco, Edge-Core, Extreme, Huawei, Juniper, NEC, Netgear, Nokia, NVIDIA, Quanta, and ZTE. In addition, the open source Host sFlow agent makes it easy to extend visibility beyond the physical network into the compute infrastructure.

Raspberry Pi 5 real-time network analytics describes how to deploy an sFlow-RT, Prometheus, and Grafana monitoring stack to monitor live network traffic.

Raspberry Pi 5 real-time network analytics

CanaKit Raspberry Pi 5 Starter Kit - Aluminum

This article describes how build an inexpensive Raspberry Pi 5 based server for real-time flow analytics using industry standard sFlow streaming telemetry. Support for sFlow is widely implemented in datacenter equipment from vendors including: A10, Arista, Aruba, Cisco, Edge-Core, Extreme, Huawei, Juniper, NEC, Netgear, Nokia, NVIDIA, Quanta, and ZTE.

In this example, we will use an 8G Raspberry Pi 5 running Raspberry Pi OS Lite (64-bit).  The easiest way to format a memory card and install the operating system is to use the Raspberry Pi Imager (shown above).

Click on EDIT SETTINGS button to customize the installation.

Set a hostname, username, and password.

Click on the SERVICES tab and select Enable SSH.  Click SAVE to save the settings and then YES to apply the settings and create a bootable micro SD card. These initial settings allow the Rasberry Pi to be accessed over the network without having to attach a screen, keyboard, and mouse.

ssh pp@192.168.4.170

Use ssh to log into Raspberry Pi (having installled the micro SD card).

sudo apt-get update && sudo apt-get -y upgrade

Update packages and OS to latest version.

curl -sSL https://get.docker.com | sh

Install Docker.

sudo usermod -aG docker $USER

Give permission to run Docker without sudo command. Exit ssh session and log in again to pick up the new settings.

docker run hello-world

Run the hello-world container to verify that docker in properly installed and running.

git clone https://github.com/sflow-rt/prometheus-grafana.git
cd prometheus-grafana
./start.sh

Start sFlow-RT, Prometheus, and Grafana using Docker compose.

Configure sFlow Agents embedded in switches, routers and servers to stream sFlow telemetry to the Raspberry Pi. The sFlow-RT Getting Started guide shows how to verify that sFlow is being received and includes tools flow and counter based analytics.

For example, the Flow Browser application lets you list attributes of network traffic that you are interested in and trend top flows with the attributes in real-time (up to the second). Defining Flows describes the flow analytics capability of sFlow-RT that can be explored.

Deploy real-time network dashboards using Docker compose describes how to configure Prometheus and Grafana to capture time series data and create custom dashboards.

The Raspberry Pi 5 is surprisingly capable, this pocket-sized server can easily monitor thousands of high speed (100G+) links, providing up to the second visibility into network flows. In this example, sFlow telemetry from 100 switches, each with 48 active 100G ports, was easily handled by the Raspberry Pi 5. Performance of the Prometheus database is likely to be the limiting factor given the relatively slow disk performance of the micro SD card, but could be improved adding an M.2 PCIe disk.

Internet eXchange Provider (IXP) Metrics

IXP Metrics is available on Github. The application provides real-time monitoring of traffic between members of an Internet eXchange Provider (IXP) network.

This article will use Arista switches as an example to illustrate the steps needed to deploy the monitoring solution, however, these steps should work for other network equipment vendors (provided you modify the vendor specific elements in this example).

git clone https://github.com/sflow-rt/prometheus-grafana.git
sed -i -e 's/prometheus/ixp-metrics/g' prometheus-grafana/env_vars
./prometheus-grafana/start.sh

The easiest way to get started is to use Docker, see Deploy real-time network dashboards using Docker compose, and deploy the sflow/ixp-metrics image bundling the IXP Metrics application.

scrape_configs:
  - job_name: sflow-rt-ixp-metrics
    metrics_path: /app/ixp-metrics/scripts/metrics.js/prometheus/txt
    static_configs:
    - targets: ['sflow-rt:8008']

Follow the directions in the article to add a Prometheus scrape task to retrieve the metrics.

sflow source-interface management 1
sflow destination 10.0.0.50
sflow polling-interval 20
sflow sample 50000
sflow run

Enable sFlow on all exchange switches, directing sFlow telemetry to the Docker host (in this case 10.0.0.50).

Use the sFlow-RT Status page to confirm that sFlow is being received from the switches. In this case 286 sFlow datagrams per second are being received from 9 switches.

The IX-F Member Export JSON Schema V1.0 is used to identify exchange members and their assigned MAC addresses. Upload the member data to the IXP Metrics Settings tab. Additional tabs are provided to locate members and MAC addresses to switch ports, query for unauthorized traffic, see real-time charts, etc.

Upload an sFlow-RT Topology. In this example, Arista eAPI can be used to query Arista switches and discover the network topology.

Use the Topology Status page to verify that sFlow telemetry is being received for all the switches and links in the topology.

sFlow-RT IXP Overall Traffic dashboard (ID: 19706) shows overall traffic in and out of exchange, breakdown of IPv4, IPv6 and ARP traffic, packet size distribution, and total number of BGP peering connections across exchange.

sFlow-RT IXP Member Traffic Top N dashboard (ID: 19707) shows peering traffic. Select a member and see trends of traffic to / from other members of the exchange.

sFlow-RT IXP Traffic Matrix dashboard (ID: 19708) displays a grid view of the traffic exchanged between members across the exchange.

Grafana Network Weathermap describes how to construct a real-time dashboard showing network topology and link utilizations.

Support for sFlow is a standard in switches used to construct Internet Exchanges. The combination of Docker, sFlow-RT, Prometheus, and Grafana provide a scaleable, cost effective, and flexible method of monitoring traffic and generating real-time dashboards.