Telemetry, analytics, and control with sFlow® standard
The talk Seeing Through the RDMA Fog: Monitoring RoCEv2 with sFlow at the recent North American Network Operator's Group (NANOG) conference describes how leveraging industry standard sFlow telemetry from data center switches provides visibility into RDMA activity in AI / ML networks.
Note: Slides are available from the talk link.
Note: AI Metrics with Prometheus and Grafana shows how to set up the monitoring stack.
The following example modifies the ddos-protect application to use sFlow-RT's httpAsync() function to send events to Loki's HTTP API.
var lokiPort = getSystemProperty("ddos_protect.loki.port") || '3100'; var lokiPush = getSystemProperty("ddos_protect.loki.push") || '/loki/api/v1/push'; var lokiHost = getSystemProperty("ddos_protect.loki.host"); function sendEvent(action,attack,target,group,protocol) { if(lokiHost) { var url = 'http://'+lokiHost+':'+lokiPort+lokiPush; lokiEvent = { streams: [ { stream: { service_name: 'ddos-protect' }, values: [[ Date.now()+'000000', action+" "+attack+" "+target+" "+group+" "+protocol, { detected_level: action == 'release' ? 'INFO' : 'WARN', action: action, attack: attack, ip: target, group: group, protocol: protocol } ]] } ] }; httpAsync({ url: url, headers: {'Content-Type':'application/json'}, operation: 'POST', body: JSON.stringify(lokiEvent), success: (response) => { if (204 != response.status) { logWarning("DDoS Loki status " + response.status); } }, error: (error) => { logWarning("DDoS Loki error " + error); } }); } if(syslogHosts.length === 0) return; var msg = {app:'ddos-protect',action:action,attack:attack,ip:target,group:group,protocol:protocol}; syslogHosts.forEach(function(host) { try { syslog(host,syslogPort,syslogFacility,syslogSeverity,msg); } catch(e) { logWarning('DDoS cannot send syslog to ' + host); } }); }The highlighted code extends the existing scripts/ddos.js script to add Loki support.
DDoS protection quickstart guide describes how to set up a DDoS mitigation solution using sFlow-RT.
By default, the dashboard shows the Last 24 Hours of traffic. Explore the data: select Last 30 Days to get a long term view, select Last 5 Minutes to get an up to the second view, click on items in a chart legend to show selected metric, drag to select an interval and zoom in.
The SDSC Expanse cluster live AI/ML metrics dashboard displays real-time metrics for workloads running on the cluster:
AI Metrics with Prometheus and Grafana describes how to quickly set up the monitoring stack for your own AI / ML network using industry standard telemetry from leading switch vendors (Arista, Cisco, Dell, Edge-Core, Juniper, HPE, NVIDIA, SONiC etc.).
The sFlow instrumentation embedded as a standard feature of data center switch hardware from all leading vendors (Arista, Cisco, Dell, Juniper, NVIDIA, etc.) provides a cost effective solution for gaining visibility into UET traffic in large production AI / ML fabrics.
docker run -p 8008:8008 -p 6343:6343/udp sflow/prometheusThe easiest way to get started is to use the pre-built sflow/prometheus Docker image to analyze the sFlow telemetry. The chart at the top of this page shows an up to the second view of UET operations using the included Flow Browser application, see Defining Flows for a list of available UET attributes. Getting Started describes how to set up the sFlow monitoring system.
Flow metrics with Prometheus and Grafana describes how collect custom network traffic flow metrics using the Prometheus time series database and include the metrics in Grafana dashboards. Use the Flow Browser to explore UET flow metrics and then configure a Prometheus scrape task to collect useful operational metrics.
sflow sampling-rate 10000 sflow polling-interval 20 sflow header-bytes 128 sflow direction both sflow drop-monitoring enable sflow enable GigabitEthernet0/8/0 sflow enable GigabitEthernet0/9/0 sflow enable GigabitEthernet0/a/0The above VPP configuration commands enable sFlow monitoring of the VPP dataplane, randomly sampling packets, periodically polling counters, and capturing dropped packets and reason codes. The measurements are send via Linux netlink messages to an instance of the open source Host sFlow agent (hsflowd) which combines the measurements and streams standard sFlow telemetry to a remote collector.
sflow {
collector { ip=192.0.2.1 udpport=6343 }
psample { group=1 egress=on }
dropmon { start=on limit=50 }
vpp { }
}The /etc/hsflowd.conf file above enables the modules needed to receive netlink messages from VPP and send the resulting sFlow telemetry to a collector at 192.0.2.1. See vpp-sflow for detailed instructions.
docker run -p 6343:6343/udp sflow/sflowtoolRun sflowtool on the sFlow collector host to verify verify that the data is being received and to see the information available in the sFlow telemetry stream. The pre-built sflow/sflowtool Docker image is the fastest way to run sflowtool.
The sFlow VPP module delivers the same industry standard network performance monitoring available in switches and routers from leading vendors, including Arista, Cisco, Dell, Juniper, NVIDIA, VyOS, etc. to provide comprehensive, network-wide, visibility.
The metrics include:
The latest version of the AI Metrics dashboard uses industry standard sFlow telemetry from network switches to monitor the number of trimmed packets to use as a congestion metric.
Ultra Ethernet Specification Update describes how the Ultra Ethernet Transport (UET) Protocol has the ability to leverage optional “packet trimming” in network switches, which allows packets to be truncated rather than dropped in the fabric during congestion events. As packet spraying causes reordering, it becomes more complicated to detect loss. Packet trimming gives the receiver and the sender an early explicit indication of congestion, allowing immediate loss recovery in spite of reordering, and is a critical feature in the low-RTT environments where UET is designed to operate.
cumulus@switch:~$ nv set system forwarding packet-trim profile packet-trim-default cumulus@switch:~$ nv config apply
NVIDIA Cumulus Linux release 5.14 for NVIDA Spectrum Ethernet Switches includes support for Packet Trimming. The above command enables packet trimming, sets the DSCP remark value to 11, sets the truncation size to 256 bytes, sets the switch priority to 4, and sets the eligibility to all ports on the switch with traffic class 1, 2, and 3. NVIDA BlueField host adapters respond to trimmed packets to ensure fast congestion recovery.
Instructions for deploying the Grafana AI Metrics dashboard are provided in the articles: AI Metrics, AI Metrics with Prometheus and Grafana and AI Metrics with Grafana Cloud.
The metrics include:
Note: Grafana Cloud has a free service tier that can be used to test this example.
Save the following compose.yml file on a system running Docker.
configs:
config.alloy:
content: |
prometheus.scrape "prometheus" {
targets = [{
__address__ = "sflow-rt:8008",
}]
forward_to = [prometheus.remote_write.grafanacloud.receiver]
metrics_path = "/app/ai-metrics/scripts/metrics.js/prometheus/txt"
scrape_interval = "10s"
}
prometheus.remote_write "grafanacloud" {
endpoint {
url = "https://<Your Grafana Cloud Prometheus Instance>/api/prom/push"
basic_auth {
username = "<Your Grafana.com User ID>"
password = "<Your Grafana.com API Token>"
}
}
}
networks:
monitoring:
driver: bridge
services:
sflow-rt:
image: sflow/ai-metrics
container_name: sflow-rt
restart: unless-stopped
ports:
- '6343:6343/udp'
- '8008:8008'
networks:
- monitoring
alloy:
image: grafana/alloy
container_name: alloy
restart: unless-stopped
configs:
- source: config.alloy
target: /etc/alloy/config.alloy
depends_on:
- sflow-rt
networks:
- monitoring
Find the settings needed to upload metrics to Prometheus by clicking on the Send
Metrics button in your Grafana Cloud account.
docker compose up -dRun the command above to start streaming metrics to Grafana Cloud. Click on the Grafana Launch button to access Grafana and add the AI Metrics dashboard (ID: 23255).
Note: Tuning Performance describes how to optimize settings for very large clusters.
Industry standard sFlow telemetry is uniquely suited monitoring AI workloads. The sFlow agents leverage instrumentation built into switch ASICs to stream randomly sampled packet headers and metadata in real-time. Sampling provides a scaleable method of monitoring the large numbers of 400G/800G links found in AI fabrics. Export of packet headers allows the sFlow collector to decode the InfiniBand Base Transport headers to extract operations and RDMA metrics. The Dropped Packet extension uses Mirror-on-Drop (MoD) / What Just Happened (WJH) capabilities in the ASIC to include packet header, location, and reason for EVERY dropped packet in the fabric.
Talk to your switch vendor about their plans to support the Transit delay and queueing extension. This extension provides visibility into queue depth and switch transit delay using instrumentation built into the ASIC.
Note: If any gauges indicate an error, click on the gauge to get specific details.
Congratulations! The configuration is now complete and you should see charts above in AI Metric application Traffic tab. In addition, the AI Metrics Grafana dashboard at the top of this page should start to populate with data.
Flow metrics with Prometheus and Grafana and Dropped packet metrics with Prometheus and Grafana describe how to define additional flow-based metrics to incorporate in Grafana dashboards.
Getting Started provides an introductions to sFlow-RT, describes how to browse metrics and traffic flows using tools included in the Docker image, and links to information on creating applications using sFlow-RT APIs.
The metrics include:
This article gives step-by-step instructions to set up the dashboard in a production environment.
git clone https://github.com/sflow-rt/prometheus-grafana.git sed -i -e 's/prometheus/ai-metrics/g' prometheus-grafana/env_vars ./prometheus-grafana/start.sh
The easiest way to get started is to use Docker, see Deploy real-time network dashboards using Docker compose, and deploy the sflow/ai-metrics image bundling the AI Metrics application to generate metrics.
scrape_configs:
- job_name: 'sflow-rt-ai-metrics'
metrics_path: /app/ai-metrics/scripts/metrics.js/prometheus/txt
scheme: http
static_configs:
- targets: [ 'sflow-rt:8008' ]
Follow the directions in
Deploy real-time network dashboards using Docker compose
to add the above Prometheus scrape task to retrieve the metrics and add the Grafana
AI Metrics
dashboard (ID: 23255).
Note: Tuning Performance describes how to optimize settings for very large clusters.
Industry standard sFlow telemetry is uniquely suited monitoring AI workloads. The sFlow agents leverage instrumentation built into switch ASICs to stream randomly sampled packet headers and metadata in real-time. Sampling provides a scaleable method of monitoring the large numbers of 400G/800G links found in AI fabrics. Export of packet headers allows the sFlow collector to decode the InfiniBand Base Transport headers to extract operations and RDMA metrics. The Dropped Packet extension uses Mirror-on-Drop (MoD) / What Just Happened (WJH) capabilities in the ASIC to include packet header, location, and reason for EVERY dropped packet in the fabric.
Talk to your switch vendor about their plans to support the Transit delay and queueing extension. This extension provides visibility into queue depth and switch transit delay using instrumentation built into the ASIC.
Note: If any gauges indicate an error, click on the gauge to get specific details.
Congratulations! The configuration is now complete and you should see charts above in AI Metric application Traffic tab. In addition, the AI Metrics Grafana dashboard at the top of this page should start to populate with data.
Flow metrics with Prometheus and Grafana and Dropped packet metrics with Prometheus and Grafana describe how to define additional flow-based metrics to incorporate in Grafana dashboards.
Getting Started provides an introductions to sFlow-RT, describes how to browse metrics and traffic flows using tools included in the Docker image, and links to information on creating applications using sFlow-RT APIs.
sudo tcpdump -i any -s 0 -w sflow.pcap udp port 6343Run the command above on the system you are using to collect sFlow data (if you aren't yet collecting sFlow, see Agents for suggested configuration settings). Type Control-C to end the capture after 5 to 10 minutes. Copy the resulting sflow.pcap file to your laptop.
docker run --rm -it -v $PWD/sflow.pcap:/sflow.pcap sflow/sflowtool \ -r /sflow.pcap -P 1Either compile the latest version of sflowtool or, as shown above, use Docker to run the pre-built sflow/sflowtool image. The -P (Playback) option replays the trace in real-time and displays the contents of each sFlow message. Running sflowtool using Docker provides additional examples, including converting the sFlow messages into JSON format for processing by a Python script.
docker run --rm -it -v $PWD/sflow.pcap:/sflow.pcap sflow/sflowtool \ -r /sflow.pcap -f 192.168.4.198/6343 -P 1The -f (forwarding) option takes an IP address and UDP port number as arguments, in this case the laptop's address, 192.168.4.198, and the standard sFlow port, 6343. Use this option to send the sFlow stream to sFlow analytics software.
The real-time dashboard is a joint InMon / Arista demonstration at The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC24) conference being held this week in Atlanta.
The conference network used in the demonstration, SCinet, is described as the most powerful and advanced network on Earth, connecting the SC community to the world.
The sFlow Packet Drop Monitoring In High Performance Networks dashboard combines telemetry from all the Arista switches in the SCinet network to provide real-time network-wide view of performance. Each of the three charts demonstrate a different type of measurement in the sFlow telemetry stream:
If you have Arista switches in your network, try enabling sFlow to gain insight into network traffic. Dropped packet notifications with Arista Networks, describes how to configure sFlow to include dropped packet notifications. Real-time network visibility is particularly relevant to AI / ML data center networks where congestion and dropped packets can result in serious performance degredation.
Flow metrics with Prometheus and Grafana describes how define flow metrics and create dashboards to trend the flow metrics over time. This article describes how the same setup can be used to define and trend metrics based on dropped packet notifications.
- job_name: sflow-rt-drops
metrics_path: /app/prometheus/scripts/export.js/flows/ALL/txt
static_configs:
- targets: ['sflow-rt:8008']
params:
metric: ['dropped_packets']
key:
- 'node:inputifindex'
- 'ifname:inputifindex'
- 'reason'
- 'stack'
- 'macsource'
- 'macdestination'
- 'null:vlan:untagged'
- 'null:[or:ipsource:ip6source]:none'
- 'null:[or:ipdestination:ip6destination]:none'
- 'null:[or:icmptype:icmp6type:ipprotocol:ip6nexthdr]:none'
label:
- 'switch'
- 'port'
- 'reason'
- 'stack'
- 'macsource'
- 'macdestination'
- 'vlan'
- 'src'
- 'dst'
- 'protocol'
value: ['frames']
dropped: ['true']
maxFlows: ['20']
minValue: ['0.001']
The Prometheus scrape configuration above is used to keep track of drop notifications. The highlighed dropped setting is used to select drop notifications for the metric (the default dropped:['false'] creates flow metrics based packet samples and is used to trend normal traffic).
Deploy real-time network dashboards using Docker compose is the simplest way to deploy an sFlow-RT, Prometheus, and Grafana stack with some basic dashboards. Install sFlow-RT Dropped Packets dashboard, code 21721, in Grafana to see the dashboard shown at the top of this page, displaying Drop Locations, Drop Reasons and Dropped Packet Details.
Visibility into dropped packets is essential for Artificial Intelligence/Machine Learning (AI/ML) workloads, where a single dropped packet can stall large scale computational tasks, idling millions of dollars worth of GPU/CPU resources, and delaying the completion of business critical workloads. Enabling real-time sFlow telemetry provides the observability into traffic flows and packet drops needed to effectively manage these networks.
The availability of the Arista EOS 4.31.4M maintenance release brings sFlow dropped packet monitoring (previously demonstrated using the 4.30.1F feature release - see SC23 Dropped packet visibility demonstration) to production networks, see EOS Life Cycle Policysflow sampling 50000 sflow polling-interval 20 sflow vrf mgmt destination 203.0.113.100 sflow vrf mgmt source-interface Management0 sflow runThe above Arista EOS commands enable sFlow counter polling and packet sampling on all ports, sending the sFlow telemetry to the sFlow analyzer at 203.0.113.100
flow tracking mirror-on-drop
sample limit 100 pps
!
tracker SFLOW
exporter SFLOW
format sflow
collector sflow
local interface Management0
no shutdown
The above commands add sFlow Dropped Packet Notification Structures to the sFlow telemetry feed using Broadcom Mirror on Drop (MoD) instrumentation. Broadcom implements mirror-on-drop in Jericho 2, Trident 3, and Tomahawk 3, or later ASICs.
 |
| CanaKit Raspberry Pi 5 Starter Kit - Aluminum |
ssh pp@192.168.4.170Use ssh to log into Raspberry Pi (having installled the micro SD card).
sudo apt-get update && sudo apt-get -y upgradeUpdate packages and OS to latest version.
curl -sSL https://get.docker.com | shInstall Docker.
sudo usermod -aG docker $USERGive permission to run Docker without sudo command. Exit ssh session and log in again to pick up the new settings.
docker run hello-worldRun the hello-world container to verify that docker in properly installed and running.
git clone https://github.com/sflow-rt/prometheus-grafana.git cd prometheus-grafana ./start.shStart sFlow-RT, Prometheus, and Grafana using Docker compose.
Finally, check out the SC23 Dropped packet visibility demonstration to learn about one of newest developments in sFlow monitoring and see a live demonstration.
The SC23-NRE-026 Standard Packet Drop Monitoring In High Performance Networks dashboard combines telemetry from all the Arista switches in the SCinet network to provide real-time network-wide view of performance. Each of the three charts demonstrate a different type of measurement in the sFlow telemetry stream:
sflow sampling 50000 sflow polling-interval 20 sflow vrf mgmt destination 2001:XXX:XXX:XXXX::XXX sflow vrf mgmt source-interface Management0 sflow extension bgp sflow runThe above Arista EOS commands enable sFlow counter polling and packet sampling on all ports, sending the sFlow telemetry to the sFlow analyzer at 2001:XXX:XXX:XXXX::XXX (IPv6 address masked for privacy).
flow tracking mirror-on-drop
sample limit 100 pps
!
tracker SC23
exporter SC23
format sflow
collector sflow
local interface Management0
no shutdown
The above commands add sFlow Dropped Packet Notification Structures to the sFlow telemetry feed. EOS 4.30.1f on Jericho 2 platforms (e.g. Arista 7804r3 at the core of SCinet diagram) is required since the implementation is based on Broadcom Mirror on Drop (MoD) instrumentation. Broadcom implements mirror-on-drop in Jericho 2, Trident 3, and Tomahawk 3, or later ASICs so it should be possible for Arista to release broad support across products incorporating these ASICs.
Industry standard sFlow telemetry is widely supported by data center switch vendors and provides the scaleable real-time visibility needed to understand and manage traffic in high performance networks. The open source Host sFlow agent extends visibility onto servers to ensure end-to-end visibility.
Visibility into dropped packets is essential for Artificial Intelligence/Machine Learning (AI/ML) workloads, where a single dropped packet can stall large scale computational tasks, idling millions of dollars worth of GPU/CPU resources, and delaying the completion of business critical workloads. Enable real-time sFlow telemetry to provide the observability needed to effectively manage these networks.
IXP Metrics is available on Github. The application provides real-time monitoring of traffic between members of an Internet eXchange Provider (IXP) network.
This article will use Arista switches as an example to illustrate the steps needed to deploy the monitoring solution, however, these steps should work for other network equipment vendors (provided you modify the vendor specific elements in this example).
git clone https://github.com/sflow-rt/prometheus-grafana.git sed -i -e 's/prometheus/ixp-metrics/g' prometheus-grafana/env_vars ./prometheus-grafana/start.sh
The easiest way to get started is to use Docker, see Deploy real-time network dashboards using Docker compose, and deploy the sflow/ixp-metrics image bundling the IXP Metrics application.
scrape_configs:
- job_name: sflow-rt-ixp-metrics
metrics_path: /app/ixp-metrics/scripts/metrics.js/prometheus/txt
static_configs:
- targets: ['sflow-rt:8008']
Follow the directions in the article to add a Prometheus scrape task to retrieve the metrics.
sflow source-interface management 1 sflow destination 10.0.0.50 sflow polling-interval 20 sflow sample 50000 sflow run
Enable sFlow on all exchange switches, directing sFlow telemetry to the Docker host (in this case 10.0.0.50).
Support for sFlow is a standard in switches used to construct Internet Exchanges. The combination of Docker, sFlow-RT, Prometheus, and Grafana provide a scaleable, cost effective, and flexible method of monitoring traffic and generating real-time dashboards.