Detecting incast congestion of RDMA flows

The recently released open source GitHub project, ai-monitor, is a proof of concept demonstrating that real-time sFlow telemetry can detect and characterize incast RoCEv2 RDMA flow collisions between GPUs during AI training.

docker run -p 6343:6343/udp -p 8008:8008 sflow/ai-monitor

The pre-built sflow/ai-monitor image is the quickest way to try out the software. The diagram at the top of this page shows test RDMA flows crossing a 4 leaf / 2 spine test network to generate an incast collision. The flows are displayed using the trace-flow application that is included in the image.

The ai-monitor application uses packet samples to detect when GPU port is oversubscribed:

1. Detect when an leaf switch egress port utilization exceeds 80%
2. Count the number of sources sending traffic to the port - single flows are ignored
3. Trace each of the flows back across the topology to its source GPU

The instant the threshold is crossed, the following incast_congestion event is generated by querying information maintained within the sFlow-RT analytics engine. Real-time elephant flow monitoring in AI / ML fabrics describes how standard sFlow from data center switches streams real-time telemetry that can be used to detect elephant flows with a few milliseconds.

{
 "event": "incast_congestion",
 "agent": "172.16.0.1",
 "ifindex": "1",
 "flows": {
  "merged": 3,
  "full": 3
 },
 "sources": [
  {
   "source": "10.0.2.1",
   "bps": 925688221883.1704
  },
  {
   "source": "10.0.3.1",
   "bps": 870534914611.966
  },
  {
   "source": "10.0.4.1",
   "bps": 784496057037.5735
  }
 ],
 "bps": 321113612931.7887,
 "time": 1793734322720,
 "node": "leaf1",
 "port": "Ethernet1/1",
 "destination": "10.0.1.1",
 "paths": [
  {
   "keys": {
    "ipsource": "10.0.2.1",
    "ipdestination": "10.0.1.1",
    "udpsourceport": "64160",
    "udpdestinationport": "4791",
    "ibbtdestinationqp": "14449313"
   },
   "path": [
    {
     "node": "leaf2",
     "agent": "172.16.0.2",
     "port": "Ethernet1/1",
     "values": [
      64
     ],
     "dataSource": "1",
     "value": 15726107505.399204
    },
    {
     "agent": "172.16.0.6",
     "remotenode": "leaf2",
     "values": [
      63
     ],
     "localnode": "spine2",
     "localport": "Ethernet2/1",
     "remoteport": "Ethernet3/1",
     "linkname": "leaf2-spine2-0",
     "dataSource": "2",
     "value": 15310584095.297842
    },
    {
     "agent": "172.16.0.1",
     "remotenode": "spine2",
     "values": [
      62
     ],
     "localnode": "leaf1",
     "localport": "Ethernet3/1",
     "remoteport": "Ethernet1/1",
     "linkname": "leaf1-spine2-0",
     "dataSource": "3",
     "value": 14456650141.853611
    }
   ]
  },
  {
   "keys": {
    "ipsource": "10.0.3.1",
    "ipdestination": "10.0.1.1",
    "udpsourceport": "51329",
    "udpdestinationport": "4791",
    "ibbtdestinationqp": "8669314"
   },
   "path": [
    {
     "node": "leaf3",
     "agent": "172.16.0.3",
     "port": "Ethernet1/1",
     "values": [
      64
     ],
     "dataSource": "1",
     "value": 14491953189.079569
    },
    {
     "agent": "172.16.0.5",
     "remotenode": "leaf3",
     "values": [
      63
     ],
     "localnode": "spine1",
     "localport": "Ethernet3/1",
     "remoteport": "Ethernet2/1",
     "linkname": "leaf3-spine1-0",
     "dataSource": "3",
     "value": 12347461757.959143
    },
    {
     "agent": "172.16.0.1",
     "remotenode": "spine1",
     "values": [
      62
     ],
     "localnode": "leaf1",
     "localport": "Ethernet2/1",
     "remoteport": "Ethernet1/1",
     "linkname": "leaf1-spine1-0",
     "dataSource": "2",
     "value": 13595310385.61916
    }
   ]
  },
  {
   "keys": {
    "ipsource": "10.0.4.1",
    "ipdestination": "10.0.1.1",
    "udpsourceport": "50140",
    "udpdestinationport": "4791",
    "ibbtdestinationqp": "2638813"
   },
   "path": [
    {
     "node": "leaf4",
     "agent": "172.16.0.4",
     "port": "Ethernet1/1",
     "values": [
      64
     ],
     "dataSource": "1",
     "value": 13670813591.56359
    },
    {
     "agent": "172.16.0.6",
     "remotenode": "leaf4",
     "values": [
      63
     ],
     "localnode": "spine2",
     "localport": "Ethernet4/1",
     "remoteport": "Ethernet3/1",
     "linkname": "leaf4-spine2-0",
     "dataSource": "4",
     "value": 14294256814.497116
    },
    {
     "agent": "172.16.0.1",
     "remotenode": "spine2",
     "values": [
      62
     ],
     "localnode": "leaf1",
     "localport": "Ethernet3/1",
     "remoteport": "Ethernet1/1",
     "linkname": "leaf1-spine2-0",
     "dataSource": "3",
     "value": 12251625078.67275
    }
   ]
  }
 ]
}

The event has three types of information:

1. The switch leaf1 and port Ethernet1/1 experiencing the incast congestion event
2. The sources of the traffic 10.0.2.1, 10.0.3.1 and 10.0.4.1
3. The paths that each QP flow takes across the fabric (sorted by TTL)

This proof of concept demonstrates how real-time analytics can be used to monitor the state of large AI / ML fabrics in real-time using standard instrumentation built into data center switches. Offloading monitoring from hosts to the network hardware frees up resources on the hosts and ensures monitoring doesn't perturb machine learning tasks.

The recent NANOG talk, Monitoring RoCEv2 with sFlow,  describes the latest developments by the sFlow community to address the requirements of AI / ML workloads, including transit delay (in nanoseconds), queue depth, dropped packet notifications, and related upgrades to the open source SONiC network operating system. In addition, sFlow's export of sampled packet headers future proofs measurements to provide visibility into new protocols such as MRC / IPv6 uSID segment routing used in the newest generation of ML supercomputing clusters, OpenAI Supercomputer networking to accelerate large scale AI training.