Calculations:
- Packet Size (4KB): 4,096 bytes × 8 bits/byte = 32,768 bits
- Line Rate Packet Speed: 400,000,000,000 bits/sec ÷ 32,768 bits = 12,207,031.25 packets per second
- Average Time per Packet: 1 ÷ 12,207,031.25 pps = 81.92 nanoseconds
- Average Sample Interval: 81.92 nanoseconds × 100,000 = 0.008192 seconds (8.192 ms)
- Default 1400 byte sFlow datagrams fills with 6 packet samples (each sample containing 128 bytes of packet header + metadata)
- Time taken to fill the datagram with 6 samples given by the Erlang-6 distribution
- Mean: 49.15ms, Median: 46.45ms, Standard Deviation: 20.07ms
- sFlow agent required to send UDP datagram as soon as buffer fills
- Network transit delay is less than 1ms and can be ignored
- Maximum delay is 1000ms. sFlow agent required to send datagram within 1 second of the first record being added
The net result is that an sFlow analyzer will receive streaming telemetry describing an elephant flow within 100ms of the flow start and on average every 50ms for the duration of the flow. The resulting 120 packet headers per second allows the analyzer to provide detailed information on individual RDMA transfers between Infiniband Queue Pairs (QP).
Varying the sampling rate has a significant impact on response time and variability. The current setting of 1-in-100,000 is a good balance between fast response, low overhead on switch, and a datagram rate that can scale to large numbers of switch ports. For example a 64 port switch with all ports handling line rate RDMA flows would generate only 1280 sFlow datagrams per second. Enabling sFlow on all switches and ports in the AI/ML fabric provides a comprehensive view of end-to-end traffic.
In practice, the 100ms fill time for one port can be treated as an upper bound since more than one switch port is typically active at a time. The combined samples from all the active ports cause the datagram fill time to be significantly reduced. The diagram shows that with five active ports fill time drops below 20ms.
The chart from AI / ML network performance metrics at scale shows the result of combining data from all the links in the fabric: The signals are highly correlated since the AI training compute / exchange cycle is synchronized across all compute nodes in the cluster. Constructive interference from combining data from all the links removes the noise in each individual signal and clearly shows the traffic pattern for the cluster.
