Transit delay and queueing

The recently finalized sFlow Transit Delay Structures extension provides visibility into the performance of packet forwarding in a switch or router using the industry standard sFlow protocol.

The diagram provides a logical representation of packet forwarding. A packet is received at an Ingress Port, the packet header is examined and a forwarding decision is made to add the packet to one of the queues associated with an Egress Port, finally the packet is removed from the queue and sent out the Egress Port to be received by the next device in the chain.

The time between sending and receiving a packet is the packet's transit delay. The transit delay is affected by the time it takes to make the forwarding decision and the time the packet spends in the queue. Identifying the specific queue selected and the number of bytes already in the queue fills out the set of performance metrics for the forwarding decision. The sFlow Transit Delay Structures extension adds these performance metrics to the metadata associated with each packet sample. 

The following output from sflowtool shows that data contained in a packet sample:

startSample ----------------------
sampleType_tag 0:1
sampleType FLOWSAMPLE
sampleSequenceNo 91159
sourceId 0:2216
meanSkipCount 400
samplePool 36463600
dropEvents 0
inputPort 2215
outputPort 2216
flowBlock_tag 0:1036
extendedType egress_queue
egress_queue_id 7
flowBlock_tag 0:1040
extendedType queue_depth
queue_depth_bytes 11354112
flowBlock_tag 0:1039
extendedType transit_delay
transit_delay_nS 839660224
flowBlock_tag 0:1
flowSampleType HEADER
headerProtocol 1
sampledPacketSize 1446
strippedBytes 4
headerLen 128
headerBytes 98-03-9B-8F-B5-CC-98-03-9B-94-C7-D5-08-00-45-16-05-94-12-C7-00-00-FE-11-B8-43-C0-00-02-02-C6-33-64-02-30-39-D4-31-05-80-D7-1D-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42
dstMAC 98039b8fb5cc
srcMAC 98039b94c7d5
IPSize 1428
ip.tot_len 1428
srcIP 192.0.2.2
dstIP 198.51.100.2
IPProtocol 17
IPTOS 22
IPTTL 254
IPID 50962
UDPSrcPort 12345
UDPDstPort 54321
UDPBytes 1408
endSample   ----------------------

The forwarding performance information is highlighted. The inputPort, outputPort, egress_queue_id, queue_depth_bytes, and transit_delay_nS values describe the performance observed by the sampled packet. The sampled packet header allows performance to be reported by specific hosts, protocols, ports, connections, etc.

Linux 4.11 kernel extends packet sampling support describes the Linux PSAMPLE interface used by the Host sFlow agent to receive packet samples. PSAMPLE has been extended in the Linux 5.13 kernel to add the performance metrics, PSAMPLE_ATTR_OUT_TC (egress queue), PSAMPLE_ATTR_OUT_TC_OCC (egress queue depth), and PSAMPLE_ATTR_LATENCY (transit delay) needed to populate the sFlow Transit Delay Structures and the Host sFlow agent now exports the performance data when it is available. The decoded sFlow record, above, was generated by Host sFlow running on a hardware switch and shows measurements made by the switch ASIC.

PSAMPLE and the Host sFlow agent are becoming the standard for sFlow monitoring of Linux based operating systems such as Cumulus Linux, DENT, and SONiC.  As ASIC vendors include the measurements in their device driver PSAMPLE support, they will automatically be included in the sFlow telemetry.

Support for the new extensions has also been added to the sFlow-RT real-time analytics engine. The open source sFlow-RT Flow Browser application shown in the screen shot above displays a real-time, up to the second, view of traffic based on the packet sample telemetry streaming from network devices (switches, routers, and hosts).

In the chart above, the value being plotted has been changed from Bits per Second and is now displaying flows with the highest transit delay (in nanoseconds). The specific device, ingress port, egress port, and egress queue are also identified. 

In the chart above, queue depth (in bytes) is displayed, showing that the nearly 12 Mbytes queue depth is responsible for the transit delay seen in the previous chart.

If the queue is full and the packet is dropped, the sFlow Dropped Packet Notification Structures extension allows the sFlow agent to report details of the dropped packet. Using sFlow to monitor dropped packets describes how the Host sFlow agent uses the Linux drop_monitor interface to implement the extension.

In the final chart above, the open source sFlow-RT Discard Browser application displays a sequence of packets being dropped by a switch as a host attempts, and fails, to establish a TCP connection. The reason for dropping the packets (an access control list) as well as device and ingress port where the packets were dropped are captured.

Transit delay and dropped packet monitoring leverage advanced instrumentation in the latest generation of network ASICs to provide valuable insight into network performance. Integration with industry standard streaming sFlow telemetry provides real-time network-wide visibility into traffic, performance, and errors.