Monitoring at 100 Gigabits/s

Chart 1: Top Connections on a 100 Gigabit link

Chart 1 shows the top connections on a 100 Gigabit Ethernet link monitored at the 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (SC11).

Chart 2: Packet counters on 100 Gigabit link

Chart 2 shows the packet rates on the link, approximately 10 million packets per second ingress and 5 million packets per second egress. The maximum packet rate on a 100 Gigabit, full duplex, link is approximately 300 million packets per second (150 million in each direction) making traffic monitoring an interesting challenge.

The article, Ok, but how much time do I have? discusses some of the challenges in monitoring at 10 Gigabit speeds: at 100 Gigabits the challenge is 10 times greater, requiring that the probe process each packet within 3 nanoseconds to provide wire-speed monitoring of a full-duplex link. Probe vendors (e.g. upcoming EndaceExtreme probe) are working to meet this challenge using custom hardware. However, the costs associated with probes and the added operational complexity of maintaining a probe-based solution is prohibitive for most applications, particularly if large numbers of links need to be monitored.

In this instance, the switch hardware, a Brocade MLXe, includes support for the sFlow traffic monitoring standard. Embedding the instrumentation in the switch hardware delivers continuous, wire-speed, monitoring of all switch ports: the switch has a total of 15.36 Terabits, 4.8 billion packets per second, of routing capacity.

Monitoring using sFlow has minimal overhead and is typically enabled on every interface on every switch to provide network-wide visibility. A central sFlow analyzer continuously monitors all the interfaces and can be queried to generate traffic reports. In this case InMon's Traffic Sentinel was used to monitor the switches in the SC11 network (SCinet) and generate the charts shown in this article.

The sFlow standard is widely supported by switch vendors. Selecting switches with sFlow support when upgrading or building out a new network provides comprehensive visibility into network performance at minimal cost. Retrofitting monitoring with probes is expensive and provides limited coverage.