Real-time visibility at 400 Gigabits/s

The chart above demonstrates real-time, up to the second, flow monitoring on a 400 gigabit per second link. The chart shows that the traffic is composed of four, roughly equal, 100 gigabit per second flows.

The data was gathered from The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC18) being held this week in Dallas. The conference network, SCinet, is described as the fastest and most powerful network in the world.

This year, the SCinet network includes recently announced 400 gigabit switches from Arista networks, see Arista Introduces 400 Gigabit Platforms. Each switch delivers 32 400G ports in a 1U form factor.

NRE-36 University of Southern California network topology for SuperComputing 2018

The switches are part of 400G demonstration network connecting USC, Caltech and StarLight booths. The chart shows traffic on a link connecting the USC and Caltech booths.

Providing the visibility needed to manage large scale high speed networks is a significant challenge. In this example, line rate traffic of 80 million packets per second is being monitored on the 400G port. The maximum packet rate for 64 byte packets on a 400 Gigabit, full duplex, link is approximately 1.2 billion packet per second (600 million in each direction). Monitoring all 32 ports requires a solution that can handle over 38 billion packets per second.

In this case, industry standard sFlow instrumentation built into the Broadcom Tomahawk 3 ASICs in the Arista switches provides line rate visibility. Real-time sFlow telemetry from all ports on all switches in the network stream to a central sFlow analyzer that provides network wide visibility. The overall bandwidth capacity delivered to SC18 exhibitors is 9.322 terabits per second.

The chart was generated using the open source Flow Trend application running on sFlow-RT. The sFlow-RT analytics software takes streaming sFlow telemetry from all the devices in the network, providing real-time visibility to orchestration, DevOps and SDN systems.