Mininet, ONOS, and segment routing

Leaf and spine traffic engineering using segment routing and SDN and CORD: Open-source spine-leaf Fabric describe a demonstration at the 2015 Open Networking Summit using the ONOS SDN controller and a physical network of 8 switches.

This article will describe how to emulate a leaf and spine network using Mininet and configure the ONOS segment routing application to provide equal cost multi-path (ECMP) routing of flows across the fabric. The Mininet Dashboard application running on the sFlow-RT real-time analytics platform is used to provide visibility into traffic flows across the emulated network.

First, run ONOS using Docker:

docker run --name onos --rm -p 6653:6653 -p 8181:8181 -d onosproject/onos

Use the graphical interface, http://onos:8181, to enable the OpenFlow Provider Suite, Network Config Host Provider, Network Config Link Provider, and Segment Routing applications. The screen shot above shows the resulting set of enabled services.

Next, install sFlow-RT and the Mininet Dashboard application on host with Mininet:

wget https://inmon.com/products/sFlow-RT/sflow-rt.tar.gz
tar -xvzf sflow-rt.tar.gz
./sflow-rt/get-app.sh sflow-rt mininet-dashboard

Start sFlow-RT:

./sflow-rt/start.sh

Download the sr.py script:

wget https://raw.githubusercontent.com/sflow-rt/onos-sr/master/sr.py

Start Mininet:

sudo env ONOS=10.0.0.73 mn --custom sr.py,sflow-rt/extras/sflow.py \
--link tc,bw=10 --topo=sr '--controller=remote,ip=$ONOS,port=6653'

The sr.py script is used to create a leaf and spine topology in Mininet and send the network configuration to the ONOS controller. The sflow.py script enables sFlow monitoring of the switches and sends the network topology to sFlow-RT.

The leaf and spine topology will appear in the ONOS web interface.

The topology will also appear in the Mininet Dashboard application:

Run an iperf test using the Mininet cli:

mininet> iperf h1 h3

The path that the traffic takes is highlighted on the Mininet Dashboard topology:

In this case the traffic flowed between leaf1 and leaf2 via spine1. Since ONOS segment routing uses equal cost multi-path routing, subsequent iperf tests may take the alternative via spine2.

Switch to the Charts tab to see traffic trend charts. In this case, the trend charts show the results of six iperf tests. The Traffic chart shows the top flows and the Topology charts show the busy links and the network diameter.

See Writing Applications for an introduction to programming sFlow-RT's analytics engine. Mininet flow analytics provides a simple example of detecting large (elephant) flows.

Real-time visibility at 400 Gigabits/s

The chart above demonstrates real-time, up to the second, flow monitoring on a 400 gigabit per second link. The chart shows that the traffic is composed of four, roughly equal, 100 gigabit per second flows.

The data was gathered from The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC18) being held this week in Dallas. The conference network, SCinet, is described as the fastest and most powerful network in the world.

This year, the SCinet network includes recently announced 400 gigabit switches from Arista networks, see Arista Introduces 400 Gigabit Platforms. Each switch delivers 32 400G ports in a 1U form factor.

NRE-36 University of Southern California network topology for SuperComputing 2018

The switches are part of 400G demonstration network connecting USC, Caltech and StarLight booths. The chart shows traffic on a link connecting the USC and Caltech booths.

Providing the visibility needed to manage large scale high speed networks is a significant challenge. In this example, line rate traffic of 80 million packets per second is being monitored on the 400G port. The maximum packet rate for 64 byte packets on a 400 Gigabit, full duplex, link is approximately 1.2 billion packet per second (600 million in each direction). Monitoring all 32 ports requires a solution that can handle over 38 billion packets per second.

In this case, industry standard sFlow instrumentation built into the Broadcom Tomahawk 3 ASICs in the Arista switches provides line rate visibility. Real-time sFlow telemetry from all ports on all switches in the network stream to a central sFlow analyzer that provides network wide visibility. The overall bandwidth capacity delivered to SC18 exhibitors is 9.322 terabits per second.

The chart was generated using the open source Flow Trend application running on sFlow-RT. The sFlow-RT analytics software takes streaming sFlow telemetry from all the devices in the network, providing real-time visibility to orchestration, DevOps and SDN systems.