Topology aware flow analytics with NVIDIA NetQ

NVIDIA Cumulus Linux 5.11 for AI / ML describes how NVIDIA 400/800G Spectrum-X switches combined with the latest Cumulus Linux release deliver enhanced real-time telemetry that is particularly relevant to the AI / machine learning workloads that Spectrum-X switches are designed to handle.

This article shows how to extract Topology from an NVIDIA fabric in order to perform advanced fabric aware analytics, for example: detect flow collisions, trace flow paths, and de-duplicate traffic.

In this example, we will use NVIDIA NetQ, a highly scalable, modern network operations toolset that provides visibility, troubleshooting, and validation of your Cumulus and SONiC fabrics in real time.

netq show lldp json

For example, the NetQ Link Layer Discovery Protocol (LLDP) service simplifies the task of gathering neighbor data from switches in the network, and with the json option, makes the output easy to process with a Python script, for example, lldp-rt.py.

The simplest way to try sFlow-RT is to use the pre-built sflow/topology Docker image that packages sFlow-RT with additional applications that are useful for monitoring network topologies.

docker run -p 6343:6343/udp -p 8008:8008 sflow/topology

Configure Cumulus Linux to steam sFlow telemetry to sFlow-RT on UDP port 6343 (the default for sFlow).

netq show lldp json | ./lldp-rt.py http://sflow-rt:8008/topology/json

The above command puts it all together, taking LLDP data from NetQ, converting it to sFlow-RT format, and posting the fabric topology to the sFlow-RT REST API.

Access the sFlow-RT web interface on port 8008. The Topology application includes a dashboard to verify that all the nodes and links in the topology are fully covered by the sFlow telemetry stream.

Getting Started is a step by step guide to sFlow-RT applications, APIs, and community support.

Emulating congestion with Containerlab

The Containerlab dashboard above shows variation in throughput in a leaf and spine network due to large "Elephant" flow collisions in an emulated network, see Leaf and spine traffic engineering using segment routing and SDN for a demonstration of the issue using physical switches.

This article describes the steps needed to emulate realistic network performance problems using Containerlab. First, using the FRRouting (FRR) open source router to build the topology provides a lightweight, high performance, routing implementation that can be used to efficiently emulate large numbers of routers using the native Linux dataplane for packet forwarding. Second, the containerlab tools netem set command can be used to introduce packet loss, delay, jitter, or restrict bandwidth of ports.

The netem tool makes use of the Linux tc (traffic control) module. Unfortunately, if you are using Docker desktop, the minimal virtual machine used to run containers does not include the tc module.

multipass launch docker

Instead, use Multipass as a convenient way to create and start an Ubuntu virtual machine with Docker support on your laptop. If you are already on a Linux system with Docker installed, skip forward to the git clone step.

multipass ls

List the multipass virtual machines.

Name                    State             IPv4             Image
docker                  Running           192.168.65.3     Ubuntu 22.04 LTS
                                          172.17.0.1

Make a note of the IP address(es) of the docker virtual machine.

multipass shell docker

Run a shell inside the docker virtual machine.

git clone https://github.com/sflow-rt/containerlab.git

Install sflow-rt/containerlab project.

cd containerlab
./run-clab

Run Containerlab using Docker.

In this example we will be using the 3 Stage Close Topology shown above.

env SAMPLING=10 containerlab deploy -t clos3.yml

Start a leaf and spine topology emulation, but use a sampling rate of 1-in-10 rather than the default of 1-in-1000. See Large flow detection for a discussion of scaling sampling rates with link speed to get consistent results between the emulation and a physical network.

./bw.py clab-clos3

Rate limit the links in the topology to 10Mbps.

./topo.py clab-clos3

Post the topology to the sFlow-RT real-time analytics container. Access the Containerlab Dashboard shown at the top of this page using a web browser to connect to http://192.168.65.3:8008/ (where 192.168.65.3 is the IP address of the docker container noted earlier).

docker exec -it clab-clos3-h1 iperf3 -c 172.16.2.2 --parallel 2

Run a series of iperf3 tests to create pairs of large flows between h1 and h2. When the flows take different paths across the fabric the total available bandwidth is 20mbps. If the flows hash onto the same path, then they share 10mbps bandwidth and the throughput is halved.

This example demonstrates that Containerlab is not restricted to emulating and validating configurations, but can also be used to emulate performance issues. In this example, the effects of large flow collisions are relevant to the performance of data center fabrics handling AI/ML workloads where large flow collisions can significantly limit performance, see RoCE networks for distributed AI training at scale.

Raspberry Pi 5 network emulation with Containerlab

The GitHub sflow-rt/containerlab project contains example network topologies for the Containerlab network emulation tool that demonstrate real-time streaming telemetry in realistic data center topologies and network configurations. The examples use the same FRRouting (FRR) engine that is part of SONiC, NVIDIA Cumulus Linux, and DENT network operating systems. Containerlab can be used to experiment before deploying solutions into production. Examples include: tracing ECMP flows in leaf and spine topologies, EVPN visibility, and automated DDoS mitigation using BGP Flowspec and RTBH controls.

Raspberry Pi 5 real-time network analytics describes how to install Docker on a Raspberry Pi 5.

docker run hello-world

Run the hello-world container to verify that Docker in properly installed and running before proceeding.

git clone https://github.com/sflow-rt/containerlab.git

Download the sflow-rt/containerlab project from GitHub.

cd containerlab
./run-clab

Start Containerlab.

containerlab deploy -t clos5.yml

Start the 5 stage leaf and spine topology shown at the top of this page. The initial launch may take a couple of minutes as the container images are downloaded for the first time. Once the images are downloaded, the topology deploys in around 10 seconds.

./topo.py clab-clos5

Push the topology to the sFlow-RT analytics software.

An instance of the sFlow-RT real-time analytics engine receives industry standard sFlow telemetry from all the switches in the network. All of the switches in the topology are configured to send sFlow to the sFlow-RT instance. In this case, Containerlab is running the pre-built sflow/clab-sflow-rt image which packages sFlow-RT with useful applications for exploring the data.

Connect to the web interface on port 8008. The sFlow-RT dashboard verifies that telemetry is being received from 10 agents (the 10 switches in the Clos fabric). See the sFlow-RT Quickstart guide for more information.

The Containerlab Dashboard (click on sFlow-RT Apps tab and containerlab-dashboard button) shows real-time dashboard displaying up to the second traffic.

docker exec -it clab-clos5-h1 iperf3 -c 172.16.4.2

Each of the hosts in the network has an iperf3 server, so running the above command will test bandwidth between h1 and h4.

docker exec -it clab-clos5-h1 iperf3 -c 2001:172:16:4::2

Generate a large IPv6 flow between h1 and h4. The traffic flows should immediately appear in the Top Flows chart. You can check the accuracy by comparing the values reported by iperf3 with those shown in the chart.

Click on the Topology tab to see a real-time weathermap of traffic flowing over the topology. See how repeated iperf3 tests take different ECMP (equal-cost multi-path) routes across the network.

docker exec -it clab-clos5-leaf1 vtysh

Linux with open source routing software (FRRouting) is an accessible alternative to vendor routing stacks (no registration / license required, no restriction on copying means you can share images on Docker Hub, no need for virtual machines). FRRouting is popular in production network operating systems (e.g. Cumulus Linux, SONiC, DENT, etc.) and the VTY shell provides an industry standard CLI for configuration, so labs built around FRR allow realistic network configurations to be explored.

docker exec -it clab-clos5-leaf1 vtysh -c "show running-config"

Use vtysh to show the running configuration on leaf1.

containerlab destroy -t clos5.yml

When you are finished, run the above command to stop the containers and free the resources associated with the emulation. Try out other topologies from the project to explore topics such as DDoS mitigation, BGP Flowspec, and EVPN.

Note: If you are building your own topologies, the Raspberry Pi 5 8G can comfortably handle topologies with up to 50 FRR/Alpine Linux nodes.

Getting Started provides an introduction to sFlow-RT analytics and APIs. Containerlab provides a useful environment for developing and testing monitoring applications for sFlow-RT before moving them into production.

Moving monitoring solutions from Containerlab to production is straightforward since sFlow is widely implemented in datacenter equipment from vendors including: A10, Arista, Aruba, Cisco, Edge-Core, Extreme, Huawei, Juniper, NEC, Netgear, Nokia, NVIDIA, Quanta, and ZTE. In addition, the open source Host sFlow agent makes it easy to extend visibility beyond the physical network into the compute infrastructure.

Raspberry Pi 5 real-time network analytics describes how to deploy an sFlow-RT, Prometheus, and Grafana monitoring stack to monitor live network traffic.

Leaf and spine network emulation on Mac OS M1/M2 systems

The GitHub sflow-rt/containerlab project contains example network topologies for the Containerlab network emulation tool that demonstrate real-time streaming telemetry in realistic data center topologies and network configurations. The examples use the same FRRouting (FRR) engine that is part of SONiC, NVIDIA Cumulus Linux, and DENT network operating systems. Containerlab can be used to experiment before deploying solutions into production. Examples include: tracing ECMP flows in leaf and spine topologies, EVPN visibility, and automated DDoS mitigation using BGP Flowspec and RTBH controls.

The Containerlab project currently has limited support for Mac OS, stating "ARM-based Macs (M1/2) are not supported, and no binaries are generated for this platform. This is mainly due to the lack of network images built for arm64 architecture as of now." However, this argument doesn't apply to the Linux based images used in these examples.

First install Docker Desktop on your Apple silicon based Mac (select the Apple Chip option).

mkdir clab
cd clab
docker run --rm -it --privileged \
  --network host --pid="host" \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /run/netns:/run/netns \
  -v $(pwd):$(pwd) -w $(pwd) \
  sflow/clab bash

Run Containerlab by typing the above commands in a terminal. This command uses a pre-built multi-architecture sflow/clab image. If you are running on an x86 platform, follow the official Containerlab Installation instructions.

git clone https://github.com/sflow-rt/containerlab.git

Download the Containerlab topologies from the sflow-rt/containerlab project.

containerlab deploy -t containerlab/clos5.yml

Start the 5 stage leaf and spine topology shown at the top of this page. The initial launch may take a couple of minutes as the container images are downloaded for the first time. Once the images are downloaded, the topology deploys in around 10 seconds.

An instance of the sFlow-RT real-time analytics engine receives industry standard sFlow telemetry from all the switches in the network. All of the switches in the topology are configured to send sFlow to the sFlow-RT instance. In this case, Containerlab is running the pre-built sflow/prometheus image which packages sFlow-RT with useful applications for exploring the data.

Connect to the web interface, http://localhost:8008. The sFlow-RT dashboard verifies that telemetry is being received from 10 agents (the 10 switches in the Clos fabric). See the sFlow-RT Quickstart guide for more information.

The screen capture shows a real-time view of traffic flowing across the network during a series iperf3 tests. Click on the sFlow-RT Apps menu and select the browse-flows application, or click here for a direct link to a chart with the settings shown above.

docker exec -it clab-clos5-h1 iperf3 -c 172.16.4.2

Each of the hosts in the network has an iperf3 server, so running the above command will test bandwidth between h1 and h4.

docker exec -it clab-clos5-leaf1 vtysh

Linux with open source routing software (FRRouting) is an accessible alternative to vendor routing stacks (no registration / license required, no restriction on copying means you can share images on Docker Hub, no need for virtual machines). FRRouting is popular in production network operating systems (e.g. Cumulus Linux, SONiC, DENT, etc.) and the VTY shell provides an industry standard CLI for configuration, so labs built around FRR allow realistic network configurations to be explored.

containerlab destroy -t containerlab/clos5.yml

When you are finished, run the above command to stop the containers and free the resources associated with the emulation. Try out other topologies from the project to explore topics such as DDoS mitigation, BGP Flowspec, and EVPN.

Moving the monitoring solution from Containerlab to production is straightforward since sFlow is widely implemented in datacenter equipment from vendors including: A10, Arista, Aruba, Cisco, Edge-Core, Extreme, Huawei, Juniper, NEC, Netgear, Nokia, NVIDIA, Quanta, and ZTE. In addition, the open source Host sFlow agent makes it easy to extend visibility beyond the physical network into the compute infrastructure.

Real-time EVPN fabric visibility

Real-time telemetry from a 5 stage Clos fabric describes lightweight emulation of realistic data center switch topologies using Containerlab. This article builds on the example to demonstrate visibility into Ethernet Virtual Private Network (EVPN) traffic as it crosses a routed leaf and spine fabric.

docker run --rm -it --privileged --network host --pid="host" \
  -v /var/run/docker.sock:/var/run/docker.sock -v /run/netns:/run/netns \
  -v ~/clab:/home/clab -w /home/clab \
  ghcr.io/srl-labs/clab bash

Start Containerlab.

curl -O https://raw.githubusercontent.com/sflow-rt/containerlab/master/evpn3.yml

Download the Containerlab topology file.

containerlab deploy -t evpn3.yml

Finally, deploy the topology.

docker exec -it clab-evpn3-leaf1 vtysh -c "show running-config"

See configuration of leaf1 switch.

Building configuration...

Current configuration:
!
frr version 8.1_git
frr defaults datacenter
hostname leaf1
no ipv6 forwarding
log stdout
!
router bgp 65001
 bgp bestpath as-path multipath-relax
 bgp bestpath compare-routerid
 neighbor fabric peer-group
 neighbor fabric remote-as external
 neighbor fabric description Internal Fabric Network
 neighbor fabric capability extended-nexthop
 neighbor eth1 interface peer-group fabric
 neighbor eth2 interface peer-group fabric
 !
 address-family ipv4 unicast
  network 192.168.1.1/32
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor fabric activate
  advertise-all-vni
 exit-address-family
exit
!
ip nht resolve-via-default
!
end

The loopback address on the switch, 192.168.1.1/32, is advertised to neighbors so that the VxLAN tunnel endpoint is known to switches in the fabric. The address-family l2vpn evpn setting exchanges bridge tables across BGP connections so that they operate as a single virtual bridge.

docker exec -it clab-evpn3-h1 ping -c 3 172.16.10.2

Ping h2 from h1

PING 172.16.10.2 (172.16.10.2): 56 data bytes
64 bytes from 172.16.10.2: seq=0 ttl=64 time=0.346 ms
64 bytes from 172.16.10.2: seq=1 ttl=64 time=0.466 ms
64 bytes from 172.16.10.2: seq=2 ttl=64 time=0.152 ms

--- 172.16.10.2 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.152/0.321/0.466 ms

The results verify that there is layer 2 connectivity between the two hosts.

docker exec -it clab-evpn3-leaf1 vtysh -c "show evpn vni"

List the Virtual Network Identifiers (VNIs) on leaf1.

VNI        Type VxLAN IF              # MACs   # ARPs   # Remote VTEPs  Tenant VRF                           
10         L2   vxlan10               2        0        1               default

We can see one virtual network, VNI 10.

docker exec -it clab-evpn3-leaf1 vtysh -c "show evpn mac vni 10"

Show the virtual bridge table for VNI 10 on leaf1.

Number of MACs (local and remote) known for this VNI: 2
Flags: N=sync-neighs, I=local-inactive, P=peer-active, X=peer-proxy
MAC               Type   Flags Intf/Remote ES/VTEP            VLAN  Seq #'s
aa:c1:ab:25:7f:a2 remote       192.168.1.2                          0/0
aa:c1:ab:25:76:ee local        eth3                                 0/0

The MAC address, aa:c1:ab:25:76:ee, is reported as locally attached to port eth3. The MAC address, aa:c1:ab:25:7f:a2, is reported as remotely accessable through the VxLAN tunnel to 192.168.1.2, the loopback address for leaf2.

The screen capture shows a real-time view of traffic flowing across the network during an iperf3 test. Connect to the sFlow-RT Flow Browser application, http://localhost:8008/app/browse-flows/html/, or click here for a direct link to a chart with the settings shown above.

The chart shows VxLAN encapsulated Ethernet packets routed across the leaf and spine fabric. The inner and outer addresses are shown, allowing the flow to be traced end-to-end. See Defining Flows for more information.

docker exec -it clab-evpn3-h1 iperf3 -c 172.16.10.2

Each of the hosts in the network has an iperf3 server, so running the above command will test bandwidth between h1 and h2. The flow should immediately appear in the Flow Browser chart.

containerlab destroy -t evpn3.yml

When you are finished, run the above command to stop the containers and free the resources associated with the emulation.

Moving the monitoring solution from Containerlab to production is straightforward since sFlow is widely implemented in datacenter equipment from vendors including: A10, Arista, Aruba, Cisco, Edge-Core, Extreme, Huawei, Juniper, NEC, Netgear, Nokia, NVIDIA, Quanta, and ZTE.

Topology aware fabric analytics

Real-time telemetry from a 5 stage Clos fabric describes how to emulate and monitor the topology shown above using Containerlab and sFlow-RT. This article extends the example to demonstrate how topology awareness enhances analytics.

docker run --rm -it --privileged --network host --pid="host" \
  -v /var/run/docker.sock:/var/run/docker.sock -v /run/netns:/run/netns \
  -v ~/clab:/home/clab -w /home/clab \
  ghcr.io/srl-labs/clab bash

Start Containerlab.

curl -O https://raw.githubusercontent.com/sflow-rt/containerlab/master/clos5.yml

Download the Containerlab topology file.

sed -i "s/prometheus/topology/g" clos5.yml

Change the sFlow-RT image from sflow/prometheus to sflow/topology in the Containerlab topology. The sflow/topology image packages sFlow-RT with useful applications that combine topology awareness with analytics.

containerlab deploy -t clos5.yml

Deploy the topology.

curl -O https://raw.githubusercontent.com/sflow-rt/containerlab/master/clos5.json

Download the sFlow-RT topology file.

curl -X PUT -H "Content-Type: application/json" -d @clos5.json \
http://localhost:8008/topology/json

Post the topology to sFlow-RT.

Connect to the sFlow-RT Topology application, http://localhost:8008/app/topology/html/. The dashboard confirms that all the links and nodes in the topology are streaming telemetry. There is currently no traffic on the network, so none of the nodes in the topology are sending flow data.

docker exec -it clab-clos5-h1 iperf3 -c 172.16.4.2

Generate traffic. You should see the Nodes No Flows number drop as switches in the traffic path stream flow data. In a production network traffic flows through all the switches and so all switches will be streaming flow telemetry.

Click on the Locate tab and enter the address 172.16.4.2. Click submit to find the location of the address. Topology awareness allows sFlow-RT to examine streaming flow data from all the links in the network and determine switch ports and MAC addresses associated with the point of ingress into the fabric.

Connect to the sFlow-RT Fabric Metrics application, http://localhost:8008/app/fabric-metrics/html/. The Traffic charts identifies each iperf3 test as a large (elephant) flow - see SDN and large flows for a discussion of the importance of large flow visibility. The Busy Links chart reports the number of links in the topology with a utilization exceeding a threshold (in this case 80% utilization).

The Ports tab contains charts showing switch ports with the highest ingress/egress utilization, packet discards, and packet errors.

Connect to the sFlow-RT Trace Flow application,  http://localhost:8008/app/trace-flow/html/.

ipsource=172.16.1.2

Enter a traffic filter and click Submit.

docker exec -it clab-clos5-h1 iperf3 -c 172.16.4.2

Run an iperf3 test and the path the traffic takes across the network topology will be immediately displayed.

Connect to the sFlow-RT Flow Browser application, http://localhost:8008/app/browse-flows/html/index.html?keys=ipsource&value=bps. Run a sequence of iperf3 tests and the traffic will be immediately shown. Defining Flows describes flow attributes that can be added to the query to examine inner / outer tunneled traffic, VxLAN, MPLS, GRE, etc.

Trace Flow demonstrated that flow telemetry is streamed by each switch port along the path from source to destination host to deliver end-to-end visibility. Simply summing the data would overestimate the size of the flow. Adding topology information allows sFlow-RT to intelligently combine data, eliminating duplicate measurements, to provide accurate measurements of traffic crossing the fabric.

containerlab destroy -t clos5.yml

When you are finished, run the above command to stop the containers and free the resources associated with the emulation.

Moving the monitoring solution from Containerlab to production is straightforward since sFlow is widely implemented in datacenter equipment from vendors including: A10, Arista, Aruba, Cisco, Edge-Core, Extreme, Huawei, Juniper, NEC, Netgear, Nokia, NVIDIA, Quanta, and ZTE. In addition, the open source Host sFlow agent makes it easy to extend visibility beyond the physical network into the compute infrastructure.

Real-time telemetry from a 5 stage Clos fabric

CONTAINERlab described how to use FRRouting and Host sFlow in a Docker container to emulate switches in a Clos (leaf/spine) fabric. The recently released open source project, https://github.com/sflow-rt/containerlab, simplifies and automates the steps needed to build and monitor topologies.

docker run --rm -it --privileged --network host --pid="host" \
  -v /var/run/docker.sock:/var/run/docker.sock -v /run/netns:/run/netns \
  -v ~/clab:/home/clab -w /home/clab \
  ghcr.io/srl-labs/clab bash

Run the above command to start Containerlab if you already have Docker installed; the ~/clab directory will be created to persist settings. Otherwise, Installation provides detailed instructions for a variety of platforms.

curl -O https://raw.githubusercontent.com/sflow-rt/containerlab/master/clos5.yml

Next, download the topology file for the 5 stage Clos fabric shown at the top of this article.

containerlab deploy -t clos5.yml

Finally, deploy the topology.

Note: The 3 stage Clos topology, clos3.yml, described in the previous article is also available.

The initial launch may take a couple of minutes as the container images are downloaded for the first time. Once the images are downloaded, the topology deploys in around 10 seconds.

An instance of the sFlow-RT real-time analytics engine receives industry standard sFlow telemetry from all the switches in the network. All of the switches in the topology are configured to send sFlow to the sFlow-RT instance. In this case, Containerlab is running the pre-built sflow/prometheus image which packages sFlow-RT with useful applications for exploring the data.

Connect to the web interface, http://localhost:8008. The sFlow-RT dashboard verifies that telemetry is being received from 10 agents (the 10 switches in the Clos fabric). See the sFlow-RT Quickstart guide for more information.

The screen capture shows a real-time view of traffic flowing across the network during a series iperf3 tests. Click on the sFlow-RT Apps menu and select the browse-flows application, or click here for a direct link to a chart with the settings shown above.

docker exec -it clab-clos5-h1 iperf3 -c 172.16.4.2

Each of the hosts in the network has an iperf3 server, so running the above command will test bandwidth between h1 and h4.

docker exec -it clab-clos5-leaf1 vtysh

Linux with open source routing software (FRRouting) is an accessible alternative to vendor routing stacks (no registration / license required, no restriction on copying means you can share images on Docker Hub, no need for virtual machines).  FRRouting is popular in production network operating systems (e.g. Cumulus Linux, SONiC, DENT, etc.) and the VTY shell provides an industry standard CLI for configuration, so labs built around FRR allow realistic network configurations to be explored.

containerlab destroy -t clos5.yml

When you are finished, run the above command to stop the containers and free the resources associated with the emulation.

Moving the monitoring solution from Containerlab to production is straightforward since sFlow is widely implemented in datacenter equipment from vendors including: A10, Arista, Aruba, Cisco, Edge-Core, Extreme, Huawei, Juniper, NEC, Netgear, Nokia, NVIDIA, Quanta, and ZTE. In addition, the open source Host sFlow agent makes it easy to extend visibility beyond the physical network into the compute infrastructure.

Topology

Real-time network and system metrics as a service describes how to use data captured from the network shown above to explore the functionality of sFlow-RT real-time analytics software. This article builds on the previous article to show how knowledge of network topology can be used to enhance analytics, see Topology for documentation.

First, follow the instructions in the previous example and start an instance of sFlow-RT using the captured sFlow.  

curl -O https://raw.githubusercontent.com/sflow-rt/fabric-view/master/demo/topology.json

Then, download the topology file for the example.

curl -X PUT -H "Content-Type: application/json" -d @topology.json \
http://localhost:8008/topology/json

Install the topology using the sFlow-RT REST API.

curl http://localhost:8008/topology/json

Retrieve the topology.

{
 "version": 0,
 "links": {
  "L1": {
   "node2": "spine1",
   "node1": "leaf1",
   "port1": "swp1s0",
   "port2": "swp49"
  },
  "L2": {
   "node2": "spine1",
   "node1": "leaf1",
   "port1": "swp1s1",
   "port2": "swp50"
  },
  "L3": {
   "node2": "spine2",
   "node1": "leaf1",
   "port1": "swp1s2",
   "port2": "swp51"
  },
  "L4": {
   "node2": "spine2",
   "node1": "leaf1",
   "port1": "swp1s3",
   "port2": "swp52"
  },
  "L5": {
   "node2": "spine2",
   "node1": "leaf2",
   "port1": "swp1s0",
   "port2": "swp49"
  },
  "L6": {
   "node2": "spine2",
   "node1": "leaf2",
   "port1": "swp1s1",
   "port2": "swp50"
  },
  "L7": {
   "node2": "spine1",
   "node1": "leaf2",
   "port1": "swp1s2",
   "port2": "swp51"
  },
  "L8": {
   "node2": "spine1",
   "node1": "leaf2",
   "port1": "swp1s3",
   "port2": "swp52"
  }
 }
}

The JSON topology structure contains the eight links connecting the leaf and spine switches in the diagram, identifying the nodes and ports associated with each link.

curl -H "Content-Type:application/json" -X PUT \
--data '{"keys":"ipsource,ipdestination",value:"bytes"}' \
http://localhost:8008/flow/srcdst/json

Now define the srcdst flow metric described in the previous article.

curl "http://localhost:8008/activeflows/TOPOLOGY/srcdst/json?aggMode=edge"

Knowledge of topology opens up additional options when querying for flows. For example, the above command only considers devices that are part of the topology and sums flows entering edge device access ports, i.e. traffic entering the the leaf switches from the servers.

[
 {
  "flowN": 1,
  "value": 248800.14028768288,
  "key": "10.4.3.2,10.4.4.2"
 },
 {
  "flowN": 1,
  "value": 176879.3798722214,
  "key": "10.4.1.2,10.4.2.2"
 },
 {
  "flowN": 1,
  "value": 526.0366052656848,
  "key": "10.4.4.2,10.4.3.2"
 },
 {
  "flowN": 1,
  "value": 375.06686598182193,
  "key": "10.4.2.2,10.4.1.2"
 }
]

The result accurately reports the amount of traffic being exchanged between the servers, discarding duplicate data reported as traffic flows traverse the links between switches.

-Dbrowse-flows.agents=TOPOLOGY -Dbrowse-flows.aggMode=edge

Adding the above arguments to the end of the command line used to start sFlow-RT configures the Flow Browser application to use the topology de-duplication method.

Click on the link below to plot a graph of the top IP Protocols using the browse-flows application (screen capture shown above):

http://localhost:8008/app/browse-flows/html/index.html?keys=ipprotocol&value=bps

Note: No data will be shown until the topology is posted to sFlow-RT.

function print(label,obj) {
  logInfo(label+"="+JSON.stringify(obj));
}

setFlow('protocol',{keys:'ipprotocol',value:'bytes'});

setIntervalHandler(function() {
  print("locate_mac",topologyLocateHostMac('000AF725C062'));
  print("locate_ip",topologyLocateHostIP('10.4.3.2'));
  print("flow_max",activeFlows('ALL','protocol',5,0,'max'));
  print("flow_sum",activeFlows('ALL','protocol',5,0,'sum'));
  print("flow_edge",activeFlows('TOPOLOGY','protocol',5,0,'edge'));
});

The demo.js script shown above uses sFlow-RT's embedded scripting API, see Writing Applications. The script defines the flow called protocol that tracks top IP Protocols and prints out the top flows with different aggregation methods. The script also demonstrates an additional capability made possible when topology is known. The topologyLocateHostMac() and topologyLocateHostIP() function locates an addresses to the edge port connecting them to the network.

-Dscript.file=$PWD/demo.js

Run the script by adding the above argument to the end of the command line used to run sFlow-RT.

2021-01-22T17:08:35-08:00 INFO: locate_mac=[{"ipaddress":"10.4.3.2","node":"leaf1","agent":"192.168.0.11","ifindex":"38","port":"swp32s1","mac":"000AF725C062"}]
2021-01-22T17:08:35-08:00 INFO: locate_ip=[{"ipaddress":"10.4.3.2","node":"leaf1","agent":"192.168.0.11","ifindex":"38","port":"swp32s1","mac":"000AF725C062"}]
2021-01-22T17:08:35-08:00 INFO: flow_max=[{"flowN":16,"agent":"192.168.0.14","value":1208583376.782055,"dataSource":"54","key":"6"}]
2021-01-22T17:08:35-08:00 INFO: flow_sum=[{"flowN":16,"value":6615963204.827695,"key":"6"}]
2021-01-22T17:08:35-08:00 INFO: flow_edge=[{"flowN":4,"value":2104039983.2917378,"key":"6"}]

The output from the script shows that the addresses were located to leaf1 port swp32s1. The flow_max and flow_sum queries don't use the topology and combine data from all 16 data sources (switch ports) that are reporting traffic. The sum mode returns the largest value since traffic is added for every data source. The max mode finds the data source reporting the largest value for the flow and reports that value (agent: 192.168.0.14, dataSource: 54). The edge mode is equivalent to the REST query used earlier.

Mininet is a network emulator that you can run on your laptop in a virtual machine (e.g. using Multipass) that provides a useful platform for building virtual topologies and exploring topology related analytics. Mininet dashboard, Mininet weathermap, and Mininet, ONOS, and segment routing provide examples.

Troubleshooting connectivity problems in leaf and spine fabrics, Black hole detection, Multi-tenant sFlow, Internet Exchange (IX) Metrics, and Leaf and spine traffic engineering using segment routing and SDN describe additional use cases for combining topology with flow analytics.

Ideally the network configuration and topology will be available in a centralized repository that can be queried to generate the information required by sFlow-RT. Alternatively, Link Layer Discovery Protocol (LLDP) data retrieved from network devices can be used to construct the topology. Fabric Visibility, Arista EOS CloudVision, and Fabric visibility with Cumulus Linux provide examples.

Mininet weathermap

Mininet dashboard is a real-time dashboard displaying traffic information from Mininet virtual networks. The screen capture demonstrates the real-time network weather map capability that was recently added to the dashboard. The torus topology is displayed and link widths are updated every second to reflect traffic. In this example a large flow between switches s1x1 and s3x3 is routed via s1x3.

The network was created using the following Mininet command:

sudo mn --custom=sflow-rt/extras/sflow.py --link tc,bw=10 \
--topo torus,3,3 --switch ovsbr,stp=1 --test iperf

In the screen capture above you can clearly see the large flow traversing switches, s4, s3, s2, s1, s9, s13, and s15 in a tree topology. The network was created using the following command:

sudo mn --custom sflow-rt/extras/sflow.py --link tc,bw=10 \
--topo tree,depth=4,fanout=2 --test iperf

The screen capture above shows a large flow traversing switches s1, s2, s3, and s4 in a linear topology. The network was created using the following command:

sudo mn --custom sflow-rt/extras/sflow.py --link tc,bw=10 \
--topo linear,4 --test iperf

It's also easy to create Custom Topologies. The following command creates the example custom topology, topo-2sw-2host.py, that ships with Mininet:

sudo mn --custom ~/mininet/custom/topo-2sw-2host.py,sflow-rt/extras/sflow.py \
--link tc,bw=10 --topo mytopo --test iperf

The final screen capture shows all three tests, identifying top flows, busiest switch ports, and showing the diameter of the network topology. This test run demonstrates the efficiency of using Mininet; creating, testing, and destroying three virtual networks, toroid (9 switches, 9 hosts, 27 links), tree (15 switches, 16 hosts, 30 links), linear (4 switches, 4 hosts, 7 links), in a few minutes, on a laptop.

For anyone curious about the technologies underpinning these examples, Mininet is a Python program that uses Open vSwitch to connect lightweight Linux containers representing hosts and switches. Open vSwitch includes industry standard sFlow instrumentation that efficiently streams telemetry to sFlow-RT real-time analytics software running the mininet-dashboard application. Mininet-dashboard uses the vis.js library to create the dynamic network diagrams.

Mininet's support of sFlow provides the same measurement technology used in physical switches, from low cost edge switches to high end chassis routers, including: A10, Aerohive, ALUe, Allied Telesis, Arista, Aruba, Big Switch, Cisco, Cumulus, Dell, D-Link, Edge-Core, Extreme, F5, Fortinet, Huawei, IP Infusion, Juniper, Netgear, OpenSwitch, Pica8, Proxim, Quanta, SMC, ZTE, and ZyXEL. A common measurement protocol makes it easy to move between emulated and physical networks, for example, Large flow detection describes how to scale sFlow configuration settings from emulated 10Mbit/s Mininet networks to 10G, 40G and 100G physical networks, and SC16 live real-time weathermaps uses the technologies described in this article to monitor a large 100Gbit/s network.

WAN optimization using real-time traffic analytics

TATA Consultancy Services white paper, Actionable Intelligence in the SDN Ecosystem: Optimizing Network Traffic through FRSA, demonstrates how real-time traffic analytics and SDN can be combined to perform real-time traffic engineering of large flows across a WAN infrastructure.

The architecture being demonstrated is shown in the diagram (this diagram has been corrected - the diagram in the white paper incorrectly states that sFlow-RT analytics software uses a REST API to poll the nodes in the topology. In fact, the nodes stream telemetry using the widely supported, industry standard, sFlow protocol, providing real-time visibility and scaleability that would be difficult to achieve using polling - see Push vs Pull).

The load balancing application receives real-time notifications of large flows from the sFlow-RT analytics software and programs the SDN Controller (in this case OpenDaylight) to push forwarding rules to the switches to direct the large flows across a specific path. Flow Aware Real-time SDN Analytics (FRSA) provides an overview of the basic ideas behind large flow traffic engineering that inspired this use case.

While OpenDaylight is used in this example, an interesting alternative for this use case would be the ONOS SDN controller running the Segment Routing application. ONOS is specifically designed with carriers in mind and segment routing is a natural fit for the traffic engineering task described in this white paper.

Leaf and spine traffic engineering using segment routing describes a demonstration combining real-time analytics and SDN control in a data center context. The demonstration was part of the recent 2015 Open Networking Summit (ONS) conference Showcase and presented in the talk, CORD: FABRIC An Open-Source Leaf-Spine L3 Clos Fabric, by Saurav Das.

ECMP visibility with Cumulus Linux

Demo: Implementing the Big Data Design Guide in the Cumulus Workbench  is a great demonstration of the power of zero touch provisioning and automation. When the switches and servers boot they automatically pick up their operating systems and configurations for the complex Equal Cost Multi-Path (ECMP) routed network shown in the diagram.

Topology discovery with Cumulus Linux looked at an alternative Multi-Chassis Link Aggregation (MLAG) configuration and shows how to extract the configuration and monitor traffic on the network using sFlow and Fabric View.

The paper Hedera: Dynamic Flow Scheduling for Data Center Networks describes the impact of colliding flows on effective ECMP cross sectional bandwidth. The paper gives an example which demonstrates that effective cross sectional bandwidth can be reduced by a factor of between 20% to 60%, depending on the number of simultaneous flows per host.

This article uses the workbench to demonstrate the effect of large "Elephant" flow collisions on network throughput. The following script running on each of the servers uses the iperf tool to generate pairs of overlapping Elephant flows:

cumulus@server1:~$ while true; do iperf -c 10.4.2.2 -t 20; sleep 20; done
------------------------------------------------------------
Client connecting to 10.4.2.2, TCP port 5001
TCP window size: 1.06 MByte (default)
------------------------------------------------------------
[  3] local 10.4.1.2 port 57234 connected with 10.4.2.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-20.0 sec  21.9 GBytes  9.41 Gbits/sec
------------------------------------------------------------
Client connecting to 10.4.2.2, TCP port 5001
TCP window size: 1.06 MByte (default)
------------------------------------------------------------
[  3] local 10.4.1.2 port 57240 connected with 10.4.2.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-20.0 sec  10.1 GBytes  4.34 Gbits/sec
------------------------------------------------------------
Client connecting to 10.4.2.2, TCP port 5001
TCP window size: 1.06 MByte (default)
------------------------------------------------------------
[  3] local 10.4.1.2 port 57241 connected with 10.4.2.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-20.0 sec  21.9 GBytes  9.41 Gbits/sec
------------------------------------------------------------

The first iperf test achieves a TCP throughput of 9.41 Gbits/sec (the maximum achievable on the 10Gbit/s network in the workbench). However, the second test only achieves a throughput of 4.34 Gbits/sec. How can this result be explained?

The Top Flows table above confirms that two simultaneous elephant flows are being tracked by Fabric View.

The Traffic charts update every second and give a fine grained view of the traffic flows over time. The charts clearly show how iperf flows vary in throughput, with the low throughput runs achieving a throughput of approximately 50% of the network capacity (these results are consistent with 20% to 60% reported in the Hedera paper).

The Performance charts show what is happening. Packets take two hops as they are routed from leaf1 to leaf2 (via spine1 or spine2). Each iperf connection is able to fully utilize the two links to achieve line rate throughput. Comparing the Total Traffic and Busy Spine Links charts shows that peak total throughput of approximately 20Gbits/sec corresponds to interval when 4 spine links are busy. The throughput is halved during intervals when the routes overlap and share 1 or 2 links (shown in gold as Collisions on the Busy Spine Links chart).

Readers might be surprised by the frequency of collisions given the number of links in the network. Packets take two hops to go from leaf1 to leaf2 - routed via spine1 or spine2. In addition, the links between switches are paired, so there are 8 possible two hop paths from leaf1 to leaf2. The explanation involves looking at the conditional probability that the second flow with overlap with the first. Suppose the first flow takes is routed to spine1 via port swp1s0 and that spine1 routes the flow to leaf2 via port swp51. If the second flow is routed via any of the 4 paths through spine2, there is no collision. However, if it is routed via spine1, there is only 1 path that avoids collisions (leaf1 port swp1s1 to spine1 port swp52). This means that there is a 5 / 8 chance of avoiding a collision, or a 3/8 (37.5%) chance that the two flow will collide. The probability of flow collisions is surprisingly high even on very large networks with many spine switches and paths (see Birthday Paradox). 

Also note the Discards trend in the Congestion and Errors section. Comparing the rate of discards with Collisions in the Busy Spine Links chart shows that discards don't occur unless there are Elephant flow collisions on the busy links.

The Discard trend lags the Collision trend because discards are reported using sFlow counters and the Collision metric are based on packet samples - see Measurement delay, counters vs. packet samples

This example demonstrates the visibility into leaf and spine fabric performance achievable using standard sFlow instrumentation built into commodity switch hardware. If you have a leaf and spine network, request a free evaluation of Fabric View to better understand your network's performance.

This small four switch leaf and spine network is composed of 12 x 10 Gbits/sec links which would require 24 x 10 Gbits/sec taps with associated probes and collector to fully monitor using traditional tools used to monitor legacy data center networks. The cost and complexity of tapping leaf and spine topologies is prohibitive. However, leaf and spine switches typically include hardware support for the sFlow measurement standard, embedding line rate visibility into every switch port for network wide coverage at no extra cost. In this example, the Fabric View analytics software is running on a commodity physical or virtual server consuming 1% CPU and 200 MBytes RAM.

Real-time analytics for leaf and spine networks is a core enabling technology for software defined networking (SDN) control mechanisms that can automatically adapt the network to rapidly changing flow patterns and dramatically improve performance.

For example, REST API for Cumulus Linux ACLs describes how and SDN controller can remotely control switches. Use cases discussed on this blog include: Elephant flow marking,  Elephant flow steering, and DDoS mitigation.

Finally, Cumulus Linux runs on open switch hardware from Agema, Dell, Edge-Core, Penguin Computing, Quanta. In addition, Hewlett-Packard recently announced that they will soon be selling a new line of open network switches built by Accton Technologies and support Cumulus Linux. The increasing availability of low cost open networking hardware running Linux creates a platform for open source and commercial software developers to quickly build and deploy innovative solutions.

Topology discovery with Cumulus Linux

Demo: Implementing the OpenStack Design Guide in the Cumulus Workbench is a great demonstration of the power of zero touch provisioning and automation. When the switches and servers boot they automatically pick up their operating systems and configurations for the complex network shown in the diagram.

REST API for Cumulus Linux ACLs describes a REST server for remotely controlling ACLs on Cumulus Linux. This article will discuss recently added topology discovery methods that allow an SDN controller to learn topology and apply targeted controls (e.g Large "Elephant" flow marking, Large flow steering, DDoS mitigation, etc.).

Prescriptive Topology Manager

Complex Topology and Wiring Validation in Data Centers describes how Cumulus Networks' prescriptive topology manager (PTM) provides a simple method of verifying and enforcing correct wiring topologies.

The following REST call converts the topology from PTM's dot notation and returns a JSON representation:

cumulus@wbench:~$ curl http://leaf1:8080/ptm

Returns the result:

{
 "links": {
  "L1": {
   "node1": "leaf1", 
   "node2": "spine1", 
   "port1": "swp1s0", 
   "port2": "swp49"
  },
  ...
 }
}

LLDP

Prescriptive Topology Manager is preferred since it ensures that the discovered topology is correct. However, PTM builds on basic Link Level Discovery Protocol (LLDP), which provides an alternative method of topology discovery.

The following REST call return the hostname:

cumulus@wbench:~$ curl http://leaf1:8080/hostname

Returns result:

"leaf1"

The following REST call returns LLDP neighbor information:

cumulus@wbench:~$ curl http://leaf1:8080/lldp/neighbors

Returns result:

{
   "lldp": [
     {
       "interface": [
         {
           "name": "eth0",
           "via": "LLDP",
           "chassis": [
             {
               "id": [
                 {
                   "type": "mac",
                   "value": "6c:64:1a:00:2e:7f"
                 }
               ],
               "name": [
                 {
                   "value": "colo-tor-3"
                 }
               ]
             }
           ],
           "port": [
             {
               "id": [
                 {
                   "type": "ifname",
                   "value": "swp10"
                 }
               ],
               "descr": [
                 {
                   "value": "swp10"
                 }
               ]
             }
           ]
         },
         ...
     }
   ]
 }

The following REST call returns LLDP configuration information:

cumulus@wbench:~$ curl http://leaf1:8080/lldp/configuration

Returns result:

{
   "configuration": [
     {
       "config": [
         {
           "tx-delay": [
             {
               "value": "30"
             }
           ],
           ...
         }
       ]
     }
   ]
 }

Topology discovery with LLDP

The script lldp.py extracts LLDP data from all the switches in the network and compiles a topology:

#!/usr/bin/env python

import sys, re, fileinput, json, requests

switch_list = ['leaf1','leaf2','spine1','spine2']

l = 0
linkdb = {}
links = {}
for switch_name in switch_list:
  # verify that lldp configuration exports hostname,ifname information
  r = requests.get("http://%s:8080/lldp/configuration" % (switch_name));
  if r.status_code != 200: continue
  config = r.json()
  lldp_hostname = config['configuration'][0]['config'][0]['hostname'][0]['value']
  if lldp_hostname != '(none)': continue
  lldp_porttype = config['configuration'][0]['config'][0]['lldp_portid-type'][0]['value']
  if lldp_porttype != 'ifname': continue
  # local hostname 
  r = requests.get("http://%s:8080/hostname" % (switch_name));
  if r.status_code != 200: continue
  host = r.json()
  # get neighbors
  r = requests.get("http://%s:8080/lldp/neighbors" % (switch_name));
  if r.status_code != 200: continue
  neighbors = r.json()
  interfaces = neighbors['lldp'][0]['interface']
  for i in interfaces:
    # local port name
    port = i['name']
    # neighboring hostname
    nhost = i['chassis'][0]['name'][0]['value']
    # neighboring port name
    nport = i['port'][0]['descr'][0]['value']
    if not host or not port or not nhost or not nport: continue
    if host < nhost:
      link = {'node1':host,'port1':port,'node2':nhost,'port2':nport}
    else:
      link = {'node1':nhost,'port1':nport,'node2':host,'port2':port}
    keystr = "%s %s -- %s %s" % (link['node1'],link['port1'],link['node2'],link['port2'])
    if keystr in linkdb:
       # check consistency
       prev = linkdb[keystr]
       if (link['node1'] != prev['node1'] 
           or link['port1'] != prev['port1']
           or link['node2'] != prev['node2']
           or link['port2'] != prev['port2']): raise Exception('Mismatched LLDP', keystr)
    else:
       linkdb[keystr] = link
       linkname = 'L%d' % (l)
       links[linkname] = link
       l += 1

top = {'links':links}               
print json.dumps(top,sort_keys=True, indent=1)

Returns result:

cumulus@wbench:~$ ./lldp.py 
{
 "links": {
  "L0": {
   "node1": "colo-tor-3", 
   "node2": "leaf1", 
   "port1": "swp10", 
   "port2": "eth0"
  }, 
  ...
 }
}

The lldp.py script and the latest version of acl_server can be found on Github, https://github.com/pphaal/acl_server/

Demonstration

Fabric visibility with Cumulus Linux demonstrates the visibility into network performance provided by Cumulus Linux support for the sFlow standard (see Cumulus Networks, sFlow and data center automation). The screen shot shows 10Gbit/s Elephant flows traversing the network shown at the top of this article. The flows between server1 and server2 were generated using iperf tests running in a continuous loop.

The acl_server and sFlow agents are installed on the leaf1, leaf2, spine1, and spine2 switches. By default, the sFlow agents automatically pick up their settings using DNS Service Discovery (DNS-SD). Adding the following entry in the wbench DNS server zone file, /etc/bind/zones/lab.local.zone, enables sFlow on the switches and directs measurements to the wbench host:

_sflow._udp     30      SRV     0 0 6343 wbench

Note: For more information on running sFlow in the Cumulus workbench, see Demo: Monitoring Traffic on Cumulus Switches with sFlow). Another point to note, this workbench setup demonstrates the visibility into Link Aggregation (LAG) provides by sFlow (see Link aggregation).

Fabric View is installed on wbench and is configured with the network topology obtained from acl_server. The web interface is accessed through the workbench reverse proxy, but access is also possible using a VPN (see Setting up OpenVPN on the Cumulus Workbench).

This workbench example automatically provisions an OpenStack cluster on the two servers along with the network to connect them. In much the same way OpenStack provides access to virtual resources, Cumulus' Remote Lab leverages the automation capabilities of open hardware to provide multi-tenant access to physical servers and networks.

Finally, Cumulus Linux runs on open switch hardware from Agema, Dell, Edge-Core, Penguin Computing, Quanta. In addition, Hewlett-Packard recently announced that they will soon be selling a new line of open network switches built by Accton Technologies and support Cumulus Linux. This article, demonstrates the flexibility that open networking offers to developers and network administrators. If you are curious, its very easy to give Cumulus Linux a try.

Fabric visibility with Arista EOS

A leaf and spine fabric is challenging to monitor. The fabric spreads traffic across all the switches and links in order to maximize bandwidth. Unlike traditional hierarchical network designs, where a small number of links can be monitored to provide visibility, a leaf and spine network has no special links or switches where running CLI commands or attaching a probe would provide visibility. Even if it were possible to attach probes, the effective bandwidth of a leaf and spine network can be as high as a Petabit/second, well beyond the capabilities of current generation monitoring tools.

The 2 minute video provides an overview of some of the performance challenges with leaf and spine fabrics and demonstrates Fabric View - a monitoring solution that leverages industry standard sFlow instrumentation in commodity data center switches to provide real-time visibility into fabric performance.

Fabric View is free to try, just register at http://www.myinmon.com/ and request an evaluation. The software requires an accurate network topology in order to characterize performance and this article will describe how to obtain the topology from a fabric of Arista Networks switches.

Arista EOS™ includes the eAPI JSON-RPC service for programmatic monitoring and control. The article Arista eAPI 101 introduces eAPI and describes how to enable the service in EOS. Enable eAPI on all the switches in the fabric.

Configure all the switches in the leaf and spine fabric to send sFlow to the Fabric View server. The following script demonstrates how sFlow can be configured programmatically using an eAPI script:

#!/usr/bin/env python

import requests
import json
import signal
from jsonrpclib import Server

switch_list = ['switch1.example.com','switch2.example.com']
username = "admin"
password = "password"

sflow_collector = "192.168.56.1"
sflow_port = "6343"
sflow_polling = "20"
sflow_sampling = "10000"

for switch_name in switch_list:
  switch = Server("https://%s:%s@%s/command-api" %
                (username, password, switch_name))
  response = switch.runCmds(1,
   ["enable",
    "configure",
    "sflow source %s" % switch_ip,
    "sflow destination %s %s" % (sflow_collector, sflow_port),
    "sflow polling-interval %s" % sflow_polling,
    "sflow sample output interface",
    "sflow sample dangerous %s" % sflow_sampling,
    "sflow run"])

Next use the following eAPI script to discover the topology:

#/usr/bin/python 
'''
Copyright (c) 2015, Arista Networks, Inc. All rights reserved.
 
Redistribution and use in source and binary forms, with or without 
modification, are permitted provided that the following conditions are met:
 
 * Redistributions of source code must retain the above copyright notice, 
   this list of conditions and the following disclaimer. 

 * Redistributions in binary form must reproduce the above copyright notice, 
   this list of conditions and the following disclaimer in the documentation 
   and/or other materials provided with the distribution. 

 * Neither the name of Arista Networks nor the names of its contributors 
   may be used to endorse or promote products derived from this software 
   without specific prior written permission.
 
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL ARISTA NETWORKS BE LIABLE FOR ANY DIRECT,
INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
'''

# v0.5 - initial version of the script to discover network topology using
# Arista eAPI and generate output in json format recognized by sFlow-RT.

from jsonrpclib import Server 
import json 
from pprint import pprint

# define switch in your topology, eapi transport protocol (http or https),
# eapi username and password
switch_list = ['switch1.example.com','switch2.example.com']
eapi_transport = 'https'
eapi_username = 'admin'
eapi_password = 'password'

debug = False

# internal variables used by the script
allports = {}
allswitches = {}
allneighbors = []
alllinks = {}

# method to populate allswitches and allports - called only from processNeighbor()
def addPort(switchname, switchIP, portname, ifindex):
 id = switchname + '>' + portname
 prt = allports.setdefault(id, { "portname": portname, "linked": False })
 if ifindex is not None:
  prt["ifindex"] = ifindex
 sw = allswitches.setdefault(switchname, { "name": switchname, "agent": switchIP, "ports": {} });
 if switchIP is not None:
  sw["agent"] = switchIP
 sw["ports"][portname] = prt

# method to collect neighbor records - called with each LLDP neighbor 
# entry as they are discovered
def processNeighbor(localname,localip,localport,localifindex,remotename,remoteport):
 addPort(localname, localip, localport,localifindex);
 addPort(remotename, None, remoteport, None);
 allneighbors.append({ "localname": localname, "localport": localport,
         "remotename": remotename, "remoteport": remoteport });

# method to remove agents that we did not discover properly, or
# that we did not intend to include in the topology.  (If we
# assigned an agent field to the switch then we assume it should stay.)
def pruneAgents():
 for nm,sw in allswitches.items():
  #if not "agent" in sw:
  if sw['agent'] == '0.0.0.0' or not sw['agent']:
   del allswitches[nm]

# method to test for a new link - called only from findLinks()
def testLink(nbor,linkno):
 swname1 = nbor["localname"]
 swname2 = nbor["remotename"]
 # one of the switches might have been pruned out
 if swname1 not in allswitches or swname2 not in allswitches:
  return False
 sw1 = allswitches[swname1]
 sw2 = allswitches[swname2]
 pname1 = nbor["localport"]
 pname2 = nbor["remoteport"]
 port1 = sw1["ports"][pname1];
 port2 = sw2["ports"][pname2];
 if not port1["linked"] and not port2["linked"]:
  # add new link
  linkid = "link" + str(linkno)
  port1["linked"] = True;
  port2["linked"] = True;
  alllinks[linkid] = {
   "node1": nbor["localname"],
   "port1": nbor["localport"],
   "node2": nbor["remotename"],
   "port2": nbor["remoteport"]
   }
  return True
 return False

# method to find unique links - call at the end once all the LLDP records have
# been processed from all the switches
def findLinks():
 linkcount = 0
 for nbor in allneighbors:
  if testLink(nbor, linkcount+1):
   linkcount += 1

# method to dump topology in json format recognized by sFlow-RT
def dumpTopology():
 topology = { "nodes": allswitches, "links": alllinks }
 print(json.dumps(topology, indent=4))

# method to get LLDP neighbors of each switch - calls processNeighbor() for each LLDP neighbor found
def getLldpNeighbors(switch_name):
 try:
  switch = Server('%s://%s:%s@%s/command-api' % (eapi_transport, eapi_username, eapi_password, switch_name))

  # Get LLDP neighbors
  commands = ["enable", "show lldp neighbors"]
  response = switch.runCmds(1, commands, 'json')
  neighbors = response[1]['lldpNeighbors']

  # Get local hostname
  commands = ["enable", "show hostname"]
  response = switch.runCmds(1, commands, 'json')
  hostname = response[1]['hostname']

  # Get SNMP ifIndexes
  commands = ["enable", "show snmp mib ifmib ifindex"]
  response = switch.runCmds(1, commands, 'json')
  interfaceIndexes = response[1]['ifIndex']

  # Get sFlow agent source address
  commands = ["enable", "show sflow"]
  response = switch.runCmds(1, commands, 'json')
  sflowAddress = response[1]['ipv4Sources'][0]['ipv4Address']
  
  # Create 2D array lldp_neighbors where each line has following entries 
  # , , , 
  lldp_neighbors = []
  for neighbor in neighbors:
   lldp_neighbors.append([neighbor['neighborDevice'].split('.')[0], 
        neighbor['port'], neighbor['neighborPort'], interfaceIndexes[neighbor['port']]])
  
  if (debug): 
   pprint(lldp_neighbors)


  # collect switches, ports and neighbor-relationships
  for row in lldp_neighbors:
   processNeighbor(hostname, 
    sflowAddress,
    row[1], # localport
    row[3], # localifindex
    row[0], # remotename
    row[2]) # remoteport

  # Print list of LLDP neighbors in human friendly format:
  #  neighbor, , connected to local  with remote 
  if debug:
   print "Switch %s has following %d neighbors:" % (hostname[1], len(neighbors))
   for i, neighbor in enumerate(lldp_neighbors):
    print "#%d neighbor, %s, connected to local %s with remote %s" % (i+1, neighbor[0], neighbor[1], neighbor[2])

 except:
  print 'Exception while connecting to %s' % switch_name
  return []


for switch in switch_list:
 getLldpNeighbors(switch)

pruneAgents()
findLinks()
dumpTopology()

The script outputs a JSON representation of the topology, for example:

{
    "nodes": {
        "leaf332": {
            "name": "leaf332", 
            "agent": "10.10.130.142", 
            "ports": {
                "Management1": {
                    "portname": "Management1", 
                    "ifindex": 999001, 
                    "linked": false
                }, 
                "Ethernet50/1": {
                    "portname": "Ethernet50/1", 
                    "ifindex": 50001, 
                    "linked": true
                }, 
                "Ethernet36": {
                    "portname": "Ethernet36", 
                    "ifindex": 36, 
                    "linked": true
                }, 
                "Ethernet51/1": {
                    "portname": "Ethernet51/1", 
                    "ifindex": 51001, 
                    "linked": true
                }, 
                "Ethernet52/1": {
                    "portname": "Ethernet52/1", 
                    "ifindex": 52001, 
                    "linked": true
                }, 
                "Ethernet49/1": {
                    "portname": "Ethernet49/1", 
                    "ifindex": 49001, 
                    "linked": true
                }, 
                "Ethernet12": {
                    "portname": "Ethernet12", 
                    "ifindex": 12, 
                    "linked": false
                }, 
                "Ethernet35": {
                    "portname": "Ethernet35", 
                    "ifindex": 35, 
                    "linked": true
                }
            }
        }, 
        "leaf259": {
            "name": "leaf259", 
            "agent": "10.10.129.220", 
            "ports": {
                "Management1": {
                    "portname": "Management1", 
                    "ifindex": 999001, 
                    "linked": false
                }, 
                "Ethernet5/1": {
                    "portname": "Ethernet5/1", 
                    "ifindex": 5001, 
                    "linked": true
                }, 
                "Ethernet29": {
                    "portname": "Ethernet29", 
                    "ifindex": 29, 
                    "linked": true
                }, 
                "Ethernet32": {
                    "portname": "Ethernet32", 
                    "ifindex": 32, 
                    "linked": true
                }, 
                "Ethernet6/1": {
                    "portname": "Ethernet6/1", 
                    "ifindex": 6001, 
                    "linked": true
                }, 
                "Ethernet31": {
                    "portname": "Ethernet31", 
                    "ifindex": 31, 
                    "linked": true
                }, 
                "Ethernet30": {
                    "portname": "Ethernet30", 
                    "ifindex": 30, 
                    "linked": true
                }, 
                "Ethernet15/1": {
                    "portname": "Ethernet15/1", 
                    "ifindex": 15001, 
                    "linked": false
                }
            }
        }, 
        "leaf331": {
            "name": "leaf331", 
            "agent": "10.10.130.141", 
            "ports": {
                "Management1": {
                    "portname": "Management1", 
                    "ifindex": 999001, 
                    "linked": false
                }, 
                "Ethernet50/1": {
                    "portname": "Ethernet50/1", 
                    "ifindex": 50001, 
                    "linked": true
                }, 
                "Ethernet36": {
                    "portname": "Ethernet36", 
                    "ifindex": 36, 
                    "linked": true
                }, 
                "Ethernet1": {
                    "portname": "Ethernet1", 
                    "ifindex": 1, 
                    "linked": false
                }, 
                "Ethernet51/1": {
                    "portname": "Ethernet51/1", 
                    "ifindex": 51001, 
                    "linked": true
                }, 
                "Ethernet52/1": {
                    "portname": "Ethernet52/1", 
                    "ifindex": 52001, 
                    "linked": true
                }, 
                "Ethernet49/1": {
                    "portname": "Ethernet49/1", 
                    "ifindex": 49001, 
                    "linked": true
                }, 
                "Ethernet11": {
                    "portname": "Ethernet11", 
                    "ifindex": 11, 
                    "linked": false
                }, 
                "Ethernet35": {
                    "portname": "Ethernet35", 
                    "ifindex": 35, 
                    "linked": true
                }
            }
        }, 
        "leaf260": {
            "name": "leaf260", 
            "agent": "10.10.129.221", 
            "ports": {
                "Management1": {
                    "portname": "Management1", 
                    "ifindex": 999001, 
                    "linked": false
                }, 
                "Ethernet11/1": {
                    "portname": "Ethernet11/1", 
                    "ifindex": 11001, 
                    "linked": false
                }, 
                "Ethernet5/1": {
                    "portname": "Ethernet5/1", 
                    "ifindex": 5001, 
                    "linked": true
                }, 
                "Ethernet29": {
                    "portname": "Ethernet29", 
                    "ifindex": 29, 
                    "linked": true
                }, 
                "Ethernet32": {
                    "portname": "Ethernet32", 
                    "ifindex": 32, 
                    "linked": true
                }, 
                "Ethernet6/1": {
                    "portname": "Ethernet6/1", 
                    "ifindex": 6001, 
                    "linked": true
                }, 
                "Ethernet31": {
                    "portname": "Ethernet31", 
                    "ifindex": 31, 
                    "linked": true
                }, 
                "Ethernet30": {
                    "portname": "Ethernet30", 
                    "ifindex": 30, 
                    "linked": true
                }
            }
        }, 
        "core210": {
            "name": "core210", 
            "agent": "10.10.129.185", 
            "ports": {
                "Ethernet3/3/1": {
                    "portname": "Ethernet3/3/1", 
                    "ifindex": 3037, 
                    "linked": false
                }, 
                "Ethernet3/6/1": {
                    "portname": "Ethernet3/6/1", 
                    "ifindex": 3073, 
                    "linked": true
                }, 
                "Ethernet3/5/1": {
                    "portname": "Ethernet3/5/1", 
                    "ifindex": 3061, 
                    "linked": true
                }, 
                "Ethernet3/2/1": {
                    "portname": "Ethernet3/2/1", 
                    "ifindex": 3025, 
                    "linked": false
                }, 
                "Ethernet3/8/1": {
                    "portname": "Ethernet3/8/1", 
                    "ifindex": 3097, 
                    "linked": true
                }, 
                "Ethernet3/1/1": {
                    "portname": "Ethernet3/1/1", 
                    "ifindex": 3013, 
                    "linked": false
                }, 
                "Management1/1": {
                    "portname": "Management1/1", 
                    "ifindex": 999011, 
                    "linked": false
                }, 
                "Ethernet3/34/1": {
                    "portname": "Ethernet3/34/1", 
                    "ifindex": 3409, 
                    "linked": false
                }, 
                "Ethernet3/31/1": {
                    "portname": "Ethernet3/31/1", 
                    "ifindex": 3373, 
                    "linked": false
                }, 
                "Ethernet3/7/1": {
                    "portname": "Ethernet3/7/1", 
                    "ifindex": 3085, 
                    "linked": true
                }
            }
        }, 
        "core212": {
            "name": "core212", 
            "agent": "10.10.129.64", 
            "ports": {
                "Ethernet3/3/1": {
                    "portname": "Ethernet3/3/1", 
                    "ifindex": 3037, 
                    "linked": false
                }, 
                "Ethernet3/12/1": {
                    "portname": "Ethernet3/12/1", 
                    "ifindex": 3145, 
                    "linked": false
                }, 
                "Ethernet3/2/1": {
                    "portname": "Ethernet3/2/1", 
                    "ifindex": 3025, 
                    "linked": false
                }, 
                "Ethernet3/13/1": {
                    "portname": "Ethernet3/13/1", 
                    "ifindex": 3157, 
                    "linked": false
                }, 
                "Ethernet3/31/1": {
                    "portname": "Ethernet3/31/1", 
                    "ifindex": 3373, 
                    "linked": false
                }, 
                "Ethernet3/32/1": {
                    "portname": "Ethernet3/32/1", 
                    "ifindex": 3385, 
                    "linked": false
                }, 
                "Ethernet3/18/1": {
                    "portname": "Ethernet3/18/1", 
                    "ifindex": 3217, 
                    "linked": true
                }, 
                "Ethernet3/28/1": {
                    "portname": "Ethernet3/28/1", 
                    "ifindex": 3337, 
                    "linked": true
                }, 
                "Ethernet3/33/1": {
                    "portname": "Ethernet3/33/1", 
                    "ifindex": 3397, 
                    "linked": false
                }, 
                "Ethernet3/5/1": {
                    "portname": "Ethernet3/5/1", 
                    "ifindex": 3061, 
                    "linked": true
                }, 
                "Ethernet3/8/1": {
                    "portname": "Ethernet3/8/1", 
                    "ifindex": 3097, 
                    "linked": true
                }, 
                "Ethernet3/34/1": {
                    "portname": "Ethernet3/34/1", 
                    "ifindex": 3409, 
                    "linked": false
                }, 
                "Ethernet3/36/1": {
                    "portname": "Ethernet3/36/1", 
                    "ifindex": 3433, 
                    "linked": false
                }, 
                "Ethernet3/35/1": {
                    "portname": "Ethernet3/35/1", 
                    "ifindex": 3421, 
                    "linked": false
                }, 
                "Ethernet3/15/1": {
                    "portname": "Ethernet3/15/1", 
                    "ifindex": 3181, 
                    "linked": true
                }, 
                "Ethernet3/7/1": {
                    "portname": "Ethernet3/7/1", 
                    "ifindex": 3085, 
                    "linked": true
                }, 
                "Ethernet3/16/1": {
                    "portname": "Ethernet3/16/1", 
                    "ifindex": 3193, 
                    "linked": true
                }, 
                "Ethernet3/17/1": {
                    "portname": "Ethernet3/17/1", 
                    "ifindex": 3205, 
                    "linked": true
                }, 
                "Management1/1": {
                    "portname": "Management1/1", 
                    "ifindex": 999011, 
                    "linked": false
                }, 
                "Ethernet3/26/1": {
                    "portname": "Ethernet3/26/1", 
                    "ifindex": 3313, 
                    "linked": true
                }, 
                "Ethernet3/25/1": {
                    "portname": "Ethernet3/25/1", 
                    "ifindex": 3301, 
                    "linked": true
                }, 
                "Ethernet3/21/1": {
                    "portname": "Ethernet3/21/1", 
                    "ifindex": 3253, 
                    "linked": false
                }, 
                "Ethernet3/11/1": {
                    "portname": "Ethernet3/11/1", 
                    "ifindex": 3133, 
                    "linked": false
                }, 
                "Ethernet3/6/1": {
                    "portname": "Ethernet3/6/1", 
                    "ifindex": 3073, 
                    "linked": true
                }, 
                "Ethernet3/27/1": {
                    "portname": "Ethernet3/27/1", 
                    "ifindex": 3325, 
                    "linked": true
                }, 
                "Ethernet3/1/1": {
                    "portname": "Ethernet3/1/1", 
                    "ifindex": 3013, 
                    "linked": false
                }, 
                "Ethernet3/23/1": {
                    "portname": "Ethernet3/23/1", 
                    "ifindex": 3277, 
                    "linked": false
                }, 
                "Ethernet3/22/1": {
                    "portname": "Ethernet3/22/1", 
                    "ifindex": 3265, 
                    "linked": false
                }
            }
        }
    }, 
    "links": {
        "link5": {
            "node1": "leaf260", 
            "node2": "core212", 
            "port2": "Ethernet3/15/1", 
            "port1": "Ethernet31"
        }, 
        "link4": {
            "node1": "leaf260", 
            "node2": "core212", 
            "port2": "Ethernet3/5/1", 
            "port1": "Ethernet30"
        }, 
        "link7": {
            "node1": "leaf259", 
            "node2": "core210", 
            "port2": "Ethernet3/6/1", 
            "port1": "Ethernet29"
        }, 
        "link6": {
            "node1": "leaf260", 
            "node2": "core212", 
            "port2": "Ethernet3/25/1", 
            "port1": "Ethernet32"
        }, 
        "link1": {
            "node1": "leaf260", 
            "node2": "leaf259", 
            "port2": "Ethernet5/1", 
            "port1": "Ethernet5/1"
        }, 
        "link3": {
            "node1": "leaf260", 
            "node2": "core210", 
            "port2": "Ethernet3/5/1", 
            "port1": "Ethernet29"
        }, 
        "link2": {
            "node1": "leaf260", 
            "node2": "leaf259", 
            "port2": "Ethernet6/1", 
            "port1": "Ethernet6/1"
        }, 
        "link9": {
            "node1": "leaf259", 
            "node2": "core212", 
            "port2": "Ethernet3/16/1", 
            "port1": "Ethernet31"
        }, 
        "link8": {
            "node1": "leaf259", 
            "node2": "core212", 
            "port2": "Ethernet3/6/1", 
            "port1": "Ethernet30"
        }, 
        "link15": {
            "node1": "leaf331", 
            "node2": "core212", 
            "port2": "Ethernet3/17/1", 
            "port1": "Ethernet51/1"
        }, 
        "link14": {
            "node1": "leaf331", 
            "node2": "core212", 
            "port2": "Ethernet3/7/1", 
            "port1": "Ethernet50/1"
        }, 
        "link17": {
            "node1": "leaf332", 
            "node2": "core210", 
            "port2": "Ethernet3/8/1", 
            "port1": "Ethernet49/1"
        }, 
        "link16": {
            "node1": "leaf331", 
            "node2": "core212", 
            "port2": "Ethernet3/27/1", 
            "port1": "Ethernet52/1"
        }, 
        "link11": {
            "node1": "leaf331", 
            "node2": "leaf332", 
            "port2": "Ethernet35", 
            "port1": "Ethernet35"
        }, 
        "link10": {
            "node1": "leaf259", 
            "node2": "core212", 
            "port2": "Ethernet3/26/1", 
            "port1": "Ethernet32"
        }, 
        "link13": {
            "node1": "leaf331", 
            "node2": "core210", 
            "port2": "Ethernet3/7/1", 
            "port1": "Ethernet49/1"
        }, 
        "link12": {
            "node1": "leaf331", 
            "node2": "leaf332", 
            "port2": "Ethernet36", 
            "port1": "Ethernet36"
        }, 
        "link20": {
            "node1": "leaf332", 
            "node2": "core212", 
            "port2": "Ethernet3/28/1", 
            "port1": "Ethernet52/1"
        }, 
        "link19": {
            "node1": "leaf332", 
            "node2": "core212", 
            "port2": "Ethernet3/18/1", 
            "port1": "Ethernet51/1"
        }, 
        "link18": {
            "node1": "leaf332", 
            "node2": "core212", 
            "port2": "Ethernet3/8/1", 
            "port1": "Ethernet50/1"
        }
    }
}

Access the Fabric View web interface at http://fabricview:8008/ and navigate to the Settings tab:

Upload the JSON topology file by clicking on the disk icon in the Topology section. Alternatively, the topology can be installed programmatically using the Fabric View REST API documented at the bottom of the Settings page.

As soon as the topology is installed, traffic data should start appearing in Fabric View. The video provides a quick walkthrough of the software features.