Thursday, August 30, 2018

Northbound Networks Zodiac GX

Mininet is widely used to emulate software defined networks (SDNs). Mininet flow analytics describes how standard sFlow telemetry, from Open vSwitch used by Mininet emulate the network, provides feedback to an SDN controller, allowing the controller to adapt the network to changing traffic, for example, to mitigate a distributed denial of service (DDoS) attack.

Northbound Networks Zodiac GX is an inexpensive open source software based switch that is ideal for experimenting with software defined networking (SDN) in a physical network setting. The small fanless package makes the switch an attractive option for desktop use. The Zodiac GX is also based on Open vSwitch, making it easy to take SDN control strategies developed on Mininet.
Enabling sFlow on the Zodiac GX is easy, navigate to the System>Startup page and add the following line to the end of the startup script (before the exit 0 line):
ovs-vsctl -- --id=@sflow create sflow agent=$OVS_BR target=$IP_CONTROLLER_1 sampling=100 polling=10 -- set bridge $OVS_BR sflow=@sflow
Reboot the switch for the changed to take effect.

Use sflowtool to verify that sFlow is arriving at the controller host and to examine the contents of the telemetry stream. Running sflowtool using Docker is a simple alternative to building the software from sources:
docker run --rm -p 8008:8008 -p 6343:6343/udp sflow/sflowtool
The text output from sflowtool can be piped into scripts to perform basic sFlow analysis.
A graphical sFlow analyzer performs the analysis tasks for you. The screen shot above shows sFlowTrend, a free sFlow analyzer that displays traffic trends. The software can be downloaded and installed or run using Docker:
docker run --rm -p 6343:6343/udp -p 8087:8087 -p 8443:8443 sflow/sflowtrend
The sFlowTrend charts update every minute. This is generally fast enough for human consumption, but real-time, up to the second, visibility is critical for SDN use cases.
The screen shot from Flow Trend shows an up to the second view of traffic. The spike in traffic is due to a 4K video being streamed from YouTube. The following command runs the software:
docker run --rm -p 6343:6343/udp -p 8008:8008 sflow/flow-trend
Flow Trend is an application running on the sFlow-RT real-time analytics platform.
Applications running on the sFlow-RT platform deliver real-time visibility to SDN, DevOps and Orchestration stacks, enabling new classes of performance aware application such as load balancing, DDoS mitigation, and workload placement.

RYU provides a framework that can be used to develop SDN applications in Python. For example, the following command runs the simple learning bridge application that ships with RYU:
docker run -it --rm -p 6633:6633 osrg/ryu ryu-manager --verbose ryu/ryu/app/
As soon as the switch connects to the controller, you should see a flurry of events as the controller programs flows on the Zodiac GX switch.

Faucet is an SDN controller for production networks implemented using RYU. Before we can use Faucet, we need to gather basic OpenFlow information from the switch.
$ ssh -t admin@ "sudo ovs-ofctl show ovslan"
admin@'s password: 
OFPT_FEATURES_REPLY (xid=0x2): dpid:000044d1fa6291b2
n_tables:254, n_buffers:256
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
 1(eth0.1): addr:44:d1:fa:62:91:b2
     config:     0
     state:      STP_FORWARD
     current:    1GB-FD AUTO_NEG
     speed: 1000 Mbps now, 0 Mbps max
 2(eth0.2): addr:44:d1:fa:62:91:b2
     config:     0
     state:      STP_FORWARD
     current:    1GB-FD AUTO_NEG
     speed: 1000 Mbps now, 0 Mbps max
 3(eth0.3): addr:44:d1:fa:62:91:b2
     config:     0
     state:      STP_FORWARD
     current:    1GB-FD AUTO_NEG
     speed: 1000 Mbps now, 0 Mbps max
 4(eth0.4): addr:44:d1:fa:62:91:b2
     config:     0
     state:      STP_FORWARD
     current:    1GB-FD AUTO_NEG
     speed: 1000 Mbps now, 0 Mbps max
 5(eth0.5): addr:44:d1:fa:62:91:b2
     config:     0
     state:      STP_FORWARD
     current:    1GB-FD AUTO_NEG
     speed: 1000 Mbps now, 0 Mbps max
 LOCAL(ovslan): addr:44:d1:fa:62:91:b2
     config:     0
     state:      0
     current:    1GB-FD AUTO_NEG
     speed: 1000 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
Connection to closed.
Update August 30, 2018: The only piece of information needed to construct the faucet config file below is the switch dpid. The Zodiac GX uses the MAC address of the switch as the dpid, so you can simply read the MAC address printed on a label on the bottom of the switch.
Now create a directory called faucet that contains the initial Faucet configuration file, faucet.yaml:
        vid: 100
        description: "office network"

        dp_id: 0x000044d1fa6291b2
        hardware: "ZodiacGX"
                name: "eth0.1"
                description: "port1"
                native_vlan: office
                name: "eth0.2"
                description: "port2"
                native_vlan: office
                name: "eth0.3"
                description: "port3"
                native_vlan: office
                name: "eth0.4"
                description: "port4"
                native_vlan: office
                name: "eth0.5"
                description: "port5"
                native_vlan: office
                name: "ovslan"
                description: "local"
                native_vlan: office
Now run Faucet:
docker run -it --rm -v $PWD/faucet/:/etc/faucet/ -v $PWD/faucet/:/var/log/faucet/ -p 6633:6653 -p 9302:9302 faucet/faucet
As soon as the switch connects to the controller, you should see events logged to the faucet.log file in the same directory as the faucet.yaml configuration file.

Faucet Documentation describes how to extend the configuration to include firewall, routing, segmentation, and network function virtualization (NFV) rules to the configuration.

The next step is integrating sFlow analytics with the controller. Writing Applications describes how write sFlow-RT applications use REST API and embedded JavaScript API. The document includes Python examples that could be embedded in RYU controller applications. Alternatively, sFlow-RT's embedded HTTP client can be used to push control actions to an SDN controller, see ONOS measurement based control for an example.

The sFlow telemetry stream contains detailed Open vSwitch performance metrics in addition to flow and interface counter data. The sFlow-RT analytics pipeline can be programmed to generate and push statistics to time series databases and dashboards, see Prometheus and Grafana and InfluxDB and Grafana.

Exporting events using syslog describes how sFlow-RT can be programmed to detect and report on traffic anomalies, sending events to Security Information and Event Management (SIEM) tools, using Splunk and Logstash as examples.

The sflow/sflow-rt Docker image provides a convenient means of developing and deploying sFlow-RT applications alongside the SDN controllers demonstrated in this article.

An important benefit of sFlow telemetry is that it decouples monitoring from the control plane. You are free to change SDN controllers, use distributed routing / switching protocols, move between network operating systems, or build your own control plane while maintaining the same level of visibility. Industry standard sFlow is widely supported by vendors, including: A10, Aerohive, ALUe, Allied Telesis, Arista, Aruba, Big Switch, Broadcom, Cisco, Cumulus, Dell, D-Link, Edge-Core, Extreme, F5, Fortinet, Huawei, IP Infusion, Juniper, Mellanox, Netgear, OpenSwitch, Pica8, Proxim, Quanta, SMC, ZTE, and ZyXEL.

Monday, August 20, 2018

RDMA over Converged Ethernet (RoCE)

RDMA over Converged Ethernet is a network protocol that allows remote direct memory access (RDMA) over an Ethernet network. One of the benefits running RDMA over Ethernet is the visibility provided by standard sFlow instrumentation embedded in the commodity Ethernet switches used to build data center leaf and spine networks where RDMA is most prevalent.

The sFlow telemetry stream includes packet headers, sampled at line rate by the switch hardware. Hardware packet sampling allows the switch to monitor traffic at line rate on all ports, keeping up with the high speed data transfers associated with RoCE.

The diagram above shows the packet headers associated with RoCEv1 and RoCEv2 packets. Decoding the InfiniBand Global Routing Header (IB GRH) and InfiniBand Base Transport Header (IB BTH) allows an sFlow analyzer to report in detail on RoCE traffic.
The sFlow-RT real-time analytics engine recently added support for RoCE by decoding InfiniBand Global Routing and InfiniBand Base Transport fields. The screen capture of the sFlow-RT Flow-Trend application shows traffic associated with an RoCEv2 connection between two hosts, and The traffic consists of SEND and ACK messages exchanged as part of a reliable connection (RC).

The standard sFlow instrumentation provides comprehensive network wide visibility into RoCE and all other applications sharing the network resources. Real-time visibility is an essential part of automating networks, providing the feedback needed to ensure that resources are efficiently allocated and rapidly identifying overloaded resources so that remediation action can be taken before significant service degradation occurs.

Thursday, July 19, 2018

ExtremeXOS 22.5.1 adds support Broadcom ASIC table utilization statistics

ExtremeXOS 22.5.1 is now available! describes added support in sFlow for "New data structures to support reporting on hardware/table utilization statistics." The feature is available on Summit X450-G2, X460-G2, X670-G2, X770, and ExtremeSwitching X440-G2, X870, X620, X690 series switches.

Figure 1 shows the packet processing pipeline of a Broadcom ASIC. The pipeline consists of a number of linked hardware tables providing bridging, routing, access control list (ACL), and ECMP forwarding group functions. Operations teams need to be able to proactively monitor table utilizations in order to avoid performance problems associated with table exhaustion.

Broadcom's sFlow specification, sFlow Broadcom Switch ASIC Table Utilization Structures, leverages the industry standard sFlow protocol to offer scaleable, multi-vendor, network wide visibility into the utilization of these hardware tables.

The following output from the open source sflowtool command line utility shows the raw table measurements (this is in addition to the extensive set of measurements already exported via sFlow by ExtremeXOS):
bcm_asic_host_entries 4
bcm_host_entries_max 8192
bcm_ipv4_entries 0
bcm_ipv4_entries_max 0
bcm_ipv6_entries 0
bcm_ipv6_entries_max 0
bcm_ipv4_ipv6_entries 9
bcm_ipv4_ipv6_entries_max 16284
bcm_long_ipv6_entries 3
bcm_long_ipv6_entries_max 256
bcm_total_routes 10
bcm_total_routes_max 32768
bcm_ecmp_nexthops 0
bcm_ecmp_nexthops_max 2016
bcm_mac_entries 3
bcm_mac_entries_max 32768
bcm_ipv4_neighbors 4
bcm_ipv6_neighbors 0
bcm_ipv4_routes 0
bcm_ipv6_routes 0
bcm_acl_ingress_entries 842
bcm_acl_ingress_entries_max 4096
bcm_acl_ingress_counters 68
bcm_acl_ingress_counters_max 4096
bcm_acl_ingress_meters 18
bcm_acl_ingress_meters_max 8192
bcm_acl_ingress_slices 3
bcm_acl_ingress_slices_max 8
bcm_acl_egress_entries 36
bcm_acl_egress_entries_max 512
bcm_acl_egress_counters 36
bcm_acl_egress_counters_max 1024
bcm_acl_egress_meters 18
bcm_acl_egress_meters_max 512
bcm_acl_egress_slices 2
bcm_acl_egress_slices_max 2
The sflowtool output is useful for troubleshooting and is easy to parse with scripts.

A convenient way to run sflowtool is to use Docker:
docker run -p 6343:6343/udp sflow/sflowtool

Ethernet Fabric Visibility

Ethernet Fabrics: Extreme Networks ExtremeFabric
Leaf and spine fabrics are challenging to monitor. The fabric spreads traffic across all the switches and links in order to maximize bandwidth. Unlike traditional hierarchical network designs, where a small number of links can be monitored to provide visibility, a leaf and spine network has no special links or switches where running CLI commands or attaching a probe would provide visibility. Even if it were possible to attach probes, the effective bandwidth of a leaf and spine network can be as high as a Petabit/second, well beyond the capabilities of current generation monitoring tools.

Fabric View solves the visibility challenge by using the industry standard sFlow instrumentation built into data center switches. Fabric View represents the fabric as if it were a single large chassis switch, treating each leaf switch as a line card and the spine switches as the backplane. The result is an intuitive tool that is easily understood by anyone familiar with traditional networks.

A demonstration can be run using Docker:
docker run --entrypoint /sflow-rt/ -p 8008:8008 sflow/fabric-view
Access the web interface on port 8008.
The first chart shows the largest TCP/UDP flows traversing the fabric (calculated from a continues stream of packet samples received from all the switches in the fabric). The chart updates every second, providing a real-time view of traffic crossing the fabric.
The last two charts are based on the hardware/table utilization statistics that are now implemented in ExtremeXOS, trending the maximum utilization of each table across all the switches in the fabric.


FabricView is one of a number of applications developed for sFlow-RT. Examples include: DDoS mitigation, Internet routing using top of rack switches, and other articles on this blog.
The sFlow-RT analytics engine receives a continuous telemetry stream from sFlow Agents embedded in network devices, hosts and applications and converts them into actionable metrics, accessible through APIs. Applications can be external, written in any language that supports HTTP/REST calls, or internal, using sFlow-RT's embedded JavaScript/ECMAScript.

Monday, July 16, 2018

Visualizing real-time network traffic flows at scale

Particle has been released on GitHub, The application is a real-time visualization of network traffic in which particles flow between hosts arranged around the edges of the screen. Particle colors represent different types of traffic.

Particles provide an intuitive representation of network packets transiting the network from source to destination. The animation slows time so that the particle takes 10 seconds (instead of milliseconds) to transit the network. Groups of particles traveling the same path represent flows of packets between the hosts. Particle size and frequency are used to indicate the intensity magnitude of the traffic flowing on a path.

Particles don't follow straight lines, instead following quadratic Bézier curves around the center of the screen. Warping particle paths toward the center of the screen ensures that all paths are of similar length and visible - even if the start and end points are on the same axis.

The example above is from a site with over 500 network switches carrying hundreds of Gigabits of traffic. Internet, Customer, Site and Datacenter hosts have been assigned to the North, East, South and West sides respectively.
The screen is updated 60 times per second for smooth animation. Active flow metrics are updated every second. Hovering over the screen freezes the animation, highlights the nearest particle, and displays details.

To try out the software, first create a configuration file to label axes and assign addresses for your network.
The above, particle.conf file, provides an example.

The simplest way to run the software is to use the pre-built Docker image.
docker run -p 8008:8008 -p 6343:6343/udp \
-v $PWD/particle.conf:/sflow-rt/particle.conf \
-e "RTPROP=-Dsystem.propertyFiles=particle.conf" \
Access the web interface on port 8008.
The Docker image also contains a random simulation of flows to demonstrate the software:
docker run -e "RTPROP=-Dparticle.demo=yes" \
-p 6343:6343/udp -p 8008:8008 sflow/particle
This particle visualization was inspired by experiments with Vizceral, see Real-time traffic visualization using Netflix Vizceral. Vizceral focuses on interactions between layered microservices.

Visualizing network traffic unique challenges that needed to be addressed. For example, in these examples the North, Internet, axis ( represents over 4 billion hosts - a number far greater than the number of pixels available on the screen. Instead of trying to represent each host individually, hosts are assigned a position proportional to their location in the range. For example, host is assigned a position half way along the axis. Assigning fixed positions to each host ensures that traffic between the hosts will always take the same path across the screen, making it easier to recognize patterns and identify changes.

Chances are you have network equipment that supports sFlow telemetry since the standard is widely supported by vendors, including: A10, Aerohive, ALUe, Allied Telesis, Arista, Aruba, Big Switch, Cisco, Cumulus, Dell, D-Link, Edge-Core, Extreme, F5, Fortinet, Huawei, IP Infusion, Juniper, Netgear, OpenSwitch, Pica8, Proxim, Quanta, SMC, ZTE, and ZyXEL. Give Particle a try and see how traffic flows on your network.

Wednesday, July 11, 2018

sFlow available on Juniper PTX series routers

sFlow functionality introduced on the PTX1000 and PTX10000 platforms—Starting in Junos OS Release 18.2R1, the PTX1000 and PTX10000 routers support sFlow, a network monitoring protocol for high-speed networks. With sFlow, you can continuously monitor tens of thousands of ports simultaneously. The mechanism used by sFlow is simple, not resource intensive, and accurate.  - New and Changed Features

The recent article, sFlow available on Juniper MX series routers, describes how Juniper is extending sFlow support to include routers to provide visibility across their entire range of switching and routing products.

Universal support for industry standard sFlow as a base Junos feature reduces the operational complexity and cost of network visibility for enterprises and service providers. Real-time streaming telemetry from campus switches, routers, and data center switches, provides centralized, real-time, end-to-end visibility needed to troubleshoot, optimize, and account for network usage.

Analytics software is a critical factor in realizing the full benefits of sFlow monitoring. Choosing an sFlow analyzer discusses important factors to consider when selecting from the range of open source and commercial sFlow analysis tools.

Monday, April 9, 2018


Logical Table Software Development Kit (SDKLT) is a new, powerful, and feature rich Software Development Kit (SDK) for Broadcom switches. SDKLT provides a new approach to switch configuration using Logical Tables.

Building the Demo App describes how to get started using a simulated Tomahawk device. Included, is a CLI that can be used to explore tables. For example, the following CLI output shows the attributes of the sFlow packet sampling table:
  Description: The MIRROR_PORT_ENCAP_SFLOW logical table is used to specify
               per-port sFlow encapsulation sample configuration.
  11 fields (1 key-type field):
        Description: Sample ingress flex sFlow packet if the generated sFlow random
                     number is greater than the threshold. A lower threshold leads to
                     higher sampling frequency.
        Description: Sample egress sFlow packet if the generated sFlow random number is
                     greater than the threshold. A lower threshold leads to
                     higher sampling frequency.
        Description: Sample ingress sFlow packet if the generated sFlow random number is
                     greater than the threshold. A lower threshold leads to
                     higher sampling frequency.
        Description: Enable to copy ingress flex sFlow packet samples to the ingress
                     mirror member using the sFlow mirror instance configuration.
        Description: Enable to copy ingress flex sFlow packet samples to CPU.
        Description: Enable to copy ingress sFlow packet samples to the ingress
                     mirror member using the sFlow mirror instance configuration.
        Description: Enable to copy ingress sFlow packet samples to CPU.
        Description: Enable to sample ingress port-based flex sFlow packets.
        Description: Enable to sample egress port-based sFlow packets.
        Description: Enable to sample ingress port-based sFlow packets.
        Description: Logical port ID.
SDKLT is a part of the OpenNSL suite, which makes it possible for the development of open network operating system projects, including: Open Network Linux, OpenSwitch, and SONiC.
The network operating system bridges the gap between applications (BGP, SNMP, sFlow, etc.) and the low level hardware capabilities accessed through the SDK. For example, OpenSwitch describes how the open source Host sFlow agent uses Control Plane Services (CPS) and Open Compute Project (OCP) Switch Abstraction Interface (SAI) to configure hardware packet sampling via vendor specific SDKs (such as OpenNSL).

Friday, April 6, 2018

sFlow available on Juniper MX series routers

sFlow support on MX Series devices—Starting in Junos OS Release 18.1R1, you can configure sFlow technology (as a sFlow agent) on a MX Series device, to continuously monitor traffic at wire speed on all interfaces simultaneously. The sFlow technology is a monitoring technology for high-speed switched or routed networks.  - New and Changed Features

Understanding How to Use sFlow Technology for Network Monitoring on a MX Series Router lists the following benefits of sFlow Technology on a MX Series Router:
  • sFlow can be used by software tools like a network analyzer to continuously monitor tens of thousands of switch or router ports simultaneously.
  • Since sFlow uses network sampling (forwarding one packet from ‘n’ number of total packets) for analysis, it is not resource intensive (for example processing, memory and more). The sampling is done at the hardware application-specific integrated circuits (ASICs) and hence it is simple and more accurate.
With the addition of the MX series routers, Juniper now supports sFlow across its entire product range:
Universal support for industry standard sFlow as a base Junos feature reduces the operational complexity and cost of network visibility for enterprises and service providers. Real-time streaming telemetry from campus switches, routers, and data center switches, provides centralized, real-time, end-to-end visibility needed to troubleshoot, optimize, and account for network usage.
Analytics software is a critical factor in realizing the full benefits of sFlow monitoring. Choosing an sFlow analyzer discusses important factors to consider when selecting from the range of open source and commercial sFlow analysis tools.

Thursday, April 5, 2018

ONOS measurement based control

ONOS traffic analytics describes how to run the ONOS SDN controller with a virtual network created using Mininet. The article also showed how to monitor network traffic using industry standard sFlow instrumentation available in Mininet and in physical switches.
This article uses the same ONOS / Mininet test bed to demonstrate how sFlow-RT real-time flow analytics can be used to push controls to the network through the ONOS REST API.  Leaf and spine traffic engineering using segment routing and SDN used real-time flow analytics to load balance an ONOS controlled physical network. In this example, we will use ONOS to filter DDoS attack traffic on a Mininet virtual network.

The following sFlow-RT script, ddos.js, detects DDoS attacks and programs ONOS filter rules to block the attacks:
var user = 'onos';
var password = 'rocks';
var onos = '';
var controls = {};


setEventHandler(function(evt) {
 // don't consider inter-switch links
 var link = topologyInterfaceToLink(evt.agent,evt.dataSource);
 if(link) return;

 // get port information
 var port = topologyInterfaceToPort(evt.agent,evt.dataSource);
 if(!port) return;

 // need OpenFlow info to create ONOS filtering rule
 if(!port.dpid || !port.ofport) return;

 // we already have a control for this flow
 if(controls[evt.flowKey]) return;

 var [ipdestination,udpsourceport] = evt.flowKey.split(',');
 var msg = {
  flows: [
    selector: {
     criteria: [

 var resp = http2({
  body: JSON.stringify(msg)

 var {deviceId,flowId} = JSON.parse(resp.body).flows[0];
 controls[evt.flowKey] = {,

 logInfo("blocking " + evt.flowKey);

setIntervalHandler(function() {
 var now =;
 for(var key in controls) {
   let rec = controls[key];

   // keep control for at least 10 seconds
   if(now - rec.time < 10000) continue;
   // keep control if threshold still triggered
   if(thresholdTriggered(rec.threshold,rec.agent,rec.metric,key)) continue;

   var resp = http2({

   delete controls[key];

   logInfo("unblocking " + key);
Some notes on the script:
  1. The ONOS REST API is used to add/remove filters that block the DDoS traffic.
  2. The controller address,, can be found on the ONOS Cluster Nodes web page.
  3. The udp_reflection flow definition is designed to detect UDP amplification attacks, e.g. DNS amplification attacks
  4. Controls are applied to the switch port where traffic enters the network
  5. The controls structure is used to keep track of state associated with deployed configuration changes so that they can be undone
  6. The intervalHandler() function is used to automatically release controls after 10 seconds - the timeout is short for the purposes of demonstration, in practical deployments the timeout would be much measured in hours
  7. For simplicity, this script is missing the error handling needed for production use. 
  8. See Writing Applications for more information.
We are going to use hping3 to simulate a DDoS attack, so install the software using the following command:
sudo apt install hping3
Run the following command to start sFlow-RT and run the ddos.js script:
env RTPROP=-Dscript.file=ddos.js ./
Next, start Mininet with ONOS:
sudo mn --custom ~/onos/tools/dev/mininet/,sflow-rt/extras/ \
--link tc,bw=10 --controller onos,1 --topo tree,2,2
Generate normal traffic between hosts h1 and h3:
mininet-onos> iperf h1 h3
The weathermap view above shows the flow crossing the network from switch s2 to s3 via s1.
Next, launch the simulated DNS amplification attack from h1 to h3:
mininet-onos> h1 hping3 --flood --udp -k -s 53 h3
The weathermap view verifies that the attack has been successfully blocked since none of the traffic is seen traversing the network.

The chart at the top of this article shows the iperf test followed by the simulated attack. The top chart shows the top flows entering the network, showing the DNS amplification attack traffic in blue. The middle chart shows traffic broken out by switch port. Here, the blue line shows the attack traffic arriving at switch s2 port s2-eth1 while the orange line shows that only a small amount of traffic is forwarded to switch s3 port s3-eth3 before the attack is blocked at switch s2 by the controller.

Mininet with ONOS and sFlow-RT is a great way to rapidly develop and test SDN applications, avoiding the time and expense involved in setting up a physical network. The application is easily moved from the Mininet virtual network to a physical network since it is based on the same industry standard sFlow telemetry generated by physical switches. In this case, using commodity switch hardware to cost effectively detect and filter massive (100's of Gbit/s) DDoS attacks.

Wednesday, April 4, 2018

ONOS traffic analytics

Open Network Operating System (ONOS) is "a software defined networking (SDN) OS for service providers that has scalability, high availability, high performance, and abstractions to make it easy to create applications and services." The open source project is hosted by the Linux Foundation.

Mininet and workflow describes how to run ONOS using the Mininet network emulator. Mininet allows virtual networks to be quickly constructed and is a simple way to experiment with ONOS. In addition, Mininet flow analytics describes how to enable industry standard sFlow streaming telemetry in Mininet, proving a simple way monitor traffic in the ONOS controlled network.

For example, the following command creates a Mininet network, controlled by ONOS, and monitored using sFlow:
sudo mn --custom ~/onos/tools/dev/mininet/,sflow-rt/extras/ \
--link tc,bw=10 --controller onos,1 --topo tree,2,2
The screen capture above shows the network topology in the ONOS web user interface.
Install Mininet dashboard to visualize the network traffic. The screen capture above shows a large flow over the same topology being displayed by ONOS, see Mininet weathermap for more examples.

In this case, the traffic was created by the following Mininet command:
mininet-onos> iperf h1 h3
The screen capture above shows top flows, busiest switch ports, and the diameter of the network topology.

The Mininet dashboard is a simple application running on the sFlow-RT analytics platform. For a more realistic example, watch the demonstration of SDN leaf and spine traffic engineering recorded at the Open Networking Summit. In the demonstration, a redundant pair of ONOS controllers implement segment routing, using OpenFlow 1.3 to control an eight switch leaf and spine network of commodity switches. Real-time flow analytics drives the dashboards in the demonstration and trigger load balancing of flows across the fabric. Leaf and spine traffic engineering using segment routing and SDN provides a more detailed explanation.

Mininet with ONOS and sFlow-RT is a great way to rapidly develop and test SDN applications, avoiding the time and expense involved in setting up a physical network.

Tuesday, April 3, 2018

Real-time baseline anomaly detection

The screen capture demonstrates the real-time baseline and anomaly detection based on industry standard sFlow streaming telemetry. The chart was generated using sFlow-RT analytics software. The blue line is an up to the second measure of traffic (measured in Bits per Second). The red and gold lines represent dynamic upper and lower limits calculated by the baseline function. The baseline function flags "high" and "low" value anomalies when values move outside the limits. In this case, a "low" value anomaly was flagged for the drop in traffic shown in the chart.

Writing Applications provides a general introduction to sFlow-RT programming. The baseline functionality is exposed through through the JavaScript API.

Create new baseline
  • name, name used to reference baseline.
  • window, the number of previous intervals to consider in calculating the limits.
  • sensitivity, the number of standard deviations used to calculate the limits.
  • repeat, the number of successive data points outside the limits before flagging anomaly 
In this example, baseline parameter values were window=180 (seconds), sensitivity=2, and repeat=3.

Update baseline
var status = baselineCheck(name,value);
  • status, "learning" while baseline is warming up (takes window intervals),  "normal" if value is in expected range, "low" if value is exceptionally low, "high" if value is exceptionally high.
  • value, latest value to check against baseline
The baselineCheck function is called periodically to update baseline statistics and check for anomalies.

Query baseline statistics
var {mean,variance,sdev,min,max} = baselineStatistics(name);
Note: Statistics are only available once the baseline has exited the "learning" status.

Reset baseline
Resets the statistics and sets state to "learning"

Delete baseline
Delete the baseline and free up associated resources.

The sFlow-RT baseline functionality is designed to be resource efficient and to converge quickly so that large numbers of baselines can be created and updated for real-time anomaly detection.

The baseline functions work best when the variable being tracked represents the activity of a large population and is relatively stable. For example, WAN traffic is generally a good candidate for baselining since it is composed of the activity of many systems and users. On the other hand, individual host activity tends to be highly variable and not well suited to baseline monitoring.
The table from Baseline contrasts two methods of baseline calculation. The baseline functionality described in this article is an example of a temporal baseline. Cluster performance metrics describes how sFlow-RT can be used to calculate statistics from large populations of devices. These functions can be used for spatial baselining and anomaly detection, for example, by finding a virtual machine in a service pool that is behaving inconsistently when compared to its peers.

Monday, April 2, 2018

Flow smoothing

The sFlow-RT real-time analytics engine includes statistical smoothing. The chart above illustrates the effect of different levels of smoothing when analyzing real-time sFlow telemetry.

The traffic generator in this example creates an alternating pattern: 1.25Mbytes/second for 30 seconds followed by a pause of 30 seconds. Smoothing time constants between 1 second and 500 seconds have been applied to generate the family of charts. The blue line is the result of 1 second smoothing and closely tracks the traffic pattern. At the other extreme, the dark red line is the result of 500 second smoothing, showing a constant 625Kbytes/second (the average of the waveform).

There is a tradeoff between responsiveness and variability (noise) when selecting the level of smoothing. Selecting a suitable smoothing level depends on the flow analytics application.

Low smoothing values are appropriate when fast response is required, for example:
Higher smoothing values are appropriate when less variability is desirable, for example:

Generating the chart

The results described in this article are easily reproduced using the testbed described in Mininet flow analytics.

The following, smooting.js, script defines a set of flows with different smoothing periods:
var times = [1,2,5,10,20,50,100,200,500];
for(var i = 0; i < times.length; i++) {
Start sFlow-RT:
env RTPROP=-Dscript.file=smoothing.js ./
Start Mininet:
sudo mn --custom extras/ --link tc,bw=10
Type the following Mininet command to open terminals on simulated hosts, h1 and h2:
mininet> xterm h1 h2
In h2 terminal window:
iperf -s
In h1 terminal window:
while true; do iperf -c -t 30; sleep 30; done
Plot the chart by opening the sFlow-RT URL:
See Writing Applications for more information.

Thursday, March 29, 2018

Mininet weathermap

Mininet dashboard is a real-time dashboard displaying traffic information from Mininet virtual networks. The screen capture demonstrates the real-time network weather map capability that was recently added to the dashboard. The torus topology is displayed and link widths are updated every second to reflect traffic. In this example a large flow between switches s1x1 and s3x3 is routed via s1x3.

The network was created using the following Mininet command:
sudo mn --custom=sflow-rt/extras/ --link tc,bw=10 \
--topo torus,3,3 --switch ovsbr,stp=1 --test iperf
In the screen capture above you can clearly see the large flow traversing switches, s4, s3, s2, s1, s9, s13, and s15 in a tree topology. The network was created using the following command:
sudo mn --custom sflow-rt/extras/ --link tc,bw=10 \
--topo tree,depth=4,fanout=2 --test iperf
The screen capture above shows a large flow traversing switches s1, s2, s3, and s4 in a linear topology. The network was created using the following command:
sudo mn --custom sflow-rt/extras/ --link tc,bw=10 \
--topo linear,4 --test iperf
It's also easy to create Custom Topologies. The following command creates the example custom topology,, that ships with Mininet:
sudo mn --custom ~/mininet/custom/,sflow-rt/extras/ \
--link tc,bw=10 --topo mytopo --test iperf
The final screen capture shows all three tests, identifying top flows, busiest switch ports, and showing the diameter of the network topology. This test run demonstrates the efficiency of using Mininet; creating, testing, and destroying three virtual networks, toroid (9 switches, 9 hosts, 27 links), tree (15 switches, 16 hosts, 30 links), linear (4 switches, 4 hosts, 7 links), in a few minutes, on a laptop.

For anyone curious about the technologies underpinning these examples, Mininet is a Python program that uses Open vSwitch to connect lightweight Linux containers representing hosts and switches. Open vSwitch includes industry standard sFlow instrumentation that efficiently streams telemetry to sFlow-RT real-time analytics software running the mininet-dashboard application. Mininet-dashboard uses the vis.js library to create the dynamic network diagrams.

Mininet's support of sFlow provides the same measurement technology used in physical switches, from low cost edge switches to high end chassis routers, including: A10, Aerohive, ALUe, Allied Telesis, Arista, Aruba, Big Switch, Cisco, Cumulus, Dell, D-Link, Edge-Core, Extreme, F5, Fortinet, Huawei, IP Infusion, Juniper, Netgear, OpenSwitch, Pica8, Proxim, Quanta, SMC, ZTE, and ZyXEL. A common measurement protocol makes it easy to move between emulated and physical networks, for example, Large flow detection describes how to scale sFlow configuration settings from emulated 10Mbit/s Mininet networks to 10G, 40G and 100G physical networks, and SC16 live real-time weathermaps uses the technologies described in this article to monitor a large 100Gbit/s network.

Thursday, March 22, 2018

OCP Summit 2018

Network telemetry was a popular topic at the recent OCP U.S. Summit 2018 in San Jose, California, with an entire afternoon track of the two day conference devoted to the subject. Videos of the talks should soon be posted on the conference web site.

The following articles on this blog cover related topics:
In addition, there were a couple of live sFlow telemetry demonstration in the conference exhibit hall.
The first was a demonstration of leaf and spine fabric visibility using white box switches running the open source Linux Foundation OpenSwitch network operating system. OpenSwitch describes how the open source Host sFlow agent enables standard sFlow instrumentation in merchant silicon based white box switches using OpenSwitch Control Plane Services (CPS), which in turn programs the silicon using the OCP Switch Abstraction Interface (SAI).

The rack in the booth contains a two spine, five leaf network. Each of the switches in the network is streaming real-time sFlow telemetry to an instance of Fabric View which is displaying real-time (up to the second) flow analytics on the monitor to the right of the picture.
The second demonstration shows a Marvell top of rack (ToR) switch with an ARM based management CPU running Linux, DPDK and SONiC. In this example, the switch is streaming sFlow telemetry to a free version of sFlowTrend which is displaying top flows in the trend chart.

Monday, March 19, 2018

Prometheus and Grafana

Prometheus is an open source time series database optimized to collect large numbers of metrics from cloud infrastructure. This article will explore how industry standard sFlow telemetry streaming supported by network devices and Host sFlow agents (Linux, Windows, FreeBSD, AIX, Solaris, Docker, Systemd, Hyper-V, KVM, Nutanix AHV, Xen) can be integrated with Prometheus.

The diagram above shows the elements of the solution: sFlow telemetry streams from hosts and switches to an instance of sFlow-RT. The sFlow-RT analytics software converts the raw measurements into metrics that are accessible through a REST API.

The following prometheus.php script mediates between the Prometheus metrics export protocol and the sFlow-RT REST API.  HTTP queries from Prometheus are translated into calls to the sFlow-RT REST API and JSON responses are converted into Prometheus metrics.
header('Content-Type: text/plain');
if(isset($_GET['labels'])) {
  $keys = htmlspecialchars($_GET["labels"]);
$vals = htmlspecialchars($_GET["values"]);
if(isset($keys)) {
  $cols = $keys.','.$vals;
} else {
  $cols = $vals;
$key_arr = explode(",",$keys);
$result = file_get_contents('http://localhost:8008/table/ALL/'.$cols.'/json');
$obj = json_decode($result,true);
foreach ($obj as $row) {
  foreach ($row as $cell) {
    if(!isset($labels)) {
      $labels = 'agent="'.$cell['agent'].'",datasource="'.$cell['dataSource'].'"';
    $name = $cell['metricName'];
    $val = $cell['metricValue'];
    if(in_array($name,$key_arr)) {
      $labels .= ','.$name.'="'.$val.'"';
    } else {
      print $name."{".$labels."} ".$val."\n";
Install prometheus.php under the web server home directory (e.g. /var/www/html) on the system running sFlow-RT and verify functionality using cURL:
$ curl "http://localhost/prometheus.php?labels=host_name&values=load_one"
load_one{agent="",datasource="2.1",host_name="server3"} 0.06
load_one{agent="",datasource="2.1",host_name="server2"} 0
load_one{agent="",datasource="2.1",host_name="spine1"} 0
load_one{agent="",datasource="2.1",host_name="spine2"} 0
load_one{agent="",datasource="2.1",host_name="leaf1"} 0
load_one{agent="",datasource="2.1",host_name="leaf2"} 0
Define metrics "scraping" jobs in the Prometheus configuration file, prometheus.yml. In this example sFlow-RT is running on host and two sets of metrics have been defined using prometheus.php interface. The sflow-rt-hosts job retrieves host metrics labeled by host_name and the sflow-rt-ifstats job retrieves network interface metrics labeled by host_name and ifname. The metric values in this example are a small selection from the extensive set of standard sFlow metrics available from sFlow-RT, see Metrics.
  scrape_interval:     15s
  evaluation_interval: 15s

  # - "first.rules"
  # - "second.rules"

  - job_name: 'sflow-rt-hosts'
    scrape_interval: 30s
    metrics_path: /prometheus.php
      labels: ['host_name']
      values: ['load_one,cpu_utilization,proc_run']
      - targets: ['']
  - job_name: 'sflow-rt-ifstats'
    scrape_interval: 30s
    metrics_path: /prometheus.php
      labels: ['host_name,ifname']
      values: ['ifinutilization,ifoututilization,ifindiscards,ifoutdiscards']
      - targets: ['']
Start Prometheus. For example, the following command shows how to run Prometheus under Docker:
docker run --name prometheus --rm -v $PWD/data:/prometheus \
-v $PWD/prometheus.yml:/etc/prometheus/prometheus.yml \
-p 9090:9090 prom/prometheus
The screen capture above shows the Prometheus web interface, accessible on port 9090.
Grafana is open source time series analysis software. The ability to pull data from many data sources and the extensive range of charting options makes Grafana an attractive tool for building operations dashboards.

The following command shows how to run Grafana under Docker:
docker run --name grafana --rm -v $PWD/data:/var/lib/grafana \
-p 3000:3000 grafana/grafana
Access the Grafana web interface on port 3000, configure a data source for the Prometheus database, and start building dashboards. The screen capture above shows the same chart built earlier using the native Prometheus interface.

Standard sFlow telemetry provides a unified method of monitoring large scale network and compute infrastructure. This example focussed on sFlow counter metrics, but sFlow also provides real-time flow information that can be used to generate flow metrics, for example, reporting on interactions between microservices.

Additional examples on this blog include:

Friday, February 2, 2018


OpenSwitch is a Linux Foundation project providing an open source white box control plane running on a standard Linux distribution. The diagram above shows the OpenSwitch architecture.

This article describes how to enable industry standard sFlow telemetry using the open source Host sFlow agent. The Host sFlow agent uses Control Plane Services (CPS) to configure sFlow instrumentation in the hardware and gather metrics. CPS in turn uses the Open Compute Project (OCP) Switch Abstraction Interface (SAI) as a vendor independent method of configuring the hardware. Hardware support for sFlow is a standard feature supported by Network Processing Unit (NPU) vendors (Barefoot, Broadcom, Cavium, Innovium, Intel, Marvell, Mellanox, etc.) and vendor neutral sFlow configuration is part of the SAI.

Installing and configuring Host sFlow agent

Installing the software is simple. Log into the switch and type the following commands:
wget --no-check-certificate
sudo dpkg -i hsflowd-opx_2.0.17-1_amd64.deb
The sFlow agent requires very little configuration, automatically monitoring all switch ports using the following default settings:

Link SpeedSampling RatePolling Interval
1 Gbit/s1-in-1,00030 seconds
10 Gbit/s1-in-10,00030 seconds
25 Gbit/s1-in-25,00030 seconds
40 Gbit/s1-in-40,00030 seconds
50 Gbit/s1-in-50,00030 seconds
100 Gbit/s1-in-100,00030 seconds

Note: The default settings ensure that large flows (defined as consuming 10% of link bandwidth) are detected within approximately 1 second - see Large flow detection

Edit the /etc/hsflowd.conf file to specify the address of an sFlow analyzer (
sflow {
  collector { ip = }
Monitoring Linux services describes how configure Host sFlow to include detailed telemetry for all services running on OpenSwitch:
  • bind9.service
  • cron.service
  • dbus.service
  • getty@tty1.service
  • getty@tty2.service
  • getty@tty3.service
  • getty@tty4.service
  • getty@tty5.service
  • getty@tty6.service
  • hsflowd.service
  • lldpd.service
  • networking.service
  • opx-alms.service
  • opx-cps.service
  • opx-front-panel-ports.service
  • opx-ip.service
  • opx-monitor-phy-media.service
  • opx-nas-shell.service
  • opx-nas.service
  • opx-nbmgr.service
  • opx-phy-media-config.service
  • opx-tmpctl.service
  • polkitd.service
  • redis-server.service
  • rsyslog.service
  • snmpd.service
  • ssh.service
  • systemd-journald.service
  • systemd-logind.service
  • systemd-udevd.service

Finally, start the Host sFlow agent:
sudo systemctl enable hsflowd
sudo systemctl start hsflowd
Using the Host sFlow agent to monitor Linux servers and switches provides a consistent set of measurements end-to-end, particularly for cloud infrastructure such as OpenStack and Docker where the network extends into the servers in the form of virtual switches and routers.

Collecting and analyzing sFlow

Visibility and the software defined data center describes the general architecture of sFlow monitoring. Standard sFlow agents embedded within the elements of the infrastructure stream essential performance metrics to management tools, ensuring that every resource in a dynamic cloud infrastructure is immediately detected and continuously monitored.

The Host sFlow agent on OpenSwitch streams standard Linux performance statistics in addition to the interface counters and packet samples that you would typically get from a networking device.
Note: Enhanced visibility into host performance is particularly important on open switch platforms since they may be running a number of user installed services that can stress the limited CPU, memory and IO resources.
For example, the following sflowtool output shows the raw data contained in an sFlow datagram:
startDatagram =================================
datagramSize 1332
unixSecondsUTC 1516946395
datagramVersion 5
agentSubId 100000
packetSequenceNo 340132
sysUpTime 17479000
samplesInPacket 7
startSample ----------------------
sampleType_tag 0:2
sampleSequenceNo 876
sourceId 2:1
counterBlock_tag 0:2001
counterBlock_tag 0:2005
disk_total 8102721536
disk_free 5178248192
disk_partition_max_used 37.77
disk_reads 25339
disk_bytes_read 562041856
disk_read_time 25380
disk_writes 3192551
disk_bytes_written 28776890368
disk_write_time 1043712
counterBlock_tag 0:2004
mem_total 2107891712
mem_free 142082048
mem_shared 0
mem_buffers 155873280
mem_cached 1611935744
swap_total 0
swap_free 0
page_in 184268
page_out 9367478
swap_in 0
swap_out 0
counterBlock_tag 0:2003
cpu_load_one 0.010
cpu_load_five 0.030
cpu_load_fifteen 0.000
cpu_proc_run 2
cpu_proc_total 167
cpu_num 2
cpu_speed 2699
cpu_uptime 3541814
cpu_user 3336490
cpu_nice 0
cpu_system 5479320
cpu_idle 2754958964
cpu_wio 168960
cpuintr 160
cpu_sintr 2717250
cpuinterrupts 656232310
cpu_contexts 1704273704
cpu_steal 0
cpu_guest 0
cpu_guest_nice 0
counterBlock_tag 0:2006
nio_bytes_in 267777
nio_pkts_in 4210
nio_errs_in 0
nio_drops_in 0
nio_bytes_out 2104528
nio_pkts_out 2227
nio_errs_out 0
nio_drops_out 0
counterBlock_tag 0:2000
hostname opx2_vm
UUID 40-d4-8b-d5-6b-29-4e-4a-be-48-d6-55-8d-f6-81-73
machine_type 3
os_name 2
os_release 3.16.0-4-amd64
endSample   ----------------------
startSample ----------------------
sampleType_tag 0:2
sampleSequenceNo 876
sourceId 0:44
counterBlock_tag 0:1005
ifName e101-001-0
counterBlock_tag 0:1
ifIndex 3
networkType 6
ifSpeed 0
ifDirection 2
ifStatus 0
ifInOctets 0
ifInUcastPkts 0
ifInMulticastPkts 0
ifInBroadcastPkts 0
ifInDiscards 0
ifInErrors 0
ifInUnknownProtos 4294967295
ifOutOctets 0
ifOutUcastPkts 0
ifOutMulticastPkts 0
ifOutBroadcastPkts 0
ifOutDiscards 0
ifOutErrors 0
ifPromiscuousMode 0
endSample   ----------------------
startSample ----------------------
sampleType_tag 0:1
sampleSequenceNo 1022129
sourceId 0:7
meanSkipCount 128
samplePool 130832512
dropEvents 0
inputPort 7
outputPort 10
flowBlock_tag 0:1
flowSampleType HEADER
headerProtocol 1
sampledPacketSize 1518
strippedBytes 4
headerLen 128
headerBytes 6C-64-1A-00-04-5E-E8-E7-32-77-E2-B5-08-00-45-00-05-DC-63-06-40-00-40-06-9E-21-0A-64-0A-97-0A-64-14-96-9A-6D-13-89-4A-0C-4A-42-EA-3C-14-B5-80-10-00-2E-AB-45-00-00-01-01-08-0A-5D-B2-EB-A5-15-ED-48-B7-34-35-36-37-38-39-30-31-32-33-34-35-36-37-38-39-30-31-32-33-34-35-36-37-38-39-30-31-32-33-34-35-36-37-38-39-30-31-32-33-34-35-36-37-38-39-30-31-32-33-34-35-36-37-38-39-30-31-32-33-34-35
dstMAC 6c641a00045e
srcMAC e8e73277e2b5
IPSize 1500
ip.tot_len 1500
IPProtocol 6
TCPSrcPort 39533
TCPDstPort 5001
TCPFlags 16
endSample   ----------------------
Note: The Linux host metrics (red), network interface counters (green), and packet sample information (blue) have been highlighted.

sflowtool has a number of additional uses:
  • Verifying that sFlow is being received correctly at the destination
  • Converting binary sFlow data into ASCII for scripted analysis (Python, Perl etc.)
  • Converting sFlow into IPFIX/NetFlow
  • Converting sFlow into PCAP format for use with tcpdump, Wireshark, etc.
  • Replicate sFlow streams for multiple collectors
  • Source code for sFlow decoder that can be used to build custom sFlow analyzer
In addition to sflowtool, there are many other open source and commercial sFlow collectors listed on

A key feature of sFlow telemetry is the low latency network-wide visibility that is possible because of the stateless nature of the measurements. Comprehensive real-time visibility is an essential building block that provides feedback for operations, automation, and control. Articles on this blog use the sFlow-RT analyzer to demonstrate use cases for real-time telemetry, including:
The programmability of an open Linux network operating system combined with real-time visibility is transformative, providing the foundation services necessary for solutions that automatically adapt the network to changing demands.

Friday, January 26, 2018

Intranet DDoS attacks

As on a Darkling Plain: Network Survival in an Age of Pervasive DDoS talk by Steinthor Bjarnason at the recent NANOG 71 conference. The talk discusses the threat that the proliferation of network connected devices in enterprises create when they are used to launch denial of service attacks. Last year's Mirai attacks are described, demonstrating the threat posed by mixed mode attacks where a compromised host is used to infect large numbers devices on the corporate network.
The first slide from the talk shows a denial attack launched against an external target, launched from infected video surveillance cameras scattered throughout the the enterprise network. The large volume of traffic fills up external WAN link and overwhelms stateful firewalls.
The second slide shows an attack targeting critical internal services that can have been identified by reconnaissance from the compromised devices. In addition, scanning activity associated with reconnaissance for additional devices can itself overload internal resources and cause outages.

In both cases, most of the critical activity occurs behind the corporate firewall, making it extremely challenging to detect and mitigate these threats.

The talk discusses a number of techniques that service providers use to secure their networks that enterprises will need to adopt in order to meet this challenge. In particular, "utilizing flow telemetry to analyze external and internal traffic. This is necessary for attack detection, classification and traceback."

Instrumentation needs to be built into every network device in order to provide the comprehensive visibility required to address these challenges. sFlow is a scaleable streaming telemetry solution built into a wide variety of devices, from low cost edge switches to high end chassis routers. Network vendors that support sFlow include: A10, Aerohive, AlexalA, ALUe, Allied Telesis, Arista, Aruba, Big Switch, Brocade, Cisco, Cumulus, DCN, Dell, D-Link, Edge-Core, Enterasys, Extreme, F5, Fortinet, HPE, Hitachi, Huawei, IBM, IP Infusion, Juniper, NEC, Netgear, OpenSwitch, Open vSwitch, Oracle, Pica8, Plexxi, Pluribus, Proxim, Quanta, Silicom, SMC, ZTE, and ZyXEL.

Selecting devices that support sFlow simplifies operations by ensuring that the visibility needed to effectively manage the network is integrated into the fabric and deployed pervasively. Attempting to add visibility later is complex, expensive, and results in limited coverage.

There are a number of examples of DDoS mitigation using sFlow on this blog. While many of the examples focus on external DDoS attacks, the techniques are equally applicable to the internal network.