Monday, April 9, 2018

SDKLT

Logical Table Software Development Kit (SDKLT) is a new, powerful, and feature rich Software Development Kit (SDK) for Broadcom switches. SDKLT provides a new approach to switch configuration using Logical Tables.

Building the Demo App describes how to get started using a simulated Tomahawk device. Included, is a CLI that can be used to explore tables. For example, the following CLI output shows the attributes of the sFlow packet sampling table:
BCMLT.0> lt list -d MIRROR_PORT_ENCAP_SFLOW
MIRROR_PORT_ENCAP_SFLOW
  Description: The MIRROR_PORT_ENCAP_SFLOW logical table is used to specify
               per-port sFlow encapsulation sample configuration.
  11 fields (1 key-type field):
    SAMPLE_ING_FLEX_RATE
        Description: Sample ingress flex sFlow packet if the generated sFlow random
                     number is greater than the threshold. A lower threshold leads to
                     higher sampling frequency.
    SAMPLE_EGR_RATE
        Description: Sample egress sFlow packet if the generated sFlow random number is
                     greater than the threshold. A lower threshold leads to
                     higher sampling frequency.
    SAMPLE_ING_RATE
        Description: Sample ingress sFlow packet if the generated sFlow random number is
                     greater than the threshold. A lower threshold leads to
                     higher sampling frequency.
    SAMPLE_ING_FLEX_MIRROR_INSTANCE
        Description: Enable to copy ingress flex sFlow packet samples to the ingress
                     mirror member using the sFlow mirror instance configuration.
    SAMPLE_ING_FLEX_CPU
        Description: Enable to copy ingress flex sFlow packet samples to CPU.
    SAMPLE_ING_MIRROR_INSTANCE
        Description: Enable to copy ingress sFlow packet samples to the ingress
                     mirror member using the sFlow mirror instance configuration.
    SAMPLE_ING_CPU
        Description: Enable to copy ingress sFlow packet samples to CPU.
    SAMPLE_ING_FLEX
        Description: Enable to sample ingress port-based flex sFlow packets.
    SAMPLE_EGR
        Description: Enable to sample egress port-based sFlow packets.
    SAMPLE_ING
        Description: Enable to sample ingress port-based sFlow packets.
    PORT_ID
        Description: Logical port ID.
SDKLT is a part of the OpenNSL suite, which makes it possible for the development of open network operating system projects, including: Open Network Linux, OpenSwitch, and SONiC.
The network operating system bridges the gap between applications (BGP, SNMP, sFlow, etc.) and the low level hardware capabilities accessed through the SDK. For example, OpenSwitch describes how the open source Host sFlow agent uses Control Plane Services (CPS) and Open Compute Project (OCP) Switch Abstraction Interface (SAI) to configure hardware packet sampling via vendor specific SDKs (such as OpenNSL).

Friday, April 6, 2018

sFlow available on Juniper MX series routers

sFlow support on MX Series devices—Starting in Junos OS Release 18.1R1, you can configure sFlow technology (as a sFlow agent) on a MX Series device, to continuously monitor traffic at wire speed on all interfaces simultaneously. The sFlow technology is a monitoring technology for high-speed switched or routed networks.  - New and Changed Features

Understanding How to Use sFlow Technology for Network Monitoring on a MX Series Router lists the following benefits of sFlow Technology on a MX Series Router:
  • sFlow can be used by software tools like a network analyzer to continuously monitor tens of thousands of switch or router ports simultaneously.
  • Since sFlow uses network sampling (forwarding one packet from ‘n’ number of total packets) for analysis, it is not resource intensive (for example processing, memory and more). The sampling is done at the hardware application-specific integrated circuits (ASICs) and hence it is simple and more accurate.
With the addition of the MX series routers, Juniper now supports sFlow across its entire product range:
Universal support for industry standard sFlow as a base Junos feature reduces the operational complexity and cost of network visibility for enterprises and service providers. Real-time streaming telemetry from campus switches, routers, and data center switches, provides centralized, real-time, end-to-end visibility needed to troubleshoot, optimize, and account for network usage.
Analytics software is a critical factor in realizing the full benefits of sFlow monitoring. Choosing an sFlow analyzer discusses important factors to consider when selecting from the range of open source and commercial sFlow analysis tools.

Thursday, April 5, 2018

ONOS measurement based control

ONOS traffic analytics describes how to run the ONOS SDN controller with a virtual network created using Mininet. The article also showed how to monitor network traffic using industry standard sFlow instrumentation available in Mininet and in physical switches.
This article uses the same ONOS / Mininet test bed to demonstrate how sFlow-RT real-time flow analytics can be used to push controls to the network through the ONOS REST API.  Leaf and spine traffic engineering using segment routing and SDN used real-time flow analytics to load balance an ONOS controlled physical network. In this example, we will use ONOS to filter DDoS attack traffic on a Mininet virtual network.

The following sFlow-RT script, ddos.js, detects DDoS attacks and programs ONOS filter rules to block the attacks:
var user = 'onos';
var password = 'rocks';
var onos = '192.168.123.1';
var controls = {};

setFlow('udp_reflection',
 {keys:'ipdestination,udpsourceport',value:'frames'});
setThreshold('udp_reflection_attack',
 {metric:'udp_reflection',value:100,byFlow:true,timeout:2});

setEventHandler(function(evt) {
 // don't consider inter-switch links
 var link = topologyInterfaceToLink(evt.agent,evt.dataSource);
 if(link) return;

 // get port information
 var port = topologyInterfaceToPort(evt.agent,evt.dataSource);
 if(!port) return;

 // need OpenFlow info to create ONOS filtering rule
 if(!port.dpid || !port.ofport) return;

 // we already have a control for this flow
 if(controls[evt.flowKey]) return;

 var [ipdestination,udpsourceport] = evt.flowKey.split(',');
 var msg = {
  flows: [
   {
    priority:4000,
    timeout:0,
    isPermanent:true,
    deviceId:'of:'+port.dpid,
    treatment:[],
    selector: {
     criteria: [
      {type:'IN_PORT',port:port.ofport},
      {type:'ETH_TYPE',ethType:'0x800'},
      {type:'IPV4_DST',ip:ipdestination+'/32'},
      {type:'IP_PROTO',protocol:'17'},
      {type:'UDP_SRC',udpPort:udpsourceport} 
     ]
    }
   }
  ]
 };

 var resp = http2({
  url:'http://'+onos+':8181/onos/v1/flows?appId=ddos',
  headers:{'Content-Type':'application/json','Accept':'application/json'},
  operation:'post',
  user:user,
  password:password,
  body: JSON.stringify(msg)
 });

 var {deviceId,flowId} = JSON.parse(resp.body).flows[0];
 controls[evt.flowKey] = {
  time:Date.now(),
  threshold:evt.thresholdID,
  agent:evt.agent,
  metric:evt.dataSource+'.'+evt.metric,
  deviceId:deviceId,
  flowId:flowId
 };

 logInfo("blocking " + evt.flowKey);
},['udp_reflection_attack']);

setIntervalHandler(function() {
 var now = Date.now();
 for(var key in controls) {
   let rec = controls[key];

   // keep control for at least 10 seconds
   if(now - rec.time < 10000) continue;
   // keep control if threshold still triggered
   if(thresholdTriggered(rec.threshold,rec.agent,rec.metric,key)) continue;

   var resp = http2({
    url:'http://'+onos+':8181/onos/v1/flows/'
        +encodeURIComponent(rec.deviceId)+'/'+encodeURIComponent(rec.flowId),
    headers:{'Accept':'application/json'},
    operation:'delete',
    user:user,
    password:password
   });

   delete controls[key];

   logInfo("unblocking " + key);
 }
});
Some notes on the script:
  1. The ONOS REST API is used to add/remove that filter out the DDoS traffic.
  2. The controller address, 192.168.123.1, can be found on the ONOS Cluster Nodes web page.
  3. The udp_reflection flow definition is designed to detect UDP amplification attacks, e.g. DNS amplification attacks
  4. Controls are applied to the switch port where traffic enters the network
  5. The controls structure is used to keep track of state associated with deployed configuration changes so that they can be undone
  6. The intervalHandler() function is used to automatically release controls after 10 seconds - the timeout is short for the purposes of demonstration, in practical deployments the timeout would be much measured in hours
  7. For simplicity, this script is missing the error handling needed for production use. 
  8. See Writing Applications for more information.
We are going to use hping3 to simulate a DDoS attack, so install the software using the following command:
sudo apt install hping3
Run the following command to start sFlow-RT and run the ddos.js script:
env RTPROP=-Dscript.file=ddos.js ./start.sh
Next, start Mininet with ONOS:
sudo mn --custom ~/onos/tools/dev/mininet/onos.py,sflow-rt/extras/sflow.py \
--link tc,bw=10 --controller onos,1 --topo tree,2,2
Generate normal traffic between hosts h1 and h3:
mininet-onos> iperf h1 h3
The weathermap view above shows the flow crossing the network from switch s2 to s3 via s1.
Next, launch the simulated DNS amplification attack from h1 to h3:
mininet-onos> h1 hping3 --flood --udp -k -s 53 h3
The weathermap view verifies that the attack has been successfully blocked since none of the traffic is seen traversing the network.

The chart at the top of this article shows the iperf test followed by the simulated attack. The top chart shows the top flows entering the network, showing the DNS amplification attack traffic in blue. The middle chart shows traffic broken out by switch port. Here, the blue line shows the attack traffic arriving at switch s2 port s2-eth1 while the orange line shows that only a small amount of traffic is forwarded to switch s3 port s3-eth3 before the attack is blocked at switch s2 by the controller.

Mininet with ONOS and sFlow-RT is a great way to rapidly develop and test SDN applications, avoiding the time and expense involved in setting up a physical network. The application is easily moved from the Mininet virtual network to a physical network since it is based on the same industry standard sFlow telemetry generated by physical switches. In this case, using commodity switch hardware to cost effectively detect and filter massive (100's of Gbit/s) DDoS attacks.

Wednesday, April 4, 2018

ONOS traffic analytics

Open Network Operating System (ONOS) is "a software defined networking (SDN) OS for service providers that has scalability, high availability, high performance, and abstractions to make it easy to create applications and services." The open source project is hosted by the Linux Foundation.

Mininet and onos.py workflow describes how to run ONOS using the Mininet network emulator. Mininet allows virtual networks to be quickly constructed and is a simple way to experiment with ONOS. In addition, Mininet flow analytics describes how to enable industry standard sFlow streaming telemetry in Mininet, proving a simple way monitor traffic in the ONOS controlled network.

For example, the following command creates a Mininet network, controlled by ONOS, and monitored using sFlow:
sudo mn --custom ~/onos/tools/dev/mininet/onos.py,sflow-rt/extras/sflow.py \
--link tc,bw=10 --controller onos,1 --topo tree,2,2
The screen capture above shows the network topology in the ONOS web user interface.
Install Mininet dashboard to visualize the network traffic. The screen capture above shows a large flow over the same topology being displayed by ONOS, see Mininet weathermap for more examples.

In this case, the traffic was created by the following Mininet command:
mininet-onos> iperf h1 h3
The screen capture above shows top flows, busiest switch ports, and the diameter of the network topology.


The Mininet dashboard is a simple application running on the sFlow-RT analytics platform. For a more realistic example, watch the demonstration of SDN leaf and spine traffic engineering recorded at the Open Networking Summit. In the demonstration, a redundant pair of ONOS controllers implement segment routing, using OpenFlow 1.3 to control an eight switch leaf and spine network of commodity switches. Real-time flow analytics drives the dashboards in the demonstration and trigger load balancing of flows across the fabric. Leaf and spine traffic engineering using segment routing and SDN provides a more detailed explanation.

Mininet with ONOS and sFlow-RT is a great way to rapidly develop and test SDN applications, avoiding the time and expense involved in setting up a physical network.

Tuesday, April 3, 2018

Real-time baseline anomaly detection

The screen capture demonstrates the real-time baseline and anomaly detection based on industry standard sFlow streaming telemetry. The chart was generated using sFlow-RT analytics software. The blue line is an up to the second measure of traffic (measured in Bits per Second). The red and gold lines represent dynamic upper and lower limits calculated by the baseline function. The baseline function flags "high" and "low" value anomalies when values move outside the limits. In this case, a "low" value anomaly was flagged for the drop in traffic shown in the chart.

Writing Applications provides a general introduction to sFlow-RT programming. The baseline functionality is exposed through through the JavaScript API.

Create new baseline
baselineCreate(name,window,sensitivity,repeat);
Where:
  • name, name used to reference baseline.
  • window, the number of previous intervals to consider in calculating the limits.
  • sensitivity, the number of standard deviations used to calculate the limits.
  • repeat, the number of successive data points outside the limits before flagging anomaly 
In this example, baseline parameter values were window=180 (seconds), sensitivity=2, and repeat=3.

Update baseline
var status = baselineCheck(name,value);
Where:
  • status, "learning" while baseline is warming up (takes window intervals),  "normal" if value is in expected range, "low" if value is exceptionally low, "high" if value is exceptionally high.
  • value, latest value to check against baseline
The baselineCheck function is called periodically to update baseline statistics and check for anomalies.

Query baseline statistics
var {mean,variance,sdev,min,max} = baselineStatistics(name);
Note: Statistics are only available once the baseline has exited the "learning" status.

Reset baseline
baselineReset(name);
Resets the statistics and sets state to "learning"

Delete baseline
baselineDelete(name);
Delete the baseline and free up associated resources.

The sFlow-RT baseline functionality is designed to be resource efficient and to converge quickly so that large numbers of baselines can be created and updated for real-time anomaly detection.

The baseline functions work best when the variable being tracked represents the activity of a large population and is relatively stable. For example, WAN traffic is generally a good candidate for baselining since it is composed of the activity of many systems and users. On the other hand, individual host activity tends to be highly variable and not well suited to baseline monitoring.
The table from Baseline contrasts two methods of baseline calculation. The baseline functionality described in this article is an example of a temporal baseline. Cluster performance metrics describes how sFlow-RT can be used to calculate statistics from large populations of devices. These functions can be used for spatial baselining and anomaly detection, for example, by finding a virtual machine in a service pool that is behaving inconsistently when compared to its peers.

Monday, April 2, 2018

Flow smoothing

The sFlow-RT real-time analytics engine includes statistical smoothing. The chart above illustrates the effect of different levels of smoothing when analyzing real-time sFlow telemetry.

The traffic generator in this example creates an alternating pattern: 1.25Mbytes/second for 30 seconds followed by a pause of 30 seconds. Smoothing time constants between 1 second and 500 seconds have been applied to generate the family of charts. The blue line is the result of 1 second smoothing and closely tracks the traffic pattern. At the other extreme, the dark red line is the result of 500 second smoothing, showing a constant 625Kbytes/second (the average of the waveform).

There is a tradeoff between responsiveness and variability (noise) when selecting the level of smoothing. Selecting a suitable smoothing level depends on the flow analytics application.

Low smoothing values are appropriate when fast response is required, for example:
Higher smoothing values are appropriate when less variability is desirable, for example:

Generating the chart

The results described in this article are easily reproduced using the testbed described in Mininet flow analytics.

The following, smooting.js, script defines a set of flows with different smoothing periods:
var times = [1,2,5,10,20,50,100,200,500];
for(var i = 0; i < times.length; i++) {
  setFlow('t_'+times[i],{value:'bytes',t:times[i]});
}
Start sFlow-RT:
env RTPROP=-Dscript.file=smoothing.js ./start.sh
Start Mininet:
sudo mn --custom extras/sflow.py --link tc,bw=10
Type the following Mininet command to open terminals on simulated hosts, h1 and h2:
mininet> xterm h1 h2
In h2 terminal window:
iperf -s
In h1 terminal window:
while true; do iperf -c 10.0.0.2 -t 30; sleep 30; done
Plot the chart by opening the sFlow-RT URL:
http://sflow-rt:8008/metric/ALL/t_1,t_2,t_5,t_10,t_20,t_50,t_100,t_200,t_500/html
See Writing Applications for more information.