Controlling large flows with OpenFlow

Performance aware software defined networking describes how sFlow and OpenFlow can be combined to create SDN applications such as  DDoS mitigation, load balancing large flows, and packet capture. This article takes a deep dive, describing how to build a test bed for developing performance aware SDN applications. The test bed is used to build and test a basic DDoS mitigation application and demonstrate how quickly the controller can adapt the network changing traffic conditions.

Test bed

Mininet uses Linux containers and Open vSwitch to allow realistic virtual networks of hosts and switches to be constructed using a virtual machine. Mininet is widely used by SDN researchers since it allows anyone with a laptop to build realistic network topologies and experiment with OpenFlow controllers and SDN applications.

The Floodlight OpenFlow Controller was selected for the test bed because its default behavior is to provide basic connectivity which can be selectively overridden using the Static Flow Pusher API. This separation allows simple performance optimizing applications to be developed since they don't need to concern themselves with maintaining connectivity and are free to focus on implementing optimizations. The Static Flow Pusher provides an example of hybrid OpenFlow (in which OpenFlow is used to selectively override forwarding decisions made by the normal switch forwarding logic). This makes it straightforward to move applications from the test bed to physical switches that support hybrid OpenFlow control. Finally, Floodlight is one of the most mature platforms used in production settings, so any applications developed on the test bed can be easily moved into production.

There are a number of ways to get started, both the Mininet and Floodlight projects distribute a pre-built virtual machine (VM) appliance (the Floodlight VM includes Mininet). Alternatively, it is straightforward to build an Ubuntu 13.04 virtual machine and install Mininet using apt-get (this is the route the author took to build a Mininet VM on a XenServer pool).

Once you have a system with Mininet and Floodlight installed, download and install sFlow-RT:

wget http://www.inmon.com/products/sFlow-RT/sflow-rt.tar.gz
tar -xvzf sflow-rt.tar.gz

And finally, this example will be using node.js as the application language. Node.js is well suited for implementing SDN applications. Its asynchronous IO model supports a high degree of parallelism, allowing the SDN application to interact with multiple controllers, monitoring systems, databases etc. without blocking, resulting in a fast and consistent response time that makes it well suited for control applications.

The following command installs node.js:

sudo apt-get install nodejs

For development, it is helpful to run each tool in a separate window so that you can see an logging messages (in a production setting processes would be daemonized).

1. Start Floodlight

cd floodlight
./floodlight.sh

2. Start Mininet, specifying Floodlight as the controller

sudo mn --controller=remote,ip=127.0.0.1
*** Creating network
*** Adding controller
*** Adding hosts:
h1 h2 
*** Adding switches:
s1 
*** Adding links:
(h1, s1) (h2, s1) 
*** Configuring hosts
h1 h2 
*** Starting controller
*** Starting 1 switches
s1 
*** Starting CLI:
mininet>

3. Configure sFlow monitoring on each of the switches:

sudo ovs-vsctl -- --id=@sflow create sflow agent=eth0  target=\"127.0.0.1:6343\" sampling=10 polling=20 -- -- set bridge s1 sflow=@sflow

4. Start sFlow-RT

cd sflow-rt
./start.sh

By default, Floodlight provides a basic layer 2 switching service, ensuring connectivity between hosts connected to the OpenFlow switches. Connectivity can be verified using the Mininet command line:

mininet> h1 ping h2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_req=1 ttl=64 time=36.7 ms
64 bytes from 10.0.0.2: icmp_req=2 ttl=64 time=0.159 ms

There are many options for using the Mininet test bed, previous articles on this blog have used Python to develop applications and different SDN controllers can be installed. For example, the PyTapDEMon series of articles describes the uses Python, POX (an OpenFlow controller written in Python) and Mininet to recreate the Microsoft DEMon SDN packet broker.

DDoS mitigation application

The following node.js script is based on the Python script described in Performance Aware Software Defined Networking.

var fs = require("fs");
var http = require('http');

var keys = 'inputifindex,ethernetprotocol,macsource,macdestination,ipprotocol,ipsource,ipdestination';
var value = 'frames';
var filter = 'sourcegroup=external&destinationgroup=internal&outputifindex!=discard';
var thresholdValue = 100;
var metricName = 'ddos';

// mininet mapping between sFlow ifIndex numbers and switch/port names
var ifindexToPort = {};
var nameToPort = {};
var path = '/sys/devices/virtual/net/';
var devs = fs.readdirSync(path);
for(var i = 0; i < devs.length; i++) {
  var dev = devs[i];
  var parts = dev.match(/(.*)-(.*)/);
  if(!parts) continue;

  var ifindex = fs.readFileSync(path + dev + '/ifindex');
  var port = {"switch":parts[1],"port":dev};
  ifindexToPort[parseInt(ifindex).toString()] = port;
  nameToPort[dev] = port;
}

var fl = { hostname: 'localhost', port: 8080 };

var groups = {'external':['0.0.0.0/0'],'internal':['10.0.0.2/32']};
var rt = { hostname: 'localhost', port: 8008 };
var flows = {'keys':keys,'value':value,'filter':filter};
var threshold = {'metric':metricName,'value':thresholdValue};

function extend(destination, source) {
  for (var property in source) {
    if (source.hasOwnProperty(property)) {
      destination[property] = source[property];
    }
  }
  return destination;
}

function jsonGet(target,path,callback) {
  var options = extend({method:'GET',path:path},target);
  var req = http.request(options,function(resp) {
    var chunks = [];
    resp.on('data', function(chunk) { chunks.push(chunk); });
    resp.on('end', function() { callback(JSON.parse(chunks.join(''))); });
  });
  req.end();
};

function jsonPut(target,path,value,callback) {
  var options = extend({method:'PUT',headers:{'content-type':'application/json'}
,path:path},target);
  var req = http.request(options,function(resp) {
    var chunks = [];
    resp.on('data', function(chunk) { chunks.push(chunk); });
    resp.on('end', function() { callback(chunks.join('')); });
  });
  req.write(JSON.stringify(value));
  req.end();
};

function jsonPost(target,path,value,callback) {
  var options = extend({method:'POST',headers:{'content-type':'application/json'},"path":path},target);
  var req = http.request(options,function(resp) {
    var chunks = [];
    resp.on('data', function(chunk) { chunks.push(chunk); });
    resp.on('end', function() { callback(chunks.join('')); });
  });
  req.write(JSON.stringify(value));
  req.end();
}

function lookupOpenFlowPort(agent,ifIndex) {
  return ifindexToPort[ifIndex];
}

function blockFlow(agent,dataSource,topKey) {
  var parts = topKey.split(',');
  var port = lookupOpenFlowPort(agent,parts[0]);
  if(!port || !port.dpid) return;
 
  var message = {"switch":port.dpid,
                 "name":"dos-1",
                 "ingress-port":port.portNumber.toString,
                 "ether-type":parts[1],
                 "protocol":parts[4],
                 "src-ip":parts[5],
                 "dst-ip":parts[6],
                 "priority":"32767",
                 "active":"true"};

  console.log("message=" + JSON.stringify(message));
  jsonPost(fl,'/wm/staticflowentrypusher/json',message,
      function(response) {
         console.log("result=" + JSON.stringify(response));
      });
}

function getTopFlows(event) {
  jsonGet(rt,'/metric/' + event.agent + '/' + event.dataSource + '.' + event.metric + '/json',
    function(metrics) {
      if(metrics && metrics.length == 1) {
        var metric = metrics[0];
        if(metric.metricValue > thresholdValue
           && metric.topKeys
           && metric.topKeys.length > 0) {
            var topKey = metric.topKeys[0].key;
            blockFlow(event.agent,event.dataSource,topKey);
        }
      }
    }
  );  
}

function getEvents(id) {
  jsonGet(rt,'/events/json?maxEvents=10&timeout=60&eventID='+ id,
    function(events) {
      var nextID = id;
      if(events.length > 0) {
        nextID = events[0].eventID;
        events.reverse();
        for(var i = 0; i < events.length; i++) {
          if(metricName == events[i].thresholdID) getTopFlows(events[i]);
        }
      }
      getEvents(nextID);  
    }
  );
}

// use port names to link dpid and port numbers from Floodlight
function getSwitches() {
  jsonGet(fl,'/wm/core/controller/switches/json',
    function(switches) { 
      for(var i = 0; i < switches.length; i++) {
        var sw = switches[i];
        var ports = sw.ports;
        for(var j = 0; j < ports.length; j++) {
          var port = nameToPort[ports[j].name];
          if(port) {
            port.dpid = sw.dpid;
            port.portNumber = ports[j].portNumber;
          }
        }
      }
      setGroup();
    }
  );
}

function setGroup() {
  jsonPut(rt,'/group/json',
    groups,
    function() { setFlows(); }
  );
}

function setFlows() {
  jsonPut(rt,'/flow/' + metricName + '/json',
    flows,
    function() { setThreshold(); }
  );
}

function setThreshold() {
  jsonPut(rt,'/threshold/' + metricName + '/json',
    threshold,
    function() { getEvents(-1); }
  ); 
}

function initialize() {
  getSwitches();
}

initialize();

The script should be fairly self explanatory to anyone familiar with JavaScript. The asynchronous style of programming in which the response to each call is handled by a callback function may be unfamiliar to non-Javascript programmers, but it is widely used in JavaScript and is the keys to node.js's low latency and ability to handle large numbers of concurrent requests. The sFlow-RT REST API calls and the basic logic for this script are explained in the Performance aware software defined networking article.

There are a couple of topics addressed in the script that warrant mention:

The OpenFlow protocol has its own way of identifying switches and ports on the network and an SDN application needs to be able to translate between the performance monitoring system's model of the network (identifying switches by their management IP addresses and ports by SNMP ifIndex numbers) and OpenFlow identifiers. Currently, there is no standard way to map between these two models and this deficiency needs to be addressed by the Open Networking Foundation, either through extensions to the configuration or OpenFlow protocols.

However, this script shows how to build these mappings in a Mininet environment by examining files in the /sys/devices/virtual/net directory and combining the information with data about the switches retrieved using Floodlight's /wm/core/controller/switches/json REST API call.

Finally, the sFlow data from Open vSwitch includes dropped packets. The sFlow-RT filter expression outputifindex!=discard is used to detect flows that aren't being blocked.

Results

This example uses a Ping Flood to demonstrate a basic denial of service attack.

The following Mininet command opens an terminal window connected to host h1:

mininet> xterm h1

Type the following command in the terminal to generate a ping flood between h1 and h2:

# ping -f 10.0.0.2

The sFlow-RT chart shows that without mitigation the ping flood generates a sustained traffic rate of around 6 thousand packets per second.

Next stop the ping flood attack and let the traffic settle down.

The following command runs the denial of service mitigation script:

nodejs mininet.js

Now start the ping flood again and see what happens.

The chart shows that the controller is able to respond quickly when the traffic flow exceeds the defined threshold of 100 packets per second. The mitigation control is applied within a second and instead of reaching a peak of 6 thousand packets per second, the attack is limited to a peak of 130 packets per second.

The ping flood attack is quickly detected by sFlow-RT, which notifies the mitigation application. The mitigation application retrieves details of the attack from the sFlow-RT in order to construct the following message, which is sent to the Floodlight's Static Flow Pusher:

message={"switch":"00:00:00:00:00:00:00:01","name":"dos-1","ether-type":"2048","protocol":"1","src-ip":"10.0.0.1","dst-ip":"10.0.0.2","priority":"32767","active":"true"}

The Floodlight controller then uses OpenFlow to push the rule to Open vSwitch which immediately starts dropping packets.

Note: The mitigation script doesn't automatically remove the control once the attack has been stopped, so the following command is needed to clear the controls on Floodlight:

curl http://localhost:8080/wm/staticflowentrypusher/clear/all/json

While far from a complete application, this example demonstrates that the sFlow and OpenFlow standard can be combined to build fast acting performance aware SDN applications that address important use cases, such as DDoS mitigation, large flow load balancing, multi-tenant performance isolation, traffic engineering, and packet capture. The Mininet platform provides a convenient way to develop, test and share applications addressing these use cases.