Frenetic, Pyretic and Resonance

Northbound APIs for traffic engineering describes some of the limitations with current OpenFlow controllers and describes some of the features needed to support traffic engineering applications. Looking for alternatives, I was excited to discover the Frenetic project, a collaborative effort between researchers at Princeton and Cornell to develop a language for developing SDN applications.

The Frenetic project has released Pyretic, an embedded implementation of Frenetic in Python. Looking further, I discovered that Pyretic is being used by the PyResonance project to implement a finite state machine (FSM) framework developed by researchers at the Georgia Institute of Technology. Resonance FSM provides a framework for an SDN controller to react to external events by changing forwarding policies. PyResonance expresses policies in the Frenetic language which can compose policies from multiple logical modules (for example forwarding, intrusion detection and access control) into OpenFlow rules which are then pushed to network switches.

This article uses the Mininet testbed described in Controlling large flows with OpenFlow to demonstrate how real-time sFlow measurements can be used to drive dynamic changes in the network using the PyResonance controller.

First install Pyretic. Next install PyResonance.

For development, it is helpful to run each tool in a separate window so that you can see an logging messages (in a production setting processes would be daemonized).

1. Start Mininet, specifying a remote controller

sudo mn --controller=remote --topo=single,3 --mac --arp

2. Start PyResonance

cd pyretic
./pyretic.py pyretic.pyresonance.resonance_simple -m i

3. Enable authorize each of the hosts

cd pyretic/pyretic/pyresonance
./sendy_json.py -i 10.0.0.1 -e auth -V authenticated
./sendy_json.py -i 10.0.0.2 -e auth -V authenticated
./sendy_json.py -i 10.0.0.3 -e auth -V authenticated

At this point all three hosts should be able to communicate. This can be verified by running the following command in the Mininet console:

mininet> pingall
*** Ping: testing ping reachability
h1 -> h2 h3 
h2 -> h1 h3 
h3 -> h1 h2

Note: It's interesting to play around with the forwarding authenticated policy. You can clear the authenticated state with the following command and run pingall again to verify that the host has been excluded from the network.

./sendy_json.py -i 10.0.0.3 -e auth -V clear

4. Configure sFlow monitoring on the vSwitch

sudo ovs-vsctl -- --id=@sflow create sflow agent=eth0  target=\"127.0.0.1:6343\" sampling=2 polling=20 -- -- set bridge s1 sflow=@sflow

Note: A low sampling rate of 1-in-2 was used because stability problems with the controller didn't allow realistic traffic levels to be used (see Results section below).

5. Start sFlow-RT

cd sflow-rt
./start.sh

DDoS mitigation application

The following node.js script is based on the script in Controlling large flows with OpenFlow.

var http = require('http');
var exec = require('child_process').exec;

var keys = 'ipsource';
var value = 'frames';
var filter = 'outputifindex!=discard&direction=ingress';
var thresholdValue = 4;
var metricName = 'ddos';

var rt = { hostname: 'localhost', port: 8008 };
var flows = {'keys':keys,'value':value,'filter':filter, t:2};
var threshold = {'metric':metricName,'value':thresholdValue, byFlow:true};

function extend(destination, source) {
  for (var property in source) {
    if (source.hasOwnProperty(property)) {
      destination[property] = source[property];
    }
  }
  return destination;
}

function jsonGet(target,path,callback) {
  var options = extend({method:'GET',path:path},target);
  var req = http.request(options,function(resp) {
    var chunks = [];
    resp.on('data', function(chunk) { chunks.push(chunk); });
    resp.on('end', function() { callback(JSON.parse(chunks.join(''))); });
  });
  req.end();
};

function jsonPut(target,path,value,callback) {
  var options = extend({method:'PUT',headers:{'content-type':'application/json'}
,path:path},target);
  var req = http.request(options,function(resp) {
    var chunks = [];
    resp.on('data', function(chunk) { chunks.push(chunk); });
    resp.on('end', function() { callback(chunks.join('')); });
  });
  req.write(JSON.stringify(value));
  req.end();
};

function sendy(address,type,state) {
  function callback(error, stdout, stderr) { };
  exec("../pyretic/pyretic/pyresonance/sendy_json.py -i " 
        + address + " -e " + type + " -V " + state, callback);  
}

function getEvents(id) {
  jsonGet(rt,'/events/json?maxEvents=10&timeout=60&eventID='+ id,
    function(events) {
      var nextID = id;
      if(events.length > 0) {
        nextID = events[0].eventID;
        events.reverse();
        var now = (new Date()).getTime();
        for(var i = 0; i < events.length; i++) {
          var evt = events[i];
          var dt = now - evt.timestamp;
          if(metricName == evt.thresholdID
            && Math.abs(dt) < 5000) {
            var flowKey = evt.flowKey;
            sendy(flowKey,"auth","clear");
          }
        }
      }
      getEvents(nextID);
    }
  );
}

function setFlows() {
  jsonPut(rt,'/flow/' + metricName + '/json',
    flows,
    function() { setThreshold(); }
  );
}

function setThreshold() {
  jsonPut(rt,'/threshold/' + metricName + '/json',
    threshold,
    function() { getEvents(-1); }
  ); 
}

function initialize() {
  setFlows();
}

initialize();

There are a few points worth noting:

1. This script is simpler than the original in Controlling large flows with OpenFlow. The application is no longer responsible for translating traffic measurements into concrete OpenFlow actions. Instead, the script expresses actions in terms of high level policy and the controller handles the translation of policy into specific OpenFlow actions.
2. The script currently calls the sendy_json.py script to implement policy changes. The Python script simply sends a json message over a TCP socket to the PyResonance controller and could easily be implemented in node.js, allowing the application to directly communicate with the controller. Alternatively, the controller could be modified to include a RESTful HTTP version of the API.
3. The low threshold value of 4 packets per second assigned because stability problems with the controller didn't allow realistic traffic levels to be used (see Results section below).

Results

The following command runs the denial of service mitigation script:

nodejs resonance.js

This example uses a Ping Flood to demonstrate a basic denial of service attack.

The following Mininet command opens a terminal window connected to host h1:

mininet> xterm h1

Start by generating small amount of traffic by typing the following command into the terminal to generate traffic between h1 and h2:

ping 10.0.0.2

Now generate a ping "flood" between h1 and h2:

ping -i 0.3 10.0.0.2

The chart shows that the controller is able to respond quickly when the traffic flow exceeds the defined threshold of 4 packets per second. The mitigation control is applied within a second, removing the host from the network.

The ping flood attack is quickly detected by sFlow-RT, which notifies the mitigation application, see Large flow detection script for a discussion of detection times and sampling rates. The mitigation application retrieves details of the attack from the sFlow-RT and executes the following policy change:

./sendy_json.py -i 10.0.0.1 -e auth -V clear

The PyResonance controller receives this message and the authorization state machine for host 10.0.0.1 is set to "not authenticated". This state change results in a new Frenetic policy blocking traffic from this address. The policy is translated into a new set of OpenFlow rules for the switch that drop packets from the blocked address, but still allow authenticated hosts to send traffic.

Note: The host can easily be re-authenticated by issuing the command:

./sendy_json.py -i 10.0.0.1 -e auth -V authenticated

While far from a complete application, this example demonstrates how the sFlow and OpenFlow standards can be combined to build fast acting performance aware SDN applications that address important use cases, such as DDoS mitigation, large flow load balancing, multi-tenant performance isolation, traffic engineering, and packet capture. The Mininet platform provides a convenient way to develop, test and share applications addressing these use cases.

The performance of the PyResonance controller was disappointing, limiting the tests to very low packet rates. Higher packet rates and large packet sizes quickly crash the controller. It isn't clear where the problem lies. Running a simple learning bridge in Pyretic is stable. The problem could be due to bugs in the PyResonance code, or PyResonance could be exposing bugs in Pyretic. It should be noted that the current version of Pyretic is a simple interpreter designed to test the Frenetic language and that the project plans to provide compilers that will proactively push rules to devices and deliver performance equivalent to custom built controllers, see Composing Software Defined Networks.  The Lithium controller project also looks interesting since they are building a controller based on finite state machines and policies that could also be a useful platform for traffic engineering. 

Aug. 29, 2013 Update: Joshua Reich, the developer of Pyretic, identified a known Python bug as the cause of the instability at high traffic levels and kindly provided a patch for Python 2.7. In addition, Hyojoon Kim, the developer of PyResonance, pointed out that the -m i option included in Step 2 of the instructions above puts the controller into reactive mode (which is much slower than proactive rule insertion). After applying the patch and dropping the -m i option, the PyResonance controller performance is now comparable to the previous tests that used the Floodlight controller, see Controlling large flows with OpenFlow. 

Oct. 1, 2013 Update: Embedding SDN applications repeats this experiment using sFlow-RT's embedding JavaScript API and demonstrates that the PyResonance controller performs at the same level as Floodlight on the Minet testbed, see Controlling large flows with OpenFlow. 

Northbound APIs for traffic engineering described some of the frustrations with current OpenFlow controllers and argued that higher levels of abstraction and mechanisms for composing policies from multiple SDN applications is critical to the long term success of software defined networking. It is exciting to see the researchers tackling these problems and it will be interesting to see how long it takes for these technologies to make their way into production quality controllers.