Sunday, August 27, 2017

Cumulus Linux 3.4 REST API

The latest Cumulus Linux 3.4 release include a REST API. This article will demonstrate how the REST API can be used to automatically deploy traffic controls based on real-time sFlow telemetry. DDoS mitigation with Cumulus Linux describes how sFlow-RT can detect Distributed Denial of Service (DDoS) attacks in real-time and deploy automated controls.

The following ddos.js script is modified to use the REST API to send Network Command Line Utility - NCLU commands to add and remove ACLs, see Installing and Managing ACL Rules with NCLU:
var user = "cumulus";
var password = "CumulusLinux!";
var thresh = 10000;
var block_minutes = 1;


setThreshold('attack',{metric:'udp_target', value:thresh, byFlow:true, timeout:10});

function restCmds(agent,cmds) {
  for(var i = 0; i < cmds.length; i++) {
    let msg = {cmd:cmds[i]};

var controls = {};
var id = 0;
setEventHandler(function(evt) {
  var key = evt.agent + ',' + evt.flowKey;
  if(controls[key]) return;

  var ifname = metric(evt.agent,evt.dataSource+".ifname")[0].metricValue;
  if(!ifname) return;

  var now = (new Date()).getTime();
  var name = 'ddos'+id++;
  var [ip,port] = evt.flowKey.split(',');
  var cmds = [
    'add acl ipv4 '+name+' drop udp source-ip any source-port '+port+' dest-ip '+ip+' dest-port any',
    'add int '+ifname+' acl ipv4 '+name+' inbound',
  controls[key] = {time:now, target: ip, port: port, agent:evt.agent, metric:evt.dataSource+'.'+evt.metric, key:evt.flowKey, name:name};
  try { restCmds(evt.agent, cmds); }
  catch(e) { logSevere('failed to add ACL, '+e); }
  logInfo('block target='+ip+' port='+port+' agent=' + evt.agent); 

setIntervalHandler(function() {
  var now = (new Date()).getTime();
  for(var key in controls) {
    if(now - controls[key].time < 1000 * 60 * block_minutes) continue;
    var ctl = controls[key];
    if(thresholdTriggered('attack',ctl.agent,ctl.metric,ctl.key)) continue;

    delete controls[key];
    var cmds = [
      'del acl ipv4 ',
    try { restCmds(ctl.agent,cmds); }
    catch(e) { logSevere('failed to remove ACL, ' + e); }
    logInfo('allow target='' port='+ctl.port+' agent='+ctl.agent);
The quickest way test the script is to use docker to run sFlow-RT:
docker run -v $PWD/ddos.js:/sflow-rt/ddos.js \
-e "RTPROP=-Dscript.file=ddos.js" \
-p 6343:6343/udp -p 8008:8008 sflow/sflow-rt
This solution can be tested using freely available software. The setup shown at the top of this article was constructed using a Cumulus VX virtual machine running on VirtualBox.  The Attacker and Target virtual machines are Linux virtual machines used to simulate the DDoS attack.

DNS amplification attack can be simulated using hping3. Run the following command on the Attacker host:
sudo hping3 --flood --udp -k -s 53
Run tcpdump on the Target host to see if the attack is getting through:
sudo tcpdump -i eth1 udp port 53
Each time an attack is launched a new ACL will be added that matches the attack signature and drops the traffic. The ACL is kept in place for at least block_minutes and removed once the attack ends. The following sFlow-RT log messages show the results:
2017-08-26T17:01:24+0000 INFO: Listening, sFlow port 6343
2017-08-26T17:01:24+0000 INFO: Listening, HTTP port 8008
2017-08-26T17:01:24+0000 INFO: ddos.js started
2017-08-26T17:03:07+0000 INFO: block target= port=53 agent=
2017-08-26T17:03:49+0000 INFO: allow target= port=53 agent=
REST API for Cumulus Linux ACLs describes the acl_server daemon that was used in the original article. The acl_server daemon is optimized for real-time performance, supporting use cases in which multiple traffic controls need to be quickly added and removed, e.g  DDoS mitigation, marking large flows, ECMP load balancing, packet brokers.

A key benefit of the openness of Cumulus Linux is that you can install software to suite your use case, other examples include: BGP FlowSpec on white box switchInternet router using Cumulus LinuxTopology discovery with Cumulus LinuxBlack hole detection, and Docker networking with IPVLAN and Cumulus Linux.