Thursday, July 13, 2017

Linux 4.11 kernel extends packet sampling support

Linux 4.11 on Linux Kernel Newbies describes the features added in the April 30, 2017 release. Of particular interest is the new netlink sampling channel:
Introduce psample, a general way for kernel modules to sample packets, without being tied to any specific subsystem. This netlink channel can be used by tc, iptables, etc. and allow to standardize packet sampling in the kernel commit
The psample netlink channel delivers sampled packet headers along with associated metadata from the Linux kernel to user space. The psample fields map directly into sFlow Version 5 sampled_header export structures:

netlink psamplesFlowDescription
PSAMPLE_ATTR_IIFINDEXinputInterface packet was received on.
PSAMPLE_ATTR_OIFINDEXoutputInterface packet was sent on.
PSAMPLE_ATTR_GROUP_SEQdropsNumber of times that the sFlow agent detected that a packet marked to be sampled was dropped due to lack of resources. Agent calculates drops by tracking discontinuities in PSAMPLE_ATTR_GROUP_SEQ
PSAMPLE_ATTR_SAMPLE_RATEsampling_rateThe Sampling Rate specifies the ratio of packets observed at the Data Source to the samples generated. For example a sampling rate of 100 specifies that, on average, 1 sample will be generated for every 100 packets observed.
PSAMPLE_ATTR_ORIGSIZEframe_lengthOriginal length of packet before sampling
PSAMPLE_ATTR_DATAheader<>Header bytes

Linux is widely used for switch network operating systems, including: Arista EOS, Cumulus Linux, Dell OS10, OpenSwitch, SONiC, and Open Network Linux. The adoption of Linux by network vendors and cloud providers is driving increased support for switch hardware by the Linux kernel community.

Hardware support for sFlow packet sampling is widely implemented in switch ASICs, including: Broadcom, Mellanox, Intel, Marvell, Barefoot Networks, Cavium, and Innovium. A standard Linux interface to ASIC sampling simplifies the implementation of sFlow agents (e.g. Host sFlow) and ensures consistent behavior across hardware platforms to deliver real-time network-wide visibility using industry standard sFlow protocol.

Wednesday, July 12, 2017

Arista eAPI

The sFlow and eAPI features of EOS (Extensible Operating System) are standard across the full range of Arista Networks switches. This article demonstrates how the real-time visibility provided by sFlow telemetry can be combined with the programmatic control of eAPI to automatically adapt the network to changing traffic conditions.

In the diagram, the sFlow-RT analytics engine receives streaming sFlow telemetry, provides real-time network-wide visibility, and automatically applies controls using eAPI to optimize forwarding, block denial of service attacks, or capture suspicious traffic.

Arista eAPI 101 describes the JSON RPC interface for programmatic control of Arista switches. The following eapi.js script shows how eAPI requests can be made using sFlow-RT's JavaScript API:
function runCmds(proto, agent, usr, pwd, cmds) {
  var req = {
    jsonrpc:'2.0',id:'sflowrt',method:'runCmds',
    params:{version:1,cmds:cmds,format:'json'}
  };
  var url = (proto || 'http')+'://'+agent+'/command-api';
  var resp = http(url,'post','application/json',JSON.stringify(req),usr,pwd);
  if(!resp) throw "no response";
  resp = JSON.parse(resp);
  if(resp.error) throw resp.error.message;
  return resp.result; 
}
The following test.js script demonstrates the eAPI functionality with a basic show request:
include('eapi.js');
var result = runCmds('http','10.0.0.90','admin','arista',['show hostname']);
logInfo(JSON.stringify(result));
Starting sFlow-RT:
env "RTPROP=-Dscript.file=test.js" ./start.sh
Running the script generates the following output:
2017-07-10T14:00:06-0700 INFO: Listening, sFlow port 6343
2017-07-10T14:00:06-0700 INFO: Listening, HTTP port 8008
2017-07-10T14:00:06-0700 INFO: test.js started
2017-07-10T14:00:06-0700 INFO: [{"fqdn":"leaf1","hostname":"leaf1"}]
2017-07-10T14:00:06-0700 INFO: test.js stopped
While retrieving information from the switch is useful, reconfiguring the switch based on real-time sFlow telemetry is much more interesting.

DDoS describes how sFlow analytics can be used to detect distributed denial of service (DDoS) attacks in real-time. EOS DirectFlow provides a flexible method of applying traffic controls and the following ddos.js script automatically detects UDP reflection/amplification attacks and uses eAPI to install DirectFlow entries to drop the attack traffic:
include('eapi.js');

var proto = 'http';
var user = 'admin';
var password = 'arista';
var thresh = 1000000;
var block_minutes = 60;

setFlow('udp_target',{keys:'ipdestination,udpsourceport',value:'frames'});

setThreshold('attack',{metric:'udp_target', value:thresh, byFlow:true});

var controls = {};
var id = 0;
setEventHandler(function(evt) {
  var key = evt.agent + ',' + evt.flowKey;
  if(controls[key]) return;

  var now = (new Date()).getTime();
  var flow = 'ddos'+id++;
  var [ip,port] = evt.flowKey.split(',');
  var cmds = [
  'enable',
  'configure',
  'directflow',
  'flow ' + flow,
  'match ethertype ip',
  'match destination ip ' + ip,
  'match ip protocol udp',
  'match source port ' + port, 
  'action drop'
  ];
  controls[key] = {time:now, target: ip, port: port, agent:evt.agent, flow:flow};
  try { runCmds(proto,evt.agent,user,password,cmds); }
  catch(e) { logSevere('failed to add filter, ' + e); }
  logInfo('block target='+ip+' port='+port+' agent=' + evt.agent);  
},['attack']);

setIntervalHandler(function() {
  var now = (new Date()).getTime();
  for(var key in controls) {
    if(now - controls[key].time < 1000 * 60 * block_minutes) continue;
    var ctl = controls[key];
    delete controls[key];
    var cmds = [
    'enable',
    'configure',
    'directflow',
    'no flow ' + ctl.flow
    ];
    try { runCmds(proto,ctl.agent,user,password,cmds); }
    catch(e) { logSevere('failed to remove filter, ' + e); }
    logInfo('allow target='+ctl.target+' port='+ctl.port+' agent='+ctl.agent);
  }
});
Some notes on the script:
  • The script is designed to work with a large number of switches, automatically applying the DirectFlow filter to the switch reporting the traffic.
  • The udp_target flow identifies the IP address targeted by the attack and the UDP source port of service being used to reflect/amplify traffic. 
  • A threshold of 1,000,000 frames per second is used to trigger an event.
  • The setEventHandler function extracts target IP address, and UDP source port from the event and uses eAPI to push a DirectFlow filter to switch (agent) identified in the event.
  • The setIntervalHandler function is responsible for removing controls after 60 minutes.
  • The script can easily be modified to use eAPI to gather additional metadata. For example, to identify leaf switches and limit filters to the edge of the network.
  • Exporting events using syslog shows how notifications can be sent to SIEM tools, e.g. Splunk, Logstash, etc.
  • InfluxDB and Grafana, Metric export to Graphite, Cloud analytics, and SignalFx, demonstrate how metrics can be pushed to local and/or cloud-based dashboards.
  • See Writing Applications for more information on sFlow-RT scripting and APIs.
The basic steps of defining a flow, setting a threshold, and then acting on events embodied in this example provide a general framework that can be applied to a wide variety of use cases: SDN and large flows, Marking large flows, SDN packet broker etc. In addition to DirectFlow, other useful EOS eAPI controls include: ACLs, route maps, static routes, null routes, packet capture etc.

Industry standard sFlow telemetry unlocks the full potential of programmable networking platforms such as Arista EOS, providing the visibility required to automatically target controls and adapt the network in real-time to changing network conditions to increase performance, reduce cost, and improve security.

Monday, July 10, 2017

Real-time DDoS mitigation using sFlow and BGP FlowSpec

Remotely Triggered Black Hole (RTBH) Routing describes how native BGP support in the sFlow-RT real-time sFlow analytics engine can be used to blackhole traffic in order to mitigate a distributed denial of service (DDoS) attack. Black hole routing is effective, but there is significant potential for collateral damage since ALL traffic to the IP address targeted by the attack is dropped.

The BGP FlowSpec extension (RFC 5575: Dissemination of Flow Specification Rules) provides a method of transmitting traffic filters that selectively block the attack traffic while allowing normal traffic to pass. BGP FlowSpec support has recently been added to sFlow-RT and this article demonstrates the new capability.

This demonstration uses the test network described in Remotely Triggered Black Hole (RTBH) Routing. The network was constructed using free components: VirtualBox, Cumulus VX, and Ubuntu LinuxBGP FlowSpec on white box switch describes how to implement basic FlowSpec support on Cumulus Linux.

The following flowspec.js sFlow-RT script detects and blocks UDP-Based Amplification attacks:
var router = '10.0.0.141';
var id = '10.0.0.70';
var as = 65141;
var thresh = 1000;
var block_minutes = 1;

setFlow('udp_target',{keys:'ipdestination,udpsourceport',value:'frames'});

setThreshold('attack',{metric:'udp_target', value:thresh, byFlow:true});

bgpAddNeighbor(router,as,id,{flowspec:true});

var controls = {};
setEventHandler(function(evt) {
  var key = evt.flowKey;
  if(controls[key]) return;

  var now = (new Date()).getTime();
  var [ip,port] = key.split(',');
  var flow = {
    'match':{
      'protocol':'=17',
      'source-port':'='+port,
      'destination': ip
    },
    'then': {'traffic-rate':0}
  };
  controls[key] = {time:now, target: ip, port: port, flow:flow};
  bgpAddFlow(router, flow);
  logInfo('block target='+ip+' port='+port);  
},['attack']);

setIntervalHandler(function() {
  var now = (new Date()).getTime();
  for(var key in controls) {
    if(now - controls[key].time < 1000 * 60 * block_minutes) continue;
    var control = controls[key];
    delete controls[key];
    bgpRemoveFlow(router,control.flow);
    logInfo('allow target='+control.target+' port='+control.port);
  }
});
See Writing Applications for more information on sFlow-RT scripting and APIs.

Start sFlow-RT:
env "RTPROP=-Dscript.file=flowspec.js -Dbgp.start=yes" ./start.sh
Simulate a DNS amplification attack using hping:
sudo hping3 --flood --udp -k -s 53 172.16.140.1
The screen capture shows the results. The left of the chart shows a simulated attack without mitigation. The attack reaches a sustained rate 30,000 packets per seconds. The right half of the chart shows an attack with automatic mitigation enabled. The target IP address and UDP source port associated with the amplification attack are immediately identified and a BGP FlowSpec filter is pushed to the upstream service provider router, sp-router, where the attack traffic is immediately dropped.

Wednesday, July 5, 2017

BGP FlowSpec on white box switch

BGP FlowSpec is a method of distributing access control lists (ACLs) using the BGP protocol. Distributed denial of service (DDoS) mitigation is an important use case for the technology, allowing a targeted network to push filters to their upstream provider to selectively remove the attack traffic.

Unfortunately, FlowSpec is currently only available on high end routing devices and so experimenting with the technology is expensive. Looking for an alternative, Cumulus Linux is an open Linux platform that allows users to install Linux packages and develop their own software.

This article describes a proof of concept implementation of basic FlowSpec functionality using ExaBGP installed on a free Cumulus VX virtual machine.  The same solution can be run on inexpensive commodity white box hardware to deliver terabit traffic filtering in a production network.

First, install latest version of ExaBGP on the Cumulus Linux switch:
curl -L https://github.com/Exa-Networks/exabgp/archive/4.0.0.tar.gz | tar zx
Now define the handler, acl.py, that will convert BGP FlowSpec updates into standard Linux netfilter/iptables entries used by Cumulus Linux to specify hardware ACLs (see Netfilter - ACLs):
#!/usr/bin/python
 
import json
import re
from os import listdir,remove
from os.path import isfile
from subprocess import Popen,STDOUT,PIPE
from sys import stdin, stdout, stderr

id = 0
acls = {}

dir = '/etc/cumulus/acl/policy.d/'
priority = '60'
prefix = 'flowspec'
bld = '.bld'
suffix = '.rules'

def commit():
  Popen(["cl-acltool","-i"],stderr=STDOUT,stdout=PIPE).communicate()[0]

def aclfile(name):
  global dir,priority,prefix,suffix
  return dir+priority+prefix+name+suffix

def handleSession(state):
  if "down" == state:
    for key,rec in acls.items():
      fn = aclfile(str(rec['id']))
      if isfile(fn):
        remove(fn)
      del acls[key]
    commit()

def buildACL(flow,action):
  acl = "[iptables]\n-A FORWARD --in-interface swp+"
  if flow.get('protocol'):
    acl = acl + " -p " + re.sub('[!<>=]','',flow['protocol'][0])
  if flow.get('source-ipv4'):
    acl = acl + " -s " + flow['source-ipv4'][0]
  if flow.get('destination-ipv4'):
    acl = acl + " -d " + flow['destination-ipv4'][0]
  if flow.get('source-port'):
    acl = acl + " --sport " + re.sub('[!<>=]','',flow['source-port'][0])
  if flow.get('destination-port'):
    acl = acl + " --dport " + re.sub('[!<>=]','',flow['destination-port'][0])
  acl = acl + " -j DROP\n"
  return acl

def handleFlow(add,flow,action):
  global id
  key = flow['string']
  if add:
    acl = buildACL(flow,action)
    id = id + 1
    acls[key] = {"acl":acl,"id":id}
    fn = aclfile(str(id))
    f = open(fn,'w')
    f.write(acl)
    f.close()
    commit()
  elif key in acls:
    rec = acls[key]
    fn = aclfile(str(rec['id']))
    if isfile(fn):
      remove(fn)
    del acls[key]
    commit()
  
while True:
  try:
     line = stdin.readline().strip()
     msg = json.loads(line)
     type = msg["type"]
     if "state" == type:
       state = msg["neighbor"]["state"]
       handleSession(state)
     elif "update" == type:
       update = msg["neighbor"]["message"]["update"] 
       if update.get('announce'):
         flow = update["announce"]["ipv4 flow"]["no-nexthop"][0]
         community = update["attribute"]["extended-community"][0]
         handleFlow(True,flow,community)
       elif update.get('withdraw'):
         flow = update["withdraw"]["ipv4 flow"][0]
         handleFlow(False,flow,None)
     else:
       pass
  except IOError:
     pass
Note: This script is a simple demonstration of the concept that has significant limitations: there is no error handling, the only action is to drop traffic, and FlowSpec comparison operators are ignored. The script is is based on the article RESTful control of Cumulus Linux ACLs.

Update July 6, 2017: An improved version of the acl.py script is now available in the ExaBGP repository on GitHub, see flow.py

Next, create the exabgp.conf file:
process acl {
   run ./acl.py;
   encoder json;
}

template {
  neighbor controller {
    family {
      ipv4 flow;
    }
    api speaking {
      processes [ acl ];
      neighbor-changes;
      receive {
         parsed;
         update;
      }
    }
  }
}

neighbor 10.0.0.70 {
  inherit controller;
  router-id 10.0.0.140;
  local-as 65140;
  peer-as 65070;
  local-address 0.0.0.0;
  connect 1179;
}
Finally, run ExaBGP:
sudo env exabgp.daemon.user=root exabgp-4.0.0/sbin/exabgp exabgp.conf
This configuration instructs ExaBGP to connect to the controller, 10.0.0.162, and prepare to receive BGP FlowSpec messages. When a BGP message is received, ExaBGP decodes the message and passes it on in the form of a JSON encoded string to the acl.py program. For example, the following FlowSpec message was sent to block an NTP reflection DDoS attack (traffic from UDP port 123) targeting host 192.168.0.1:
{
 "exabgp": "4.0.0",
 "time": 1498868955.31,
 "host": "tor-router",
 "pid": 3854,
 "ppid": 3849,
 "counter": 6,
 "type": "update",
 "neighbor": {
  "address": {
   "local": "0.0.0.0",
   "peer": "10.0.0.70"
  },
  "asn": {
   "local": "65140",
   "peer": "65070"
  },
  "direction": "receive",
  "message": {
   "update": {
    "attribute": {
     "origin": "igp",
     "as-path": [
      65140
     ],
     "confederation-path": [],
     "local-preference": 100,
     "extended-community": [
      9225060886715040000
     ]
    },
    "announce": {
     "ipv4 flow": {
      "no-nexthop": [
       {
        "destination-ipv4": [
         "192.168.0.1/32"
        ],
        "protocol": [
         "=udp"
        ],
        "source-port": [
         "=123"
        ],
        "string": "flow destination-ipv4 192.168.0.1/32 protocol =udp source-port =123"
       }
      ]
     }
    }
   }
  }
 }
}
The acl.py script wrote the file /etc/cumulus/acl/policy.d/60flowspec1.rules with an iptables representation of the FlowSpec message:
[iptables]
-A FORWARD --in-interface swp+ -p udp -d 192.168.0.1/32 --sport 123 -j DROP
The acl.py script also invoked the cl-acltool command to install the new rule, which can be verified using iptables:
sudo iptables --list FORWARD
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
DROP       all  --  240.0.0.0/5          anywhere            
DROP       all  --  loopback/8           anywhere            
DROP       all  --  base-address.mcast.net/4  anywhere            
DROP       all  --  255.255.255.255      anywhere            
DROP       udp  --  anywhere             192.168.0.1          udp spt:ntp
The attack traffic is now dropped by the switch. In a hardware switch, this entry would be pushed by Cumulus Linux into the hardware, filtering the traffic at line rate to provide Terabit traffic filtering.