Wednesday, September 20, 2017

Flow Trend

The open source sflow-rt/flow-trend project displays a real-time trend chart of network traffic that updates every second. Defining Flows describes how to break out traffic by different traffic attributes, including: addresses, ports, VLANs, protocols, countries, DNS names, etc.
docker run -p 6343:6343/udp -p 8008:8008 sflow/flow-trend
The simplest way to run the software is using the docker. Configure network devices to send standard sFlow telemetry to Flow Trend. Access the web user interface on port 8008.

Sunday, September 10, 2017

Real-time traffic visualization using Netflix Vizceral

The open source sflow-rt/vizceral project demonstrates how real-time sFlow network telemetry can be presented using Netflix Vizceral. The central dot represents the Internet (all non-local addresses). The surrounding dots represents addresses grouped into sites, data centers, buildings etc. The animated particle flows represent packet flows with colors indicating packet type: TCP/UDP shown in blue, ICMP shown in yellow, and all other traffic in red.
Click on a node to zoom in to show packets flowing up and down the protocol stack. Press the ESC key to unzoom.

The simplest way to run the software is to use the pre-built Docker image:
docker run -p 6343:6343/udp -p 8008:8008 sflow/vizceral
The Docker image also contains demo data based on Netflix's public cloud infrastructure:
docker run -e "RTPROP=-Dviz.demo=yes" -p 8008:8008 sflow/vizceral
In this case, the detailed view shows messages flowing between microservices running in the Amazon public cloud. Similar visibility could be obtained by deploying Host sFlow agents with associated modules for web and application servers and modifying sflow/vizceral to present the application transaction flows. In private data centers, sFlow support in load balancers  (F5, A10) provides visibility into interactions between application tiers. See Microservices for more information on using sFlow to instrument microservice architectures.
Collecting Docker Swarm service metrics describes how meta data about services running on Docker Swarm can be combined with sFlow telemetry to generate service level metrics. A similar approach could be taken to display Docker Swarm service interactions using Vizceral. Using network visibility to measure flows between services greatly simplifies the monitoring task, avoiding the challenge of adding instrumentation to each container.

Tuesday, September 5, 2017

Troubleshooting connectivity problems in leaf and spine fabrics

Introducing data center fabric, the next-generation Facebook data center network describes the benefits of moving to a leaf and spine network architecture. The diagram shows how the leaf and spine architecture creates many paths between each pair of hosts. Multiple paths increase available bandwidth and resilience against the loss of a link or a switch. While most networks don't have the scale requirements of Facebook, smaller scale leaf and spine designs deliver high bandwidth, low latency, networking to support cloud workloads (e.g. vSphere, OpenStack, Docker, Hadoop, etc.).

Unlike traditional hierarchical network designs, where a small number of links can be monitored to provide visibility, a leaf and spine network has no special links or switches where running CLI commands or attaching a probe would provide visibility. Even if it were possible to attach probes, the effective bandwidth of a leaf and spine network can be as high as a Petabit/second, well beyond the capabilities of current generation monitoring tools.

Fortunately, industry standard sFlow monitoring technology is built into the commodity switch hardware used to build leaf and spine networks. Enabling sFlow telemetry on all the switches in the network provides centralized, real-time, visibility into network traffic.
Fabric View describes an open source application running on the sFlow-RT real-time analytics engine. The Fabric View application provides an overview of the health of the entire leaf and spine fabric, tracking flows and counters on all links and summarizing information in a set of fabric level metrics and dashboards. In addition, Black hole detection describes how to detect routing anomalies in the fabric using the forwarding information included in the sFlow telemetry stream.

The sFlow sampling mechanism implemented in the switches is a highly scaleable method of passively collecting traffic information. However,  analyzing failed connections can be a challenge since very few packets are generated and the chance of sampling these packets is small. The traditional tools used to diagnose connectivity issues, ping and traceroute, are of limited value in a leaf and spine network since they only test a single path and are likely to miss the path that is experiencing difficulties.

An alternative method of addressing the multi-path tracing problem is to enable filtered packet capture on each switch, programming the filters to capture the packets of interest. However, this method can be slow and complex since every switch needs to be configured for each test and the switch configurations need to be cleared after the test has been completed.

This article explores how the hping3 tool can be used with sFlow to trace packet paths across the fabric and detect where they are being lost. The following Python script,, uses sFlow-RT's REST API to program a flow to watch for a specific flow and print the links that it traverses:
#!/usr/bin/env python
import argparse
import requests
import json
import signal
from random import randint

def sig_handler(signal,frame):
signal.signal(signal.SIGINT, sig_handler)

parser = argparse.ArgumentParser()
parser.add_argument('filter', help='sFlow-RT flow filter, e.g. "ipsource="')
args = parser.parse_args()

rt = 'http://localhost:8008'
name = 'trace' + str(randint(0,10000))

flow = {'keys':'link:inputifindex','value':'frames',

flowurl = rt+'/flows/json?maxFlows=100&timeout=60&name='+name
flowID = -1
while 1 == 1:
  r = requests.get(flowurl+'&flowID='+str(flowID))
  if r.status_code != 200: break
  flows = r.json()
  if len(flows) == 0: continue

  flowID = flows[0]["flowID"]
  for f in flows:
    print f['flowKeys']
Note: See RESTflow for a description of the sFlow-RT REST API.

First run the following Python script, supplying a filter to select the packets of interest:
./ 'ipsource='
Note: Identifying characteristics of failed connections may be inferable from application error logs. Otherwise, running packet capture on the affected host (tcpdump/wireshark) can identify the network attributes of interest.

Next, log into the host that is having connectivity problems and generate traffic matching the flow:
sudo hping3 -c 100000 -i u100 --udp -k -s 1111 -p 53
Note: The above command sends 100,000 packets at a rate of 1 packet every 100 microseconds (i.e. at a rate of 10,000 packets per second).  Select a packet rate that will not disturb production traffic on the network and make sure to send enough packets so that at least one packet will be sampled on each link. For example, for 10G links the packet sampling rate should be around 1-in-10,000 so generating 100,000 packets means that there is a 99.995% chance that a link carrying the flow will generate at least 1 sample (the probability is easily calculated using the Binomal distribution, see Wolfram Alpha).

The script will start printing links traversed by the flow immediately they are detected (typically in less than a second after starting the test):
./ 'ipsource='
The above example traced the single path traversed by a specific connection. To explore all paths, drop the source port and hping3 will cycle through source ports and the traffic should be visible on all the equal cost paths (provided that a layer 4 hash function has been selected by the switches).

Drop the source port from the filter:
./ 'ipsource='
Drop the -k and -s options from the hping3 command:
sudo hping3  -c 100000 -i u100 --udp -p 53
The open source trace-flow application is a graphical version of the script written using sFlow-RT's JavaScript API (see Writing Applications). The screen capture above displayed the path for the test traffic within a second of the start of test.

Continuous network-wide monitoring of leaf and spine networks using sFlow leverages the capabilities of commodity switch hardware and provides centralized visibility that simplifies network operation and trouble shooting.

Sunday, August 27, 2017

Cumulus Linux 3.4 REST API

The latest Cumulus Linux 3.4 release include a REST API. This article will demonstrate how the REST API can be used to automatically deploy traffic controls based on real-time sFlow telemetry. DDoS mitigation with Cumulus Linux describes how sFlow-RT can detect Distributed Denial of Service (DDoS) attacks in real-time and deploy automated controls.

The following ddos.js script is modified to use the REST API to send Network Command Line Utility - NCLU commands to add and remove ACLs, see Installing and Managing ACL Rules with NCLU:
var user = "cumulus";
var password = "CumulusLinux!";
var thresh = 10000;
var block_minutes = 1;


setThreshold('attack',{metric:'udp_target', value:thresh, byFlow:true, timeout:10});

function restCmds(agent,cmds) {
  for(var i = 0; i < cmds.length; i++) {
    let msg = {cmd:cmds[i]};

var controls = {};
var id = 0;
setEventHandler(function(evt) {
  var key = evt.agent + ',' + evt.flowKey;
  if(controls[key]) return;

  var ifname = metric(evt.agent,evt.dataSource+".ifname")[0].metricValue;
  if(!ifname) return;

  var now = (new Date()).getTime();
  var name = 'ddos'+id++;
  var [ip,port] = evt.flowKey.split(',');
  var cmds = [
    'add acl ipv4 '+name+' drop udp source-ip any source-port '+port+' dest-ip '+ip+' dest-port any',
    'add int '+ifname+' acl ipv4 '+name+' inbound',
  controls[key] = {time:now, target: ip, port: port, agent:evt.agent, metric:evt.dataSource+'.'+evt.metric, key:evt.flowKey, name:name};
  try { restCmds(evt.agent, cmds); }
  catch(e) { logSevere('failed to add ACL, '+e); }
  logInfo('block target='+ip+' port='+port+' agent=' + evt.agent); 

setIntervalHandler(function() {
  var now = (new Date()).getTime();
  for(var key in controls) {
    if(now - controls[key].time < 1000 * 60 * block_minutes) continue;
    var ctl = controls[key];
    if(thresholdTriggered('attack',ctl.agent,ctl.metric,ctl.key)) continue;

    delete controls[key];
    var cmds = [
      'del acl ipv4 ',
    try { restCmds(ctl.agent,cmds); }
    catch(e) { logSevere('failed to remove ACL, ' + e); }
    logInfo('allow target='' port='+ctl.port+' agent='+ctl.agent);
The quickest way test the script is to use docker to run sFlow-RT:
docker run -v $PWD/ddos.js:/sflow-rt/ddos.js \
-e "RTPROP=-Dscript.file=ddos.js" \
-p 6343:6343/udp -p 8008:8008 sflow/sflow-rt
This solution can be tested using freely available software. The setup shown at the top of this article was constructed using a Cumulus VX virtual machine running on VirtualBox.  The Attacker and Target virtual machines are Linux virtual machines used to simulate the DDoS attack.

DNS amplification attack can be simulated using hping3. Run the following command on the Attacker host:
sudo hping3 --flood --udp -k -s 53
Run tcpdump on the Target host to see if the attack is getting through:
sudo tcpdump -i eth1 udp port 53
Each time an attack is launched a new ACL will be added that matches the attack signature and drops the traffic. The ACL is kept in place for at least block_minutes and removed once the attack ends. The following sFlow-RT log messages show the results:
2017-08-26T17:01:24+0000 INFO: Listening, sFlow port 6343
2017-08-26T17:01:24+0000 INFO: Listening, HTTP port 8008
2017-08-26T17:01:24+0000 INFO: ddos.js started
2017-08-26T17:03:07+0000 INFO: block target= port=53 agent=
2017-08-26T17:03:49+0000 INFO: allow target= port=53 agent=
REST API for Cumulus Linux ACLs describes the acl_server daemon that was used in the original article. The acl_server daemon is optimized for real-time performance, supporting use cases in which multiple traffic controls need to be quickly added and removed, e.g  DDoS mitigation, marking large flows, ECMP load balancing, packet brokers.

A key benefit of the openness of Cumulus Linux is that you can install software to suite your use case, other examples include: BGP FlowSpec on white box switchInternet router using Cumulus LinuxTopology discovery with Cumulus LinuxBlack hole detection, and Docker networking with IPVLAN and Cumulus Linux.

Thursday, July 13, 2017

Linux 4.11 kernel extends packet sampling support

Linux 4.11 on Linux Kernel Newbies describes the features added in the April 30, 2017 release. Of particular interest is the new netlink sampling channel:
Introduce psample, a general way for kernel modules to sample packets, without being tied to any specific subsystem. This netlink channel can be used by tc, iptables, etc. and allow to standardize packet sampling in the kernel commit
The psample netlink channel delivers sampled packet headers along with associated metadata from the Linux kernel to user space. The psample fields map directly into sFlow Version 5 sampled_header export structures:

netlink psamplesFlowDescription
PSAMPLE_ATTR_IIFINDEXinputInterface packet was received on.
PSAMPLE_ATTR_OIFINDEXoutputInterface packet was sent on.
PSAMPLE_ATTR_GROUP_SEQdropsNumber of times that the sFlow agent detected that a packet marked to be sampled was dropped due to lack of resources. Agent calculates drops by tracking discontinuities in PSAMPLE_ATTR_GROUP_SEQ
PSAMPLE_ATTR_SAMPLE_RATEsampling_rateThe Sampling Rate specifies the ratio of packets observed at the Data Source to the samples generated. For example a sampling rate of 100 specifies that, on average, 1 sample will be generated for every 100 packets observed.
PSAMPLE_ATTR_ORIGSIZEframe_lengthOriginal length of packet before sampling
PSAMPLE_ATTR_DATAheader<>Header bytes

Linux is widely used for switch network operating systems, including: Arista EOS, Cumulus Linux, Dell OS10, OpenSwitch, SONiC, and Open Network Linux. The adoption of Linux by network vendors and cloud providers is driving increased support for switch hardware by the Linux kernel community.

Hardware support for sFlow packet sampling is widely implemented in switch ASICs, including: Broadcom, Mellanox, Intel, Marvell, Barefoot Networks, Cavium, and Innovium. A standard Linux interface to ASIC sampling simplifies the implementation of sFlow agents (e.g. Host sFlow) and ensures consistent behavior across hardware platforms to deliver real-time network-wide visibility using industry standard sFlow protocol.

Wednesday, July 12, 2017

Arista eAPI

The sFlow and eAPI features of EOS (Extensible Operating System) are standard across the full range of Arista Networks switches. This article demonstrates how the real-time visibility provided by sFlow telemetry can be combined with the programmatic control of eAPI to automatically adapt the network to changing traffic conditions.

In the diagram, the sFlow-RT analytics engine receives streaming sFlow telemetry, provides real-time network-wide visibility, and automatically applies controls using eAPI to optimize forwarding, block denial of service attacks, or capture suspicious traffic.

Arista eAPI 101 describes the JSON RPC interface for programmatic control of Arista switches. The following eapi.js script shows how eAPI requests can be made using sFlow-RT's JavaScript API:
function runCmds(proto, agent, usr, pwd, cmds) {
  var req = {
  var url = (proto || 'http')+'://'+agent+'/command-api';
  var resp = http(url,'post','application/json',JSON.stringify(req),usr,pwd);
  if(!resp) throw "no response";
  resp = JSON.parse(resp);
  if(resp.error) throw resp.error.message;
  return resp.result; 
The following test.js script demonstrates the eAPI functionality with a basic show request:
var result = runCmds('http','','admin','arista',['show hostname']);
Starting sFlow-RT:
env "RTPROP=-Dscript.file=test.js" ./
Running the script generates the following output:
2017-07-10T14:00:06-0700 INFO: Listening, sFlow port 6343
2017-07-10T14:00:06-0700 INFO: Listening, HTTP port 8008
2017-07-10T14:00:06-0700 INFO: test.js started
2017-07-10T14:00:06-0700 INFO: [{"fqdn":"leaf1","hostname":"leaf1"}]
2017-07-10T14:00:06-0700 INFO: test.js stopped
While retrieving information from the switch is useful, reconfiguring the switch based on real-time sFlow telemetry is much more interesting.

DDoS describes how sFlow analytics can be used to detect distributed denial of service (DDoS) attacks in real-time. EOS DirectFlow provides a flexible method of applying traffic controls and the following ddos.js script automatically detects UDP reflection/amplification attacks and uses eAPI to install DirectFlow entries to drop the attack traffic:

var proto = 'http';
var user = 'admin';
var password = 'arista';
var thresh = 1000000;
var block_minutes = 60;


setThreshold('attack',{metric:'udp_target', value:thresh, byFlow:true});

var controls = {};
var id = 0;
setEventHandler(function(evt) {
  var key = evt.agent + ',' + evt.flowKey;
  if(controls[key]) return;

  var now = (new Date()).getTime();
  var flow = 'ddos'+id++;
  var [ip,port] = evt.flowKey.split(',');
  var cmds = [
  'flow ' + flow,
  'match ethertype ip',
  'match destination ip ' + ip,
  'match ip protocol udp',
  'match source port ' + port, 
  'action drop'
  controls[key] = {time:now, target: ip, port: port, agent:evt.agent, flow:flow};
  try { runCmds(proto,evt.agent,user,password,cmds); }
  catch(e) { logSevere('failed to add filter, ' + e); }
  logInfo('block target='+ip+' port='+port+' agent=' + evt.agent);  

setIntervalHandler(function() {
  var now = (new Date()).getTime();
  for(var key in controls) {
    if(now - controls[key].time < 1000 * 60 * block_minutes) continue;
    var ctl = controls[key];
    delete controls[key];
    var cmds = [
    'no flow ' + ctl.flow
    try { runCmds(proto,ctl.agent,user,password,cmds); }
    catch(e) { logSevere('failed to remove filter, ' + e); }
    logInfo('allow target='' port='+ctl.port+' agent='+ctl.agent);
Some notes on the script:
  • The script is designed to work with a large number of switches, automatically applying the DirectFlow filter to the switch reporting the traffic.
  • The udp_target flow identifies the IP address targeted by the attack and the UDP source port of service being used to reflect/amplify traffic. 
  • A threshold of 1,000,000 frames per second is used to trigger an event.
  • The setEventHandler function extracts target IP address, and UDP source port from the event and uses eAPI to push a DirectFlow filter to switch (agent) identified in the event.
  • The setIntervalHandler function is responsible for removing controls after 60 minutes.
  • The script can easily be modified to use eAPI to gather additional metadata. For example, to identify leaf switches and limit filters to the edge of the network.
  • Exporting events using syslog shows how notifications can be sent to SIEM tools, e.g. Splunk, Logstash, etc.
  • InfluxDB and Grafana, Metric export to Graphite, Cloud analytics, and SignalFx, demonstrate how metrics can be pushed to local and/or cloud-based dashboards.
  • See Writing Applications for more information on sFlow-RT scripting and APIs.
The basic steps of defining a flow, setting a threshold, and then acting on events embodied in this example provide a general framework that can be applied to a wide variety of use cases: SDN and large flows, Marking large flows, SDN packet broker etc. In addition to DirectFlow, other useful EOS eAPI controls include: ACLs, route maps, static routes, null routes, packet capture etc.

Industry standard sFlow telemetry unlocks the full potential of programmable networking platforms such as Arista EOS, providing the visibility required to automatically target controls and adapt the network in real-time to changing network conditions to increase performance, reduce cost, and improve security.

Monday, July 10, 2017

Real-time DDoS mitigation using sFlow and BGP FlowSpec

Remotely Triggered Black Hole (RTBH) Routing describes how native BGP support in the sFlow-RT real-time sFlow analytics engine can be used to blackhole traffic in order to mitigate a distributed denial of service (DDoS) attack. Black hole routing is effective, but there is significant potential for collateral damage since ALL traffic to the IP address targeted by the attack is dropped.

The BGP FlowSpec extension (RFC 5575: Dissemination of Flow Specification Rules) provides a method of transmitting traffic filters that selectively block the attack traffic while allowing normal traffic to pass. BGP FlowSpec support has recently been added to sFlow-RT and this article demonstrates the new capability.

This demonstration uses the test network described in Remotely Triggered Black Hole (RTBH) Routing. The network was constructed using free components: VirtualBox, Cumulus VX, and Ubuntu LinuxBGP FlowSpec on white box switch describes how to implement basic FlowSpec support on Cumulus Linux.

The following flowspec.js sFlow-RT script detects and blocks UDP-Based Amplification attacks:
var router = '';
var id = '';
var as = 65141;
var thresh = 1000;
var block_minutes = 1;


setThreshold('attack',{metric:'udp_target', value:thresh, byFlow:true});


var controls = {};
setEventHandler(function(evt) {
  var key = evt.flowKey;
  if(controls[key]) return;

  var now = (new Date()).getTime();
  var [ip,port] = key.split(',');
  var flow = {
      'destination': ip
    'then': {'traffic-rate':0}
  controls[key] = {time:now, target: ip, port: port, flow:flow};
  bgpAddFlow(router, flow);
  logInfo('block target='+ip+' port='+port);  

setIntervalHandler(function() {
  var now = (new Date()).getTime();
  for(var key in controls) {
    if(now - controls[key].time < 1000 * 60 * block_minutes) continue;
    var control = controls[key];
    delete controls[key];
    logInfo('allow target='' port='+control.port);
See Writing Applications for more information on sFlow-RT scripting and APIs.

Start sFlow-RT:
env "RTPROP=-Dscript.file=flowspec.js -Dbgp.start=yes" ./
Simulate a DNS amplification attack using hping:
sudo hping3 --flood --udp -k -s 53
The screen capture shows the results. The left of the chart shows a simulated attack without mitigation. The attack reaches a sustained rate 30,000 packets per seconds. The right half of the chart shows an attack with automatic mitigation enabled. The target IP address and UDP source port associated with the amplification attack are immediately identified and a BGP FlowSpec filter is pushed to the upstream service provider router, sp-router, where the attack traffic is immediately dropped.

Wednesday, July 5, 2017

BGP FlowSpec on white box switch

BGP FlowSpec is a method of distributing access control lists (ACLs) using the BGP protocol. Distributed denial of service (DDoS) mitigation is an important use case for the technology, allowing a targeted network to push filters to their upstream provider to selectively remove the attack traffic.

Unfortunately, FlowSpec is currently only available on high end routing devices and so experimenting with the technology is expensive. Looking for an alternative, Cumulus Linux is an open Linux platform that allows users to install Linux packages and develop their own software.

This article describes a proof of concept implementation of basic FlowSpec functionality using ExaBGP installed on a free Cumulus VX virtual machine.  The same solution can be run on inexpensive commodity white box hardware to deliver terabit traffic filtering in a production network.

First, install latest version of ExaBGP on the Cumulus Linux switch:
curl -L | tar zx
Now define the handler,, that will convert BGP FlowSpec updates into standard Linux netfilter/iptables entries used by Cumulus Linux to specify hardware ACLs (see Netfilter - ACLs):
import json
import re
from os import listdir,remove
from os.path import isfile
from subprocess import Popen,STDOUT,PIPE
from sys import stdin, stdout, stderr

id = 0
acls = {}

dir = '/etc/cumulus/acl/policy.d/'
priority = '60'
prefix = 'flowspec'
bld = '.bld'
suffix = '.rules'

def commit():

def aclfile(name):
  global dir,priority,prefix,suffix
  return dir+priority+prefix+name+suffix

def handleSession(state):
  if "down" == state:
    for key,rec in acls.items():
      fn = aclfile(str(rec['id']))
      if isfile(fn):
      del acls[key]

def buildACL(flow,action):
  acl = "[iptables]\n-A FORWARD --in-interface swp+"
  if flow.get('protocol'):
    acl = acl + " -p " + re.sub('[!<>=]','',flow['protocol'][0])
  if flow.get('source-ipv4'):
    acl = acl + " -s " + flow['source-ipv4'][0]
  if flow.get('destination-ipv4'):
    acl = acl + " -d " + flow['destination-ipv4'][0]
  if flow.get('source-port'):
    acl = acl + " --sport " + re.sub('[!<>=]','',flow['source-port'][0])
  if flow.get('destination-port'):
    acl = acl + " --dport " + re.sub('[!<>=]','',flow['destination-port'][0])
  acl = acl + " -j DROP\n"
  return acl

def handleFlow(add,flow,action):
  global id
  key = flow['string']
  if add:
    acl = buildACL(flow,action)
    id = id + 1
    acls[key] = {"acl":acl,"id":id}
    fn = aclfile(str(id))
    f = open(fn,'w')
  elif key in acls:
    rec = acls[key]
    fn = aclfile(str(rec['id']))
    if isfile(fn):
    del acls[key]
while True:
     line = stdin.readline().strip()
     msg = json.loads(line)
     type = msg["type"]
     if "state" == type:
       state = msg["neighbor"]["state"]
     elif "update" == type:
       update = msg["neighbor"]["message"]["update"] 
       if update.get('announce'):
         flow = update["announce"]["ipv4 flow"]["no-nexthop"][0]
         community = update["attribute"]["extended-community"][0]
       elif update.get('withdraw'):
         flow = update["withdraw"]["ipv4 flow"][0]
  except IOError:
Note: This script is a simple demonstration of the concept that has significant limitations: there is no error handling, the only action is to drop traffic, and FlowSpec comparison operators are ignored. The script is is based on the article RESTful control of Cumulus Linux ACLs.

Update July 6, 2017: An improved version of the script is now available in the ExaBGP repository on GitHub, see

Next, create the exabgp.conf file:
process acl {
   run ./;
   encoder json;

template {
  neighbor controller {
    family {
      ipv4 flow;
    api speaking {
      processes [ acl ];
      receive {

neighbor {
  inherit controller;
  local-as 65140;
  peer-as 65070;
  connect 1179;
Finally, run ExaBGP:
sudo env exabgp.daemon.user=root exabgp-4.0.0/sbin/exabgp exabgp.conf
This configuration instructs ExaBGP to connect to the controller,, and prepare to receive BGP FlowSpec messages. When a BGP message is received, ExaBGP decodes the message and passes it on in the form of a JSON encoded string to the program. For example, the following FlowSpec message was sent to block an NTP reflection DDoS attack (traffic from UDP port 123) targeting host
 "exabgp": "4.0.0",
 "time": 1498868955.31,
 "host": "tor-router",
 "pid": 3854,
 "ppid": 3849,
 "counter": 6,
 "type": "update",
 "neighbor": {
  "address": {
   "local": "",
   "peer": ""
  "asn": {
   "local": "65140",
   "peer": "65070"
  "direction": "receive",
  "message": {
   "update": {
    "attribute": {
     "origin": "igp",
     "as-path": [
     "confederation-path": [],
     "local-preference": 100,
     "extended-community": [
    "announce": {
     "ipv4 flow": {
      "no-nexthop": [
        "destination-ipv4": [
        "protocol": [
        "source-port": [
        "string": "flow destination-ipv4 protocol =udp source-port =123"
The script wrote the file /etc/cumulus/acl/policy.d/60flowspec1.rules with an iptables representation of the FlowSpec message:
-A FORWARD --in-interface swp+ -p udp -d --sport 123 -j DROP
The script also invoked the cl-acltool command to install the new rule, which can be verified using iptables:
sudo iptables --list FORWARD
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
DROP       all  --          anywhere            
DROP       all  --  loopback/8           anywhere            
DROP       all  --  anywhere            
DROP       all  --      anywhere            
DROP       udp  --  anywhere             udp spt:ntp
The attack traffic is now dropped by the switch. In a hardware switch, this entry would be pushed by Cumulus Linux into the hardware, filtering the traffic at line rate to provide Terabit traffic filtering.

Monday, June 19, 2017

Remotely Triggered Black Hole (RTBH) Routing

The screen shot demonstrates real-time distributed denial of service (DDoS) mitigation. Automatic mitigation was disabled for the first simulated attack (shown on the left of the chart).  The attack reaches a sustained packet rate of 1000 packets per second for a period of 60 seconds. Next, automatic mitigation was enabled and a second attack launched. This time, as soon as the traffic crosses the threshold (the horizontal red line), a BGP remote trigger message is sent to router, which immediately drops the traffic.
The diagram shows the test setup. The network was built out of freely available components: CumulusVX switches and Ubuntu 16.04 servers running under VirtualBox.

The following configuration is installed on the ce-router:
router bgp 65140
 bgp router-id
 neighbor remote-as 65140
 neighbor port 1179
 neighbor remote-as 65141
 address-family ipv4 unicast
  neighbor allowas-in
  neighbor route-map blackhole-in in
ip community-list standard blackhole permit 65535:666
route-map blackhole-in permit 20
 match community blackhole
 match ip address prefix-len 32
 set ip next-hop
The ce-router peers with the upstream service provider router (sp-router as well as with sFlow-RT.  A route-map is used to filter updates from sFlow-RT (, matching the well-known blackhole community 65535:666, and setting the null route next-hop The route-map also ensures that only /32 prefixes are accepted. In production, the route-map should also filter out the addresses of critical infrastructure (router IP addresses etc.).

An additional route-map will typically be required to select the blackhole routes to propagate upstream, re-mapping the community to meet the upstream service provider's policy, e.g. Hurricane Electric uses community 6939:666.

In addition, the ce-router is configured to send sFlow to the controller (, see Switch configurations.

Run the DDoS mitigation application on server using docker:
docker run --net=host -p 6343:6343/udp -p 8008:8008 -p 1179:1179 -e "RTPROP=-Dddos_blackhole.router=" sflow/ddos-blackhole
Alternatively, an RPM or DEB packaged version of sFlow-RT can be downloaded and installed on the server and the ddos-blackhole application can be installed. For example, on Ubuntu:
dpkg -i sflow-rt_2.0-1195.deb
/usr/local/sflow-rt/ sflow-rt ddos-blackhole
Edit the configuration file, /usr/local/sflow-rt/conf.d/sflow-rt.conf:
Next, start the daemon:
service sflow-rt start
In either case, the remainder of the configuration is handled through the web interface, accessible via

Click on the Settings tab and upload the following IP Address Groups file groups.json:
 "external": [
 "private": [
 "multicast": [
 "web": [
The groups identify addresses that are external (possible attackers) and local (possible targets). By default, traffic to the externalprivate, multicast and exclude groups will not trigger actions. Any additional group names, in this case web, are blackhole candidates.

Note: Only traffic from the external group to backhole candidate groups (web) is shown on the Charts tab (and considered for DDoS detection and mitigation).

The following command on sp-host simulates an ICMP flood attack on ce-host:
ping -f
The following messages should appear in the sFlow-RT logs:
2017-06-17T00:17:39+0000 INFO: Listening, BGP port 1179
2017-06-17T00:17:40+0000 INFO: Listening, sFlow port 6343
2017-06-17T00:17:40+0000 INFO: Listening, HTTP port 8008
2017-06-17T00:17:40+0000 INFO: app/ddos-blackhole/scripts/ddos.js started
2017-06-17T00:17:40+0000 INFO: app/ddos-blackhole/scripts/stats.js started
2017-06-17T00:17:46+0000 INFO: BGP open 41692
2017-06-17T00:18:13+0000 INFO: DDoS blocking
2017-06-17T00:20:25+0000 INFO: DDoS allowing
The screen capture at the top of this article shows that the time between the attack being launched and successfully blocked is just a few of seconds.

Wednesday, March 29, 2017

Arista EOS telemetry

Arista EOS switches support industry standard sFlow telemetry, enabling hardware instrumentation supported by merchant silicon to export hardware interface counters and flow data. The latest release of the open source Host sFlow agent has been ported to EOS, augmenting the telemetry with standard host CPU, memory, and disk IO metrics.

Linux as a Switch Operating System: Five Lessons Learned identifies benefits of using Linux as the basis for EOS. In this context, the Linux operating system made it easy to port the Host sFlow agent, use standard Linux package management (RPM Package Manager), and gather metrics using standard Linux APIs. A new eAPI module automatically synchronizes the Host sFlow daemon with the EOS sFlow configuration.

The following sflowtool output shows the additional metrics contributed by a Host sFlow agent installed on an Arista switch:
startDatagram =================================
datagramSize 704
unixSecondsUTC 1490843418
datagramVersion 5
agentSubId 100000
packetSequenceNo 714
sysUpTime 0
samplesInPacket 1
startSample ----------------------
sampleType_tag 0:2
sampleSequenceNo 714
sourceId 2:1
counterBlock_tag 0:2001
counterBlock_tag 0:2010
udpInDatagrams 1459
udpNoPorts 16
udpInErrors 0
udpOutDatagrams 4765
udpRcvbufErrors 0
udpSndbufErrors 0
udpInCsumErrors 0
counterBlock_tag 0:2009
tcpRtoAlgorithm 1
tcpRtoMin 200
tcpRtoMax 120000
tcpMaxConn 4294967295
tcpActiveOpens 102
tcpPassiveOpens 100
tcpAttemptFails 0
tcpEstabResets 0
tcpCurrEstab 8
tcpInSegs 19930
tcpOutSegs 19804
tcpRetransSegs 0
tcpInErrs 0
tcpOutRsts 2
tcpInCsumErrors 0
counterBlock_tag 0:2008
icmpInMsgs 1606
icmpInErrors 0
icmpInDestUnreachs 16
icmpInTimeExcds 0
icmpInParamProbs 0
icmpInSrcQuenchs 0
icmpInRedirects 0
icmpInEchos 1590
icmpInEchoReps 0
icmpInTimestamps 0
icmpInAddrMasks 0
icmpInAddrMaskReps 0
icmpOutMsgs 0
icmpOutErrors 1606
icmpOutDestUnreachs 0
icmpOutTimeExcds 16
icmpOutParamProbs 0
icmpOutSrcQuenchs 0
icmpOutRedirects 0
icmpOutEchos 0
icmpOutEchoReps 0
icmpOutTimestamps 1590
icmpOutTimestampReps 0
icmpOutAddrMasks 0
icmpOutAddrMaskReps 0
counterBlock_tag 0:2007
ipForwarding 2
ipDefaultTTL 64
ipInReceives 24685
ipInHdrErrors 0
ipInAddrErrors 42
ipForwDatagrams 0
ipInUnknownProtos 0
ipInDiscards 0
ipInDelivers 23025
ipOutRequests 26170
ipOutDiscards 0
ipOutNoRoutes 0
ipReasmTimeout 0
ipReasmReqds 0
ipReasmOKs 0
ipReasmFails 0
ipFragOKs 4
ipFragFails 0
ipFragCreates 8
counterBlock_tag 0:2005
disk_total 1907843072
disk_free 1083969536
disk_partition_max_used 43.18
disk_reads 16549
disk_bytes_read 1337825280
disk_read_time 7420
disk_writes 412
disk_bytes_written 1159168
disk_write_time 216
counterBlock_tag 0:2004
mem_total 1938849792
mem_free 85483520
mem_shared 0
mem_buffers 106614784
mem_cached 735801344
swap_total 0
swap_free 0
page_in 830716
page_out 566
swap_in 0
swap_out 0
counterBlock_tag 0:2003
cpu_load_one 0.070
cpu_load_five 0.060
cpu_load_fifteen 0.050
cpu_proc_run 0
cpu_proc_total 221
cpu_num 1
cpu_speed 2698
cpu_uptime 17265
cpu_user 272510
cpu_nice 50
cpu_system 178050
cpu_idle 16279880
cpu_wio 550
cpuintr 461060
cpu_sintr 41840
cpuinterrupts 5458397
cpu_contexts 5338141
cpu_steal 0
cpu_guest 0
cpu_guest_nice 0
counterBlock_tag 0:2006
nio_bytes_in 8149749
nio_pkts_in 115730
nio_errs_in 0
nio_drops_in 0
nio_bytes_out 4996846
nio_pkts_out 28451
nio_errs_out 0
nio_drops_out 0
counterBlock_tag 0:2000
hostname leaf1
UUID 33-28-66-a5-82-27-43-49-a5-f1-c1-ba-cc-6c-1d-d3
machine_type 2
os_name 2
os_release 3.4.43.Ar-4170906.4180F
endSample   ----------------------
endDatagram   =================================
There are a number of additional open source and commercial sFlow collectors available.
For example, the diagram shows how new and existing cloud based or locally hosted orchestration, operations, and security tools can leverage the sFlow-RT analytics service to gain real-time visibility.

Installing Host sFlow agent on an Arista switch

The following steps download and install the Host sFlow agent on an Arista switch and direct the telemetry stream to collector

1. Install the Host sFlow agent (hsflowd)
eos# copy extension:
eos# extension hsflowd-eos-2.0.9-1.i686.rpm
eos# bash sudo service hsflowd start
eos# copy installed-extensions boot-extensions
2. Enable eAPI, see eAPI and Unix Domain Socket
eos(config)# management api http-commands
eos(config-mgmt-api-http-cmds)# protocol unix-socket
eos(config-mgmt-api-http-cmds)# no shutdown
3. Configure switch to run hsflowd on startup:
eos(config)# event-handler hsflowd
eos(config-handler-hsflowd)# trigger on-boot
eos(config-handler-hsflowd)# action bash sudo service hsflowd start
eos(config-handler-hsflowd)# delay 60
eos(config-handler-hsflowd)# asynchronous
4. Configure sFlow Introduction to Managing EOS Devices – Setting up Management
eos(config)# sflow source-interface Management1
eos(config)# sflow destination
eos(config)# sflow run
The host metrics should immediately begin to be received at the sFlow collector.

Monday, March 20, 2017


Maximum Performance from Acropolis Hypervisor and Open vSwitch describes the network architecture within a Nutanix converged infrastructure appliance - see diagram above. This article will explore how the Host sFlow agent can be deployed to enable sFlow instrumentation in the Open vSwitch (OVS)  and deliver streaming network and system telemetry from nodes in a Nutanix cluster.
This article is based on a single hardware node running Nutanix Community Edition (CE), built following the instruction in Part I: How to setup a three-node NUC Nutanix CE cluster. If you don't have hardware readily available, the article, 6 Nested Virtualization Resources To Get You Started With Community Edition, describes how to run Nutanix CE as a virtual machine.
The sFlow standard is widely supported by network equipment vendors, which combined with sFlow from each Nutanix appliance, delivers end to end visibility in the Nutanix cluster. The following screen captures from the free sFlowTrend tool are representative examples of the data available from the Nutanix appliance.
The Network > Top N chart displays the top flows traversing OVS. In this case an HTTP connection is responsible for most of the traffic. Inter-VM and external traffic flows traverse OVS and are efficiently monitored by the embedded sFlow instrumentation.
The Hosts > CPU utilization chart shows an increase in CPU utilization due to the increased traffic.
The Hosts > Disk IO shows the Write operations associated with connection.

Installing Host sFlow agent on Nutanix appliance

The following steps install Host sFlow on a Nutanix device:

First log into the Nutanix host as root.

Next, find the latest version of the Centos 7 RPM on and use the following commands to download and install the software:
rpm -ivh hsflowd-centos7-2.0.8-1.x86_64.rpm
rm hsflowd-centos7-2.0.8-1.x86_64.rpm
Edit the /etc/hsflowd.conf file to direct sFlow telemetry to collector, enable KVM monitoring (virtual machine stats), and push sFlow configuration to OVS (network stats):
sflow {
  # collectors:
  collector { ip= udpport=6343 }
  # Open vSwitch sFlow configuration:
  ovs { }
  # KVM (libvirt) hypervisor and VM monitoring:
  kvm { }
Now start the Host sFlow daemon:
systemctl enable hsflowd.service
systemctl start hsflowd.service
Data will immediately start to appear in sFlowTrend.

Wednesday, February 22, 2017


A QUIC update on Google’s experimental transport describes some of the benefits of  the QUIC (Quick UDP Internet Connections) protocol that is now the default transport when Google's Chrome browser connects to Google services (gmail, search, etc.). Given the over 50% market share of the Chrome browser (NetMarketShare) and the popularity of Google services, it is important to be aware of the QUIC protocol and to start tracking its use of network resources.

An easy way to see if you have any QUIC traffic on your network is to use the standard sFlow instrumentation built into network switches. Configure the switches to send sFlow telemetry to an sFlow collector for visibility into network traffic.

For example, use Docker to run the sFlow-RT active-flows application to analyze the sFlow data stream:
docker run -p 6343:6343/udp -p 8008:8008 -d sflow/top-flows
Access the web interface at http://localhost:8008/ and enter the following Flow Specification to monitor QUICK flows:
Note: Real-time domain name lookups describes how sFlow-RT incorporates DNS (Domain Name Service) requests in its real-time analytics pipeline so that traffic flows can be identified by domain name.

The resulting top flows table is shown in the screen capture above. The Google addresses are identifiable by the domain names (What is and it appears that all the traffic is flowing to or from Google services (as one would expect). However, it would be nice to be able to be notified of QUIC traffic that is not associated with Google since this could represent a threat.

The following quic.js script generates events for QUIC traffic to non-Google domains:

setFlowHandler(function(rec) {
Note: Writing Applications gives an overview of sFlow-RT's embedded script API. The script logs events.

Run the script using the following command:
docker run -v `pwd`/quic.js:/sflow-rt/quic.js \
-e "RTPROP=-Ddns.servers=resolv.conf -Dscript.file=quic.js" \
-p 8008:8008 -p 6343:6343/udp sflow/top-flows
The article Exporting events using syslog shows how the script could be modified export events via syslog to SIEM tools such as Logstash and Splunk.

Monday, January 23, 2017

Telegraf, InfluxDB, Chronograf, and Kapacitor

The InfluxData TICK (Telegraf, InfluxDB, Chronograf, Kapacitor) provides a full set of integrated metrics tools, including an agent to export metrics (Telegraf), a time series database to collect and store the metrics (InfluxDB), a dashboard to display metrics (Chronograf), and a data processing engine (Kapacitor). Each of the tools is open sourced and can be used together or separately.
This article will show how industry standard sFlow agents embedded within the data center infrastructure can provide Telegraf metrics to InfluxDB. The solution uses sFlow-RT as a proxy to convert sFlow metrics into their Telegraf equivalent form so that they are immediately visible through the default Chronograf dashboards (Using a proxy to feed metrics into Ganglia described a similar approach for sending metrics to Ganglia).

The following telegraf.js script instructs sFlow-RT to periodically export host metrics to InfluxDB:
var influxdb = "";

function sendToInfluxDB(msg) {
  if(!msg || !msg.length) return;
  var req = {
  req.error = function(e) {
    logWarning('InfluxDB POST failed, error=' + e);
  try { httpAsync(req); }
  catch(e) {
    logWarning('bad request ' + req.url + ' ' + e);

var metric_names = [

var ntoi;
function mVal(row,name) {
  if(!ntoi) {
    ntoi = {};
    for(var i = 0; i < metric_names.length; i++) {
      ntoi[metric_names[i]] = i;
  return row[ntoi[name]].metricValue;

setIntervalHandler(function() {
  var i,r,msg = [];
  var vals = table('ALL',metric_names);
  for(i = 0; i < vals.length; i++) {
    r = vals[i];

    // Telegraf System plugin metrics
      +' load1='+mVal(r,'load_one')
      +' uptime='+mVal(r,'uptime')+'i');

    // Telegraf CPU plugin metrics
      +' usage_user='+(mVal(r,'cpu_user')||0)
Some notes on the script:
  1. The sentToInfluxDB() function uses the Writing data using the HTTP API to POST metrics to InfluxDB.
  2. The setIntervalHandler function retrieves a table of metrics from sFlow-RT every 15 seconds and formats them to use the same names and tags as Telegraf.
  3. The script implements Telegraf System and CPU plugin functionality.
  4. Additional metrics can easily be added to proxy additional Telegraf plugins.
  5. Writing applications provides an overview of the sFlow-RT APIs.
Start gathering metrics:
docker run -v `pwd`/telegraf.js:/sflow-rt/telegraf.js \
-e "RTPROP=-Dscript.file=telegraf.js" \
-p 8008:8008 -p 6343:6343/udp sflow/sflow-rt
Accessing the Chronograf home page brings up a table of hosts with their status and CPU load:
Clicking on the leaf1 host displays a dashboard trending key performance metrics:
Pre-processing the metrics using sFlow-RT's real-time streaming analytics engine can greatly increase scaleability by selectively exporting metrics and calculating higher level summary statistics in order to reduce the amount of data logged to the time series database. The analytics pipeline can also augment the metrics with additional metadata.
For example, Collecting Docker Swarm service metrics demonstrates how sFlow-RT can monitor dynamic service pools running under Docker Swarm and write summary statistics to InfluxDB. In this case Grafana was used to build metrics dashboard instead of Chronograf.

The open source Host sFlow agent exports an extensive range of standard sFlow metrics and has been ported to a wide range of platforms. Standard metrics describes how standardization helps reduce operational complexity. The overlap between standard sFlow metrics and Telegraf base plugin metrics makes the task of proxying straightforward.
The Host sFlow agent (and sFlow agents embedded in network switches and routers) goes beyond simple metrics export to provide detailed visibility into network traffic and articles on this blog demonstrate how sFlow-RT analytics software can be configured to generate detailed traffic flow metrics that can be streamed into InfluxDB, logged (e.g. Exporting events using syslog), or trigger control actions (e.g. DDoS mitigationDocker 1.12 swarm mode elastic load balancing).