Friday, July 12, 2019

Arista BGP FlowSpec

The video of a talk by Peter Lundqvist from DKNOG9 describes BGP FlowSpec, use cases, and details of Arista's implementation.

FlowSpec for real-time control and sFlow telemetry for real-time visibility is a powerful combination that can be used to automate DDoS mitigation and traffic engineering. The article, Real-time DDoS mitigation using sFlow and BGP FlowSpec, gives an example using the sFlow-RT analytics software.

EOS 4.22 includes support for BGP FlowSpec. This article uses a virtual machine running vEOS-4.22 to demonstrate how to configure FlowSpec and sFlow so that the switch can be controlled by an sFlow-RT application (such as the DDoS mitigation application referenced earlier).

The following output shows the EOS configuration statements related to sFlow and FlowSpec:
service routing protocols model multi-agent
sflow sample 16384
sflow polling-interval 30
sflow destination
sflow run
interface Ethernet1
   flow-spec ipv4 ipv6
interface Management1
   ip address
ip routing
router bgp 65096
   neighbor remote-as 65070
   neighbor transport remote-port 1179
   neighbor send-community extended
   neighbor maximum-routes 12000 
   address-family flow-spec ipv4
      neighbor activate
   address-family flow-spec ipv6
      neighbor activate
The following JavaScript statement configures the FlowSpec connection on the sFlow-RT side:
The FlowSpec functionality is exposed through sFlow-RT's REST API.
The sFlow-RT REST API Explorer is a simple way to exercise the FlowSpec functionality. In this case we are going to push a rule that blocks traffic from UDP port 53 targeted at host This type of rule is typically used to block a DNS amplification attack.

The following output on the switch verifies that the rule has been received:
localhost#sho bgp flow-spec ipv4 detail
BGP Flow Specification rules for VRF default
Router identifier, local AS number 65096
BGP Flow Specification Matching Rule for;*;IP:17;SP:53;
 Rule identifier: 3851506952
 Matching Rule:
   Destination Prefix:
   Source Prefix: *
   IP Protocol: 17
   Source Port: 53
 Paths: 1 available
    from (
      Origin IGP, metric -, localpref 100, weight 0, valid, external, best
      Actions: Drop
In practice the process of adding and removing filtering rules can be completely automated by an sFlow-RT application. The combination of real-time sFlow analytics with the real-time control provided by FlowSpec allows DDoS attacks to be detected and blocked within seconds.

Friday, June 14, 2019

Mininet flow analytics with custom scripts

Mininet flow analytics describes how to use the helper script that ships with the sFlow-RT analytics engine to enable sFlow telemetry, e.g.
sudo mn --custom sflow-rt/extras/ --link tc,bw=10 \
--topo tree,depth=2,fanout=2
Mininet, ONOS, and segment routing provides an example using a Custom Topology, e.g.
sudo env ONOS= mn --custom,sflow-rt/extras/ \
--link tc,bw=10 --topo=sr '--controller=remote,ip=$ONOS,port=6653'
This article describes how to incorporate sFlow monitoring in a fully custom Mininet script. Consider the following script based on Working with Mininet:
from mininet.topo import Topo
from import Mininet
from mininet.util import dumpNodeConnections
from mininet.log import setLogLevel

class SingleSwitchTopo(Topo):
    "Single switch connected to n hosts."
    def build(self, n=2):
        switch = self.addSwitch('s1')
        # Python's range(N) generates 0..N-1
        for h in range(n):
            host = self.addHost('h%s' % (h + 1))
            self.addLink(host, switch)

def simpleTest():
    "Create and test a simple network"
    topo = SingleSwitchTopo(n=4)
    net = Mininet(topo)
    print "Dumping host connections"
    print "Testing bandwidth between h1 and h4"
    h1, h4 = net.get( 'h1', 'h4' )
    net.iperf( (h1, h4) )

if __name__ == '__main__':
    # Tell mininet to print useful information
Add the highlighted lines to incorporate sFlow telemetry:
from mininet.topo import Topo
from import Mininet
from mininet.util import dumpNodeConnections
from mininet.log import setLogLevel
from mininet.util import customClass
from import TCLink

# Compile and run sFlow helper script
# - configures sFlow on OVS
# - posts topology to sFlow-RT

# Rate limit links to 10Mbps
link = customClass({'tc':TCLink}, 'tc,bw=10')

class SingleSwitchTopo(Topo):
    "Single switch connected to n hosts."
    def build(self, n=2):
        switch = self.addSwitch('s1')
        # Python's range(N) generates 0..N-1
        for h in range(n):
            host = self.addHost('h%s' % (h + 1))
            self.addLink(host, switch)

def simpleTest():
    "Create and test a simple network"
    topo = SingleSwitchTopo(n=4)
    net = Mininet(topo,link=link)
    print "Dumping host connections"
    print "Testing bandwidth between h1 and h4"
    h1, h4 = net.get( 'h1', 'h4' )
    net.iperf( (h1, h4) )

if __name__ == '__main__':
    # Tell mininet to print useful information
When running the script the highlighted output confirms that sFlow has been enabled and the topology has been posted to sFlow-RT:
pp@mininet:~$ sudo ./
*** Creating network
*** Adding controller
*** Adding hosts:
h1 h2 h3 h4 
*** Adding switches:
*** Adding links:
(10.00Mbit) (10.00Mbit) (h1, s1) (10.00Mbit) (10.00Mbit) (h2, s1) (10.00Mbit) (10.00Mbit) (h3, s1) (10.00Mbit) (10.00Mbit) (h4, s1) 
*** Configuring hosts
h1 h2 h3 h4 
*** Starting controller
*** Starting 1 switches
s1 ...(10.00Mbit) (10.00Mbit) (10.00Mbit) (10.00Mbit) 
*** Enabling sFlow:
*** Sending topology
Dumping host connections
h1 h1-eth0:s1-eth1
h2 h2-eth0:s1-eth2
h3 h3-eth0:s1-eth3
h4 h4-eth0:s1-eth4
Testing bandwidth between h1 and h4
*** Iperf: testing TCP bandwidth between h1 and h4 
*** Results: ['6.32 Mbits/sec', '6.55 Mbits/sec']
*** Stopping 1 controllers
*** Stopping 4 links
*** Stopping 1 switches
*** Stopping 4 hosts
h1 h2 h3 h4 
*** Done
Mininet dashboard and Mininet weathermap describe the sFlow-RT Mininet Dashboard application shown at the top of this article. The tool provides a real-time visualization of traffic flowing over the Mininet network. Writing Applications describes how to develop custom analytics applications for sFlow-RT.

Wednesday, May 8, 2019

Secure forwarding of sFlow using ssh

Typically sFlow datagrams are sent unencrypted from agents embedded in switches and routers to a local collector/analyzer. Sending sFlow datagrams over the management VLAN or out of band management network generally provides adequate isolation and security within the site. Inter-site traffic within an organization is typically carried over a virtual private network (VPN) which encrypts the data and protects it from eavesdropping.

This article describes a simple method of carrying sFlow datagrams over an encrypted ssh connection which can be useful in situations where a VPN is not available, for example, sending sFlow to an analyzer in the public cloud, or to an external consultant.

The diagram shows the elements of the solution. A collector on the site receives sFlow datagrams from the network devices and uses the script to convert the datagrams into line delimited hexadecimal strings that are sent over an ssh connection to another instance of running on the analyzer that converts the hexadecimal strings back to sFlow datagrams.

The following Python script accomplishes the task:

import socket
import sys
import argparse

parser = argparse.ArgumentParser(description='Serialize/deserialize sFlow')
parser.add_argument('-c', '--collector', default='')
parser.add_argument('-s', '--server')
parser.add_argument('-p', '--port', type=int, default=6343)
args = parser.parse_args()


if(args.server != None):
  while True:
    line = sys.stdin.readline()
    if not line:
    buf = bytearray.fromhex(line[:-1])
    sock.sendto(buf, (args.server, args.port))
  while True:
    buf = sock.recv(2048)
    if not buf:
    print buf.encode('hex')
Create a user account on both the collector and analyzer machines, in this example the user is pp. Next copy the script to both machines.

If you log into the collector machine, following command will send sFlow to the analyzer machine:
./ | ssh pp@analyzer './ -s'
If you log into the analyzer machine, the following command will retrieve sFlow from the collector machine:
ssh pp@collector './' | ./ -s
If a permanent connection is required, it is relatively straightforward to create a daemon using systemd. In this example, the service is being installed on the collector machine by performing the following steps:
First log into the collector generate an ssh key:
Next, install the key on the analyzer system:
ssh-copy-id pp@analyzer
Now create the systemd service file, /etc/systemd/system/sflow-tunnel.service:
Description=sFlow tunnel

ExecStart=/bin/sh -c "/home/pp/ | /usr/bin/ssh pp@analyzer './ -s'"

Finally, use the systemctl command to enable and start the daemon:
sudo systemctl enable sflow-tunnel.service
sudo systemctl start sflow-tunnel.service
A simple way to confirm that sFlow is arriving on the analyzer machine is to use sflowtool.

There are numerous articles on this blog describing how the sFlow-RT analytics software can be used to integrate sFlow telemetry with popular metrics and SIEM (security information and event management) tools.

Tuesday, April 23, 2019

Prometheus exporter

Prometheus is an open source time series database optimized to collect large numbers of metrics from cloud infrastructure. This article will explore how industry standard sFlow telemetry streaming supported by network devices (Arista, Aruba, Cisco, Dell, Huawei, Juniper, etc.) and Host sFlow agents (Linux, Windows, FreeBSD, AIX, Solaris, Docker, Systemd, Hyper-V, KVM, Nutanix AHV, Xen) can be integrated with Prometheus to extend visibility into the network.

The diagram above shows the elements of the solution: sFlow telemetry streams from hosts and switches to an instance of sFlow-RT. The sFlow-RT analytics software converts the raw measurements into metrics that are accessible through a REST API. The sflow-rt/prometheus application extends the REST API to include native Prometheus exporter functionality allowing Prometheus to retrieve metrics. Prometheus stores metrics in a time series database that can be queries by Grafana to build dashboards.

The Docker sflow/prometheus image provides a simple way to run the application:
docker run --name sflow-rt -p 8008:8008 -p 6343:6343/udp -d sflow/prometheus
Configure sFlow agents to send data to the collector,, on port 6343.

Verify that the metrics are available using cURL:
$ curl
ifinucastpkts{agent="",datasource="2",host="server",ifname="enp3s0"} 9.44
ifoutdiscards{agent="",datasource="2",host="server",ifname="enp3s0"} 0
ifoutbroadcastpkts{agent="",datasource="2",host="server",ifname="enp3s0"} 0
ifinerrors{agent="",datasource="2",host="server",ifname="enp3s0"} 0
If the sFlow agents don't provide host and ifname information, enable SNMP to retrieve sysName and ifName data to populate these fields:
docker run --name sflow-rt -p 8008:8008 -p 6343:6343/udp -d sflow/prometheus \
By default SNMP version 2c will be used with the public community string. Additional System Properties can be used to override these defaults.
Now define a metrics "scraping" job in the Prometheus configuration file, prometheus.yml
  scrape_interval:     15s
  evaluation_interval: 15s

  # - "first.rules"
  # - "second.rules"

  - job_name: 'sflow-rt'
    metrics_path: /app/prometheus/scripts/export.js/dump/ALL/ALL/txt
      - targets: ['']
Now start Prometheus:
docker run --name prometheus --rm -v $PWD/data:/prometheus \
-v $PWD/prometheus.yml:/etc/prometheus/prometheus.yml \
-p 9090:9090 -d prom/prometheus
The screen capture above shows the Prometheus web interface (accessed on port 9090).
Grafana is open source time series analysis software. The ability to pull data from many data sources and the extensive range of charting options makes Grafana an attractive tool for building operations dashboards.

The following command shows how to run Grafana under Docker:
docker run --name grafana -v $PWD/data:/var/lib/grafana \
-p 3000:3000 -d grafana/grafana
Access the Grafana web interface on port 3000, configure a data source for the Prometheus database, and start building dashboards. The screen capture above shows the same chart built earlier using the native Prometheus interface.

Wednesday, March 6, 2019


Loggly is a cloud logging and and analysis platform. This article will demonstrate how to integrate network events generated from industry standard sFlow instrumentation build into network switches.
Loggly offers a free 14 day evaluation, so you can try this example at no cost.
ICMP unreachable describes how monitoring ICMP destination unreachable messages can help identify misconfigured hosts and scanning behavior. The article uses the sFlow-RT real-time analytics software to process the raw sFlow and report on unreachable messages.

The following script, loggly.js, modifies the sFlow-RT script from the article to send events to the Loggly HTTP/S Event Endpoint:
var token = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx';
var url = ''+token+'/tag/http/';

var keys = [

for (var i = 0; i < keys.length; i++) {
  var key = keys[i];
  setFlow(key, {
    keys:'macsource,ipsource,macdestination,ipdestination,' + key,

setFlowHandler(function(rec) {
  var keys = rec.flowKeys.split(',');
  var msg = {,

  try { http(url,'post','application/json',JSON.stringify(msg)); }
  catch(e) { logWarning(e); };
}, keys);
Some notes on the script:
  1. Modify the script to use the correct token for your Loggly account.
  2. Including MAC addresses can help identify hosts even if they spoof IP addresses
  3. See Writing Applications for more information.
Run the script using the sflow/sflow-rt docker image:
docker run -p 6343:6343/udp -v $PWD/loggly.js:/loggly.js \
sflow/sflow-rt -Dscript.file=/loggly.js
Events should now start appearing in Loggly.
The Loggly Live Tail page can be used to verify that the logs are being received. The screen capture at the start of this article shows a chart trending events by the host that triggered them, identifying as the source of the network scan.

The loggly.js script can easily be modified to track and log different types of network activity. For example, Blacklists describes how to download a set of blacklisted addresses, match traffic against the blacklist and generate events for the matches.

Intranet DDoS attacks describes the threats posed by IoT (Internet of Things) devices and the need for visibility throughout the network in order to tackle these threats. Incorporating sFlow in the monitoring strategy extends visibility beyond the firewalls to the entire network.

In addition to generating events, sFlow analytics can be used to deliver performance metrics. The article, Cloud analytics, describes how to use sFlow-RT to send performance metrics to the Librato cloud service - also part of Solarwinds.