REST API for Cumulus Linux ACLs

RESTful control of Cumulus Linux ACLs included a proof of concept script that demonstrated how to remotely control iptables entries in Cumulus Linux.  Cumulus Linux in turn converts the standard Linux iptables rules into the hardware ACLs implemented by merchant silicon switch ASICs to deliver line rate filtering.

Previous blog posts demonstrated how remote control of Cumulus Linux ACLs can be used for DDoS mitigation and Large "Elephant" flow marking.

A more advanced version of the script is now available on GitHub:

https://github.com/pphaal/acl_server/

The new script adds the following features:

1. It now runs as a daemon.
2. Exceptions generated by cl-acltool are caught and handled
3. Rules are compiled asynchronously, reducing response time of REST calls
4. Updates are batched, supporting hundreds of operations per second

The script doesn't provide any security, which may be acceptable if access to the REST API is limited to the management port, but is generally unacceptable for production deployments.

Fortunately, Cumulus Linux is a open Linux distribution that allows additional software components to be installed. Rather than being forced to add authentication and encryption to the script, it is possible to install additional software and leverage the capabilities of a mature web server such as Apache. The operational steps needed to secure access to Apache are well understood and the large Apache community ensures that security issues are quickly identified and addressed.

This article will demonstrate how Apache can be used to proxy REST operations for the acl_server script, allowing familiar Apache features to be applied to secure access to the ACL service.

Download the acl_server script from GitHub

wget https://raw.githubusercontent.com/pphaal/acl_server/master/acl_server

Change the following line to limit access to requests made by other processes on the switch:

server = HTTPServer(('',8080), ACLRequestHandler)

Limiting access to localhost, 127.0.0.1:

server = HTTPServer(('127.0.0.1',8080), ACLRequestHandler)

Next, install the script on the switch:

sudo mv acl_server /etc/init.d/
sudo chown root:root /etc/init.d/acl_server
sudo chmod 755 /etc/init.d/acl_server
sudo service acl_server start
sudo update-rc.d acl_server start

Now install Apache:

sudo sh -c "echo 'deb http://ftp.us.debian.org/debian  wheezy main contrib' \
>>/etc/apt/sources.list.d/deb.list"
sudo apt-get update
sudo apt-get install apache2

Next enable the Apache proxy module:

sudo a2enmod proxy proxy_http

Create an Apache configuration file /etc/apache2/conf.d/acl_server with the following contents:

<ifmodule mod_proxy.c>
  ProxyRequests off
  ProxyVia off
  ProxyPass        /acl/ http://127.0.0.1:8080/acl/
  ProxyPassReverse /acl/ http://127.0.0.1:8080/acl/
</ifmodule>

Make any additional changes to the Apache configuration to encrypt and authenticate requests.

Finally, restart Apache:

sudo service apache2 restart

These above steps are easily automated using tools like Puppet or Ansible that are available for Cumulus Linux.

The following examples demonstrate the REST API.

Create an ACL

curl -H "Content-Type:application/json" -X PUT --data '["[iptables]","-A FORWARD --in-interface swp+ -d 10.10.100.10 -p udp --sport 53 -j DROP"]' http://10.0.0.233/acl/ddos1

ACLs are sent as a JSON encoded array of strings. Each string will be written as a line in a file stored under /etc/cumulus/acl/policy.d/ - See Cumulus Linux: Netfilter - ACLs. For example, the rule above will be written to the file 50rest-ddos1.rules with the following content:

[iptables]
-A FORWARD --in-interface swp+ -d 10.10.100.10 -p udp --sport 53 -j DROP

This iptables rule blocks all traffic from UDP port 53 (DNS) to host 10.10.100.10. This is the type of rule that might be inserted to block a DNS amplification attack.

Retrieve an ACL

curl http://10.0.0.233/acl/ddos1

Returns the result:

["[iptables]", "-A FORWARD --in-interface swp+ -d 10.10.100.10 -p udp --sport 53 -j DROP"]

List ACLs

curl http://10.0.0.233/acl/

Returns the result:

["ddos1"]

Delete an ACL

curl -X DELETE http://10.0.0.233/acl/ddos1

Delete all ACLs

curl -X DELETE http://10.0.0.233/acl/

Note this doesn't delete all the ACLs, just the ones created using the REST API. All default ACLs or manually created ACLs are inaccessible through the REST API.

The acl_server batches and compiles changes after the HTTP requests complete. Batching has the benefit of increasing throughput and reducing request latency, but makes it difficult to track compilation errors since they are reported later. The acl_server catches the output and status when running cl-acltool and attaches an HTTP Warning header to subsequent requests to indicate that the last compilation failed:

HTTP/1.0 204 No Content
Server: BaseHTTP/0.3 Python/2.7.3
Date: Thu, 12 Feb 2015 05:31:06 GMT
Accept: application/json
Content-Type: application/json
Warning: 199 - "check lasterror"

The output of cl-acltool can be retrieved:

curl http://10.0.0.233/acl/lasterror

returns the result:

{"returncode": 255, "lines": [...]}

The REST API is intended to be used by automation systems and so syntax problems with the ACLs they generate should be rare and are the result of a software bug. A controller using this API should check responses for the presence of the last error Warning, log the lasterror information so that the problem can be debugged, and finally delete all the rules created through the REST API to restore the system to its default state.

While this REST API could be used as a convenient way to manually push an ACL to a switch, the API is intended to be part of automation solutions that combine real-time traffic analytics with automated control. Cumulus Linux includes standard sFlow measurement support, delivering real-time network wide visibility to drive solutions that include: DDoS mitigation, enforcing black lists, marking large flows, ECMP load balancing, packet brokers etc.

Fabric visibility with Cumulus Linux

A leaf and spine fabric is challenging to monitor. The fabric spreads traffic across all the switches and links in order to maximize bandwidth. Unlike traditional hierarchical network designs, where a small number of links can be monitored to provide visibility, a leaf and spine network has no special links or switches where running CLI commands or attaching a probe would provide visibility. Even if it were possible to attach probes, the effective bandwidth of a leaf and spine network can be as high as a Petabit/second, well beyond the capabilities of current generation monitoring tools.

The 2 minute video provides an overview of some of the performance challenges with leaf and spine fabrics and demonstrates Fabric View - a monitoring solution that leverages industry standard sFlow instrumentation in commodity data center switches to provide real-time visibility into fabric performance.

Fabric View is free to try, just register at http://www.myinmon.com/ and request an evaluation. The software requires an accurate network topology in order to characterize performance and this article will describe how to obtain the topology from a Cumulus Networks fabric.

Complex Topology and Wiring Validation in Data Centers describes how Cumulus Networks' prescriptive topology manager (PTM) provides a simple method of verifying and enforcing correct wiring topologies. The following ptm.py script converts the topology from PTM's dot notation to the JSON representation used by Fabric View:

#!/usr/bin/env python

import sys, re, fileinput, requests, json

url = sys.argv[1]
top = {'links':{}}

def dequote(s):
  if (s[0] == s[-1]) and s.startswith(("'", '"')):
    return s[1:-1]
  return s

l = 1
for line in fileinput.input(sys.argv[2:]):
  link = re.search('([\S]+):(\S+)\s*(--|->)\s*(\S+):([^\s;,]+)',line)
  if link:
    s1 = dequote(link.group(1))
    p1 = dequote(link.group(2))
    s2 = dequote(link.group(4))
    p2 = dequote(link.group(5))
    linkname = 'L%d' % (l)
    l += 1
    top['links'][linkname] = {'node1':s1,'port1':p1,'node2':s2,'port2':p2}

requests.put(url,data=json.dumps(top),headers={'content-type':'application/json'})

The following example demonstrates how to use the script,converting the file topology.dot to JSON and posting the result to the Fabric View server running on host fabricview:

./ptm.py http://fabricview:8008/script/fabric-view.js/topology/json topology.dot

Cumulus Networks, sFlow and data center automation describes how enable sFlow on Cumulus Linux. Configure all the switches in the leaf and spine fabric to send sFlow to the Fabric View server and you should immediately start to see data through the web interface, http://fabricview:8008/. The video provides a quick walkthrough of the software features.

DDoS flood protection

Denial of Service attacks represents a significant impact to on-going operations of many businesses. When most revenue is derived from on-line operation, a DDoS attack can put a company out of business. There are many flavors of DDoS attacks, but the objective is always the same: to saturate a resource, such as a router, switch, firewall or web server, with multiple simultaneous and bogus requests, from many different sources. These attacks generate large volumes of traffic, 100Gbit/s attacks are now common, making mitigation a challenge.

The 3 minute video demonstrates Flood Protect - a DDoS mitigation solution that leverages industry standard sFlow instrumentation in commodity data center switches to provide real-time detection and mitigation of DDoS attacks. Flood Protect is an application running on InMon's Switch Fabric Accelerator SDN controller. Other applications provide visibility and accelerate fabric performance applying controls reduce latency and increase throughput.

An early version of Flood Protect won the 2014 SDN Idol competition in a joint demonstration with Brocade Networks.

Visit sFlow.com to learn more, evaluate pre-release versions of these products, or discuss requirements.

Stop thief!

The Host-sFlow project recently added added CPU steal to the set of CPU metrics exported.

steal (since Linux 2.6.11)
       (8) Stolen time, which is the time spent in other operating systems
       when running in a virtualized environment

Keeping close track of the stolen time metric is particularly import when running managing virtual machines in a public cloud. For example, Netflix and Stolen Time includes the discussion:

So how does Netflix handle this problem when using Amazon’s Cloud? Adrian admits that they tracked this statistic so closely that when an instance crossed a stolen time threshold the standard operating procedure at Netflix was to kill the VM and start it up on a different hypervisor. What Netflix realized over time was that once a VM was performing poorly because another VM was crashing the party, usually due to a poorly written or compute intensive application hogging the machine, it never really got any better and their best learned approach was to get off that machine.

The following articles describe how to monitor public cloud instances using Host sFlow agents:

• Amazon Elastic Compute Cloud (EC2)
• Rackspace cloudservers

The CPU steal metric is particularly relevant to Network Function Virtualization (NFV). Virtual appliances implementing network functions such as load balancing are particularly sensitive to stolen CPU cycles that can severely impact application response times. Application Delivery Controller (ADC) vendors export sFlow metrics from their physical and virtual appliances - sFlow leads convergence of multi-vendor application, server, and network performance management.  The addition of CPU steal to the set of sFlow metrics exported by virtual appliances will allow the NFV orchestration tools to better optimize service pools.

InfluxDB and Grafana

Cluster performance metrics describes how to use sFlow-RT to calculate metrics and post them to Graphite. This article will describe how to use sFlow with the InfluxDB time series database and Grafana dashboard builder.

The diagram shows the measurement pipeline. Standard sFlow measurements from hosts, hypervisors, virtual machines, containers, load balancers, web servers and network switches stream to the sFlow-RT real-time analytics engine. Over 40 vendors implement the sFlow standard and compatible products are listed on sFlow.org. The open source Host sFlow agent exports standard sFlow metrics from hosts. For additional background, the Velocity conference talk provides an introduction to sFlow and case study from a large social networking site.

It is possible to simply convert the raw sFlow metrics into InfluxDB metrics. The sflow2graphite.pl script provides an example that can be modified to support InfluxDB's native format, or used unmodified with the InfluxDB Graphite input plugin. However, there are scaleability advantages to placing the sFlow-RT analytics engine in front of the time series database. For example, in large scale cloud environments the metrics for each member of a dynamic pool isn't necessarily worth trending since virtual machines are frequently added and removed. Instead, sFlow-RT tracks all the members of the pool, calculates summary statistics for the pool, and logs the summary statistics to the time series database. This pre-processing can significantly reduce storage requirements, reducing costs and increasing query performance. The sFlow-RT analytics software also calculates traffic flow metrics, hot/missed Memcache keys, top URLs, exports events via syslog to Splunk, Logstash etc. and provides access to detailed metrics through its REST API. 

First install InfluxDB - in this case the software has been installed on host 10.0.0.30.

Next install sFlow-RT:

wget http://www.inmon.com/products/sFlow-RT/sflow-rt.tar.gz
tar -xvzf sflow-rt.tar.gz
cd sflow-rt

Edit the init.js script and add the following lines (modifying the dbURL to send metrics to the InfluxDB instance):

var dbURL = "http://10.0.0.30:8086/db/inmon/series?u=root&p=root";

setIntervalHandler(function() {
  var metrics = ['min:load_one','q1:load_one','med:load_one',
                 'q3:load_one','max:load_one'];
  var vals = metric('ALL',metrics,{os_name:['linux']});
  var body = [];
  for each (var val in vals) {
     body.push({name:val.metricName,columns:['val'],points:[[val.metricValue]]});
  }
  http(dbURL,'post', 'application/json', JSON.stringify(body));
} , 15);

Now start sFlow-RT:

./start.sh

The script makes an sFlow-RT metrics() query every 15 seconds and posts the results to InfluxDB.

The screen capture shows InfluxDB's SQL like query language and a basic query demonstrating that the metrics are being logged in the database. However, the web interface is rudimentary and a dashboard builder simplifies querying and presentation of the time series data.

Grafana is a powerful HTML 5 dashboard building tool that supports InfluxDB, Graphite, and OpenTSDB.

The screen shot shows the Grafana query builder, offering simple drop down menus that make it easy to build complex charts. The resulting chart, shown below, can be combined with additional charts to build a custom dashboard.

The sFlow standard delivers the comprehensive instrumentation of data center infrastructure and is easily integrated with DevOps tools - see Visibility and the software defined data center

Update January 31, 2016:

The InfluxDB REST API changed with version 0.9 and the above sFlow-RT script will no longer work. The new API is described in Creating a database using the HTTP API. The following version of the script has been updated to use the new API:

var dbURL = "http://10.0.0.30:8086/write?db=mydb";

setIntervalHandler(function() {
  var metrics = ['min:load_one','q1:load_one','med:load_one',
                 'q3:load_one','max:load_one'];
  var vals = metric('ALL',metrics,{os_name:['linux']});
  var body = [];
  for each (var val in vals) {
     body.push(val.metricName.replace(/[^a-zA-Z0-9_]/g,'_') + ' value=' + val.metricValue);
  }
  try { http(dbURL,'post', 'text/plain', body.join('\n')); }
  catch(e) { logWarning('http error ' + e); }
} , 15);

Update April 27, 2016

The sFlow-RT software no longer ships with an init.js file.

Instead, create an influxdb.js file in the sFlow-RT home directory and add the JavaScript code. Next, edit the start.sh file to add a script.file=influxdb.js option, i.e.

RT_OPTS="-Dscript.file=influxdb.js -Dsflow.port=6343 -Dhttp.port=8008"

The script should be loaded when sFlow-RT is started.

Monitoring leaf and spine fabric performance

A leaf and spine fabric is challenging to monitor. The fabric spreads traffic across all the switches and links in order to maximize bandwidth. Unlike traditional hierarchical network designs, where a small number of links can be monitored to provide visibility, a leaf and spine network has no special links or switches where running CLI commands or attaching a probe would provide visibility. Even if it were possible to attach probes, the effective bandwidth of a leaf and spine network can be as high as a Petabit/second, well beyond the capabilities of current generation monitoring tools.

The 2 minute video provides an overview of some of the performance challenges with leaf and spine fabrics and demonstrates Fabric View - a monitoring solution that leverages industry standard sFlow instrumentation in commodity data center switches to provide real-time visibility into fabric performance. Fabric View is an application running on InMon's Switch Fabric Accelerator SDN controller. Other applications can automatically respond to problems and apply controls to protect against DDoS attacks, reduce latency and increase throughput.

Visit sFlow.com to learn more, evaluate pre-release versions of these products, or discuss requirements.

Open vSwitch 2014 Fall Conference

Open vSwitch is an open source software virtual switch that is popular in cloud environments such as OpenStack. Open vSwitch is a standard Linux component that forms the basis of a number of commercial and open source solutions for network virtualization, tenant isolation, and network function virtualization (NFV) - implementing distributed virtual firewalls and routers.

The recent Open vSwitch 2014 Fall Conference agenda included a wide variety speakers addressing a range of topics, including: large scale operation experiences at Rackspace, implementing stateful firewalls, Docker networking,  and acceleration technologies (Intel DPDK and Netmap/VALE).

The video above is a recording of the following sFlow related talk from the conference:

Traffic visibility and control with sFlow (Peter Phaal, InMon)
sFlow instrumentation has been included in Open vSwitch since version 0.99.1 (released 25 Jan 2010). This talk will introduce the sFlow architecture and discuss how it differs from NetFlow/IPFIX, particularly in regards to delivering real-time flow analytics to an SDN controller. The talk will demonstrate that sFlow measurements from Open vSwitch are identical to sFlow measurements made in hardware on bare metal switches, providing unified, end-to-end, measurement across physical and virtual networks. Finally, Open vSwitch / Mininet will be used to demonstrate Elephant flow detection and marking using a combination of sFlow and OpenFlow.

Slides and videos for all the conference talks will soon be available on the Open vSwitch web site.