 |
| Figure 1: Elephants and Mice |
This article demonstrates a self contained real-time Elephant flow marking solution that leverages the visibility and control features of Cumulus Linux.
SDN fabric controller for commodity data center switches provides some background on the capabilities of the commodity switch hardware used to run Cumulus Linux. The article describes how the measurement and control capabilities of the hardware can be used to maximize data center fabric performance:
- Visibility: Cumulus recently added sFlow support, providing the network wide visibility needed for control - see Cumulus Networks, sFlow and data center automation.
- Control: Cumulus Linux Netfilter (ACLs) describes how standard Linux ebtables / iptables configuration files are used to manage the ACL/policy capabilities of the hardware.
For example, the following command creates a filter called ddos1 to drop a DNS amplification attack:
curl -H "Content-Type:application/json" -X PUT --data \ '["[iptables]",\ "-A FORWARD --in-interface swp+ -d 10.10.100.10 -p udp --sport 53 -j DROP"]' \ http://10.0.0.233:8080/acl/ddos1The filter can be retrieved:
curl http://10.0.0.233:8080/acl/ddos1The following command lists the filter names:
curl http://10.0.0.233:8080/acl/The filter can be deleted:
curl -X DELETE http://10.0.0.233:8080/acl/ddos1Finally, all filters can be deleted:
curl -X DELETE http://10.0.0.233:8080/acl/Running the following Python script on the Cumulus switches provides a simple proof of concept implementation of the REST API:
#!/usr/bin/env python
from BaseHTTPServer import BaseHTTPRequestHandler,HTTPServer
from os import listdir,remove
from os.path import isfile
from json import dumps,loads
from subprocess import Popen,STDOUT,PIPE
import re
class ACLRequestHandler(BaseHTTPRequestHandler):
uripat = re.compile('^/acl/([a-z0-9]+)$')
dir = '/etc/cumulus/acl/policy.d/'
priority = '50'
prefix = 'rest-'
suffix = '.rules'
filepat = re.compile('^'+priority+prefix+'([a-z0-9]+)\\'+suffix+'$')
def commit(self):
Popen(["cl-acltool","-i"],stderr=STDOUT,stdout=PIPE).communicate()[0]
def aclfile(self,name):
return self.dir+self.priority+self.prefix+name+self.suffix
def wheaders(self,status):
self.send_response(status)
self.send_header('Content-Type','application/json')
self.end_headers()
def do_PUT(self):
m = self.uripat.match(self.path)
if None != m:
name = m.group(1)
len = int(self.headers.getheader('content-length'))
data = self.rfile.read(len)
lines = loads(data)
fn = self.aclfile(name)
f = open(fn,'w')
f.write('\n'.join(lines) + '\n')
f.close()
self.commit()
self.wheaders(201)
else:
self.wheaders(404)
def do_DELETE(self):
m = self.uripat.match(self.path)
if None != m:
name = m.group(1)
fn = self.aclfile(name)
if isfile(fn):
remove(fn)
self.commit()
self.wheaders(204)
elif '/acl/' == self.path:
for file in listdir(self.dir):
m = self.filepat.match(file)
if None != m:
remove(self.dir+file)
self.commit()
self.wheaders(204)
else:
self.wheaders(404)
def do_GET(self):
m = self.uripat.match(self.path)
if None != m:
name = m.group(1)
fn = self.aclfile(name)
if isfile(fn):
result = [];
with open(fn) as f:
for line in f:
result.append(line.rstrip('\n'))
self.wheaders(200)
self.wfile.write(dumps(result))
else:
self.wheaders(404)
elif '/acl/' == self.path:
result = []
for file in listdir(self.dir):
m = self.filepat.match(file)
if None != m:
name = m.group(1)
result.append(name)
self.wheaders(200)
self.wfile.write(dumps(result))
else:
self.wheaders(404)
if __name__ == '__main__':
server = HTTPServer(('',8080), ACLRequestHandler)
server.serve_forever()
Some notes on building a production ready solution:
- Add authentication
- Add error handling
- Script needs to run as a daemon
- Scaleability could be improved by asynchronously committing rules in batches
- Latency could be improved through use of persistent connections (SPDY, websocket)
The following sFlow-RT controller application implements large flow marking using sFlow measurements from the switch and control of ACLs using the REST API:
include('extras/json2.js');
// Define large flow as greater than 100Mbits/sec for 1 second or longer
var bytes_per_second = 100000000/8;
var duration_seconds = 1;
var id = 0;
var controls = {};
setFlow('tcp',
{keys:'ipsource,ipdestination,tcpsourceport,tcpdestinationport',
value:'bytes', filter:'direction=ingress', t:duration_seconds}
);
setThreshold('elephant',
{metric:'tcp', value:bytes_per_second, byFlow:true, timeout:4,
filter:{ifspeed:[1000000000]}}
);
setEventHandler(function(evt) {
if(controls[evt.flowKey]) return;
var rulename = 'mark' + id++;
var keys = evt.flowKey.split(',');
var acl = [
'[iptables]',
'# mark Elephant',
'-t mangle -A FORWARD --in-interface swp+ -s ' + keys[0] + ' -d ' + keys[1]
+ ' -p tcp --sport ' + keys[2] + ' --dport ' + keys[3]
+ ' -j SETQOS --set-dscp 10 --set-cos 5'
];
http('http://'+evt.agent+':8080/acl/'+rulename,
'put','application/json',JSON.stringify(acl));
controls[evt.flowKey] = {
agent:evt.agent,
dataSource:evt.dataSource,
rulename:rulename,
time: (new Date()).getTime()
};
},['elephant']);
setIntervalHandler(function() {
for(var flowKey in controls) {
var ctx = controls[flowKey];
var val = flowValue(ctx.agent,ctx.dataSource + '.tcp',flowKey);
if(val < 100) {
http('http://'+ctx.agent+':8080/acl/'+ctx.rulename,'delete');
delete controls[flowKey];
}
}
},5);
The following command line argument load the script:-Dscript.file=clmark.jsSome notes on the script:
- The 100Mbits/s threshold for large flows was selected because it represents 10% of the bandwidth of the 1Gigabit access ports on the network
- The setFlow filter specifies ingress flows since the goal is to mark flows as they enter the network
- The setThreshold filter specifies that thresholds are only applied to 1Gigabit access ports
- The event handler function triggers when new Elephant flows are detected, creating and installing an ACL to mark packets in the flow with a dscp value of 10 and a cos value of 5
- The interval handler function runs every 5 seconds and removes ACLs for flows that have completed
while true; do iperf -c 10.100.10.152 -i 20 -t 20; sleep 20; doneThe following screen capture shows a basic test setup and results:
No comments:
Post a Comment