Wednesday, July 5, 2017

BGP FlowSpec on white box switch

BGP FlowSpec is a method of distributing access control lists (ACLs) using the BGP protocol. Distributed denial of service (DDoS) mitigation is an important use case for the technology, allowing a targeted network to push filters to their upstream provider to selectively remove the attack traffic.

Unfortunately, FlowSpec is currently only available on high end routing devices and so experimenting with the technology is expensive. Looking for an alternative, Cumulus Linux is an open Linux platform that allows users to install Linux packages and develop their own software.

This article describes a proof of concept implementation of basic FlowSpec functionality using ExaBGP installed on a free Cumulus VX virtual machine.  The same solution can be run on inexpensive commodity white box hardware to deliver terabit traffic filtering in a production network.

First, install latest version of ExaBGP on the Cumulus Linux switch:
curl -L https://github.com/Exa-Networks/exabgp/archive/4.0.0.tar.gz | tar zx
Now define the handler, acl.py, that will convert BGP FlowSpec updates into standard Linux netfilter/iptables entries used by Cumulus Linux to specify hardware ACLs (see Netfilter - ACLs):
#!/usr/bin/python
 
import json
import re
from os import listdir,remove
from os.path import isfile
from subprocess import Popen,STDOUT,PIPE
from sys import stdin, stdout, stderr

id = 0
acls = {}

dir = '/etc/cumulus/acl/policy.d/'
priority = '60'
prefix = 'flowspec'
bld = '.bld'
suffix = '.rules'

def commit():
  Popen(["cl-acltool","-i"],stderr=STDOUT,stdout=PIPE).communicate()[0]

def aclfile(name):
  global dir,priority,prefix,suffix
  return dir+priority+prefix+name+suffix

def handleSession(state):
  if "down" == state:
    for key,rec in acls.items():
      fn = aclfile(str(rec['id']))
      if isfile(fn):
        remove(fn)
      del acls[key]
    commit()

def buildACL(flow,action):
  acl = "[iptables]\n-A FORWARD --in-interface swp+"
  if flow.get('protocol'):
    acl = acl + " -p " + re.sub('[!<>=]','',flow['protocol'][0])
  if flow.get('source-ipv4'):
    acl = acl + " -s " + flow['source-ipv4'][0]
  if flow.get('destination-ipv4'):
    acl = acl + " -d " + flow['destination-ipv4'][0]
  if flow.get('source-port'):
    acl = acl + " --sport " + re.sub('[!<>=]','',flow['source-port'][0])
  if flow.get('destination-port'):
    acl = acl + " --dport " + re.sub('[!<>=]','',flow['destination-port'][0])
  acl = acl + " -j DROP\n"
  return acl

def handleFlow(add,flow,action):
  global id
  key = flow['string']
  if add:
    acl = buildACL(flow,action)
    id = id + 1
    acls[key] = {"acl":acl,"id":id}
    fn = aclfile(str(id))
    f = open(fn,'w')
    f.write(acl)
    f.close()
    commit()
  elif key in acls:
    rec = acls[key]
    fn = aclfile(str(rec['id']))
    if isfile(fn):
      remove(fn)
    del acls[key]
    commit()
  
while True:
  try:
     line = stdin.readline().strip()
     msg = json.loads(line)
     type = msg["type"]
     if "state" == type:
       state = msg["neighbor"]["state"]
       handleSession(state)
     elif "update" == type:
       update = msg["neighbor"]["message"]["update"] 
       if update.get('announce'):
         flow = update["announce"]["ipv4 flow"]["no-nexthop"][0]
         community = update["attribute"]["extended-community"][0]
         handleFlow(True,flow,community)
       elif update.get('withdraw'):
         flow = update["withdraw"]["ipv4 flow"][0]
         handleFlow(False,flow,None)
     else:
       pass
  except IOError:
     pass
Note: This script is a simple demonstration of the concept that has significant limitations: there is no error handling, the only action is to drop traffic, and FlowSpec comparison operators are ignored. The script is is based on the article RESTful control of Cumulus Linux ACLs.

Update July 6, 2017: An improved version of the acl.py script is now available in the ExaBGP repository on GitHub, see flow.py

Next, create the exabgp.conf file:
process acl {
   run ./acl.py;
   encoder json;
}

template {
  neighbor controller {
    family {
      ipv4 flow;
    }
    api speaking {
      processes [ acl ];
      neighbor-changes;
      receive {
         parsed;
         update;
      }
    }
  }
}

neighbor 10.0.0.70 {
  inherit controller;
  router-id 10.0.0.140;
  local-as 65140;
  peer-as 65070;
  local-address 0.0.0.0;
  connect 1179;
}
Finally, run ExaBGP:
sudo env exabgp.daemon.user=root exabgp-4.0.0/sbin/exabgp exabgp.conf
This configuration instructs ExaBGP to connect to the controller, 10.0.0.162, and prepare to receive BGP FlowSpec messages. When a BGP message is received, ExaBGP decodes the message and passes it on in the form of a JSON encoded string to the acl.py program. For example, the following FlowSpec message was sent to block an NTP reflection DDoS attack (traffic from UDP port 123) targeting host 192.168.0.1:
{
 "exabgp": "4.0.0",
 "time": 1498868955.31,
 "host": "tor-router",
 "pid": 3854,
 "ppid": 3849,
 "counter": 6,
 "type": "update",
 "neighbor": {
  "address": {
   "local": "0.0.0.0",
   "peer": "10.0.0.70"
  },
  "asn": {
   "local": "65140",
   "peer": "65070"
  },
  "direction": "receive",
  "message": {
   "update": {
    "attribute": {
     "origin": "igp",
     "as-path": [
      65140
     ],
     "confederation-path": [],
     "local-preference": 100,
     "extended-community": [
      9225060886715040000
     ]
    },
    "announce": {
     "ipv4 flow": {
      "no-nexthop": [
       {
        "destination-ipv4": [
         "192.168.0.1/32"
        ],
        "protocol": [
         "=udp"
        ],
        "source-port": [
         "=123"
        ],
        "string": "flow destination-ipv4 192.168.0.1/32 protocol =udp source-port =123"
       }
      ]
     }
    }
   }
  }
 }
}
The acl.py script wrote the file /etc/cumulus/acl/policy.d/60flowspec1.rules with an iptables representation of the FlowSpec message:
[iptables]
-A FORWARD --in-interface swp+ -p udp -d 192.168.0.1/32 --sport 123 -j DROP
The acl.py script also invoked the cl-acltool command to install the new rule, which can be verified using iptables:
sudo iptables --list FORWARD
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
DROP       all  --  240.0.0.0/5          anywhere            
DROP       all  --  loopback/8           anywhere            
DROP       all  --  base-address.mcast.net/4  anywhere            
DROP       all  --  255.255.255.255      anywhere            
DROP       udp  --  anywhere             192.168.0.1          udp spt:ntp
The attack traffic is now dropped by the switch. In a hardware switch, this entry would be pushed by Cumulus Linux into the hardware, filtering the traffic at line rate to provide Terabit traffic filtering.

12 comments:

  1. Thank you - a slightly improved version of this code is now in the ExaBGP repository:
    https://github.com/Exa-Networks/exabgp/blob/master/lib/exabgp/application/flow.py

    ReplyDelete
    Replies
    1. Thanks! I have updated the article with a link.

      Delete
  2. This switch should be connected directly with the Router ?
    This switch should be in middle of Router <-> Server ?

    ReplyDelete
    Replies
    1. The best placement for the switch depends on your network topology. If you are a service provider and want to offer protection to your downstream customers, then the switches would be at the edge of the network connecting to the customer WAN links.

      If you are are trying to protect a data center or campus then you want to have the switch as close to the upstream link as possible since this is the traffic you wan to filter.

      For most setups you are probably better off using REST API for Cumulus Linux ACLs. The FlowSpec implementation shown in this article is useful for experimenting with FlowSpec, or to integrate with existing FlowSpec controllers.

      Delete
    2. Hello Peter,

      Actually our router don't support BGP Flow Spec. but we want to setup DDoS Scrubbing in our whole network. just wondering how to do the setup.

      Can you guide us in the Setup ?

      We using Brocade MLXe As core router, Brocade VDX as Edgerouter and Brocade ICX as Rack Switch. MLXe have Uplinks from Upstreams, and we have 2x VDX in redundant mode, both VDX getting uplinks from MLXe. and then forwarding traffics to Rack Switches.

      We filtering most attacks using ACL but need SYN Flood filtering system which can filter all the traffics before it reach the servers inside our network.

      Delete
    3. SYN floods typically don't generate the amount of traffic associated with UDP attacks. Layer 4+ attacks are better mitigated at the load balancer or on the servers. Have you tried using SYN cookies?

      Delete
    4. Hello Peter,

      Yes we do use SYNPROXY in Centos7, but its not supported in centos6, and using syn cookies not much improvement, only SYNPROXY works in much better way. though we already filter spoofed ips, so not much traffic pass our security acl's, but still need a ddos filtering system.

      Delete
  3. So now which switches or which chipsets that support a large amount of ACL entries? Ideally, hundreds of thousands.

    ReplyDelete
    Replies
    1. The Cumulus Linux Netfilter - ACLs documentation lists the number of ACLs supported for the supported merchant silicon platforms. The maximum is in the thousands, not hundreds of thousands.

      CloudFlare DDoS Mitigation Pipeline describes using BPF filters on hosts to implement large numbers of complex filters. You could use sFlow analytics to construct a hybrid approach, pushing filters for high volume attacks to hardware switches and implementing low volume filtering in software. sFlow-RT's BGP support could be used steer attack traffic through the scrubbing switches and/or servers by modifying the Remotely Triggered Black Hole (RTBH) Routing application to steer attack traffic rather than drop it.

      Delete
  4. Hi Peter,
    I have a question about how to send message after running ExaBGP. I saved the FlowSpec message file you attached at the end of the article to msg.json. And I opened another terminal running "cat msg.json > exabgp-4.0.10/run/exabgp.in", but I received the error :

    21:09:02 | 32509 | api | command from process not understood : {
    21:09:02 | 32509 | api | command from process not understood : "exabgp": "4.0.10" ,
    .....

    Could you help with how to send bgpflowspec message after running Exabgp? Thanks

    ReplyDelete
  5. Hi Peter,

    Could you explain how to send bgp flowspec message? I run Exabgp in one terminal and saved the json file you attached at the end to one file msg.json and run the command "cat msg.json > exabgp-4.0.10/run/exabgp.in" in the other terminal. But I received error:
    20:52:35 | 31542 | api | command from process not understood : {
    20:52:35 | 31542 | api | command from process not understood : "exabgp": "4.0.0" ,
    20:52:35 | 31542 | api | command from process not understood : "time": 1498868955.31 ,
    20:52:35 | 31542 | api | command from process not understood : "host": "tor-router" ,

    Could you explain more on how to send bgp flowspec message to Exabgp? Thanks

    ReplyDelete
    Replies
    1. The JSON message shown in the article was a log of internal message received by the acl.py script embedded in exabgp. In this case the BGP FlowSpec message was sent by sFlow-RT, see Real-time DDoS mitigation using sFlow and BGP FlowSpec.

      You should consult the exabgp documentation for configuration examples for sending flowspec messages.

      Delete