Sunday, April 6, 2014

DDoS mitigation hybrid OpenFlow controller

Performance optimizing hybrid OpenFlow controller describes the growing split in the SDN controller market between edge controllers using virtual switches to deliver network virtualization (e.g. VMware NSX, Nuage Networks, Juniper Contrail, etc.) and fabric controllers that optimize performance of the physical network. The article provides an example using InMon's sFlow-RT controller to detect and mark large "elephant" flows so that they don't interfere with latency sensitive small "mice" flows.

This article describes an additional example, using the sFlow-RT controller to implement the ONS 2014 SDN Idol winning distributed denial of service (DDoS) mitigation solution - Real-time SDN Analytics for DDoS mitigation.
Figure 1: ISP/IX Market Segment
Figure 1 shows how service providers are ideally positioned to mitigate large flood attacks directed at their customers. The mitigation solution involves an SDN controller that rapidly detects and filters out attack traffic and protects the customer's Internet access.
Figure 2: Novel DDoS Mitigation solution using Real-time SDN Analytics
Figure 2 shows the elements of the control system in the SDN Idol demonstration. The addition of an embedded OpenFlow controller in sFlow-RT allows the entire DDoS mitigation system to be collapsed into the following sFlow-RT JavaScript application:
// Define large flow as greater than 100Mbits/sec for 1 second or longer
var bytes_per_second = 100000000/8;
var duration_seconds = 1;

var idx = 0;

setFlow('udp_target',
 {keys:'ipdestination,udpsourceport',
  value:'bytes', filter:'direction=egress', t:duration_seconds}
);

setThreshold('attack',
 {metric:'udp_target', value:bytes_per_second, byFlow:true, timeout:2, 
  filter:{ifspeed:[1000000000]}}
);

setEventHandler(function(evt) {
 var agent = evt.agent;
 var ports = ofInterfaceToPort(agent);
 if(ports && ports.length == 1) {
  var dpid = ports[0].dpid;
  var id = "drop" + idx++;
  var k = evt.flowKey.split(',');
  var rule= {
   priority:500, idleTimeout:20, hardTimeout:3600,
   match:{dl_type:2048, nw_proto:17, nw_dst:k[0], tp_src:k[1]},
   actions:[]
  };
  setOfRule(dpid,id,rule);
 }
},['attack']);
The following command line arguments load the script and enable OpenFlow on startup:
-Dscript.file=ddos.js -Dopenflow.start=yes
Some notes on the script:
  1. The 100Mbits/s threshold for large flows was selected because it represents 10% of the bandwidth of the 1Gigabit access ports on the network
  2. The setFlow filter specifies egress flows since the goal is to filter flows as converge on customer facing egress ports.
  3. The setThreshold filter specifies that thresholds are only applied to 1Gigabit access ports
  4. The OpenFlow rule generated in setEventHandler matches the destination address and source port associated with the DDoS attack and includes an idleTimeout of 20 seconds and a hardTimeout of 3600 seconds. This means that OpenFlow rules are automatically removed by the switch when the flow becomes idle without any further intervention from the controller. If the attack is still in progress when the hardTimeout expires and the rule is removed, the attack will be immediately be detected by the controller and a new rule will be installed.
The nping tool can be used to simulate DDoS attacks to test the application. The following script simulates a series of DNS reflection attacks:
while true; do nping --udp --source-port 53 --data-length 1400 --rate 2000 --count 700000 --no-capture --quiet 10.100.10.151; sleep 40; done
The following screen capture shows a basic test setup and results:
The chart at the top right of the screen capture shows attack traffic mixed with normal traffic arriving at the edge switch. The switch sends a continuous stream of measurements to the sFlow-RT controller running the DDoS mitigation application. When an attack is detected, an OpenFlow rule is pushed to the switch to block the traffic. The chart at the bottom right trends traffic on the protected customer link, showing that normal traffic is left untouched, but attack traffic is immediately detected and removed from the link.
Note: While this demonstration only used a single switch, the solution easily scales to hundreds of switches and thousands of edge ports.
This example, along with the large flow marking example, demonstrates that basing the sFlow-RT fabric controller on widely supported sFlow and OpenFlow standards and including an open, standards based, programming environment (JavaScript / ECMAScript) makes sFlow-RT an ideal platform for rapidly developing and deploying traffic engineering SDN applications in existing networks.

16 comments:

  1. Great post! I am currently trying to recreate this experiment. I have sflow-rt up and running with the custom options (ddos.js and openflowcontroller start) as well as mininet (running the default 2 host topology with remote controller). I see feedback in the terminal that is running sflow and I know that the sflow-rt controller and mininet are talking to each other as I get a successful pingall.

    Overall, I think it's working but I can't tell for sure. I see that you have visualized your traffic with graphs and I was wondering if there was a way to do that same thing in sflow-rt web interface. (I tried to go to http://localhost:8008/metric/ALL/udp_target/html and unfortunately I see nothing) . I really appreciate the help and above all for the interesting article. Have a great week!

    ReplyDelete
    Replies
    1. The sFlow from Open vSwitch on Mininet is ingress sampled. Try removing the filter in the setFlow command. You could also put a logging statement after the setOfRule statement to let you know that the rule has been triggered, e.g. logInfo("blocking " + k[0]);

      Delete
  2. Hi Peter,

    Thanks for your post. I'm incredibly new to all of this and I realized that I wasn't setting any configuration for the mininet's openvswitch (i.e. "sudo ovs-vsctl -- --id=@sflow create sflow agent=eth0 target=\"127.0.0.1:6343\" sampling=10 polling=20 -- -- set bridge s1 sflow=@sflow" which was extracted from one of the other blog posts). I'll give this and your suggestions a shot when I get home.

    ReplyDelete
    Replies
    1. You might find the leafandspine script that now ships with sFlow-RT easier to use - see Mininet integrated hybrid OpenFlow testbed

      Delete
  3. Thanks for the replies Peter. I am making some progress, I am able to see the graph (see here: http://imgur.com/vba0xl5). I am also getting feedback from my log message which I inserted as you advised below. I also modified some of the code above to get it to work correctly for me (mainly bumped bytes_per_second down to 10 Mbps, removed egress filter on setFlow, and finally I removed the filter from the setThreshold function)

    -----------------------------

    // Define large flow as greater than 100Mbits/sec for 1 second or longer
    var bytes_per_second = 10000000/8;
    var duration_seconds = 1;

    var idx = 0;

    setFlow('udp_target',
    {keys:'ipdestination,udpsourceport',
    value:'bytes', t:duration_seconds}
    );

    setThreshold('attack',
    {metric:'udp_target', value:bytes_per_second, byFlow:true, timeout:2}
    );

    setEventHandler(function(evt) {
    var agent = evt.agent;
    var ports = ofInterfaceToPort(agent);
    if(ports && ports.length == 1) {
    var dpid = ports[0].dpid;
    var id = "drop" + idx++;
    var k = evt.flowKey.split(',');
    var rule= {
    priority:5, idleTimeout:20, hardTimeout:3600,
    match:{dl_type:2048, nw_proto:17, nw_dst:k[0], tp_src:k[1]},
    actions:[]
    };
    setOfRule(dpid,id,rule);
    logInfo("blocking " + k[0]);
    }
    },['attack']);


    ----------------------------------


    I am also getting output when I run ovs-ofctl dump-flows like so:

    sudo ovs-ofctl dump-flows s1
    NXST_FLOW reply (xid=0x4):
    cookie=0x2, duration=6.801s, table=0, n_packets=0, n_bytes=0, idle_timeout=20, hard_timeout=3600, idle_age=6, priority=5,udp,nw_dst=10.0.0.2,tp_src=53 actions=drop
    cookie=0x0, duration=323.261s, table=0, n_packets=7297726, n_bytes=10523142088, idle_age=0, priority=10 actions=NORMAL

    So it appears that ddos mitigation flow is getting pushed to the switch but I do not see any drop in the graph. It just continues to show the attack traffic as normal. Any thoughts on what I am missing? Thanks again Peter for the incredibly quick replies!

    ReplyDelete
    Replies
    1. Open vSwitch samples packets before filtering, so you see the discarded packets. If you add the following filter to your setFlow function, filter:'outputifindex!=discard', then you should be able to see the flow being dropped.

      Delete
    2. Great everything works now as expected. Thanks a lot!

      Delete
  4. Hi,
    I am too trying the same experiment but when I run the nping command to produce DNS reflection attack that is mentioned above in the article, it throws me an error saying
    libnsock mksock_bind_addr(): Bind to 0.0.0.0:53 failed (IOD #34517): Permission denied (13)

    I'm not sure why this error pops up. Am I missing something..? I would appreciate if I can get a help regarding this.
    Thanks

    ReplyDelete
  5. Yes I'm having the same issue as Yujjit Abijaiy, any solutions? Thanks

    ReplyDelete
    Replies
    1. You need to be running as root to open port 53.

      Delete
    2. Thank you for your prompt reply Peter. I've managed to follow the steps provided by you and TheDoc McFly, but apparently there's no traffic being dropped. I figured that it has something to do with the controller. I'm using 'sudo mn' and when I start sFlow './start.sh':

      2015-05-16T08:32:23-0400 SEVERE: OF: could not start controller: Address already in use

      I really appreciate any help you can provide. Thanks!

      Delete
    3. The message indicates that there is another OpenFlow controller running. You need to disable the controller so that sFlow-RT can open the OpenFlow port and receive OpenFlow connections.

      Delete
    4. May I know how to disable the controller? I'm using 'sudo mn --controller=remote' but when I start sFlow the following appeared:

      2015-05-16T14:54:31-0400 INFO: OF: connected to 127.0.0.1:42110 using OF 1.3
      2015-05-16T14:54:31-0400 WARNING: OF1.3: error from 127.0.0.1:42110: type = OFPET_BAD_REQUEST, code = OFPBRC_BAD_STATS, cause = OFStatisticsMessage [type=TABLE_FEATURES, flags=0, data=[]]

      And is this correct (start.sh)?:

      RT_OPTS="-Dsflow.port=6343 -Dhttp.port=8008"
      SCRIPTS="-Dscript.file=ddos.js -Dopenflow.controller.start=yes"

      I look forward to your reply. Thank You

      Delete
    5. You can ignore the warning. It looks like you have successfully connected Mininet to the sFlow-RT controller.

      Delete
    6. Sorry Peter but I forgot to mention that pingall is unsuccessful. Any ideas?

      Delete
    7. This article used a physical switch that supports hybrid OpenFlow. When you use Mininet, you need to simulate hybrid OpenFlow by installing rules that provide a forwarding path for packets. You can use the leafandspine script that ships with sFlow-RT to set up the default OpenFlow rules:
      Hybrid OpenFlow ECMP testbed

      <a href="

      Delete