Sunday, January 6, 2013

Performance aware software defined networking

Figure 1: Performance aware SDN applications with sFlow-RT
The article, Software defined networking, described reasons for including the sFlow standard as the visibility protocol in an SDN stack. InMon's sFlow-RT module delivers real-time network, host and application visibility to SDN applications, making possible the development of new classes of performance aware application, such as load balancing and denial of service protection.

Figure 1 shows how sFlow-RT fits into an SDN stack. The sFlow-RT module receives a continuous stream of sFlow datagrams from network devices and converts them into actionable metrics, accessible through a REST (Representational State Transfer) API. REST Applications make use of the API to detect changing network traffic and adapt the network by pushing controls to an OpenFlow controller (for example, the open source Floodlight OpenFlow Controller). The controller uses the OpenFlow protocol to communicate with the network devices and reconfigure their forwarding behavior.

Note: While this example focuses on network visibility, sFlow integrates network, server and application monitoring and sFlow-RT delivers real-time visibility into these metrics, allowing the SDN controller to be server and application aware.
Figure 2: Denial of service protection example
Figure 2 shows a simplified REST Application for protecting against denial of service attacks. The numbered arrows indicate REST commands used by the application to monitor the network and deploy controls.

1. define address groups
Address groups (defined using CIDR notation) are a useful way of categorizing traffic. In this case, allowing internal and external addresses to be identified:
curl -H "Content-Type:application/json" -X PUT --data "{external:['0.0.0.0/0'], internal:['10.0.0.0/8']}" http://localhost:8008/group/json
2. define flows
Flows are defined by naming the packet attributes used to group packets into flows (keys), a value to associate with the flow, and a filter to select specific traffic. In this case we are interested in defining incoming flows, i.e. from external source addresses to internal destination addresses:
curl -H "Content-Type:application/json" -X PUT --data "{keys:'ipsource,ipdestination', value:'frames', filter:'sourcegroup=external&destinationgroup=internal'}" http://localhost:8008/flow/incoming/json
3. define thresholds
The following command defines a threshold of 1000 packets per second on any incoming flow:
curl -H "Content-Type:application/json" -X PUT --data "{metric:'incoming', value:1000}" http://localhost:8008/threshold/incoming/json
4. receive threshold event
The application polls for events, using "long polling" to receive asynchronous notification of new events. The following command asks for any events after eventID=4, the most recent event received, waiting up to 60 seconds for a result:
curl "http://localhost:8008/events/json?eventID=4&timeout=60"
The following event shows that an incoming flow generating 1,531 packets per second has exceeded the threshold:
[{
 "agent": "10.0.0.16",
 "dataSource": "4",
 "eventID": 5,
 "metric": "incoming",
 "threshold": 1000,
 "thresholdID": "incoming",
 "timestamp": 1357169369479,
 "value": 1531.149418835524
}]
5. monitor flow
Additional information about the flow is retrieved using the information from the event, including: agent, datasource and metric:
curl http://localhost:8008/metric/10.0.0.16/4.incoming/json
The following result shows that the value is still increasing and identifies the specific flow that exceeded the threshold, a flow from external address 192.168.1.1 to internal address 10.0.0.151:
[{
 "agent": "10.0.0.16",
 "dataSource": "4",
 "metricName": "incoming",
 "metricValue": 1582.93965044338071,
 "topKeys": [
  {
   "key": "192.168.1.1,10.0.0.151",
   "updateTime": 1357169662500,
   "value": 1582.93965044338071
  },
  {
   "key": "192.168.1.4,10.0.0.151",
   "updateTime": 1357169665500,
   "value": 46.552918457198984
  }
 ],
 "updateTime": 1357169665500
}]
6. deploy control
The OpenFlow controller is instructed to drop traffic from the external attacker (192.168.1.1)

7. monitor flow
Continue to monitor the flow to verify that the control has taken effect.

8. release control
At some later time the control is removed in order to release flow table entries tied up in blocking the attack. If the attacker returns, a new event will be generated, triggering a new control.

The following Python script, ddos.py, combines the steps to demonstrate a simple application:
import requests
import json

groups = {'external':['0.0.0.0/0'],'internal':['10.0.0.0/8']}
flows = {'keys':'ipsource,ipdestination','value':'frames','filter':'sourcegroup=
external&destinationgroup=internal'}
threshold = {'metric':'incoming','value':1000}

target = 'http://localhost:8008'

r = requests.put(target + '/group/json',data=json.dumps(groups))
r = requests.put(target + '/flow/incoming/json',data=json.dumps(flow
s))
r = requests.put(target + '/threshold/incoming/json',data=json.dumps
(threshold))

eventurl = target + '/events/json?maxEvents=10&timeout=60'
eventID = -1
while 1 == 1:
  r = requests.get(eventurl + '&eventID=' + str(eventID))
  if r.status_code != 200: break
  events = r.json()
  if len(events) == 0: continue

  eventID = events[0]["eventID"]
  for e in events:
    if 'incoming' == e['metric']:
      r = requests.get(target + '/metric/' + e['agent'] + '/' + e['dataSource'] + '.' + e['metric'] + '/json')
      metric = r.json()
      if len(metric) > 0:
        print metric[0]["topKeys"][0]["key"]
Running the script generates the following output within seconds of a large flow starting:
$ python ddos.py 
192.168.1.1,10.0.0.151
sFlow-RT is free for non-commercial use. Please try out the software. Comments, questions and general discussion of performance aware SDN are welcome on the sFlow-RT group.

37 comments:

  1. I have a problem in using sFlow together with FloodLight. My goal is to find the traffic on each port of OVS Network. So I used sFlow to get the statistics, but sFlow gives me the ifindex(as SNMP) of each port, in FloodLight I only have the port number(May be 1\2\3), how can I find the relationship between the ifindex and the port number. Should I use another SNMP daemon? Thanks very much!

    ReplyDelete
    Replies
    1. Finding the mapping between OpenFlow switch port numbers and SNMP ifIndex numbers is difficult, but I think it is an important problem to solve, particularly when looking at hybrid environments in which some traffic is switched conventionally and other traffic is controlled using OpenFlow. It's important to be able to combine performance data collected using sFlow and SNMP with OpenFlow configuration and performance data.

      I don't think there is currently a simple answer, either OpenFlow needs to provide a way of retrieving ifIndex numbers associated with ports, or the OpenFlow bridge/port ID information needs to be exported as part of the sFlow and SNMP (define OpenFlow MIB?) standards. There were some discussions in the past, but the OpenFlow standard has evolved considerably since then (see OpenFlow mailing list, or the sFlow mailing).

      Delete
    2. Can you please tell me where exactly can I find the options for filter and keys? I've looked through the specifications but it's not that obvious for me where to find them. I am trying to get statistics per switch, like the number of flows passing through the switch, number of packets, etc. If you know an example that can help me with this please let me know. Thanks!

      Delete
    3. The /flowkeys/html or /flowkeys/json pages list the tokens that are being decoded from the sFlow feed and these tokens can be used to create the filter: and/or keys: arguments in a flow definition. There are a number of examples on this blog, just click on the sFlow-RT label.

      In addition to exporting packet samples, the sFlow protocol pushes the standard set of interface counters that you would get by SNMP polling. The counters that are being received will be listed under the /metrics/html or /metrics/json pages. These can be retrieved using the metric or dump REST calls.

      Documentation for the API is included with the software, just visit the /html/api.html and /html/script.html pages.

      Delete
  2. Hi,

    the python script get a 415 error when trying to push data to group/json.

    Other put seems ok, any idea of what's happening ? (ip range validation failing ?)

    Thanks for your article :)

    ReplyDelete
    Replies
    1. It sounds like the JSON object is badly formed. Did you cut and paste the script from this article? The flows assignment should be a single line.

      The extras/ddos.py script that installs with sFlow-RT is the original script from this article. It should run without error and is a better starting point.

      Delete
  3. I've cut&paste, it dont work, and the extras/ddos.py get the same 415 error :(

    I event tryed to pass an empty dict and I get the same error ...

    ReplyDelete
    Replies
    1. Can you provide a short example of the code that is failing?

      Delete
    2. Here it is : http://dpaste.org/RtTT8/

      Delete
    3. I haven't been able to reproduce the problem:
      $ python simple.py
      200

      What version of sFlow-RT are you running? Does it work if you use the curl command? You can turn up logging by editing the resources/config/logging.properties file (set the level to FINE). Let me know if any exceptions stand out.

      Delete
    4. Ok ...

      So I should have try to do the request with curl, because it works.

      I've digged a little, and findout that the python-requests packaged with debian wheezy is *very* old. If I install a more recent one, everything is ok.

      Sorry about this, and thanks for your support !

      Delete
  4. Another thought - perhaps the content-type needs to be specified in the put command?

    headers = {'content-type':'application/json'}
    r = requests.put(target + '/group/json',data=json.dumps(groups),headers=headers)

    ReplyDelete
  5. Hi,

    I have managed to reproduce the results, however I would like to know how the Floodlight controller is used here,can you please elaborate more on how the controller can be used, also if possible I would like to get packet headers, so how can I do that ?

    Thank you in advance.

    ReplyDelete
    Replies
    1. Controlling large flows with OpenFlow contains a more complete example using Floodlight's static flow pusher. If you are interested in DDoS mitigation, then you might also want to also look at null routing large attacks. The DDoS describes a system that uses BGP to null route / redirect attacks.

      Delete
    2. Thanks for your reply.

      I actually had a look at those two articles, but I am more interested in looking at packet headers or samples (using sFlow terminology), and then use floodlight or any controller with a northbound APIs to insert flow rules in the network, these articles only handle counters. I don't have much background programming Nodejs so I am planning to do that in python. So, do you have any recommendations.



      Delete
    3. Actually, most of the articles are focused on analyzing packet samples and identifying packet header attributes. You specify the packet attributes you are interested in monitoring and then sFlow-RT generates additional counters and flow records. The /flowkeys/json page will show you the packet header fields that sFlow-RT is able to decode. You can combine header attributes to define flows (the Python example above defines ipsource,ipdestination flows). If you try the example above and access the URL /dump/ALL/incoming/json, you will see the top flows reported by each switch port.

      The Bay Areas Network Virtualization talk describes the REST API in more detail (the slides are linked in the article). The talk presented a number of cURL examples showing how to define and monitor flows.

      Delete
    4. running elephant.py shows the result , when there are large flows in the network, but running ddos.py does not perform any action, Can you tell Peter ? Thanks

      Delete
  6. I copy and paste the python script and use pip install requests to install requests module, but when I type python ddos.py it was not work and display message like
    File "ddos.py", line 16, in
    r = requests.get(eventurl + '&eventID=' + str(eventID))
    File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 44, in request

    ...etc
    what should I do to resolve this problem?

    ReplyDelete
    Replies
    1. Have you tried the version of the script in the /extras directory of the sFlow-RT distribution?

      Delete
    2. sir i am also getting the same problem.. what should i do

      Delete
  7. Hi, I'd like to know if I can use sFlow-rt with ONOS controller to detect large flow

    ReplyDelete
  8. Hi, can I use sFlow-rt with ONOS controller to detect large flow? plz

    ReplyDelete
    Replies
    1. The Fabric View application demonstrates tracking of large flows in an ECMP fabric. A modified version of this application was used with ONOS in the CORD: Open-source spine-leaf Fabric demonstration.

      Mininet integrated hybrid OpenFlow testbed demonstrates large flow detection using sFlow-RT with Mininet.

      Delete
  9. i am running the script but nothing is happenning.. the traffic is not settling down..

    ReplyDelete
    Replies
    1. The REST API for specifying groups and applying groups to addresses has changed. You need to name the group in the PUT command and then sourcegroup in the flow definition is replaced by the function group:ipsource:groupname

      Delete
    2. have you found the solution Rajeev ?

      Delete
  10. Hello,
    Running following script, python ddos.py, does not generate any results,, even if there are large flows in the network? why is that ?

    ReplyDelete
  11. running elephant.py shows the result , when there are large flows in the network, but running ddos.py does not perform any action, Can you tell Peter ? Thanks

    ReplyDelete
    Replies
    1. The ddos.py script need to be modified. Please read the discussion above about the REST API changes regarding specifying and applying address groups.

      Delete
  12. Thanks Peter for your reply,
    I did the changes in flows and request.put as you mentioned but still it didn't mitigate the traffic ,
    Can you help me out kindy ,
    I changed the code as :
    flows = {'keys':'ipsource,ipdestination','value':'frames','filter':'group:ipsource:incoming=external&group:ipdestination:incoming=internal'}
    and
    r = requests.put(target + '/group/incoming/json',data=json.dumps(groups))

    but it didnt work still..

    ReplyDelete
    Replies
    1. Have you found a solution to this?if yes can you please share

      Delete
  13. There are a number of more recent examples on this blog, see DoS

    ReplyDelete
  14. I want to do try this using floodlight Vm can someone guide me through the steps??

    ReplyDelete