Thursday, May 19, 2016

Mininet flow analytics

Mininet is free software that creates a realistic virtual network, running real kernel, switch and application code, on a single machine (VM, cloud or native), in seconds. Mininet is useful for development, teaching, and research. Mininet is also a great way to develop, share, and experiment with OpenFlow and Software-Defined Networking systems.

This article shows how standard sFlow instrumentation built into Mininet can be combined with sFlow-RT analytics software to provide real-time traffic visibility for Mininet networks. Augmenting Mininet with sFlow telemetry realistically emulates the instrumentation built into most vendor's switch hardware, provides visibility into Mininet experiments, and opens up new areas of research (e.g. SDN and large flows).

The following papers are a small selection of projects using sFlow-RT:
In order to make it easier to get started, the latest release of sFlow-RT includes a Mininet helper script sflow.py that automates sFlow configuration. The following example shows how to use the script and build a simple application in Python.

Install sFlow-RT on the Mininet host:
wget https://inmon.com/products/sFlow-RT/sflow-rt.tar.gz
tar -xvzf sflow-rt.tar.gz
cd sflow-rt
./start.sh
In a second terminal, add the --custom argument to the Mininet command line. For example, the following command builds a depth 2 tree topology with link bandwidths of 10Mbit/s
cd sflow-rt
sudo mn --custom extras/sflow.py --link tc,bw=10 --topo tree,depth=2,fanout=2
The sflow.py script extends Mininet, automatically enabling sFlow on each of the switches in the topology, and posting a JSON representation of the Mininet topology using sFlow-RT's REST API.

Traffic engineering of large "Elephant" flows is an active area of research. The following Python script, elephant.py, demonstrates how Elephant flows can be detected using sFlow-RT REST API calls:
#!/usr/bin/env python
import requests
import json

rt = 'http://127.0.0.1:8008'

flow = {'keys':'link:inputifindex,ipsource,ipdestination','value':'bytes'}
requests.put(rt+'/flow/pair/json',data=json.dumps(flow))

threshold = {'metric':'pair','value':1000000/8,'byFlow':True,'timeout':1}
requests.put(rt+'/threshold/elephant/json',data=json.dumps(threshold))

eventurl = rt+'/events/json?thresholdID=elephant&maxEvents=10&timeout=60'
eventID = -1
while 1 == 1:
  r = requests.get(eventurl + "&eventID=" + str(eventID))
  if r.status_code != 200: break
  events = r.json()
  if len(events) == 0: continue

  eventID = events[0]["eventID"]
  events.reverse()
  for e in events:
    print e['flowKey']
Some notes on the script:
  • The link:inputifindex function in the flow definition identifies the link in the topology associated with the ingress port on the Mininet switch, see Defining Flows
  • The script defines an Elephant flow as a flow that consumes 10% of the link bandwidth. In this example Mininet was configured with a link bandwidth of 10Mbit/s so an Elephant is a flow that exceeds 1Mbit/s. Since the specified flow measures traffic in bytes/second the threshold needs to be converted by bytes/second (dividing by 8).
  • The sFlow-RT REST API uses long-polling as a method of asynchronously pushing events to the client. The events HTTP request blocks until there are new events or a timeout occurs. The client immediately reconnects after receiving a response to wait for further events. 
See Writing Applications for additional information.

Start the script:
$ ./elephant.py
Run an iperf test using the Mininet CLI:
mininet> iperf h1 h3
*** Iperf: testing TCP bandwidth between h1 and h3
*** Results: ['9.06 Mbits/sec', '9.98 Mbits/sec']
The following results should appear as soon as the flow is detected:
$ ./elephant.py 
s1-s2,10.0.0.1,10.0.0.3
s1-s3,10.0.0.1,10.0.0.3
The output identifies the links carrying the flow between h1 and h3 and shows the IP addresses of the hosts.
The sFlow-RT web interface provides basic charting capabilities. The chart above shows a sequence of iperf tests.

The Python script can easily be modified to address a number of interesting use cases. Instead of simply printing events, a REST call can be made to an OpenFlow controller (POX, OpenDaylight, Floodlight, ONOS, etc) to apply controls to mark, mirror, load balance, rate limit, or block flows.

57 comments:

  1. Please Help me, i face this error " Caught exception.Cleaning up.... SyntaxError: invalid syntax (sflow.py, line 25) , when i run "sudo mn --custom extras/sflow.py --link tc,bw=10 --topo tree,depth=2,fanout=2" .
    Regreds..

    ReplyDelete
    Replies
    1. I just tried it with sFlow-RT version 2.0-r1121 on an Ubuntu 14.04 system running Mininet 2.1.0 and Python 2.7.6 without any error. What is your setup?

      Delete
  2. How does I make REST calls to POX for adding a flow??

    ReplyDelete
    Replies
    1. POX is written in Python, so you should be able to modify the example to run as a POX application and access native POX APIs.

      The following project looks like it could be extended to add/remove flows via REST API:
      https://github.com/festradasolano/pox-jsonrest

      Delete
  3. Hi Peter,
    Where i can find the sflow.py?

    Thank you,

    ReplyDelete
  4. I don't see it ...

    floodlight@floodlight:~/sflow-rt/extras$ ls
    README init.d leafandspine.py tail_log.py
    demo.pcap leafandspine-hybrid.js tail_flows.py topflows.py
    floodlight@floodlight:~/sflow-rt/extras$ ls -l
    total 4772

    ReplyDelete
    Replies
    1. Which version of sFlow-RT are you using? The script is in http://www.inmon.com/products/sFlow-RT/sflow-rt.tar.gz, not the DEB or RPM distributions.

      Delete
  5. how to use sflow with pox controller for load balancing?

    ReplyDelete
  6. Hi Peter, how do I get the switch dpid from the inputifindex ?

    ReplyDelete
    Replies
    1. There are a number of methods. For example, the following query gets the dpid of ifindex 2 on agent 10.0.0.202:
      /metric/10.0.0.202/2.of_dpid/json

      The following query finds the OpenFlow port:
      /metric/10.0.0.202/2.of_port/json

      If you are writing an embedded JavaScript application, then the following function returns the dpid and openflow port:
      ofInterfaceToPort(agent,ifIndex)

      Delete
  7. Hi Peter
    How can i check the flow table entries of Mininet ovs using sflow-rt?

    ReplyDelete
    Replies
    1. sFlow-RT cannot report the flow table entries. You can use the "ovs-ofctl dump-flows" command to get the active flows.

      Delete
  8. Hi Peter, How to get all metrics of one switch ?

    ReplyDelete
    Replies
    1. Normally you would use the following REST API query:
      http://localhost:8008/metric/10.0.0.20/json
      or the JavaScript equivalent, metrics('10.0.0.20')

      However, with Mininet all the logical switches are represented by a single instance of Open vSwitch and so you need to query the interfaces associated with the selected Mininet switch. The sflow.py script posts the Mininet topology to sFlow-RT. You can query the topology using the /topology/json REST API call, or the getTopology() JavaScript equivalent.

      The nodes section in the topology lists the Mininet switches and their ports. Each port has an ifIndex and you can use the ifIndex numbers to identify switch specific metrics, e.g. 1.ifinoctets is a counter associated with ifIndex=1.

      Delete
  9. Hi Peter,
    I was working on routing the flows after detection. Can you suggest me any tutorial? I am struggling finding one. I wrote an algorithm and need to implement it now. Thanks in advance.

    ReplyDelete
    Replies
    1. What are you using as your controller? How you re-route depends on the available APIs.

      Delete
    2. I am planning to use POX or RYU controller. So do you mean there are API's which are already built for re-route?

      Delete
  10. I am planning to use RYU or POX controller. So do you mean there are already API's for the controllers which can help me to re-route the flows?

    ReplyDelete
    Replies
    1. I don't believe RYU or POX has a REST API for programming flows. ONOS measurement based control describes how to use the ONOS controller.

      You could write your own Python extension to RYU or POX and access sFlow-RT's REST API to retrieve flow metrics, see Writing Applications for information on sFlow-RT REST APIs.

      Delete
  11. Hi Peter,
    When I change the fanout value to more than 2 then elephant.py does not detect or print any output. Is it because the script is unable to read the topology?

    sudo mn --custom extras/sflow.py --link tc,bw=10 --topo tree,depth=2,fanout=2

    ReplyDelete
    Replies
    1. I just tested the script and it appears to work. Are you using the latest version of sFlow-RT? You might want to load the Mininet Dashboard application. It can now display topology as well, see Mininet weathermap.

      If you run your iperf test between h1 and h2 then the traffic doesn't cross any inter-switch links and so the elephant.py script won't generate any events. Try to do the iperf test between h1 and h3

      Delete
  12. Hi peter ..
    How to see the ryu in sflow

    ReplyDelete
  13. how to connect two controller in custom topology of mininet

    ReplyDelete
  14. how we can use use dataset in mininet and miniedit environment kindly guide me as possible??

    ReplyDelete
  15. I have started sflow-rt,
    when running the command
    sudo mn --custom extras/sflow.py --link tc,bw=10 --topo tree,depth=2,fanout=2

    Getting Error

    Caught exception. Cleaning up...

    ImportError: No module named requests

    ReplyDelete
  16. Hi Peter
    I cannot make it working following your manual. I got following error:

    TypeError: request() got an unexpected keyword argument 'json'

    I have tried requests module v. 2.2.1 and 2.22.1 with no luck.
    My system: 14.04.1-Ubuntu, Mininet 2.2.1, sFlow-RT (believe it's the lates as I can't determine the version)

    Please help

    ReplyDelete
    Replies
    1. Hi Peter, I also met same issue, could you give us a hand.

      *** Enabling sFlow:
      s1 s2 s3
      *** Sending topology
      --------------------------------------------------------------------------------
      Caught exception. Cleaning up...

      TypeError: request() got an unexpected keyword argument 'json'

      Delete
    2. You need to update the Python requests library to version 2.4.2 or later.

      pip install requests --upgrade

      Delete
    3. Hi Peter,
      Python 2.7.6, 14.04.1-Ubuntu, Mininet 2.2.1.. Still facing same issue. What else I'm missing?

      Delete
    4. What version of the requests module are you using? It needs to be 2.4.2 or later.

      Delete
    5. Hi Peter,
      pip --version
      pip 1.5.4 from /usr/lib/python2.7/dist-packages (python 2.7)

      How to update python requests library?

      Delete
    6. pip install requests --upgrade

      Delete
    7. Cannot fetch index base URL https://pypi.python.org/simple/
      Could not find any downloads that satisfy the requirement requests in /usr/lib/python2.7/dist-packages
      Downloading/unpacking requests
      Cleaning up...
      No distributions at all found for requests in /usr/lib/python2.7/dist-packages
      Storing debug log for failure in /root/.pip/pip.log

      Delete
    8. Hi Peter,

      pip install --index-url=https://pypi.org/simple/ requests --upgrade
      Downloading/unpacking requests from https://files.pythonhosted.org/packages/45/1e/0c169c6a5381e241ba7404532c16a21d86ab872c9bed8bdcd4c423954103/requests-2.24.0-py2.py3-none-any.whl#sha256=fe75cc94a9443b9246fc7049224f75604b113c36acb93f87b80ed42c44cbb898
      Cleaning up...
      Exception:
      Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
      status = self.run(options, args)
      File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 278, in run
      requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
      File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1198, in prepare_files
      do_download,
      File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1376, in unpack_url
      self.session,
      File "/usr/lib/python2.7/dist-packages/pip/download.py", line 546, in unpack_http_url
      resp = session.get(target_url, stream=True)
      File "/usr/share/python-wheels/requests-2.2.1-py2.py3-none-any.whl/requests/sessions.py", line 479, in get
      return self.request('GET', url, **kwargs)
      File "/usr/lib/python2.7/dist-packages/pip/download.py", line 237, in request
      return super(PipSession, self).request(method, url, *args, **kwargs)
      File "/usr/share/python-wheels/requests-2.2.1-py2.py3-none-any.whl/requests/sessions.py", line 467, in request
      resp = self.send(prep, **send_kwargs)
      File "/usr/share/python-wheels/requests-2.2.1-py2.py3-none-any.whl/requests/sessions.py", line 570, in send
      r = adapter.send(request, **kwargs)
      File "/usr/share/python-wheels/requests-2.2.1-py2.py3-none-any.whl/requests/adapters.py", line 385, in send
      raise SSLError(e)
      SSLError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

      Storing debug log for failure in /root/.pip/pip.log

      Delete
    9. Try the following to upgrade the SSL certicates:
      pip2.7 install --upgrade httplib2

      Delete
  17. hello where i can put the elephant.py script? , do i need to run the simple_switch13.py and ofctl_rest.py the ryu applications with the elephant.py?

    ReplyDelete
    Replies
    1. Just run the elephant.py script anywhere on the host running sFlow-RT. You could run it remotely by editing rt variable at the top of the script. This article uses the default Open vSwitch controller so you don't need the other scripts you mentioned.

      Delete
  18. @ peter thanks for replying i did this steps i put the elephant.py inside flow-rt/app/mininet-dashboard
    first :- i wrote in the first terminal cd flow-rt
    ./start.sh
    second:- i wrote in the second terminal the topology (sudo mn --custom extras/sflow.py --link tc,bw=10 --topo tree,depth=2,fanout=2)
    third:- in the third terminal i wrote cd flow-rt/app/mininet-dashboard
    then ./elephant.py
    are these steps above right or wrong ? did i miss steps

    ReplyDelete
    Replies
    1. Normally, the sflow-rt/app/ directory is used for embedded applications, see Writing Applications. Starting sflow-rt is not going to automatically start elephant.py.

      The steps you have, seem fine. In the following order, and in separate terminals:
      1. start sflow-rt
      2. start mininet
      3. start elephant.py

      Delete
    2. @peter i follow the steps but the elephant.py scripts i put it in the sflow-rt/app/mininet-dashboard not in the sflow-rt/app is that right?

      Delete
    3. I would recommend putting the elephant.py script in your home directory, i.e. ~/elephant.py, but you can put it wherever you want, it is not an integrated sFlow-RT applications so it doesn't need to be in the sflow-rt/app directory.

      Delete
  19. @ can i use the ryu controller application instead of default ovs switch controller with elephant.py script?

    ReplyDelete
    Replies
    1. Yes, the elephant.py script should work if you use the Ryu controller. Ryu measurement based control provides an example using Ryu, Mininet, and sFlow-RT.

      Delete
  20. @peter i follow the steps but when i go to localhost:8008/metric i got http error 404,

    ReplyDelete
    Replies
    1. The /metric/ALL/pair/html page was removed in sFlow-RT 3.0. You can use the browse-metrics app instead, see sFlow-RT 3.0 released.

      Delete
  21. Does this tools will support to test memory usage of ryu controller. If anyone has knowledge, whether any tools is available to find the memory consumption of openflow switches which store the flow rule.

    ReplyDelete
    Replies
    1. The sFlow from Open vSwitch includes key performance statistics, Open vSwitch performance monitoring. I don't know what instrumentation the ryu controller provides.

      Delete
  22. How to edit ./elephant.py, so that threshold is set 40% or 50% instead of 10%. Currently i have edit the script as follows:
    threshold = {'metric':'pair','value':1000000/8,'byFlow':True,'timeout':5}
    However, changing the value from 1 to 5 is not working for me.

    PS. Thanks for such great forum. I started from scratch and was able to successfully plot traffic on Mininet dashboard along with detecting flows that exceed 1 mb.

    ReplyDelete
    Replies
    1. The value property is the threshold. In this case the value is in bytes per second, so 1000000/8 is 1Mbit/s converted to bytes/second. The Mininet link speed is set to 10Mbit/s, so this is the 10% threshold. A 50% threshold would be 5000000/8.

      The timeout value is the hysteresis for the threshold (in seconds). This means that the value needs to drop below the threshold for 5 seconds before it can re-trigger.

      Delete
    2. Thanks that worked for me, also can you guide me how can i perform actions like dropping data in case of threshold s crossed instead of only print the flow key

      Delete
  23. Hi peter
    how to generate a continuous requests to the server in mininet and Ryu controller in order to measure server response time please??

    ReplyDelete
  24. Hi...I am doing a simple academic project on sdn wherein i have built a simple tree topology using mininet. I am performing DDOS ICMP flood attack and want to measure cpu and memory stats for individual hosts. How can it be done ?

    ReplyDelete
    Replies
    1. I am sorry I don't have an answer. You might try the Mininet mailing list.

      Delete