Raspberry Pi real-time network analytics

The Raspberry Pi model 3b is not much bigger than a credit card, costs $35, runs Linux, has a 1G RAM, and powerful 4 core 64 bit ARM processor. This article will demonstrate how to turn the Raspberry Pi into a Terribit/second real-time network analytics engine capable of monitoring hundreds of switches and thousands of switch ports.

The diagram shows how the sFlow-RT real-time analytics engine receives a continuous telemetry stream from industry standard sFlow instrumentation build into network, server and application infrastructure and delivers analytics through APIs and can easily be integrated with a wide variety of on-site and cloud, orchestration, DevOps and Software Defined Networking (SDN) tools.

A future article will examine how the Host sFlow agent can be used to efficiently stream measurements from large numbers of inexpensive Rasberry Pi devices ($5 for model Zero) to the sFlow-RT collector to monitor and control the "Internet of Things" (IoT).

The following instructions show how to install sFlow-RT on Raspbian Jesse (the Debian Linux based Raspberry Pi operating system).

wget http://www.inmon.com/products/sFlow-RT/sflow-rt_2.0-1092.deb
sudo dpkg -i --ignore-depends=openjdk-7-jre-headless sflow-rt_2.0-1092.deb

We are ignoring the dependency on openjdk and will use the default Raspbian Java 1.8 version instead.

Next, edit /usr/local/sflow-rt/conf.d/sflow-rt.jvm and replace the default settings with the following:

-Xms600M
-Xmx600M
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode

These new settings reduce the requested memory to fit within the 1G on the Raspberry Pi and leave some memory for system tasks. The G1GC garbage collector not available on ARM Java implementation, so we will use incremental concurrent mark and sweep instead.

Start the sFlow-RT daemon:

sudo service sflow-rt start

The sFlow-RT web interface should now be accessible at http://<raspberrypi_ip>:8008/

Finally, Agents provides information on configuring devices to send sFlow to the Raspberry Pi analyzer. Visit the http://<raspberrypi_ip>:8008/agents/html page to verify that data is being received.

Writing Applications provides an overview of the sFlow-RT APIs. For example, run the following Python script on the Raspberry Pi to log traffic flows that exceed 100Mbits/second:

#!/usr/bin/env python
import requests
import json

rt = 'http://127.0.0.1:8008'

flow = {'keys':'ipsource,ipdestination','value':'bytes'}
requests.put(rt+'/flow/pair/json',data=json.dumps(flow))

threshold = {'metric':'pair','value':100000000/8,'byFlow':True,'timeout':1}
requests.put(rt+'/threshold/elephant/json',data=json.dumps(threshold))

eventurl = rt+'/events/json?thresholdID=elephant&maxEvents=10&timeout=60'
eventID = -1
while 1 == 1:
  r = requests.get(eventurl + "&eventID=" + str(eventID))
  if r.status_code != 200: break
  events = r.json()
  if len(events) == 0: continue

  eventID = events[0]["eventID"]
  events.reverse()
  for e in events:
    print e['flowKey']

In addition, there are a number of open source sFlow-RT Applications available on the Downloads page (e.g. Mininet dashboard, DDoS mitigation, etc.) and articles describing use cases for sFlow-RT on this blog.