InfluxDB and Grafana

Cluster performance metrics describes how to use sFlow-RT to calculate metrics and post them to Graphite. This article will describe how to use sFlow with the InfluxDB time series database and Grafana dashboard builder.

The diagram shows the measurement pipeline. Standard sFlow measurements from hosts, hypervisors, virtual machines, containers, load balancers, web servers and network switches stream to the sFlow-RT real-time analytics engine. Over 40 vendors implement the sFlow standard and compatible products are listed on sFlow.org. The open source Host sFlow agent exports standard sFlow metrics from hosts. For additional background, the Velocity conference talk provides an introduction to sFlow and case study from a large social networking site.

It is possible to simply convert the raw sFlow metrics into InfluxDB metrics. The sflow2graphite.pl script provides an example that can be modified to support InfluxDB's native format, or used unmodified with the InfluxDB Graphite input plugin. However, there are scaleability advantages to placing the sFlow-RT analytics engine in front of the time series database. For example, in large scale cloud environments the metrics for each member of a dynamic pool isn't necessarily worth trending since virtual machines are frequently added and removed. Instead, sFlow-RT tracks all the members of the pool, calculates summary statistics for the pool, and logs the summary statistics to the time series database. This pre-processing can significantly reduce storage requirements, reducing costs and increasing query performance. The sFlow-RT analytics software also calculates traffic flow metrics, hot/missed Memcache keys, top URLs, exports events via syslog to Splunk, Logstash etc. and provides access to detailed metrics through its REST API. 

First install InfluxDB - in this case the software has been installed on host 10.0.0.30.

Next install sFlow-RT:

wget http://www.inmon.com/products/sFlow-RT/sflow-rt.tar.gz
tar -xvzf sflow-rt.tar.gz
cd sflow-rt

Edit the init.js script and add the following lines (modifying the dbURL to send metrics to the InfluxDB instance):

var dbURL = "http://10.0.0.30:8086/db/inmon/series?u=root&p=root";

setIntervalHandler(function() {
  var metrics = ['min:load_one','q1:load_one','med:load_one',
                 'q3:load_one','max:load_one'];
  var vals = metric('ALL',metrics,{os_name:['linux']});
  var body = [];
  for each (var val in vals) {
     body.push({name:val.metricName,columns:['val'],points:[[val.metricValue]]});
  }
  http(dbURL,'post', 'application/json', JSON.stringify(body));
} , 15);

Now start sFlow-RT:

./start.sh

The script makes an sFlow-RT metrics() query every 15 seconds and posts the results to InfluxDB.

The screen capture shows InfluxDB's SQL like query language and a basic query demonstrating that the metrics are being logged in the database. However, the web interface is rudimentary and a dashboard builder simplifies querying and presentation of the time series data.

Grafana is a powerful HTML 5 dashboard building tool that supports InfluxDB, Graphite, and OpenTSDB.

The screen shot shows the Grafana query builder, offering simple drop down menus that make it easy to build complex charts. The resulting chart, shown below, can be combined with additional charts to build a custom dashboard.

The sFlow standard delivers the comprehensive instrumentation of data center infrastructure and is easily integrated with DevOps tools - see Visibility and the software defined data center

Update January 31, 2016:

The InfluxDB REST API changed with version 0.9 and the above sFlow-RT script will no longer work. The new API is described in Creating a database using the HTTP API. The following version of the script has been updated to use the new API:

var dbURL = "http://10.0.0.30:8086/write?db=mydb";

setIntervalHandler(function() {
  var metrics = ['min:load_one','q1:load_one','med:load_one',
                 'q3:load_one','max:load_one'];
  var vals = metric('ALL',metrics,{os_name:['linux']});
  var body = [];
  for each (var val in vals) {
     body.push(val.metricName.replace(/[^a-zA-Z0-9_]/g,'_') + ' value=' + val.metricValue);
  }
  try { http(dbURL,'post', 'text/plain', body.join('\n')); }
  catch(e) { logWarning('http error ' + e); }
} , 15);

Update April 27, 2016

The sFlow-RT software no longer ships with an init.js file.

Instead, create an influxdb.js file in the sFlow-RT home directory and add the JavaScript code. Next, edit the start.sh file to add a script.file=influxdb.js option, i.e.

RT_OPTS="-Dscript.file=influxdb.js -Dsflow.port=6343 -Dhttp.port=8008"

The script should be loaded when sFlow-RT is started.