Tuesday, December 9, 2014

InfluxDB and Grafana

Cluster performance metrics describes how to use sFlow-RT to calculate metrics and post them to Graphite. This article will describe how to use sFlow with the InfluxDB time series database and Grafana dashboard builder.

The diagram shows the measurement pipeline. Standard sFlow measurements from hosts, hypervisors, virtual machines, containers, load balancers, web servers and network switches stream to the sFlow-RT real-time analytics engine. Over 40 vendors implement the sFlow standard and compatible products are listed on sFlow.org. The open source Host sFlow agent exports standard sFlow metrics from hosts. For additional background, the Velocity conference talk provides an introduction to sFlow and case study from a large social networking site.
It is possible to simply convert the raw sFlow metrics into InfluxDB metrics. The sflow2graphite.pl script provides an example that can be modified to support InfluxDB's native format, or used unmodified with the InfluxDB Graphite input plugin. However, there are scaleability advantages to placing the sFlow-RT analytics engine in front of the time series database. For example, in large scale cloud environments the metrics for each member of a dynamic pool isn't necessarily worth trending since virtual machines are frequently added and removed. Instead, sFlow-RT tracks all the members of the pool, calculates summary statistics for the pool, and logs the summary statistics to the time series database. This pre-processing can significantly reduce storage requirements, reducing costs and increasing query performance. The sFlow-RT analytics software also calculates traffic flow metrics, hot/missed Memcache keys, top URLsexports events via syslog to Splunk, Logstash etc. and provides access to detailed metrics through its REST API
First install InfluxDB - in this case the software has been installed on host 10.0.0.30.

Next install sFlow-RT:
wget http://www.inmon.com/products/sFlow-RT/sflow-rt.tar.gz
tar -xvzf sflow-rt.tar.gz
cd sflow-rt
Edit the init.js script and add the following lines (modifying the dbURL to send metrics to the InfluxDB instance):
var dbURL = "http://10.0.0.30:8086/db/inmon/series?u=root&p=root";

setIntervalHandler(function() {
  var metrics = ['min:load_one','q1:load_one','med:load_one',
                 'q3:load_one','max:load_one'];
  var vals = metric('ALL',metrics,{os_name:['linux']});
  var body = [];
  for each (var val in vals) {
     body.push({name:val.metricName,columns:['val'],points:[[val.metricValue]]});
  }
  http(dbURL,'post', 'application/json', JSON.stringify(body));
} , 15);
Now start sFlow-RT:
./start.sh
The script makes an sFlow-RT metrics() query every 15 seconds and posts the results to InfluxDB.
The screen capture shows InfluxDB's SQL like query language and a basic query demonstrating that the metrics are being logged in the database. However, the web interface is rudimentary and a dashboard builder simplifies querying and presentation of the time series data.

Grafana is a powerful HTML 5 dashboard building tool that supports InfluxDB, Graphite, and OpenTSDB.
The screen shot shows the Grafana query builder, offering simple drop down menus that make it easy to build complex charts. The resulting chart, shown below, can be combined with additional charts to build a custom dashboard.
The sFlow standard delivers the comprehensive instrumentation of data center infrastructure and is easily integrated with DevOps tools - see Visibility and the software defined data center

Update January 31, 2016:

The InfluxDB REST API changed with version 0.9 and the above sFlow-RT script will no longer work. The new API is described in Creating a database using the HTTP API. The following version of the script has been updated to use the new API:
var dbURL = "http://10.0.0.30:8086/write?db=mydb";

setIntervalHandler(function() {
  var metrics = ['min:load_one','q1:load_one','med:load_one',
                 'q3:load_one','max:load_one'];
  var vals = metric('ALL',metrics,{os_name:['linux']});
  var body = [];
  for each (var val in vals) {
     body.push(val.metricName.replace(/[^a-zA-Z0-9_]/g,'_') + ' value=' + val.metricValue);
  }
  try { http(dbURL,'post', 'text/plain', body.join('\n')); }
  catch(e) { logWarning('http error ' + e); }
} , 15);
Update April 27, 2016

The sFlow-RT software no longer ships with an init.js file.

Instead, create an influxdb.js file in the sFlow-RT home directory and add the JavaScript code. Next, edit the start.sh file to add a script.file=influxdb.js option, i.e.
RT_OPTS="-Dscript.file=influxdb.js -Dsflow.port=6343 -Dhttp.port=8008"
The script should be loaded when sFlow-RT is started.

17 comments:

  1. There is not a lot of documentation for InfluxDB or sFlow-RT. I get an error when I start sFlow-RT: WARNING: init.js init.js#26 IO error java.io.IOException: Server returned HTTP response code: 401 for URL: http://10.144.52.246:8086/db/test_sflow2/series?u=***$p=*** which I assumed meant the userid did not have the right privileges but I can do simple InfluxDB commands from my browser. I tried to enable http.log in sFlow-RT but I have not found the right syntax for start.sh.

    Any suggestions? Can you provide more details on how this works?

    ReplyDelete
    Replies
    1. Try accessing the URL using cURL. You should also check the InfluxDB logs to see if there is an indication why the request was rejected. I suspect that your userID or password contains characters that need to be escaped in the URL.

      Delete
  2. Hi Peter,
    I'm running into some issues debugging my init.js script. Would you mind posting your entire init.js script so I can see that for reference?
    Thanks!

    ReplyDelete
    Replies
    1. That was the entire script. What are the errors you are seeing? A good way to debug is to use logInfo() to print variables. For example, logInfo(JSON.stringify(body)) to see what message is being pushed to InfluxDB.

      This example requires that you have Host sFlow agents on your servers (otherwise there will be no metrics to export). You can verify that metrics are arriving at sFlow-RT by making sure that the servers show up as agents under the sFlow-RT /agents/html page. Click on an agent to see the metric values it is exporting.

      Delete
  3. This comment has been removed by the author.

    ReplyDelete
  4. Hi there, just for reference - sFlow-RT doesn't work with InfluxDB v0.9 due to the change in protocol.

    To get around this, I've wrapped sflowtool in a python script, which can then be made to run in the background, read from stdout and post to InfluxDB.

    Here is the source:
    https://github.com/ab77/beastcraft-telemetry/blob/master/traffic.py

    Hope this helps someone..

    -- ab1

    ReplyDelete
    Replies
    1. Thank you for pointing out the InfluxDB v9.0 API change. I have updated the article to include a version of the sFlow-RT script that works with the new API.

      Delete
  5. I know this is a bit of a long shot, but I'm not having issues with this, I went and grabbed v0.9.6 influxdb in hopes that my errors would cease but, not so much, i get the following error.

    WARNING: http error InternalError: IO error java.io.IOException: Server returned HTTP response code: 400 for URL: http://influx:8086/write?db=mydb (init.js#25)

    ReplyDelete
  6. Did you create the mydb database? Here is a link to another example sending data to InfluxDB https://github.com/sflow-rt/ix-metrics/blob/master/scripts/metrics.js#L170-L225

    ReplyDelete
  7. I did however I am still getting the same error. I'll try and take a look at the other example this evening.

    ReplyDelete
  8. Hello Peter,
    I tried to follow your article but I am getting this error after run sflow-rt.
    WARNING: http error InternalError: not found (influxdb.js#11)
    Could you please let me know your suggestion?

    ReplyDelete
    Replies
    1. You need to create the database "mydb" in InfluxDB before sFlow-RT can write to it. You can cURL to create the database:

      curl -i -XPOST http://localhost:8086/query --data-urlencode "q=CREATE DATABASE mydb"

      See https://docs.influxdata.com/influxdb/v1.2/guides/writing_data/

      Delete
    2. This comment has been removed by the author.

      Delete
    3. This comment has been removed by the author.

      Delete
    4. Thanks Peter. Now I can see sFlow-RT data (max/med/q1/q3_load_one) on both InfluxDB and Grafana. Could you please suggest how can I export all metrics of sFlow-RT (eg. OVS traffic, hsflowd metrics) to InfluxDB?

      Delete
    5. There are a number of examples on this blog, http://blog.sflow.com/search/label/InfluxDB. You need to select the metrics you want to export and decide how you want to label and tag them for InfluxDB. Writing Applications provides an overview of the metrics query functions.

      Delete
    6. Thanks for your useful links, Peter.

      Delete