Metric export to Graphite

Figure 1: Cluster performance metrics

Cluster performance metrics describes how sFlow-RT can be used to calculate summary metrics for cluster performance. The article includes a Python script that polls sFlow-RT's REST API and then sends metrics to to Graphite. In this article sFlow-RT's internal scripting API will be used to send metrics directly to Graphite.

Figure 2: Components of sFlow-RT

The following script (graphite.js) re-implements the Python example (generating a sum of the load_one metric for a cluster of Linux machines) in JavaScript using sFlow-RT built-in functions for retrieving metrics and sending them to Graphite:

// author: Peter
// version: 1.0
// date: 11/23/2013
// description: Log metrics to Graphite

include('extras/json2.js');

var graphiteServer = "10.0.0.151";
var graphitePort = null;

var errors = 0;
var sent = 0;
var lastError;

setIntervalHandler(function() {
  var names = ['sum:load_one'];
  var prefix = 'linux.';
  var vals = metric('ALL',names,{os_name:['linux']});
  var metrics = {};
  for(var i = 0; i < names.length; i++) {
    metrics[prefix + names[i]] = vals[i].metricValue;
  }
  try { 
    graphite(graphiteServer,graphitePort,metrics);
    sent++;
  } catch(e) {
    errors++;
    lastError = e.message;
  }
} , 15);

setHttpHandler(function() {
  var message = { 'errors':errors,'sent':sent };
  if(lastError) message.lastError = lastError;
  return JSON.stringify(message);
});

The interval handler function runs every 15 seconds and retrieves the set of metrics in the names array (in this case just one metrics, but multiple metrics could be retrieved). The names are then converted into a Graphite friendly form (prefixing each metric with the token linux. so that they can be easily grouped) and then sent to the Graphite collector running on 10.0.0.151 using the default TCP port 2003. The script also keeps track of any errors and makes them available through the URL /script/graphite.js/json

The following command line argument loads the script on startup:

-Dscript.file=graphite.js

The following Graphite screen capture below shows a trend of the metric:

There are a virtually infinite number of core and derived metrics that can be collected by sFlow-RT using standard sFlow instrumentation embedded in switches, servers and applications throughout the data center. For example Packet loss describes the importance of collecting network packet loss metrics and including them in performance dashboards.

Figure 2: Visibility and the software defined data center

While having access to all these metrics is extremely useful, not all of them need to be stored in Graphite. Using sFlow-RT to calculate and selectively export high value metrics reduces pressure on the time series database, while still allowing any of the remaining metrics to be polled using the REST API when needed.

Finally, metrics export is only one of many applications for sFlow data, some of which have been described on this blog. The data center wide visibility provided by sFlow-RT supports orchestration tools and allows them to automatically optimize the allocation of compute, storage and application resources and the placement of loads on these resources.

Cluster performance metrics

The diagram shows three clusters of servers in a data center (e.g. web, Memcache, Hadoop, or application cluster). Each cluster may consist of hundreds of individual servers. Host sFlow agents on the physical or virtual hosts efficiently export standard metrics that can be processed by a wide range of performance analysis tools, including: Graphite, Ganglia, sFlowTrend, log analyzers, etc.

While many tools are capable of consuming sFlow metrics directly, not all tools have the scaleability to handle large clusters, or the ability to calculate summary statistics characterizing cluster performance (rather than simply reporting on each member of the cluster).

This article describes how the sFlow-RT analytics engine is used to collect sFlow from large clusters and report on overall cluster health. The diagram illustrates how sFlow-RT is deployed, consuming sFlow measurements from the servers and making summary statistics available through its REST API so that they can be used to populate a performance dashboard like Graphite.

The following summary statistics are supported by sFlow-RT:

• max: Largest value
• min: Smallest value
• sum: Total value
• avg: Average value
• var: Variance
• sdev: Standard deviation
• med: Median value
• q1: First quartile
• q2: Second quartile (same as med:)
• q3: Third quartile
• iqr: Inter-quartile range (i.e. q3 - q1)

Any programming language capable of making HTTP requests (Perl, Python, Java, Javascript, bash, etc.) can be used to retrieve metrics from sFlow-RT using the URL:

http://server:8008/metric/agents/metrics/json?filter

Where:

• server The host running sFlow-RT
• agents A semicolon separated list of host addresses or names, or ALL to include all hosts.
• metrics A comma separated list of metrics to retrieve.
• filter A filter to further restrict the hosts to include in the query.

Use of the API is best illustrated by a few examples:

http://localhost:8008/metric/web1;web2;web3/avg:load_one,avg:http_method_get/json

produces the results:

[
 {
  "metricN": 3,
  "metricName": "avg:load_one",
  "metricValue": 0.10105350773249354,
  "updateTime": 1360040842222
 },
 {
  "metricN": 3,
  "metricName": "avg:http_method_get",
  "metricValue": 54.015954359255026,
  "updateTime": 1360040842721
 }
]

The following query uses a filter to select servers whose hostname starts with the prefix "mem":

http://localhost:8008/metric/ALL/med:bytes_out,iqr:bytes_out/json?host_name=mem*

The following Python script polls sFlow-RT for cluster statistics every 60 seconds and posts the results to a Graphite collector (10.0.0.151):

import requests
import json
import time
import socket

sock = socket.socket()
sock.connect(("10.0.0.151",2003))

url = 'http://localhost:8008/metric/ALL/sum:load_one/json'
while 1 == 1:
  r = requests.get(url)
  if r.status_code != 200: break
  vals = r.json()
  if len(vals) == 0: continue
  for v in vals:
    mname  = v["metricName"]
    mvalue = v["metricValue"]
    mtime  = v["updateTime"] / 1000
    message = 'clusterB.%s %f %i\n' % (mname,mvalue,mtime)
    sock.sendall(message)
    time.sleep(60)

Finally, sFlow (and sFlow-RT) is not limited to monitoring server metrics. The switches connecting the servers in the clusters can also be monitored (the sFlow standard is supported by most switch vendors). The network can quickly become a bottleneck as cluster size increases and it is important to track metrics such as link utilization, packet discards etc. that can result in severe performance degradation. In addition, sFlow instrumentation is available for Apache, NGINX, Java, Memcached and custom applications - providing details such as URLs, response times, status codes, etc., and tying application, server and network performance together to provide a comprehensive view of performance.

Graphite

The Graphite realtime charting software provides a flexible way to store and plot time series data. Graphite was originally developed by Orbitz.com, but is now an open source project that is growing in popularity among system administrators. However, while Graphite is very good at recording metrics and building dashboards, it provides no native monitoring capabilities, instead relying on user provided agents to make measurements.

The open source Host sFlow agent is an exact complement to Graphite, providing a lightweight, portable agent that exports standard metrics from a wide variety of systems. Deploying Host sFlow agents with a Graphite collector offers a complete, highly scalable monitoring solution.

The diagram shows the essential components of the solution. The Host sFlow agents continuously send metrics to the Graphite collector. The sflow2graphite script converts the binary sFlow data into Graphite's text based messages and submits them to the Graphite server.

First install sflowtool on the Graphite server. The sflow2graphite script requires sflowtool to receive binary sFlow datagrams and convert them into a text based representation. Next, download the sflow2graphite.pl script from http://sflow2graphite.googlecode.com.

Next run the following command on the Graphite server:

./sflow2graphite.pl

Now, configure the Host sFlow agents to send sFlow to the Graphite server. Within minutes, metrics from the hosts should start appearing in Graphite.

Note: If you are already collecting sFlow data with another tool, like Ganglia, then you may want to consider forwarding sFlow from the existing collector. Alternatively, you may want to extract the data from the existing collector and feed it into Graphite - in the case of Ganglia there are a number of solutions available.

It is worth noting that performance metrics may be gauges or counters. In the case of a gauge (like a system load average), Graphite will automatically display a useful chart. In the case of a counter metric (like total disk bytes read), Graphite will display the increasing total by default.

The screen capture above shows how you can apply a derivative function to the metric to convert a counter into a rate. The following charts trends disk bytes read/written as a bytes/second rate:

Finally, Graphite uses a hierarchical naming convention for metrics. If the default metrics names generated by the sflow2graphite script doesn't fit your naming hierarchy, simply edit the table of names in the sflow2graphite script to apply your own names.