Real-time Kubernetes cluster monitoring example

The Sunburst GPU chart updates every second to show a real-time view of the share of GPU resources being consumed by namespaces operating on the Nautilus hyperconverged Kubernetes cluster. The Nautilus cluster tightly couples distributes storage, GPU, and CPU resources to share among the participating research organizations.

The Sunburst Process chart provides an up to the second view of the cluster-wide share of CPU resources used by each namespace.

The Sunburst DNS chart shows a real-time view of network activity generated by each namespace. The chart is produced by looking up DNS names for network addresses observed in packet flows using the Kubernetes DNS service. The domain names contain information about the namespace, service, and node generating the packets. Most traffic is exchanges between nodes within the cluster (identified as local). The external (not local) traffic is also shown by DNS name.

The Sunburst Protocols chart shows the different network protocols being used to communicate between nodes in the cluster. The chart shows the IP over IP tunnel traffic used for network virtualization.

Clicking on a segment in the Sunburst Protocols chart allows the selected traffic to be examined in detail using the Flow Browser. In this example, DNS names are again used to translate raw packet flow data into inter-namespace flows. See Defining Flows for information on the flow analytics capabilities that can be explored using the browse-flows application.

The Discard Browser provides a detailed view of any network packets dropped in the cluster. In this chart inter-namespace dropped packets are displayed, identifying the haproxy service as the largest source of dropped packets. 

The final chart shows an up to the second view of the average power consumed by a GPU in the cluster (approximately 250 Watts per GPU).

The diagram shows the elements of the monitoring solution. Host sFlow agents deployed on each Node in the Kubernetes Cluster stream standard sFlow telemetry to an instance of the sFlow-RT real-time analytics software that provides cluster wide metrics through a REST API, where they can be viewed, or imported into time series databases like Prometheus and trended in dashboards using tools like Grafana.

Note: sFlow is widely supported by network switches and routers. Enable sFlow monitoring in the physical network infrastructure for end-to-end visibility.

Create the following sflow-rt.yml file to deploy the pre-built sflow/prometheus Docker image, bundling sFlow-RT with the applications used in this article:

apiVersion: v1
kind: Service
metadata:
  name: sflow-rt-sflow
spec:
  type: NodePort
  selector:
    name: sflow-rt
  ports:
    - protocol: UDP
      port: 6343
---
apiVersion: v1
kind: Service
metadata:
  name: sflow-rt-rest
spec:
  type: LoadBalancer
  selector:
    name: sflow-rt
  ports:
    - protocol: TCP
      port: 8008
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sflow-rt
spec:
  replicas: 1
  selector:
    matchLabels:
      name: sflow-rt
  template:
    metadata:
      labels:
        name: sflow-rt
    spec:
      containers:
      - name: sflow-rt
        image: sflow/prometheus:latest
        ports:
          - name: http
            protocol: TCP
            containerPort: 8008
          - name: sflow
            protocol: UDP
            containerPort: 6343

Run the following command to deploy the service:

kubectl apply -f sflow-rt.yml

Now create the following host-sflow.yml file to deploy the pre-built sflow/host-sflow Docker image:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: host-sflow
spec:
  selector:
    matchLabels:
      name: host-sflow
  template:
    metadata:
      labels:
        name: host-sflow
    spec:
      restartPolicy: Always
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: host-sflow
        image: sflow/host-sflow:latest
        env:
          - name: COLLECTOR
            value: "sflow-rt-sflow"
          - name: SAMPLING
            value: "10"
          - name: NET
            value: "host"
          - name: DROPMON
            value: "enable"
        volumeMounts:
          - mountPath: /var/run/docker.sock
            name: docker-sock
            readOnly: true
      volumes:
        - name: docker-sock
          hostPath:
            path: /var/run/docker.sock

Run the following command to deploy the agents:

kubectl apply -f host-sflow.yml

Telemetry should immediately start streaming as a Host sFlow agent is started on each node in the cluster.

Note: Exporting GPU performance metrics from the NVIDIA GPUs in the Nautilus cluster requires a special version of the Host sFlow agent built using the NVIDIA supplied Docker image that includes GPU drivers, see https://gitlab.nrp-nautilus.io/prp/sflow/

Access the sFlow-RT web user interface to confirm that telemetry is being received.

The sFlow-RT Status page confirms that telemetry is being received from all 180 nodes in the cluster.

Note: If you don't currently have access to a production Kubernetes cluster, you can experiment with this solution using Docker Desktop, see Kubernetes testbed.

The charts shown in this article are accessed via the sFlow-RT Apps tab.

The sFlow-RT applications are designed to explore the available metrics, but don't provide persistent storage. Prometheus export functionality allows metrics to be recorded in a time series database to drive operational dashboards, see  Flow metrics with Prometheus and Grafana.

Collecting Docker Swarm service metrics

This article demonstrates how to address the challenge of monitoring dynamic Docker Swarm deployments and track service performance metrics using existing on-premises and cloud monitoring tools like Ganglia, Graphite, InfluxDB, Grafana, SignalFX, Librato, etc.

In this example, Docker Swarm is used to deploy a simple web service on a four node cluster:

docker service create --replicas 2 -p 80:80 --name apache httpd:2.4

Next, the following script tests the agility of monitoring systems by constantly changing the number of replicas in the service:

#!/bin/bash
while true
do
  docker service scale apache=$(( ( RANDOM % 20 )  + 1 ))
  sleep 30
done

The above test is easy to set up and is a quick way to stress test monitoring systems and reveal accuracy and performance problems when they are confronted with container workloads.

Many approaches to gathering and recording metrics were developed for static environments and have a great deal of difficulty tracking rapidly changing container-based service pools without missing information, leaking resources, and slowing down. For example, each new container in Docker Swarm has unique name, e.g. apache.16.17w67u9157wlri7trd854x6q0. Monitoring solutions that record container names, or even worse, index data by container name, will suffer from bloated databases and resulting slow queries.

The solution is to insert a stream processing analytics stage in the metrics pipeline that delivers a consistent set of service level metrics to existing tools.

The asynchronous metrics export method implemented in the open source Host sFlow agent is part of the solution, sending a real-time telemetry stream to a centralized sFlow collector which is then able to deliver a comprehensive view of all services deployed on the Docker Swarm cluster.

The sFlow-RT real-time analytics engine completes the solution by converting the detailed per instance metrics into service level statistics which are in turn streamed to a time series database where they drive operational dashboards.

For example, the following swarmmetrics.js script computes cluster and service level metrics and exports them to InfluxDB:

var docker = "https://10.0.0.134:2376/services";
var certs = '/tls/';

var influxdb = "http://10.0.0.50:8086/write?db=docker"

var clustermetrics = [
  'avg:load_one',
  'max:cpu_steal',
  'sum:node_domains'
];

var servicemetrics = [
  'avg:vir_cpu_utilization',
  'avg:vir_bytes_in',
  'avg:vir_bytes_out'
];

function sendToInfluxDB(msg) {
  if(!msg || !msg.length) return;

  var req = {
    url:influxdb,
    operation:'POST',
    headers:{"Content-Type":"text/plain"},
    body:msg.join('\n')
  };
  req.error = function(e) {
    logWarning('InfluxDB POST failed, error=' + e);
  }
  try { httpAsync(req); }
  catch(e) {
    logWarning('bad request ' + req.url + ' ' + e);
  }
}

function clusterMetrics(nservices) {
  var vals = metric(
    'ALL', clustermetrics,
    {'node_domains':['*'],'host_name':['vx*host*']}
  );
  var msg = [];
  msg.push('swarm.services value='+nservices);
  msg.push('nodes value='+(vals[0].metricN || 0));
  for(var i = 0; i < vals.length; i++) {
    let val = vals[i];
    msg.push(val.metricName+' value='+ (val.metricValue || 0));
  } 
  sendToInfluxDB(msg);
}

function serviceMetrics(name, replicas) {
  var vals = metric(
    'ALL', servicemetrics,
    {'vir_host_name':[name+'\\.*'],'vir_cpu_state':['running']}
  );
  var msg = [];
  msg.push('replicas_configured,service='+name+' value='+replicas);
  msg.push('replicas_measured,service='+name+' value='+(vals[0].metricN || 0));
  for(var i = 0; i < vals.length; i++) {
    let val = vals[i];
    msg.push(val.metricName+',service='+name+' value='+(val.metricValue || 0));
  }
  sendToInfluxDB(msg);
}

setIntervalHandler(function() {
  var i, services, service, spec, name, replicas, res;
  try { services = JSON.parse(http2({url:docker, certs:certs}).body); }
  catch(e) { logWarning("cannot get " + docker + " error=" + e); }
  if(!services || !services.length) return;

  clusterMetrics(services.length);

  for(i = 0; i < services.length; i++) {
    service = services[i];
    if(!service) continue;
    spec = service["Spec"];
    if(!spec) continue;
    name = spec["Name"];
    if(!name) continue;
 
    replicas = spec["Mode"]["Replicated"]["Replicas"];
    serviceMetrics(name, replicas);
  }
},10);

Some notes on the script:

1. Only a few representative metrics are being monitored, many more are available, see Metrics.
2. The setIntervalHandler function is run every 10 seconds. The function queries Docker REST API for the current list of services and then calculates summary statistics for each service. The summary statistics are then pushed to InfluxDB via a REST API call.
3. Cluster performance metrics describes the set of summary statistics that can be calculated.
4. Writing Applications provides additional information on sFlow-RT scripting and REST APIs.

Start gathering metrics:

docker run -v `pwd`/tls:/tls -v `pwd`/swarmmetrics.js:/sflow-rt/swarmmetrics.js \
-e "RTPROP=-Dscript.file=swarmmetrics.js" \
-p 8008:8008 -p 6343:6343/udp sflow/sflow-rt

The results are shown in the Grafana dashboard at the top of this article. The charts show 30 minutes of data. The top Replicas by Service chart compares the number of replicas configured for each service with the number of container instances that the monitoring system is tracking. The chart demonstrates that the monitoring system is accurately tracking the rapidly changing service pool and able to deliver reliable metrics. The middle Network IO by Service chart shows a brief spike in network activity whenever the number of instances in the apache service is increased. Finally, the bottom Cluster Size chart confirms that all four nodes in the Swarm cluster are being monitored.

This solution is extremely scaleable. For example, increasing the size of the cluster from 4 to 1,000 nodes increases the amount of raw data that sFlow-RT needs to process to accurately calculate service metrics, but has have no effect on the amount of data sent to the time series database and so there is no increase in storage requirements or query response time.

Pre-processing the stream of raw data reduces the cost of the monitoring solution, either in terms of the resources required by an on-premises monitoring solutions, or the direct costs of cloud based solutions which charge per data point per minute per month. In this case the raw telemetry stream contains hundreds of thousands of potential data points per minute per host - filtering and summarizing the data reduces monitoring costs by many orders of magnitude.

This example can easily be modified to send data into any on-premises or cloud based backend, examples in this blog include: SignalFX, Librato, Graphite and Ganglia. In addition, Docker 1.12 swarm mode elastic load balancing describes how the same architecture can be used to dynamically resize service pools to meet changing demand.

Docker 1.12 swarm mode elastic load balancing

Docker Built-In Orchestration Ready For Production: Docker 1.12 Goes GA describes the native swarm mode feature that integrates cluster management, virtual networking, and policy based deployment of services.

This article will demonstrate how real-time streaming telemetry can be used to construct an elastic load balancing solution that dynamically adjusts service capacity to match changing demand.

Getting started with swarm mode describes the steps to configure a swarm cluster. For example, following command issued on any of the Manager nodes deploys a web service on the cluster:

docker service create --replicas 2 -p 80:80 --name apache httpd:2.4

And the following command raises the number of containers in the service pool from 2 to 4:

docker service scale apache=4

Asynchronous Docker metrics describes how sFlow telemetry provides the real-time visibility required for elastic load balancing. The diagram shows how streaming telemetry allows the sFlow-RT controller to determine the load on the service pool so that it can use the Docker service API to automatically increase or decrease the size of the pool as demand changes. Elastic load balancing of the service pools ensures consistent service levels by adding additional resources if demand increases. In addition, efficiency is improved by releasing resources when demand drops so that they can be used by other services. Finally, global visibility into all resources and services makes it possible to load balance between services, reducing service pools for non-critical services to release resources during peak demand.

The first step is to install and configure Host sFlow agents on each of the nodes in the Docker swarm cluster. The following /etc/hsflowd.conf file configures Host sFlow to monitor Docker and send sFlow telemetry to a designated collector (in this case 10.0.0.162):

sflow {
  sampling = 400
  polling = 10
  collector { ip = 10.0.0.162 } 
  docker { }
  pcap { dev = docker0 }
  pcap { dev = docker_gwbridge }
}

Note: The configuration file is identical for all nodes in the cluster making it easy to automate the installation and configuration of sFlow monitoring using  Puppet, Chef, Ansible, etc.

Verify that the sFlow measurements are arriving at the collector node (10.0.0.162) using sflowtool:

docker -p 6343:6343/udp sflow/sflowtool

The following elb.js script implements elastic load balancer functionality using the sFlow-RT real-time analytics engine:

var api = "https://10.0.0.134:2376";
var certs = '/tls/';
var service = 'apache';

var replicas_min = 1;
var replicas_max = 10;
var util_min = 0.5;
var util_max = 1;
var bytes_min = 50000;
var bytes_max = 100000;
var enabled = false;

function getInfo(name) {
  var info = null;
  var url = api+'/services/'+name;
  try { info = JSON.parse(http2({url:url, certs:certs}).body); }
  catch(e) { logWarning("cannot get " + url + " error=" + e); }
  return info;
}

function setReplicas(name,count,info) {
  var version = info["Version"]["Index"];
  var spec = info["Spec"];
  spec["Mode"]["Replicated"]["Replicas"]=count;
  var url = api+'/v1.24/services/'+info["ID"]+'/update?version='+version;
  try {
    http2({
      url:url, certs:certs, method:'POST',
      headers:{'Content-Type':'application/json'},
      body:JSON.stringify(spec)
    });
  }
  catch(e) { logWarning("cannot post to " + url + " error=" + e); }
  logInfo(service+" set replicas="+count);
}

var hostpat = service+'\\.*';
setIntervalHandler(function() {
  var info = getInfo(service);
  if(!info) return;

  var replicas = info["Spec"]["Mode"]["Replicated"]["Replicas"];
  if(!replicas) {
    logWarning("no active members for service=" + service);
    return;
  }

  var res = metric(
    'ALL', 'avg:vir_cpu_utilization,avg:vir_bytes_in,avg:vir_bytes_out',
    {'vir_host_name':[hostpat],'vir_cpu_state':['running']}
  );

  var n = res[0].metricN;

  // we aren't seeing all the containers (yet)
  if(replicas !== n) return;

  var util = res[0].metricValue;
  var bytes = res[1].metricValue + res[2].metricValue;

  if(!enabled) return;

  // load balance
  if(replicas < replicas_max && (util > util_max || bytes > bytes_max)) {
    setReplicas(service,replicas+1,info);
  }
  else if(replicas > replicas_min && util < util_min && bytes < bytes_min) {
    setReplicas(service,replicas-1,info);
  }
},2);

setHttpHandler(function(req) {
  enabled = req.query && req.query.state && req.query.state[0] === 'enabled';
  return enabled ? "enabled" : "disabled";
});

Some notes on the script:

1. The setReplicas(name,count,info) function uses the Docker Remote API to implement functionality equivalent to the docker service scale name=count command shown earlier. The REST API is accessible at https://10.0.0.134:2376 in this example.
2. The setIntervalHandler() function runs every 2 seconds, retrieving metrics for the service pool and scaling the number of replicas in the service up or down based on thresholds.
3. The setHttpHandler() function exposes a simple REST API for enabling / disabling the load balancer functionality. The API can easily be extended to all thresholds to be set, to report statistics, etc.
4. Certificates, key.pem, cert.pem, and ca.pem, required to authenticate API requests must be present in the /tls/ directory.
5. The thresholds are set to unrealistically low values for the purpose of this demonstration.
6. The script can easily be extended to load balance multiple services simultaneously.
7. Writing Applications provides additional information on sFlow-RT scripting.

Run the controller:

docker run -v `pwd`/tls:/tls -v `pwd`/elb.js:/sflow-rt/elb.js \
 -e "RTPROP=-Dscript.file=elb.js" -p 8008:8008 -p 6343:6343/udp -d sflow/sflow-rt

The autoscaling functionality can be enabled:

curl "http://localhost:8008/script/elb.js/json?state=enabled"

and disabled:

curl "http://localhost:8008/script/elb.js/json?state=disabled"

using the REST API exposed by the script.

The chart above shows the results of a simple test to demonstrate the elastic load balancer function. First, ab - Apache HTTP server benchmarking tool was used to generate load on the apache service running under Docker swarm:

ab -rt 60 -n 300000 -c 4 http://10.0.0.134/

Next, the test was repeated with the elastic load balancer enabled. The chart clearly shows that the load balancer is keeping the average network load on each container under control.

2016-09-24T00:57:10+0000 INFO: Listening, sFlow port 6343
2016-09-24T00:57:10+0000 INFO: Listening, HTTP port 8008
2016-09-24T00:57:10+0000 INFO: elb.js started
2016-09-24T01:00:17+0000 INFO: apache set replicas=2
2016-09-24T01:00:23+0000 INFO: apache set replicas=3
2016-09-24T01:00:27+0000 INFO: apache set replicas=4
2016-09-24T01:00:33+0000 INFO: apache set replicas=5
2016-09-24T01:00:41+0000 INFO: apache set replicas=6
2016-09-24T01:00:47+0000 INFO: apache set replicas=7
2016-09-24T01:00:59+0000 INFO: apache set replicas=8
2016-09-24T01:01:29+0000 INFO: apache set replicas=7
2016-09-24T01:01:33+0000 INFO: apache set replicas=6
2016-09-24T01:01:35+0000 INFO: apache set replicas=5
2016-09-24T01:01:39+0000 INFO: apache set replicas=4
2016-09-24T01:01:43+0000 INFO: apache set replicas=3
2016-09-24T01:01:45+0000 INFO: apache set replicas=2
2016-09-24T01:01:47+0000 INFO: apache set replicas=1

The sFlow-RT log shows that containers are added to the apache service to handle the increased load and removed once demand decreases.

This example relied on a small subset of the information available from the sFlow telemetry stream. In addition to container resource utilization, the Host sFlow agent exports an extensive set of metrics from the nodes in the Docker swarm cluster. If the nodes are virtual machines running in a public or private cloud, the metrics can be used to perform elastic load balancing of the virtual machine pool making up the cluster, increasing the cluster size if demand increases and reducing cluster size when demand decreases. In addition, poorly performing instances can be detected and removed from the cluster (see Stop thief! for an example).

The sFlow agents also efficiently report on traffic flowing within and between microservices running on the swarm cluster. For example, the following command:

docker run -p 6343:6343/udp -p 8008:8008 -d sflow/top-flows

launches the top-flows application to show an up to the second view of active flows in the network.

Comprehensive real-time analytics is critical to effectively managing agile container-bases infrastructure. Open source Host sFlow agents provide a lightweight method of instrumenting the infrastructure that unifies network and system monitoring to deliver a full set of standard metrics to performance management applications.

Network visibility with Docker

Microservices describes the critical role that network visibility provides as a common point of reference for monitoring, managing and securing the interactions between the numerous and diverse distributed service instances in a microservices deployment.

Industry standard sFlow is well placed to give network visibility into the Docker infrastructure used to support microservices. The sFlow standard is widely supported by data center switch vendors (Cisco, Arista, Juniper, Dell, HPE, Brocade, Cumulus, etc.)  providing a cost effective and scaleable method of monitoring the physical network infrastructure. In addition, Linux bridge, macvlan, ipvlan, adapters described how sFlow is also an efficient means of leveraging instrumentation built into the Linux kernel to extend visibility into Docker host networking.

The following commands build the Host sFlow binary package from sources on an Ubuntu 14.04 system:

sudo apt-get update
sudo apt-get install build-essential
sudo apt-get install libpcap-dev
sudo apt-get install wget
wget https://github.com/sflow/host-sflow/archive/v1.29.1.tar.gz
tar -xvzf v1.29.1.tar.gz
cd host-sflow-1.29.1
make DOCKER=yes PCAP=yes deb

This resulting hsflowd_1.29.1-1_amd64.deb package can be copied and installed on all the hosts in the Docker cluster using configuration management tools such as Puppet, Chef, Ansible, etc.

This article will explore the alternative of deploying sFlow agents as Docker containers.

Create a directory for the project and edit the Dockerfile:

mkdir hsflowd
cp hsflowd_1.29.1-1_amd64.deb hsflowd
cd hsflowd
printf "sflow {\n dnssd=on\n pcap { dev = docker0 }\n}" > hsflowd.conf
vi Dockerfile

Add the following contents to Dockerfile:

FROM   ubuntu:trusty
RUN    apt-get update && apt-get install -y libpcap0.8 docker.io
ADD    hsflowd_1.29.1-1_amd64.deb /tmp
RUN    dpkg -i /tmp/hsflowd_1.29.1-1_amd64.deb
ADD    hsflowd.conf /etc/hsflowd.conf
CMD    /etc/init.d/hsflowd start && tail -f /dev/null

Build the project:

docker build -t hsflowd .

Run the service:

docker run --pid=host --uts=host --net=host \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /sys/fs/cgroup/:/sys/fs/cgroup/:ro -d hsflowd

In this example, DNS Service Discovery (DNS-SD), is being used as the configuration method for the sFlow agents. Adding the following entry to DNS zone file allows the agents to automatically discover the designated sFlow analyzers, analytics1 and analytics2, and configuration parameters:

_sflow._udp   30  SRV     0 0 6343  analytics1
_sflow._udp   30  SRV     0 0 6343  analytics2
_sflow._udp   30  TXT     (
"txtvers=1"
"sampling=400"
"polling=20"
)

As soon as the container starts, the sFlow agent will make a DNS request to find the sFlow analyzers, which can themselves be packaged as Docker containers. Network and system analytics as a Docker microservice describes how sFlow analytics can be packaged as a RESTful service and integrated with a wide variety of on-site and cloud, orchestration, DevOps and Software Defined Networking (SDN) tools.

Any change to the entries in the zone file will be automatically picked up by the sFlow agents.

The agent has been configured for Docker bridged networking, monitoring traffic through bridge docker0. For macvlan or ipvlan networking, change the pcap setting from docker0 to eth0.

One of the major advantages of packaging the sFlow agents and analytics components as Docker containers is that large scale deployments can be automated using Docker Compose with Swarm, deploying sFlow agents on every node in the Swarm cluster to deliver real-time cluster-wide visibility into the resource consumption and communication patterns of all microservices running on the cluster.

SignalFx

SignalFx is an example of a cloud based analytics service. SignalFx provides a REST API for uploading metrics and a web portal that it simple to combine and trend data and build and share dashboards.

This article describes a proof of concept demonstrating how SignalFx's cloud service can be used to cost effectively monitor large scale cloud infrastructure by leveraging standard sFlow instrumentation. SignalFx offers a free 14 day trial, making it easy to evaluate solutions based on this demonstration.

The diagram shows the measurement pipeline. Standard sFlow measurements from hosts, hypervisors, virtual machines, containers, load balancers, web servers and network switches stream to the sFlow-RT real-time analytics engine. Metrics are pushed from sFlow-RT to SignalFx using the REST API.

Over 40 vendors implement the sFlow standard and compatible products are listed on sFlow.org. The open source Host sFlow agent exports standard sFlow metrics from hosts, virtual machines and containers and local services. For additional background, the Velocity conference talk provides an introduction to sFlow and case study from a large social networking site.

SignalFx's service is priced based on the number of data points that they need to store and they estimate a cost of $15 per host per month to record comprehensive host statistics at 10 second granularity. Collecting metrics from a cluster of 1,000 hosts would cost as much as $15,000 per month.

There are important scaleability and cost advantages to placing the sFlow-RT analytics engine in front of the metrics collection service. For example, in large scale cloud environments the metrics for each member of a dynamic pool isn't necessarily worth trending since virtual machines are frequently added and removed. Instead, sFlow-RT tracks all the members of the pool, calculates summary statistics for the pool, and logs the summary statistics. This pre-processing can significantly reduce storage requirements, reducing costs and increasing query performance. The sFlow-RT analytics software also calculates traffic flow metrics, hot/missed Memcache keys, top URLs, exports events via syslog to Splunk, Logstash etc. and provides access to detailed metrics through its REST API.

The following steps were involved in setting up the proof of concept.

First register for free trial at SignalFx.com.

Download and install sFlow-RT.

Create a signalfx.js script in the sFlow-RT home directory with the following lines (use the token from your SignalFx account):

var url = "https://ingest.signalfx.com/v2/datapoint";
var token = "YOUR_APP_API_TOKEN";

setIntervalHandler(function() {
  var metrics = ['min:load_one','q1:load_one','med:load_one',
                 'q3:load_one','max:load_one'];
  var vals = metric('ALL',metrics,{os_name:['linux']});
  var gauges = [];
  for each (var val in vals) {
     gauges.push({
       metric: val.metricName.replace(/[^a-zA-Z0-9_]/g,"_"),
       dimensions:{cluster:"Linux"},
       value: val.metricValue
     });
  }
  var body = {"gauge":gauges};
  var req = {
    url:url,
    operation:'post',
    headers: {
      'Content-Type':'application/json',
      'X-SF-TOKEN':token
    },
    body: JSON.stringify(body)
  };
  try { http2(req); }
  catch(e) { logWarning("metric upload failed " + e); }
} , 10); 

Add the following sFlow-RT configuration entry to load the script:

script.file=signalfx.js

Now start sFlow-RT. Cluster performance metrics describes the summary metrics that sFlow-RT can calculate. In this case, the load average minimum, maximum, and quartiles for the cluster are being calculated and pushed to SignalFx every minute.

Install Host sFlow agents on the physical or virtual machines in your cluster and direct them to send metrics to the sFlow-RT host. The installation steps can be easily automated using orchestration tools like Puppet, Chef, Ansible, etc.

Physical and virtual switches in the cluster can be configured to send sFlow to sFlow-RT in order to add traffic metrics to the mix, exporting metrics that characterizing traffic between service tiers etc. However, in public cloud environments, traffic flow information is typically not available. The articles, Amazon Elastic Compute Cloud (EC2) and Rackspace cloudservers describe how Host sFlow agents can be configured to monitor traffic between virtual machines in the cloud.

Metrics should start appearing in SignalFx as soon as the Host sFlow agents are started.

In this example, sFlow-RT is exporting 5 metrics to summarize the cluster performance, reducing the total monthly cost of monitoring the 1,000 host cluster to less than $15 per month. Of course there are likely to be more metrics that you will want to track, but the ability to selectively log high value metrics provides a way to control costs and maximize benefits.

If you are managing physical infrastructure then sFlow provides a simple way to incorporate network telemetry. For example, add the following metrics to the script to summarize network health:

• max:ifinutilization
• max:ifoututilization
• sum:ifindiscards
• sum:ifinerrors
• sum:ifoutdiscards
• sum:ifouterrors

A network connecting 1,000 physical hosts would have considerably more than 1,000 switch ports and summarizing the per port statistics greatly reduces the cost of monitoring the network. For a catalog of network, host, and application metrics, see Metrics.

Cloud analytics

Librato is an example of a cloud based analytics service (now part of SolarWinds). Librato provides an easy to use REST API for pushing metrics into their cloud service. The web portal makes it simple to combine and trend data and build and share dashboards.

This article describes a proof of concept demonstrating how Librato's cloud service can be used to cost effectively monitor large scale cloud infrastructure by leveraging standard sFlow instrumentation. Librato offers a free 30 day trial, making it easy to evaluate solutions based on this demonstration.

The diagram shows the measurement pipeline. Standard sFlow measurements from hosts, hypervisors, virtual machines, containers, load balancers, web servers and network switches stream to the sFlow-RT real-time analytics engine. Metrics are pushed from sFlow-RT to Librato using the REST API.

Over 40 vendors implement the sFlow standard and compatible products are listed on sFlow.org. The open source Host sFlow agent exports standard sFlow metrics from hosts. For additional background, the Velocity conference talk provides an introduction to sFlow and case study from a large social networking site.

Librato's service is priced based on the number of data points that they need to store. For example, a Host sFlow agent reports approximately 50 measurements per node. Collecting all the measurements from a cluster of 100 servers would generate 5000 metrics and cost $1,000 per month if metrics are stored at 15 second intervals.

There are important scaleability and cost advantages to placing the sFlow-RT analytics engine in front of the metrics collection service. For example, in large scale cloud environments the metrics for each member of a dynamic pool isn't necessarily worth trending since virtual machines are frequently added and removed. Instead, sFlow-RT tracks all the members of the pool, calculates summary statistics for the pool, and logs the summary statistics. This pre-processing can significantly reduce storage requirements, reducing costs and increasing query performance. The sFlow-RT analytics software also calculates traffic flow metrics, hot/missed Memcache keys, top URLs, exports events via syslog to Splunk, Logstash etc. and provides access to detailed metrics through its REST API.

The following steps were involved in setting up the proof of concept.

First register for free trial at Librato.com.

Find or build a server with Java 1.7+ and install sFlow-RT:

wget http://www.inmon.com/products/sFlow-RT/sflow-rt.tar.gz
tar -xvzf sflow-rt.tar.gz
cd sflow-rt

Edit the init.js script and add the following lines (modifying the user and token from your Librato account):

var url = "https://metrics-api.librato.com/v1/metrics";
var user = "first.last@mycompany.com";
var token = "55add91c806fb5f634ad1a334789a32e8d10a597815e6865aa84f0749324450e";

setIntervalHandler(function() {
  var metrics = ['min:load_one','q1:load_one','med:load_one',
                 'q3:load_one','max:load_one'];
  var vals = metric('ALL',metrics,{os_name:['linux']});
  var gauges = {};
  for each (var val in vals) {
     gauges[val.metricName] = {
       "value": val.metricValue,
       "source": "Linux_Pool"
     };
  }
  var body = {"gauges":gauges};
  http(url,'post', 'application/json', JSON.stringify(body), user, token);
} , 15); 

Now start sFlow-RT:

./start.sh

Cluster performance metrics describes the summary metrics that sFlow-RT can calculate. In this case, the load average minimum, maximum, and quartiles for the cluster are being calculated and pushed to Librato every 15 seconds.

Install Host sFlow agents on the physical or virtual machines in your cluster and direct them to send metrics to the sFlow-RT host. The installation steps can be easily automated using orchestration tools like Puppet, Chef, Ansible, etc.

Physical and virtual switches in the cluster can be configured to send sFlow to sFlow-RT in order to add traffic metrics to the mix, exporting metrics that characterizing traffic between service tiers etc. However, in public cloud environments, traffic flow information is typically not available. The articles, Amazon Elastic Compute Cloud (EC2) and Rackspace cloudservers describe how Host sFlow agents can be configured to monitor traffic between virtual machines in the cloud.

Metrics should start appearing in Librato as soon as the Host sFlow agents are started.

In this example, sFlow-RT is exporting 5 metrics to summarize the cluster performance, reducing the total monthly cost of monitoring the cluster from $1,000 to $1. Of course there are likely to be more metrics that you will want to track, but the ability to selectively log high value metrics provides a way to control costs and maximize benefits.

InfluxDB and Grafana

Cluster performance metrics describes how to use sFlow-RT to calculate metrics and post them to Graphite. This article will describe how to use sFlow with the InfluxDB time series database and Grafana dashboard builder.

The diagram shows the measurement pipeline. Standard sFlow measurements from hosts, hypervisors, virtual machines, containers, load balancers, web servers and network switches stream to the sFlow-RT real-time analytics engine. Over 40 vendors implement the sFlow standard and compatible products are listed on sFlow.org. The open source Host sFlow agent exports standard sFlow metrics from hosts. For additional background, the Velocity conference talk provides an introduction to sFlow and case study from a large social networking site.

It is possible to simply convert the raw sFlow metrics into InfluxDB metrics. The sflow2graphite.pl script provides an example that can be modified to support InfluxDB's native format, or used unmodified with the InfluxDB Graphite input plugin. However, there are scaleability advantages to placing the sFlow-RT analytics engine in front of the time series database. For example, in large scale cloud environments the metrics for each member of a dynamic pool isn't necessarily worth trending since virtual machines are frequently added and removed. Instead, sFlow-RT tracks all the members of the pool, calculates summary statistics for the pool, and logs the summary statistics to the time series database. This pre-processing can significantly reduce storage requirements, reducing costs and increasing query performance. The sFlow-RT analytics software also calculates traffic flow metrics, hot/missed Memcache keys, top URLs, exports events via syslog to Splunk, Logstash etc. and provides access to detailed metrics through its REST API. 

First install InfluxDB - in this case the software has been installed on host 10.0.0.30.

Next install sFlow-RT:

wget http://www.inmon.com/products/sFlow-RT/sflow-rt.tar.gz
tar -xvzf sflow-rt.tar.gz
cd sflow-rt

Edit the init.js script and add the following lines (modifying the dbURL to send metrics to the InfluxDB instance):

var dbURL = "http://10.0.0.30:8086/db/inmon/series?u=root&p=root";

setIntervalHandler(function() {
  var metrics = ['min:load_one','q1:load_one','med:load_one',
                 'q3:load_one','max:load_one'];
  var vals = metric('ALL',metrics,{os_name:['linux']});
  var body = [];
  for each (var val in vals) {
     body.push({name:val.metricName,columns:['val'],points:[[val.metricValue]]});
  }
  http(dbURL,'post', 'application/json', JSON.stringify(body));
} , 15);

Now start sFlow-RT:

./start.sh

The script makes an sFlow-RT metrics() query every 15 seconds and posts the results to InfluxDB.

The screen capture shows InfluxDB's SQL like query language and a basic query demonstrating that the metrics are being logged in the database. However, the web interface is rudimentary and a dashboard builder simplifies querying and presentation of the time series data.

Grafana is a powerful HTML 5 dashboard building tool that supports InfluxDB, Graphite, and OpenTSDB.

The screen shot shows the Grafana query builder, offering simple drop down menus that make it easy to build complex charts. The resulting chart, shown below, can be combined with additional charts to build a custom dashboard.

The sFlow standard delivers the comprehensive instrumentation of data center infrastructure and is easily integrated with DevOps tools - see Visibility and the software defined data center

Update January 31, 2016:

The InfluxDB REST API changed with version 0.9 and the above sFlow-RT script will no longer work. The new API is described in Creating a database using the HTTP API. The following version of the script has been updated to use the new API:

var dbURL = "http://10.0.0.30:8086/write?db=mydb";

setIntervalHandler(function() {
  var metrics = ['min:load_one','q1:load_one','med:load_one',
                 'q3:load_one','max:load_one'];
  var vals = metric('ALL',metrics,{os_name:['linux']});
  var body = [];
  for each (var val in vals) {
     body.push(val.metricName.replace(/[^a-zA-Z0-9_]/g,'_') + ' value=' + val.metricValue);
  }
  try { http(dbURL,'post', 'text/plain', body.join('\n')); }
  catch(e) { logWarning('http error ' + e); }
} , 15);

Update April 27, 2016

The sFlow-RT software no longer ships with an init.js file.

Instead, create an influxdb.js file in the sFlow-RT home directory and add the JavaScript code. Next, edit the start.sh file to add a script.file=influxdb.js option, i.e.

RT_OPTS="-Dscript.file=influxdb.js -Dsflow.port=6343 -Dhttp.port=8008"

The script should be loaded when sFlow-RT is started.

Metric export to Graphite

Figure 1: Cluster performance metrics

Cluster performance metrics describes how sFlow-RT can be used to calculate summary metrics for cluster performance. The article includes a Python script that polls sFlow-RT's REST API and then sends metrics to to Graphite. In this article sFlow-RT's internal scripting API will be used to send metrics directly to Graphite.

Figure 2: Components of sFlow-RT

The following script (graphite.js) re-implements the Python example (generating a sum of the load_one metric for a cluster of Linux machines) in JavaScript using sFlow-RT built-in functions for retrieving metrics and sending them to Graphite:

// author: Peter
// version: 1.0
// date: 11/23/2013
// description: Log metrics to Graphite

include('extras/json2.js');

var graphiteServer = "10.0.0.151";
var graphitePort = null;

var errors = 0;
var sent = 0;
var lastError;

setIntervalHandler(function() {
  var names = ['sum:load_one'];
  var prefix = 'linux.';
  var vals = metric('ALL',names,{os_name:['linux']});
  var metrics = {};
  for(var i = 0; i < names.length; i++) {
    metrics[prefix + names[i]] = vals[i].metricValue;
  }
  try { 
    graphite(graphiteServer,graphitePort,metrics);
    sent++;
  } catch(e) {
    errors++;
    lastError = e.message;
  }
} , 15);

setHttpHandler(function() {
  var message = { 'errors':errors,'sent':sent };
  if(lastError) message.lastError = lastError;
  return JSON.stringify(message);
});

The interval handler function runs every 15 seconds and retrieves the set of metrics in the names array (in this case just one metrics, but multiple metrics could be retrieved). The names are then converted into a Graphite friendly form (prefixing each metric with the token linux. so that they can be easily grouped) and then sent to the Graphite collector running on 10.0.0.151 using the default TCP port 2003. The script also keeps track of any errors and makes them available through the URL /script/graphite.js/json

The following command line argument loads the script on startup:

-Dscript.file=graphite.js

The following Graphite screen capture below shows a trend of the metric:

There are a virtually infinite number of core and derived metrics that can be collected by sFlow-RT using standard sFlow instrumentation embedded in switches, servers and applications throughout the data center. For example Packet loss describes the importance of collecting network packet loss metrics and including them in performance dashboards.

Figure 2: Visibility and the software defined data center

While having access to all these metrics is extremely useful, not all of them need to be stored in Graphite. Using sFlow-RT to calculate and selectively export high value metrics reduces pressure on the time series database, while still allowing any of the remaining metrics to be polled using the REST API when needed.

Finally, metrics export is only one of many applications for sFlow data, some of which have been described on this blog. The data center wide visibility provided by sFlow-RT supports orchestration tools and allows them to automatically optimize the allocation of compute, storage and application resources and the placement of loads on these resources.

Cluster performance metrics

The diagram shows three clusters of servers in a data center (e.g. web, Memcache, Hadoop, or application cluster). Each cluster may consist of hundreds of individual servers. Host sFlow agents on the physical or virtual hosts efficiently export standard metrics that can be processed by a wide range of performance analysis tools, including: Graphite, Ganglia, sFlowTrend, log analyzers, etc.

While many tools are capable of consuming sFlow metrics directly, not all tools have the scaleability to handle large clusters, or the ability to calculate summary statistics characterizing cluster performance (rather than simply reporting on each member of the cluster).

This article describes how the sFlow-RT analytics engine is used to collect sFlow from large clusters and report on overall cluster health. The diagram illustrates how sFlow-RT is deployed, consuming sFlow measurements from the servers and making summary statistics available through its REST API so that they can be used to populate a performance dashboard like Graphite.

The following summary statistics are supported by sFlow-RT:

• max: Largest value
• min: Smallest value
• sum: Total value
• avg: Average value
• var: Variance
• sdev: Standard deviation
• med: Median value
• q1: First quartile
• q2: Second quartile (same as med:)
• q3: Third quartile
• iqr: Inter-quartile range (i.e. q3 - q1)

Any programming language capable of making HTTP requests (Perl, Python, Java, Javascript, bash, etc.) can be used to retrieve metrics from sFlow-RT using the URL:

http://server:8008/metric/agents/metrics/json?filter

Where:

• server The host running sFlow-RT
• agents A semicolon separated list of host addresses or names, or ALL to include all hosts.
• metrics A comma separated list of metrics to retrieve.
• filter A filter to further restrict the hosts to include in the query.

Use of the API is best illustrated by a few examples:

http://localhost:8008/metric/web1;web2;web3/avg:load_one,avg:http_method_get/json

produces the results:

[
 {
  "metricN": 3,
  "metricName": "avg:load_one",
  "metricValue": 0.10105350773249354,
  "updateTime": 1360040842222
 },
 {
  "metricN": 3,
  "metricName": "avg:http_method_get",
  "metricValue": 54.015954359255026,
  "updateTime": 1360040842721
 }
]

The following query uses a filter to select servers whose hostname starts with the prefix "mem":

http://localhost:8008/metric/ALL/med:bytes_out,iqr:bytes_out/json?host_name=mem*

The following Python script polls sFlow-RT for cluster statistics every 60 seconds and posts the results to a Graphite collector (10.0.0.151):

import requests
import json
import time
import socket

sock = socket.socket()
sock.connect(("10.0.0.151",2003))

url = 'http://localhost:8008/metric/ALL/sum:load_one/json'
while 1 == 1:
  r = requests.get(url)
  if r.status_code != 200: break
  vals = r.json()
  if len(vals) == 0: continue
  for v in vals:
    mname  = v["metricName"]
    mvalue = v["metricValue"]
    mtime  = v["updateTime"] / 1000
    message = 'clusterB.%s %f %i\n' % (mname,mvalue,mtime)
    sock.sendall(message)
    time.sleep(60)

Finally, sFlow (and sFlow-RT) is not limited to monitoring server metrics. The switches connecting the servers in the clusters can also be monitored (the sFlow standard is supported by most switch vendors). The network can quickly become a bottleneck as cluster size increases and it is important to track metrics such as link utilization, packet discards etc. that can result in severe performance degradation. In addition, sFlow instrumentation is available for Apache, NGINX, Java, Memcached and custom applications - providing details such as URLs, response times, status codes, etc., and tying application, server and network performance together to provide a comprehensive view of performance.

Using Ganglia to monitor GPU performance

The Ganglia charts show GPU health and performance metrics collected using sFlow, see GPU performance monitoring. The combination of Ganglia and sFlow provides a highly scaleable solution for monitoring the performance of large GPU based compute clusters, eliminating the need to poll for GPU metrics. Instead, all the host and GPU metrics are efficiently pushed directly to the central Ganglia collector.

The screen capture shows the new GPU metrics, including:

• Processes
• GPU Utilization
• Memory R/W Utilization
• ECC Errors
• Power
• Temperature

The article, Ganglia 3.2 released, describes the basic steps needed to configure Ganglia as an sFlow collector. Once configured, Ganglia will automatically discover and track new servers as they are added to the network.

Note: Support for the GPU metrics is currently only available in Ganglia if you compile gmond from the latest development sources.

GPU performance monitoring

NVIDIA's Compute Unified Device Architecture (CUDA™) dramatically increases computing performance by harnessing the power of the graphics processing unit (GPU). Recently, NVIDIA published the sFlow NVML GPU Structures specification, defining a standard set of metrics for reporting GPU health and performance, and extended the Host sFlow agent to export the GPU metrics.

The following displays the sFlow metrics using sflowtool, the GPU metrics are highlighted:

[pp@test] /usr/local/bin/sflowtool
startDatagram =================================
datagramSourceIP 10.0.0.150
datagramSize 512
unixSecondsUTC 1346360234
datagramVersion 5
agentSubId 100000
agent 10.0.0.150
packetSequenceNo 1
sysUpTime 3000
samplesInPacket 1
startSample ----------------------
sampleType_tag 0:2
sampleType COUNTERSSAMPLE
sampleSequenceNo 1
sourceId 2:1
counterBlock_tag 0:2001
adaptor_0_ifIndex 1
adaptor_0_MACs 1
adaptor_0_MAC_0 000000000000
adaptor_1_ifIndex 2
adaptor_1_MACs 1
adaptor_1_MAC_0 e0cb4e98f891
adaptor_2_ifIndex 3
adaptor_2_MACs 1
adaptor_2_MAC_0 e0cb4e98f890
counterBlock_tag 0:2005
disk_total 145102770176
disk_free 46691696640
disk_partition_max_used 76.06
disk_reads 477615
disk_bytes_read 13102692352
disk_read_time 2227298
disk_writes 2370522
disk_bytes_written 193176428544
disk_write_time 445531146
counterBlock_tag 0:2004
mem_total 12618829824
mem_free 2484174848
mem_shared 0
mem_buffers 971259904
mem_cached 8214761472
swap_total 12580810752
swap_free 12580810752
page_in 6400433
page_out 94324428
swap_in 0
swap_out 0
counterBlock_tag 5703:1
nvml_device_count 1
nvml_processes 0
nvml_gpu_mS 0
nvml_mem_mS 0
nvml_mem_bytes_total 6441598976
nvml_mem_bytes_free 6429614080
nvml_ecc_errors 0
nvml_energy_mJ 74569
nvml_temperature_C 54
nvml_fan_speed_pc 30
counterBlock_tag 0:2003
cpu_load_one 0.040
cpu_load_five 0.240
cpu_load_fifteen 0.350
cpu_proc_run 0
cpu_proc_total 229
cpu_num 8
cpu_speed 1600
cpu_uptime 896187
cpu_user 21731800
cpu_nice 120230
cpu_system 5686620
cpu_idle 2844149774
cpu_wio 2992230
cpuintr 570
cpu_sintr 222180
cpuinterrupts 166594944
cpu_contexts 266986130
counterBlock_tag 0:2006
nio_bytes_in 0
nio_pkts_in 0
nio_errs_in 0
nio_drops_in 0
nio_bytes_out 0
nio_pkts_out 0
nio_errs_out 0
nio_drops_out 0
counterBlock_tag 0:2000
hostname test0
UUID 00000000000000000000000000000000
machine_type 3
os_name 2
os_release 2.6.35.14-106.fc14.x86_64
endSample   ----------------------
endDatagram   =================================

Note: Currently only the Linux version of Host sFlow includes the GPU support and the agent needs to be compiled from sources on a system that includes the NVML library.

The inclusion of GPU metrics in Host sFlow offers an extremely scaleable, lightweight solution for monitoring compute cluster performance. In addition to exporting a comprehensive set of standard performance metrics, the Host sFlow agent also offers a convenient API for exporting custom application metrics.

The sFlow standard isn't limited to monitoring compute resources; most network switch vendors include sFlow support, providing detailed visibility into cluster communication patterns and network utilization. Combining sFlow from switches, servers and applications delivers a comprehensive view of cluster performance.

Hadoop

Hadoop is an open source software for storing and processing large data sets using a cluster of commodity servers. This article shows how sFlow agents can be installed to monitor the performance of a Hadoop cluster.

Note: This example is based on the commercially Hadoop distribution from Cloudera. The virtual machine downloads from Cloudera are a convenient way to experiment with Hadoop - providing a fully functional Hadoop system on a single virtual machine.

First, install Host sFlow on each node in the Hadoop cluster, see Installing Host sFlow on a Linux server. The Host sFlow agents export standard CPU, memory, disk and network statistics from the servers. In this case, the Host sFlow agent was installed on the Cloudera virtual machine.

Since Hadoop is implemented in Java, the next step involves installing and configuring the sFlow java agent, see Java virtual machine. In this case, the Hadoop java virtual machines are configured in the /etc/hadoop/conf/hadoop-env.sh file. The following fragment from hadoop-env.sh highlights the additional java command line options needed to implement sFlow monitoring of each Hadoop daemon:

# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-javaagent:/etc/hadoop/conf/sflowagent.jar -Dsflow.
hostname=hadoop.namenode -Dsflow.dsindex=50070 -Dcom.sun.management.jmxremote $H
ADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-javaagent:/etc/hadoop/conf/sflowagent.jar
 -Dsflow.hostname=hadoop.secondarynamenode -Dsflow.dsindex=50090 -Dcom.sun.manag
ement.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-javaagent:/etc/hadoop/conf/sflowagent.jar -Dsflow.
hostname=hadoop.datanode -Dsflow.dsindex=50075 -Dcom.sun.management.jmxremote $H
ADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS="-javaagent:/etc/hadoop/conf/sflowagent.jar -Dsflow.
hostname=hadoop.balancer -Dsflow.dsindex=50060 -Dcom.sun.management.jmxremote $H
ADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS="-javaagent:/etc/hadoop/conf/sflowagent.jar -Dsflo
w.hostname=hadoop.jobtracker -Dsflow.dsindex=50030 -Dcom.sun.management.jmxremote
 $HADOOP_JOBTRACKER_OPTS"

Note: The sflowagent.jar file was placed in the /etc/hadoop/conf directory. Each daemon was given a descriptive sflow.hostname value corresponding to the daemon name. In addition, a unique sflow.dsindex value must be assigned to each daemon - the index values only need to be unique within a single server - all the servers in the cluster can share the same configuration settings. In this case the port that each daemon listens on is used as the index.

The Host sFlow and Java sFlow agents share configuration, see Host sFlow distributed agent, simplifying the task of configuring sFlow monitoring within each server and across the cluster.

Note: If you are using Ganglia monitor performance of the cluster, then you need to set multiple_jvm_instances = yes since there is more than one Java daemon running on each server, see Using Ganglia to monitor Java virtual machines.

Finally, Hadoop is sensitive to network congestion and generates large traffic volumes. Fortunately, most switch vendors support the sFlow standard. Combining sFlow from the Hadoop servers with sFlow from the switches offers comprehensive visibility into the performance of a Hadoop cluster.

Using Ganglia to monitor virtual machine pools

The Ganglia charts show virtual machine performance metrics collected using sFlow. Enabling sFlow monitoring on each node in a virtual machine pool provides a highly scalable solution for monitoring performance. Embedded sFlow monitoring in the hypervisors simplifies deployments by eliminating the need to poll for metrics. Instead, virtual machine metrics are pushed directly from each node to the central Ganglia collector. Currently sFlow agents are available for XCP (Xen Cloud Platform), Citrix XenServer and KVM/libvirt virtualization platforms, see http://host-sflow.sourceforge.net/. In addition, an sFlow agent has been demonstrated for the upcoming Windows 8 version of Hyper-V.

The article, Ganglia 3.2 released, describes the basic steps needed to configure Ganglia as an sFlow collector. Once configured, Ganglia will automatically discover and track new servers and virtual machines as they are added to the network.

Note: To try out Ganglia's sFlow/virtual machine reporting, you will need to download Ganglia 3.3.

By default, Ganglia will automatically start displaying the virtual machine metrics. However, there is an optional configuration setting available in the gmond.conf file that can be used to modify how Ganglia handles the sFlow virtual machine metrics.

sflow{
  accept_vm_metrics = yes
}

Setting the accept_vm_metrics flag to no will cause Ganglia to ignore sFlow virtualization metrics.

Ganglia and sFlow offers a comprehensive view of the performance of a virtual machine pools, providing not just virtualization related metrics, but also the server CPU, memory, disk and network IO performance metrics needed to fully characterize pool performance.

Note: Visibility into network performance is an essential part of managing a virtual machine pool since virtual machines rely on the network for storage, local communication and Internet access. Network performance also affects other critical pool operations like virtual machine migration and backup. Support for the sFlow standard by most switch vendors delivers the necessary end to end network visibility and further simplifies management by including the network in an integrated monitoring solution.

Using Ganglia to monitor Memcache clusters

The Ganglia charts show Memcache performance metrics collected using sFlow. Enabling sFlow monitoring in Memcache servers provides a highly scalable solution for monitoring the performance of large Memcache clusters. Embedded sFlow monitoring simplifies deployments by eliminating the need to poll for metrics. Instead, metrics are pushed directly from each Memcache server to the central Ganglia collector. Currently, there is an implementation of sFlow for Memcached, see http://host-sflow.sourceforge.net/relatedlinks.php.

The article, Ganglia 3.2 released, describes the basic steps needed to configure Ganglia as an sFlow collector. Once configured, Ganglia will automatically discover and track new Memcache servers as they are added to the network.

Note: To try out Ganglia's sFlow/Memcache reporting, you will need to download Ganglia 3.3.

By default, Ganglia will automatically start displaying the Memcache metrics. However, there are two optional configuration settings available in the gmond.conf file that can be used to modify how Ganglia handles the sFlow Memcache metrics.

sflow{
  accept_memcache_metrics = no
  multiple_memcache_instances = no
}

Setting the accept_memcache_metrics flag to no will cause Ganglia to ignore sFlow Memcache metrics.

The multiple_memcache_instances setting must be set to yes in cases where there are multiple Memcache instances running on each server in the cluster. Each Memcache instance will be identified by the server port included in the title of the charts. For example, the following chart is reporting on the Memcache server listening on port 11211 on host ganglia:

Ganglia and sFlow offers a comprehensive view of the performance of a cluster of Memcache servers, providing not just Memcache related metrics, but also the server CPU, memory, disk and network IO performance metrics needed to fully characterize cluster performance.

Note: A Memcache sFlow agent does more than simply export performance counters, it also exports detailed data on Memcache operations that can be used to monitor hot keys, missed keys, top clients etc. The operation data complements the counter data displayed in Ganglia, helping to identify the root cause of problems. For example, Ganglia was showing that the Memcache miss rate was high and an examination of the transactions identified a mistyped key in the application code as the root cause. In addition, Memcache performance is critically dependent on network latency and packet loss - here again, sFlow provides the necessary visibility since most switch vendors already include support for the sFlow standard.

Using Ganglia to monitor Java virtual machines

The Ganglia charts show the standard sFlow Java virtual machine metrics. The combination of Ganglia and sFlow provides a highly scalable solution for monitoring the performance of clustered Java application servers. The sFlow Java agent for stand-along Java services, or Tomcat sFlow for web-based servlets, simplify deployments by eliminating the need to poll for metrics using a Java JMX client. Instead, metrics are pushed directly from each Java virtual machine to the central Ganglia collector.

Note: The Tomcat sFlow agent also allows Ganglia to report HTTP performance metrics.

The article, Ganglia 3.2 released, describes the basic steps needed to configure Ganglia as an sFlow collector. Once configured, Ganglia will automatically discover and track new servers as they are added to the network. The articles, Java virtual machine and Tomcat, describes the steps needed to instrument existing Java applications and Apache Tomcat servlet engines respectively. In both cases the sFlow agent is included when starting the Java virtual machine and requires minimal configuration and no change to the application code.

Note: To try out Ganglia's sFlow/Java reporting, you will need to download Ganglia 3.3.

By default, Ganglia will automatically start displaying the Java virtual machine metrics. However, there are two optional configuration settings available in the gmond.conf file that can be used to modify how Ganglia handles the sFlow Java metrics.

sflow{
  accept_jvm_metrics = yes
  multiple_jvm_instances = no
}

Setting the accept_jvm_metrics flag to no will cause Ganglia to ignore Java virtual machine metrics.

The multiple_jvm_instances setting must be set to yes in cases where there are multiple Java virtual machine instances running on each server in the cluster. Charts associated with each java virtual machine instance will be identified by a unique "hostname" included in the title of its charts. For example, the following chart is identified as being associated with the apache-tomcat java virtual machine on host xenvm4.sf.inmon.com:

Ganglia and sFlow offers a comprehensive view of the performance of a cluster of Java servers, providing not just Java related metrics, but also the server CPU, memory, disk and network IO performance metrics needed to fully characterize cluster performance.