Using a proxy to feed metrics into Ganglia

The GitHub gmond-proxy project demonstrates how a simple proxy can be used to map metrics retrieved through a REST API into Ganglia's gmond TCP protocol.

The diagram shows the elements of the Ganglia monitoring system. The Ganglia server contains runs the gmetad daemon that polls for data from gmond instances and stores time series data. Trend charts are presented through the web interface. The transparent gmond-proxy replaces a native gmond daemon and delivers metrics in response to gmetad's polling requests.

The following commands install the proxy on the sFlow collector - an Ubuntu 14.04 system that is already runnig sFlow-RT:

wget https://raw.githubusercontent.com/sflow-rt/gmond-proxy/master/gmond_proxy.py
sudo mv gmond_proxy.py /etc/init.d/
sudo chown root:root /etc/init.d/gmond_proxy.py
sudo chmod 755 /etc/init.d/gmond_proxy.py
sudo service gmond_proxy.py start
sudo update-rc.d gmond_proxy.py start

The following commands install Ganglia's gmetad collector and web user interface on the Ganglia server - an Ubuntu 14.04 system:

sudo apt-get install gmetad
sudo apt-get install ganglia-webfrontend
cp /etc/ganglia-webfrontend/apache.conf /etc/apache2/sites-enabled

Next edit the /etc/ganglia/gmetad.conf file and configure the proxy as a data source:

data_source "my cluster" sflow-rt

Restart the Apache and gmetad daemons:

sudo service gmetad restart
sudo service apache2 restart

The Ganglia web user interface, shown in the screen capture, is now available at http://server/ganglia/

Ganglia natively supports sFlow, so what are some of the benefits of using the proxy? Firstly, the proxy allows metrics to be filtered, reducing the amount of data logged and increasing the scaleability of the Ganglia collector. Secondly, sFlow-RT generates traffic flow metrics, making them available to Ganglia. Finally, Ganglia is typically used in conjunction with additional monitoring tools that can all be driven using the analytics stream generated by sFlow-RT.

The diagram above shows how the sFlow-RT analytics engine is used to deliver metrics and events to cloud based and on-site DevOps tools, see: Cloud analytics,  InfluxDB and Grafana, Metric export to Graphite, and Exporting events using syslog. There are important scaleability and cost advantages to placing the sFlow-RT analytics engine in front of metrics collection applications as shown in the diagram. For example, in large scale cloud environments the metrics for each member of a dynamic pool are not necessarily worth trending since virtual machines are frequently added and removed. Instead, sFlow-RT can be configured to track all the members of the pool, calculate summary statistics for the pool, and log summary statistics. This pre-processing can significantly reduce storage requirements, reduce costs and increase query performance.