sFlow: Ganglia

Showing posts with label Ganglia. Show all posts

Friday, December 16, 2016

Using Ganglia to monitor Linux services

The screen capture from the Ganglia monitoring tool shows metrics for services running on a Linux host. Monitoring Linux services describes how the open source Host sFlow agent has been extended to export standard Virtual Node metrics from services running under systemd. Ganglia already supports these standard metrics and the article Using Ganglia to monitor virtual machine pools describes the configuration steps needed to enable this feature.

Thursday, January 21, 2016

Podcast with Nick Buraglio and Brent Salisbury

"Have you seen sFlow options in your router configuration or flow collector? Are you looking for alternatives to SNMP or NetFlow? Have you been curious about the instrumentation of your new white box or virtual switch? Yes? Then you will probably enjoy learning more about sFlow!"

Non-Blocking #1: SFlow With Peter Phaal Of InMon And SFlow.Org is a discussion between Brent Salisbury (networkstatic.net), Nick Buraglio (forwardingplane.net), and Peter Phaal (blog.sflow.com).

Web sites and tools mentioned in the podcast:

sFlow.org
Devices that support sFlow
Software to analyze sFlow
sFlow.org mailing list
sFlow structures
blog.sflow.com (incorrectly referenced as blog.sflow.org in the podcast)
Host sFlow
sflowtool

The podcast touches on a number of topics that have been explored in greater detail on this blog. The topics are listed in roughly the order they are mentioned in the podcast:

Tuesday, December 8, 2015

Using a proxy to feed metrics into Ganglia

The GitHub gmond-proxy project demonstrates how a simple proxy can be used to map metrics retrieved through a REST API into Ganglia's gmond TCP protocol.

The diagram shows the elements of the Ganglia monitoring system. The Ganglia server contains runs the gmetad daemon that polls for data from gmond instances and stores time series data. Trend charts are presented through the web interface. The transparent gmond-proxy replaces a native gmond daemon and delivers metrics in response to gmetad's polling requests.

The following commands install the proxy on the sFlow collector - an Ubuntu 14.04 system that is already runnig sFlow-RT:

wget https://raw.githubusercontent.com/sflow-rt/gmond-proxy/master/gmond_proxy.py
sudo mv gmond_proxy.py /etc/init.d/
sudo chown root:root /etc/init.d/gmond_proxy.py
sudo chmod 755 /etc/init.d/gmond_proxy.py
sudo service gmond_proxy.py start
sudo update-rc.d gmond_proxy.py start

The following commands install Ganglia's gmetad collector and web user interface on the Ganglia server - an Ubuntu 14.04 system:

sudo apt-get install gmetad
sudo apt-get install ganglia-webfrontend
cp /etc/ganglia-webfrontend/apache.conf /etc/apache2/sites-enabled

Next edit the /etc/ganglia/gmetad.conf file and configure the proxy as a data source:

data_source "my cluster" sflow-rt

Restart the Apache and gmetad daemons:

sudo service gmetad restart
sudo service apache2 restart

The Ganglia web user interface, shown in the screen capture, is now available at http://server/ganglia/

Ganglia natively supports sFlow, so what are some of the benefits of using the proxy? Firstly, the proxy allows metrics to be filtered, reducing the amount of data logged and increasing the scaleability of the Ganglia collector. Secondly, sFlow-RT generates traffic flow metrics, making them available to Ganglia. Finally, Ganglia is typically used in conjunction with additional monitoring tools that can all be driven using the analytics stream generated by sFlow-RT.

The diagram above shows how the sFlow-RT analytics engine is used to deliver metrics and events to cloud based and on-site DevOps tools, see: Cloud analytics, InfluxDB and Grafana, Metric export to Graphite, and Exporting events using syslog. There are important scaleability and cost advantages to placing the sFlow-RT analytics engine in front of metrics collection applications as shown in the diagram. For example, in large scale cloud environments the metrics for each member of a dynamic pool are not necessarily worth trending since virtual machines are frequently added and removed. Instead, sFlow-RT can be configured to track all the members of the pool, calculate summary statistics for the pool, and log summary statistics. This pre-processing can significantly reduce storage requirements, reduce costs and increase query performance.

Thursday, February 26, 2015

Broadcom ASIC table utilization metrics, DevOps, and SDN

Figure 1: Two-Level Folded CLOS Network Topology Example

Figure 1 from the Broadcom white paper, Engineered Elephant Flows for Boosting Application Performance in Large-Scale CLOS Networks, shows a data center leaf and spine topology. Leaf and spine networks are seeing rapid adoption since they provide the scaleability needed to cost effectively deliver the low latency, high bandwidth interconnect for cloud, big data, and high performance computing workloads.

Broadcom Trident ASICs are popular in white box, brite-box and branded data center switches from a wide range of vendors, including: Accton, Agema, Alcatel-Lucent, Arista, Cisco, Dell, Edge-Core, Extreme, Hewlett-Packard, IBM, Juniper, Penguin Computing, and Quanta.

Figure 2: OF-DPA Programming Pipeline for ECMP

Figure 2 shows the packet processing pipeline of a Broadcom ASIC. The pipeline consists of a number of linked hardware tables providing bridging, routing, access control list (ACL), and ECMP forwarding group functions. Operations teams need to be able to proactively monitor table utilizations in order to avoid performance problems associated with table exhaustion.

Broadcom's recently released sFlow specification, sFlow Broadcom Switch ASIC Table Utilization Structures, leverages the industry standard sFlow protocol to offer scaleable, multi-vendor, network wide visibility into the utilization of these hardware tables.

Support for the new extension has just been added to the open source Host sFlow agent, which runs on Cumulus Linux, a Debian based Linux distribution that supports open switch hardware from Agema, Dell, Edge-Core, Penguin Computing, Quanta. Hewlett-Packard recently announced that they will soon be selling a new line of open network switches built by Accton Technologies and supporting Cumulus Linux.

The speed with which this new features can be delivered on hardware from the wide range of vendors supporting Cumulus Linux is a powerful illustration of the power of open networking. While support for the Broadcom ASIC table extension has been checking into the Host sFlow trunk it hasn't yet made it into the Cumulus Networks binary repositories. However, Cumulus Linux is an open platform, so users are free to download sources, compile and install the latest software version direct from SourceForge.

The following output from the open source sflowtool command line utility shows the raw table measurements (this is in addition to the extensive set of sFlow measurements already exported via sFlow on Cumulus Linux):

bcm_asic_host_entries 4
bcm_host_entries_max 8192
bcm_ipv4_entries 0
bcm_ipv4_entries_max 0
bcm_ipv6_entries 0
bcm_ipv6_entries_max 0
bcm_ipv4_ipv6_entries 9
bcm_ipv4_ipv6_entries_max 16284
bcm_long_ipv6_entries 3
bcm_long_ipv6_entries_max 256
bcm_total_routes 10
bcm_total_routes_max 32768
bcm_ecmp_nexthops 0
bcm_ecmp_nexthops_max 2016
bcm_mac_entries 3
bcm_mac_entries_max 32768
bcm_ipv4_neighbors 4
bcm_ipv6_neighbors 0
bcm_ipv4_routes 0
bcm_ipv6_routes 0
bcm_acl_ingress_entries 842
bcm_acl_ingress_entries_max 4096
bcm_acl_ingress_counters 68
bcm_acl_ingress_counters_max 4096
bcm_acl_ingress_meters 18
bcm_acl_ingress_meters_max 8192
bcm_acl_ingress_slices 3
bcm_acl_ingress_slices_max 8
bcm_acl_egress_entries 36
bcm_acl_egress_entries_max 512
bcm_acl_egress_counters 36
bcm_acl_egress_counters_max 1024
bcm_acl_egress_meters 18
bcm_acl_egress_meters_max 512
bcm_acl_egress_slices 2
bcm_acl_egress_slices_max 2

The sflowtool output is useful for troubleshooting and is easy to parse with scripts.

DevOps

The diagram shows how the sFlow-RT analytics engine is used to deliver metrics and events to cloud based and on-site DevOps tools, see: Cloud analytics, InfluxDB and Grafana, Cloud Analytics, Metric export to Graphite, and Exporting events using syslog.

For example, the following sFlow-RT application simplifies monitoring of the leaf and spine network by combining measurements from all the switches, identifying the switch with the maximum utilization of each table, pushing the summaries to operations dashboard every 15 seconds, and sending syslog events immediately when any table exceeds 80% utilization:

var network_wide_metrics = [
  'max:bcm_host_utilization',
  'max:bcm_mac_utilization',
  'max:bcm_ipv4_ipv6_utilization',
  'max:bcm_total_routes_utilization',
  'max:bcm_ecmp_nexthops_utilization',
  'max:bcm_acl_ingress_utilization',
  'max:bcm_acl_ingress_meters_utilization',
  'max:bcm_acl_ingress_counters_utilization',
  'max:bcm_acl_egress_utilization',
  'max:bcm_acl_egress_meters_utilization',
  'max:bcm_acl_egress_counters_utilization'
];

var max_utilization = 80;

setIntervalHandler(function() {
  var vals = metric('ALL',network_wide_metrics);
  var graphite_metrics = {};
  for each (var val in vals) {
    if(!val.hasOwnProperty('metricValue')) continue;

    // generate syslog events for over utilized tables
    if(val.metricValue >= max_utilization) {
       var event = {
         "asic_table":val.metricName,
         "utilization":val.metricValue,
         "switchIP":val.agent
       };
       try {
         syslog(
           '10.0.0.1', // syslog collector: splunk>, logstash, etc.
           514,        // syslog port
           16,         // facility = local0
           5,          // severity = notice
           event
        );
      } catch(e) { logWarning("syslog() failed " + e); }
    }

    // add metric to graphite set
    graphite_metrics["network.podA."+val.metricName] = val.metricValue;
  }

  // sent metrics to graphite
  try {
    graphite(
      '10.0.0.151',  // graphite server
      2003,          // graphite carbon UDP port
      graphite_metrics
    );
  } catch(e) { logWarning("graphite() failed " + e); }
},15);

The following screen capture shows the graphs starting to appear in Graphite:

Real-time traffic analytics

The table utilization metrics are only a part of the visibility that sFlow provides into the performance of a leaf and spine network.

A leaf and spine fabric is challenging to monitor. The fabric spreads traffic across all the switches and links in order to maximize bandwidth. Unlike traditional hierarchical network designs, where a small number of links can be monitored to provide visibility, a leaf and spine network has no special links or switches where running CLI commands or attaching a probe would provide visibility. Even if it were possible to attach probes, the effective bandwidth of a leaf and spine network can be as high as a Petabit/second, well beyond the capabilities of current generation monitoring tools.

Scaleable traffic measurement is possible because Broadcom ASICs implement hardware support for sFlow monitoring, providing cost effective, line rate visibility that is build into the switches and scales to all port speeds (1G, 10G, 25G, 40G, 50G, 100G, ...) and the high port counts found in large leaf and spine networks.

The 2 minute video provides an overview of some of the performance challenges with leaf and spine fabrics and demonstrates Fabric View - a monitoring solution that leverages industry standard sFlow instrumentation in commodity data center switches to provide real-time visibility into fabric performance. Fabric visibility with Cumulus Linux describes how to set up Fabric View to monitor a Cumulus Linux leaf and spine network.

SDN

Real-time network analytics are a fundamental driver for a number of important SDN use cases, allowing the SDN controller to rapidly detect changes in traffic and respond by applying active controls. SDN fabric controller for commodity data center switches describes how control of the ACL table is the key feature needed to to build scaleable SDN solutions.

REST API for Cumulus Linux ACLs describes open source software to allow an SDN controller to centrally manage the ACL tables on a large scale network of switches running Cumulus Linux.

The ability to install software on the switches is transformative, allowing third party developers and network operators transparent access to the full capabilities of the switch and build solutions that efficiently handle automation challenges.

A number of SDN use cases have been demonstrated that build on Cumulus Linux to leverage the real-time visibility and control capabilities of the switch ASIC:

Visit the sFlow.com web site to learn more about SDN control of leaf and spine networks.

Finally, the SDN use cases make extensive use of the ACL table and so this brings us full circle to the importance of the Broadcom sFlow extension providing visibility into the utilization of table resources.

Friday, October 19, 2012

Using Ganglia to monitor GPU performance

The Ganglia charts show GPU health and performance metrics collected using sFlow, see GPU performance monitoring. The combination of Ganglia and sFlow provides a highly scaleable solution for monitoring the performance of large GPU based compute clusters, eliminating the need to poll for GPU metrics. Instead, all the host and GPU metrics are efficiently pushed directly to the central Ganglia collector.

The screen capture shows the new GPU metrics, including:

Processes
GPU Utilization
Memory R/W Utilization
ECC Errors
Power
Temperature

The article, Ganglia 3.2 released, describes the basic steps needed to configure Ganglia as an sFlow collector. Once configured, Ganglia will automatically discover and track new servers as they are added to the network.

Note: Support for the GPU metrics is currently only available in Ganglia if you compile gmond from the latest development sources.

Saturday, October 6, 2012

Thread pools

Figure 1: Thread pool

The thread pool pattern, illustrated in figure 1, is common to many parallel processing applications. A number of worker threads, organized in a thread pool, take tasks from a task queue. Once a thread has completed a task, it waits for a new task to appear on the task queue. Keeping track of the number of active threads in the pool is essential. If tasks wait in the queue because there aren't enough workers then requests will be delayed, or possibly dropped if the queue fills up.

The recently finalized sFlow Application Structures specification defines a standard set of metrics for reporting on thread pools:

Active Threads The number of threads in the thread pool that are actively processing a request.
Idle Threads The number of threads in the thread pool that are waiting for a request.
Maximum Threads The maximum number of threads that can exist in the thread pool.
Delayed Tasks The number of tasks that could not be served immediately, but spent time in the task queue.
Dropped Tasks The number of tasks that were dropped because the task queue was full.

The Apache web server uses a thread pool and is a useful demonstration of the value of the sFlow thread pool metrics. The Apache thread pool can be accessed using mod_status, which makes the thread pool visible as a web page. The following screen capture shows the server-status page generated by mod_status:

The grid of characters is used to visualize the the state of the pool (referred to as the "scoreboard"), each cell in the grid represents a slot for a thread and the size of the grid shows the maximum number of threads that are permitted in the pool. The summary line above the grid states that 6 requests are currently being processed and that there are 69 idle workers (i.e. there are six "W" characters and sixty nine "_" characters in the grid).

While the server-status page isn't designed to be machine readable, the information is critical and there are numerous performance monitoring tools that make HTTP requests and extract the worker pool statistics from the text. A much more efficient way to retrieve the information is to use the Apache sFlow module, which in addition to reporting the thread pool statistics will export HTTP counters, URLs, response times, status codes, etc.

The article, Using Ganglia to monitor web farms, describes how to use the open source Ganglia performance monitoring software to collect and report on web server clusters using sFlow. Ganglia now includes support for the sFlow thread pool metrics.

Figure 2: Ganglia chart showing active threads from an Apache web server

Figure 2 trends the number of active workers in the pool. If the number of active workers approaches the maximum allowed, then additional servers may need to be added to the cluster. An increase in active threads could also indicate a performance problem with backend systems (a slow database holding up worker threads) or may be the result of a Denial of Service (DoS) attack (e.g. Sloloris).

Monitoring thread pools using sFlow is very useful, but only scratches the surface of what is possible. The sFlow standard is widely support be network equipment vendors and can be combined with sFlow metrics from hosts, services and applications to provide a comprehensive view of data center performance.

Wednesday, February 1, 2012

Ganglia 3.3 released

Ganglia 3.2 was the first release to include native sFlow support. The latest Ganglia 3.3 release includes a new web user interface and adds support for additional sFlow metrics:

The Host sFlow distributed agent efficiently exports metrics from Windows, Linux and FreeBSD servers as well as Hyper-V, XenServer, XCP and Xen hypervisors. Additional sFlow agents are available for Java, Apache, Tomcat, NGINX, node.js and Memcached.

Sunday, January 1, 2012

Using Ganglia to monitor virtual machine pools

The Ganglia charts show virtual machine performance metrics collected using sFlow. Enabling sFlow monitoring on each node in a virtual machine pool provides a highly scalable solution for monitoring performance. Embedded sFlow monitoring in the hypervisors simplifies deployments by eliminating the need to poll for metrics. Instead, virtual machine metrics are pushed directly from each node to the central Ganglia collector. Currently sFlow agents are available for XCP (Xen Cloud Platform), Citrix XenServer and KVM/libvirt virtualization platforms, see http://host-sflow.sourceforge.net/. In addition, an sFlow agent has been demonstrated for the upcoming Windows 8 version of Hyper-V.

The article, Ganglia 3.2 released, describes the basic steps needed to configure Ganglia as an sFlow collector. Once configured, Ganglia will automatically discover and track new servers and virtual machines as they are added to the network.

Note: To try out Ganglia's sFlow/virtual machine reporting, you will need to download Ganglia 3.3.

By default, Ganglia will automatically start displaying the virtual machine metrics. However, there is an optional configuration setting available in the gmond.conf file that can be used to modify how Ganglia handles the sFlow virtual machine metrics.

sflow{
  accept_vm_metrics = yes
}

Setting the accept_vm_metrics flag to no will cause Ganglia to ignore sFlow virtualization metrics.

Ganglia and sFlow offers a comprehensive view of the performance of a virtual machine pools, providing not just virtualization related metrics, but also the server CPU, memory, disk and network IO performance metrics needed to fully characterize pool performance.

Note: Visibility into network performance is an essential part of managing a virtual machine pool since virtual machines rely on the network for storage, local communication and Internet access. Network performance also affects other critical pool operations like virtual machine migration and backup. Support for the sFlow standard by most switch vendors delivers the necessary end to end network visibility and further simplifies management by including the network in an integrated monitoring solution.

Friday, December 30, 2011

Using Ganglia to monitor Memcache clusters

The Ganglia charts show Memcache performance metrics collected using sFlow. Enabling sFlow monitoring in Memcache servers provides a highly scalable solution for monitoring the performance of large Memcache clusters. Embedded sFlow monitoring simplifies deployments by eliminating the need to poll for metrics. Instead, metrics are pushed directly from each Memcache server to the central Ganglia collector. Currently, there is an implementation of sFlow for Memcached, see http://host-sflow.sourceforge.net/relatedlinks.php.

The article, Ganglia 3.2 released, describes the basic steps needed to configure Ganglia as an sFlow collector. Once configured, Ganglia will automatically discover and track new Memcache servers as they are added to the network.

Note: To try out Ganglia's sFlow/Memcache reporting, you will need to download Ganglia 3.3.

By default, Ganglia will automatically start displaying the Memcache metrics. However, there are two optional configuration settings available in the gmond.conf file that can be used to modify how Ganglia handles the sFlow Memcache metrics.

sflow{
  accept_memcache_metrics = no
  multiple_memcache_instances = no
}

Setting the accept_memcache_metrics flag to no will cause Ganglia to ignore sFlow Memcache metrics.

The multiple_memcache_instances setting must be set to yes in cases where there are multiple Memcache instances running on each server in the cluster. Each Memcache instance will be identified by the server port included in the title of the charts. For example, the following chart is reporting on the Memcache server listening on port 11211 on host ganglia:

Ganglia and sFlow offers a comprehensive view of the performance of a cluster of Memcache servers, providing not just Memcache related metrics, but also the server CPU, memory, disk and network IO performance metrics needed to fully characterize cluster performance.

Note: A Memcache sFlow agent does more than simply export performance counters, it also exports detailed data on Memcache operations that can be used to monitor hot keys, missed keys, top clients etc. The operation data complements the counter data displayed in Ganglia, helping to identify the root cause of problems. For example, Ganglia was showing that the Memcache miss rate was high and an examination of the transactions identified a mistyped key in the application code as the root cause. In addition, Memcache performance is critically dependent on network latency and packet loss - here again, sFlow provides the necessary visibility since most switch vendors already include support for the sFlow standard.

Thursday, December 29, 2011

Using Ganglia to monitor Java virtual machines

The Ganglia charts show the standard sFlow Java virtual machine metrics. The combination of Ganglia and sFlow provides a highly scalable solution for monitoring the performance of clustered Java application servers. The sFlow Java agent for stand-along Java services, or Tomcat sFlow for web-based servlets, simplify deployments by eliminating the need to poll for metrics using a Java JMX client. Instead, metrics are pushed directly from each Java virtual machine to the central Ganglia collector.

Note: The Tomcat sFlow agent also allows Ganglia to report HTTP performance metrics.

The article, Ganglia 3.2 released, describes the basic steps needed to configure Ganglia as an sFlow collector. Once configured, Ganglia will automatically discover and track new servers as they are added to the network. The articles, Java virtual machine and Tomcat, describes the steps needed to instrument existing Java applications and Apache Tomcat servlet engines respectively. In both cases the sFlow agent is included when starting the Java virtual machine and requires minimal configuration and no change to the application code.

Note: To try out Ganglia's sFlow/Java reporting, you will need to download Ganglia 3.3.

By default, Ganglia will automatically start displaying the Java virtual machine metrics. However, there are two optional configuration settings available in the gmond.conf file that can be used to modify how Ganglia handles the sFlow Java metrics.

sflow{
  accept_jvm_metrics = yes
  multiple_jvm_instances = no
}

Setting the accept_jvm_metrics flag to no will cause Ganglia to ignore Java virtual machine metrics.

The multiple_jvm_instances setting must be set to yes in cases where there are multiple Java virtual machine instances running on each server in the cluster. Charts associated with each java virtual machine instance will be identified by a unique "hostname" included in the title of its charts. For example, the following chart is identified as being associated with the apache-tomcat java virtual machine on host xenvm4.sf.inmon.com:

Ganglia and sFlow offers a comprehensive view of the performance of a cluster of Java servers, providing not just Java related metrics, but also the server CPU, memory, disk and network IO performance metrics needed to fully characterize cluster performance.

Wednesday, December 28, 2011

Using Ganglia to monitor web farms

The Ganglia charts show HTTP performance metrics collected using sFlow. Enabling sFlow monitoring in web servers provides a highly scalable solution for monitoring the performance of large web farms. Embedded sFlow monitoring simplifies deployments by eliminating the need to poll for metrics or tail log files. Instead, metrics are pushed directly from each web server to the central Ganglia collector. Currently, there are implementation of sFlow for Apache, NGINX, Tomcat and node.js web servers, see http://www.sflow.net/relatedlinks.php.

The article, Ganglia 3.2 released, describes the basic steps needed to configure Ganglia as an sFlow collector. Once configured, Ganglia will automatically discover and track new web servers as they are added to the network.

Note: To try out Ganglia's sFlow/HTTP reporting, you will need to download Ganglia 3.3.

By default, Ganglia will automatically start displaying the HTTP metrics. However, there are two optional configuration settings available in the gmond.conf file that can be used to modify how Ganglia handles the sFlow HTTP metrics.

sflow{
  accept_http_metrics = yes
  multiple_http_instances = no
}

Setting the accept_http_metrics flag to no will cause Ganglia to ignore sFlow HTTP metrics.

The multiple_http_instances setting must be set to yes in cases where there are multiple HTTP instances running on each server in the cluster. Charts associated with each HTTP instance are identified by the server port included in the title of its charts. For example, the following chart is reporting on the web server listening on port 8080 on host xenvm4.sf.inmon.com:

Ganglia and sFlow provide a comprehensive view of the performance of a cluster of web servers, providing not just HTTP related metrics, but also the server CPU, memory, disk and network IO performance metrics needed to fully characterize cluster performance.

Note: An HTTP sFlow agent does more than simply export performance counters, it also exports detailed transaction data that can be used to monitor top URLs, top Referers, top clients, response times etc. The transaction data complements the counter data displayed in Ganglia, helping to identify the root cause of problems. For example, Ganglia was showing a sudden increase in HTTP requests and an examination of the transactions demonstrated that the increase was a denial of service attack, identifying the targeted URL and the list of attacker IP addresses.

Monday, July 11, 2011

Ganglia and cloud performance

The Ganglia 3.2 release includes support for collecting and displaying server performance metrics sent using the sFlow standard. Ganglia's focus has traditionally been to monitor clusters and grids, however, the scalability and automatic discovery capabilities of Ganglia also make it highly suited to monitoring pools of virtual machines.

Visibility in the cloud discusses the different challenges managing virtual machines hosted within a public cloud and management of the cloud infrastructure. The article, Rackspace cloudservers, shows how Ganglia and sFlow can be used to monitor the performance of virtual machines hosted in a public cloud. This article examines how Ganglia and sFlow can be used by service providers and private cloud operators to monitor the performance of the cloud infrastructure.

Currently sFlow agents are available for XCP (Xen Cloud Platform), Citrix XenServer and KVM/libvirt virtualization platforms. When monitoring a hypervisor using sFlow, Ganglia will display the following hypervisor specific metrics in addition to the familiar CPU, memory, disk and network statistics:

The Domain Count trends the number of virtual machines running on the server. The Hypervisor Free Memory chart shows how much free memory is available to run additional virtual machines.

In addition, sFlow also reports basic CPU, Memory, Disk I/O and Network I/O for every virtual machine running on the hypervisor without the need to install agents on the virtual machines. However, these additional statistics are currently discarded by default since the Ganglia user interface expects every server to report a common set of metrics and the data available from virtual machines is limited, resulting in missing charts. In addition, sFlow uniquely identifies virtual machines by their UUID (Universally Unique Identifier) but Ganglia currently expects hosts to be identified by IP addresses and hostnames (which may not be known for virtual machines).

Ganglia 3.2 provides an experimental override, allowing the additional per virtual machine performance metrics to be collected. The following entries in the Ganglia gmond configuration file (/etc/gmond.conf) configures sFlow monitoring and enables the additional per virtual machine performance metrics:

globals {
/* Listen, but don't send metrics */
  mute = yes
  deaf = no
  ...

/* sFlow channel */
udp_recv_channel {
  port = 6343
}

/* Enable virtual machine statistics */
sflow {
  accept_vm_metrics = yes
}

Once enabled, each virtual machine will appear as a member of the cluster. Selecting a virtual machine displays its metrics:

Note: The virtual machine metrics reported by sFlow are consistent with libvirt.

Ganglia has great potential for monitoring virtual machine pools. The experimental support for virtual machine monitoring in Ganglia 3.2 provides a starting point, laying the foundation for further development.

Thursday, July 7, 2011

Ganglia 3.2 released

The open source Ganglia Monitoring System is widely used to monitor high-performance computing systems such as clusters and Grids. The latest Ganglia 3.2 release includes native support for the sFlow standard. This article describes some of the benefits of using sFlow for cluster monitoring and describes how to configure Ganglia as an sFlow analyzer.

The diagram shows the elements of the solution. Each server sends sFlow to the Ganglia gmond process which builds an in-memory database containing the server statistics. The Ganglia gmetad process periodically queries the gmond database and updates trend charts that are made available through a web interface. The sFlow server performance data seamlessly integrates with Ganglia since the standard sFlow server metrics are based on Ganglia's core set of metrics.

Note: The metrics for all the servers in the cluster can be retrieved as an XML document by connecting to gmond (the default port is 8649). This API (used by gmetad) provides a simple way for performance monitoring tools that rely on a polling model to retrieve the sFlow metrics, for example Vladimir Vuksan's Use your trending data for alerting article describes how Nagios can use this API to retrieve metrics.

The sFlow solution is very similar to using Ganglia in a unicast configuration where gmond agents, installed on each server in the cluster, are configured to periodically send metrics to a central gmond instance that builds the database of cluster performance. The sFlow solution simply replaces the gmond agents in the cluster with sFlow agents.

Host sFlow is a free, open source, sFlow agent implementation. The Host sFlow agent reports on the performance of physical and virtual servers and currently supports Linux, FreeBSD and Windows servers as well as the Citrix XenServer, XCP (Xen Cloud Platform) and KVM/libvirt virtualization platforms.

Why use Host sFlow instead of gmond to monitor servers?

Lightweight Eliminating collector functionality reduces the size and complexity of the Host sFlow agent. The reduced overhead is particularly important when monitoring resource constrained environments like hypervisors.
Portable The Host sFlow agent has minimal software dependencies and is easily ported to different platforms, including a native Windows implementation.
Efficient The sFlow protocol efficiently packs all the standard Ganglia metrics in a single UDP datagram. Gmond requires over 32 datagrams to send the same information.
Standard The standard metrics exported by Host sFlow agents allows performance monitoring tools to share data, eliminating the need for wasteful duplication of agents.
Metrics The standard set of sFlow metrics includes the core Ganglia metrics as well as additional disk I/O, swap, interrupt and virtual machine statistics.

One of the strengths of Ganglia is the ability to easily add new metrics. While the Host sFlow agent doesn't support the addition of custom metrics, the Ganglia gmetric command line tool provides a simple way add custom metrics. For example, the following script exports the number of users currently logged into a system:

/usr/bin/gmetric --name Current_Users --value `who |wc -l` --type int32 --unit current_users

Running the command periodically using crontab allows Ganglia to track the metric. For embedded applications, the embeddedgmetric project provides C/C++, Python, PHP, Perl and Java libraries for sending Ganglia metrics.

The following entries in the gmond configuration file (/etc/gmond.conf) configures gmond to run in collector only mode, listening for sFlow data on UDP port 6343 (the standard sFlow port):

globals {
/* Listen, but don't send metrics */
  mute = yes
  deaf = no
  ...

}

/* channel to receive gmetric messages */
udp_recv_channel {
  port = 8649
}

/* channel to receive sFlow */
/* 6343 is the default sFlow port, an explicit sFlow    */
/* configuration section is needed to override default  */ 
udp_recv_channel {
  port = 6343
}

/* channel to service requests for XML data from gmetad */
tcp_accept_channel {
  port = 8649
}

Note: Delete all modules, collection_group and include sections from the gmond configuration file in this example and in the following examples since gmond is being used simply to collect sFlow metrics and doesn't need to load modules to generate metrics.

In the Ganglia architecture, each cluster is monitored by a separate gmond process. If more than one cluster is to be monitored, then it is possible to run multiple gmond processes on a single server, each with its own configuration file. For example, if Host sFlow agents on the first cluster are sending to port 6343, then Host sFlow agents on the second cluster should be configured to send to a different port, say 6344. The following settings will configure the second gmond to listen on the non-standard port.

globals {
/* Listen, but don't send metrics */
  mute = yes
  deaf = no
  ...

}

/* channel to receive gmetric messages */
udp_recv_channel {
  port = 8650
}

/* channel to receive sFlow */
udp_recv_channel {
  port = 6344
}

/* Change sFlow channel to non-standard port 6344 */
sflow {
  udp_port = 6344
}

/* channel to service requests for XML data from gmetad */
tcp_accept_channel {
  port = 8650
}

Note: The non-standard port setting is only required if both gmond processes are running on a single server. If each cluster is monitored by a separate server then the Host sFlow agents on each cluster simply need to be configured to send to the collector for their cluster.

Another alternative is to assign multiple IP addresses to the server, one per cluster. In this case Host sFlow agents in the first cluster will be configured to send to one address and agents in the second cluster to a different address. The following settings show how gmond can be configured to listen for sFlow on a specific IP address (e.g. 10.0.0.22):

globals {
/* Listen, but don't send metrics */
  mute = yes
  deaf = no
  ...

}

/* channel to receive sFlow */
udp_recv_channel {
  port = 6343
  bind = 10.0.0.22
}

The integration of network, system and application monitoring (see sFlow Host Structures) makes sFlow ideally suited for converged infrastructure monitoring. Using a single multi-vendor standard for both network and system performance monitoring reduces complexity and provides the integrated view of performance needed for effective management (see Management silos).

Sunday, January 30, 2011

Rackspace cloudservers

The article, Visibility in the cloud, provides a general discussion of how to monitor cloud infrastructure. This article uses the Rackspace cloudservers™ hosting service to provide a concrete example of implementing sFlow monitoring in a public cloud.

There are a number of APIs and tools available for managing large cloud server deployments in the Rackspace cloud. However, the web interface provides the quickest solution for setting up the small number of cloud servers used in this example:

In this example, three cloud servers have been created: two Fedora Linux servers and a Windows 2003 server. The following diagram shows the network topology that connects the cloud servers:

Each cloud server is provides with a public IP address and a private IP address. The private network is intended for inter-server communication and there are no usage charges. Bandwidth on the public network is metered and usage-based charges apply.

In this example, the sFlow analyzer has been installed on the server, Web. In order to provide sFlow monitoring, open source Host sFlow agents were installed on the Linux and Windows cloud servers. The sFlow agents were configured to send sFlow to the private address on Web (10.180.164.230).

By default, Rackspace creates Linux cloud servers with a restrictive firewall configuration. The firewall configurations were modified (changes shown in red) to implement packet sampling and allow sFlow datagrams to be received from the private network interface (eth1).

[root@web ~]# more /etc/sysconfig/iptables
# Firewall configuration written by system-config-firewall
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -m statistic --mode random --probability 0.01 -j ULOG --ulog-nlgroup 1
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -p udp --dport 6343 -i eth1 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -m statistic --mode random --probability 0.01 -j ULOG --ulog-nlgroup 1
COMMIT

Note: On Linux systems, Host sFlow uses the iptables ULOG facility to monitor network traffic, see ULOG for a more detailed discussion.

The Host sFlow agents were configured to poll counters every 30 seconds and pick up the packet samples via ULOG, sending the resulting sFlow to collector, 10.180.164.230:

[root@web ~]# more /etc/hsflowd.conf 
sflow {
  DNSSD = off
  polling = 30
  sampling = 400

  collector {
    ip = 10.180.164.230
  }

  ulogGroup = 1
  ulogProbability = 0.01
}

Two sFlow analyzers were installed on cloud servers in order to demonstrate different aspects of sFlow analysis: the open source Ganglia cluster monitoring application and the commercial Traffic Sentinel application from InMon Corp. Both applications are easily installed on a Linux cloud server. Both tools also provide web-based interfaces, making them well suited to cloud deployment.

An advantage of using the sFlow standard for server monitoring is that it provides a multi-vendor solution. Windows and Linux servers export standard metrics that link network and system performance and allow a wide variety of analysis applications to be used.

The following web browser screen shot shows Ganglia displaying the performance of the cloud servers:

The charts present a cluster-wide view of performance, with statistics combined from all the servers.

Drilling down to an individual server provides a detailed view of the server's performance:

Traffic Sentinel provides similar functionality when presenting server performance. The following screen shows a cluster-wide view of performance:

In addition, the top servers page, shown below, provides a real-time view comparing the performance of the busiest servers in the cluster.

The sFlow standard originated as a way to monitor network performance and is supported by most switch vendors. The following chart demonstrates some of the visibility into network traffic available using sFlow:

The chart shows a protocol breakdown of the network traffic to the cloud servers. For a more detailed view, the following application map shows how network monitoring can be used to track the complex relationships between the cloud servers:

In addition to monitoring server and network performance, sFlow can also be used to monitor performance of the scale-out applications that are typically deployed in the cloud, including: web farms, memcached and membase clusters.

The sFlow standard is extremely well suited for cloud performance monitoring. The scalability of sFlow allows tens of thousands of cloud servers to be centrally monitored. With sFlow, data is continuously sent from the cloud servers to the sFlow analyzer, providing a real-time view of performance across the cloud.

The sFlow push model is much more efficient than typical monitoring architectures that require the management system to periodically poll servers for statistics. Polling breaks down in highly dynamic cloud environments where servers can appear and disappear. With sFlow, cloud servers are automatically discovered and continuously monitored as soon as they are created. The sFlow messages act as a server heartbeat, providing rapid notification when a server is deleted and stops sending sFlow.

Finally, sFlow provides the detailed, real-time, visibility into network, server and application performance needed to manage performance and control costs. For anyone interested in more information on sFlow, the sFlow presentation provides a strategic view of the role that sFlow monitoring plays in converged, virtualized and cloud environments.

Monday, October 25, 2010

Ganglia

The open source Ganglia Monitoring System is widely used to monitor high-performance computing systems such as clusters and Grids. The recent addition of sFlow support makes Ganglia an attractive option for monitoring servers in cloud computing environments (see Cloud-scale performance monitoring).

/* sFlow channel */
udp_recv_channel {
  port = 6343
}