Comparing sFlow and NetFlow in a vSwitch

As virtualization shifts the network edge from top of rack switches to software virtual switches running on the hypervisors; visibility in the virtual switching layer is essential in order to provide network, server and storage management teams with the information needed to coordinate resources and ensure optimal performance.

The recent release of Citrix XenServer 6.0 provides an opportunity for a side-by-side comparison of sFlow and NetFlow monitoring technologies since both protocols are supported by the Open vSwitch that is now the default XenServer network stack.

The diagram above shows the experimental setup. Traffic between the virtual machines VM1 and VM2 passes through the Virtual Switch where sFlow and NetFlow measurements are simultaneously generated. The sFlow is sent to an sFlow Analyzer (InMon sFlowTrend) and the NetFlow to a NetFlow Analyzer (SolarWinds Real-Time NetFlow Analyzer). Both tools are running in tandem making it is easy to perform side by side comparisons to see differences in the visibility that NetFlow and sFlow provide into the same underlying traffic.

Note: XenServer 6.0, sFlowTrend and Real-Time NetFlow Analyzer are all available at no charge, making it easy for anyone to reproduce these tests.

Configuration

The Host sFlow supplemental pack was installed to automate sFlow configuration of the Open vSwitch and to export standard sFlow Host metrics. The following /etc/hsflowd.conf file sets the packet sampling rate to 1-in-400, counter polling interval to 20 seconds and sends sFlow to sFlowTrend running on host 10.0.0.42 and listening on UDP port 6343.

sflow{
  DNSSD = off
  polling = 20
  sampling = 400
  collector{
    ip = 10.0.0.42
    udpport = 6343
  }
}

The following command was used to manually configure NetFlow monitoring, sending NetFlow to the Real-Time NetFlow Analyzer running on host 10.0.0.42 and listening on UDP port 2055:

ovs−vsctl −− set Bridge xenbr0 netflow=@nf \
−− −−id=@nf create NetFlow targets=\"10.0.0.42:2055\" active−timeout=60

Results

The following charts show the top protocols measured using sFlow and NetFlow:

Top Protocols in sFlowTrend

Top Protocols in Real-Time NetFlow Analyzer

Looking at the two charts, both show similar average traffic levels. The sFlowTrend chart shows the ingress Memcache (TCP:11211) traffic at between 0.7 and 0.9 Mb/s. Looking at the Real-time NetFlow Analyzer total traffic table, 464.41Mb were seen over the last 11 minutes 47 seconds, giving an average rate of 0.66 Mb/s. The sFlowTrend measurements are consistently higher since they include the bandwidth consumed by layer 2 headers whereas NetFlow only reports on layer 3 bytes. However, the layer 2 overhead can be estimated by assuming that an additional 18 bytes per packet (MAC source, MAC destination, type and CRC) and multiplying by the total packets count (492,036), resulting in an additional  0.1 Mb/s which brings the NetFlow measurement to 0.76Mb/s, putting it into agreement with the sFlow measurements.

Note: The overhead associated with Ethernet headers and tunneling protocols can represent a significant fraction of overall bandwidth. By exporting packet headers, sFlow provides detailed information on the encapsulations and their overhead. NetFlow does not provide a direct measure of total bandwidth.

The periodic, 60 second, spikes in traffic shown on the NetFlow Analyzer chart are an artifact of the way NetFlow reports on long running connections. With NetFlow, packet and byte counters are maintained for each connection in a flow cache within the switch. When the connection terminates, a flow record is generated containing the connection information and counters. The active-timeout setting in the NetFlow configuration is used to ensure visibility into long running connections, causing the switch to periodically export NetFlow records for active connections. In contrast, sFlow does not use a flow cache, instead sampled packet headers are continually exported, resulting in real-time charts that more accurately reflect the traffic trend.

In addition, exporting packet headers allows an sFlow analyzer to monitor all types of traffic flowing across the switch; note the ARP and IPv6 traffic displayed in sFlowTrend in addition to the TCP/UDP flows. Visibility into layer 2 traffic is particularly important in switched environments where protocols such as DHCP/BOOTP, STP, LLDP and ARP need to be closely managed. sFlow also provides visibility into networked storage, including Ethernet SAN technologies (e.g. FCoE or AoE), that typically dominates bandwidth usage in the data center. Looking forward, there are a number of tunneling protocols being developed to connect virtual switches, including: GRE, mpls, VPLS, VXLAN and NVGRE. As new protocols are deployed on the network they are easily monitored without any change to exiting sFlow agents ensuring end-to-end visibility across the physical and virtual network.

In contrast, NetFlow relies on the switch to decode the traffic. In this case the switch is exporting NetFlow version 5 which only exports records for IPv4 traffic. The NetFlow analyzer is thus only able to report on IPv4 protocols, all other traffic is invisible. This limitation is not unique to Open vSwitch; NetFlow version 5 is the most widely supported version of NetFlow in network devices and is also the version exported by VMware vSphere 5.0.

The next two charts show top connections flowing through the virtual switch:

Top Connections in sFlowTrend

Top Connections in Real-Time NetFlow Analyzer

The Top Connections charts further demonstrate the limitation in NetFlow visibility where only IPv4 flows are shown. The sFlow analyzer is able to report in detail on all types of traffic flowing through the switch, in this case showing details of IPv6 traffic in addition to IPv4 flows.

The next two charts show interface utilization and packet counts from sFlowTrend:

Link Utilization in sFlowTrend

Link Counters in sFlowTrend

This type of interface trending is a staple of network management, but obtaining the information is challenging in virtual environments. While SNMP is typically used to obtain this information from network equipment, servers are much less likely to be managed using SNMP and so SNMP polling is often not an option. In addition, there may be large numbers of virtual ports associated with each physical switch port. In a virtual environment with 10,000 physical switch ports you might need to monitor as many as 200,000 virtual ports. Even if SNMP agents were installed on all the servers, SNMP polling does not scale well to large numbers of interfaces. The integrated counter polling mechanism built into sFlow provides scalable monitoring of the utilization of every switch port in the network, both physical and virtual, quickly identifying problems wherever they may occur in the network.

In contrast, NetFlow only reports on traffic flows so neither of these charts is available in the NetFlow Analyzer. The remaining charts are based on sFlow counter data so there are no corresponding NetFlow Analyzer charts.

The next sFlowTrend chart shows the CPU load on the hypervisor:

Hypervisor CPU in sFlowTrend

The virtual switch is a software component running on the hypervisor, thus if the hypervisor is overloaded, then network performance will degrade. The sFlow counter polling mechanism extends to system performance counters in addition to the interface counters shown earlier, allowing the sFlow analyzer to display hypervisor CPU utilization. In this case the chart shows a small spike in system CPU utilization corresponds to the spike in traffic at 9:52AM.

The next sFlowTrend chart shows a trend in disk IO on the virtual machine:

Virtual Machine Disk IO in sFlowTrend

This chart shows that the burst in iSCSI traffic shown in the Top Protocols chart corresponds to a spike in read activity on the virtual machine. Again, sFlow's counter push mechanism efficiently exports information about the performance of virtual machines, allowing the interaction between network and system activity to be understood.

Comments

NetFlow provides limited visibility, focusing on layer 3 network connections. The NetFlow architecture relies on complex functionality within the switches and the complexity of configuring and maintaining NetFlow adds to operational costs and limits scalability. For example, gaining visibility into IPv6 traffic requires firmware (and often hardware) upgrades to the network infrastructure that can be challenging in large scale, always-on, cloud environments.

In contrast, adding support for additional protocols in sFlow requires no change to the network infrastructure, but is simply a matter of upgrading the sFlow analyzer. The sFlow architecture eliminates complexity from the agents, increasing scalability and reducing the operational costs associated with configuration and maintenance. sFlow provides comprehensive visibility into network and system resources needed to manage performance in virtualized and cloud environments.

XenServer 6.0 supplemental pack

Today, the press release, Citrix Optimizes XenServer for the Cloud Era, announced the release of XenServer 6.0. Of particular interest is the inclusion of Open vSwitch (OVS) as the default network stack for XenServer. OVS includes support for the sFlow standard, seamlessly integrates monitoring of physical and virtual networking to provide the end-to-end visibility needed to manage cloud performance.

The Host sFlow agent is available as a XenServer 6.0 supplemental pack, offering a simple way to enable sFlow in OVS and further simplifying operations by exporting standard server and virtual machine statistics. Unifying network and system monitoring provides the integrated view of performance needed for the coordinated management of resources that is essential in virtualized environments.

This article describes the steps needed to install the supplemental pack and configure sFlow monitoring.

Note:  The default vSwitch in XenServer 6.0 includes sFlow support. See the XenServer 5.6FP1/FP2 instructions to enable sFlow on the older platforms.

First, download the Host sFlow XenServer supplemental pack (xenserver60-hsflowd-X_XX.iso).

Then, either copy the file to the host and run the following commands:

mkdir /tmp/iso 
mount -o loop xenserver60-hsflowd-X_XX.iso /tmp/iso 
cd /tmp/iso 
./install.sh 
cd 
umount /tmp/iso

Alternatively, burn the ISO file onto a CD and run the following commands to install:

mount /dev/cdrom /mnt
cd /mnt
./install.sh
cd
umount /mnt

Next, use the following commands to start the monitoring daemons:

service hsflowd start
service sflowovsd start

The following steps configure all the sFlow agents to sample packets at 1-in-512, poll counters every 20 seconds and send sFlow to an analyzer (10.0.0.50) over UDP using the default sFlow port (6343).

Note: A previous posting discussed the selection of sampling rates.

The default configuration method used for sFlow is DNS-SD; enter the following DNS settings in the site DNS server:

analyzer A 10.0.0.50

_sflow._udp SRV 0 0 6343 analyzer
_sflow._udp TXT (
"txtvers=1"
"polling=20"
"sampling=512"
)

Note: These changes must be made to the DNS zone file corresponding to the search domain in the XenServer /etc/resolv.conf file. If you need to modify the search domain, do not edit the resolv.conf file directly since the changes will be lost on a system reboot, instead either follow the directions in How to Add a Permanent Search Domain Entry in the Resolv.conf File of a XenServer Host, or simply edit the DNSSD_domain setting in the /etc/hsflowd.conf file to specify the domain to use to retrieve DNS-SD settings.

Once the sFlow settings are added to the DNS server, they will be automatically picked up by the Host sFlow agents. If you need to change the sFlow settings, simply change them on the DNS server and the change will automatically be applied to all the XenServer systems in the data center.

Alternatively, manual configuration is an option if you do not want to use DNS-SD. Simply edit the Host sFlow agent configuration file, /etc/hsflowd.conf, on each Xenserver:

sflow{
  DNSSD = off
  polling = 20
  sampling = 512
  collector{
    ip = 10.0.0.50
    udpport = 6343
  }
}

After editing the configuration file you will need to restart the Host sFlow agent:

service hsflowd restart

An sFlow analyzer is needed to receive the sFlow data and report on performance (see Choosing an sFlow analyzer). The free sFlowTrend analyzer is a great way to get started, see sFlowTrend adds server performance monitoring to see examples.

Microsoft Hyper-V

This week, InMon Corp. demonstrated sFlow monitoring in Microsoft Hyper-V during the Microsoft BUILD conference in Anaheim, California. InMon and Microsoft are working together to integrate sFlow support in Hyper-V 3.0, the hypervisor that will be included in the upcoming Windows Server 8 and Windows 8 releases.

The video above describes the integration of sFlow monitoring in Hyper-V.  The extensible virtual switch is a key component of the Hyper-V 3.0 release. The following diagram from the video shows how sFlow is implemented as virtual switch extension.

The sFlow extension is inserted into the packet forwarding path, randomly sampling packets that flow through the virtual switch. sFlow's packet sampling mechanism is extremely lightweight, delivering detailed visibility while ensuring minimal impact on forwarding performance. Sampled packets are sent to a userspace sFlow agent that exports the sampled packet headers along with physical and virtual NIC interface counters, server and virtual machine cpu, memory and I/O performance to an sFlow collector.

The photo above is of a live demonstration of the Hyper-V sFlow extension. The screen shows a real-time chart of network traffic flowing between virtual machines generated using the free sFlowTrend collector.

The sFlow standard is widely supported by switch vendors. Including sFlow monitoring in the Hyper-V virtual switch simplifies network management by allowing a common set of tools to monitor physical and virtual network performance. sFlow in Hyper-V further simplifies management by exporting server and virtual machine statistics. Unifying network and system monitoring provides the integrated view of performance needed for the coordinated management that is essential in virtualized environments. In addition to monitoring data center resources, sFlow can also be used to monitor performance within public clouds (see Rackspace cloudservers and Amazon EC2), providing the integrated visibility needed to optimize workload placement and minimise operating costs in hybrid cloud environments.

For public or private cloud operators, support for sFlow in the Hyper-V extensible switch and the Open vSwitch embeds visibility in leading commercial hypervisors (Microsoft Hyper-V and Citrix XenServer), open source hypervisors (Xen Cloud Platform (XCP), Xen, KVM, Proxmox VE and VirtualBox) and cloud management systems (OpenStack, OpenQRM and OpenNebula), providing the scalable visibility needed for accounting and billing, operational control and cost management.

Java virtual machine

The jmx-sflow-agent project is an open source implementation of sFlow monitoring for Java Virtual Machines (JVM). Instrumenting Java using sFlow provides scalable, real-time monitoring for applications such as Hadoop, Cassandra and Tomcat that typically involve large numbers of virtual machines.

The sFlow Host Structures extension describes a set of standard metrics for monitoring physical and virtual machines. The jmx-sflow-agent extends the set of physical server metrics exported by a Host sFlow agent which exports the physical server metrics.

The jmx-sflow-agent exports the generic virtual machine structures. In addition, the jmx-sflow-agent uses the Java Management Extensions (JMX) interface to collect performance statistics about java threads, heap/non-heap memory, garbage collection, compilation and class loading. These additional metrics are exported along with the generic virtual machine statistics.

The jmx-sflow-agent software is designed to integrate with the Host sFlow agent to provide a complete picture of server performance. Download, install and configure Host sFlow before proceeding to install the tomcat-sflow-valve - see Installing Host sFlow on a Linux Server. There are a number of options for analyzing cluster performance using Host sFlow, including Ganglia and sFlowTrend.

Next, download the sflowagent.jar file from https://github.com/sflow/jmx-sflow-agent. Copy the sflowagent.jar file into the same directory as your Java application. Enabling the sFlow instrumentation involves including -javaagent:sflowagent.jar argument when starting the Java application.

The following example shows how instrumentation is added to MyApp:

java -javaagent:sflowagent.jar\
     -Dsflow.hostname=vm.test\
     -Dsflow.uuid=564dd23c-453d-acd2-fa64-85da86713620\
     -Dsflow.dsindex=80001\
     MyApp

Arguments can be passed to the sFlow module as system properties:

• sflow.hostname, optionally assign a "hostname" to identify the virtual machine. Choose a naming strategy that is helpful in identifying virtual machine instances, for example: "hadoop.node1" etc. The hostname is exported in the sFlow host_descr structure.
• sflow.uuid, optionally assign a UUID to the virtual machine so that it can be uniquely identified. The UUID is exported in the sFlow host_descr structure.
• sflow.dsindex, uniquely identifies the data source associated with this virtual machine on this server. The dsindex number only needs to be set if more than one virtual machine is running on the server. For virtual machines offering network services, use TCP/UDP port number associated with the service as the dsindex, otherwise, use numbers in the range 65536-99999 to avoid clashes with other sFlow agents on the server.

The sFlow module shares Host sFlow's configuration and will automatically send data to the set of sFlow analyzers specified in the Host sFlow configuration file, picking up any configuration changes without the need to restart the Java application.

Finally, the real potential of of the Java virtual machine sFlow agent is as part of a broader sFlow monitoring system providing scaleable, real-time, visibility into applications, servers, storage and networking across the entire data center.

node.js

The node-sflow-module project is an open source implementation of sFlow monitoring for node.js, an open source event-based environment environment for creating network applications that built on Google's high performance V8 JavaScript Engine.

The advantage of using sFlow is the scalability it offers for monitoring the performance of large web server clusters or load balancers where request rates are high and conventional logging solutions generate too much data or impose excessive overhead. Real-time monitoring of HTTP provides essential visibility into the performance of large-scale, complex, multi-layer services constructed using Representational State Transfer (REST) architectures. In addition, monitoring HTTP services using sFlow is part of an integrated performance monitoring solution that provides real-time visibility into applications, servers and switches (see sFlow Host Structures).

The node-sflow-module software (sflow.js) is designed to integrate with the Host sFlow agent to provide a complete picture of server performance. Download, install and configure Host sFlow before proceeding to install node.js - see Installing Host sFlow on a Linux Server. There are a number of options for analyzing cluster performance using Host sFlow, including Ganglia and sFlowTrend.

Next, download the sflow.js file from http://node-sflow-module.googlecode.com/. Copy the sflow.js file into the same directory as your node.js application. Including the sFlow instrumentation is a one line change to the application, see the simple Hello World example below:

var http = require("http");
require("./sflow.js").instrument(http);

http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('Hello World\n');
}).listen(1337, "127.0.0.1");
console.log('Server running at http://127.0.0.1:1337/');

Once installed, the sflow.js will stream measurements to a central sFlow Analyzer. Currently the only software that can decode HTTP sFlow is sflowtool. Download, compile and install the latest sflowtool sources on the system your are using to receive sFlow from the servers in the node.js cluster.

Running sflowtool will display output of the form:

[pp@pcentos ~]$ sflowtool
startDatagram =================================
datagramSourceIP 10.0.0.112
datagramSize 116
unixSecondsUTC 1314458638
datagramVersion 5
agentSubId 8124
agent 10.0.0.112
packetSequenceNo 1
sysUpTime 22002
samplesInPacket 1
startSample ----------------------
sampleType_tag 0:2
sampleType COUNTERSSAMPLE
sampleSequenceNo 1
sourceId 3:8124
counterBlock_tag 0:2201
http_method_option_count 0
http_method_get_count 2
http_method_head_count 0
http_method_post_count 0
http_method_put_count 0
http_method_delete_count 0
http_method_trace_count 0
http_methd_connect_count 0
http_method_other_count 0
http_status_1XX_count 0
http_status_2XX_count 2
http_status_3XX_count 0
http_status_4XX_count 0
http_status_5XX_count 0
http_status_other_count 0
endSample   ----------------------
endDatagram   =================================
startDatagram =================================
datagramSourceIP 10.0.0.112
datagramSize 236
unixSecondsUTC 1314458652
datagramVersion 5
agentSubId 8124
agent 10.0.0.112
packetSequenceNo 2
sysUpTime 35729
samplesInPacket 1
startSample ----------------------
sampleType_tag 0:1
sampleType FLOWSAMPLE
sampleSequenceNo 0
sourceId 3:8124
meanSkipCount 6
samplePool 6
dropEvents 0
inputPort 0
outputPort 1073741823
flowBlock_tag 0:2201
flowSampleType http
http_method 2
http_protocol 1001
http_uri /
http_host 10.0.0.112:8124
http_useragent Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.
http_bytes 0
http_duration_uS 0
http_status 200
flowBlock_tag 0:2100
extendedType socket4
socket4_ip_protocol 6
socket4_local_ip 10.0.0.112
socket4_remote_ip 10.1.1.60
socket4_local_port 8124
socket4_remote_port 52609
endSample   ----------------------
endDatagram   =================================

The -H option causes sflowtool to output the HTTP request samples using the combined log format:

[pp@pcentos ~]$ sflowtool -H
10.1.1.60 - - [27/Aug/2011:08:26:52 -0700] "GET / HTTP/1.1" 200 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534."

Converting sFlow to combined logfile format allows existing log analyzers to be used to analyze the sFlow data. For example, the following commands use sflowtool and webalizer to create reports:

The resulting webalizer report shows top URLs:

Finally, the real potential of HTTP sFlow is as part of a broader performance management system providing real-time visibility into applications, servers, storage and networking across the entire data center.

Ganglia and cloud performance

The Ganglia 3.2 release includes support for collecting and displaying server performance metrics sent using the sFlow standard. Ganglia's focus has traditionally been to monitor clusters and grids, however, the scalability and automatic discovery capabilities of Ganglia also make it highly suited to monitoring pools of virtual machines.

Visibility in the cloud discusses the different challenges managing virtual machines hosted within a public cloud and management of the cloud infrastructure. The article, Rackspace cloudservers, shows how Ganglia and sFlow can be used to monitor the performance of virtual machines hosted in a public cloud. This article examines how Ganglia and sFlow can be used by service providers and private cloud operators to monitor the performance of the cloud infrastructure.

Currently sFlow agents are available for XCP (Xen Cloud Platform), Citrix XenServer and KVM/libvirt virtualization platforms. When monitoring a hypervisor using sFlow, Ganglia will display the following hypervisor specific metrics in addition to the familiar CPU, memory, disk and network statistics:

The Domain Count trends the number of virtual machines running on the server. The Hypervisor Free Memory chart shows how much free memory is available to run additional virtual machines.

In addition, sFlow also reports basic CPU, Memory, Disk I/O and Network I/O for every virtual machine running on the hypervisor without the need to install agents on the virtual machines. However, these additional statistics are currently discarded by default since the Ganglia user interface expects every server to report a common set of metrics and the data available from virtual machines is limited, resulting in missing charts. In addition, sFlow uniquely identifies virtual machines by their UUID (Universally Unique Identifier) but Ganglia currently expects hosts to be identified by IP addresses and hostnames (which may not be known for virtual machines).

Ganglia 3.2 provides an experimental override, allowing the additional per virtual machine performance metrics to be collected. The following entries in the Ganglia gmond configuration file (/etc/gmond.conf) configures sFlow monitoring and enables the additional per virtual machine performance metrics:

globals {
/* Listen, but don't send metrics */
  mute = yes
  deaf = no
  ...

/* sFlow channel */
udp_recv_channel {
  port = 6343
}

/* Enable virtual machine statistics */
sflow {
  accept_vm_metrics = yes
}

Once enabled, each virtual machine will appear as a member of the cluster. Selecting a virtual machine displays its metrics:

Note: The virtual machine metrics reported by sFlow are consistent with libvirt.

Ganglia has great potential for monitoring virtual machine pools. The experimental support for virtual machine monitoring in Ganglia 3.2 provides a starting point, laying the foundation for further development.

Ganglia 3.2 released

The open source Ganglia Monitoring System is widely used to monitor high-performance computing systems such as clusters and Grids. The latest Ganglia 3.2 release includes native support for the sFlow standard. This article describes some of the benefits of using sFlow for cluster monitoring and describes how to configure Ganglia as an sFlow analyzer.

The diagram shows the elements of the solution. Each server sends sFlow to the Ganglia gmond process which builds an in-memory database containing the server statistics. The Ganglia gmetad process periodically queries the gmond database and updates trend charts that are made available through a web interface. The sFlow server performance data seamlessly integrates with Ganglia since the standard sFlow server metrics are based on Ganglia's core set of metrics.

Note: The metrics for all the servers in the cluster can be retrieved as an XML document by connecting to gmond (the default port is 8649). This API (used by gmetad) provides a simple way for performance monitoring tools that rely on a polling model to retrieve the sFlow metrics, for example Vladimir Vuksan's Use your trending data for alerting article describes how Nagios can use this API to retrieve metrics.

The sFlow solution is very similar to using Ganglia in a unicast configuration where gmond agents, installed on each server in the cluster, are configured to periodically send metrics to a central gmond instance that builds the database of cluster performance. The sFlow solution simply replaces the gmond agents in the cluster with sFlow agents.

Host sFlow is a free, open source, sFlow agent implementation. The Host sFlow agent reports on the performance of physical and virtual servers and currently supports Linux, FreeBSD and Windows servers as well as the Citrix XenServer, XCP (Xen Cloud Platform) and KVM/libvirt virtualization platforms.

Why use Host sFlow instead of gmond to monitor servers?

• Lightweight Eliminating collector functionality reduces the size and complexity of the Host sFlow agent. The reduced overhead is particularly important when monitoring resource constrained environments like hypervisors.
• Portable The Host sFlow agent has minimal software dependencies and is easily ported to different platforms, including a native Windows implementation.
• Efficient The sFlow protocol efficiently packs all the standard Ganglia metrics in a single UDP datagram. Gmond requires over 32 datagrams to send the same information.
• Standard The standard metrics exported by Host sFlow agents allows performance monitoring tools to share data, eliminating the need for wasteful duplication of agents. 
• Metrics The standard set of sFlow metrics includes the core Ganglia metrics as well as additional disk I/O, swap, interrupt and virtual machine statistics. 

One of the strengths of Ganglia is the ability to easily add new metrics. While the Host sFlow agent doesn't support the addition of custom metrics, the Ganglia gmetric command line tool provides a simple way add custom metrics. For example, the following script exports the number of users currently logged into a system:

/usr/bin/gmetric --name Current_Users --value `who |wc -l` --type int32 --unit current_users

Running the command periodically using crontab allows Ganglia to track the metric. For embedded applications, the embeddedgmetric project provides C/C++, Python, PHP, Perl and Java libraries for sending Ganglia metrics.

The following entries in the gmond configuration file (/etc/gmond.conf) configures gmond to run in collector only mode, listening for sFlow data on UDP port 6343 (the standard sFlow port):

globals {
/* Listen, but don't send metrics */
  mute = yes
  deaf = no
  ...

}

/* channel to receive gmetric messages */
udp_recv_channel {
  port = 8649
}

/* channel to receive sFlow */
/* 6343 is the default sFlow port, an explicit sFlow    */
/* configuration section is needed to override default  */ 
udp_recv_channel {
  port = 6343
}

/* channel to service requests for XML data from gmetad */
tcp_accept_channel {
  port = 8649
}

Note: Delete all modules, collection_group and include sections from the gmond configuration file in this example and in the following examples since gmond is being used simply to collect sFlow metrics and doesn't need to load modules to generate metrics.

In the Ganglia architecture, each cluster is monitored by a separate gmond process. If more than one cluster is to be monitored, then it is possible to run multiple gmond processes on a single server, each with its own configuration file. For example, if Host sFlow agents on the first cluster are sending to port 6343, then Host sFlow agents on the second cluster should be configured to send to a different port, say 6344. The following settings will configure the second gmond to listen on the non-standard port.

globals {
/* Listen, but don't send metrics */
  mute = yes
  deaf = no
  ...

}

/* channel to receive gmetric messages */
udp_recv_channel {
  port = 8650
}

/* channel to receive sFlow */
udp_recv_channel {
  port = 6344
}

/* Change sFlow channel to non-standard port 6344 */
sflow {
  udp_port = 6344
}

/* channel to service requests for XML data from gmetad */
tcp_accept_channel {
  port = 8650
}

Note: The non-standard port setting is only required if both gmond processes are running on a single server. If each cluster is monitored by a separate server then the Host sFlow agents on each cluster simply need to be configured to send to the collector for their cluster.

Another alternative is to assign multiple IP addresses to the server, one per cluster. In this case Host sFlow agents in the first cluster will be configured to send to one address and agents in the second cluster to a different address. The following settings show how gmond can be configured to listen for sFlow on a specific IP address (e.g.  10.0.0.22):

globals {
/* Listen, but don't send metrics */
  mute = yes
  deaf = no
  ...

}

/* channel to receive sFlow */
udp_recv_channel {
  port = 6343
  bind = 10.0.0.22
}

The integration of network, system and application monitoring (see sFlow Host Structures) makes sFlow ideally suited for converged infrastructure monitoring. Using a single multi-vendor standard for both network and system performance monitoring reduces complexity and provides the integrated view of performance needed for effective management (see Management silos).