XenServer 6.0 supplemental pack

Today, the press release, Citrix Optimizes XenServer for the Cloud Era, announced the release of XenServer 6.0. Of particular interest is the inclusion of Open vSwitch (OVS) as the default network stack for XenServer. OVS includes support for the sFlow standard, seamlessly integrates monitoring of physical and virtual networking to provide the end-to-end visibility needed to manage cloud performance.

The Host sFlow agent is available as a XenServer 6.0 supplemental pack, offering a simple way to enable sFlow in OVS and further simplifying operations by exporting standard server and virtual machine statistics. Unifying network and system monitoring provides the integrated view of performance needed for the coordinated management of resources that is essential in virtualized environments.

This article describes the steps needed to install the supplemental pack and configure sFlow monitoring.

Note:  The default vSwitch in XenServer 6.0 includes sFlow support. See the XenServer 5.6FP1/FP2 instructions to enable sFlow on the older platforms.

First, download the Host sFlow XenServer supplemental pack (xenserver60-hsflowd-X_XX.iso).

Then, either copy the file to the host and run the following commands:

mkdir /tmp/iso 
mount -o loop xenserver60-hsflowd-X_XX.iso /tmp/iso 
cd /tmp/iso 
./install.sh 
cd 
umount /tmp/iso

Alternatively, burn the ISO file onto a CD and run the following commands to install:

mount /dev/cdrom /mnt
cd /mnt
./install.sh
cd
umount /mnt

Next, use the following commands to start the monitoring daemons:

service hsflowd start
service sflowovsd start

The following steps configure all the sFlow agents to sample packets at 1-in-512, poll counters every 20 seconds and send sFlow to an analyzer (10.0.0.50) over UDP using the default sFlow port (6343).

Note: A previous posting discussed the selection of sampling rates.

The default configuration method used for sFlow is DNS-SD; enter the following DNS settings in the site DNS server:

analyzer A 10.0.0.50

_sflow._udp SRV 0 0 6343 analyzer
_sflow._udp TXT (
"txtvers=1"
"polling=20"
"sampling=512"
)

Note: These changes must be made to the DNS zone file corresponding to the search domain in the XenServer /etc/resolv.conf file. If you need to modify the search domain, do not edit the resolv.conf file directly since the changes will be lost on a system reboot, instead either follow the directions in How to Add a Permanent Search Domain Entry in the Resolv.conf File of a XenServer Host, or simply edit the DNSSD_domain setting in the /etc/hsflowd.conf file to specify the domain to use to retrieve DNS-SD settings.

Once the sFlow settings are added to the DNS server, they will be automatically picked up by the Host sFlow agents. If you need to change the sFlow settings, simply change them on the DNS server and the change will automatically be applied to all the XenServer systems in the data center.

Alternatively, manual configuration is an option if you do not want to use DNS-SD. Simply edit the Host sFlow agent configuration file, /etc/hsflowd.conf, on each Xenserver:

sflow{
  DNSSD = off
  polling = 20
  sampling = 512
  collector{
    ip = 10.0.0.50
    udpport = 6343
  }
}

After editing the configuration file you will need to restart the Host sFlow agent:

service hsflowd restart

An sFlow analyzer is needed to receive the sFlow data and report on performance (see Choosing an sFlow analyzer). The free sFlowTrend analyzer is a great way to get started, see sFlowTrend adds server performance monitoring to see examples.

Microsoft Hyper-V

This week, InMon Corp. demonstrated sFlow monitoring in Microsoft Hyper-V during the Microsoft BUILD conference in Anaheim, California. InMon and Microsoft are working together to integrate sFlow support in Hyper-V 3.0, the hypervisor that will be included in the upcoming Windows Server 8 and Windows 8 releases.

The video above describes the integration of sFlow monitoring in Hyper-V.  The extensible virtual switch is a key component of the Hyper-V 3.0 release. The following diagram from the video shows how sFlow is implemented as virtual switch extension.

The sFlow extension is inserted into the packet forwarding path, randomly sampling packets that flow through the virtual switch. sFlow's packet sampling mechanism is extremely lightweight, delivering detailed visibility while ensuring minimal impact on forwarding performance. Sampled packets are sent to a userspace sFlow agent that exports the sampled packet headers along with physical and virtual NIC interface counters, server and virtual machine cpu, memory and I/O performance to an sFlow collector.

The photo above is of a live demonstration of the Hyper-V sFlow extension. The screen shows a real-time chart of network traffic flowing between virtual machines generated using the free sFlowTrend collector.

The sFlow standard is widely supported by switch vendors. Including sFlow monitoring in the Hyper-V virtual switch simplifies network management by allowing a common set of tools to monitor physical and virtual network performance. sFlow in Hyper-V further simplifies management by exporting server and virtual machine statistics. Unifying network and system monitoring provides the integrated view of performance needed for the coordinated management that is essential in virtualized environments. In addition to monitoring data center resources, sFlow can also be used to monitor performance within public clouds (see Rackspace cloudservers and Amazon EC2), providing the integrated visibility needed to optimize workload placement and minimise operating costs in hybrid cloud environments.

For public or private cloud operators, support for sFlow in the Hyper-V extensible switch and the Open vSwitch embeds visibility in leading commercial hypervisors (Microsoft Hyper-V and Citrix XenServer), open source hypervisors (Xen Cloud Platform (XCP), Xen, KVM, Proxmox VE and VirtualBox) and cloud management systems (OpenStack, OpenQRM and OpenNebula), providing the scalable visibility needed for accounting and billing, operational control and cost management.

Java virtual machine

The jmx-sflow-agent project is an open source implementation of sFlow monitoring for Java Virtual Machines (JVM). Instrumenting Java using sFlow provides scalable, real-time monitoring for applications such as Hadoop, Cassandra and Tomcat that typically involve large numbers of virtual machines.

The sFlow Host Structures extension describes a set of standard metrics for monitoring physical and virtual machines. The jmx-sflow-agent extends the set of physical server metrics exported by a Host sFlow agent which exports the physical server metrics.

The jmx-sflow-agent exports the generic virtual machine structures. In addition, the jmx-sflow-agent uses the Java Management Extensions (JMX) interface to collect performance statistics about java threads, heap/non-heap memory, garbage collection, compilation and class loading. These additional metrics are exported along with the generic virtual machine statistics.

The jmx-sflow-agent software is designed to integrate with the Host sFlow agent to provide a complete picture of server performance. Download, install and configure Host sFlow before proceeding to install the tomcat-sflow-valve - see Installing Host sFlow on a Linux Server. There are a number of options for analyzing cluster performance using Host sFlow, including Ganglia and sFlowTrend.

Next, download the sflowagent.jar file from https://github.com/sflow/jmx-sflow-agent. Copy the sflowagent.jar file into the same directory as your Java application. Enabling the sFlow instrumentation involves including -javaagent:sflowagent.jar argument when starting the Java application.

The following example shows how instrumentation is added to MyApp:

java -javaagent:sflowagent.jar\
     -Dsflow.hostname=vm.test\
     -Dsflow.uuid=564dd23c-453d-acd2-fa64-85da86713620\
     -Dsflow.dsindex=80001\
     MyApp

Arguments can be passed to the sFlow module as system properties:

• sflow.hostname, optionally assign a "hostname" to identify the virtual machine. Choose a naming strategy that is helpful in identifying virtual machine instances, for example: "hadoop.node1" etc. The hostname is exported in the sFlow host_descr structure.
• sflow.uuid, optionally assign a UUID to the virtual machine so that it can be uniquely identified. The UUID is exported in the sFlow host_descr structure.
• sflow.dsindex, uniquely identifies the data source associated with this virtual machine on this server. The dsindex number only needs to be set if more than one virtual machine is running on the server. For virtual machines offering network services, use TCP/UDP port number associated with the service as the dsindex, otherwise, use numbers in the range 65536-99999 to avoid clashes with other sFlow agents on the server.

The sFlow module shares Host sFlow's configuration and will automatically send data to the set of sFlow analyzers specified in the Host sFlow configuration file, picking up any configuration changes without the need to restart the Java application.

Finally, the real potential of of the Java virtual machine sFlow agent is as part of a broader sFlow monitoring system providing scaleable, real-time, visibility into applications, servers, storage and networking across the entire data center.

node.js

The node-sflow-module project is an open source implementation of sFlow monitoring for node.js, an open source event-based environment environment for creating network applications that built on Google's high performance V8 JavaScript Engine.

The advantage of using sFlow is the scalability it offers for monitoring the performance of large web server clusters or load balancers where request rates are high and conventional logging solutions generate too much data or impose excessive overhead. Real-time monitoring of HTTP provides essential visibility into the performance of large-scale, complex, multi-layer services constructed using Representational State Transfer (REST) architectures. In addition, monitoring HTTP services using sFlow is part of an integrated performance monitoring solution that provides real-time visibility into applications, servers and switches (see sFlow Host Structures).

The node-sflow-module software (sflow.js) is designed to integrate with the Host sFlow agent to provide a complete picture of server performance. Download, install and configure Host sFlow before proceeding to install node.js - see Installing Host sFlow on a Linux Server. There are a number of options for analyzing cluster performance using Host sFlow, including Ganglia and sFlowTrend.

Next, download the sflow.js file from http://node-sflow-module.googlecode.com/. Copy the sflow.js file into the same directory as your node.js application. Including the sFlow instrumentation is a one line change to the application, see the simple Hello World example below:

var http = require("http");
require("./sflow.js").instrument(http);

http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('Hello World\n');
}).listen(1337, "127.0.0.1");
console.log('Server running at http://127.0.0.1:1337/');

Once installed, the sflow.js will stream measurements to a central sFlow Analyzer. Currently the only software that can decode HTTP sFlow is sflowtool. Download, compile and install the latest sflowtool sources on the system your are using to receive sFlow from the servers in the node.js cluster.

Running sflowtool will display output of the form:

[pp@pcentos ~]$ sflowtool
startDatagram =================================
datagramSourceIP 10.0.0.112
datagramSize 116
unixSecondsUTC 1314458638
datagramVersion 5
agentSubId 8124
agent 10.0.0.112
packetSequenceNo 1
sysUpTime 22002
samplesInPacket 1
startSample ----------------------
sampleType_tag 0:2
sampleType COUNTERSSAMPLE
sampleSequenceNo 1
sourceId 3:8124
counterBlock_tag 0:2201
http_method_option_count 0
http_method_get_count 2
http_method_head_count 0
http_method_post_count 0
http_method_put_count 0
http_method_delete_count 0
http_method_trace_count 0
http_methd_connect_count 0
http_method_other_count 0
http_status_1XX_count 0
http_status_2XX_count 2
http_status_3XX_count 0
http_status_4XX_count 0
http_status_5XX_count 0
http_status_other_count 0
endSample   ----------------------
endDatagram   =================================
startDatagram =================================
datagramSourceIP 10.0.0.112
datagramSize 236
unixSecondsUTC 1314458652
datagramVersion 5
agentSubId 8124
agent 10.0.0.112
packetSequenceNo 2
sysUpTime 35729
samplesInPacket 1
startSample ----------------------
sampleType_tag 0:1
sampleType FLOWSAMPLE
sampleSequenceNo 0
sourceId 3:8124
meanSkipCount 6
samplePool 6
dropEvents 0
inputPort 0
outputPort 1073741823
flowBlock_tag 0:2201
flowSampleType http
http_method 2
http_protocol 1001
http_uri /
http_host 10.0.0.112:8124
http_useragent Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.
http_bytes 0
http_duration_uS 0
http_status 200
flowBlock_tag 0:2100
extendedType socket4
socket4_ip_protocol 6
socket4_local_ip 10.0.0.112
socket4_remote_ip 10.1.1.60
socket4_local_port 8124
socket4_remote_port 52609
endSample   ----------------------
endDatagram   =================================

The -H option causes sflowtool to output the HTTP request samples using the combined log format:

[pp@pcentos ~]$ sflowtool -H
10.1.1.60 - - [27/Aug/2011:08:26:52 -0700] "GET / HTTP/1.1" 200 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534."

Converting sFlow to combined logfile format allows existing log analyzers to be used to analyze the sFlow data. For example, the following commands use sflowtool and webalizer to create reports:

The resulting webalizer report shows top URLs:

Finally, the real potential of HTTP sFlow is as part of a broader performance management system providing real-time visibility into applications, servers, storage and networking across the entire data center.

Ganglia and cloud performance

The Ganglia 3.2 release includes support for collecting and displaying server performance metrics sent using the sFlow standard. Ganglia's focus has traditionally been to monitor clusters and grids, however, the scalability and automatic discovery capabilities of Ganglia also make it highly suited to monitoring pools of virtual machines.

Visibility in the cloud discusses the different challenges managing virtual machines hosted within a public cloud and management of the cloud infrastructure. The article, Rackspace cloudservers, shows how Ganglia and sFlow can be used to monitor the performance of virtual machines hosted in a public cloud. This article examines how Ganglia and sFlow can be used by service providers and private cloud operators to monitor the performance of the cloud infrastructure.

Currently sFlow agents are available for XCP (Xen Cloud Platform), Citrix XenServer and KVM/libvirt virtualization platforms. When monitoring a hypervisor using sFlow, Ganglia will display the following hypervisor specific metrics in addition to the familiar CPU, memory, disk and network statistics:

The Domain Count trends the number of virtual machines running on the server. The Hypervisor Free Memory chart shows how much free memory is available to run additional virtual machines.

In addition, sFlow also reports basic CPU, Memory, Disk I/O and Network I/O for every virtual machine running on the hypervisor without the need to install agents on the virtual machines. However, these additional statistics are currently discarded by default since the Ganglia user interface expects every server to report a common set of metrics and the data available from virtual machines is limited, resulting in missing charts. In addition, sFlow uniquely identifies virtual machines by their UUID (Universally Unique Identifier) but Ganglia currently expects hosts to be identified by IP addresses and hostnames (which may not be known for virtual machines).

Ganglia 3.2 provides an experimental override, allowing the additional per virtual machine performance metrics to be collected. The following entries in the Ganglia gmond configuration file (/etc/gmond.conf) configures sFlow monitoring and enables the additional per virtual machine performance metrics:

globals {
/* Listen, but don't send metrics */
  mute = yes
  deaf = no
  ...

/* sFlow channel */
udp_recv_channel {
  port = 6343
}

/* Enable virtual machine statistics */
sflow {
  accept_vm_metrics = yes
}

Once enabled, each virtual machine will appear as a member of the cluster. Selecting a virtual machine displays its metrics:

Note: The virtual machine metrics reported by sFlow are consistent with libvirt.

Ganglia has great potential for monitoring virtual machine pools. The experimental support for virtual machine monitoring in Ganglia 3.2 provides a starting point, laying the foundation for further development.

Ganglia 3.2 released

The open source Ganglia Monitoring System is widely used to monitor high-performance computing systems such as clusters and Grids. The latest Ganglia 3.2 release includes native support for the sFlow standard. This article describes some of the benefits of using sFlow for cluster monitoring and describes how to configure Ganglia as an sFlow analyzer.

The diagram shows the elements of the solution. Each server sends sFlow to the Ganglia gmond process which builds an in-memory database containing the server statistics. The Ganglia gmetad process periodically queries the gmond database and updates trend charts that are made available through a web interface. The sFlow server performance data seamlessly integrates with Ganglia since the standard sFlow server metrics are based on Ganglia's core set of metrics.

Note: The metrics for all the servers in the cluster can be retrieved as an XML document by connecting to gmond (the default port is 8649). This API (used by gmetad) provides a simple way for performance monitoring tools that rely on a polling model to retrieve the sFlow metrics, for example Vladimir Vuksan's Use your trending data for alerting article describes how Nagios can use this API to retrieve metrics.

The sFlow solution is very similar to using Ganglia in a unicast configuration where gmond agents, installed on each server in the cluster, are configured to periodically send metrics to a central gmond instance that builds the database of cluster performance. The sFlow solution simply replaces the gmond agents in the cluster with sFlow agents.

Host sFlow is a free, open source, sFlow agent implementation. The Host sFlow agent reports on the performance of physical and virtual servers and currently supports Linux, FreeBSD and Windows servers as well as the Citrix XenServer, XCP (Xen Cloud Platform) and KVM/libvirt virtualization platforms.

Why use Host sFlow instead of gmond to monitor servers?

• Lightweight Eliminating collector functionality reduces the size and complexity of the Host sFlow agent. The reduced overhead is particularly important when monitoring resource constrained environments like hypervisors.
• Portable The Host sFlow agent has minimal software dependencies and is easily ported to different platforms, including a native Windows implementation.
• Efficient The sFlow protocol efficiently packs all the standard Ganglia metrics in a single UDP datagram. Gmond requires over 32 datagrams to send the same information.
• Standard The standard metrics exported by Host sFlow agents allows performance monitoring tools to share data, eliminating the need for wasteful duplication of agents. 
• Metrics The standard set of sFlow metrics includes the core Ganglia metrics as well as additional disk I/O, swap, interrupt and virtual machine statistics. 

One of the strengths of Ganglia is the ability to easily add new metrics. While the Host sFlow agent doesn't support the addition of custom metrics, the Ganglia gmetric command line tool provides a simple way add custom metrics. For example, the following script exports the number of users currently logged into a system:

/usr/bin/gmetric --name Current_Users --value `who |wc -l` --type int32 --unit current_users

Running the command periodically using crontab allows Ganglia to track the metric. For embedded applications, the embeddedgmetric project provides C/C++, Python, PHP, Perl and Java libraries for sending Ganglia metrics.

The following entries in the gmond configuration file (/etc/gmond.conf) configures gmond to run in collector only mode, listening for sFlow data on UDP port 6343 (the standard sFlow port):

globals {
/* Listen, but don't send metrics */
  mute = yes
  deaf = no
  ...

}

/* channel to receive gmetric messages */
udp_recv_channel {
  port = 8649
}

/* channel to receive sFlow */
/* 6343 is the default sFlow port, an explicit sFlow    */
/* configuration section is needed to override default  */ 
udp_recv_channel {
  port = 6343
}

/* channel to service requests for XML data from gmetad */
tcp_accept_channel {
  port = 8649
}

Note: Delete all modules, collection_group and include sections from the gmond configuration file in this example and in the following examples since gmond is being used simply to collect sFlow metrics and doesn't need to load modules to generate metrics.

In the Ganglia architecture, each cluster is monitored by a separate gmond process. If more than one cluster is to be monitored, then it is possible to run multiple gmond processes on a single server, each with its own configuration file. For example, if Host sFlow agents on the first cluster are sending to port 6343, then Host sFlow agents on the second cluster should be configured to send to a different port, say 6344. The following settings will configure the second gmond to listen on the non-standard port.

globals {
/* Listen, but don't send metrics */
  mute = yes
  deaf = no
  ...

}

/* channel to receive gmetric messages */
udp_recv_channel {
  port = 8650
}

/* channel to receive sFlow */
udp_recv_channel {
  port = 6344
}

/* Change sFlow channel to non-standard port 6344 */
sflow {
  udp_port = 6344
}

/* channel to service requests for XML data from gmetad */
tcp_accept_channel {
  port = 8650
}

Note: The non-standard port setting is only required if both gmond processes are running on a single server. If each cluster is monitored by a separate server then the Host sFlow agents on each cluster simply need to be configured to send to the collector for their cluster.

Another alternative is to assign multiple IP addresses to the server, one per cluster. In this case Host sFlow agents in the first cluster will be configured to send to one address and agents in the second cluster to a different address. The following settings show how gmond can be configured to listen for sFlow on a specific IP address (e.g.  10.0.0.22):

globals {
/* Listen, but don't send metrics */
  mute = yes
  deaf = no
  ...

}

/* channel to receive sFlow */
udp_recv_channel {
  port = 6343
  bind = 10.0.0.22
}

The integration of network, system and application monitoring (see sFlow Host Structures) makes sFlow ideally suited for converged infrastructure monitoring. Using a single multi-vendor standard for both network and system performance monitoring reduces complexity and provides the integrated view of performance needed for effective management (see Management silos).

Five W's

The Five W's are the set of questions that a news report must answer in order to be considered complete:

• What happened (what is the story)?
• Who is it about?
• When did it take place?
• Where did it take place?
• Why did it happen?
• How did it happen?

These questions provide a good framework for solving performance management problems. The following example demonstrates how the network-wide visibility provided by the sFlow standard makes it easy to quickly answer each question in order to detect, diagnose and eliminate performance problems.

Note: The free sFlowTrend tool is used to demonstrate problem solving using sFlow, but there are many other tools to choose from.

What?

What happened? Threshold violations on interface counters provide the notification that there is a problem. sFlow provides an extremely efficient method of collecting interface counters, monitoring every interface in the network, allowing prompt detection of performance problems.

This screen capture of the sFlowTrend dashboard shows that a problem with excessive unicast packets has been detected. There are many devices and interfaces in this network, the next question is who reported the problem? Clicking on the bar provides the following answer.

Who?

Who is reporting the problem? The following table sorts the switches to show which ones are seeing excessive unicast traffic. Comparing switches provides a baseline making it easy to see whether the problem is widespread, or localized to specific devices. 

Note: Many monitoring systems are hierarchical, counters are polled locally and notifications of threshold violations are sent to the central management system. The problem with this approach is that the underlying data needed to put the event into context is lost. The sFlow architecture centralizes monitoring - performance counters from all devices are centrally collected and thresholds calculations are performed by the collector. sFlow makes it simple to drill down and compare the statistics underlying any notification, making it much easier to troubleshoot problems. 

Drilling down further, the following table shows individual interfaces sorted to show which interface is seeing excessive traffic. 

Now that we know where the problem is, the next question is when did it start?

When?

Again, because sFlow centralizes all the critical performance data, follow up is straightforward. Counter trends on any link can be displayed and the following chart was obtained by drilling down on the interface highlighted in the screen above.

This chart shows that a 2 minute spike in traffic occurred around 10 minutes ago. The chart shows that link utilization has returned to normal levels so there is no need for immediate action. However, it is worth identifying why the spike occurred and assess if it is likely to occur again.

Why? How?

Interface counters are only one type of data exported by sFlow. sFlow agents also export real-time traffic information. The two types of data complement one another, counters allow performance anomalies to be quickly detected and traffic information provides the detail needed to identify the root cause of the problem.

By flipping the chart from Utilization to a Top Connections, sFlowTrend uses the sFlow traffic measurements to break down the traffic into the top connections responsible for the traffic.

The chart shows why the traffic spiked. Multiple TCP connections to port 80 (web) from ganglia.sf.inmon.com were responsible for the spike in traffic. The top two connections are to fedora.mirrors.tds.net, providing a clue as to the type of traffic.

How did the spike happen? It seems likely that a system update was run on the ganglia.sf.inmon.com server. Given the timing of the traffic, this looks like an unscheduled update. It would be a good idea to talk to the system administrator for the server and suggest off-peak times to schedule the updates so that they don't interfere with peak business hours traffic.

Where?

What if the spike was ongoing and we couldn't contact the system administrator of the server to shut down the update? In this case it is very important to be able to locate the server on the network in order to take action.

When an sFlow agent reports on network traffic it also includes information on the packet path across the device. Combining data from all the switches allows an sFlow analyzer to discover and track network topology and the location of each host on the network.

Clicking on an address in sFlowTrend provides the current location.

In this case, the host ganglia.sf.inmon.com is located on port A13 on switch 10.0.0.244. Knowing where the server is located allows the network administrator to log into the switch and take corrective action, blocking or rate limiting the traffic.

The article, Network edge, described how the process of detecting traffic problems and applying controls to the switches can be fully automated. Automation is particularly important in large scale environments where manual intervention is labor intensive and slow.

Finally, the sFlow standard is widely supported by network equipment vendors, providing simple, scalable, end-to-end monitoring of wired and wireless networking as well as servers, virtual machines and applications running on the network. Comprehensive, integrated visibility is the key to simplifying management and controlling costs in networked environments.