XenServer 6.0 supplemental pack

Today, the press release, Citrix Optimizes XenServer for the Cloud Era, announced the release of XenServer 6.0. Of particular interest is the inclusion of Open vSwitch (OVS) as the default network stack for XenServer. OVS includes support for the sFlow standard, seamlessly integrates monitoring of physical and virtual networking to provide the end-to-end visibility needed to manage cloud performance.

The Host sFlow agent is available as a XenServer 6.0 supplemental pack, offering a simple way to enable sFlow in OVS and further simplifying operations by exporting standard server and virtual machine statistics. Unifying network and system monitoring provides the integrated view of performance needed for the coordinated management of resources that is essential in virtualized environments.

This article describes the steps needed to install the supplemental pack and configure sFlow monitoring.

Note:  The default vSwitch in XenServer 6.0 includes sFlow support. See the XenServer 5.6FP1/FP2 instructions to enable sFlow on the older platforms.

First, download the Host sFlow XenServer supplemental pack (xenserver60-hsflowd-X_XX.iso).

Then, either copy the file to the host and run the following commands:

mkdir /tmp/iso 
mount -o loop xenserver60-hsflowd-X_XX.iso /tmp/iso 
cd /tmp/iso 
./install.sh 
cd 
umount /tmp/iso

Alternatively, burn the ISO file onto a CD and run the following commands to install:

mount /dev/cdrom /mnt
cd /mnt
./install.sh
cd
umount /mnt

Next, use the following commands to start the monitoring daemons:

service hsflowd start
service sflowovsd start

The following steps configure all the sFlow agents to sample packets at 1-in-512, poll counters every 20 seconds and send sFlow to an analyzer (10.0.0.50) over UDP using the default sFlow port (6343).

Note: A previous posting discussed the selection of sampling rates.

The default configuration method used for sFlow is DNS-SD; enter the following DNS settings in the site DNS server:

analyzer A 10.0.0.50

_sflow._udp SRV 0 0 6343 analyzer
_sflow._udp TXT (
"txtvers=1"
"polling=20"
"sampling=512"
)

Note: These changes must be made to the DNS zone file corresponding to the search domain in the XenServer /etc/resolv.conf file. If you need to modify the search domain, do not edit the resolv.conf file directly since the changes will be lost on a system reboot, instead either follow the directions in How to Add a Permanent Search Domain Entry in the Resolv.conf File of a XenServer Host, or simply edit the DNSSD_domain setting in the /etc/hsflowd.conf file to specify the domain to use to retrieve DNS-SD settings.

Once the sFlow settings are added to the DNS server, they will be automatically picked up by the Host sFlow agents. If you need to change the sFlow settings, simply change them on the DNS server and the change will automatically be applied to all the XenServer systems in the data center.

Alternatively, manual configuration is an option if you do not want to use DNS-SD. Simply edit the Host sFlow agent configuration file, /etc/hsflowd.conf, on each Xenserver:

sflow{
  DNSSD = off
  polling = 20
  sampling = 512
  collector{
    ip = 10.0.0.50
    udpport = 6343
  }
}

After editing the configuration file you will need to restart the Host sFlow agent:

service hsflowd restart

An sFlow analyzer is needed to receive the sFlow data and report on performance (see Choosing an sFlow analyzer). The free sFlowTrend analyzer is a great way to get started, see sFlowTrend adds server performance monitoring to see examples.

Microsoft Hyper-V

This week, InMon Corp. demonstrated sFlow monitoring in Microsoft Hyper-V during the Microsoft BUILD conference in Anaheim, California. InMon and Microsoft are working together to integrate sFlow support in Hyper-V 3.0, the hypervisor that will be included in the upcoming Windows Server 8 and Windows 8 releases.

The video above describes the integration of sFlow monitoring in Hyper-V.  The extensible virtual switch is a key component of the Hyper-V 3.0 release. The following diagram from the video shows how sFlow is implemented as virtual switch extension.

The sFlow extension is inserted into the packet forwarding path, randomly sampling packets that flow through the virtual switch. sFlow's packet sampling mechanism is extremely lightweight, delivering detailed visibility while ensuring minimal impact on forwarding performance. Sampled packets are sent to a userspace sFlow agent that exports the sampled packet headers along with physical and virtual NIC interface counters, server and virtual machine cpu, memory and I/O performance to an sFlow collector.

The photo above is of a live demonstration of the Hyper-V sFlow extension. The screen shows a real-time chart of network traffic flowing between virtual machines generated using the free sFlowTrend collector.

The sFlow standard is widely supported by switch vendors. Including sFlow monitoring in the Hyper-V virtual switch simplifies network management by allowing a common set of tools to monitor physical and virtual network performance. sFlow in Hyper-V further simplifies management by exporting server and virtual machine statistics. Unifying network and system monitoring provides the integrated view of performance needed for the coordinated management that is essential in virtualized environments. In addition to monitoring data center resources, sFlow can also be used to monitor performance within public clouds (see Rackspace cloudservers and Amazon EC2), providing the integrated visibility needed to optimize workload placement and minimise operating costs in hybrid cloud environments.

For public or private cloud operators, support for sFlow in the Hyper-V extensible switch and the Open vSwitch embeds visibility in leading commercial hypervisors (Microsoft Hyper-V and Citrix XenServer), open source hypervisors (Xen Cloud Platform (XCP), Xen, KVM, Proxmox VE and VirtualBox) and cloud management systems (OpenStack, OpenQRM and OpenNebula), providing the scalable visibility needed for accounting and billing, operational control and cost management.

Java virtual machine

The jmx-sflow-agent project is an open source implementation of sFlow monitoring for Java Virtual Machines (JVM). Instrumenting Java using sFlow provides scalable, real-time monitoring for applications such as Hadoop, Cassandra and Tomcat that typically involve large numbers of virtual machines.

The sFlow Host Structures extension describes a set of standard metrics for monitoring physical and virtual machines. The jmx-sflow-agent extends the set of physical server metrics exported by a Host sFlow agent which exports the physical server metrics.

The jmx-sflow-agent exports the generic virtual machine structures. In addition, the jmx-sflow-agent uses the Java Management Extensions (JMX) interface to collect performance statistics about java threads, heap/non-heap memory, garbage collection, compilation and class loading. These additional metrics are exported along with the generic virtual machine statistics.

The jmx-sflow-agent software is designed to integrate with the Host sFlow agent to provide a complete picture of server performance. Download, install and configure Host sFlow before proceeding to install the tomcat-sflow-valve - see Installing Host sFlow on a Linux Server. There are a number of options for analyzing cluster performance using Host sFlow, including Ganglia and sFlowTrend.

Next, download the sflowagent.jar file from https://github.com/sflow/jmx-sflow-agent. Copy the sflowagent.jar file into the same directory as your Java application. Enabling the sFlow instrumentation involves including -javaagent:sflowagent.jar argument when starting the Java application.

The following example shows how instrumentation is added to MyApp:

java -javaagent:sflowagent.jar\
     -Dsflow.hostname=vm.test\
     -Dsflow.uuid=564dd23c-453d-acd2-fa64-85da86713620\
     -Dsflow.dsindex=80001\
     MyApp

Arguments can be passed to the sFlow module as system properties:

• sflow.hostname, optionally assign a "hostname" to identify the virtual machine. Choose a naming strategy that is helpful in identifying virtual machine instances, for example: "hadoop.node1" etc. The hostname is exported in the sFlow host_descr structure.
• sflow.uuid, optionally assign a UUID to the virtual machine so that it can be uniquely identified. The UUID is exported in the sFlow host_descr structure.
• sflow.dsindex, uniquely identifies the data source associated with this virtual machine on this server. The dsindex number only needs to be set if more than one virtual machine is running on the server. For virtual machines offering network services, use TCP/UDP port number associated with the service as the dsindex, otherwise, use numbers in the range 65536-99999 to avoid clashes with other sFlow agents on the server.

The sFlow module shares Host sFlow's configuration and will automatically send data to the set of sFlow analyzers specified in the Host sFlow configuration file, picking up any configuration changes without the need to restart the Java application.

Finally, the real potential of of the Java virtual machine sFlow agent is as part of a broader sFlow monitoring system providing scaleable, real-time, visibility into applications, servers, storage and networking across the entire data center.