sFlow Host Structures

The completed sFlow Host Structures specification has been published by sFlow.org, extending the sFlow standard to include physical and virtual server performance metrics. The specification describes a coherent framework that builds on the sFlow metrics exported by most switch vendors, linking network, server and application performance monitoring to provide an integrated picture of performance.

The diagram above shows how the packet header information exported by network devices is used to link network performance with performance metrics collected from servers and applications. The packet header contains MAC addresses corresponding to physical and virtual server network adapter cards as well as TCP/UDP socket information identifying individual application instances. Collecting sFlow data from the network devices provides an sFlow analyzer with a real-time map of the physical and logical relationships between entities on the network (see Packet paths and Application mapping).

A server exporting sFlow performance metrics includes an additional structure containing the MAC addresses associated with each of its network adapters. The inclusion of the MAC addresses provides a common key linking server performance metrics (CPU, Memory, I/O etc.) to network performance measurements (network flows, link utilizations, etc.), providing a complete picture of the server's performance (see Hybrid server monitoring and UUID)

The sFlow Host Structures specification builds on the scalable "counter push" mechanism that is used by network devices to export standard interface counters (see Link utilization). Most operating systems already maintain performance counter to track CPU, memory and I/O performance. The sFlow Host Structures specification leverages work done by the Ganglia project to define a common set of metrics across different operating systems, including: Windows, Linux (Fedora/RedHat/CentOS, Debian, Gentoo, SuSE/OpenSuSE), Solaris, FreeBSD, NetBSD, OpenBSD, DragonflyBSD and AIX. The extension of sFlow to include server performance metrics integrates network and system monitoring to deliver a data center wide view of performance (see Top servers and Cluster performance).

For virtual machine performance metrics, the sFlow Host Structures specification draws on definitions from the libvirt project which has defined a standard set of metrics that can be collected from a wide variety of virtualization platforms, including: Xen, QEMU, KVM, LXC, OpenVZ, User Mode Linux, VirtualBox, VMWare ESX and GSX. Again, the MAC addresses associated with each virtual machine are exported along with its performance metrics so that the virtual machine's performance can be linked to its network activity.

The sFlow Host Structures document also describes the extension of sFlow's sampling mechanism to include application transaction sampling. Examples of application level transactions include: HTTP requests to a web server, NFS/CIFS requests to a file server, memcached requests and operations performed by a Hadoop cluster. An application sFlow agent samples completed transactions, capturing information about each completed request, including: size, duration, type, URL, file name etc. Each application transaction sample is linked to the network through the inclusion of TCP/UDP socket information which can be matched to packet header information from network devices.

What clearly distinguishes sFlow from other monitoring technologies is the integrated, end-to-end, view of performance that it offers. Integration exponentially increases the value of information by making it actionable. For example, identifying that an application is running slowly isn't enough to solve the performance problem. However, if you also know that the server hosting the application is seeing poor disk performance, can link the disk performance to a slow NFS server, can identify the other clients of the NFS server and finally determine that all the request are competing for access to a single file, then you are in a position to take action. It is this ability to link data together, combined with the scalability to monitor every resource in the data center that makes sFlow revolutionary.

OpenStack

The recently launched OpenStack project aims to provide an open source stack for cloud computing service providers. The project's backers include NASA and Rackspace, along with Citrix, Dell, NTT Data, Peer1, Intel, AMD and a number of other companies.

The OpenStack project is focused on the tools needed to manage and deploy cloud services on a large scale. By creating an ecosystem of service providers sharing common standards and open source tools, the project aims to create an environment that increases acceptance of cloud computing by eliminating the threat of vendor or service provider lock in. The project is hypervisor agnostic, targeting KVM, Xen and XenServer with the initial release.

A previous blog entry described the  Xen Cloud Platform. The Open Stack and Xen Cloud Platform projects are largely complementary and since both projects share a number of major contributors, the efforts should be well coordinated. For example, a critical part of any cloud computing architecture is the virtualization and isolation of networking among tenants in the cloud. The Xen Cloud Platform has already adopted the Open vSwitch since it provides the open, standards-based, visibility and control needed to manage cloud networking. Based on comments from Citrix (a participant in both projects), it appears that OpenStack project will also incorporate the Open vSwitch as part of its networking stack.

The Open vSwitch supports the sFlow standard, extending the network visibility provided by most switch vendors into the virtualization layer. The sFlow standard is uniquely placed to become the standard of choice for cloud performance monitoring. The scalability of sFlow monitoring allows all the physical and virtual switches in a large cloud data center to be centrally monitored, providing the visibility needed to manage performance and account for network usage.

The recent extension of sFlow into server monitoring (see server) delivers the "single pane of glass" visibility into the network, storage and system resources that cloud service providers need to optimize service, reduce costs and charge for metered services.

Configuring Allied Telesis switches

The recent Allied Ware Plus 2.1.1 release adds sFlow support to Allied Telesis switches.

The following commands configure an Allied Telesis switch to sample packets at 1-in-512, poll counters every 30 seconds and send sFlow to an analyzer (10.0.0.50) over UDP using the default sFlow port (6343):

awplus> enable
awplus# configure terminal
awplus(config)# sflow collector ip 10.0.0.50 port 6343
awplus(config)# interface port1.0.1-port1.0.24
awplus(config-if)# sflow sampling-rate 512
awplus(config-if)# sflow polling-interval 30
awplus(config-if)# set sflow collector ip 10.0.0.50
awplus(config-if)# exit
awplus(config)# sflow enable

A previous posting discussed the selection of sampling rates. Additional information can be found on the Allied Telesis web site.

See Trying out sFlow for suggestions on getting started with sFlow monitoring and reporting.

RMON (4 groups)

Diagram highlighting 4 RMON groups supported by most switch vendors

If you look carefully at the data sheets for almost any managed switch you are likely to see RMON mentioned as a network management feature with one of the following qualifiers: mini-RMON, RMON (4 groups), RMON (groups 1,2,3 and 9) or RMON (statistics, history, alarm and event). The RMON feature is almost never used, it's a bit like the human appendix, a remnant left behind by evolution.

The RMON standard was developed by the IETF during the early 1990's to provide an SNMP interface to probes used for remotely monitoring Ethernet and Token Ring LANs. At the time, LANs consisted of coax cables that where shared by a number of hosts. Repeaters were used to connect the the cables and extend the network. In this environment, a single RMON probe would see all the traffic on the shared network, providing complete network visibility.

In the mid 1990's demand for bandwidth increased and switches started to become popular. However, while segmenting the network using switches helped improve performance, segmentation dramatically increased the number of probes needed to monitor the network since a probe was required for each segment. Many customers depended on the visibility that RMON probes provided and switch vendors felt pressure to provide embedded RMON functionality.

The RMON standard defines 20 different types (groups) of measurement, including: traffic matrices, top talkers, top protocols, trending etc. Implementing all these features on a switch is difficult, requiring a significant area on the switch ASIC, resources that the switch vendors wanted to allocate to more advanced switching features like QoS, rate limiting, VLANs etc.

Four RMON groups were identified that were easy to implement, requiring minimal ASIC resources. Since the RMON standard allowed a vendor to claim RMON compliance by implementing any of the RMON groups, many switch vendors decided to implement the four RMON groups in order to be able to market their products as supporting the RMON standard. The proliferation of devices, many with very limited capabilities, all claiming RMON compliance undermined the value of the RMON standard and it has fallen out of favor as a network monitoring technology.

Today, even though hardly anyone uses the four RMON groups they are now part of the design of most switch ASICs and leaving the feature in is easier than going to the trouble of redesigning the chip to remove it.

In 2001, the sFlow standard was developed to address the need to monitor network traffic in switched LAN environments. The sFlow standard describes a minimum set of functions (packet sampling and counter polling) that are easily implemented in a switch ASIC. Requiring that all sFlow compliant switches implement these features, ensures that every sFlow compliant switch delivers the full range of features needed for network visibility.

Diagram highlighting RMON functional areas addressed by sFlow

The sFlow monitoring architecture provides the full range of traffic monitoring functions by shifting complexity from the switches to a central sFlow analyzer (see Choosing an sFlow analyzer). The architecture has proven successful and today most switch vendors embed sFlow monitoring.

DNS-SD

The host-sflow agent is a free, open source, implementation of Host sFlow. Host sFlow extends the visibility provided by the sFlow standard, currently supported by most switch vendors, to include server performance metrics. Adopting a single multi-vendor standard for network and system monitoring simplifies data center management, providing "single pane of glass" visibility in converged data center environments (see Management silos).

Extending sFlow monitoring to hosts poses some configuration management challenges that are not easily addressed using the current approaches to sFlow configuration in switches:

1. Scalability, a data center with 100 switches may contain thousands of servers. With virtualization, there may be many virtual machines running on each physical server, pushing the total server count even higher. Automation is essential in order to deploy timely configuration changes and to ensure configuration consistency across all the servers.
2. Diversity, each host-sflow agent may be running on different operating systems (e.g. Windows, Linux) or a hypervisor (e.g. XenServer, ESXi). While you can generally rely on a switch to support SNMP and provide remote CLI access (using telnet or ssh), these services cannot be relied upon for multi-vendor, multi-OS server configuration.

In order to address these challenges, host-sflow agents can be configured using the DNS-SD (DNS Server Discovery) protocol. DNS (Domain Name System) client functionality is universally supported on servers and DNS-SD is a method of using DNS to perform server discovery and distribute configuration information.

Using DNS-SD, configuring all the servers in the data center is as simple as adding a few lines of text to the site's DNS zone file. The centrally managed configuration is then distributed to all the servers using DNS.

The following DNS configuration settings instruct all the servers to poll counters every 20 seconds, sending sFlow to two sFlow analyzers, server and chow:

_sflow._udp SRV 0 0 6343 server
_sflow._udp SRV 0 0 6343 chow
_sflow._udp TXT (
"txtvers=1"
"polling=20"
)

In addition to providing a simple way to configure sFlow on servers, DNS-SD offers an attractive alternative to using CLI or SNMP to configure sFlow, particularly in data centers. DNS-SD offers a simple way to configure all sFlow agents in the physical network (see Multi-vendor support), virtual network (see Open vSwitch and Virtual routing), servers (see Cluster performance and Top servers) and virtual servers (see Host sFlow) in a coordinated and scalable way.

Application mapping

Understanding the complex relationships between applications and the server, network and storage resources on which they depend is critical for effective management. Keeping track of these relationships is particularly challenging in converged data center environments where virtual machine migration, scale out storage and elastic service pools create large scale, rapidly changing relationships between the application components.

Current application discover and dependency mapping tools often require considerable manual configuration effort and can quickly fall behind in dynamic environments. Stale or incomplete dependency maps are of limited value in supporting the real-time performance, security and trouble shooting tasks that challenge data center managers.

Scalable, real-time visibility into network, system and storage relationships is one of the benefits of adopting the sFlow standard for converged data center monitoring. Most switch vendors implement sFlow in their switch hardware, providing integrated, wire-speed visibility into traffic between servers, throughout the data center, without the need for agents (see Hybrid server monitoring).

The application dependency map shown at the top of this page tracks application dependencies in real time using sFlow data (see Choosing an sFlow analyzer). The dependency map is only a small part of the integrated data center visibility provided by sFlow. The sFlow standard integrates network, storage and server monitoring to deliver a seamless picture of data center performance.

FCoE

The Fiber Channel over Ethernet (FCoE) protocol provides a method of carrying Storage Area Network (SAN) traffic over a switched Ethernet network. FCoE adds to the list of options for converged storage and data networking (see Networked storage) that provide the flexibility needed to support virtualization and cloud computing.

Network visibility is critical for managing performance in a converged network. The sFlow standard embeds monitoring in Ethernet switches and provides visibility into all types of traffic flowing over the converged network (see Packet headers), including new protocols such as FCoE. The chart at the top of this page was created using sFlow data from a switch and shows a real-time view of FCoE traffic flowing through the switch (see Choosing an sFlow analyzer).

Most data center switch vendors already support the sFlow standard. Selecting switches with sFlow support provides the visibility needed for effective management and control of a converged network.