Host sFlow 1.0 Released

(image from Host sFlow)

The Host sFlow project has released an open source agent implementing the sFlow Host Structures specification. The current stable release (version 1.0) supports: Linux, Windows, XenServer and Xen. The project is working to add support for additional platforms, including: AIX, HPUX, OS X, FreeBSD, OpenBSD, NetBSD, VMWare, KVM and Hyper-V.

The Host sFlow agent provides a highly scalable solution for monitoring clusters of servers (see Top servers and Cluster performance). The combination of sFlow in top of rack (ToR) or end of row (EoR) switches and Host sFlow agents installed on servers delivers visibility into the performance of large scale data center workloads (see Hybrid server monitoring).

The combination of Open vSwitch and the Host sFlow agent provide a lightweight, scalable, performance monitoring solution for open source virtualization (Xen, XenServer and KVM) and cloud platforms (OpenStack and Xen Cloud Platform).

The sFlow Host Structures specification has only recently been finalized. When looking for an sFlow analyzer (see Choosing an sFlow analyzer), ask if the vendor supports all optional and extended sFlow fields (including packet headers, interface counters and host structures). If possible, arrange for an evaluation and test the solution in a large scale trial.

Top servers

The image above shows the output of the Linux "top" command. Each row in the table corresponds to a process and the values in the row indicate how much of the system resources (memory and CPU) are consumed by the process. Sorting the table quickly identifies the top consumers of system resources.

Identifying "top" processes is a staple of system management and most operating systems have a tool that displays a sorted table of processes (e.g. Unix top, Windows Task Manager, OS X Activity Monitor).

When managing a data center full of servers, a top servers tool provides similar benefits, rapidly identifying servers with performance problems.

In the top servers table shown above, each row corresponds to a server in the data center. Sorting the table by server load quickly finds the most heavily loaded servers.

The challenge in constructing a data center wide top servers table is finding a scalable way to collect performance metrics from all the servers in the data center so that the metrics can be combined and sorted in a single table.

The screen capture shows actual data collected from over 1,000 servers. A Host sFlow agent was installed on each server. The agent is an open source implementation of the Host sFlow standard currently being developed at sFlow.org. The agent requires minimal server resource, only 50K of memory and negligible CPU. The combined network traffic from all 1,000+ Host sFlow agents is a little over 100K bits per second.

The Host sFlow agents provide the sFlow analyzer with a real-time view of the load on all the servers in the data center, making it possible to construct the data center wide top servers table.

Host sFlow combined with sFlow monitoring built into the network switches (see Hybrid server monitoring) provides a complete picture of the performance of each server in the data center. The traffic visibility from the switches provides context for a server's performance metrics, identifying the clients making use of its services and the back end resources that it depends on.

Open vSwitch

(diagram from Open vSwitch)

The Open vSwitch provides advanced switching capabilities for virtual servers. Currently the Open vSwitch supports Linux, Xen/XenServer, KVM and Virtual Box. The open source software is designed to be easily portable and is expected to support additional platforms in the future. The Open vSwitch is designed to integrate switching across multiple physical servers, providing an open source alternative to proprietary virtual switches such as VMWare's distributed switch and Cisco's Nexus 1000v.

The recent integration of sFlow traffic monitoring in the Open vSwitch extends the visibility into virtual servers, ensuring data center visibility and control.

Note: The Open vSwitch demonstrates how to integrate the reference sFlow agent code with a virtual switch or network adapter. Integrating sFlow requires minimal support in the "fast path" requiring only packet sampling and packet counters.

The following lines, added to the Open vSwitch configuration file (ovs-vswitchd.conf), configure sampling packets at 1-in-512, polling counters every 20 seconds and sending sFlow to an analyzer (10.0.0.50) over UDP using the default sFlow port (6343):

sflow.<bridgename>.agent    = eth0
sflow.<bridgename>.host     = 10.0.0.50:6343
sflow.<bridgename>.sampling = 512
sflow.<bridgename>.polling  = 20
sflow.<bridgename>.header   = 128

Note: Type "man ovs-vswitchd.conf" for a full list of configuration options. A previous posting discussed the selection of sampling rates.

The following screen capture, from the free sFlowTrend application, demonstrates the visibility provided by sFlow in the Open vSwitch:

All traffic is visible, traffic between virtual machines, and from the virtual machines to the outside world. In addition, sFlow is able to report on all the protocols on the network (note the layer 2, TCP and IPv6 flows in the chart), as well as information on VLANs and layer 2 priorities that is essential for managing switched traffic.

The second screen capture shows a bandwidth trend for a virtual adapter on the vSwitch:

This type of interface trending is a staple of network management, but obtaining the information is challenging in virtual environments. While SNMP is typically used to obtain this information from network equipment, servers are much less likely to be managed using SNMP and so SNMP polling is often not an option. In addition, there may be large numbers of virtual ports associated with each physical switch port. In a virtual environment with 10,000 physical switch ports you might need to monitor as many as 200,000 virtual ports. Even if SNMP agents were installed on all the servers, SNMP polling does not scale well to large numbers of interfaces. The integrated counter polling mechanism built into sFlow provides scalable monitoring of the utilization of every switch port in the network, both physical and virtual, quickly identifying problems wherever they may occur in the network.

Download Open vSwitch and sFlowTrend to evaluate the benefits of visibility in the virtualization layer.

Finally, the Open vSwitch also supports the OpenFlow to allow centralized control of switch forwarding. The combination of sFlow and OpenFlow in the vSwitches delivers visibility and control of the network edge.

Feb. 15, 2011 Update: The configuration steps shown in this article are no longer correct, more recent versions of the Open vSwitch use the ovs-vsctl command instead. The easiest way to manage the sFlow configuration of an Open vSwitch is to install the open source Host sFlow agent which will automatically manage sFlow settings in the Open vSwitch. For recent information on the Open vSwitch, click on the vSwitch label below.

Developer resources

(photo from sFlow.org)

The challenge of providing data center visibility and control requires that all network devices integrate the instrumentation needed for visibility and that software developers deliver solutions that convert the measurements into actionable intelligence.

Meeting this challenge involves a broad community of individuals and companies participating in the sFlow consortium and developing products based on the sFlow standard. The rapid changes taking place in data center networking are accelerating the adoption of sFlow and introducing large numbers of developers to sFlow technology. This article provides links to resources intended to help new developers get started with sFlow.

The sFlow.org web site is the first place developers should look for information, whether adding sFlow support to a network device, or developing an application to analyze sFlow. Join the sFlow mailing list to ask questions and connect with the sFlow community.

Familiarity with the sFlow version 5 standard is essential, even if you are using one of the following software libraries. A detailed understanding of the standard is critical when integrating the agent software with switch hardware to ensure that data is correctly collected and exported. When developing an sFlow analyzer, an understanding of the measurements ensures that the data is correctly interpreted.

Agent developers should not base their agent implementations on RFC 3176. The RFC describes sFlow version 4 and has been superseded by version 5. However, when developing an sFlow analyzer, both versions of sFlow should be supported since customers may have equipment exporting sFlow version 4.

In addition to the standard, the paper Packet Sampling Basics, is worth reading in order to understand the theory behind sFlow's sampling mechanism.

The following resources are available for agent developers:

1. Developer Tools, contains a free, open-source implementation of an sFlow agent that is the basis for a number of open source and commercial sFlow agent implementations. Building an agent using the sFlow Agent library saves time and ensures that the sFlow information is correctly and efficiently encoded. In addition to the agent source code, tools for testing and validating the correct function of the agent are included.
2. Virtual Probe, a free, open-source, software probe for monitoring virtual switch traffic based on the sFlow Agent library.
3. sFlow Toolkit, contains the sflowtool command line utility that decodes and prints sFlow records.
4. sFlowTrend, a free, graphical sFlow analyzer. Testing the agent with sFlowTrend makes it easy to visually check the accuracy of the sFlow agent and ensure that it is correctly reporting traffic flows and interface counters.

The first challenge facing anyone interested in writing an sFlow analyzer is to get access to a source of sFlow data. In order to obtain realistic sFlow data, it is best if the sFlow is generated by devices installed in a production network. The sFlow standard is widely supported by switch vendors, so check to see if you have access to an sFlow capable device. If an sFlow capable switch is not available, the software agents listed above provide another alternative. Installing the agent software on a server will monitor traffic on the server, generating realistic sFlow data.

The following software tools are helpful for sFlow analyzer developers:

1. sFlow Toolkit, contains a command line tool, sflowtool, that decodes sFlow and can be used with scripting languages to perform sFlow analysis. The source code for sflowtool is a useful starting point for developing a C language sFlow analyzer.
2. Net::sFlow, open-source Perl library for decoding sFlow version 4 and 5 datagrams.
3. jsFlow, open-source Java library for decoding sFlow version 5 datagrams.
4. Wireshark, open-source protocol analyzer. Familiarity with protocol analysis is an important skill for an sFlow developer since sFlow exports packet headers and decoding the packet headers is the responsibility of the sFlow analyzer. Examining traffic in Wireshark is a great way to develop a better understanding of protocol analysis and the information that can be extracted from the packet header.

Finally, please share links to additional resources by posting comments to this article.

Trying out sFlow

If you are interested in network-wide visibility and want to start experimenting with sFlow, take a look at your network and see if any of the switches are sFlow capable. Most switch vendors support sFlow, including: Brocade, Hewlett-Packard, Juniper Networks, Extreme Networks, Force10 Networks, 3Com, D-Link, Alcatel-Lucent, H3C, Hitachi, NEC AlaxalA, Allied Telesis and Comtec (for a complete list of switches, see sFlow.org).

If you don't already have switches with sFlow support, consider purchasing a switch to experiment with. There are a number of inexpensive switches with sFlow support (check the list of switches on sFlow.org), alternatively you may be able to pick up a used switch on eBay.

Finally, the open source Host sFlow agent can be used to host traffic and traffic between virtual machines on a virtual server (Xen®, VMware®, KVM).

Once you have access to a source of sFlow data, you will need an sFlow analyzer. The sFlowTrend application (shown above) is a free, purpose built, sFlow analyzer that will allow you to try out the full range of sFlow functionality, including:

• decoding and filtering on data from packet headers (including VLANs, priorities, MAC addresses, Ethernet types, as well as TCP/IP fields)
  
• accurate analysis, trending and reporting of packet samples
• trending of sFlow counters
• support for sFlow MIB to automatically configure sFlow on switches

Many traffic analyzers claim support for sFlow, but provide only partial support. It is worth starting with sFlowTrend to see the full capabilities of sFlow and to gain experience with sFlow monitoring before evaluating larger scale solutions.

Future posts on this blog will use sFlowTrend to demonstrate how sFlow monitoring can be used to solve common network problems. Downloading a copy of sFlowTrend will allow you to try the different strategies on your own network.