Saturday, January 7, 2012

Host sFlow distributed agent

The sFlow standard uses a distributed push model in which each host autonomously sends a continuous stream of performance metrics to a central sFlow collector. The push model is highly scalable and is particularly suited to cloud environments.

The distributed architecture extends to the sFlow agents within a host. The following diagram shows how sFlow agent coordinate on each host to provide a comprehensive and consistent view of performance.

Typically, all hosts in the data center will share identical configuration settings. The Host sFlow daemon, hsflowd, provides a single point of configuration. By default hsflowd uses DNS Service Discover (DNS-SD) to learn configuration settings. Alternatively, settings can be distributed using orchestration software such as Puppet or Chef to update each host's hsflowd configuration file, /etc/hsflowd.conf, and restart the daemon.

The hsflowd daemon writes configuration information that it receives via DNS-SD or through its configuration file to the /etc/ file. Other sFlow agents running on the host automatically detect changes to the /etc/ file and apply the configuration settings.

The sFlow protocol allows each agent to operate autonomously and send an independent stream of metrics to the sFlow collector. Distributing monitoring among the agents eliminates dependencies and synchronization challenges that would increase the complexity of the agents.

Each of the sFlow agents is responsible for exporting a different set of metrics:
  • hsflowd The Host sFlow daemon exports CPU, memory, disk and network IO performance metrics and can also export per virtual machine statistics when run on a Xen, XenServer or KVM hypervisor. In addition, traffic monitoring using iptables/LOG is supported on Linux platforms.
  • Open vSwitch The Open vSwitch is "the default switch in XenServer 6.0, the Xen Cloud Platform and also supports Xen, KVM, Proxmox VE and VirtualBox. It has also been integrated into many virtual management systems including OpenStack, openQRM, and OpenNebula." Enabling the build-in sFlow monitoring on the virtual switch offers the same visibility as sFlow on physical switches and provides a unified end-to-end view of network performance across the physical and virtual infrastructure. The sflowovsd daemon ships with Host sFlow and is used to configure sFlow monitoring on the Open vSwitch using the ovs-vsctl command. Similar integrated sFlow support has also been demonstrated for the Microsoft extensible virtual switch that is part of the upcoming Windows 8 version of Hyper-V.
  • java The Java sFlow agent exports performance statistics about java threads, heap/non-heap memory, garbage collection, compilation and class loading.
  • httpd An sFlow agent embedded in the web server exports key performance metrics along with detailed transaction data that can be used to monitor top URLs, top Referers, top clients, response times etc. Currently, there are implementation of sFlow for Apache, NGINX, Tomcat and node.js web servers, see
  • Memcached An sFlow agent embedded in the Memcache server exports performance metrics along with detailed data on Memcache operations that can be used to monitor hot keys, missed keys, top clients etc. Currently, there is an implementation of sFlow for Memcached, see
Together the agents running on each host, along with sFlow agents embedded within the network infrastructure form an integrated monitoring system that provides a unified view view of network, system, storage and application performance that can scale to hundreds of thousands of servers.


  1. Hi,
    I am considering the sflow solution in a cloud enviroment but with a minor (?) modification. the infrastructure (hypervisor + ovs switch ) are managed by one entitity while the end users who install application (i.e. apache memcached) are a different entity. How is tis possible to expose the monitor data in a homogenous way so that users can easily fit in without let them (end-users) replicating the infrastructure's work ?

    1. The article Visibility in the cloud discusses the administrative challenges of instrumenting public/private clouds. In a private cloud where you have control over the hypervisors and the virtual machines, you would install hsflowd on the hypervisor to manage the sFlow setting in Open vSwitch (this shouldn't interfere with configuration and forwarding configuration associated with OpenFlow) and report basic CPU, memory, disk metrics for the hypervisor and VMs. You would also install hsflowd on the VMs (along with Apache, Memcached sub-agents etc). DNS-SD provides an efficient way to orchestrate settings consistently among all the hsflowd instances.