Ubuntu 18.04

Ubuntu 18.04 comes with Linux kernel version 4.15. This version of the kernel includes efficient in-kernel packet sampling that can be used to provide network visibility for production servers running network heavy workloads, see Berkeley Packet Filter (BPF).

This article provides instructions for installing and configuring the open source Host sFlow agent to remotely monitor servers using the industry standard sFlow protocol. The sFlow-RT real-time analyzer is used to demonstrate the capabilities of sFlow telemetry.

Find the latest Host sFlow version on the Host sFlow download page.

wget https://github.com/sflow/host-sflow/releases/download/v2.0.25-3/hsflowd-ubuntu18_2.0.25-3_amd64.deb
sudo dpkg -i hsflowd-ubuntu18_2.0.25-3_amd64.deb
sudo systemctl enable hsflowd

The above commands download and install the software.

sflow {
  collector { ip=10.0.0.30 }
  pcap { speed=1G-1T }
  tcp { }
  systemd { }
}

Edit the /etc/hsflowd.conf file. The above example sends sFlow to a collector at 10.0.0.30, enables packet sampling on all network adapters, adds TCP performance information, and exports metrics for Linux services. See Configuring Host sFlow for Linux for the complete set of configuration options.

sudo systemctl restart hsflowd

Restart the Host sFlow daemon to start streaming telemetry to the collector.

sflow {
  dns-sd { domain=.sf.inmon.com }
  pcap { speed=1G-1T }
  tcp { }
  systemd { }
}

Alternatively, if you have control over a DNS domain, you can use DNS SRV records to advertise sFlow collector address(es). In the above example, the sf.inmon.com domain will be queried for collectors.

sflow-rt          A       10.0.0.30
_sflow._udp   60  SRV     0 0 6343  sflow-rt

The above entries from the sf.inmon.com zone file directs the sFlow to sflow-rt.sf.inmon.com (10.0.0.30). If you change the collector, all Host sFlow agents will pick up the change within 60 seconds (the DNS time to live specified in the SRV entry).

Now that the Host sFlow agent has been configured, it's time to install an sFlow collector on server 10.0.0.30, which we will assume is also running Ubuntu 18.04.

First install Java.

sudo apt install openjdk-11-jre-headless

Next, install the latest version of sFlow-RT along with browse-metrics, browse-flows and prometheus applications.

LATEST=`wget -qO - https://inmon.com/products/sFlow-RT/latest.txt`
wget https://inmon.com/products/sFlow-RT/sflow-rt_$LATEST.deb
sudo dpkg -i sflow-rt_$LATEST.deb
sudo /usr/local/sflow-rt/get-app.sh sflow-rt browse-metrics
sudo /usr/local/sflow-rt/get-app.sh sflow-rt browse-flows
sudo /usr/local/sflow-rt/get-app.sh sflow-rt prometheus
sudo systemctl enable sflow-rt
sudo systemctl start sflow-rt

Finally, allow sFlow and HTTP requests through the firewall.

sudo ufw allow 6343/udp
sudo ufw allow 8008/tcp

System Properties describes configuration options that can be set in the /usr/local/sflow-rt/conf.d/sflow-rt.conf file. See Download and install for instructions on securing access to sFlow-RT as well as links to additional applications.

Use a web browser to connect to http://10.0.0.30:8008/ to access the sFlow-RT web interface (shown above). The Status page confirms that sFlow is being received.

curl http://10.0.0.30:8008/prometheus/metrics/ALL/ALL/txt

The above command retrieves metrics for all the hosts in Prometheus export format.

Configure Prometheus or InfluxDB to periodically retrieve and store metrics. The following examples demonstrate the use of Grafana to query a Prometheus database to populate dashboards: sFlow-RT Network Interfaces, sFlow-RT Countries and Networks, and sFlow-RT Health.

CentOS 8

CentOS 8 / RHEL 8 come with Linux kernel version 4.18. This version of the kernel includes efficient in-kernel packet sampling that can be used to provide network visibility for production servers running network heavy workloads, see Berkeley Packet Filter (BPF).

This article provides instructions for installing and configuring the open source Host sFlow agent to remotely monitor servers using the industry standard sFlow protocol. The sFlow-RT real-time analyzer is used to demonstrate the capabilities of sFlow telemetry.

Find the latest Host sFlow version on the Host sFlow download page.

wget https://github.com/sflow/host-sflow/releases/download/v2.0.26-3/hsflowd-centos8-2.0.26-3.x86_64.rpm
sudo rpm -i hsflowd-centos8-2.0.26-3.x86_64.rpm
sudo systemctl enable hsflowd

The above commands download and install the software.

sflow {
  collector { ip=10.0.0.30 }
  pcap { speed=1G-1T }
  tcp { }
  systemd { }
}

Edit the /etc/hsflowd.conf file. The above example sends sFlow to a collector at 10.0.0.30, enables packet sampling on all network adapters, adds TCP performance information, and exports metrics for Linux services. See Configuring Host sFlow for Linux for the complete set of configuration options.

sudo systemctl restart hsflowd

Restart the Host sFlow daemon to start streaming telemetry to the collector.

sflow {
  dns-sd { domain=.sf.inmon.com }
  pcap { speed=1G-1T }
  tcp { }
  systemd { }
}

Alternatively, if you have control over a DNS domain, you can use DNS SRV records to advertise sFlow collector address(es). In the above example, the sf.inmon.com domain will be queried for collectors.

sflow-rt          A       10.0.0.30
_sflow._udp   60  SRV     0 0 6343  sflow-rt

The above entries from the sf.inmon.com zone file directs the sFlow to sflow-rt.sf.inmon.com (10.0.0.30). If you change the collector, all Host sFlow agents will pick up the change within 60 seconds (the DNS time to live specified in the SRV entry).

Now that the Host sFlow agent has been configured, it's time to install an sFlow collector on server 10.0.0.30, which we will assume is also running CentOS 8.

First install Java.

sudo yum install java-11-openjdk-headless

Next, install the latest version of sFlow-RT along with browse-metrics, browse-flows and prometheus applications.

LATEST=`wget -qO - https://inmon.com/products/sFlow-RT/latest.txt`
wget https://inmon.com/products/sFlow-RT/sflow-rt-$LATEST.noarch.rpm
sudo rpm -i sflow-rt-$LATEST.noarch.rpm
sudo /usr/local/sflow-rt/get-app.sh sflow-rt browse-metrics
sudo /usr/local/sflow-rt/get-app.sh sflow-rt browse-flows
sudo /usr/local/sflow-rt/get-app.sh sflow-rt prometheus
sudo systemctl enable sflow-rt
sudo systemctl start sflow-rt

Finally, allow sFlow and HTTP requests through the firewall.

sudo firewall-cmd --permanent --zone=public --add-port=6343/udp
sudo firewall-cmd --permanent --zone=public --add-port=8008/tcp
sudo firewall-cmd --reload

System Properties describes configuration options that can be set in the /usr/local/sflow-rt/conf.d/sflow-rt.conf file. See Download and install for instructions on securing access to sFlow-RT as well as links to additional applications.

Use a web browser to connect to http://10.0.0.30:8008/ to access the sFlow-RT web interface (shown above). The Status page confirms that sFlow is being received.

curl http://10.0.0.30:8008/prometheus/metrics/ALL/ALL/txt

The above command retrieves metrics for all the hosts in Prometheus export format.

Configure Prometheus or InfluxDB to periodically retrieve and store metrics. The following examples demonstrate the use of Grafana to query a Prometheus database to populate dashboards: sFlow-RT Network Interfaces, sFlow-RT Countries and Networks, and sFlow-RT Health.

Systemd traffic marking

Monitoring Linux services describes how the open source Host sFlow agent exports metrics from services launched using systemd, the default service manager on most recent Linux distributions. In addition, the Host sFlow agent efficiently samples network traffic using Linux kernel capabilities: PCAP/BPF, nflog, and ulog.

This article describes a recent extension to the Host sFlow systemd module, mapping sampled traffic to the individual services the generate or consume them. The ability to color traffic by application greatly simplifies service discovery and service dependency mapping; making it easy to see how services communicate in a multi-tier application architecture.

The following /etc/hsflowd.conf file configures the Host sFlow agent, hsflowd, to sampling packets on interface eth0, monitor systemd services and mark the packet samples, and track tcp performance:

sflow {
  collector { ip = 10.0.0.70 }
  pcap { dev = eth0 }
  systemd { markTraffic = on }
  tcp { }
}

The diagram above illustrates how the Host sFlow agent is able to efficiently monitor and classify traffic. In this case both the Host sFlow agent and an Apache web server are are running as services managed by systemd. A network connection , shown in red, to the HTTP service. In this case, configuring the pcap module to monitor interface eth0 on the server programs a Berkeley Packet Filter (BPF) that randomly samples packets in the Linux kernel and provides copies (shown as the dotted red line) to the Host sFlow agent. In addition, the Host sFlow agent queries systemd to obtain a list of running services and the resources allocated to them. Further Linux kernel tables are queried to identify the network sockets that were opened by each service.

The Host sFlow then attaches an additional record to exported packet samples to indicate the services generating or consuming the packets:

/* Traffic source/sink entity reference */
/* opaque = flow_data; enterprise = 0; format = 2210 */
/* Set Data source to all zeroes if unknown
struct extended_entities {
 sflow_data_source_expanded src_ds;    /* Data Source associated with
                                          packet source */
 sflow_data_source_expanded dst_ds;    /* Data Source associated with
                                          packet destination */
}

Note: The data source references point to the performance metrics exported by the systemd module, see Monitoring Linux services.

Finally, enabling the tcp module adds delay, retransmit, loss, and reordering information to the sampled packet, see Network performance monitoring.

The screen capture above shows network traffic colored by service name. The chart colors traffic associated with the httpd.service in blue, remote login traffic associated with the sshd.service in red, BGP traffic associated with the bird.service in gold, and traffic to the inmsfd.service in green.

The chart was generated using the open source Flow Trend application running on the sFlow-RT real-time analytics platform. The chart is the result of the following flow definition:

host:[or:dssource:dsdestination]:vir_host_name

The host: function is used to join information from the sampled flow with telemetry reported for each of the services. Additional keys can be added to the flow definition to break out the traffic by network addresses, quality of service, or any of the many properties reported by sFlow, see Defining Flows for additional information.

Networking on the host has been referred to as the "Goldilocks Zone" because the host provides context that is unavailable in network switches and routers. The sFlow standard defines measurements that network, host, and application entities send in a continuous telemetry stream to analytics software that can combine the data to provide a comprehensive end-to-end view of activity, see sFlow Host Structures.

Using Ganglia to monitor Linux services

The screen capture from the Ganglia monitoring tool shows metrics for services running on a Linux host. Monitoring Linux services describes how the open source Host sFlow agent has been extended to export standard Virtual Node metrics from services running under systemd. Ganglia already supports these standard metrics and the article Using Ganglia to monitor virtual machine pools describes the configuration steps needed to enable this feature.

Monitoring Linux services

Mainstream Linux distributions have moved to systemd to manage daemons (e.g. httpd, sshd, etc.). The diagram illustrates how systemd runs each daemon within its own container so that it can maintain tight control of the daemon's resources.

This article describes how to use the open source Host sFlow agent to gather telemetry from daemons running under systemd.

Host sFlow systemd monitoring exports a standard set of metrics for each systemd service - the sFlow Host Structures extension defines metrics for Virtual Nodes (virtual machines, containers, etc.) that are used to export Xen, KVM, Docker, and Java resource usage. Exporting the standard metrics for systemd services provides interoperability with sFlow analyzers, allowing them to report on Linux services using existing virtual node monitoring capabilities.

While running daemons within containers helps systemd maintain control of the resources, it also provides a very useful abstraction for monitoring. For example, a single service (like the Apache web server) may consist of dozens of processes. Reporting on container level metrics abstracts away the per-process details and gives a view of the total resources consumed by the service. In addition, service metadata (like the service name) provides a useful way of identifying and grouping services, for example, making it easy to report on total CPU consumed by the web service across a pool of servers.

Systemd monitoring is easy to set up.

First download and install the latest software release.

Next, enable the systemd module by adding the highlighted line in the /etc/hsflowd.conf file:

sflow{
  collector{ ip=10.0.0.1 }
  systemd{}
}

This is a minimal configuration that sends sFlow telemetry to a collector running on host 10.0.0.1. The Host sFlow agent is capable of gathering an extensive set of network, system and application level metrics. See Configuring Host sFlow for Linux for a full set of options.

Finally, start the agent:

sudo systemctl enable hsflowd.service
sudo systemctl start hsflowd.service

For the best accuracy, enable systemd cgroup accounting by adding the following entries to the /etc/systemd/system.conf file and rebooting the server:

DefaultCPUAccounting=yes
DefaultBlockIOAccounting=yes
DefaultMemoryAccounting=yes

The Host sFlow agent will automatically detect when cgroup accounting has been enabled. However, if cgroup accounting hasn't been enabled, it is still able to compute and export statistics, although it might miss contributions from short lived processes.

Once the agents have been configured, verify that sFlow telemetry is being received at the collector using sflowtool. The simplest way to run sflowtool is using Docker:

docker run -p 6343:6343/udp sflow/sflowtool

The following output shows the statistics exported for the apache2 service:

startSample ----------------------
sampleType_tag 0:2
sampleType COUNTERSSAMPLE
sampleSequenceNo 50
sourceId 3:112270
counterBlock_tag 0:2103
vdsk_capacity 0
vdsk_allocation 0
vdsk_available 0
vdsk_rd_req 0
vdsk_rd_bytes 0
vdsk_wr_req 0
vdsk_wr_bytes 0
vdsk_errs 0
counterBlock_tag 0:2102
vmem_memory 16674816
vmem_maxMemory 0
counterBlock_tag 0:2101
vcpu_state 1
vcpu_cpu_mS 180
vcpu_cpuCount 0
counterBlock_tag 0:2002
parent_dsClass 2
parent_dsIndex 1
counterBlock_tag 0:2000
hostname apache2.service
UUID 92-53-c6-17-60-65-52-a2-ac-f7-76-cb-7b-63-d9-23
machine_type 3
os_name 2
os_release 4.4.0-45-generic
endSample   ----------------------

Install Host sFlow agents on all the hosts in the data center for comprehensive visibility.

Berkeley Packet Filter (BPF)

Linux bridge, macvlan, ipvlan, adapters discusses how industry standard sFlow technology, widely supported by data center switch vendors, has been extended to provide network visibility into the Linux data plane. This article explores how sFlow's lightweight packet sampling mechanism has been implemented on Linux network adapters.

Linux Socket Filtering aka Berkeley Packet Filter (BPF) describes the recently added prandom_u32() function that allows packets to be randomly sampled in the Linux kernel for efficient monitoring of production traffic.

Background: Enhancing Network Intrusion Detection With Integrated Sampling and Filtering, Jose M. Gonzalez and Vern Paxson, International Computer Science Institute Berkeley, discusses the motivation for adding random sampling BPF and the email thread [PATCH] filter: added BPF random opcode describes the Linux implementation and includes an interesting discussion of the motivation for the patch.

The following code shows how the open source Host sFlow agent implements random 1-in-256 packet sampling as a BPF program:

ld rand
mod #256
jneq #1, drop
ret #-1
drop: ret #0

A JIT for packet filters discusses the Linux Just In Time (JIT) compiler for BFP programs, delivering native machine code performance for compiled filters.

Minimizing cost of visibility describes why low overhead monitoring is an essential component for increasing efficiency in cloud infrastructure. The combination of BPF packet sampling with standard sFlow export provides a low overhead method of delivering real-time network visibility into large scale cloud infrastructure.

Network visibility with Docker

Microservices describes the critical role that network visibility provides as a common point of reference for monitoring, managing and securing the interactions between the numerous and diverse distributed service instances in a microservices deployment.

Industry standard sFlow is well placed to give network visibility into the Docker infrastructure used to support microservices. The sFlow standard is widely supported by data center switch vendors (Cisco, Arista, Juniper, Dell, HPE, Brocade, Cumulus, etc.)  providing a cost effective and scaleable method of monitoring the physical network infrastructure. In addition, Linux bridge, macvlan, ipvlan, adapters described how sFlow is also an efficient means of leveraging instrumentation built into the Linux kernel to extend visibility into Docker host networking.

The following commands build the Host sFlow binary package from sources on an Ubuntu 14.04 system:

sudo apt-get update
sudo apt-get install build-essential
sudo apt-get install libpcap-dev
sudo apt-get install wget
wget https://github.com/sflow/host-sflow/archive/v1.29.1.tar.gz
tar -xvzf v1.29.1.tar.gz
cd host-sflow-1.29.1
make DOCKER=yes PCAP=yes deb

This resulting hsflowd_1.29.1-1_amd64.deb package can be copied and installed on all the hosts in the Docker cluster using configuration management tools such as Puppet, Chef, Ansible, etc.

This article will explore the alternative of deploying sFlow agents as Docker containers.

Create a directory for the project and edit the Dockerfile:

mkdir hsflowd
cp hsflowd_1.29.1-1_amd64.deb hsflowd
cd hsflowd
printf "sflow {\n dnssd=on\n pcap { dev = docker0 }\n}" > hsflowd.conf
vi Dockerfile

Add the following contents to Dockerfile:

FROM   ubuntu:trusty
RUN    apt-get update && apt-get install -y libpcap0.8 docker.io
ADD    hsflowd_1.29.1-1_amd64.deb /tmp
RUN    dpkg -i /tmp/hsflowd_1.29.1-1_amd64.deb
ADD    hsflowd.conf /etc/hsflowd.conf
CMD    /etc/init.d/hsflowd start && tail -f /dev/null

Build the project:

docker build -t hsflowd .

Run the service:

docker run --pid=host --uts=host --net=host \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /sys/fs/cgroup/:/sys/fs/cgroup/:ro -d hsflowd

In this example, DNS Service Discovery (DNS-SD), is being used as the configuration method for the sFlow agents. Adding the following entry to DNS zone file allows the agents to automatically discover the designated sFlow analyzers, analytics1 and analytics2, and configuration parameters:

_sflow._udp   30  SRV     0 0 6343  analytics1
_sflow._udp   30  SRV     0 0 6343  analytics2
_sflow._udp   30  TXT     (
"txtvers=1"
"sampling=400"
"polling=20"
)

As soon as the container starts, the sFlow agent will make a DNS request to find the sFlow analyzers, which can themselves be packaged as Docker containers. Network and system analytics as a Docker microservice describes how sFlow analytics can be packaged as a RESTful service and integrated with a wide variety of on-site and cloud, orchestration, DevOps and Software Defined Networking (SDN) tools.

Any change to the entries in the zone file will be automatically picked up by the sFlow agents.

The agent has been configured for Docker bridged networking, monitoring traffic through bridge docker0. For macvlan or ipvlan networking, change the pcap setting from docker0 to eth0.

One of the major advantages of packaging the sFlow agents and analytics components as Docker containers is that large scale deployments can be automated using Docker Compose with Swarm, deploying sFlow agents on every node in the Swarm cluster to deliver real-time cluster-wide visibility into the resource consumption and communication patterns of all microservices running on the cluster.

Docker network visibility demonstration

The 2 minute live demonstration shows how the open source Host sFlow agent can be used to efficiently monitor Docker networking in production environments. The demonstration shows real-time tracking of 30Gbit/s traffic flows using less than 1% of a single processor core.

Linux bridge, macvlan, ipvlan, adapters

The open source Host sFlow project added a feature to efficiently monitor traffic on Linux host network interfaces: network adapters, Linux bridge, macvlan, ipvlan, etc. Implementation of high performance sFlow traffic monitoring is made possible by the inclusion of random packet sampling support in the Berkeley Packet Filter (BPF) implementation in recent Linux kernels (3.19 or later).

In addition to the new BPF capability, hsflowd has a couple of other ways to monitor traffic:

• iptables, add a statistic rule to the iptables firewall to add traffic monitoring
• Open vSwitch, has built-in sFlow instrumentation that can be configured by hsflowd.

The BPF sampling mechanism is less complex to configure than iptables and can be used to monitor any Linux network device, including: network adapters (e.g. eth0) and the Linux bridge (e.g. docker0). Monitoring a network adapter also provides visibility into lightweight macvlan and ipvlan network virtualization technologies that are likely to become more prevalent in the Linux container ecosystem, see Using Docker with macvlan Interfaces.

The following commands build and install hsflowd on an Ubuntu 14.03 host:

sudo apt-get update
sudo apt-get install build-essential
sudo apt-get install libpcap-dev
sudo apt-get install git
git clone https://github.com/sflow/host-sflow
cd host-sflow
make
sudo make install

Installing Host sFlow on a Linux server provides basic instructions for configuring the Host sFlow agent (hsflowd). To monitor traffic on the host, edit the /etc/hsflowd.conf file configure the sFlow collector and enable packet sampling on eth0

pcap { dev = eth0 }

Now start the daemon:

sudo hsflowd start

At this point packet traversing eth0 will be sampled and sent out as part of the standard sFlow telemetry stream sent to an sFlow analyzer. For example, using sFlow-RT with the top-flows application as the sFlow analyzer generated the top flows table below.

There a numerous server monitoring agents available in the open source community that will export similar host statistics (CPU, memory, disk) to the Host sFlow agent. Host sFlow differs by also including network traffic visibility using the same packet sampling mechanism supported by most data center switches. Significant advances are extending the visibility into the physical network, for example, Broadcom BroadView Instrumentation tracks buffer utilization and microbursts that effect application performance.

A common standard for monitoring physical and virtual network and server infrastructure reduces operational complexity. Visibility into network activity is critical to understanding the performance of scale out applications that drive large amounts of East-West traffic. Host sFlow, along with support for sFlow in the physical network, delivers scaleable data center wide telemetry to SDN and DevOps tools so that they can better orchestrate the allocation of resources to maximize performance and reduce costs.

Podcast with Nick Buraglio and Brent Salisbury

"Have you seen sFlow options in your router configuration or flow collector? Are you looking for alternatives to SNMP or NetFlow? Have you been curious about the instrumentation of your new white box or virtual switch? Yes? Then you will probably enjoy learning more about sFlow!"

Non-Blocking #1: SFlow With Peter Phaal Of InMon And SFlow.Org is a discussion between Brent Salisbury (networkstatic.net), Nick Buraglio (forwardingplane.net), and Peter Phaal (blog.sflow.com).

Web sites and tools mentioned in the podcast:

1. sFlow.org
2. Devices that support sFlow
3. Software to analyze sFlow
4. sFlow.org mailing list
5. sFlow structures
6. blog.sflow.com (incorrectly referenced as blog.sflow.org in the podcast)
7. Host sFlow
8. sflowtool

The podcast touches on a number of topics that have been explored in greater detail on this blog. The topics are listed in roughly the order they are mentioned in the podcast:

1. Widespread support for sFlow among switch vendors
2. Disaggregated flow cache
3. ULOG
4. Push vs Pull
5. sFlow vs SNMP for interface counters
6. Broadcom ASIC table utilization metrics, DevOps, and SDN
7. Broadcom BroadView Instrumentation
8. Rapidly detecting large flows, sFlow vs. NetFlow/IPFIX
9. SDN and large flows
10. Probes
11. Packet headers
12. Network virtualization visibility demo
13. History of sFlow
14. Standards
15. Open vSwitch performance monitoring
16. Wireless
17. Prescriptive vs descriptive standards (sFlow / IPFIX)
18. RMON (4 groups)
19. Observability
20. Host sFlow distributed agent
21. Host sFlow data model
22. Multi-tenant traffic in virtualized network environments
23. Workload placement
24. SDN router using merchant silicon top of rack switch
25. White box Internet router PoC
26. Active Route Manager
27. Leaf and spine traffic engineering using segment routing and SDN
28. CORD: Open-source spine-leaf Fabric (demo from 2015 Open Networking Summit)
29. sflowtool
30. sflowtool for packet caputure
31. sflowtool with Wireshark

Stop thief!

The Host-sFlow project recently added added CPU steal to the set of CPU metrics exported.

steal (since Linux 2.6.11)
       (8) Stolen time, which is the time spent in other operating systems
       when running in a virtualized environment

Keeping close track of the stolen time metric is particularly import when running managing virtual machines in a public cloud. For example, Netflix and Stolen Time includes the discussion:

So how does Netflix handle this problem when using Amazon’s Cloud? Adrian admits that they tracked this statistic so closely that when an instance crossed a stolen time threshold the standard operating procedure at Netflix was to kill the VM and start it up on a different hypervisor. What Netflix realized over time was that once a VM was performing poorly because another VM was crashing the party, usually due to a poorly written or compute intensive application hogging the machine, it never really got any better and their best learned approach was to get off that machine.

The following articles describe how to monitor public cloud instances using Host sFlow agents:

• Amazon Elastic Compute Cloud (EC2)
• Rackspace cloudservers

The CPU steal metric is particularly relevant to Network Function Virtualization (NFV). Virtual appliances implementing network functions such as load balancing are particularly sensitive to stolen CPU cycles that can severely impact application response times. Application Delivery Controller (ADC) vendors export sFlow metrics from their physical and virtual appliances - sFlow leads convergence of multi-vendor application, server, and network performance management.  The addition of CPU steal to the set of sFlow metrics exported by virtual appliances will allow the NFV orchestration tools to better optimize service pools.

Dell, Cumulus, Open Source, Open Standards, and Unified Management

On Thursday, at Network Field Day 7, Arpit Joshipura described Dell's networking strategy. He started by polling the delegates to see which topics were most on their mind.

The first topic raised by many of the delegates was the recently announced Dell/Cumulus partnership (listed as Open NW on the white board), see Dell Unlocks New Era for Open Networking, Decouples Hardware and Software. Next on the list was an interest in Dell's Open Source networking strategy, understanding Dell's Differentiation strategy, and plans for L3.

Dell's open networking strategy is described at time marker 14:55 in the video. Dell was one of the first vendors to move to merchant silicon, now they are opening up the switch platform, allowing customers to choose from standard merchant silicon based switch platforms (Broadcom, Intel) and switch software (currently FTOS / Cumulus).

Arpit suggests that customers will choose Cumulus Linux as the operating system for the layer 3 features and because they can use the same expertise and tools (Puppet, Chef etc.) to manage Linux servers and the switches connecting them. He also suggested that customers would choose FTOS for legacy networks and layer 2 features. Support for the Open Networking Install Environment (ONIE) allows customers to load different switch operating systems on the hardware. This is the same model as Dell uses when selling servers, allowing customers to choose hardware (Intel/AMD), software (Windows, SUSE, Red Hat), and obtain support from Dell. Arpit summarizes the strategy, "Michael Dell did this on PCs, he did it on servers and I think we are in the best position to do it for networking."

The recent talk, It Ain't Software Defined until you Unbundle the Platform, by JR Rivers Co-Founder/CEO of Cumulus Networks, at the Silicon Valley Software Defined Networking Group captures the vision. While the number of hardware and software choices is currently limited, both the Dell and Cumulus talks are clear that Cumulus Linux is the first of many software choices, other likely future candidates include: Broadcom Fastpath, Big Switch's Switch Light Linux, Pluribus OpenNetvisor, Pica8 PicOS, etc. On the hardware front, expect a greater variety of switching platforms, ranging from familiar top of rack configurations to others that look more like servers with the CPU and memory resources to implement application functions like content distribution, caching, load balancing etc.

Open switching platforms and merchant silicon are part of a set of accelerating trends that are driving toward a common set of standards and APIs that deliver the data center wide visibility and control needed to deliver agile, self optimizing, software defined data centers - see Drivers for growth.

Metric export to Graphite

Figure 1: Cluster performance metrics

Cluster performance metrics describes how sFlow-RT can be used to calculate summary metrics for cluster performance. The article includes a Python script that polls sFlow-RT's REST API and then sends metrics to to Graphite. In this article sFlow-RT's internal scripting API will be used to send metrics directly to Graphite.

Figure 2: Components of sFlow-RT

The following script (graphite.js) re-implements the Python example (generating a sum of the load_one metric for a cluster of Linux machines) in JavaScript using sFlow-RT built-in functions for retrieving metrics and sending them to Graphite:

// author: Peter
// version: 1.0
// date: 11/23/2013
// description: Log metrics to Graphite

include('extras/json2.js');

var graphiteServer = "10.0.0.151";
var graphitePort = null;

var errors = 0;
var sent = 0;
var lastError;

setIntervalHandler(function() {
  var names = ['sum:load_one'];
  var prefix = 'linux.';
  var vals = metric('ALL',names,{os_name:['linux']});
  var metrics = {};
  for(var i = 0; i < names.length; i++) {
    metrics[prefix + names[i]] = vals[i].metricValue;
  }
  try { 
    graphite(graphiteServer,graphitePort,metrics);
    sent++;
  } catch(e) {
    errors++;
    lastError = e.message;
  }
} , 15);

setHttpHandler(function() {
  var message = { 'errors':errors,'sent':sent };
  if(lastError) message.lastError = lastError;
  return JSON.stringify(message);
});

The interval handler function runs every 15 seconds and retrieves the set of metrics in the names array (in this case just one metrics, but multiple metrics could be retrieved). The names are then converted into a Graphite friendly form (prefixing each metric with the token linux. so that they can be easily grouped) and then sent to the Graphite collector running on 10.0.0.151 using the default TCP port 2003. The script also keeps track of any errors and makes them available through the URL /script/graphite.js/json

The following command line argument loads the script on startup:

-Dscript.file=graphite.js

The following Graphite screen capture below shows a trend of the metric:

There are a virtually infinite number of core and derived metrics that can be collected by sFlow-RT using standard sFlow instrumentation embedded in switches, servers and applications throughout the data center. For example Packet loss describes the importance of collecting network packet loss metrics and including them in performance dashboards.

Figure 2: Visibility and the software defined data center

While having access to all these metrics is extremely useful, not all of them need to be stored in Graphite. Using sFlow-RT to calculate and selectively export high value metrics reduces pressure on the time series database, while still allowing any of the remaining metrics to be polled using the REST API when needed.

Finally, metrics export is only one of many applications for sFlow data, some of which have been described on this blog. The data center wide visibility provided by sFlow-RT supports orchestration tools and allows them to automatically optimize the allocation of compute, storage and application resources and the placement of loads on these resources.

Velocity Conference talk

This talk from the O'Reilly 2012 Velocity Conference presents a case study describing how Tagged.com uses sFlow to monitor their application infrastructure.

Some of the topics covered include:

• Introduction to sFlow
• Using Ganglia as an sFlow collector
• Monitoring Apache web tier
• Monitoring Memcache clusters
• Monitoring Java applications
• Using sflowtool to develop custom tools

The recently published O'Reilly book, Monitoring with Ganglia, includes the Tagged case study and a chapter on configuring Ganglia to collect sFlow metrics. 

Thread pools

Figure 1: Thread pool

The thread pool pattern, illustrated in figure 1, is common to many parallel processing applications. A number of worker threads, organized in a thread pool, take tasks from a task queue. Once a thread has completed a task, it waits for a new task to appear on the task queue. Keeping track of the number of active threads in the pool is essential. If tasks wait in the queue because there aren't enough workers then requests will be delayed, or possibly dropped if the queue fills up.

The recently finalized sFlow Application Structures specification defines a standard set of metrics for reporting on thread pools:

• Active Threads The number of threads in the thread pool that are actively processing a request.
• Idle Threads The number of threads in the thread pool that are waiting for a request.
• Maximum Threads The maximum number of threads that can exist in the thread pool.
• Delayed Tasks The number of tasks that could not be served immediately, but spent time in the task queue.
• Dropped Tasks The number of tasks that were dropped because the task queue was full.

The Apache web server uses a thread pool and is a useful demonstration of the value of the sFlow thread pool metrics. The Apache thread pool can be accessed using mod_status, which makes the thread pool visible as a web page. The following screen capture shows the server-status page generated by mod_status:

The grid of characters is used to visualize the the state of the pool (referred to as the "scoreboard"), each cell in the grid represents a slot for a thread and the size of the grid shows the maximum number of threads that are permitted in the pool. The summary line above the grid states that 6 requests are currently being processed and that there are 69 idle workers (i.e. there are six "W" characters and sixty nine "_" characters in the grid).

While the server-status page isn't designed to be machine readable, the information is critical and there are numerous performance monitoring tools that make HTTP requests and extract the worker pool statistics from the text. A much more efficient way to retrieve the information is to use the Apache sFlow module, which in addition to reporting the thread pool statistics will export HTTP counters, URLs, response times, status codes, etc.

The article, Using Ganglia to monitor web farms, describes how to use the open source Ganglia performance monitoring software to collect and report on web server clusters using sFlow. Ganglia now includes support for the sFlow thread pool metrics.

Figure 2: Ganglia chart showing active threads from an Apache web server

Figure 2 trends the number of active workers in the pool. If the number of active workers approaches the maximum allowed, then additional servers may need to be added to the cluster. An increase in active threads could also indicate a performance problem with backend systems (a slow database holding up worker threads) or may be the result of a Denial of Service (DoS) attack (e.g. Sloloris).

Monitoring thread pools using sFlow is very useful, but only scratches the surface of what is possible. The sFlow standard is widely support be network equipment vendors and can be combined with sFlow metrics from hosts, services and applications to provide a comprehensive view of data center performance.

Packets and Flows

Figure 1: Sending a picture over a packet switched network

Figure 1 illustrates how data is transferred over a packet switched network. Host A is in the process of transferring a picture to Host B. The picture has been broken up into parts and each part is sent as a separate packet. Three packets containing parts 8, 9 and 10 are in transit and are in the process of being forwarded by switches Z, Y and X respectively.

In the example, Host A is responsible for breaking up the picture into parts and transmitting the packets. Host B is responsible for re-constructing the picture, detecting parts that are missing, corrupted, or delivered out of order and sending acknowledgement packets back to Host A, which is then responsible for resending packets if necessary.

The packet switches are responsible for transmitting, or forwarding, the packets. Each packet switch examines the destination address (e.g. To: B) and sends the packet on a link that will take the packet closer to its destination. The switches are unaware of the contents of the packets they forward, in this case the picture fragments and part numbers. Figure 1 only shows packets relating to the image transfer between Host A and Host B, but in reality the switches will be simultaneously forwarding packets from many other hosts.

Figure 2: Sorting mail

The mail sorting room shown in Figure 2 is a good analogy for the function performed by a packet switch. Letters arrive in the sorting room and are quickly placed into pigeon holes based on destination. The mail sorters don't know or care what's in the letters, they are focused on quickly reading the destination address on each envelope and placing the letter in a pigeon hole along with other letters to the same region so that the letters can be sent to another sorting facility closer to the destination.

In a packet switched network, each host and switch has a different perspective on data transfers and maintains different state in order to perform its task. Managing the performance of the communication system requires a correct understanding of the nature of the task that each element is responsible for and a way to monitor how effectively that task is being performed.

As an example of a poorly fitting model, consider the concept of "flow records" that are often presented as an intuitive way to monitor and understand traffic on packet switched networks. Continuing our example, the data transfer would be represented by two flow records, one accounting for packets from Host A to Host B and another accounting for packets from Host B to Host A.

Figure 3: Telephone operator

There is an inherent appeal in flow records since they are similar to the familiar "call records" that you see on a telephone bill, recording the number dialed, the time the call started and the call duration. However, as the switchboard and patch cords demonstrate in Figure 3, telephones networks are circuit switched, i.e. a dedicated circuit is set up between the two telephones involving all the switches in the path. It is easy to see how a circuit switch might easily generate call records by considering the manual operator. The operator just needs to note when they connected the call using the patch cord, who they connected to, and when they terminated the call by pulling the plug.

Viewing packet switches through the lens of a circuit oriented measurement is misleading. Start by considering the steps that the mail sorter in Figure 2 would have to go through in order to create flow records. The mail sorter would be requiring to keep track of the From: and To: address information on each letter, count the number letters that Host A sent to Host B, open the letters and peek inside to decide whether the letter was part of an existing conversation or the start of a new conversation. This task is extremely cumbersome and error prone, and the resulting flow records don't monitor the task that the mail sorter is actually performing; for example, flow records won't tell you how many postcards, letters and parcels were sorted.

Packet and circuit switched networks have very different characteristics and an effective monitoring system will collect measurements that are relevant to the performance of the network:

• Circuit switches have a limited number of connections that they can handle and if more calls are attempted, calls are blocked (i.e. receive a busy signal). Blocking probabilities and sizing of circuit switches are analyzed using Erlang calculations.
• Packet switches don't block. Instead packets are interleaved as they are forwarded. If the number of packets arriving exceeds the forwarding capacity of the switch, then packets may be delayed as they wait to be serviced, or be discarded if there are too many packets already waiting in the queue. Queuing delays and packet discard probabilities are analyzed using queuing theory.

To make the example in Figure 1 concrete, make Host A is an Apache web server, Host B a laptop running a web browser, and the picture transfer a response associated with the HTTP request http://www.lolcats.com/popular/159.html

The following table compares the switch, host and application measurements provided by sFlow and NetFlow:

	sFlow	NetFlow
Switch	Each switch exports packet oriented measurements, exporting interface counters and randomly sampled packet headers and associated forwarding decisions.	Switches exports connection oriented flow records that include source address, destination address, protocol, bytes, packets and duration. Note: Many switches aren't capable of making these measurements and so go unmonitored.
Host	The server exports standard host metrics, including: CPU, memory and disk performance.	None. NetFlow is generally only implemented in network devices.
Application	The web server exports standard HTTP metrics that include request counts and randomly sampled web requests, providing detailed information such as the URL, referrer, server address, client address, user, browser, request bytes, response bytes, status and duration. The web server also reports maximum, idle and active workers.	None. NetFlow is typically only implemented in network devices.

NetFlow takes a network centric view of measurement and tries to infer application behavior by examining packets in the network. NetFlow imposes a stateful, connection oriented, model on core devices that should be stateless. Unfortunately, the resulting flow measurements aren't a natural fit for packet switches, providing a distorted view of the operation of these devices. For example, the switch makes forwarding decisions on a packet by packet basis and these decisions can change over the lifetime of a flow. The packet oriented measurements made by sFlow accurately capture forwarding decisions, but flow oriented measurement can be misleading. Another example building on the mail sorting analogy in Figure 2; packet oriented measurements support analysis of small, large and jumbo frames (postcards, letters and parcels), but this detail is lost in flow records.

Flows are an abstraction that is useful for understanding end-to-end traffic traversing the packet switches. However, the flow abstraction describes connections created by the communication end points and to properly measure connection performance, one needs to instrument those end points. Hosts are responsible for initiating and terminating flows and are a natural place to report flows, but the traditional flow model ignores important detail that the host can provide. For example, the host is in a position to include important details about services such as user names, URLs, response times, and status codes as well as information about the computational resources needed to deliver the services; information that is essential for managing service capacity, utilization and response times.

While NetFlow is network centric and tries to infer information about applications from network packets (which is becoming increasingly difficult as more traffic is encrypted), the sFlow standard takes a systems approach, exposing information from network, servers and applications in order to provide a comprehensive view of performance.

Measurement isn't simply about producing pretty charts. The ultimate goal is to be able to act on the measurements and control performance. Control requires a model of behavior that allow performance to be predicted and measurements that characterize demand and show how close the system performance matches the predictions. The sFlow standard is well suited to automation, providing comprehensive measurements based on models of network, server and application performance. The Data center convergence, visibility and control presentation describes the critical role that measurement plays in managing costs and optimizing performance.

Host sFlow progress

The open source Host sFlow project has made significant progress over the last two years. The rapidly increasing number of downloads reflects the projects maturity and the unique functionality delivered by the sFlow standard in unifying network, server and application performance monitoring of large scale cloud environments.

Starting with Linux, the Host sFlow project has added support for FreeBSD, Solaris and Windows. Installed on hypervisors, Host sFlow reports on the performance of the hypervisor and all the virtual machines. Currently supported hypervisors include: Microsoft Hyper-V, Citrix XenServer, Xen Cloud Platform (XCP), KVM and libvirt. In addition, the Host sFlow agent is extensible and a growing number of projects implement sFlow instrumentation in virtual switches (Open vSwitch, Hyper-V extensible virtual switch), popular web servers (Apache, NGINX), Memcached, application servers (Java, Tomcat) and even in-house applications written in scripting languages (PHP, Python, Ruby, Perl).

Popular open source projects such as Ganglia and Graphite offer scalable collection and reporting of sFlow metrics from large scale compute, virtual machine pools, web farms, Java application servers and Memcache clusters.

Deploying Host sFlow agents on servers extends the sFlow monitoring built into networking devices from leading vendors, including: IBM, HP, Dell, Cisco, Juniper, Brocade, F5, Alcatel-Lucent, Arista Networks, Allied Telesis, Extreme Networks, Fortinet, Hitachi, Huawei, NEC, ZTE and others. The combination of sFlow in servers and switches delivers integrated, end-to-end visibility into cloud computing, software defined networking and converged storage.

GPU performance monitoring

NVIDIA's Compute Unified Device Architecture (CUDA™) dramatically increases computing performance by harnessing the power of the graphics processing unit (GPU). Recently, NVIDIA published the sFlow NVML GPU Structures specification, defining a standard set of metrics for reporting GPU health and performance, and extended the Host sFlow agent to export the GPU metrics.

The following displays the sFlow metrics using sflowtool, the GPU metrics are highlighted:

[pp@test] /usr/local/bin/sflowtool
startDatagram =================================
datagramSourceIP 10.0.0.150
datagramSize 512
unixSecondsUTC 1346360234
datagramVersion 5
agentSubId 100000
agent 10.0.0.150
packetSequenceNo 1
sysUpTime 3000
samplesInPacket 1
startSample ----------------------
sampleType_tag 0:2
sampleType COUNTERSSAMPLE
sampleSequenceNo 1
sourceId 2:1
counterBlock_tag 0:2001
adaptor_0_ifIndex 1
adaptor_0_MACs 1
adaptor_0_MAC_0 000000000000
adaptor_1_ifIndex 2
adaptor_1_MACs 1
adaptor_1_MAC_0 e0cb4e98f891
adaptor_2_ifIndex 3
adaptor_2_MACs 1
adaptor_2_MAC_0 e0cb4e98f890
counterBlock_tag 0:2005
disk_total 145102770176
disk_free 46691696640
disk_partition_max_used 76.06
disk_reads 477615
disk_bytes_read 13102692352
disk_read_time 2227298
disk_writes 2370522
disk_bytes_written 193176428544
disk_write_time 445531146
counterBlock_tag 0:2004
mem_total 12618829824
mem_free 2484174848
mem_shared 0
mem_buffers 971259904
mem_cached 8214761472
swap_total 12580810752
swap_free 12580810752
page_in 6400433
page_out 94324428
swap_in 0
swap_out 0
counterBlock_tag 5703:1
nvml_device_count 1
nvml_processes 0
nvml_gpu_mS 0
nvml_mem_mS 0
nvml_mem_bytes_total 6441598976
nvml_mem_bytes_free 6429614080
nvml_ecc_errors 0
nvml_energy_mJ 74569
nvml_temperature_C 54
nvml_fan_speed_pc 30
counterBlock_tag 0:2003
cpu_load_one 0.040
cpu_load_five 0.240
cpu_load_fifteen 0.350
cpu_proc_run 0
cpu_proc_total 229
cpu_num 8
cpu_speed 1600
cpu_uptime 896187
cpu_user 21731800
cpu_nice 120230
cpu_system 5686620
cpu_idle 2844149774
cpu_wio 2992230
cpuintr 570
cpu_sintr 222180
cpuinterrupts 166594944
cpu_contexts 266986130
counterBlock_tag 0:2006
nio_bytes_in 0
nio_pkts_in 0
nio_errs_in 0
nio_drops_in 0
nio_bytes_out 0
nio_pkts_out 0
nio_errs_out 0
nio_drops_out 0
counterBlock_tag 0:2000
hostname test0
UUID 00000000000000000000000000000000
machine_type 3
os_name 2
os_release 2.6.35.14-106.fc14.x86_64
endSample   ----------------------
endDatagram   =================================

Note: Currently only the Linux version of Host sFlow includes the GPU support and the agent needs to be compiled from sources on a system that includes the NVML library.

The inclusion of GPU metrics in Host sFlow offers an extremely scaleable, lightweight solution for monitoring compute cluster performance. In addition to exporting a comprehensive set of standard performance metrics, the Host sFlow agent also offers a convenient API for exporting custom application metrics.

The sFlow standard isn't limited to monitoring compute resources; most network switch vendors include sFlow support, providing detailed visibility into cluster communication patterns and network utilization. Combining sFlow from switches, servers and applications delivers a comprehensive view of cluster performance.

System boundary

Credit: Wikimedia

A critical step in developing a control strategy for a system is deciding where to draw the system boundary. For example, in the data center, network management typically draws the boundary to include all the networking devices and their interconnections. The boundary is drawn to include the access switch ports, but exclude the servers and network adapters. System management is the mirror image, drawing the boundary to include servers and network adapters and exclude the network.

Given the system boundary drawn around the network, network planning and design treats the demand on the network as immutable and focusses on creating a network topology that will handle projected demand. Similarly, system capacity management is concerned with providing enough computational resources to handle application workloads while largely ignoring the network - treating it as a uniform "cloud" that will support the whatever demand is placed on it.

The move to scale-out applications, virtualization and convergence aims to improve efficiency by creating a flexible, scalable computing infrastructure that can adapt to changing demand. However, these architectural changes challenge the traditional separation between network and system management. For example, the following diagram shows how moving a virtual machine can dramatically alter network traffic.

Figure 1 Moving a virtual machine can increase network traffic

Changes in network traffic patterns can also have a profound effect on application performance. With convergence, networked storage increases demand for bandwidth and also makes applications sensitive to network congestions.

Redrawing the system boundary to include the network, servers, storage and applications provides the increased span of control needed to deliver efficient cloud services. For example, instead of treating the demand on the network as immutable, it becomes possible to move virtual machines in order to reshape network traffic patterns.

Figure 2 Moving a virtual machine to reduce network traffic

Generally, moving computation closer to data improves application performance by reducing latency and increasing bandwidth as well as reducing the overall load on the network. For example, the Hadoop distributed computing software is rack aware, since "network traffic between different nodes with in the same rack is much more desirable than network traffic across the racks."

The sFlow standard is well suited to automation, providing a unified measurement system that includes network, system and application performance. The Data center convergence, visibility and control presentation describes the critical role that measurement plays in managing costs and optimizing performance.

Windows Server 8 beta

The public beta of Windows Server 8 is now available for download. Windows Server 8 contains significant enhancements to support virtualization and cloud computing. This article describes how sFlow standard monitoring can be installed and configured to manage the performance of large scale Windows Server 8 cloud deployments.

With sFlow monitoring, each server sends performance measurements to a central analysis application - in this example the sFlow analyzer has address 10.0.0.50 and each Windows 8 server will be configured to send sFlow to this address.

First download the hsflowd-win8-X.XX.X-x64.msi file from sflow.net. Double click on the file to launch the installer:

Click on the Next button.

Click the check box to accept the license and then click on the Next button.

Click the Next button to accept the default installation location.

Enter the IP address of the sFlow collector, a Counter polling interval and a Sampling rate.  In this case the sFlow collector has address 10.0.0.50. The default polling interval of 20 seconds and default sampling rate of 1 in 256 should be suitable for most installations, see Sampling rates for more information on choosing sampling and polling setting.

Click on the Next button to accept the settings.

Click on the Install button to start the installation. After a few moments you will be prompted to install the network monitoring driver, see Hyper-V extensible virtual switch.

Click to check the Always trust software from "InMon Corp." box and click on the Install button to install the driver. Once the installation is complete, you should see the following window.

Click on the Finish button to complete the installation.

There is one final step to configure monitoring of the virtual switch. On the Start screen, type hyper-v manager to launch the Hyper-V Manager application.

Click on the Virtual Switch Manager... in the Actions column on the right to launch the Virtual Switch Manager.

Expand each virtual switch, click on Extensions and enable the sFlow packet sampling filter (in this case for vSwitch1). At this point the sFlow software installation and configuration steps are complete and you should start seeing data in your sFlow analyzer.

Use the Registry Editor if you need to change the sFlow configuration settings: the IP address of the sFlow analyzer, the polling interval, or the packet sampling rate. On the Start screen, type regedit to launch the Registry Editor.

Double-click on a setting to edit its value. Once the settings have been changed, the sFlow agent needs to be restarted before the changes will take effect.

On the Start screen type services to run the Services manager.

Select the Host sFlow Agent service and click on the Restart link (or use the right button menu option).

The following charts use the free sFlowTrend tool to demonstrate some of the performance metrics reported from Windows Server 8. However, there are many other open source and commercial sFlow analysis applications, see sFlow Collectors.

The following chart shows the top network connections flowing through the virtual switch.

It is easy to see that the bulk of the traffic is associated with Windows file sharing. The chart provides a minute-by-minute view of network activity based on measurements made by the driver installed in the virtual switch.

The next chart shows the CPU utilization on the server.

The processor utilization is averaging at just under 15%, with a brief peak of 40%. Based on this chart, it looks like there is plenty of capacity to run additional virtual machines.

The final chart in this example shows disk activity for the server and each virtual machine.

The physical servers and the virtual machines are all shown, along with key disk performance and usage metrics. Again, the numbers suggest that additional virtual machines could be added without impacting performance - the disk is only 26% used, there are only 3 virtual machines deployed, and there is very little disk IO activity.

The sFlow standard is widely supported by switch vendors. Using sFlow to monitor the Windows Server 8 extensible virtual switch simplifies management by allowing a common set of tools to monitor physical and virtual network performance. Performance management is further simplified since sFlow also provides an efficient way of monitoring server and virtual machine statistics. Unifying network and system monitoring provides the integrated view of performance needed for the coordinated management that is essential in virtualized environments. In addition to monitoring data center resources, sFlow can also be used to monitor performance within public clouds (see Rackspace cloudservers and Amazon EC2), providing the integrated visibility needed to optimize workload placement and minimise operating costs in hybrid cloud environments.

For public or private cloud operators, support for sFlow in the Hyper-V extensible switch and the Open vSwitch embeds visibility in leading commercial hypervisors (Microsoft Hyper-V and Citrix XenServer), open source hypervisors (Xen Cloud Platform (XCP), Xen, KVM, Proxmox VE and VirtualBox) and cloud management systems (OpenStack, OpenQRM and OpenNebula), providing the scalable visibility needed for accounting and billing, operational control and cost management.