Berkeley Packet Filter (BPF)

Linux bridge, macvlan, ipvlan, adapters discusses how industry standard sFlow technology, widely supported by data center switch vendors, has been extended to provide network visibility into the Linux data plane. This article explores how sFlow's lightweight packet sampling mechanism has been implemented on Linux network adapters.

Linux Socket Filtering aka Berkeley Packet Filter (BPF) describes the recently added prandom_u32() function that allows packets to be randomly sampled in the Linux kernel for efficient monitoring of production traffic.

Background: Enhancing Network Intrusion Detection With Integrated Sampling and Filtering, Jose M. Gonzalez and Vern Paxson, International Computer Science Institute Berkeley, discusses the motivation for adding random sampling BPF and the email thread [PATCH] filter: added BPF random opcode describes the Linux implementation and includes an interesting discussion of the motivation for the patch.

The following code shows how the open source Host sFlow agent implements random 1-in-256 packet sampling as a BPF program:

ld rand
mod #256
jneq #1, drop
ret #-1
drop: ret #0

A JIT for packet filters discusses the Linux Just In Time (JIT) compiler for BFP programs, delivering native machine code performance for compiled filters.

Minimizing cost of visibility describes why low overhead monitoring is an essential component for increasing efficiency in cloud infrastructure. The combination of BPF packet sampling with standard sFlow export provides a low overhead method of delivering real-time network visibility into large scale cloud infrastructure.

Docker network visibility demonstration

The 2 minute live demonstration shows how the open source Host sFlow agent can be used to efficiently monitor Docker networking in production environments. The demonstration shows real-time tracking of 30Gbit/s traffic flows using less than 1% of a single processor core.

Linux bridge, macvlan, ipvlan, adapters

The open source Host sFlow project added a feature to efficiently monitor traffic on Linux host network interfaces: network adapters, Linux bridge, macvlan, ipvlan, etc. Implementation of high performance sFlow traffic monitoring is made possible by the inclusion of random packet sampling support in the Berkeley Packet Filter (BPF) implementation in recent Linux kernels (3.19 or later).

In addition to the new BPF capability, hsflowd has a couple of other ways to monitor traffic:

• iptables, add a statistic rule to the iptables firewall to add traffic monitoring
• Open vSwitch, has built-in sFlow instrumentation that can be configured by hsflowd.

The BPF sampling mechanism is less complex to configure than iptables and can be used to monitor any Linux network device, including: network adapters (e.g. eth0) and the Linux bridge (e.g. docker0). Monitoring a network adapter also provides visibility into lightweight macvlan and ipvlan network virtualization technologies that are likely to become more prevalent in the Linux container ecosystem, see Using Docker with macvlan Interfaces.

The following commands build and install hsflowd on an Ubuntu 14.03 host:

sudo apt-get update
sudo apt-get install build-essential
sudo apt-get install libpcap-dev
sudo apt-get install git
git clone https://github.com/sflow/host-sflow
cd host-sflow
make
sudo make install

Installing Host sFlow on a Linux server provides basic instructions for configuring the Host sFlow agent (hsflowd). To monitor traffic on the host, edit the /etc/hsflowd.conf file configure the sFlow collector and enable packet sampling on eth0

pcap { dev = eth0 }

Now start the daemon:

sudo hsflowd start

At this point packet traversing eth0 will be sampled and sent out as part of the standard sFlow telemetry stream sent to an sFlow analyzer. For example, using sFlow-RT with the top-flows application as the sFlow analyzer generated the top flows table below.

There a numerous server monitoring agents available in the open source community that will export similar host statistics (CPU, memory, disk) to the Host sFlow agent. Host sFlow differs by also including network traffic visibility using the same packet sampling mechanism supported by most data center switches. Significant advances are extending the visibility into the physical network, for example, Broadcom BroadView Instrumentation tracks buffer utilization and microbursts that effect application performance.

A common standard for monitoring physical and virtual network and server infrastructure reduces operational complexity. Visibility into network activity is critical to understanding the performance of scale out applications that drive large amounts of East-West traffic. Host sFlow, along with support for sFlow in the physical network, delivers scaleable data center wide telemetry to SDN and DevOps tools so that they can better orchestrate the allocation of resources to maximize performance and reduce costs.

System boundary

Credit: Wikimedia

A critical step in developing a control strategy for a system is deciding where to draw the system boundary. For example, in the data center, network management typically draws the boundary to include all the networking devices and their interconnections. The boundary is drawn to include the access switch ports, but exclude the servers and network adapters. System management is the mirror image, drawing the boundary to include servers and network adapters and exclude the network.

Given the system boundary drawn around the network, network planning and design treats the demand on the network as immutable and focusses on creating a network topology that will handle projected demand. Similarly, system capacity management is concerned with providing enough computational resources to handle application workloads while largely ignoring the network - treating it as a uniform "cloud" that will support the whatever demand is placed on it.

The move to scale-out applications, virtualization and convergence aims to improve efficiency by creating a flexible, scalable computing infrastructure that can adapt to changing demand. However, these architectural changes challenge the traditional separation between network and system management. For example, the following diagram shows how moving a virtual machine can dramatically alter network traffic.

Figure 1 Moving a virtual machine can increase network traffic

Changes in network traffic patterns can also have a profound effect on application performance. With convergence, networked storage increases demand for bandwidth and also makes applications sensitive to network congestions.

Redrawing the system boundary to include the network, servers, storage and applications provides the increased span of control needed to deliver efficient cloud services. For example, instead of treating the demand on the network as immutable, it becomes possible to move virtual machines in order to reshape network traffic patterns.

Figure 2 Moving a virtual machine to reduce network traffic

Generally, moving computation closer to data improves application performance by reducing latency and increasing bandwidth as well as reducing the overall load on the network. For example, the Hadoop distributed computing software is rack aware, since "network traffic between different nodes with in the same rack is much more desirable than network traffic across the racks."

The sFlow standard is well suited to automation, providing a unified measurement system that includes network, system and application performance. The Data center convergence, visibility and control presentation describes the critical role that measurement plays in managing costs and optimizing performance.

Hybrid server monitoring

(image from Fiber Channel over Ethernet (FCoE) Primer)

Current trends toward convergence tightly link networking, storage and system performance. In a converged environment, system administrators need to be aware of network traffic linking applications, storage and users in order to avoid performance problems. The dynamic application environment created virtual machine migration, scale out storage and elastic service pools requires that server administrators be aware of network I/O in order to optimize performance and avoid creating problems through poor workload placement choices.

Server operating systems and hardware integrate the instrumentation needed to monitor CPU, memory and disk performance. However, server network adapters typically lack the hardware support needed for traffic monitoring, leaving system administrators with a very limited view of server network I/O. Without hardware support, the network monitoring tools that are available to system administrators are typically used only for troubleshooting since using the tools operationally would adversely impact server performance.

Solving the problem of poor server network visibility requires a broader perspective. Depending on the data center network topology, each server is attached to a blade, top of rack (ToR) or an end of row (EoR) switch. The diagram above illustrates the one-to-one relationship between network adapter and the switch connecting the server to storage and networking resources. Monitoring traffic on a server's switch port provides a complete picture of server network I/O.

Switch vendors recognize the need for network-wide visibility and most have implemented hardware support for the sFlow standard in their data center switches. Combining performance metrics from the server with network visibility from the adjacent switch creates a hybrid monitoring solution that exploits the strengths of existing server and switch instrumentation to provide a complete picture of system performance.

Similar challenges exist in virtual server environments. The integration of sFlow traffic monitoring in the virtual switch (e.g. Xen Cloud Platform) with system performance metrics obtained from virtual machines provides a complete picture of cloud performance. The emerging VEPA standard allows much of the virtual switch functionality to be offloaded from the server software to the adjacent physical switch hardware. VEPA will be a firmware upgrade for most switches so selecting a switch with sFlow support today provides visibility into physical server network I/O and also provides an upgrade path to extend visibility into the virtualization layer as VEPA becomes available.

One challenge remains to widely implementing this hybrid monitoring strategy. Currently, performance monitoring of servers is highly fragmented. Each server hardware, operating system, and system management vendor creates their own agents and software for performance monitoring, none of which interoperate. The emerging sFlow host and power extensions define standard export formats so that performance management tools can easily combine network and server measurements to build a complete picture of data center performance.

Open vSwitch

(diagram from Open vSwitch)

The Open vSwitch provides advanced switching capabilities for virtual servers. Currently the Open vSwitch supports Linux, Xen/XenServer, KVM and Virtual Box. The open source software is designed to be easily portable and is expected to support additional platforms in the future. The Open vSwitch is designed to integrate switching across multiple physical servers, providing an open source alternative to proprietary virtual switches such as VMWare's distributed switch and Cisco's Nexus 1000v.

The recent integration of sFlow traffic monitoring in the Open vSwitch extends the visibility into virtual servers, ensuring data center visibility and control.

Note: The Open vSwitch demonstrates how to integrate the reference sFlow agent code with a virtual switch or network adapter. Integrating sFlow requires minimal support in the "fast path" requiring only packet sampling and packet counters.

The following lines, added to the Open vSwitch configuration file (ovs-vswitchd.conf), configure sampling packets at 1-in-512, polling counters every 20 seconds and sending sFlow to an analyzer (10.0.0.50) over UDP using the default sFlow port (6343):

sflow.<bridgename>.agent    = eth0
sflow.<bridgename>.host     = 10.0.0.50:6343
sflow.<bridgename>.sampling = 512
sflow.<bridgename>.polling  = 20
sflow.<bridgename>.header   = 128

Note: Type "man ovs-vswitchd.conf" for a full list of configuration options. A previous posting discussed the selection of sampling rates.

The following screen capture, from the free sFlowTrend application, demonstrates the visibility provided by sFlow in the Open vSwitch:

All traffic is visible, traffic between virtual machines, and from the virtual machines to the outside world. In addition, sFlow is able to report on all the protocols on the network (note the layer 2, TCP and IPv6 flows in the chart), as well as information on VLANs and layer 2 priorities that is essential for managing switched traffic.

The second screen capture shows a bandwidth trend for a virtual adapter on the vSwitch:

This type of interface trending is a staple of network management, but obtaining the information is challenging in virtual environments. While SNMP is typically used to obtain this information from network equipment, servers are much less likely to be managed using SNMP and so SNMP polling is often not an option. In addition, there may be large numbers of virtual ports associated with each physical switch port. In a virtual environment with 10,000 physical switch ports you might need to monitor as many as 200,000 virtual ports. Even if SNMP agents were installed on all the servers, SNMP polling does not scale well to large numbers of interfaces. The integrated counter polling mechanism built into sFlow provides scalable monitoring of the utilization of every switch port in the network, both physical and virtual, quickly identifying problems wherever they may occur in the network.

Download Open vSwitch and sFlowTrend to evaluate the benefits of visibility in the virtualization layer.

Finally, the Open vSwitch also supports the OpenFlow to allow centralized control of switch forwarding. The combination of sFlow and OpenFlow in the vSwitches delivers visibility and control of the network edge.

Feb. 15, 2011 Update: The configuration steps shown in this article are no longer correct, more recent versions of the Open vSwitch use the ovs-vsctl command instead. The easiest way to manage the sFlow configuration of an Open vSwitch is to install the open source Host sFlow agent which will automatically manage sFlow settings in the Open vSwitch. For recent information on the Open vSwitch, click on the vSwitch label below.

SR-IOV

The diagram (source Intel: Virtual Machine Direct Connect (VMDc)) illustrates the close relationship between the network adapter and the virtual switch in providing networking to virtual machines. Currently most virtual server systems use a software virtual switch (vSwitch) to share access to the network adapter among the virtual machines (VMs).

The Single Root I/O Virtualization (SR-IOV) standard being implemented by 10G network adapter vendors provides hardware support for virtualization, allowing virtual machines to directly share the network adapter without involving the software virtual switch. Hardware sharing improves performance and frees CPU cycles to be used by the virtual machines. Software is still needed to configure and manage the switching function on the network adapter and integrate it with the management of the virtual machines. Virtual switch software is evolving, offloading performance critical functions to the network adapter, while continuing to provide the management interface.

Maintaining visibility and control of the edge is a critical component of an effective data center management strategy. Since virtual switches provide the first layer of switching in a virtualized environment, they comprise the network edge. Integrating the virtual switches into the overall network management system is essential.

Previously, the role of the VEPA protocol in integrating software virtual switches with hardware switches was discussed. VEPA support in the network adapter offers integration between a hardware switch and the network adapter. Most switch vendors support sFlow for traffic monitoring, and the combination of sFlow and VEPA would provide visibility and control of virtual machine traffic.

Ultimately, the evolving functionality of the network adapter/virtual switch is likely to deliver the visibility, performance, security and quality of service capabilities needed from the network edge. This trend is illustrated by the roadmap for the Open vSwitch, which includes support for both sFlow and SR-IOV on its roadmap, along with OpenFlow for control of the edge.