Lasers!

Cool! You mean that I actually have frickin' switches with frickin' laser beams attached to their frickin' ports?

Dr. Evil is right, lasers are cool! The draft sFlow Optical Interface Structures specification exports metrics gathered from instrumentation built into Small Form-factor Pluggable (SFP) and Quad Small Form-factor Pluggable (QSFP) optics modules. This article provides some background on optical modules and discusses the value of including optical metrics in the sFlow telemetry stream exported from switches and hosts.

Pluggable optical modules are intelligent devices that do more than simply convert between optical and electrical signals. The functional diagram below shows the elements within a pluggable optical module.

The transmit and receive functions are shown in the upper half of the diagram. Incoming optical signals are received on a fiber and amplified as they are converted to electrical signals that can be handled by the switch. Transmit data drives the modulation of a laser diode which transmits the optical signal down a fiber.

The bottom half of the diagram shows the management components. Power, voltage and temperature sensors are monitored and the results are written into registers in an EEPROM that are accessible via a management interface.

The proposed sFlow extension standardizes the export of the module sensor data so that they can be exported along with switch port interface counters. A standard structure ensures multi-vendor interoperability and including the optical metrics as part of the sFlow export provides a scaleable method of monitoring all the optical modules in the network.

While the measurements from a single module are useful, the value increases when measurements from all modules can be combined at the sFlow collector. For example, matching pairs of modules allows problems with the transmitter, receiver and the cable connecting them to be isolated.

The sFlow counter push mechanism is an extremely efficient method of monitoring at scale and can collect metrics from all the optical modules in the data center. Combining data from all the modules makes it easier to find outliers.

Finally, incorporating optics monitoring as part of the comprehensive sFlow telemetry stream allows optical metrics to be correlated with switch port, traffic flow and application performance metrics. For example, an increase in application response time can be traced to the paths that the traffic takes across the network, packet discard rates on the ports, and signal strength from the optical sensors, to find a marginal fiber link.

Cisco adds sFlow support to Nexus 9K series

Cisco adds support for the sFlow standard in the Cisco Nexus 9000 Series 7.0(3)I2(1) NX-OS Release. Combined with the Nexus 3000/3100 series, which have included sFlow support since NX-OS 5.0(3)U4(1),  Cisco now offers cost effective, built-in, visibility across the full spectrum of data center switches.

Cisco network engineers might not be familiar with the multi-vendor sFlow technology since it is a relatively new addition to Cisco products. The article, Cisco adds sFlow support, describes some of the key features of sFlow and contrasts them to Cisco NetFlow.

Nexus 9000 switches can be operated in NX-OS mode or ACI mode:

• NX-OS mode includes a number of open features such as sFlow, Python, NX-API, and Bash that integrate with an open ecosystem of orchestration tools such as Puppet, Chef, CFEngine, and Ansible. "By embracing the open culture of development and operations (DevOps) and creating a more Linux-like environment in the Cisco Nexus 9000 Series, Cisco enables IT departments with strong Linux skill sets to meet business needs efficiently," Cisco Nexus 9000 Series Switches: Integrate Programmability into Your Data Center. Open APIs are becoming increasingly popular, preventing vendor lock-in, and allowing organizations to benefit from the rapidly increasing range of open hardware and software solutions to reduce costs and increase agility.
• ACI mode is a closed solution that relies on proprietary hardware and places the switches under the control of Cisco's APIC (Application Policy Infrastructure Controller) - eliminating many of the features, including sFlow, available in NX-OS mode. The ACI solution is more expensive and the closed platform locks customers into Cisco hardware and solutions.

SDN fabric controllers compares tightly coupled (ACI) and loosely federated (NX-OS) approaches to virtualizing data center networking and there are a number of articles on this blog exploring use cases for real-time sFlow analytics in the data center.

Snowflakes, IPFIX, NetFlow and sFlow

Snow flakes by Wilson Bentley

Each snowflake is unique and beautiful. However, while such immense diversity is attractive in nature, variation in data center management standards results in operational complexity, making it difficult to implement the automation and control needed to effectively manage at scale.

The following table examines the approaches taken by the IPFIX and sFlow standards by contrasting how they handle four basic aspects of measurement.

Note: The IPFIX standard is based on Cisco's NetFlow™ version 9 protocol and most of the points of comparison apply equally to NetFlow.

	IPFIX	sFlow
Packet	IPFIX currently defines over 50 fields relating to packet header (see IP Flow Information Export (IPFIX) Entities): protocolIdentifier ipClassOfService tcpControlBits sourceTransportPort sourceIPv4Address destinationTransportPort destinationIPv4Address sourceIPv6Address destinationIPv6Address flowLabelIPv6 icmpTypeCodeIPv4 igmpType sourceMacAddress vlanId ipVersion ipv6ExtensionHeaders destinationMacAddress icmpTypeCodeIPv6 icmpTypeIPv4 icmpCodeIPv4 icmpTypeIPv6 icmpCodeIPv6 udpSourcePort udpDestinationPort tcpSourcePort tcpDestinationPort tcpSequenceNumber tcpAcknowledgementNumber tcpWindowSize tcpUrgentPointer tcpHeaderLength ipHeaderLength totalLengthIPv4 payloadLengthIPv6 ipTTL nextHeaderIPv6 ipDiffServCodePoint ipPrecedence fragmentFlags ipPayloadLength udpMessageLength isMulticast ipv4IHL ipv4Options tcpOptions ipTotalLength ethernetHeaderLength ethernetPayloadLength ethernetTotalLength dot1qVlanId dot1qPriority dot1qCustomerVlanId dot1qCustomerPriority ethernetType The IPFIX standard does not require vendors to support all the fields, each vendor is free to export any combination of fields that they choose, none of the fields are mandatory. The result is that each vendor and each product produces unique and incompatible data.	The sFlow standard specifies a single way to report packet attributes, the packet header, ensuring that every vendor and product produces compatible results. Every sFlow compatible device deployed since the sFlow standard was published in 2001 provides visibility into every protocol that has ever, or will ever, run over Ethernet. The packet header includes all the protocol fields exported by IPFIX as well as fields associated with emerging protocols such as FCoE, AoE, TRILL, NVGRE and VxLAN that have yet to by defined in IPFIX.
Time	IPFIX has over 30 elements that can be used to represent time (see IP Flow Information Export (IPFIX) Entities): flowEndSysUpTime flowStartSysUpTime flowStartSeconds flowEndSeconds flowStartMilliseconds flowEndMilliseconds flowStartMicroseconds flowEndMicroseconds flowStartNanoseconds flowEndNanoseconds flowStartDeltaMicroseconds flowEndDeltaMicroseconds flowDurationMilliseconds flowDurationMicroseconds observationTimeSeconds observationTimeMilliseconds observationTimeMicroseconds observationTimeNanoseconds monitoringIntervalStartMilliSeconds monitoringIntervalEndMilliSeconds collectionTimeMilliseconds maxExportSeconds maxFlowEndSeconds minExportSeconds minFlowStartSeconds maxFlowEndMicroseconds maxFlowEndMilliseconds maxFlowEndNanoseconds minFlowStartMicroseconds minFlowStartMilliseconds minFlowStartNanoseconds The IPFIX standard allows vendors to report time using these elements in any combination, or to omit timestamps altogether. In order to report time consistently, every agent must have a real-time clock and be time synchronized. Finally, it is left up the vendors to decide how often to export data and so an IPFIX collector must understand each vendor's implementation in order to be certain that it has received all the data and detect data loss.	The sFlow standard requires that data be sent immediately. The stateless nature of the protocol means that data can be combined and timestamps added by the central sFlow collector without any need for timestamps or time synchronization among the agents. Note: The sFlow datagrams do contain a time stamp, the agent uptime in milliseconds at the time the datagram was sent.
Sampling	IPFIX currently defines eight different algorithms for packet sampling (see IANA Packet Sampling Parameters): Systematic count-based Sampling Systematic time-based Sampling Random n-out-of-N Sampling Uniform probabilistic Sampling Property match Filtering Hash based Filtering using BOB Hash based Filtering using IPSX Hash based Filtering using CRC Vendors are not required to implement any of these algorithms and are free to invent their own sampling schemes (see NetFlow-lite). In addition, many of the standard algorithms can be shown to be inaccurate.	The sFlow standard mandates a single, statistically valid, sampling algorithm. All sFlow compliant vendors and products, implement the same algorithm and produce accurate, interoperable results.
URL	There is no-standard IPFIX element for exporting a URL. However, IPFIX does allow vendor extensions, resulting in multiple schemes for exporting URL data. Examples include: nProbe URLs are additional fields that can be included as flow keys when configuring the probe. Dell SonicWall URLs are included in an HTTP specific table and link to flow records. Citrix AppFlow URLs are included in an HTTP request table with links to additional HTTP response and ingress/egress connection tables.  In each case, in addition to the URL element itself being vendor specific, the information model associated with the exported URLs is also unique, reflecting the internal architecture of the exporting device.	The sFlow standard mandates a set of HTTP counters and transaction attributes that ensures consistent reporting from HTTP aware entities such as web servers (Apache, Tomcat, NGINX etc.) and load balancers (F5 etc.), irrespective of vendor or internal architecture. Each URL is exported as part of the standard transaction record that includes: client IP, server IP, referrer, authuser, user-agent, mime-type, status, request-bytes, response-bytes, response time. In addition, the sFlow standard defines a unified data model that links measurements from network devices, servers and application instances to provide a comprehensive, data center wide, view of performance.

From the examples in the table, it is apparent that IPFIX and sFlow standards take two very different approaches. The IPFIX standard is descriptive, defining a standard set of attributes that vendors can use to describe the information that they choose to export. The result is that vendors use IPFIX to differentiate each product, reporting a unique and inconsistent set of measurements based on its internal architecture and product features. In contrast, the sFlow standard is prescriptive, defining a set of measurements that every vendor must implement. While IPFIX provides a way to describe each "snowflake", the sFlow standard results from vendors working together to identifying common measurements and implement them in an interoperable way.

Henry Ford transformed the auto industry by moving from hand-made, custom parts to standardized components and processes that allowed for mass production. The data center is undergoing a similar transformation, from small, static, custom environments to large scale, commoditized, flexible, cloud architectures. The sFlow standard delivers the universal performance measurements needed for automation, enjoys broad vendor support, and along with other disruptive technologies like 10G Ethernet, merchant silicon, Software Defined Networking (SDN), OpenFlow, networked storage and virtualization is enabling this transformation.

Vendor support

Cisco's recent support for the sFlow standard should come as no surprise. The graph trends the rapid growth in vendor support for sFlow over the last decade. Today, in addition to Cisco, virtually every other major vendor ships products with sFlow, including: HP, IBM, Dell, Juniper, Brocade, Arista, Huawei, Hitachi, AlaxalA, NEC, Alcatel-Lucent, Fortinet, D-Link, NETGEAR, Extreme Networks, Allied Telesis, ZTE, ZyXEL and LG-ERICCSON.

Growth would have been even faster but industry consolidation has combined a number of sFlow vendors; 3Com and H3C are now combined with ProCurve in Hewlett-Packard, Blade Network Technologies is now part of IBM and Force10 joins PowerConnect as part of Dell. However, this consolidation of US vendors is more than offset by adoption of the sFlow standard among emerging Asian vendors, including: Huawei, ZTE and Edge-Core Networks. Additionally, the graph doesn't count merchant silicon vendors, including Broadcom, Marvell and Intel, that implement sFlow support in the ASICs used by many of the switch vendors.

The rise in vendor support for sFlow was initially driven adoption of 1G Ethernet and more recent growth has been driven by the accelerating deployment of 10G Ethernet. Looking forward, the growth in number of vendors will slow down - there are very few vendors left that do not support sFlow. However, expect vendors to expand the range of products that support sFlow as new 10G, 40G and 100G Ethernet switches are developed to address increasing demand for bandwidth. Also expect to see increased support for sFlow in wireless networks.

Finally, the sFlow standard provides the end-to-end, multi-vendor visibility needed for effective control of resources in the data center and new technologies like OpenFlow and Software Defined Networking (SDN) are unlocking this potential by allowing networks to automatically adapt to the changing real-time traffic patterns reported by sFlow.

Cisco adds sFlow support

Cisco Nexus 3000 series switches

Cisco added support for the sFlow standard in the latest NX-OS 5.0(3)U4(1) release for Nexus 3000 series switches. The Nexus 3000 series are the first Cisco switches based on merchant silicon, which includes hardware support for sFlow, offering scalable, wire-speed, monitoring of all traffic flowing throughout entire networks of Nexus 3000 series switches.

Example: sFlowTrend Top connections chart

The article, 10 Gigabit Ethernet, describes the trend toward 10 Gigabit networking and the critical role that top of rack switches play in next generation data center architectures. Most organisations are predicted to upgrade to 10 Gigabit top of rack switches within the next two years in order to support the demands of virtualization and cloud computing. With the addition of Cisco, all leading switch vendors now have 10 Gigabit top of rack switches that support the sFlow standard, making sFlow the obvious choice when selecting a vendor neutral performance monitoring solution for large scale cloud environments.

Since the Nexus 3000 series switches are the first Cisco products with sFlow, the rest of this article is addressed to Cisco network administrators who are likely to be unfamiliar with sFlow technology. As a Cisco network administrator, you are likely to have experience with using Cisco's Switched Port Analyzer (SPAN) technology to selectively monitor traffic in Cisco edge switches and with Cisco's Netflow technology for monitoring TCP/IP traffic in Cisco routers.

By adding sFlow support to the Nexus 3000 series, Cisco eliminates the need for probes, providing wire-speed 10 Gigabit monitoring of all switch ports - the functional equivalent of forty-eight 10 Gigabit probes and four 40 Gigabit probes in a Nexus 3064 - embedded in the switch hardware at no extra cost. If you are familiar with RMON probes, sFlow is functionally equivalent to deploying an RMON probe for each switch port.

Based on the name, you might think that sFlow is just another version of Cisco Netflow. However, this is not the case - sFlow differs significantly from NetFlow and understanding these differences is important if you want to get the most out of sFlow:

1. sFlow exports interface counters, eliminating the need for SNMP polling - extremely useful when you have tens of thousands of edge switch ports to monitor.
2. sFlow exports packet headers not flow records. By exporting packet headers, sFlow is able to provide full layer 2 - 7 visibility into all types of traffic flowing at the network edge, including: MAC addresses, VLANs, TRILL, tunnels (GRE, VXLAN etc.), Ethernet SAN traffic (FCoE and AoE), IPv6 in addition to the TCP/IP information typically reported by NetFlow. You can even use sFlow with Wireshark for remote packet capture.
3. sFlow is highly scalable. Unlike NetFlow, which is typically enabled on selected links at the core, sFlow is enabled on every port, on every switch, for full end-to-end network visibility. The sFlow measurements are implemented in silicon and won't impact switch CPU. The scalability of sFlow allows tens of thousands of 10G switch ports in the top of rack switches, as well as their 40 Gigabit uplink ports, to be centrally monitored. In addition, sFlow is available in 100 Gigabit switches, ensuring visibility as higher speed interconnects are deployed to support the growing 10 Gigabit edge.
4. sFlow is easy to configure and manage. Eliminating complexity is essential for large scale web 2.0, big data, virtualization and cloud deployments.
5. sFlow is a multi-vendor standard supported by almost every network equipment vendor. You can mix and match Cisco Nexus 3000 series switches with best in class solutions from other vendors and still maintain comprehensive, interoperable, data center wide visibility.
6. sFlow is not just for switches. The sFlow standard also provides visibility into server, storage, virtual machine and application performance, helping to break down management silos by providing a consistent view of performance to operations and development teams (see DevOps).
7. sFlow functionality is determined by the choice of sFlow analyzer. With Flexible NetFlow, much of the analysis is performed on the network device, limiting the functionality of NetFlow collectors to simply recording the data and generating reports. As a result, NetFlow collectors end up being fairly generic in functionality. In contrast, sFlow shifts analysis from the switches to a central sFlow analyzer which determines how to process the data and present the results, see Choosing an sFlow analyzer. The result is a greater diversity of solutions and there is likely to be an sFlow analyzer that is particularly well adapted to your requirements. While many NetFlow collectors claim sFlow support, their support tends to be limited, ignoring sFlow specific features and treating sFlow as if it were basic NetFlow version 5.

Trying out sFlow is easy, just upgrade to the latest NX-OS release, configure sFlow export, and install the free sFlowTrend analyzer to gain real-time visibility - providing immediate answers to the Who, What, Where, When, Why and How questions that are the key to effective management.

Merchant silicon

The following chart, from Commoditization of Ethernet Switches: How Value is Flowing into Silicon, shows the rapidly increasing market share of network switches based on Broadcom, Marvell and Intel (Fulcrum)  chipsets (often referred to as "merchant silicon") as switch vendors move from proprietary ASICs to off-the-shelf designs.

Off-the-shelf vs. Internal Silicon Design

As an example, many vendors now base their 10 Gigabit top of rack switches on Broadcom chipsets. Often vendors don't disclose when they are using merchant silicon, however, based on news reports, similarities in specifications and rumors, the following switches appear to use similar Broadcom chipsets: IBM BNT RackSwitch G8264, Juniper QFX3500, Cisco Nexus 3064, Arista 7050S-64, HP 5900-AF, Alcatel-Lucent Omniswitch 6900 and Dell Force10 S4810.

In addition to reducing costs, the move to merchant silicon helps increase multi-vendor interoperability and support for standards. For example, the sFlow standard is widely implemented in merchant chipsets and the adoption of merchant silicon for 10 Gigabit top of rack switches has greatly increased the presence of sFlow in data centers. The Network World article, OpenFlow, Merchant Silicon, and the Future of Networking, suggests that the rising popularity of merchant silicon is also helping to drive adoption of the OpenFlow standard.

Together, the sFlow and OpenFlow standards transform data center networking by providing the integrated visibility and control needed to adapt to changing workloads in converged, virtualized and cloud environments.

Standard metrics

This presentation describes the role that standard server performance metrics play in increasing the scalability and reducing the operational complexity of performance monitoring in large data centers.

The presentation uses popular performance monitoring tools: Nagios, Ganglia, Collectd, Cacti and Munin to demonstrate the complexity of managing each application's agents on multiple platforms and servers. The tools are then used demonstrate that a core set of metrics is widely recognized and broadly supported. The presentation goes on to show how an agent exporting these standard metrics allows performance monitoring tools to share data, eliminating the need for wasteful duplication. Finally, by including server performance metrics in the sFlow standard, server performance monitoring becomes part of an integrated solution that includes networking, servers and applications.

sFlow Host Structures

The completed sFlow Host Structures specification has been published by sFlow.org, extending the sFlow standard to include physical and virtual server performance metrics. The specification describes a coherent framework that builds on the sFlow metrics exported by most switch vendors, linking network, server and application performance monitoring to provide an integrated picture of performance.

The diagram above shows how the packet header information exported by network devices is used to link network performance with performance metrics collected from servers and applications. The packet header contains MAC addresses corresponding to physical and virtual server network adapter cards as well as TCP/UDP socket information identifying individual application instances. Collecting sFlow data from the network devices provides an sFlow analyzer with a real-time map of the physical and logical relationships between entities on the network (see Packet paths and Application mapping).

A server exporting sFlow performance metrics includes an additional structure containing the MAC addresses associated with each of its network adapters. The inclusion of the MAC addresses provides a common key linking server performance metrics (CPU, Memory, I/O etc.) to network performance measurements (network flows, link utilizations, etc.), providing a complete picture of the server's performance (see Hybrid server monitoring and UUID)

The sFlow Host Structures specification builds on the scalable "counter push" mechanism that is used by network devices to export standard interface counters (see Link utilization). Most operating systems already maintain performance counter to track CPU, memory and I/O performance. The sFlow Host Structures specification leverages work done by the Ganglia project to define a common set of metrics across different operating systems, including: Windows, Linux (Fedora/RedHat/CentOS, Debian, Gentoo, SuSE/OpenSuSE), Solaris, FreeBSD, NetBSD, OpenBSD, DragonflyBSD and AIX. The extension of sFlow to include server performance metrics integrates network and system monitoring to deliver a data center wide view of performance (see Top servers and Cluster performance).

For virtual machine performance metrics, the sFlow Host Structures specification draws on definitions from the libvirt project which has defined a standard set of metrics that can be collected from a wide variety of virtualization platforms, including: Xen, QEMU, KVM, LXC, OpenVZ, User Mode Linux, VirtualBox, VMWare ESX and GSX. Again, the MAC addresses associated with each virtual machine are exported along with its performance metrics so that the virtual machine's performance can be linked to its network activity.

The sFlow Host Structures document also describes the extension of sFlow's sampling mechanism to include application transaction sampling. Examples of application level transactions include: HTTP requests to a web server, NFS/CIFS requests to a file server, memcached requests and operations performed by a Hadoop cluster. An application sFlow agent samples completed transactions, capturing information about each completed request, including: size, duration, type, URL, file name etc. Each application transaction sample is linked to the network through the inclusion of TCP/UDP socket information which can be matched to packet header information from network devices.

What clearly distinguishes sFlow from other monitoring technologies is the integrated, end-to-end, view of performance that it offers. Integration exponentially increases the value of information by making it actionable. For example, identifying that an application is running slowly isn't enough to solve the performance problem. However, if you also know that the server hosting the application is seeing poor disk performance, can link the disk performance to a slow NFS server, can identify the other clients of the NFS server and finally determine that all the request are competing for access to a single file, then you are in a position to take action. It is this ability to link data together, combined with the scalability to monitor every resource in the data center that makes sFlow revolutionary.

Power

Electricity Counter

The paper, No "Power" Struggles: Coordinated Multi-level Power Management for the Data Center, examines different power control strategies and suggests that power savings of 20% - 60% are possible using feedback control to optimize and consolidate virtual machine locations and workloads. The paper concludes, "Our results indicate that effective coordination is likely to be more important in future environments with richer diversity in workloads and increased emphasis on power reduction."

The sFlow energy metering extension, currently being developed as part of the sFlow standard, builds on the proven scalability of sFlow's counter polling mechanism to provide real-time metering of all switches, PoE ports, servers, blades and processors in the data center.

Incorporating power metering in sFlow integrates power and temperature information with workload and performance statistics from network devices and servers to deliver the real-time measurements needed for coordinated control and optimization of power usage throughout the data center.

Convergence

The diagram shows technologies that are part of the drive toward converge of storage, server and network technologies in the data center.

Virtualization and the need to support virtual machine mobility (e.g. vMotion/XenMotion/Xen Live Migration) is driving the adoption of large, flat, high-speed, layer-2, switched Ethernet fabrics in the data center. A layer-2 fabric allows a virtual machine to keeps its IP address and maintain network connections when it moves (performing a "live" migration).

Networked storage (iSCSI, NFS, FCoE) is also needed to support virtual machine mobility. In addition, moving storage from dedicated SANs to a converged Ethernet fabric reduced cabling and networking costs and improves flexibility. However, migration of storage traffic onto a converged Ethernet fabric dramatically increases bandwidth demands and is one of the factors accelerating the adoption of 10/40/100G Ethernet.

Ethernet standards are evolving to address the needs of convergence. As well as higher speeds, IEEE data center bridging standards add support for lossless Ethernet to improve storage performance and support Fiber Channel over Ethernet (FCoE). The performance and stability of large layer 2 Ethernets is being addressed through new protocols such as shortest path bridging.

Changes are not limited to storage and data networking. Server architectures are changing as convergence blurs the line between the server and the network, extending the Ethernet fabric into servers through blade switches, network adapters and virtual switches.

The current siloed approach to managing storage, servers and networks no longer works in a converged environment where each of these areas is so closely inter-dependent. Fortunately, convergence to an Ethernet fabric brings with it the data center visibility needed to manage the converged data center.

The sFlow standard, implemented by most vendor's Ethernet switches, simplifies management by providing the unified visibility and control needed to fully realize the benefits of virtualization and convergence.

Virtual routing

The diagram shows networking elements within a virtual server. The server's physical network adapters connect to LAN switches that provide a high speed, flat, layer 2, fabric connecting servers and storage in the data center. Virtual switches provide shared access to the physical adapters, connecting the virtual network adapters in the virtual machines to the physical network.

Current hierarchical network designs confine routing to specialized hardware at the core of the network.  It is worth re-examining the place of routing given the changes in data center architecture brought about by convergence and virtualization. What if routing could be virtualized?

The performance of software routers running on commodity x86 hardware is improving,  Vyatta recently announced 20 Gbps routing performance from their software routers. With network adapter support for virtualization (e.g. SR-IOV), it is now feasible to implement high-performance routing and firewall functionality in virtual machines.

Virtualization of routing offers a number of advantages:

1. Virtualization allows services to be replicated and deployed where they are needed in the virtual infrastructure. A virtual router can easily be replicated to provide redundancy or add capacity.
2. Virtual routing can provide better reliability and lower costs by making use of the general purpose virtual server infrastructure, eliminating the need for expensive, specialized router hardware.
3. Distributing routing to the edge of the network reduces pressure on the core and improves scalability. 

There benefits don't just apply to routing, many other specialized devices can also be virtualized, including firewalls, load balancers, proxies etc. Virtualization of layer 3-7 network devices on a high performance converged Ethernet fabric offers a flexible and dynamic infrastructure that can easily be reconfigured to meet changing demands.

To illustrate the potential of virtualized networking, consider the example of a hosted data center. In a typical hosted data center, customers have racks or partial racks of equipment installed in the data center. A typical customer will have their own router, firewall, load balancer and servers installed in the rack. A virtual rack can be constructed by deploying routing and firewall virtual machines along with general purpose virtual machines that the customer can use to deploy their applications. A virtual rack can be provisioned and maintained automatically, providing customers with much more responsive service while reducing operating costs. In addition, virtualization allows higher customer densities per physical rack, increasing the revenue that can be generated per rack.

The benefits aren't restricted to service provider networks. In enterprise data centers, the flexibility of virtualized networking allows for more efficient management and utilization of resources. However, a barrier to realizing these benefits is the current siloed approach to data center management. Close coordination is needed between network and system management teams. For example, who would be responsible for provisioning and configuring a virtual router? This type of cross functional task is a challenge for most organizations.

Integrated traffic monitoring provides the visibility needed for effective management of virtualized networks. The diagram shows some of the data paths that are possible in a virtual stack: the red line shows traffic between two physical VLANs connected by a virtual router and the gold line shows traffic routed between two virtual machines hosted on the same server. In order to provide network visibility, every networking device, physical or virtual needs to include integrated traffic monitoring so that all traffic paths can be observed. Shared visibility into all resources in the data center ensures that each group (network, systems and storage) is aware of its impact on shared resources, eliminates finger pointing, improves coordination and lays the foundation for automating control.

There are many proprietary and standard technologies for embedded traffic monitoring. Broadly speaking these fall into two classes, TCP/IP flow monitoring built into many routers (e.g. Cisco NetFlow) and multi-protocol packet-based monitoring built into most switches. Convergence in both the LAN (data center bridging) and the WAN (Metro Ethernet and Carrier Ethernet) is taking place using Ethernet technologies, making the sFlow standard the logical choice for visibility since it enjoys broad, multi-vendor support and is already built into most vendor's Ethernet products. Just as convergence to Ethernet simplifies connectivity, convergence to sFlow standard monitoring built into Ethernet devices simplifies management of the converged network.

Products incorporating the sFlow standard provide visibility throughout the physical switch, virtual switch, virtual router and cloud layers, delivering the end to end visibility needed to realize the full benefits of virtualization and convergence.

Standards

Data center convergence and virtualization offer the promise of improved efficiency and flexibility. Developing a network visibility strategy when planning the network upgrades needed to support convergence is essential since only network equipment with embedded traffic monitoring will provide the data center wide visibility essential for effective control.

This article examines the proprietary and standards-based protocols for embedded traffic monitoring as well as their current status and level of support.

Before looking at the standards, it's important to understand how current traffic monitoring protocols fit in the protocol stack. The article, sFlow and NetFlow, describes how sFlow operates at layer 2 (switches) and Cisco NetFlow (and variants from other vendors such as j-Flow, NetStream, LFAP etc.) operates at layer 3 (routers).

There are two organizations responsible for most networking standards:

1. IETF (Internet Engineering Task Force) is responsible for layer 3-7 protocols (IP, routing, DNS, telnet, smtp etc.)
2. IEEE (Institute of Electrical and Electronics Engineers) is responsible for layer 2 protocols (Ethernet, switching, 802.11 etc.)

The IETF has recently developed a standard alternative to proprietary IP flow monitoring protocols (Cisco NetFlow, j-Flow, LFAP, NetStream etc.). The IPFIX standard was created in order "to transfer IP flow data from IPFIX exporters to collectors." This focus on IP flow export is consistent with the IETF's responsibility for the TCP/IP suite of protocols and explicitly avoids addressing layer 2 monitoring (switches). The IPFIX standard was published in 2008. Unfortunately, IPFIX has found little support among router vendors, who continue to implement proprietary solutions.

The IEEE publishes the standards for Ethernet and bridging/switching, and is currently developing the set of standards for Data Center Bridging (DCB) that are driving data center convergence. The IEEE would seem to be the natural place to standardize a protocol for monitoring switched Ethernet traffic, however, the IEEE's focus is on the mechanics of layer 2 connectivity and network management protocols have not been a priority.

The sFlow.org industry consortium was formed in order to develop a multi-vendor standard to address the need for network visibility in layer 2 devices. Many of the members of sFlow.org actively participate in the IEEE standards process, ensuring that sFlow is well matched to the challenge of monitoring current and emerging IEEE standard networks (Ethernet, 40/100G, DCB etc.). The sFlow standard was published in 2001 and it is now implemented by most switch vendors (see sFlow.org).

In selecting a standard for data center visibility, it is important to understand how data center networks are changing. The traditional three layer architecture, in which traffic is moved up to the core and back, does not scale well. Instead, the trend is to integrate access and aggregation layer switches into a flat layer 2 network using shortest-path bridging (IEEE 802.1aq) so that traffic bypasses the core to deliver the increased scalability, bandwidth and reduced latency needed to support converged data center workloads.

The shift in data center network architecture requires a corresponding shift in network management. Instead of monitoring and controlling at layer 3 in the core routers, visibility and control functions move to layer 2 switches and the network edge.

sFlow is the only standard specifically designed for embedded monitoring of layer 2 devices. Selecting switches with embedded sFlow provides a low cost, scalable means of obtaining layer 2-7 visibility into all traffic flowing over the switched network (including storage traffic). Building a network visibility strategy around the sFlow standard maximizes the choice of vendors and ensures interoperable monitoring in mixed vendor environments, eliminating vendor lock-in and facilitating "best in class" product selection.

VEPA

A virtual switch is a software component of a virtual server, providing network connectivity to virtual machines (VMs). The challenge with virtual switches is integrating them into the rest of the network in order to maintain visibility and control.

The diagram shows how the emerging VEPA (Virtual Ethernet Port Aggregator) standard addresses this challenge by ensuring that packets from the virtual machines connected to the virtual switch (shown in green) also pass through an adjacent hardware switch (Bridge). In a blade server, the adjacent hardware switch would be the blade switch. If stand-alone servers are used, then the adjacent hardware switch would be the top of rack switch.

Passing traffic through the hardware switch offloads tasks such as rate limiting and access control lists (ACLs), simplifying the virtual switch and freeing CPU cycles that can be used by the virtual machines.

The sFlow standard is widely supported by switch vendors. Selecting blade switches and top of rack switches with sFlow and VEPA support will offer visibility and control of the network edge.

Multi-vendor support

Multi-vendor support of the sFlow standard has been increasing rapidly over the last 5 years. Initially published by InMon Corp. as RFC 3176, sFlow has grown into the leading, multi-vendor, standard for monitoring high-speed switched networks. The growth in vendor support of sFlow has been driven by the move to 1G and more recently 10G Ethernet switches. The sFlow.org industry consortium, responsible for developing and promoting the sFlow standard, lists the large number of switches that implement sFlow. The switch vendors supporting sFlow now include: HP, IBM, Dell, Brocade, Juniper, BLADE, 3Com, H3C, Force10, Hitachi, AlaxalA, NEC, Alcatel-Lucent, D-Link, Extreme Networks, Allied Telesis and Comtec.

Broad vendor support delivers the network-wide visibility that is essential for managing the convergence of voice, data and storage on the campus and in the data center. It is likely that you have products from one or more of these vendors - if you would like more information on options to extend visibility in your network, ask them about sFlow.

September 11, 2012 Update: An updated version of this article describes the continued growth (nearly doubling) of sFlow support among vendors over the last couple of years, including Cisco's recent support for sFlow.

LAN and WAN

There is widespread confusion about the differences between sFlow and NetFlow and they are often simply referred to collectively as xFlow, implying that the two technologies are interchangeable. The sFlow and NetFlow posting described some of the technical differences between the two technologies, in particular, describing how sFlow operates at the Switch/Ethernet/LAN level and NetFlow operates at the Router/IP/WAN level. This division helps understand where the two technologies fit in the market.

The chart breaks down networking into switching and routing on the x-axis and network speed on the y-axis and then plots the application areas for sFlow and NetFlow. Since sFlow is built into switch ASICs, it offers monitoring solutions that span the full range of layer 2-3 switching products, from inexpensive switches aimed at office and small business environments, to the most demanding applications in supercomputer data centers, Internet exchange points and digital effects render farms. NetFlow is typically found in enterprise class routers. Since performance critical components of NetFlow are often implemented in software, NetFlow isn't widely used for monitoring at the high end of the router market (tier-1 ISPs). The cost of NetFlow enabled equipment limits its use at the low end of the router market.

Dividing the market into routing and switching products and the related applications helps explain why some vendors support sFlow while others support NetFlow. The chart also explains why a vendor might offer sFlow on their switch products and NetFlow on their router products. In practice, most networks blend switching and routing in order to meet the varied requirements of the different services running on the network. In many cases, a network monitoring strategy that embraces both sFlow and NetFlow delivers the most complete visibility into network activity.

Why an sFlow blog?

sFlow has been quietly emerging as the leading multi-vendor standard for monitoring traffic in switched networks. The first sFlow capable switches were shipped in 2001. Since then vendor support has been increasing each year and now switches from Brocade, Hewlett-Packard, Juniper Networks, Extreme Networks, 3Com, D-Link, Alcatel-Lucent, H3C, Hitachi, NEC, AlaxalA, Allied Telesis and Comtec have embedded sFlow. Chances are you already have switches in your network that support sFlow (for a complete list of switches that support sFlow, see Network Equipment).

sFlow monitoring solutions provide traffic visibility for the full range of switched networks, from an organization running a large, busy 10G network (see Amsterdam Internet Exchange) to a small office trying to see who is hogging bandwidth on their T1 link (see sFlowTrend).

Through the postings in this blog, we hope to increase awareness of sFlow and the way it can be used to solve problems that challenge network administrators every day.