Cisco adds sFlow support to Nexus 9K series

Cisco adds support for the sFlow standard in the Cisco Nexus 9000 Series 7.0(3)I2(1) NX-OS Release. Combined with the Nexus 3000/3100 series, which have included sFlow support since NX-OS 5.0(3)U4(1),  Cisco now offers cost effective, built-in, visibility across the full spectrum of data center switches.

Cisco network engineers might not be familiar with the multi-vendor sFlow technology since it is a relatively new addition to Cisco products. The article, Cisco adds sFlow support, describes some of the key features of sFlow and contrasts them to Cisco NetFlow.

Nexus 9000 switches can be operated in NX-OS mode or ACI mode:

• NX-OS mode includes a number of open features such as sFlow, Python, NX-API, and Bash that integrate with an open ecosystem of orchestration tools such as Puppet, Chef, CFEngine, and Ansible. "By embracing the open culture of development and operations (DevOps) and creating a more Linux-like environment in the Cisco Nexus 9000 Series, Cisco enables IT departments with strong Linux skill sets to meet business needs efficiently," Cisco Nexus 9000 Series Switches: Integrate Programmability into Your Data Center. Open APIs are becoming increasingly popular, preventing vendor lock-in, and allowing organizations to benefit from the rapidly increasing range of open hardware and software solutions to reduce costs and increase agility.
• ACI mode is a closed solution that relies on proprietary hardware and places the switches under the control of Cisco's APIC (Application Policy Infrastructure Controller) - eliminating many of the features, including sFlow, available in NX-OS mode. The ACI solution is more expensive and the closed platform locks customers into Cisco hardware and solutions.

SDN fabric controllers compares tightly coupled (ACI) and loosely federated (NX-OS) approaches to virtualizing data center networking and there are a number of articles on this blog exploring use cases for real-time sFlow analytics in the data center.

Drivers for growth

This article examines the factors that are continuing to accelerate adoption of the sFlow measurement standard as the universal source of analytics in the data center, including: rising popularity of merchant silicon based switches, open switch operating systems and platforms, virtual switching, network virtualization, and integration of real-time sFlow analytics in orchestration stacks to create automated self-optimizing data centers.

Two years ago the article Merchant silicon described the broad adoption of the Broadcom Trident ASIC by switch vendors. This trend is picking up pace with the rapid adoption of the new Trident II ASIC (announced last year, but only available in volume this Fall). Vendors don't typically disclose when they use merchant silicon, however, based on news reports, similarities in specifications and rumors, the following switches appear to use Broadcom Trident II chipsets: Extreme Summit X770, HP 5930, Dell S6000, Cumulus HCL partners (Agema, Edge-Core, Penguin Computing and Quanta), Arista 7250X and 7500E series, Cisco Nexus 3100 and 9000 series, Juniper QFX 3500 series and Nuage 7850 VSG.

Note: While most of the Broadcom based switches listed already support sFlow, a few vendors have yet to enable the feature in their firmware. If you have, or are considering, Broadcom based switches in your data center, ask your vendor when they plan to enable sFlow. A list of switches with sFlow support is maintained on sFlow.org.

Merchant silicon lowers the barriers to entering the networking market in much the same way as standardizing on x86 compute platforms commoditized hardware and made it possible for a large number of PC manufacturers to emerge. The second component driving this trend is the availability of switch operating systems (Broadcom FASTPATH, Cumulus Linux, Big Switch's Switch Light Linux, Pluribus OpenNetvisor, Pica8 PicOS, etc.) that further reduce the barrier to entry. Another project to watch is the Open Compute Project's efforts to define an open switch hardware platform - if successful, it will create high volume standard hardware platform and competition between hardware vendors that will drive down hardware costs and increase the market for switch operating systems and the ecosystem of software running on those platforms - analogous to Windows and Linux running on x86 and their respective application ecosystems.

The slide from Bruce Davie's keynote address at Open Server Summit 2013, Network Virtualization: What it is, Why it Matters, shows the rapid transition from a physical edge in which physical servers are attached to physical switch ports, to a virtual edge in which virtual machines are attached to virtual switches. Second generation virtual switches are starting to enter the market, delivering increased performance and integrating support for overlays and network virtualization. In SDN market predictions for New Year: NFV, OpenFlow, Open vSwitch boom, Eric Hanselman, chief analyst at 451 Research states, "The improved scalability of Open vSwitch 2.0 will affect the numerous SDN vendors who use it as an OpenFlow agent on switches or as an endpoint in overlay technologies. These vendors include high-profile players such as VMware Inc. and startups such as Midokura and Pica8."

Accelerating adoption of virtual switching is helping to drive sFlow growth since support for the standard is integrated in virtual switches:

• Open vSwitch, integrated in the mainstream Linux kernel and an integral part of many commercial and open source virtualization platforms, including: VMware/Nicira NSX, OpenStack, Xen Cloud Platform, XenServer, and KVM.
• Hyper-V Virtual Switch, part of Window Server 2012
• IBM Distributed Virtual Switch 5000V
• HP FlexFabric Virtual Switch 5900v

The article Visibility and the software defined data center describes how the sFlow standard has been extended to include not just the network, but server and application resources as well. For example, growing support for sFlow in web servers (Apache, NGINX, Tomcat) and load balancers (F5 BIG IP, HAproxy) extends visibility to include application response time, URLs, response codes etc. Best of Velocity 2012: The sFlow Standard describes how sFlow analytics integrate into the DevOps tool stack to provide scaleable, real-time monitoring of application resources.

So far this article has described the widespread support for the sFlow measurement standard within the data center infrastructure. The remainder of the article explores the rise in automation and the vital role that real-time analytics is poised to play in orchestration stacks.

As SDN solutions move from pilot to large scale deployments, attention is shifting from using SDN merely to configure networking, to optimizing performance and increasing efficiency. There is also a clear move to an integrated view of orchestration that includes networking, servers, storage and applications, going beyond SDN to create what VMware calls the Software Defined Data Center (SDDC), Cisco terms the Application Centric Infrastructure (ACI), and Microsoft refers to as the Cloud OS.

The following articles demonstrate the growing awareness among industry leaders about the importance of analytics as they develop their cloud orchestration controllers:

1. Of Mice and Elephants by Martin Casado and Justin Pettit with input from Bruce Davie, Teemu Koponen, Brad Hedlund, Scott Lowe, and T. Sridhar - VMware
2. Software Defined Networking on VMWare with Scott Lowe on RunAs Radio - VMware
3. How Software-defined Networking is rewriting the rules of application delivery by Senior Vice President and General Manager, HP Networking - Hewlett-Packard
4. Networking Without Limits: SDN by Brad Anderson, Corporate Vice President, Windows Server & System Center - Microsoft
5. Where the puck is going: analytics by Mike Bushong - Plexxi
6. Wandl and Cariden: Is There a Real Value? by Tom Nolle - CIMI Corp.

The article Workload placement describes this author's take on the strategic value of analytics and orchestration as a way to transform the economics of cloud computing by more densely packing workloads in the data center.

Recent breakthroughs in real-time sFlow analysis incorporated in the sFlow-RT analytics engine delivers timely, comprehensive, and actionable metrics through a programmatic interface. Expect to see this technology incorporated in next generation self optimizing orchestration solutions in 2014.

Performance Aware SDN describes the theory behind analytics driven orchestration. The talk describes how fast controller response, programmatic configuration interfaces such as OpenFlow, and consistent instrumentation of all the elements being orchestrated are pre-requisites for feedback control.

The requirement for complete measurement coverage by next generation orchestration systems will create a strong demand for sFlow instrumented infrastructure since sFlow is the only widely supported multi-vendor standard that spans network, server and application resources and delivers the low latency and scaleability required for adaptive control.

Vendor support

Cisco's recent support for the sFlow standard should come as no surprise. The graph trends the rapid growth in vendor support for sFlow over the last decade. Today, in addition to Cisco, virtually every other major vendor ships products with sFlow, including: HP, IBM, Dell, Juniper, Brocade, Arista, Huawei, Hitachi, AlaxalA, NEC, Alcatel-Lucent, Fortinet, D-Link, NETGEAR, Extreme Networks, Allied Telesis, ZTE, ZyXEL and LG-ERICCSON.

Growth would have been even faster but industry consolidation has combined a number of sFlow vendors; 3Com and H3C are now combined with ProCurve in Hewlett-Packard, Blade Network Technologies is now part of IBM and Force10 joins PowerConnect as part of Dell. However, this consolidation of US vendors is more than offset by adoption of the sFlow standard among emerging Asian vendors, including: Huawei, ZTE and Edge-Core Networks. Additionally, the graph doesn't count merchant silicon vendors, including Broadcom, Marvell and Intel, that implement sFlow support in the ASICs used by many of the switch vendors.

The rise in vendor support for sFlow was initially driven adoption of 1G Ethernet and more recent growth has been driven by the accelerating deployment of 10G Ethernet. Looking forward, the growth in number of vendors will slow down - there are very few vendors left that do not support sFlow. However, expect vendors to expand the range of products that support sFlow as new 10G, 40G and 100G Ethernet switches are developed to address increasing demand for bandwidth. Also expect to see increased support for sFlow in wireless networks.

Finally, the sFlow standard provides the end-to-end, multi-vendor visibility needed for effective control of resources in the data center and new technologies like OpenFlow and Software Defined Networking (SDN) are unlocking this potential by allowing networks to automatically adapt to the changing real-time traffic patterns reported by sFlow.

Cisco adds sFlow support

Cisco Nexus 3000 series switches

Cisco added support for the sFlow standard in the latest NX-OS 5.0(3)U4(1) release for Nexus 3000 series switches. The Nexus 3000 series are the first Cisco switches based on merchant silicon, which includes hardware support for sFlow, offering scalable, wire-speed, monitoring of all traffic flowing throughout entire networks of Nexus 3000 series switches.

Example: sFlowTrend Top connections chart

The article, 10 Gigabit Ethernet, describes the trend toward 10 Gigabit networking and the critical role that top of rack switches play in next generation data center architectures. Most organisations are predicted to upgrade to 10 Gigabit top of rack switches within the next two years in order to support the demands of virtualization and cloud computing. With the addition of Cisco, all leading switch vendors now have 10 Gigabit top of rack switches that support the sFlow standard, making sFlow the obvious choice when selecting a vendor neutral performance monitoring solution for large scale cloud environments.

Since the Nexus 3000 series switches are the first Cisco products with sFlow, the rest of this article is addressed to Cisco network administrators who are likely to be unfamiliar with sFlow technology. As a Cisco network administrator, you are likely to have experience with using Cisco's Switched Port Analyzer (SPAN) technology to selectively monitor traffic in Cisco edge switches and with Cisco's Netflow technology for monitoring TCP/IP traffic in Cisco routers.

By adding sFlow support to the Nexus 3000 series, Cisco eliminates the need for probes, providing wire-speed 10 Gigabit monitoring of all switch ports - the functional equivalent of forty-eight 10 Gigabit probes and four 40 Gigabit probes in a Nexus 3064 - embedded in the switch hardware at no extra cost. If you are familiar with RMON probes, sFlow is functionally equivalent to deploying an RMON probe for each switch port.

Based on the name, you might think that sFlow is just another version of Cisco Netflow. However, this is not the case - sFlow differs significantly from NetFlow and understanding these differences is important if you want to get the most out of sFlow:

1. sFlow exports interface counters, eliminating the need for SNMP polling - extremely useful when you have tens of thousands of edge switch ports to monitor.
2. sFlow exports packet headers not flow records. By exporting packet headers, sFlow is able to provide full layer 2 - 7 visibility into all types of traffic flowing at the network edge, including: MAC addresses, VLANs, TRILL, tunnels (GRE, VXLAN etc.), Ethernet SAN traffic (FCoE and AoE), IPv6 in addition to the TCP/IP information typically reported by NetFlow. You can even use sFlow with Wireshark for remote packet capture.
3. sFlow is highly scalable. Unlike NetFlow, which is typically enabled on selected links at the core, sFlow is enabled on every port, on every switch, for full end-to-end network visibility. The sFlow measurements are implemented in silicon and won't impact switch CPU. The scalability of sFlow allows tens of thousands of 10G switch ports in the top of rack switches, as well as their 40 Gigabit uplink ports, to be centrally monitored. In addition, sFlow is available in 100 Gigabit switches, ensuring visibility as higher speed interconnects are deployed to support the growing 10 Gigabit edge.
4. sFlow is easy to configure and manage. Eliminating complexity is essential for large scale web 2.0, big data, virtualization and cloud deployments.
5. sFlow is a multi-vendor standard supported by almost every network equipment vendor. You can mix and match Cisco Nexus 3000 series switches with best in class solutions from other vendors and still maintain comprehensive, interoperable, data center wide visibility.
6. sFlow is not just for switches. The sFlow standard also provides visibility into server, storage, virtual machine and application performance, helping to break down management silos by providing a consistent view of performance to operations and development teams (see DevOps).
7. sFlow functionality is determined by the choice of sFlow analyzer. With Flexible NetFlow, much of the analysis is performed on the network device, limiting the functionality of NetFlow collectors to simply recording the data and generating reports. As a result, NetFlow collectors end up being fairly generic in functionality. In contrast, sFlow shifts analysis from the switches to a central sFlow analyzer which determines how to process the data and present the results, see Choosing an sFlow analyzer. The result is a greater diversity of solutions and there is likely to be an sFlow analyzer that is particularly well adapted to your requirements. While many NetFlow collectors claim sFlow support, their support tends to be limited, ignoring sFlow specific features and treating sFlow as if it were basic NetFlow version 5.

Trying out sFlow is easy, just upgrade to the latest NX-OS release, configure sFlow export, and install the free sFlowTrend analyzer to gain real-time visibility - providing immediate answers to the Who, What, Where, When, Why and How questions that are the key to effective management.

Standards

Data center convergence and virtualization offer the promise of improved efficiency and flexibility. Developing a network visibility strategy when planning the network upgrades needed to support convergence is essential since only network equipment with embedded traffic monitoring will provide the data center wide visibility essential for effective control.

This article examines the proprietary and standards-based protocols for embedded traffic monitoring as well as their current status and level of support.

Before looking at the standards, it's important to understand how current traffic monitoring protocols fit in the protocol stack. The article, sFlow and NetFlow, describes how sFlow operates at layer 2 (switches) and Cisco NetFlow (and variants from other vendors such as j-Flow, NetStream, LFAP etc.) operates at layer 3 (routers).

There are two organizations responsible for most networking standards:

1. IETF (Internet Engineering Task Force) is responsible for layer 3-7 protocols (IP, routing, DNS, telnet, smtp etc.)
2. IEEE (Institute of Electrical and Electronics Engineers) is responsible for layer 2 protocols (Ethernet, switching, 802.11 etc.)

The IETF has recently developed a standard alternative to proprietary IP flow monitoring protocols (Cisco NetFlow, j-Flow, LFAP, NetStream etc.). The IPFIX standard was created in order "to transfer IP flow data from IPFIX exporters to collectors." This focus on IP flow export is consistent with the IETF's responsibility for the TCP/IP suite of protocols and explicitly avoids addressing layer 2 monitoring (switches). The IPFIX standard was published in 2008. Unfortunately, IPFIX has found little support among router vendors, who continue to implement proprietary solutions.

The IEEE publishes the standards for Ethernet and bridging/switching, and is currently developing the set of standards for Data Center Bridging (DCB) that are driving data center convergence. The IEEE would seem to be the natural place to standardize a protocol for monitoring switched Ethernet traffic, however, the IEEE's focus is on the mechanics of layer 2 connectivity and network management protocols have not been a priority.

The sFlow.org industry consortium was formed in order to develop a multi-vendor standard to address the need for network visibility in layer 2 devices. Many of the members of sFlow.org actively participate in the IEEE standards process, ensuring that sFlow is well matched to the challenge of monitoring current and emerging IEEE standard networks (Ethernet, 40/100G, DCB etc.). The sFlow standard was published in 2001 and it is now implemented by most switch vendors (see sFlow.org).

In selecting a standard for data center visibility, it is important to understand how data center networks are changing. The traditional three layer architecture, in which traffic is moved up to the core and back, does not scale well. Instead, the trend is to integrate access and aggregation layer switches into a flat layer 2 network using shortest-path bridging (IEEE 802.1aq) so that traffic bypasses the core to deliver the increased scalability, bandwidth and reduced latency needed to support converged data center workloads.

The shift in data center network architecture requires a corresponding shift in network management. Instead of monitoring and controlling at layer 3 in the core routers, visibility and control functions move to layer 2 switches and the network edge.

sFlow is the only standard specifically designed for embedded monitoring of layer 2 devices. Selecting switches with embedded sFlow provides a low cost, scalable means of obtaining layer 2-7 visibility into all traffic flowing over the switched network (including storage traffic). Building a network visibility strategy around the sFlow standard maximizes the choice of vendors and ensures interoperable monitoring in mixed vendor environments, eliminating vendor lock-in and facilitating "best in class" product selection.

Multi-vendor support

Multi-vendor support of the sFlow standard has been increasing rapidly over the last 5 years. Initially published by InMon Corp. as RFC 3176, sFlow has grown into the leading, multi-vendor, standard for monitoring high-speed switched networks. The growth in vendor support of sFlow has been driven by the move to 1G and more recently 10G Ethernet switches. The sFlow.org industry consortium, responsible for developing and promoting the sFlow standard, lists the large number of switches that implement sFlow. The switch vendors supporting sFlow now include: HP, IBM, Dell, Brocade, Juniper, BLADE, 3Com, H3C, Force10, Hitachi, AlaxalA, NEC, Alcatel-Lucent, D-Link, Extreme Networks, Allied Telesis and Comtec.

Broad vendor support delivers the network-wide visibility that is essential for managing the convergence of voice, data and storage on the campus and in the data center. It is likely that you have products from one or more of these vendors - if you would like more information on options to extend visibility in your network, ask them about sFlow.

September 11, 2012 Update: An updated version of this article describes the continued growth (nearly doubling) of sFlow support among vendors over the last couple of years, including Cisco's recent support for sFlow.

Trying out sFlow

If you are interested in network-wide visibility and want to start experimenting with sFlow, take a look at your network and see if any of the switches are sFlow capable. Most switch vendors support sFlow, including: Brocade, Hewlett-Packard, Juniper Networks, Extreme Networks, Force10 Networks, 3Com, D-Link, Alcatel-Lucent, H3C, Hitachi, NEC AlaxalA, Allied Telesis and Comtec (for a complete list of switches, see sFlow.org).

If you don't already have switches with sFlow support, consider purchasing a switch to experiment with. There are a number of inexpensive switches with sFlow support (check the list of switches on sFlow.org), alternatively you may be able to pick up a used switch on eBay.

Finally, the open source Host sFlow agent can be used to host traffic and traffic between virtual machines on a virtual server (Xen®, VMware®, KVM).

Once you have access to a source of sFlow data, you will need an sFlow analyzer. The sFlowTrend application (shown above) is a free, purpose built, sFlow analyzer that will allow you to try out the full range of sFlow functionality, including:

• decoding and filtering on data from packet headers (including VLANs, priorities, MAC addresses, Ethernet types, as well as TCP/IP fields)
  
• accurate analysis, trending and reporting of packet samples
• trending of sFlow counters
• support for sFlow MIB to automatically configure sFlow on switches

Many traffic analyzers claim support for sFlow, but provide only partial support. It is worth starting with sFlowTrend to see the full capabilities of sFlow and to gain experience with sFlow monitoring before evaluating larger scale solutions.

Future posts on this blog will use sFlowTrend to demonstrate how sFlow monitoring can be used to solve common network problems. Downloading a copy of sFlowTrend will allow you to try the different strategies on your own network.

Why an sFlow blog?

sFlow has been quietly emerging as the leading multi-vendor standard for monitoring traffic in switched networks. The first sFlow capable switches were shipped in 2001. Since then vendor support has been increasing each year and now switches from Brocade, Hewlett-Packard, Juniper Networks, Extreme Networks, 3Com, D-Link, Alcatel-Lucent, H3C, Hitachi, NEC, AlaxalA, Allied Telesis and Comtec have embedded sFlow. Chances are you already have switches in your network that support sFlow (for a complete list of switches that support sFlow, see Network Equipment).

sFlow monitoring solutions provide traffic visibility for the full range of switched networks, from an organization running a large, busy 10G network (see Amsterdam Internet Exchange) to a small office trying to see who is hogging bandwidth on their T1 link (see sFlowTrend).

Through the postings in this blog, we hope to increase awareness of sFlow and the way it can be used to solve problems that challenge network administrators every day.