OpenFlow

(diagram from OpenFlow: Enabling innovation in Campus Networks)

OpenFlow is a new protocol for controlling how packets are forwarded through switches. Currently switches implement control and forwarding logic within the same device. OpenFlow makes use of existing forwarding hardware but moves the control logic out of the switch to a central controller. The architecture provides fine grain control of traffic, making it possible to engineer the network to implement customized security and performance management strategies.

An interesting video demo shows how OpenFlow can be used to maintain a live connection between two laptops as they move around a wireless network. Similar challenges exist in data centers in maintaining connectivity as virtual machines migrate.

The combination of OpenFlow and sFlow offers exciting possibilities, combining the network wide visibility of sFlow with the centralized control capabilities of OpenFlow creates a dynamic feedback control system that can be used to create intelligent, self optimizing networks.

Data center traffic

(chart from  Flyways To De-Congest Data Networks)

Very few studies of data center traffic have been published since the challenge of instrumentation and the confidentiality of the data create significant obstacles for researchers.

The following papers are some of the few that contain traffic data from corporate data centers:

• A First Look at Modern Enterprise Traffic (2005), broad comparison of enterprise and Internet traffic based on traces captured at Lawrence Berkeley National Laboratory
• How Healthy are Today's Enterprise Networks? (2008), uses data collected from end hosts to assess the number of unsuccessful connection attempts in an enterprise network
• Understanding Data Center Traffic Characteristics (2009), surveys 19 corporate data centers
• The Nature of Datacenter Traffic: Measurements & Analysis (2009), provides detailed information on Hadoop/Map-Reduce network traffic 
• What Goes Into a Data Center? (2009), overview of data center components and management challenges, including: power, servers, networking and software
• Flyways To De-Congest Data Networks (2009), examines congestion in a data center network

The papers are worth reading in their entirety, but some highlights stand out:

• "The characteristics of traffic inside the Internet enterprises remain almost wholly unexplored"
• "80% of packets stay in data center ... Trend is towards even more internal communication"
• 90 - 95% of the network devices in large data centers are edge devices
• "We find that utilization is significantly higher in the core than in the aggregation and edge layers"
• "Given the large number of unused links (40% are never used), an ideal traffic engineering scheme would split traffic across the over-utilized and under-utilized links"
• "Our data shows that a map-reduce style data mining workload results in sparse demand matrices"
• "At any time only a few ToR [Top of Rack] switches are bottlenecked"
• "Today, computation constrained by network"

In order to remove the network bottlenecks that can affect the performance of applications in the data center, a number of architectures have been developed to create a non-blocking data center network, including Fat-tree and VL2.

(table from Flyways To De-Congest Data Networks)

However, eliminating over-subscription in the network is expensive, ranging from 2 to 5 times the cost of a conventional network design. In addition, the cost to power and manage the increased number of links and switches adds significantly to the operating cost of the data center.

Applying this same strategy to the road system would be the equivalent of connecting every town and city with an 8-lane freeway, no matter how small or remote the town. In practice, traffic studies guide development and roads are built where they are needed to satisfy demand. A similar, measurement-based, approach can be applied to network design.

The measurement studies show that networks already contain many unused and under-utilized links. Instead of using the brute force approach of adding capacity, an alternative strategy is to use network visibility to utilize existing bandwidth more intelligently and target upgrades where they are most needed.

Data center visibility is made possible by the sFlow standard, currently supported by most switch vendors. Network switches with sFlow deliver the visibility into utilization, topology and traffic needed for effective control of data center resources, ensuring that the benefits of convergence and virtualization can be fully realized.

Standards

Data center convergence and virtualization offer the promise of improved efficiency and flexibility. Developing a network visibility strategy when planning the network upgrades needed to support convergence is essential since only network equipment with embedded traffic monitoring will provide the data center wide visibility essential for effective control.

This article examines the proprietary and standards-based protocols for embedded traffic monitoring as well as their current status and level of support.

Before looking at the standards, it's important to understand how current traffic monitoring protocols fit in the protocol stack. The article, sFlow and NetFlow, describes how sFlow operates at layer 2 (switches) and Cisco NetFlow (and variants from other vendors such as j-Flow, NetStream, LFAP etc.) operates at layer 3 (routers).

There are two organizations responsible for most networking standards:

1. IETF (Internet Engineering Task Force) is responsible for layer 3-7 protocols (IP, routing, DNS, telnet, smtp etc.)
2. IEEE (Institute of Electrical and Electronics Engineers) is responsible for layer 2 protocols (Ethernet, switching, 802.11 etc.)

The IETF has recently developed a standard alternative to proprietary IP flow monitoring protocols (Cisco NetFlow, j-Flow, LFAP, NetStream etc.). The IPFIX standard was created in order "to transfer IP flow data from IPFIX exporters to collectors." This focus on IP flow export is consistent with the IETF's responsibility for the TCP/IP suite of protocols and explicitly avoids addressing layer 2 monitoring (switches). The IPFIX standard was published in 2008. Unfortunately, IPFIX has found little support among router vendors, who continue to implement proprietary solutions.

The IEEE publishes the standards for Ethernet and bridging/switching, and is currently developing the set of standards for Data Center Bridging (DCB) that are driving data center convergence. The IEEE would seem to be the natural place to standardize a protocol for monitoring switched Ethernet traffic, however, the IEEE's focus is on the mechanics of layer 2 connectivity and network management protocols have not been a priority.

The sFlow.org industry consortium was formed in order to develop a multi-vendor standard to address the need for network visibility in layer 2 devices. Many of the members of sFlow.org actively participate in the IEEE standards process, ensuring that sFlow is well matched to the challenge of monitoring current and emerging IEEE standard networks (Ethernet, 40/100G, DCB etc.). The sFlow standard was published in 2001 and it is now implemented by most switch vendors (see sFlow.org).

In selecting a standard for data center visibility, it is important to understand how data center networks are changing. The traditional three layer architecture, in which traffic is moved up to the core and back, does not scale well. Instead, the trend is to integrate access and aggregation layer switches into a flat layer 2 network using shortest-path bridging (IEEE 802.1aq) so that traffic bypasses the core to deliver the increased scalability, bandwidth and reduced latency needed to support converged data center workloads.

The shift in data center network architecture requires a corresponding shift in network management. Instead of monitoring and controlling at layer 3 in the core routers, visibility and control functions move to layer 2 switches and the network edge.

sFlow is the only standard specifically designed for embedded monitoring of layer 2 devices. Selecting switches with embedded sFlow provides a low cost, scalable means of obtaining layer 2-7 visibility into all traffic flowing over the switched network (including storage traffic). Building a network visibility strategy around the sFlow standard maximizes the choice of vendors and ensures interoperable monitoring in mixed vendor environments, eliminating vendor lock-in and facilitating "best in class" product selection.

Large Hadron Collider

LHCb (Photo Credit: CERN)

The Large Hadron Collider at CERN has been in the news as it comes online.

An interesting paper (Management of the LHCb Network Based on SCADA System) describes the data collection network associated with the LHCb experiment. High-speed switched Ethernet networks are used to collect measurements from the experiment and to control its operation. The paper states that, "Sophisticated monitoring of both networks at all levels is essential for the successful operation of the experiment."

The network monitoring system uses sFlow to measure network utilization on the core switches. "Because there are so many ports in the core switches, the SNMP query of interface counters takes a long time and occupies a lot CPU and memory resource."

The distributed counter polling mechanism in sFlow provides a highly scalable alternative to SNMP polling, delivering reliable monitoring in the most demanding environments. Network visibility in the data center is equally important and sFlow provides the scalability and performance needed to maintain effective control of high-speed data center networks.

Vyatta

Vyatta's open-source software router provides a way of adding routing and security functions to virtual environments (Xen, VMWare and Hyper-V).

Vyatta recently announced the availability of the Core 6 Alpha release, adding sFlow support. This early alpha release (the production version in due in early 2010) can be downloaded now. If you want to try it out, the bootable live CD version offers a quick way to see how the router works without dedicating a server to the project.

The following configuration sets the sampling rate on interface eth0 to 1-in-100 and sends sFlow to a collector at 10.0.0.50:

system {
 flow-accounting {
  interface eth0 {
   sampling-rate 100
  }
  sflow {
   agentid 0
   server 10.0.0.50 {
   }
  }
 }
}

Note: A previous posting discussed the selection of sampling rates. Additional information can be found on the Vyatta web site.

The screen capture was generated using sFlowTrend to monitor the Vyatta router. Both the Vyatta router software and sFlowTrend can be downloaded at no charge, offering an inexpensive way to try out sFlow.

Making sure that all network devices (both physical and virtual) support sFlow provides the data center wide visibility needed for effective control of network resources. sFlow is already supported by most hardware vendors and Vyatta's support for sFlow extends visibility into the virtual network.

Probes

The RMON (Remote MONitoring) standard was developed in the early 1990's to standardize network monitoring devices (usually referred to as "probes"). At the time, Ethernet LANs consisted of coax cables that where shared by a number of hosts. Repeaters were used to connect the the cables and extend the network. In this environment, a single RMON probe would see all the traffic on the shared network, providing complete network visibility.

Multi-port switches started to become popular in the mid 1990's and SPAN/mirror ports were added to switches to continue to allow probe-based monitoring. The increasing number of ports per switch and the increasing port speeds has made the use of probes a challenge. The need for embedded instrumentation was becoming clear.

In the late 1990's, Cisco introduced the NetFlow protocol, embedding L3-4 monitoring in routers and in 2001 the sFlow protocol was introduced, embedding L2-7 monitoring in switches. Interest in network visibility has accelerated the adoption of the sFlow standard among switch vendors, further limiting the role of probes.

The chart clearly shows the trend toward embedded monitoring. Google Insight for Search was used to trend the popularity of the search terms sFlow, NetFlow, RMON and Probe compared to overall searches relating to Network Monitoring & Management. The sFlow and NetFlow lines track closely and exceed general interest in Network Monitoring & Management, indicating that they are increasingly important topics. The RMON and Probe lines track closely with each other and show a rapid decline compared to Network Monitoring & Management, indicating declining interest in probes as monitoring shifts from probes to embedded instrumentation.

Current trends toward data center convergence increase the need for visibility and control. Complete network visibility is likely to involve both NetFlow and sFlow. NetFlow provides visibility into routers while sFlow extends visibility into the increasingly important switching layer, including: virtual servers, blade servers and edge switches.

Note: The overall downward trend in all the lines results from the increasing population of Internet users. As more people use the Internet, the proportion of Internet users interested in any one topic is diluted. This is particularly true of technical topics. In the past, the technical barriers to using the Internet skewed the user population and resulted in more searches relating to technical topics. Now that everyone is online, the majority of searches relate to more populist topics.

Network visibility

The chart, created using Google News Timeline, trends the growing interest in "network visibility" from 1990 to 2008. The dot-com bubble (2000) shows up as a clear spike, followed by a plateau. Starting in 2005 interest has accelerated with strong annual gains.

The sFlow standard, designed to provide network-wide visibility, was introduced in 2002 and the rapid growth in multi-vendor support for sFlow closely tracks this chart.

Finally, current trends toward data center convergence and virtualization create new management challenges that continue to drive interest in the visibility and control made possible by sFlow.