Convergence

The diagram shows technologies that are part of the drive toward converge of storage, server and network technologies in the data center.

Virtualization and the need to support virtual machine mobility (e.g. vMotion/XenMotion/Xen Live Migration) is driving the adoption of large, flat, high-speed, layer-2, switched Ethernet fabrics in the data center. A layer-2 fabric allows a virtual machine to keeps its IP address and maintain network connections when it moves (performing a "live" migration).

Networked storage (iSCSI, NFS, FCoE) is also needed to support virtual machine mobility. In addition, moving storage from dedicated SANs to a converged Ethernet fabric reduced cabling and networking costs and improves flexibility. However, migration of storage traffic onto a converged Ethernet fabric dramatically increases bandwidth demands and is one of the factors accelerating the adoption of 10/40/100G Ethernet.

Ethernet standards are evolving to address the needs of convergence. As well as higher speeds, IEEE data center bridging standards add support for lossless Ethernet to improve storage performance and support Fiber Channel over Ethernet (FCoE). The performance and stability of large layer 2 Ethernets is being addressed through new protocols such as shortest path bridging.

Changes are not limited to storage and data networking. Server architectures are changing as convergence blurs the line between the server and the network, extending the Ethernet fabric into servers through blade switches, network adapters and virtual switches.

The current siloed approach to managing storage, servers and networks no longer works in a converged environment where each of these areas is so closely inter-dependent. Fortunately, convergence to an Ethernet fabric brings with it the data center visibility needed to manage the converged data center.

The sFlow standard, implemented by most vendor's Ethernet switches, simplifies management by providing the unified visibility and control needed to fully realize the benefits of virtualization and convergence.

Virtual routing

The diagram shows networking elements within a virtual server. The server's physical network adapters connect to LAN switches that provide a high speed, flat, layer 2, fabric connecting servers and storage in the data center. Virtual switches provide shared access to the physical adapters, connecting the virtual network adapters in the virtual machines to the physical network.

Current hierarchical network designs confine routing to specialized hardware at the core of the network.  It is worth re-examining the place of routing given the changes in data center architecture brought about by convergence and virtualization. What if routing could be virtualized?

The performance of software routers running on commodity x86 hardware is improving,  Vyatta recently announced 20 Gbps routing performance from their software routers. With network adapter support for virtualization (e.g. SR-IOV), it is now feasible to implement high-performance routing and firewall functionality in virtual machines.

Virtualization of routing offers a number of advantages:

1. Virtualization allows services to be replicated and deployed where they are needed in the virtual infrastructure. A virtual router can easily be replicated to provide redundancy or add capacity.
2. Virtual routing can provide better reliability and lower costs by making use of the general purpose virtual server infrastructure, eliminating the need for expensive, specialized router hardware.
3. Distributing routing to the edge of the network reduces pressure on the core and improves scalability. 

There benefits don't just apply to routing, many other specialized devices can also be virtualized, including firewalls, load balancers, proxies etc. Virtualization of layer 3-7 network devices on a high performance converged Ethernet fabric offers a flexible and dynamic infrastructure that can easily be reconfigured to meet changing demands.

To illustrate the potential of virtualized networking, consider the example of a hosted data center. In a typical hosted data center, customers have racks or partial racks of equipment installed in the data center. A typical customer will have their own router, firewall, load balancer and servers installed in the rack. A virtual rack can be constructed by deploying routing and firewall virtual machines along with general purpose virtual machines that the customer can use to deploy their applications. A virtual rack can be provisioned and maintained automatically, providing customers with much more responsive service while reducing operating costs. In addition, virtualization allows higher customer densities per physical rack, increasing the revenue that can be generated per rack.

The benefits aren't restricted to service provider networks. In enterprise data centers, the flexibility of virtualized networking allows for more efficient management and utilization of resources. However, a barrier to realizing these benefits is the current siloed approach to data center management. Close coordination is needed between network and system management teams. For example, who would be responsible for provisioning and configuring a virtual router? This type of cross functional task is a challenge for most organizations.

Integrated traffic monitoring provides the visibility needed for effective management of virtualized networks. The diagram shows some of the data paths that are possible in a virtual stack: the red line shows traffic between two physical VLANs connected by a virtual router and the gold line shows traffic routed between two virtual machines hosted on the same server. In order to provide network visibility, every networking device, physical or virtual needs to include integrated traffic monitoring so that all traffic paths can be observed. Shared visibility into all resources in the data center ensures that each group (network, systems and storage) is aware of its impact on shared resources, eliminates finger pointing, improves coordination and lays the foundation for automating control.

There are many proprietary and standard technologies for embedded traffic monitoring. Broadly speaking these fall into two classes, TCP/IP flow monitoring built into many routers (e.g. Cisco NetFlow) and multi-protocol packet-based monitoring built into most switches. Convergence in both the LAN (data center bridging) and the WAN (Metro Ethernet and Carrier Ethernet) is taking place using Ethernet technologies, making the sFlow standard the logical choice for visibility since it enjoys broad, multi-vendor support and is already built into most vendor's Ethernet products. Just as convergence to Ethernet simplifies connectivity, convergence to sFlow standard monitoring built into Ethernet devices simplifies management of the converged network.

Products incorporating the sFlow standard provide visibility throughout the physical switch, virtual switch, virtual router and cloud layers, delivering the end to end visibility needed to realize the full benefits of virtualization and convergence.

802.1aq and TRILL

There are a number of drivers increasing demand for bandwidth in the data center:

• Multi-core processors and blade servers greatly increase computational density, creating a corresponding demand for bandwidth.
• Networked storage increases demand for bandwidth.
• Virtualization and server consolidation ensures that servers are fully utilized, further increasing demand for bandwidth.

Virtual machine mobility (e.g. VMWare vMotion, Citrix XenMotion or Xen Live Migration) require a large, flat layer 2 network so that machines can be moved without reconfiguring their network settings. The increasing size of the layer 2 network, combined with the increasing demand for bandwidth challenges the scalability of current Ethernet switching technologies.

The diagram illustrates the problem. Currently, Ethernet uses the spanning tree protocol to determine forwarding paths. The tree structure forces traffic to the network core, creating a bottleneck. In addition, the tree structure doesn't allow traffic to flow on backup links, further limiting usable bandwidth. An alternative forwarding technique (used by routers), is to select shortest paths through the network. Shortest path forwarding allows traffic flows to bypass the core, reducing the bottleneck. The added benefit of shortest path forwarding is its ability to make use of all the links, including backup links, further increasing capacity.

There are two emerging standards for shortest path forwarding in switches:

• TRILL (Transparent Interconnect of Lots of Links)
• IEEE 802.1aq (Shortest Path Bridging)

Both TRILL and 802.1aq use IS-IS (Intermediate System to Intermediate System) routing to select shortest paths through the network. The protocols are very similar, but are being proposed by two different standards bodies: TRILL is being developed by IETF and 802.1aq by the IEEE.

It is surprising to see the IETF working on a LAN bridging protocol. The IETF is responsible for Internet protocols (TCP/IP, routing) and the IEEE is responsible for LAN protocols (802.11, Ethernet, bridging/switching). Adopting the IEEE 802.1aq standard makes the most sense, since it will ensure interoperability with the IEEE data center bridging standards being developed to support FCoE and facilitate data center convergence.

Finally, while more efficient network topologies will help increase network capacity, the days of relying on network over-provisioning are over. Much tighter control of bandwidth is going to be required in order to cope with converged data center workloads. Selecting switches that support the sFlow standard provides the visibility and control needed to manage the increasing demand for bandwidth.

Ethernet growth

There are a number of drivers increasing demand for bandwidth in the data center:

• Multi-core processors and blade servers greatly increase computational density, creating a corresponding demand for bandwidth.
• Networked storage increases demand for bandwidth.
• Virtualization and server consolidation ensures that servers are fully utilized, further increasing demand for bandwidth.

The chart shows a projection of the adoption of higher speed Ethernet server interconnects as the increasing demand for bandwidth drives current deployment of 10G Ethernet and accelerates adoption of emerging 40G and 100G Ethernet products.

The scalability, performance and relatively low cost makes Ethernet the clear choice of networking technology for providing converged SAN and LAN connectivity. The sFlow standard, already supported by most Ethernet switch vendors, provides the network visibility that is essential for managing data centers in this rapidly evolving environment.

LAN and WAN

There is widespread confusion about the differences between sFlow and NetFlow and they are often simply referred to collectively as xFlow, implying that the two technologies are interchangeable. The sFlow and NetFlow posting described some of the technical differences between the two technologies, in particular, describing how sFlow operates at the Switch/Ethernet/LAN level and NetFlow operates at the Router/IP/WAN level. This division helps understand where the two technologies fit in the market.

The chart breaks down networking into switching and routing on the x-axis and network speed on the y-axis and then plots the application areas for sFlow and NetFlow. Since sFlow is built into switch ASICs, it offers monitoring solutions that span the full range of layer 2-3 switching products, from inexpensive switches aimed at office and small business environments, to the most demanding applications in supercomputer data centers, Internet exchange points and digital effects render farms. NetFlow is typically found in enterprise class routers. Since performance critical components of NetFlow are often implemented in software, NetFlow isn't widely used for monitoring at the high end of the router market (tier-1 ISPs). The cost of NetFlow enabled equipment limits its use at the low end of the router market.

Dividing the market into routing and switching products and the related applications helps explain why some vendors support sFlow while others support NetFlow. The chart also explains why a vendor might offer sFlow on their switch products and NetFlow on their router products. In practice, most networks blend switching and routing in order to meet the varied requirements of the different services running on the network. In many cases, a network monitoring strategy that embraces both sFlow and NetFlow delivers the most complete visibility into network activity.

Networked storage

A previous posting discussed how sFlow is used to provide visibility in the data center. This post looks more closely at the challenge posed by networked storage.

There are many good reasons to use networked storage: the storage resources can be shared, replicated and backed up independently of the systems that use them. In a virtual server environment, using networked storage for the virtual machine images simplifies the replication of virtual machines and the migration of virtual machines between servers (e.g. VMWare vMotion, Citrix XenMotion or Xen Live Migration).

In addition, the migrating of storage from a dedicated storage area network (SAN) to a single Converged Enhanced Ethernet (CEE) network promises to reduce cost and create a more flexible data center infrastructure. However, this migration also places additional demands on the LAN infrastructure.

Regardless of the type of networked storage (iSCSI, NFS, AoE or FCoE), the management of network bandwidth is critical to successful deployment and operation. For example, the chart above shows site-wide traffic from a large campus network broken out by protocol. The storage traffic (iSCSI) is clearly the largest load on the network, dwarfing the amount of web (HTTP) traffic.

The visibility into network traffic provided by sFlow is critical to effectively managing network resources. If the network is poorly provisioned, congestion associated with storage traffic will degrade quality of service (QoS) for other applications on the network and impair system performance since network congestion will also manifest itself as slow disk performance.

Network visibility in the data center

Current trends toward Virtualization, Converged Enhanced Ethernet (CEE), Fibre Channel over Ethernet (FCoE), Service Oriented Architectures (SOA) and Cloud Computing are part of a broader re-architecture of the data center in which enterprise applications are decomposed into simpler elements that can be deployed, moved, replicated and connected using high-speed switched Ethernet.

The following example, shown in the diagram, illustrates the management challenges faced in this new environment. A system manager decides to move a virtual machine from one server to another. The system management tools show that there is plenty of capacity on the destination server and this looks like a safe move. Unfortunately, the move causes the storage traffic, which had previously been confined to a single switch, to congest links across the data center causing system wide performance problems.

In this new environment the traditional siloed approach in which different teams manage the network, storage and servers does not work. An integrated approach to management is needed if the full benefits of a converged data center are to be achieved. Ensuring network-wide visibility into the storage, network and services running in the data center, their traffic volumes and dependencies is a critical component of an integrated management strategy.

In order to achieve data center wide visibility, every layer of the data center network, including the core, distribution, top of rack and blade server switches, needs to be instrumented. This might seem like a daunting (and expensive!) challenge. However, most vendors have integrated sFlow into their switch products. The sFlow standard provides a proven solution that is available in network products from the leading computer and network vendors, including HP, IBM, Dell, Brocade, BLADE, Juniper, Force10 and 3Com (for a complete list, see sFlow.org). Making sFlow a requirement when building out a new data center is a sound investment, adding very little to the cost of the network, but ensuring that the visibility needed safely deploy, optimize and scale up new services is available to the operations team.