AoE

(credit What is Ethernet SAN & AoE?) 

The ATA over Ethernet (AoE) protocol provides a method of carrying Storage Area Network (SAN) traffic over a switched Ethernet network. AoE provides networked storage that supports virtualization and cloud computing applications.

An AoE SAN consists of commodity hardware: SATA disks, Ethernet adapters and switches. The result is a low cost, high performance networked storage solution. The article, What is Ethernet SAN & AoE?, includes technical details and a discussion of the benefits of AoE as a SAN technology.

Visibility into network activity is essential for maintaining optimal performance in a converged network since storage traffic  tends to dominate network workloads. Fortunately, one of the benefits of using an Ethernet SAN is increased network visibility since most switch vendors implement the sFlow standard in their switch hardware.

One of the unique features of the sFlow standard is that any sFlow capable switch will monitor all types of traffic flowing across the network, including non-IP SAN protocols such as AoE and FCoE. The sFlow data exported by the switches includes packet header information that permits an sFlow analyzer to report on new types of traffic as they are deployed on the network without any need to upgrade switch firmware or hardware.

The following chart trends AoE operations on a SAN using sFlow data from Ethernet switches:

The chart shows the detailed visibility into AoE servers, clients, operations and targets available through sFlow monitoring. In addition, sFlow is being used to monitor the performance of the systems attached to the Ethernet SAN. Using sFlow to monitor servers links MAC addresses associated with AoE connections to server host names, UUIDs and performance metrics. The chart above demonstrates the additional context that integrated network and server monitoring provides; showing that while there are two different server MAC addresses (FAA69ECD3D6F and EABF12E86CD0), they are in fact associated with a single storage server (UUID e9d0e087-51c7-429e-9841-4439db35f3be).

The following charts make use of sFlow data from the server to trend server load:

In this case, it appears that the storage server is the performance bottleneck. Upgrading the server, or moving to a dedicated AoE storage appliance (e.g. Coraid EtherDrive) should significantly boost performance.

Network convergence using a high-speed, flat, layer 2 fabric creates a simple, flexible infrastructure that supports virtualization and cloud computing. Choosing switches that support the sFlow standard provides the unified view of network and system performance needed for effective control of data center resources. For additional information, the Data center convergence, visibility and control presentation describes the critical role that measurement plays in managing costs and optimizing performance.

FCoE

The Fiber Channel over Ethernet (FCoE) protocol provides a method of carrying Storage Area Network (SAN) traffic over a switched Ethernet network. FCoE adds to the list of options for converged storage and data networking (see Networked storage) that provide the flexibility needed to support virtualization and cloud computing.

Network visibility is critical for managing performance in a converged network. The sFlow standard embeds monitoring in Ethernet switches and provides visibility into all types of traffic flowing over the converged network (see Packet headers), including new protocols such as FCoE. The chart at the top of this page was created using sFlow data from a switch and shows a real-time view of FCoE traffic flowing through the switch (see Choosing an sFlow analyzer).

Most data center switch vendors already support the sFlow standard. Selecting switches with sFlow support provides the visibility needed for effective management and control of a converged network.

Ethernet growth

There are a number of drivers increasing demand for bandwidth in the data center:

• Multi-core processors and blade servers greatly increase computational density, creating a corresponding demand for bandwidth.
• Networked storage increases demand for bandwidth.
• Virtualization and server consolidation ensures that servers are fully utilized, further increasing demand for bandwidth.

The chart shows a projection of the adoption of higher speed Ethernet server interconnects as the increasing demand for bandwidth drives current deployment of 10G Ethernet and accelerates adoption of emerging 40G and 100G Ethernet products.

The scalability, performance and relatively low cost makes Ethernet the clear choice of networking technology for providing converged SAN and LAN connectivity. The sFlow standard, already supported by most Ethernet switch vendors, provides the network visibility that is essential for managing data centers in this rapidly evolving environment.

Networked storage

A previous posting discussed how sFlow is used to provide visibility in the data center. This post looks more closely at the challenge posed by networked storage.

There are many good reasons to use networked storage: the storage resources can be shared, replicated and backed up independently of the systems that use them. In a virtual server environment, using networked storage for the virtual machine images simplifies the replication of virtual machines and the migration of virtual machines between servers (e.g. VMWare vMotion, Citrix XenMotion or Xen Live Migration).

In addition, the migrating of storage from a dedicated storage area network (SAN) to a single Converged Enhanced Ethernet (CEE) network promises to reduce cost and create a more flexible data center infrastructure. However, this migration also places additional demands on the LAN infrastructure.

Regardless of the type of networked storage (iSCSI, NFS, AoE or FCoE), the management of network bandwidth is critical to successful deployment and operation. For example, the chart above shows site-wide traffic from a large campus network broken out by protocol. The storage traffic (iSCSI) is clearly the largest load on the network, dwarfing the amount of web (HTTP) traffic.

The visibility into network traffic provided by sFlow is critical to effectively managing network resources. If the network is poorly provisioned, congestion associated with storage traffic will degrade quality of service (QoS) for other applications on the network and impair system performance since network congestion will also manifest itself as slow disk performance.

Network visibility in the data center

Current trends toward Virtualization, Converged Enhanced Ethernet (CEE), Fibre Channel over Ethernet (FCoE), Service Oriented Architectures (SOA) and Cloud Computing are part of a broader re-architecture of the data center in which enterprise applications are decomposed into simpler elements that can be deployed, moved, replicated and connected using high-speed switched Ethernet.

The following example, shown in the diagram, illustrates the management challenges faced in this new environment. A system manager decides to move a virtual machine from one server to another. The system management tools show that there is plenty of capacity on the destination server and this looks like a safe move. Unfortunately, the move causes the storage traffic, which had previously been confined to a single switch, to congest links across the data center causing system wide performance problems.

In this new environment the traditional siloed approach in which different teams manage the network, storage and servers does not work. An integrated approach to management is needed if the full benefits of a converged data center are to be achieved. Ensuring network-wide visibility into the storage, network and services running in the data center, their traffic volumes and dependencies is a critical component of an integrated management strategy.

In order to achieve data center wide visibility, every layer of the data center network, including the core, distribution, top of rack and blade server switches, needs to be instrumented. This might seem like a daunting (and expensive!) challenge. However, most vendors have integrated sFlow into their switch products. The sFlow standard provides a proven solution that is available in network products from the leading computer and network vendors, including HP, IBM, Dell, Brocade, BLADE, Juniper, Force10 and 3Com (for a complete list, see sFlow.org). Making sFlow a requirement when building out a new data center is a sound investment, adding very little to the cost of the network, but ensuring that the visibility needed safely deploy, optimize and scale up new services is available to the operations team.