Saturday, May 29, 2010


How do you identify the network traffic associated with a physical or virtual server?  The need to answer this type of question becomes critical as data center convergence demands close coordination between the network, storage and system teams (see Management silos). For example, suppose a system manager is considering migrating a virtual machine and wants to make sure that the move won't be disruptive to the network (see Network visibility in the data center). Providing an answer to this question can be extremely difficult without the right tools.

The diagram shows the complex relationship between virtual machines, virtual network adapters, virtual switches, physical adapters and the data center network of physical switches. A single virtual machine may have more than one virtual network adapter, each of which is connected to a different virtual switch. Each virtual switch may be connected by one or more physical network adapters to more than one physical switch. Each physical and virtual machine may have an IP address for each of its network adapters. In addition, there may be large numbers of additional IP addresses allocated to physical or virtual machines in order to support applications (for example a web server may have an IP address associated with each web site that it hosts). Finally, the task is made even more complicated by virtual machine mobility since traffic data from multiple locations needs to be combined to report on the virtual machine.

The first step in meeting this challenge must be addressed at the time the data center network is designed. Data center networks are currently being upgraded in order to provide the flat, high-speed, layer 2 fabric needed to facilitate virtual machine mobility. As part of the network refresh, selecting switches with embedded traffic monitoring adds little to the cost of the network, but provides the instrumentation needed for network-wide visibility.

The sFlow standard is specifically designed for embedded monitoring of layer 2 devices. Selecting switches with embedded sFlow provides a low cost, scalable means of obtaining layer 2-7 visibility into all traffic flowing over the switched network (including storage traffic). Building a network visibility strategy around the sFlow standard maximizes the choice of vendors and ensures interoperable monitoring in mixed vendor environments, eliminating vendor lock-in and facilitating "best in class" product selection.

However, even with complete visibility into the network traffic, it is still challenging to identify which traffic belongs to a particular virtual machine. Looking at network traffic you can see MAC addresses and IP addresses, but you don't know which addresses belong to which virtual machine. Complicating matters further, the set of addresses associated with a particular virtual machine may change over time. What is needed is a single identifier that is uniquely associated with the virtual machine.

System administrators face this challenge every day. By associating a UUID (Universally Unique Identifier) with each virtual and physical machine they are able to uniquely identify that machine, even if its network configuration is changed.

Previous articles have looked at the visibility into system performance that Host sFlow provides (see Cluster performance and Top servers). In addition, to the performance counters, Host sFlow exports the UUID corresponding to each physical and virtual machine, along with the MAC addresses associated with each of its physical or virtual network adapters. An sFlow analyzer (see Choosing an sFlow analyzer) can combine sFlow from hosts and switches to provide a complete picture network activity.

From the network managers point of view, the UUID becomes just another attribute of network traffic that can be filtered and reported on. With a common language it is easy for network and system managers to coordinate their activities. This time around, the system manger can ask the question, "What is the network impact of moving virtual machine, UUID:a45a256c-c24b-2976-a931-8f2b90b1e026?"

Given the UUID, the sFlow analyzer can trend traffic for the virtual machine, irrespective of its location and even during migrations. The sFlow monitoring system is always aware of the current location of the virtual machine and the MAC and IP addresses associated with it.  The trend chart above shows a real-time picture of traffic associated with the virtual machine. It's easy to see that the bulk of the traffic is NFS storage traffic associated with the storage server (

To assess the impact of moving the virtual machine, it is useful to visualize the traffic clustered by physical switch. The chart above shows traffic flows associated with the virtual machine clustered by switch. The virtual machine and NFS server are clearly attached to different switches. Ideally, you would like to minimize the number of network hops that the storage traffic takes (in order to reduce load on the network core and reduce disk IO latency for the virtual machine). Suggesting that the virtual machine by migrated to the same switch as the NFS server, or a location nearby, would accomplish this goal.

The close coordination of network, system and storage management captured in this example is critical for successful management of converged data centers (see Management silos). This level of coordination is only possible because the data center was designed with visibility in mind. The switches, virtual switches and servers have all been instrumented with sFlow in order to provide an integrated, data center wide, view of performance.

Saturday, May 15, 2010

Configuring Open vSwitch

A cloud data center consists of large numbers of physical servers, each running a hypervisor with one or more virtual switches connecting the physical network to virtual machines (see Anatomy of an open source cloud). Coordinating and managing the configuration of the large numbers of virtual switches in a cloud data center is a complex task.

In order to address this challenge, the Open vSwitch (part of the Xen Cloud Platform) configuration mechanism has moved to using the OpenFlow configuration protocol. The OpenFlow configuration protocol makes it possible to centrally manage the configuration of all the Open vSwitch instances in the cloud in a way that was not possible using the previous configuration file method.

The ovs-vsctl command line tool (installed as part of the Open vSwitch package) can be used to issue configuration commands to the vSwitch.

The following commands configure an Open vSwitch to sample packets at 1-in-512, poll counters every 30 seconds and send sFlow to an analyzer ( over UDP using the default sFlow port (6343):

SFLOWUUID=`ovs-vsctl create sFlow agent=eth0 target=\"\" header=128 sampling=512 polling=30`

Find the set of bridges that have been created:

ovs-vsctl list br

Apply the sFlow settings to each bridge (in this example, xenbr0 and xenbr1):

ovs-vsctl set bridge xenbr0 sflow=$SFLOWUUID
ovs-vsctl set bridge xenbr1 sflow=$SFLOWUUID

A previous posting discussed the selection of sampling rates. Additional information can be found on the Open vSwitch web site.

See Open vSwitch for additional information on monitoring the vSwitch using sFlow.

Finally, sFlow monitoring of the vSwitches is part of an overall cloud performance monitoring strategy. The sFlow standard unifies cloud data center management by providing visibility into the physical network (see Multi-vendor support), virtual network (see Open vSwitch and Virtual routing), servers (see Cluster performance and Top servers), virtual servers (see Host sFlow) and storage (see Networked storage) to provide complete end-to-end visibility into cloud performance.

Feb. 15, 2011 Update: Updated information on managing the Open vSwitch on different virtualization platforms has been posted to this blog, just click on the vSwitch label below to see the most recent articles.