|Figure 1: Network virtualization (credit Brad Hedlund)|
|Figure 2: Physical and virtual packet paths|
|Figure 3: Apparent virtual network traffic matrix|
|Figure 4: Relative cost of different topologies (from Flyways To De-Congest Data Networks)|
In fact, the traffic matrix isn't random, it just appears random because the virtual machines have been randomly scattered around the data center by the network virtualization layer. Consider an important use case for network virtualization - multi-tenant isolation. Virtual networks are created for each tenant and configured to isolate and protect tenants from each other in the public cloud. Virtual machines assigned to each tenant are free to communicate among themselves, but are prevented for communicating with other tenants in the data center.
|Figure 5: Traffic matrix within and between tenants|
- The largest tenant occupies a small fraction of the total area in the traffic matrix.
- Tenant size rapidly decreases with most tenants being much smaller than the largest few.
- The traffic matrix is extremely sparse.
|Figure 6: Traffic from large map / reduce cluster|
Note: There is a striking similarity between the traffic matrices in figures 5 and 6. The reason for the strong diagonal in the Hadoop traffic matrix is that the Hadoop scheduler is topologically aware, assigning compute tasks to nodes that are close to the storage they are going to operate on, and orchestrating storage replication in order to minimise non-local transfers. However, when this workload is run over a virtualized network, the virtual machines are scattered, turning this highly localized and efficient traffic pattern into randomly distributed traffic.Apart from Hadoop, how else might a large tenant use the network? It's worth focusing on large tenants since their workloads are likely to be the hardest to accomodate. Netflix is one of the largest and most sophisticated tenants in the Amazon Elastic Compute Cloud (EC2) and the presentation, Dynamically Scaling Netflix in the Cloud provides some interesting insights into their use of cloud resources.
|Figure 7: Netflix elastic load balancing pools|
|Figure 8: Elastic load balancing|
|Figure 9: Variation in number of Netflix instances over a 24 hour period|
Elastic load balancing is a service provided by the underlying infrastructure, the service provider is aware of the pools and the members within each pool. Since it's in the nature of a load balancing pool that each instance has similar traffic pattern to its peers, observing the communication patterns of active pool members would allow a topology aware orchestration controller to select poorly placed VMs when making a removal decision and add new VMs in locations that are close to their peers.
Note: Netflix maintains a base number of reserved instances (reserved instances are the least expensive option, provided you can keep them busy) and uses this "free" capacity for analytics tasks (Hadoop) during off peak periods. Exposing basic locality information to tenants would allow them to better configure topology aware workloads like Hadoop, delivering improved performance and reducing traffic on the shared physical network.Multi-tenancy is just one application of network virtualization. However, the general concept of creating multiple virtual networks implies constraints on communication patterns and a location aware virtual network controller will be able to reduce network loads, improve application performance, and increase scaleability by placing nodes that communicate together topologically close to each other.
There are challenges dealing with large tenants since they may have large groups of machines that need to be provided with high bandwidth communication. Helios: A Hybrid Electrical/Optical Switch Architecture for Modular Data Centers describes some of the limitations of fixed configuration networks and describes how optical networking can be used to flexibly allocate bandwidth where it is needed.
|Figure 10: Demonstrating AWESOME in the Pursuit of the Optical Data Center|
However flexible the network, coordinated management of storage, virtual machine and networking resources is required to fully realize the flexibility and efficiency promised by cloud data centers. The paper, Joint VM Placement and Routing for Data Center Traffic Engineering, shows that jointly optimizing network and server resources can yield significant benefits.
Note: Vint Cerf recently revealed that Google has re-engineering its data center networks to use OpenFlow based software defined networks, possibly bringing networking under the coordinated control of their data center resource management system. In addition, one of the authors of the Helios paper, Amin Vahdat, is a distinguished engineer at Google and has described Google's use of optical networking and OpenFlow in the context of WAN traffic engineering; it would be surprising if Google weren't applying similar techniques within their data centers.Comprehensive measurement is an essential, but often overlooked, component of an adaptive architecture. The controller cannot optimally place workloads if the traffic matrix, link utilizations, and server loads are not known. The widely supported sFlow standard addresses the requirement for pervasive visibility by embedding instrumentation within physical and virtual switches, and in the servers and applications making use of the network to provide the integrated view of performance needed for unified control.
Finally, there are significant challenges to realizing revolutionary improvements in datacenter flexibility and scaleability, many of which aren't technical. Network virtualization, management silos and missed opportunities discusses how inflexible human organizational structures are being reflected in the data center architectures proposed by industry consortia. The article talks about Open Stack, but the recently formed Open Daylight consortium seems to have similar issues, freezing in place existing architectures that offer incremental benefits, rather than providing the flexibility needed for radical innovation and improvement.