Saturday, February 23, 2013

SDN and large flows

Figure 1 Long vs Short flows (from The Nature of Datacenter Traffic: Measurements & Analysis)
Load balancing LAG/ECMP groups proposed using the combination of sFlow and OpenFlow to address the challenge that existing link aggregation (LAG) and equal cost multi-path routing (ECMP) architectures. An important question to ask is, how fast does a software defined networking load balancer have to be in order to significantly improve network performance? If load balancing decisions need to be made in the order of nanoseconds or microseconds, then software defined networking isn't a feasible approach, however, if decisions can be made in the order of seconds then software defined networking is an attractive approach because of the increased flexibility offered by an external software controller.

The chart in Figure 1 was taken from the paper, The Nature of Datacenter Traffic: Measurement & Analysis, which provides a comprehensive analysis of network traffic patterns in a large scale data center environment running a realistic workload. The chart shows that most traffic flows are short lived, over 50% of flows last less than 1 second. However, very little of the bandwidth is consumed by these short flows. Most bandwidth is consumed by the small number of long lived flows, flows with a duration between 10 seconds and 1000 seconds.
Figure 2: ECMP vs SDN/sFlow (from DevoFlow: Scaling Flow Management for High-Performance Networks)
The chart in Figure 2 is taken from the paper, DevoFlow: Scaling Flow Management for High-Performance Networks. This paper presents simulation results to examine the effect of using sFlow and OpenFlow to load balance long lived flows. The chart uses ECMP load balancing as a baseline (the light blue bars) and compares alternative approaches to load balancing traffic on a fat-tree/CLOS network topology. The results of load balancing based on sFlow traffic measurements (packets sampled at 1-in-1000 on 1G links) is shown by the brick red bars. The simulation demonstrates that load balancing of large flows can significantly improve throughput over ECMP.

Note: A CLOS network is a best case for ECMP since it offers the largest number of alternative equal cost paths. One would expect dynamic routing of large flows to deliver even greater improvements on non-CLOS networks.

DevoFlow refers to the paper, Hedera: Dynamic Flow Scheduling for Data Center Networks, for a definition of large flows - which defines a "large flow" as a flow that consumes 10% of a link's total bandwidth. For example, when monitoring 1Gigabit links, a large flow would be defined as any flow exceeding 100 Mbits/second. For a 10 Gigabit link, a large flow would be defined as any flow exceeding 1 Gigabits/second. The choice of the 1-in-1000 sampling rate in the DevoFlow article was selected to allow large flows on their 1Gigabit links to be detected within 1 second.

The sampling based scheme easily scales to higher speeds, since a sampling rate of 1-in-10,000 would detect large flows within a second on 10Gigabit links, a sampling rate of 1-in-100,000 would detect large flows within a second on 100Gigabit links etc. In each case the monitoring load on a central controller would be the same, i.e. the monitoring overhead to drive load balancing is small and doesn't go up with network speed. For more information, see Scalability and accuracy of packet sampling.

The detection speed of 1 second makes sense given the bi-modal distribution shown in Figure 1. Ignoring flows that last less than a second means that the controller can ignore the short flows (which consume very little bandwidth and are handled well by existing hardware load balancing techniques). Reacting to these short flows would consume controller resources and be counter productive since the flows would like end before any controls would take effect. If the controller can react within a few seconds to large flows it will have effective control over 90% of the bandwidth consumed on the network.

Note: The paper, Estimating the Volume of Elephant Flows under Packet Sampling, describes how sampling reduces the resources needed to detect large flows.

A skeptical reader might have noticed that the papers referenced in this article so far all relate to a map/reduce (e.g. Hadoop) workload and be concerned about the general applicability software defined load balancing.
Figure 3: Peak Period Aggregate Traffic Composition (North America, Mobile Access) 
For another proof point, consider Figure 3, from Sandvine's Global Internet Phenomena Report 2H 2012. The chart shows that 72% of peak period downstream bandwidth in the North America consistes of large flows (comprising Real-Time Entertainment such as Netflix, Hulu, YouTube etc. and Filesharing). NetFlix alone accounts for 33% of all North American primetime downstream bandwidth.
Figure 4: Peak Period Aggregate Traffic Composition (North America, Mobile Access)
Figure 4 shows that large flows also dominate bandwidth consumption by mobile devices. However, on mobile platforms YouTube streaming dominates, accounting for nearly 31% of peak period mobile bandwidth.

Netflix hosts its service within Amazon EC2, therefore it's not unreasonable to expect that the network bandwidth within the Amazon cloud is strongly driven by large video flows (along with other related activities that also generate large flows: transcoding video files, off peak Amazon Elastic MapReduce, etc. - see Dynamically Scaling Netflix in the Cloud).

Other research papers have examined the impact of large flows on total bandwidth consumption:
From all the evidence, it is clear that load balancing of large flows offers a significant opportunity to improve network performance in data centers, wide area and access networks, both wired and wireless.
Figure 5 Performance aware software defined networking
Figure 5 shows the elements of the software defined load balancing system described in Load balancing LAG/ECMP groups. This is a flexible architecture that could be used to load balance large flows in many different environments, including data center, WAN, wireless, and carrier networks. An important benefit of software defined networking is that control decisions are no longer limited to algorithms offered by network equipment providers - an SDN load balancer can take into account business concerns relating to value, cost, security, licensing etc., selecting traffic paths to maximise business benefit.

All the components needed to take SDN load balancing into the mainstream are now in place. The sFlow standard is a mature, robust, scalable, measurement technology that is almost universally supported by switch vendors and vendor support for OpenFlow is rapidly increasing - so finding network equipment that supports both sFlow and OpenFlow is not difficult. OpenFlow controllers are readily available and InMon's sFlow-RT real-time analytics engine detects large flows and provides the APIs needed to drive load balancing SDN applications. Load balancing is poised to be a killer application that will drive SDN into the mainstream.

No comments:

Post a Comment