The incident report describes an external distributed denial of service (DDoS) attack that was launched some time before 9:30. The effects of the attack started to be detected by the measurement system at 9:30 and it took until 10:00 fully identify the attack and start planning a response. The plan to null route the traffic was implemented at 10:09 and the incident was fully resolved at 10:29.
SDN and delay discusses the components of delay in a feedback control loop and includes the above timeline. Applying the timeline to the DDoS incident identifies the following components of response delay:
- Measurement delay, 30 minutes
- Planning delay, 9 minutes
- Configuration delay, not broken out, included in planning delay
- Response delay, < 20 minutes
- Loop delay, 59 minutes
While automation is an important part of the OpenStack (and other cloud orchestration systems), current architectures don't include the feedback mechanisms and coordinated controls needed for effective multi-tenant performance isolation, Network virtualization, management silos, and missed opportunities.
The key to building responsive performance optimizing controllers is a pervasive, scaleable, real-time monitoring system. The sFlow instrumentation embedded within the physical and virtual switches (in this case Open vSwitch), load balancers and hypervisors enables real-time monitoring of the entire cloud data center.
The next step is to integrate the real-time analytics into the orchestration system. The article performance aware software defined networking describes the basic elements of a performance optimizing controller.
DDoS describes a fully automated system for DDoS mitigation with a loop delay of around 10 seconds, i.e. it is able to detect, characterize, null route and eliminate an attack within 10 seconds (over 300 times faster than the manual process). The controller is fast enough to prevent the attack from fully developing, cutting the peak traffic by a factor of 4.
Controlling large flows with OpenFlow describes an experimental controller that can mitigate a denial of service attack in 2 seconds.
Denial of service mitigation is just one example of multi-tenant performance isolation. There are many types of application that tenants run within their cloud deployments that stress the infrastructure. The articles Multi-tenant traffic in virtualized network environments, Pragmatic software defined networking and Resource allocation look at some of the architectural issues involved in managing cloud performance.