Docker Built-In Orchestration Ready For Production: Docker 1.12 Goes GA describes the native swarm mode feature that integrates cluster management, virtual networking, and policy based deployment of services.
This article will demonstrate how real-time streaming telemetry can be used to construct an elastic load balancing solution that dynamically adjusts service capacity to match changing demand.
Getting started with swarm mode describes the steps to configure a swarm cluster. For example, following command issued on any of the Manager nodes deploys a web service on the cluster:
docker service create --replicas 2 -p 80:80 --name apache httpd:2.4And the following command raises the number of containers in the service pool from 2 to 4:
docker service scale apache=4Asynchronous Docker metrics describes how sFlow telemetry provides the real-time visibility required for elastic load balancing. The diagram shows how streaming telemetry allows the sFlow-RT controller to determine the load on the service pool so that it can use the Docker service API to automatically increase or decrease the size of the pool as demand changes. Elastic load balancing of the service pools ensures consistent service levels by adding additional resources if demand increases. In addition, efficiency is improved by releasing resources when demand drops so that they can be used by other services. Finally, global visibility into all resources and services makes it possible to load balance between services, reducing service pools for non-critical services to release resources during peak demand.
The first step is to install and configure Host sFlow agents on each of the nodes in the Docker swarm cluster. The following /etc/hsflowd.conf file configures Host sFlow to monitor Docker and send sFlow telemetry to a designated collector (in this case 10.0.0.162):
sflow {
sampling = 400
polling = 10
collector { ip = 10.0.0.162 }
docker { }
pcap { dev = docker0 }
pcap { dev = docker_gwbridge }
}
Note: The configuration file is identical for all nodes in the cluster making it easy to automate the installation and configuration of sFlow monitoring using Puppet, Chef, Ansible, etc.Verify that the sFlow measurements are arriving at the collector node (10.0.0.162) using sflowtool:
docker -p 6343:6343/udp sflow/sflowtoolThe following elb.js script implements elastic load balancer functionality using the sFlow-RT real-time analytics engine:
var api = "https://10.0.0.134:2376";
var certs = '/tls/';
var service = 'apache';
var replicas_min = 1;
var replicas_max = 10;
var util_min = 0.5;
var util_max = 1;
var bytes_min = 50000;
var bytes_max = 100000;
var enabled = false;
function getInfo(name) {
var info = null;
var url = api+'/services/'+name;
try { info = JSON.parse(http2({url:url, certs:certs}).body); }
catch(e) { logWarning("cannot get " + url + " error=" + e); }
return info;
}
function setReplicas(name,count,info) {
var version = info["Version"]["Index"];
var spec = info["Spec"];
spec["Mode"]["Replicated"]["Replicas"]=count;
var url = api+'/v1.24/services/'+info["ID"]+'/update?version='+version;
try {
http2({
url:url, certs:certs, method:'POST',
headers:{'Content-Type':'application/json'},
body:JSON.stringify(spec)
});
}
catch(e) { logWarning("cannot post to " + url + " error=" + e); }
logInfo(service+" set replicas="+count);
}
var hostpat = service+'\\.*';
setIntervalHandler(function() {
var info = getInfo(service);
if(!info) return;
var replicas = info["Spec"]["Mode"]["Replicated"]["Replicas"];
if(!replicas) {
logWarning("no active members for service=" + service);
return;
}
var res = metric(
'ALL', 'avg:vir_cpu_utilization,avg:vir_bytes_in,avg:vir_bytes_out',
{'vir_host_name':[hostpat],'vir_cpu_state':['running']}
);
var n = res[0].metricN;
// we aren't seeing all the containers (yet)
if(replicas !== n) return;
var util = res[0].metricValue;
var bytes = res[1].metricValue + res[2].metricValue;
if(!enabled) return;
// load balance
if(replicas < replicas_max && (util > util_max || bytes > bytes_max)) {
setReplicas(service,replicas+1,info);
}
else if(replicas > replicas_min && util < util_min && bytes < bytes_min) {
setReplicas(service,replicas-1,info);
}
},2);
setHttpHandler(function(req) {
enabled = req.query && req.query.state && req.query.state[0] === 'enabled';
return enabled ? "enabled" : "disabled";
});
Some notes on the script:
- The setReplicas(name,count,info) function uses the Docker Remote API to implement functionality equivalent to the docker service scale name=count command shown earlier. The REST API is accessible at https://10.0.0.134:2376 in this example.
- The setIntervalHandler() function runs every 2 seconds, retrieving metrics for the service pool and scaling the number of replicas in the service up or down based on thresholds.
- The setHttpHandler() function exposes a simple REST API for enabling / disabling the load balancer functionality. The API can easily be extended to all thresholds to be set, to report statistics, etc.
- Certificates, key.pem, cert.pem, and ca.pem, required to authenticate API requests must be present in the /tls/ directory.
- The thresholds are set to unrealistically low values for the purpose of this demonstration.
- The script can easily be extended to load balance multiple services simultaneously.
- Writing Applications provides additional information on sFlow-RT scripting.
docker run -v `pwd`/tls:/tls -v `pwd`/elb.js:/sflow-rt/elb.js \ -e "RTPROP=-Dscript.file=elb.js" -p 8008:8008 -p 6343:6343/udp -d sflow/sflow-rtThe autoscaling functionality can be enabled:
curl "http://localhost:8008/script/elb.js/json?state=enabled"and disabled:
curl "http://localhost:8008/script/elb.js/json?state=disabled"using the REST API exposed by the script.
ab -rt 60 -n 300000 -c 4 http://10.0.0.134/Next, the test was repeated with the elastic load balancer enabled. The chart clearly shows that the load balancer is keeping the average network load on each container under control.
2016-09-24T00:57:10+0000 INFO: Listening, sFlow port 6343 2016-09-24T00:57:10+0000 INFO: Listening, HTTP port 8008 2016-09-24T00:57:10+0000 INFO: elb.js started 2016-09-24T01:00:17+0000 INFO: apache set replicas=2 2016-09-24T01:00:23+0000 INFO: apache set replicas=3 2016-09-24T01:00:27+0000 INFO: apache set replicas=4 2016-09-24T01:00:33+0000 INFO: apache set replicas=5 2016-09-24T01:00:41+0000 INFO: apache set replicas=6 2016-09-24T01:00:47+0000 INFO: apache set replicas=7 2016-09-24T01:00:59+0000 INFO: apache set replicas=8 2016-09-24T01:01:29+0000 INFO: apache set replicas=7 2016-09-24T01:01:33+0000 INFO: apache set replicas=6 2016-09-24T01:01:35+0000 INFO: apache set replicas=5 2016-09-24T01:01:39+0000 INFO: apache set replicas=4 2016-09-24T01:01:43+0000 INFO: apache set replicas=3 2016-09-24T01:01:45+0000 INFO: apache set replicas=2 2016-09-24T01:01:47+0000 INFO: apache set replicas=1The sFlow-RT log shows that containers are added to the apache service to handle the increased load and removed once demand decreases.
This example relied on a small subset of the information available from the sFlow telemetry stream. In addition to container resource utilization, the Host sFlow agent exports an extensive set of metrics from the nodes in the Docker swarm cluster. If the nodes are virtual machines running in a public or private cloud, the metrics can be used to perform elastic load balancing of the virtual machine pool making up the cluster, increasing the cluster size if demand increases and reducing cluster size when demand decreases. In addition, poorly performing instances can be detected and removed from the cluster (see Stop thief! for an example).
docker run -p 6343:6343/udp -p 8008:8008 -d sflow/top-flowslaunches the top-flows application to show an up to the second view of active flows in the network.
Comprehensive real-time analytics is critical to effectively managing agile container-bases infrastructure. Open source Host sFlow agents provide a lightweight method of instrumenting the infrastructure that unifies network and system monitoring to deliver a full set of standard metrics to performance management applications.
No comments:
Post a Comment