Custom events

Measuring Page Load Speed with Navigation Timing describes the standard instrumentation built into web browsers. This article will use navigation timing as an example to demonstrate how custom sFlow events augment standard sFlow instrumentation embedded in network devices, load balancers, hosts and web servers.

The JQuery script can be embedded in a web page to provide timing information:

$(window).load(function(){
 var samplingRate = 10;
 if(samplingRate !== 1 && Math.random() > (1/samplingRate)) return;

 setTimeout(function(){
   if(window.performance) {
     var t = window.performance.timing;
     var msg = {
       sampling_rate : samplingRate,
       t_url         : {type:"string",value:window.location.href},
       t_useragent   : {type:"string",value:navigator.userAgent},
       t_loadtime    : {type:"int32",value:t.loadEventEnd-t.navigationStart},
       t_connecttime : {type:"int32",value:t.responseEnd-t.requestStart} 
     };
     $.ajax({
       url:"/navtiming.php",
       method:"PUT",
       contentType:"application/json",
       data:JSON.stringify(msg) 
     });
    }
  }, 0);
});

The script supports random sampling. In this case a samplingRate of 10 means that, on average, 1-in-10 page hits will generate a measurement record. Measurement records are sent back to the server where the navtiming.php script acts as a gateway, augmenting the measurements and sending them as custom sFlow events.

<?php
$rawInput = file_get_contents("php://input");
$rec = json_decode($rawInput);
$rec->datasource = "navtime";
$rec->t_ip = array("type" => "ip", "value" => $_SERVER['REMOTE_ADDR']);

$msg=array("rtflow"=>$rec);
$sock = fsockopen("udp://localhost",36343,$errno,$errstr);
if(! $sock) { return; }
fwrite($sock, json_encode($msg));
fclose($sock);
?>

In this case the remote IP address associated with the client browser is added to the measurement before it is formatted as a JSON rtflow message and sent to the Host sFlow agent (hsflowd) running on the web server host. The Host sFlow agent encodes the data as an sFlow structure and sends it to the sFlow collector as part of the telemetry stream.

The following sflowtool output verifies that the metrics are being received at the sFlow Analyzer:

startSample ----------------------
sampleType_tag 4300:1003
sampleType RTFLOW
rtflow_datasource_name navtime
rtflow_sampling_rate 1
rtflow_sample_pool 0
rtflow t_url = (string) "http://10.0.0.84/index.html"
rtflow t_useragent = (string) "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36"
rtflow t_loadtime = (int32) 115
rtflow t_connecttime = (int32) 27
rtflow t_ip = (ip) 10.1.1.63
endSample   ----------------------

A more interesting way to consume the data is to use sFlow-RT. For example, the following REST API call programs the sFlow-RT analytics pipeline to create a metric that tracks average t_loadtime by URL:

curl -H "Content-Type:application/json" -X PUT --data '{keys:"t_url",value:"avg:t_loadtime",t:15}' http://localhost:8008/flow/urlloadtime/json

The following query can be used to retrieves the resulting metric value:

curl http://localhost:8008/metric/ALL/urlloadtime/json
[{
 "agent": "10.0.0.84",
 "dataSource": "navtime",
 "lastUpdate": 1807,
 "lastUpdateMax": 1807,
 "lastUpdateMin": 1807,
 "metricN": 1,
 "metricName": "urlloadtime",
 "metricValue": 11.8125,
 "topKeys": [{
  "key": "http://10.0.0.84/index.html",
  "lastUpdate": 1807,
  "value": 11.8125
 }]
}]

RESTflow describes the sFlow-RT REST API used to create flow definitions and access flow based metrics and Defining Flows provides reference material.

Installing mod-sflow provides a rich set of transaction and counter metrics from the Apache web server, including information on worker threads, see Thread pools.

Telemetry from Apache, Host sFlow and the custom events are all combined at the sFlow analyzer. For example, the following query pulls together the load average on the server, with Apache thread pool utilization and URL load times:

curl http://localhost:8008/metric/10.0.0.84/load_one,workers_utilization,urlloadtime/json
[
 {
  "agent": "10.0.0.84",
  "dataSource": "2.1",
  "lastUpdate": 23871,
  "lastUpdateMax": 23871,
  "lastUpdateMin": 23871,
  "metricN": 1,
  "metricName": "load_one",
  "metricValue": 0
 },
 {
  "agent": "10.0.0.84",
  "dataSource": "3.80",
  "lastUpdate": 4123,
  "lastUpdateMax": 4123,
  "lastUpdateMin": 4123,
  "metricN": 1,
  "metricName": "workers_utilization",
  "metricValue": 0.390625
 },
 {
  "agent": "10.0.0.84",
  "dataSource": "navtime",
  "lastUpdate": 8821,
  "lastUpdateMax": 8821,
  "lastUpdateMin": 8821,
  "metricN": 1,
  "metricName": "urlloadtime",
  "metricValue": 91.81072992715491,
  "topKeys": [{
   "key": "http://10.0.0.84/index.html",
   "lastUpdate": 8821,
   "value": 91.81072992715491
  }]
 }
]

The article, Cluster performance metrics, describes the metric API in more detail. Additional sFlow-RT APIs can be used to send data to a variety of DevOps tools, including: Ganglia, Graphite, InfluxDB and Grafana, Logstash, Splunk, cloud analytics services.

Finally, standard sFlow instrumentation is also widely implemented by physical and virtual network devices. Combining data from all these sources provides a comprehensive real-time view of applications and the compute, storage and networking resources that the applications depend on.