Real-time web analytics

The diagram shows a typical scale out web service with a load balancer distributing requests among a pool of web servers. The sFlow HTTP Structures standard is supported by commercial load balancers, including F5 and A10, and open source load balancers and web servers, including HAProxy, NGINX, Apache, and Tomcat.

The simplest way to try out the examples in this article is to download sFlow-RT and install the Host sFlow agent and Apache mod-sflow instrumentation on a Linux web server.

The following sFlow-RT metrics report request rates based on the standard sFlow HTTP counters:

• http_method_option
• http_method_get
• http_method_head
• http_method_post
• http_method_put
• http_method_delete
• http_method_trace
• http_method_connect
• http_method_other
• http_status_1xx
• http_status_2xx
• http_status_3xx
• http_status_4xx
• http_status_5xx
• http_status_other
• http_requests

In addition, mod-sflow exports the following standard thread pool metrics:

• workers_active
• workers_idle
• workers_max
• workers_utilization
• req_delayed
• req_dropped

Cluster performance metrics describes how sFlow-RT's REST API is used to compute summary statistics for a pool of servers. For example, the following query calculates the cluster wide total request rates:

http://localhost:8008/metric/ALL/sum:http_method_get,sum:http_method_post/json

More interesting is that the sFlow telemetry stream also includes randomly sampled HTTP request records with the following attributes:

• protocol
• serveraddress
• serveraddress6
• serverport
• clientaddress
• clientaddress6
• clientport
• proxyprotocol
• proxyserveraddress
• proxyserveraddress6
• proxyserverport
• proxyclientaddress
• proxyclientaddress6
• proxyclientport
• httpmethod
• httpprotocol
• httphost
• httpuseragent
• httpxff
• httpauthuser
• httpmimetype
• httpurl
• httpreferer
• httpstatus
• bytes
• req_bytes
• resp_bytes
• duration
• requests

The sFlow-RT analytics pipeline is programmable. Defining Flows describes how to compute additional metrics based on the sampled requests. For example, the following flow definition creates a new metric called image_bytes that tracks the volume of image data in HTTP responses as a bytes/second value calculated over a 10 second window:

setFlow('image_bytes', {value:'resp_bytes',t:10,filter:'httpmimetype~image/.*'});

The new metric can be queries in exactly the same way as the counter based metrics above, e.g.:

http://localhost:8008/metric/ALL/sum:image_bytes/json

The uri: function is used to extract parts of the httpurl or httpreferer URL fields. The following attributes can be extracted:

• normalized
• scheme
• user
• authority
• host
• port
• path
• file
• extension
• query
• fragment
• isabsolute
• isopaque

For example, the following flow definition creates a metric called game_reqs that tracks the requests/second hitting the URL path with prefix /games:

setFlow('games_reqs', {value:'requests',t:10,filter:'uri:httpurl:path~/games/.*'});

Define flow keys to identify slowest requests, most popular URLs, etc. For example, the following definition tracks the top 5 longest duration requests:

setFlow('slow_reqs', {keys:'httpurl',value:'duration',t:10,n:5});

The following query retrieves the result:

$ curl "http://localhost:8008/activeflows/ALL/slow_reqs/json?maxFlows=5"
[
 {
  "dataSource": "3.80",
  "flowN": 1,
  "value": 117009.24305622398,
  "agent": "10.0.0.150",
  "key": "/login.php"
 },
 {
  "dataSource": "3.80",
  "flowN": 1,
  "value": 7413.476263017302,
  "agent": "10.0.0.150",
  "key": "/games/animals.php"
 },
 {
  "dataSource": "3.80",
  "flowN": 1,
  "value": 4486.286259806839,
  "agent": "10.0.0.150",
  "key": "/games/puzzles.php"
 },
 {
  "dataSource": "3.80",
  "flowN": 1,
  "value": 2326.33482623333,
  "agent": "10.0.0.150",
  "key": "/sales/buy.php"
 },
 {
  "dataSource": "3.80",
  "flowN": 1,
  "value": 276.3486100676183,
  "agent": "10.0.0.150",
  "key": "/index.php"
 }
]

Sampled records are a useful complement to counter based metrics, making it possible to disaggregate counts and identify root causes. For example, suppose a spike in errors is identified through the http_status_4xx or http_status_5xx metrics. The following flow definition breaks out the most frequent failed requests by specific URL and error code:

setFlow('err_reqs', {keys:'httpurl,httpstatus',value:'requests',t:10,n:5,
  filter:'range:httpstatus:400=true'});

Finally, the real-time HTTP analytics don't exist in isolation. The diagram shows how the sFlow-RT real-time analytics engine receives a continuous telemetry stream from sFlow instrumentation build into network, server and application infrastructure and delivers analytics through APIs and can easily be integrated with a wide variety of on-site and cloud, orchestration, DevOps and Software Defined Networking (SDN) tools.