Real-time web analytics

The diagram shows a typical scale out web service with a load balancer distributing requests among a pool of web servers. The sFlow HTTP Structures standard is supported by commercial load balancers, including F5 and A10, and open source load balancers and web servers, including HAProxy, NGINX, Apache, and Tomcat.

The simplest way to try out the examples in this article is to download sFlow-RT and install the Host sFlow agent and Apache mod-sflow instrumentation on a Linux web server.

The following sFlow-RT metrics report request rates based on the standard sFlow HTTP counters:

• http_method_option
• http_method_get
• http_method_head
• http_method_post
• http_method_put
• http_method_delete
• http_method_trace
• http_method_connect
• http_method_other
• http_status_1xx
• http_status_2xx
• http_status_3xx
• http_status_4xx
• http_status_5xx
• http_status_other
• http_requests

In addition, mod-sflow exports the following standard thread pool metrics:

• workers_active
• workers_idle
• workers_max
• workers_utilization
• req_delayed
• req_dropped

Cluster performance metrics describes how sFlow-RT's REST API is used to compute summary statistics for a pool of servers. For example, the following query calculates the cluster wide total request rates:

http://localhost:8008/metric/ALL/sum:http_method_get,sum:http_method_post/json

More interesting is that the sFlow telemetry stream also includes randomly sampled HTTP request records with the following attributes:

• protocol
• serveraddress
• serveraddress6
• serverport
• clientaddress
• clientaddress6
• clientport
• proxyprotocol
• proxyserveraddress
• proxyserveraddress6
• proxyserverport
• proxyclientaddress
• proxyclientaddress6
• proxyclientport
• httpmethod
• httpprotocol
• httphost
• httpuseragent
• httpxff
• httpauthuser
• httpmimetype
• httpurl
• httpreferer
• httpstatus
• bytes
• req_bytes
• resp_bytes
• duration
• requests

The sFlow-RT analytics pipeline is programmable. Defining Flows describes how to compute additional metrics based on the sampled requests. For example, the following flow definition creates a new metric called image_bytes that tracks the volume of image data in HTTP responses as a bytes/second value calculated over a 10 second window:

setFlow('image_bytes', {value:'resp_bytes',t:10,filter:'httpmimetype~image/.*'});

The new metric can be queries in exactly the same way as the counter based metrics above, e.g.:

http://localhost:8008/metric/ALL/sum:image_bytes/json

The uri: function is used to extract parts of the httpurl or httpreferer URL fields. The following attributes can be extracted:

• normalized
• scheme
• user
• authority
• host
• port
• path
• file
• extension
• query
• fragment
• isabsolute
• isopaque

For example, the following flow definition creates a metric called game_reqs that tracks the requests/second hitting the URL path with prefix /games:

setFlow('games_reqs', {value:'requests',t:10,filter:'uri:httpurl:path~/games/.*'});

Define flow keys to identify slowest requests, most popular URLs, etc. For example, the following definition tracks the top 5 longest duration requests:

setFlow('slow_reqs', {keys:'httpurl',value:'duration',t:10,n:5});

The following query retrieves the result:

$ curl "http://localhost:8008/activeflows/ALL/slow_reqs/json?maxFlows=5"
[
 {
  "dataSource": "3.80",
  "flowN": 1,
  "value": 117009.24305622398,
  "agent": "10.0.0.150",
  "key": "/login.php"
 },
 {
  "dataSource": "3.80",
  "flowN": 1,
  "value": 7413.476263017302,
  "agent": "10.0.0.150",
  "key": "/games/animals.php"
 },
 {
  "dataSource": "3.80",
  "flowN": 1,
  "value": 4486.286259806839,
  "agent": "10.0.0.150",
  "key": "/games/puzzles.php"
 },
 {
  "dataSource": "3.80",
  "flowN": 1,
  "value": 2326.33482623333,
  "agent": "10.0.0.150",
  "key": "/sales/buy.php"
 },
 {
  "dataSource": "3.80",
  "flowN": 1,
  "value": 276.3486100676183,
  "agent": "10.0.0.150",
  "key": "/index.php"
 }
]

Sampled records are a useful complement to counter based metrics, making it possible to disaggregate counts and identify root causes. For example, suppose a spike in errors is identified through the http_status_4xx or http_status_5xx metrics. The following flow definition breaks out the most frequent failed requests by specific URL and error code:

setFlow('err_reqs', {keys:'httpurl,httpstatus',value:'requests',t:10,n:5,
  filter:'range:httpstatus:400=true'});

Finally, the real-time HTTP analytics don't exist in isolation. The diagram shows how the sFlow-RT real-time analytics engine receives a continuous telemetry stream from sFlow instrumentation build into network, server and application infrastructure and delivers analytics through APIs and can easily be integrated with a wide variety of on-site and cloud, orchestration, DevOps and Software Defined Networking (SDN) tools.

Network and system analytics as a Docker service

The diagram shows how new and existing cloud based or locally hosted orchestration, operations, and security tools can leverage the sFlow-RT analytics service to gain real-time visibility. Network visibility with Docker describes how to install open source sFlow agents to monitor network activity in a Docker environment in order to gain visibility into Docker Microservices.

The sFlow-RT analytics software is now on Docker Hub, making it easy to deploy real-time sFlow analytics as a Docker service:

docker run -p 8008:8008 -p 6343:6343/udp -d sflow/sflow-rt

Configure standard sFlow Agents to stream telemetry to the analyzer and retrieve analytics using the REST API on port 8008.

Increase memory from default 1G to 2G:

docker run -e "RTMEM=2G" -p 8008:8008 -p 6343:6343/udp -d sflow/sflow-rt

Set System Property to enable country lookups when Defining Flows:

docker run -e "RTPROP=-Dgeo.country=resources/config/GeoIP.dat" -p 8008:8008 -p 6343:6343/udp -d sflow/sflow-rt

Run sFlow-RT Application. Drop the -d option while developing an application to see output of logging commands and use control-c to stop the container.

docker run -v /Users/pp/my-app:/sflow-rt/app/my-app -p 8008:8008 -p 6343:6343/udp -d sflow/sflow-rt

A simple Dockerfile can be used to generate a new image that includes the application:

FROM sflow/sflow-rt:latest
COPY /Users/pp/my-app /sflow-rt/app

Similarly, a Dockerfile can be used to generate a new image from published applications. Any required System Properties can also be set in the Dockerfile.

FROM sflow/sflow-rt:latest
ENV RTPROP="-Dgeo.country=resources/config/GeoIP.dat"
RUN /sflow-rt/get-app.sh sflow-rt top-flows

This solution is extremely scaleable, a single sFlow-RT instance can monitor thousands of servers and the network devices connecting them.