Real-time traffic visualization using Netflix Vizceral

The open source sflow-rt/vizceral project demonstrates how real-time sFlow network telemetry can be presented using Netflix Vizceral. The central dot represents the Internet (all non-local addresses). The surrounding dots represents addresses grouped into sites, data centers, buildings etc. The animated particle flows represent packet flows with colors indicating packet type: TCP/UDP shown in blue, ICMP shown in yellow, and all other traffic in red.

Click on a node to zoom in to show packets flowing up and down the protocol stack. Press the ESC key to unzoom.

The simplest way to run the software is to use the pre-built Docker image:

docker run -p 6343:6343/udp -p 8008:8008 sflow/vizceral

The Docker image also contains demo data based on Netflix's public cloud infrastructure:

docker run -e "RTPROP=-Dviz.demo=yes" -p 8008:8008 sflow/vizceral

In this case, the detailed view shows messages flowing between microservices running in the Amazon public cloud. Similar visibility could be obtained by deploying Host sFlow agents with associated modules for web and application servers and modifying sflow/vizceral to present the application transaction flows. In private data centers, sFlow support in load balancers  (F5, A10) provides visibility into interactions between application tiers. See Microservices for more information on using sFlow to instrument microservice architectures.

Collecting Docker Swarm service metrics describes how meta data about services running on Docker Swarm can be combined with sFlow telemetry to generate service level metrics. A similar approach could be taken to display Docker Swarm service interactions using Vizceral. Using network visibility to measure flows between services greatly simplifies the monitoring task, avoiding the challenge of adding instrumentation to each container.

sFlow leads convergence of multi-vendor application, server, and network performance management

Over the last six months, leading Application Delivery Controller (ADC) vendors F5 and A10 have added support for the sFlow standard to their respective TMOS and ACOS operating systems, making multi-vendor, real-time application layer visibility available in approximately 50% of commercial ADC market.

Figure 1: Best of Velocity 2012, The sFlow Standard

Equally important is the availability of sFlow support in leading open source web servers, load balancers, applications servers, hypervisors and operating systems, including: Apache, NGINX, Tomcat, Java, HAproxy, Hyper-V, Xen, KVM, Linux, Windows, Solaris, FreeBSD and AIX. The combination sFlow in ADCs and the application infrastructure behind them provides comprehensive end to end visibility in multi-tier, scale-out, application architectures.

Figure 1 shows the strategic role that ADCs (load balancers) play in controlling the flow of application requests, regulating admission, filtering, directing loads, and virtualizing services. RESTful control of ADCs combined with real-time visibility provides a powerful capability for flexing resources as demand changes, reducing costs and increasing performance as resources are closely matched to workloads.

What is unusual about diagram is the inclusion of the network. Application architects often give little thought to the network since its complexity is conveniently hidden behind APIs. Unfortunately, it is in the nature of scale-out applications that their performance is tightly coupled to that of the network. In addition, the network is shared between application tiers, allowing performance problems to propagate.

Figure 2: sFlow drivers for growth

Application visibility and control in the ADC space along with near universal support for sFlow among switch vendors combines with Software Defined Networking (SDN) to transform application performance management by orchestrating all the elements of the data center to deliver a comprehensive performance management solution, what VMware calls the Software Defined Data Center (SDDC), Cisco terms the Application Centric Infrastructure (ACI), and Microsoft refers to as the Cloud OS.

Figure 3: Visibility and the software defined data center

Recent breakthroughs in real-time sFlow analysis incorporated in the sFlow-RT analytics engine delivers comprehensive, timely, and actionable metrics through a programmatic interface. Expect to see this technology incorporated in next generation self optimizing orchestration solutions in 2014.

Performance Aware SDN describes the theory behind analytics driven orchestration. The talk describes how fast controller response, programmatic configuration interfaces such as OpenFlow, and consistent instrumentation of all the elements being orchestrated are pre-requisites for feedback control.

The requirement for complete measurement coverage by next generation orchestration systems will create a strong demand for sFlow instrumented infrastructure since sFlow is the only widely supported multi-vendor standard that spans network, server and application resources and delivers the low latency and scaleability required for adaptive control.

F5 BIG-IP LTM and TMOS 11.4.0

The latest TMOS 11.4.0 release for F5's BIG-IP Local Traffic Manager (LTM) includes comprehensive L2-7 support for sFlow, from packet sampling and interface counters to application response times, URLs and status codes (see Monitoring BIG-IP System Traffic with sFlow).

Load balancers are used to virtualize scale out service pools: clients connect to a virtual IP address and service port associated with the load balancer which selects a member of the server pool to handle the request. This architecture provides operational flexibility, allowing servers to be added and removed from the pool as demand changes.

The load balancer is uniquely positioned to provide information on the overall performance of the entire service pool and link the performance seen by clients with the behavior of individual servers in the pool. The advantage of using sFlow to monitor performance is the scalability it offers when request rates are high and conventional logging solutions generate too much data or impose excessive overhead. In addition, monitoring HTTP services using sFlow is part of an integrated monitoring system that spans the data center, providing real-time visibility into application, server and network performance.

Once configured, BIG IP will stream measurements to a central sFlow Analyzer. Download, compile and install the sflowtool on the system your are using to receive sFlow to see the raw data and verify that the measurements are being received.

Running sflowtool will display output of the form:

startDatagram =================
datagramSourceIP 10.0.0.153
datagramSize 564
unixSecondsUTC 1370017719
datagramVersion 5
agentSubId 3
agent 10.0.0.153
packetSequenceNo 16
sysUpTime 1557816000
samplesInPacket 2
startSample ----------------------
sampleType_tag 0:2
sampleType COUNTERSSAMPLE
sampleSequenceNo 1
sourceId 3:2
counterBlock_tag 0:2201
http_method_option_count 0
http_method_get_count 71
http_method_head_count 0
http_method_post_count 0
http_method_put_count 0
http_method_delete_count 0
http_method_trace_count 0
http_methd_connect_count 0
http_method_other_count 2
http_status_1XX_count 0
http_status_2XX_count 26
http_status_3XX_count 24
http_status_4XX_count 23
http_status_5XX_count 0
http_status_other_count 0
endSample   -------------------
startSample -------------------
sampleType_tag 0:1
sampleType FLOWSAMPLE
sampleSequenceNo 1
sourceId 3:2
meanSkipCount 1
samplePool 1
dropEvents 0
inputPort 352
outputPort 1073741823
flowBlock_tag 0:2102
extendedType proxy_socket4
proxy_socket4_ip_protocol 6
proxy_socket4_local_ip 10.0.0.153
proxy_socket4_remote_ip 10.0.0.150
proxy_socket4_local_port 40451
proxy_socket4_remote_port 80
flowBlock_tag 0:2100
extendedType socket4
socket4_ip_protocol 6
socket4_local_ip 10.0.0.153
socket4_remote_ip 10.0.0.70
socket4_local_port 80
socket4_remote_port 40451
flowBlock_tag 0:2206
flowSampleType http
http_method 2
http_protocol 1001
http_uri /index.html
http_host 10.10.10.250
http_referrer http://asdfasdfasdf.asdf
http_useragent curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
http_authuser Aladdin
http_mimetype text/html; charset=UTF-8
http_request_bytes 340
http_bytes 8778
http_duration_uS 1930
http_status 200
endSample   ----------------------
endDatagram ======================

There are two types of sFlow record shown: COUNTERSAMPLE and FLOWSAMPLE data. The counters are useful for trending overall performance using tools like Ganglia and Graphite. Using sflowtool to output combined logfile format makes the data available to most logfile analyzers.

Note: The highlighted IP addresses in the FLOWSAMPLE correspond to addresses in the diagram and illustrate how request records from the proxy link clients to the back end servers.

A native sFlow analyzer like sFlowTrend can combine the counters, flows and host performance metrics to provide an integrated view of performance.

Installing sFlow agents on the backend web servers further extends visibility: implementations are available for Apache, NGINX, Tomcat and node.js. Application logic running on the servers can also be instrumented with sFlow, see Scripting languages. Back end Memcache, Java and virtualization pools can also be instrumented with sFlow. sFlow agents embedded in physical and virtual switches provide end-to-end visibility across the network.

Comprehensive visibility in multi-tiered environments allows the powerful control capabilities of the load balancers to be used to greatest effect: regulating traffic between tiers, protecting overloaded backend systems, defending against denial of service attacks, moving resources from over provisioned pools to under provisioned pools.

HAProxy

The sflow/haproxy project is an implementation of the sFlow HTTP standard for the open source HAProxy high performance TCP/HTTP load balancer. Load balancers are used to virtualize scale out service pools: clients connect to a virtual IP address and service port associated with the load balancer which selects a member of the server pool to handle the request. This architecture provides operational flexibility, allowing servers to be added and removed from the pool as demand changes.

The load balancer is uniquely positioned to provide information on the overall performance of the entire service pool and link the performance seen by clients with the behavior of individual servers in the pool. The advantage of using sFlow to monitor performance is the scalability it offers when request rates are high and conventional logging solutions generate too much data or impose excessive overhead. In addition, monitoring HTTP services using sFlow is part of an integrated monitoring system that spans the data center, providing real-time visibility into application, server and network performance.

The sflow/haproxy software is designed to integrate with the Host sFlow agent to provide a complete picture of proxy performance. Download, install and configure Host sFlow before proceeding to install sflow/haproxy - see Installing Host sFlow on a Linux Server.

Note: the sflow/haproxy agent picks up its configuration from the Host sFlow agent. The sampling.http setting can be used to override the default sampling setting to set a specific sampling rate for HTTP requests.

The following commands download and install the sFlow instrumented version of HAProxy on a Linux server:

git clone https://github.com/sflow/haproxy.git
cd haproxy
make TARGET=linux26 USE_SFLOW=yes
make install

Once installed and configured, HAProxy will stream measurements to a central sFlow Analyzer. Download, compile and install the sflowtool on the system your are using to receive sFlow to see the raw data and verify that the measurements are being received.

Running sflowtool will display output of the form:

$ sflowtool
startDatagram =================================
datagramSourceIP 10.0.0.153
datagramSize 564
unixSecondsUTC 1368058148
datagramVersion 5
agentSubId 80
agent 10.0.0.153
packetSequenceNo 23
sysUpTime 417000
samplesInPacket 2
startSample ----------------------
sampleType_tag 0:2
sampleType COUNTERSSAMPLE
sampleSequenceNo 1
sourceId 3:80
counterBlock_tag 0:2201
http_method_option_count 0
http_method_get_count 71
http_method_head_count 0
http_method_post_count 0
http_method_put_count 0
http_method_delete_count 0
http_method_trace_count 0
http_methd_connect_count 0
http_method_other_count 2
http_status_1XX_count 0
http_status_2XX_count 26
http_status_3XX_count 24
http_status_4XX_count 23
http_status_5XX_count 0
http_status_other_count 0
endSample   ----------------------
startSample ----------------------
sampleType_tag 0:1
sampleType FLOWSAMPLE
sampleSequenceNo 71
sourceId 3:80
meanSkipCount 1
samplePool 71
dropEvents 0
inputPort 0
outputPort 1073741823
flowBlock_tag 0:2102
extendedType proxy_socket4
proxy_socket4_ip_protocol 6
proxy_socket4_local_ip 0.0.0.0
proxy_socket4_remote_ip 10.0.0.150
proxy_socket4_local_port 0
proxy_socket4_remote_port 80
flowBlock_tag 0:2100
extendedType socket4
socket4_ip_protocol 6
socket4_local_ip 0.0.0.0
socket4_remote_ip 10.0.0.70
socket4_local_port 0
socket4_remote_port 57642
flowBlock_tag 0:2206
flowSampleType http
http_method 2
http_protocol 1001
http_uri GET /games/animals.php HTTP/1.1
http_host 10.0.0.153
http_referrer http://10.0.0.153/games/
http_useragent Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/537.31 (KHTML
http_mimetype text/html; charset=UTF-8
http_request_bytes 346
http_bytes 487
http_duration_uS 13000
http_status 200
endSample   ----------------------
endDatagram   =================================

There are two types of sFlow record shown: COUNTERSAMPLE and FLOWSAMPLE data. The counters are useful for trending overall performance using tools like Ganglia and Graphite. Using sflowtool to output combined logfile format makes the data available to most logfile analyzers.

Note: The highlighted IP addresses in the FLOWSAMPLE correspond to addresses in the diagram and illustrate how request records from the proxy link clients to the back end servers. 

A native sFlow analyzer like sFlowTrend can combine the counters, flows and host performance metrics to provide an integrated view of performance.

Installing sFlow agents on the backend web servers further extends visibility: implementations are available for Apache, NGINX, Tomcat and node.js. Application logic running on the servers can also be instrumented with sFlow, see Scripting languages. Back end Memcache, Java and virtualization pools can also be instrumented with sFlow. sFlow agents embedded in physical and virtual switches provide visibility into the network.

Comprehensive end to end visibility in multi-tiered environments allows the powerful control capabilities of the load balancers to be used to greatest effect: regulating traffic between tiers, protecting overloaded backend systems, defending against denial of service attacks, moving resources from over provisioned pools to under provisioned pools.

The sFlow-RT real-time analytics engine makes the full set of sFlow metrics accessible through a RESTful API so that they can be used to drive automation. A future article will explore how sFlow metrics can be used to control HAProxy behavior (by issuing UnixSocketCommands).