Network and system analytics as a Docker microservice

Microservices describes why the industry standard sFlow instrumentation embedded within cloud infrastructure is uniquely able to provide visibility into microservice deployments.

The sFlow-RT analytics engine is well suited to deployment as a Docker microservice since the application is stateless and presents network and system analytics as a RESTful service.

The following steps demonstrate how to create a containerized deployment of sFlow-RT.

First, create a directory for the project and edit the Dockerfile:

mkdir sflow-rt
cd sflow-rt
vi Dockerfile

Add the following contents to Dockerfile:

FROM   centos:centos6
RUN    yum install -y java-1.7.0-openjdk
RUN    rpm -i http://www.inmon.com/products/sFlow-RT/sflow-rt-2.0-1072.noarch.rpm
EXPOSE 8008 6343/udp
CMD    /etc/init.d/sflow-rt start && tail -f /dev/null

Build the project:

docker build -t sflow-rt .

Run the service:

docker run -p 8008:8008 -p 6343:6343/udp -d sflow-rt

Access the API at http://docker_host:8008/ to verify that the service is running.

Update July 22, 2016: sFlow-RT can now be run from Docker Hub, see sflow/sflow-rt for instructions.

Now configure sFlow agents to send data to the docker_host on port 6343:

• Switch configurations
• Install Host sFlow agents to monitor hosts, hypervisors, Docker, etc.

The following articles provide examples of using the sFlow-RT REST API:

• RESTflow
• Cluster performance metrics

The diagram shows how new and existing cloud based or locally hosted orchestration, operations, and security tools can leverage sFlow-RT's analytics service to gain real-time visibility. The solution is extremely scaleable, a single sFlow-RT instance can monitor thousands of servers and the network devices connecting them.

Microservices

Figure 1: Visibility and the software defined data center

In the land of microservices, the network is the king(maker) by Sudip Chakrabarti, Lightspeed Venture Partners, makes the case that visibility into network traffic is the key to monitoring, managing and securing applications that are composed of large numbers of communicating services running in virtual machines or containers.

While I genuinely believe that the network will play an immensely strategic role in the microservices world, inspecting and storing billions of API calls on a daily basis will require significant computing and storage resources. In addition, deep packet inspection could be challenging at line rates; so, sampling, at the expense of full visibility, might be an alternative. Finally, network traffic analysis must be combined with service-level telemetry data (that we already collect today) in order to get a comprehensive and in-depth picture of the distributed application.

Sampling isn't just an alternative, sampling is the key to making large scale microservice visibility a reality. Shrink ray describes how sampling acts as a scaling function, reducing the task of monitoring large scale microservice infrastructure from an intractable measurement and big data problem to a lightweight real-time data center wide visibility solution for monitoring, managing, optimizing and securing the infrastructure.

Figure 2: sFlow Host Structures

Industry standard sFlow is the multi-vendor method for distributed sampling of network traffic. The sFlow standard is model based - models of entities such as interfaces, switches, routers, forwarding state, hosts, virtual machines, messages, etc. are used to define standard measurements that describe their operation. Standardized measurements embedded within the infrastructure ensure consistent reporting that is independent of the specific vendors and application stacks deployed in the data center. Push vs Pull describes how sFlow's push based streaming telemetry addresses the challenge of monitoring large scale cloud environments where services and hosts are constantly being added, removed, started and stopped. In addition, sFlow Host Structures describes how the data model allows telemetry streams from independent sources in network, server and application entities to be combined at the sFlow receiver to provide end to end visibility into the microservice interactions and the compute and networking services on which they depend.

The challenge in delivering network visibility to microservice management tools is not technical - the solution is fully deployable today:

• Applications - e.g. Apache, NGINX, Tomcat, HAproxy, ADC (F5, A10, ..), Memcache, ...
• Virtual Servers - e.g. Xen, Hyper-V, KVM, Docker, JVM, ...
• Virtual Network - e.g. Open vSwitch, Linux Bridge, macvlan, ...
• Servers - e.g. Linux, Windows, FreeBSD, Solaris, AIX
• Network - e.g. Cisco Nexus 9k/3k, Arista, Juniper QFX/EX, Dell, HPE, Brocade, Cumulus, Big Switch, Pica8, Quanta, ... – visit sFlow.org for a complete list

Network, system and application teams working together can enable sFlow instrumentation that is already embedded throughout the infrastructure to achieve comprehensive visibility into microservice interactions.

Incorporating sFlow analytics into the microservices architecture is straightforward. The sFlow-RT analytics engine processes the raw telemetry streams, combines data using the data model, and delivers visibility as a REST based microservice that is easily consumed by new and existing cloud based or locally hosted orchestration, operations, and security tools.