Streaming telemetry

The OpenConfig project has been getting a lot of attention lately.  A number of large network operators, lead by Google, are developing "a consistent set of vendor-neutral data models (written in YANG) based on actual operational needs from use cases and requirements from multiple network operators."

The OpenConfig project extends beyond configuration, "Streaming telemetry is a new paradigm for network monitoring in which data is streamed from devices continuously with efficient, incremental updates. Operators can subscribe to the specific data items they need, using OpenConfig data models as the common interface."

Anees Shaikh's Network Field Day talk provides an overview of OpenConfig and includes an example that demonstrates how configuration and state are combined in a single YANG data model. In the example, read/write config attributes used to configure a network interface (name, description, MTU, operational state) are combined with the state attributes needed to verify the configuration (MTU, name, description, oper-status, last-change) and collect metrics (in-octets, in-ucast-pkts, in-broadcast-pkts, ...).

Anees positions OpenConfig streaming telemetry mechanism as an attractive alternative to polling for metrics using Simple Network Management Protocol (SNMP) - see Push vs Pull for a detailed comparison between pushing (streaming) and pulling (polling) metrics.

Streaming telemetry is not unique to OpenConfig. Industry standard sFlow is a streaming telemetry alternative to SNMP that has seen rapid vendor adoption over the last decade. Drivers for growth discusses how the rise of merchant silicon and virtualization have accelerated adoption of sFlow, particularly in data centers.

	sFlow	OpenConfig Telemetry	SNMP
Organization	sFlow.org	OpenConfig.net	IETF
Users	General Purpose	Large Service Providers	General Purpose
Scope	Data Plane, Control Plane	Management Plane, Control Plane	Control Plane
Vendor Support	40+ (see sFlow.org)	1 (Cisco IOS XR )	Near universal
Models	structure definitions	YANG models	Management Information Base
Encoding	XDR (RFC 4506)	protobufs, JSON, NetConf	ASN.1
Transport	UDP	UDP, HTTP	UDP
Mode	Push	Push	Pull

The table compares sFlow and OpenConfig Telemetry. There are a number of similarities: sFlow and OpenConfig are both driven by participation based organizations that publish standards to ensure multi-vendor interoperability, and both push standard sets of metrics using standard encodings over widely supported transport protocols.

However, important differences result from OpenConfig's exclusive focus on large service provider configuration and monitoring requirements. Telemetry is tied to the hierarchical YANG configuration models, making it easy to correlate operational and configured state, but limiting the scope of monitoring to the management and control planes of devices that are configured using OpenConfig.

In contrast, sFlow is a management and control plane agnostic monitoring technology (i.e. a device may be configured using CLI, NetConf, JSON RPC, OpenConfig, etc. to use any control plane OpenFlow, BGP, OSPF, TRILL, spanning tree, etc.). In addition, sFlow is primarily concerned with the data and control planes, i.e. capturing information about packets and forwarding actions.

The article, Management, control and data planes in network devices and systems, by Ivan Pepelnjak, provides background on data, control, and management plane terminology.

Gathering data plane telemetry requires hardware support and merchant silicon vendors (Broadcom, Cavium/XPliant, Intel/Fulcrum, Marvell etc.) include sFlow instrumentation in their switching/routing ASICs. Embedded hardware support allows sFlow to efficiently stream standardized data plane telemetry from all devices in a large multi-vendor network.

To conclude, sFlow and OpenConfig shouldn't be viewed as competing technologies. Instead, the complementary capabilities of sFlow and OpenConfig and shared architectural model make it easy to combine sFlow and OpenConfig into an integrated management solution that unifies monitoring and control of management, control, and data planes.

UDP packet replication using Open vSwitch

UDP protocols such as sFlow, syslog, NetFlow, IPFIX and SNMP traps, have many advantages for large scale network and system monitoring, see Push vs Pull.  In a typical deployment each managed element is configured to send UDP packets to a designated collector (specified by an IP address and port). For example, in a simple sFlow monitoring system all the switches might be configured to send sFlow data to UDP port 6343 on the host running the sFlow analysis application. Complex deployments may require multiple analysis applications, for example: a first application providing analytics for software defined networking, a second focused on host performance, a third addressing packet capture and security, and the fourth looking at application performance. In addition, a second copy of each application may be required for redundancy. The challenge is getting copies of the data to all the application instances in an efficient manner.

There are a number of approaches to replicating UDP data, each with limitations:

1. IP Multicast - if the data is sent to an IP multicast address then each application could subscribe to the multicast channel receive a copy of the data. This sounds great in theory, but in practice configuring and maintaining IP multicast connectivity can be a challenge. In addition, all the agents and collectors would need to support IP multicast. In addition, IP multicast also doesn't address the situation where you have multiple applications running on single host and so each application has to be receive the UDP data on a different port.
2. Replicate at source - each agent could be configured to send a copy of the data to each application. Replicating at source is a configuration challenge (all agents need to be reconfigured if you add an additional application). This approach is also wasteful of bandwidth - multiple copies of the same data are send across the network.
3. Replicate at destination - a UDP replicator, or "samplicator" application receives the stream of UDP messages, copies them and resends them to each of the applications. This functionality may be deployed as a stand alone application, or be an integrated function within an analysis application. The replicator application is a single point of failure - if it is shut down none of the applications receive data. The replicator adds delay to the measurements and at high data rates can significantly increase UDP loss rate as the datagrams are received, sent, and received again. 

This article will examine a fourth option, using software defined networking (SDN) techniques to replicate and distribute data within the network. The Open vSwitch is implemented in the Linux kernel and includes OpenFlow and network virtualization features that will be used to build the replication network.

First, you will need a server (or virtual machine) running a recent version of Linux. Next download and install Open vSwitch.

Next, configure the Open vSwitch to handle networking for the server:

ovs-vsctl add-br br0
ovs-vsctl add-port br0 eth0
ifconfig eth0 0
ifconfig br0 10.0.0.1/24

Now configure the UDP agents to send their data to 10.0.0.1. You should be able to run a collector application for each service port (e.g. sFlow 6343, syslog 514, etc.).

The first case to consider is replicating the datagrams to a second port on the server (sending packets to App 1 and App 2 in the diagram). First, use the ovs-vsctl command to list the OpenFlow port numbers on the virtual switch:

% ovs-vsctl --format json --columns name,ofport list Interface
{"data":[["eth0",1],["br1",65534]],"headings":["name","ofport"]}

We are interested in replicating packets received on eth0 and output shows that the corresponding OpenFlow port is 1.

The Open vSwitch provides a command line utility ovs-ofctl that uses the OpenFlow protocol to configure forwarding rules in the vSwitch. The following OpenFlow rule will replicate sFlow datagrams:

in_port=1 dl_type=0x0800 nw_proto=17 tp_dst=6343 actions=LOCAL,mod_tp_dst:7343,normal

The match part of the rule looks for packets received on port 1 (in_port=1), where the Ethernet type is IPv4 (dl_type=0x0800), the IP protocol is UDP (nw_protocol=17), and the destination UDP port is 6343 (tp_dst=6343). The actions section of the rule is the key to building the replication function. The LOCAL action delivers the original packet as intended. The destination port is then changed to 7343 (mod_tp_dst:7343) and the modified packet is sent through the normal processing path to be delivered to the application.

Save this rule to a file, say replicate.txt, and then use ovs-ofctl to apply the rule to br0:

ovs-ofctl add-flows br0 replicate.txt

At this point a second sFlow analysis application listening for sFlow datagrams on port 7343 should start receiving data - sflowtool is a convenient way to verify that the packets are being received:

sflowtool -p 7343

The second case to consider is replicating the datagrams to a remote host (sending packets to App 3 in the diagram).

in_port=1 dl_type=0x0800 nw_proto=17 tp_dst=6343 actions=LOCAL,mod_tp_dst:7343,normal,mod_nw_src:10.0.0.1,mod_nw_dst:10.0.0.2,normal

The extended rule includes additional actions that modify the source address of the packets (mod_nw_src:10.0.0.1) and the destination IP address (mod_nw_dst:10.0.0.2) and sends the packet through the normal processing path. Since we are relying on the routing functionality in the Linux stack to deliver the packet, make sure that routing in enabled - see How to Enable IP Forwarding in Linux.

Unicast reverse path filtering (uRPF) is mechanism that routers use to drop spoofed packets (i.e. packets where the source address doesn't belong to the subnet on the access port the packet was received on). uRPF should be enabled wherever practical because spoofing is used in a variety of security and denial of service attacks, e.g. DNS amplification attacks. By modifying the IP source address to be the address of the forwarding host (10.0.0.1) rather than the original source IP address the OpenFlow rule ensures that the packet will pass through uRPF filters, both on the host and on the access router. Rewriting the sFlow source address does not cause any problems because the sFlow protocol identifies the original source of the data within its payload and doesn't rely on the IP source address. However, other UDP protocols (for example, NetFlow/IPFIX) rely on the IP source address to identify the source of the data. In this case, removing the mod_nw_src action will leave the IP source address unchanged, but the packet may well be dropped by uRPF filters. Newer Linux distributions implement strict uRPF by default, however it can be disabled if necessary, see Reverse Path Filtering.

This article has only scratched the surface of capabilities of the Open vSwitch. In situations where passing the raw packets across the network isn't feasible the Open vSwitch can be configured to send the packets over a tunnel (sending packets to App 4 in the diagram). Tunnels, in conjunction with OpenFlow, can be used to create a virtual UDP distribution overlay network with its own addressing scheme and topology - Open vSwitch is used by a number of network virtualization vendors (e.g. VMware NSX). In addition, more complex filters can also be implemented, forwarding datagrams based on source subnet to different collectors etc.

The replication functions don't need to be performed in software in the virtual switch. OpenFlow rules can be pushed to OpenFlow capable hardware switches which can perform the replication, or source based forwarding functions at wire speed. A full blown controller based solution isn't necessarily required, the ovs-ofctl command can be used to push OpenFlow rules to physical switches.

More generally, building flexible UDP datagram distribution and replication networks is an interesting use case for software defined networking. The power of software defined networking is that you can adapt the network behavior to suit the needs of the application - in this case overcoming the limitations of existing UDP distribution solutions by modifying the behavior of the network.

Finding elephants

The Blind Men and the Elephant
John Godfrey Saxe (1816-1887)

It was six men of Indostan
To learning much inclined,
Who went to see the Elephant
(Though all of them were blind),
That each by observation
Might satisfy his mind.

The First approached the Elephant,
And happening to fall
Against his broad and sturdy side,
At once began to bawl:
"God bless me! but the Elephant
Is very like a wall!"

The Second, feeling of the tusk,
Cried, "Ho, what have we here,
So very round and smooth and sharp?
To me 'tis mighty clear
This wonder of an Elephant
Is very like a spear!"

The Third approached the animal,
And happening to take
The squirming trunk within his hands,
Thus boldly up and spake:
"I see," quoth he, "the Elephant
Is very like a snake!"

The Fourth reached out an eager hand,
And felt about the knee
"What most this wondrous beast is like
Is mighty plain," quoth he:
"'Tis clear enough the Elephant
Is very like a tree!"

The Fifth, who chanced to touch the ear,
Said: "E'en the blindest man
Can tell what this resembles most;
Deny the fact who can,
This marvel of an Elephant
Is very like a fan!"

The Sixth no sooner had begun
About the beast to grope,
Than seizing on the swinging tail
That fell within his scope,
"I see," quoth he, "the Elephant
Is very like a rope!"

And so these men of Indostan
Disputed loud and long,
Each in his own opinion
Exceeding stiff and strong,
Though each was partly in the right,
And all were in the wrong!

MORAL.

So oft in theologic wars,
The disputants, I ween,
Rail on in utter ignorance
Of what each other mean,
And prate about an Elephant
Not one of them has seen!

This poem is just one version of the popular Bind Men and an Elephant story. There are many conclusions that you can draw from the story, but this version captures the idea that you can't properly understand something unless you have seen it in its entirety.

Similar arguments occur among data center operations teams advocating for different performance monitoring technologies such as NetFlow, IPFIX, SNMP, WMI, JMX, libvirt etc. Many of these arguments arise because teams are siloed, with each group arguing from their narrow perspective and dismissing the concerns of other teams. While narrowly focussed tools have their place, they miss the big picture.

Cloud-based architectures tightly couple network, storage and compute resources into a large scale, flexible, platform for delivering application services. The article, System boundary, describes the need for a holistic approach to organizing and managing data center resources. A comprehensive, cloud-oriented, approach to monitoring is essential for troubleshoot performance problems, automating operations and fully exploiting the potential of cloud architectures to increase efficiency by adapting to changing demand.

The sFlow standard addresses the challenge of cloud monitoring by embedding instrumentation throughout the cloud infrastructure in order to provide a comprehensive, real-time, view of the performance of all the individual network, server and application resources as well as the cloud as a whole. The sFlow architecture is designed to see elephants, not just individual body parts. It's only by gaining this comprehensive view of performance that large scale cloud data center environments can be properly understood and managed.

Push vs Pull

Push-me-pull-you from Doctor Doolittle

There are two major performance monitoring architectures:

• Push, metrics are periodically sent by each monitored system to a central collector. Examples of push architectures include: sFlow, Ganglia, Graphite, collectd and StatsD.
• Pull, a central collector periodically requests metrics from each monitored system. Examples of pull architectures include: SNMP, JMX, WMI and libvirt.

The remainder of this article will explore some of the strengths and weaknesses of push and pull architectures:

	Push	Pull
Discovery	Agent automatically sends metrics as soon as it starts up, ensuring that it is immediately detected and continuously monitored. Speed of discovery is independent of number of agents.	Discovery requires collector to periodically sweep address space to find new agents. Speed of discovery depends on discovery sweep interval and size of address space.
Scalability	Polling task fully distributed among agents, resulting in linear scalability. Lightweight central collector listens for updates and stores measurements. Minimal work for agents to periodically send fixed set of measurements. Agents are stateless, exporting data as soon as it is generated.	Workload on central poller increases with the number of devices polled. Additional work on poller to generate requests and maintaining session state in order to match requests and responses. Additional work for agents to parse and process requests. Agents often required to maintain state so that metrics can be retrieved at a later time by the poller.
Security	Push agents are inherently secure against remote attacks since they do not listen for network connections.	Polling protocol can potentially open up system to remote access and denial of service attacks.
Operational Complexity	Minimal configuration required for agents: polling interval and address of collector. Firewalls need to be configured for unidirectional communication of measurements from agents to collector.	Poller needs to be configured with list of devices to poll, security credentials to access the devices and the set of measurements to retrieve. Firewalls need to be configured to allow bi-directional communication between poller and agents.
Latency	The low overhead and distributed nature of the push model permits measurement to be sent more frequently, allowing the management system to quickly react to changes. In addition, many push protocols, like sFlow, are implemented on top of UDP, providing non-blocking, low-latency transport of measurements.	The lack of scalability in polling typically means that measurements are retrieved less often, resulting in a delayed view of performance that makes the management system less responsive to changes. The two way communication involved in polling increases latency as connections are established and authenticated before measurements can be retrieved.
Flexibility	Relatively inflexible: pre-determined, fixed set of measurements are periodically exported.	Flexible: poller can ask for any metric at any time.

The push model is particularly attractive for large scale cloud environments where services and hosts are constantly being added, removed, started and stopped. Maintaining lists of devices to poll for statistics in these environments is challenging and the discovery, scalability, security, low-latency and the simplicity of the push model make it a clear winner.

The sFlow standard is particularly well suited to large scale monitoring of cloud infrastructures, delivering the comprehensive visibility into the performance of network, compute and application resources needed for effective management and control.

In practice, a hybrid approach provides the best overall solution. The core set of standard metrics needed to manage performance and detect problems is pushed using sFlow and a pull protocol is used to retrieve diagnostic information from specific devices when a problem is detected.

Large Hadron Collider

LHCb (Photo Credit: CERN)

The Large Hadron Collider at CERN has been in the news as it comes online.

An interesting paper (Management of the LHCb Network Based on SCADA System) describes the data collection network associated with the LHCb experiment. High-speed switched Ethernet networks are used to collect measurements from the experiment and to control its operation. The paper states that, "Sophisticated monitoring of both networks at all levels is essential for the successful operation of the experiment."

The network monitoring system uses sFlow to measure network utilization on the core switches. "Because there are so many ports in the core switches, the SNMP query of interface counters takes a long time and occupies a lot CPU and memory resource."

The distributed counter polling mechanism in sFlow provides a highly scalable alternative to SNMP polling, delivering reliable monitoring in the most demanding environments. Network visibility in the data center is equally important and sFlow provides the scalability and performance needed to maintain effective control of high-speed data center networks.

Sampling rates

A previous posting discussed the scalability and accuracy of packet sampling and the advantages of packet sampling for network-wide visibility.

Selecting a suitable packet sampling rate is an important part of configuring sFlow on a switch. The table gives suggested values that should work well for general traffic monitoring in most networks. However, if traffic levels are unusually high the sampling rate may be decreased (e.g. use 1 in 5000 instead of 1 in 2000 for 10Gb/s links).

Configure sFlow monitoring on all interfaces on the switch for full visibility. Packet sampling is implemented in hardware so all the interfaces can be monitored with very little overhead.

Finally, select a suitable counter polling interval so that link utilizations can be accurately tracked. Generally the polling interval should be set to export counters at least twice as often as the data will be reported (see Nyquist-Shannon sampling theory for an explanation). For example, to trend utilization with minute granularity, select a polling interval of between 20 and 30 seconds. Don't be concerned about setting relatively short polling intervals; counter polling with sFlow is very efficient, allowing more frequent polling with less overhead than is possible with SNMP.

sFlow MIB

Configuring switches through the switch command line interface (CLI) can be complex and time consuming, especially when monitoring needs to be configured on every switch in order to achieve network-wide visibility.

The sFlow MIB provides a way for an sFlow analysis application to use SNMP to automatically configure sFlow settings on the switches that it wants to monitor. Since the sFlow MIB is an optional part of the sFlow standard, not all sFlow capable switches can be configured using SNMP. However, HP ProCurve and Alcatel-Lucent switches support the sFlow MIB making it easy to quickly try out sFlow monitoring using the free sFlowTrend application. The screen capture above shows the sFlowTrend setting needed to enable SNMP configuration of sFlow.

Many traffic analyzers that claim sFlow support do not support the sFlow MIB. If you have switches that support the sFlow MIB, then selecting an analyzer that supports the sFlow MIB will ensure a successful deployment.

Future posts on this blog will describe the configuration commands needed to enable sFlow on additional vendor's switches.

Link utilization

One of the basic tasks in monitoring network traffic is to accurately track the utilization of links in your network. A managed switch will provide a standard set of counters for each interface that can be retrieved retrieved periodically using SNMP and used to trend link utilization, packet rates, errors and discards.

sFlow provides an alternative to SNMP counter polling. The sFlow agent in the switch will periodically send, or "push" its own counters to the central collector. Pushing counters is much more efficient than than retrieving them using SNMP, requiring 10-20 times fewer network packets to retrieve the same information. The sFlow protocol uses XDR to encode the counters. XDR is much simpler to encode and decode than the ASN1 encoding that the SNMP protocol uses, so the CPU load on the switches and the collectors is also significantly reduced. Finally, distributing the counter polling task among the switches further reduces the load on the central collector.

The benefits of using sFlow to retrieve interface statistics become clear when you monitor large networks. Instead of requiring 5-10 servers dedicated to SNMP polling, a single sFlow analyzer can collect counters from all the interfaces in the network, providing a centralized view of utilization throughout the network, rapidly identifying any areas of congestion.