Showing posts with label IPFIX. Show all posts
Showing posts with label IPFIX. Show all posts

Tuesday, May 19, 2026

Fixing Ghost Drops: How eBPF Rescued IPv6 Telemetry


A customer complains that they aren't getting IPFIX flow data from a router.

Use socat to check that IPFIX is being received (IANA assigned port for IPFIX is 4739):

socat -b 0 -dd -u UDP6-RECV:4739 - 2>&1
Output demonstrates that at least some IPFIX messages can be received when listening on port 4739.
2026/05/15 22:46:32 socat[108419] N using stdout for writing
2026/05/15 22:46:32 socat[108419] N starting data transfer loop with FDs [5,5] and [1,1]
2026/05/15 22:46:33 socat[108419] N received packet with 0 bytes from AF=10 [fec0:0000:0000:0000:0001:000c:2744:69f1]:50978
2026/05/15 22:46:33 socat[108419] N received packet with 0 bytes from AF=10 [fec0:0000:0000:0000:0001:000c:2744:69f1]:50978
Use tcpdump to check for IPFIX packets. This gives visibility into packets before the host network stack, so you can see packets before they are dropped by host network stack or host firewall
tcpdump -i enp0s3 -n udp port 4739
The output shows that IPFIX datagrams are being received from a second source, fec0::1:c:2744:69f0, but they aren't showing up in the socat output, so the Linux kernel must be dropping them for some reason.
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp0s3, link-type EN10MB (Ethernet), snapshot length 262144 bytes
21:09:57.217821 IP6 fec0::1:c:2744:69f1.50978 > fec0::27ff:fe8d:4f0b.ipfix: UDP, length 1432
21:09:57.217921 IP6 fec0::1:c:2744:69f0.50978 > fec0::27ff:fe8d:4f0b.ipfix: UDP, length 1428
A check of the host firewall and reverse path filtering settings don't explain the drops, so take a more detailed look with tcpdump with the -v (verbose) option.
tcpdump -i enp0s3 -nv udp port 4739
This time we see that packets from fec0::1:c:2744:69f0 have a bad UDP checksum.
dropped privs to tcpdump
tcpdump: listening on enp0s3, link-type EN10MB (Ethernet), snapshot length 262144 bytes
22:07:36.443823 IP6 (flowlabel 0x1991c, hlim 64, next-header UDP (17) payload length: 1230) fec0::1:c:2744:69f1.50978 > fec0::27ff:fe8d:4f0b.ipfix: [udp sum ok] UDP, length 1222
22:07:37.356528 IP6 (flowlabel 0x1991c, hlim 64, next-header UDP (17) payload length: 1008) fec0::1:c:2744:69f0.50978 > fec0::27ff:fe8d:4f0b.ipfix: [bad udp cksum 0xfc60 -> 0xfb60!] UDP, length 1000
Verify by looking at the UDP error counter:
nstat -az Udp6InErrors
Results show an increasing error counter, confirming that the packets are being dropped by the Linux kernel.
#kernel
Udp6InErrors                    5106               0.0
Unfortunately, simple fixes that might work for IPv4 (disabling/ignoring UDP checksum) are not available for IPv6, see RFC 8200 Internet Protocol, Version 6 (IPv6) Specification
Unlike IPv4, the default behavior when UDP packets are originated by an IPv6 node is that the UDP checksum is not optional. That is, whenever originating a UDP packet, an IPv6 node must compute a UDP checksum over the packet and the pseudo-header, and, if that computation yields a result of zero, it must be changed to hex FFFF for placement in the UDP header. IPv6 receivers must discard UDP packets containing a zero checksum and should log the error.
Ideally the router would send IPFIX with correctly computed UDP checksum and on closer examination it looks that some IFPIX messages from the router do have a correct UDP checksum while others do not. The messages with the correct checksum originate from the routers management CPU, while the messages with incorrect checksum are being directly generated in hardware by the routing chip.

This isn't too surprising - if this were IPv4 export, the hardware could set the UDP checksum to zero (since it is optional) and there would be no issue, However, with IPv6 the mandatory checksum must be computed - a complex calculation that is likely to produce errors.

Computation of the UDP checksum involves creating an IPv6 pseudo header (shown above) and then calculating the checksum over the IPv6 pseudo header, the UDP header, and the UDP payload.

Ideally, the router vendor will fix the issue, but it may not be possible if there are hardware limitations, or it may take time if a fix isn't seen as a priority, so a workaround is needed.

This is where eBPF comes to the rescue! The GitHub fix-udp6-checksum project uses an eBPF program, fix_checksum.c, to compute and rewrite the UDP checksum before the IPFIX packet is handed to the Linux network stack.

Dowload and compile the fix_checksum.c program on a system running Docker:

git clone https://github.com/inmoncorp/fix-udp6-checksum.git
cd fix-udp6-checksum
./build.sh

Copy the resulting fix_checksum.o file to the IPFIX collector.

Install the eBPF program on interface enp0s3

sudo tc qdisc add dev enp0s3 clsact
sudo tc filter add dev enp0s3 ingress bpf da obj fix_checksum.o sec tc/ingress

Check to see that the filter has been installed

sudo tc -s filter show dev enp0s3 ingress
The output shows that the filter is installed and that the eBPF Just-In-Time (JIT) compiler has run for maximum performance.
filter protocol all pref 49151 bpf chain 0 
filter protocol all pref 49151 bpf chain 0 handle 0x1 fix_checksum.o:[tc/ingress] direct-action not_in_hw id 230 name fix_ipfix_check tag c6a96524b9f80adb jited
Finally, using socat to verify that the missing IPFIX data is being received:
2026/05/18 14:06:02 socat[945320] N using stdout for writing
2026/05/18 14:06:02 socat[945320] N starting data transfer loop with FDs [5,5] and [1,1]
2026/05/18 14:06:03 socat[945320] N received packet with 0 bytes from AF=10 [fec0:0000:0000:0000:0001:000c:2744:69f1]:50978
2026/05/18 14:06:03 socat[945320] N received packet with 0 bytes from AF=10 [fec0:0000:0000:0000:0001:000c:2744:69f0]:50978
eBPF is a game changer, providing the ability to change how Linux processes packets without having to build a new kernel or even reboot a production system. What would otherwise have been a major issue is transformed into a relatively straightforward fix and a happy customer.

Thursday, September 15, 2022

Low latency flow analytics


Real-time analytics on network flow data with Apache Pinot describes LinkedIn's flow ingestion and analytics pipeline for sFlow and IPFIX exports from network devices. The solution uses Apache Kafka message queues to connect LinkedIn's InFlow flow analyzer with the Apache Pinot datastore to support low latency queries. The article describes the scale of the monitoring system, InFlow receives 50k flows per second from over 100 different network devices on the LinkedIn backbone and edge devices and states InFlow requires storage of tens of TBs of data with a retention of 30 days. The article concludes, Following the successful onboarding of flow data to a real-time table on Pinot, freshness of data improved from 15 mins to 1 minute and query latencies were reduced by as much as 95%.
The sFlow-RT real-time analytics engine provides a faster, simpler, more scaleable, alternative for flow monitoring. sFlow-RT  radically simplifies the measurement pipeline, combining flow collection, enrichment, and analytics in a single programmable stage. Removing pipeline stages improves data freshness — flow measurements represent an up to the second view of traffic flowing through the monitored network devices. The improvement from minute to sub-second data freshness enhances automation use cases such as automated DDoS mitigation, load balancing, and auto-scaling (see Delay and stability).

An essential step in improving data freshness is eliminating IPFIX and migrating to an sFlow only network monitoring system. Rapidly detecting large flows, sFlow vs. NetFlow/IPFIX describes how the on-device flow cache component of IPFIX/NetFlow measurements adds an additional stage (and additional latency, anywhere from 1 to 30 minutes) to the flow analytics pipeline. On the other hand, sFlow is stateless, eliminates the flow cache, and guarantees sub-second freshness. Moving from IPFIX/NetFlow is straightforward as most of the leading router vendors now include sFlow support, see Real-time flow telemetry for routers. In addition, sFlow is widely supported in switches, making it possible to efficiently monitor every device in the data center.

Removing pipeline stages increases scaleability and reduces the cost of the solution. A single instance of sFlow-RT can monitor 10's of thousands of network devices and over a million flows per second, replacing large numbers of scale-out pipeline instances. Removing the need for distributed message queues improves resilience by decoupling the flow monitoring system from the network so that it can reliably deliver the visibility needed to manage network congestion "meltdown" events — see Who monitors the monitoring systems?

Introducing a programmable flow analytics stage before storing the data significantly reduces storage requirements (and improves query latency) since only metrics of interest will be computed and stored (see Flow metrics with Prometheus and Grafana).

The LinkedIn paper mentions that they are developing an eBPF Skyfall agent for monitoring hosts. The open source Host sFlow agent extends sFlow to hosts and can be deployed at scale today, see Real-time Kubernetes cluster monitoring example.

docker run -p 8008:8008 -p 6343:6343/udp sflow/prometheus
Trying out sFlow-RT is easy. For example, run the above command to start the analyzer using the pre-built sflow/prometheus Docker image. Configure sFlow agents to stream telemetry to the analyzer and access the through the web interface on port 8008 (see Getting Started). The default settings should work for most small to moderately sized networks. See Tuning Performance for tips on optimizing performance for larger sites.

Thursday, June 2, 2016

OVS Orbit podcast with Ben Pfaff

OVS Orbit Episode 6 is a wide ranging discussion between Ben Pfaff and Peter Phaal of the industry standard sFlow measurement protocol, implementation of sFlow in Open vSwitch, network analytics use cases and application areas supported by sFlow, including: OpenStack, Open Network Virtualization (OVN), DDoS mitigation, ECMP load balancing, Elephant and Mice flows, Docker containers, Network Function Virtualization (NFV), and microservices.

Follow the link to see listen to the podcast, read the extensive show notes, follow related links, and to subscribe to the podcast.

Friday, May 6, 2016

sFlow to IPFIX/NetFlow

RESTflow explains how the sFlow architecture shifts the flow cache from devices to external software and describes how the sFlow-RT REST API can be used to program and query flow caches. Exporting events using syslog describes how flow records can be exported using the syslog protocol to Security Information and Event Management (SIEM) tools such as Logstash and and Splunk. This article demonstrates how sFlow-RT can be used to define and export the flows using the IP Flow Information eXport (IPFIX) protocol (the IETF standard based on NetFlow version 9).

For example, the following command defines a cache that will maintain flow records for TCP flows on the network, capturing IP source and destination addresses, source and destination port numbers and the bytes transferred and sending flow records to address 10.0.0.162:
curl -H "Content-Type:application/json" -X PUT --data \ '{"keys":"ipsource,ipdestination,tcpsourceport,tcpdestinationport", \
"value":"bytes", "ipfixCollectors":["10.0.0.162"]}' \
http://localhost:8008/flow/tcp/json
Running Wireshark's tshark command line utility on 10.0.0.162 verifies that flows are being received:
# tshark -i eth0 -V udp port 4739
Running as user "root" and group "root". This could be dangerous.
Capturing on lo
Frame 1 (134 bytes on wire, 134 bytes captured)
    Arrival Time: Aug 24, 2013 10:44:06.096082000
    [Time delta from previous captured frame: 0.000000000 seconds]
    [Time delta from previous displayed frame: 0.000000000 seconds]
    [Time since reference or first frame: 0.000000000 seconds]
    Frame Number: 1
    Frame Length: 134 bytes
    Capture Length: 134 bytes
    [Frame is marked: False]
    [Protocols in frame: eth:ip:udp:cflow]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IP (0x0800)
Internet Protocol, Src: 10.0.0.162 (10.0.0.162), Dst: 10.0.0.162 (10.0.0.162)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 120
    Identification: 0x0000 (0)
    Flags: 0x02 (Don't Fragment)
        0.. = Reserved bit: Not Set
        .1. = Don't fragment: Set
        ..0 = More fragments: Not Set
    Fragment offset: 0
    Time to live: 64
    Protocol: UDP (0x11)
    Header checksum: 0x2532 [correct]
        [Good: True]
        [Bad : False]
    Source: 10.0.0.162 (10.0.0.162)
    Destination: 10.0.0.162 (10.0.0.162)
User Datagram Protocol, Src Port: 56109 (56109), Dst Port: ipfix (4739)
    Source port: 56109 (56109)
    Destination port: ipfix (4739)
    Length: 100
    Checksum: 0x15b9 [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
Cisco NetFlow/IPFIX
    Version: 10
    Length: 92
    Timestamp: Aug 24, 2013 10:44:06.000000000
        ExportTime: 1377366246
    FlowSequence: 74
    Observation Domain Id: 0
    Set 1
        Template FlowSet: 2
        FlowSet Length: 40
        Template (Id = 258, Count = 8)
            Template Id: 258
            Field Count: 8
            Field (1/8)
                .000 0000 1000 0010 = Type: exporterIPv4Address (130)
                Length: 4
            Field (2/8)
                .000 0000 1001 0110 = Type: flowStartSeconds (150)
                Length: 4
            Field (3/8)
                .000 0000 1001 0111 = Type: flowEndSeconds (151)
                Length: 4
            Field (4/8)
                .000 0000 0000 1000 = Type: IP_SRC_ADDR (8)
                Length: 4
            Field (5/8)
                .000 0000 0000 1100 = Type: IP_DST_ADDR (12)
                Length: 4
            Field (6/8)
                .000 0000 1011 0110 = Type: TCP_SRC_PORT (182)
                Length: 2
            Field (7/8)
                .000 0000 1011 0111 = Type: TCP_DST_PORT (183)
                Length: 2
            Field (8/8)
                .000 0000 0101 0101 = Type: BYTES_TOTAL (85)
                Length: 8
    Set 2
        DataRecord (Template Id): 258
        DataRecord Length: 36
        Flow 1
            ExporterAddr: 10.0.0.20 (10.0.0.20)
            [Duration: 65.000000000 seconds]
                StartTime: Aug 24, 2013 10:43:01.000000000
                EndTime: Aug 24, 2013 10:44:06.000000000
            SrcAddr: 10.0.0.16 (10.0.0.16)
            DstAddr: 10.0.0.20 (10.0.0.20)
            SrcPort: 48859
            DstPort: 443
            Octets: 228045
The output demonstrates how the flow cache definition is exported as an IPFIX Template and the individual flow records are exported as one or more Flow entries within a DataRecord.

What might not be apparent is that the single configuration command to sFlow-RT enabled network wide monitoring of TCP connections, even in a network containing hundreds of physical switches, thousands of virtual switches, different switch models, multiple vendors etc. In contrast, if devices maintain their own flow caches then each switch needs to be re-configured whenever monitoring requirements change - typically a time consuming and complex manual process, see Software defined analytics.
While IPFIX provides a useful method of exporting IP flow records to legacy monitoring solutions, logging flow records is only a small subset of the applications for sFlow analytics. The real-time networking, server, and application analytics provided by sFlow-RT delivers actionable data through APIs and can easily be integrated with a wide variety of on-site and cloud, orchestration, DevOps and Software Defined Networking (SDN) tools.

Monday, December 1, 2014

Open vSwitch 2014 Fall Conference


Open vSwitch is an open source software virtual switch that is popular in cloud environments such as OpenStack. Open vSwitch is a standard Linux component that forms the basis of a number of commercial and open source solutions for network virtualization, tenant isolation, and network function virtualization (NFV) - implementing distributed virtual firewalls and routers.

The recent Open vSwitch 2014 Fall Conference agenda included a wide variety speakers addressing a range of topics, including: large scale operation experiences at Rackspace, implementing stateful firewalls, Docker networking,  and acceleration technologies (Intel DPDK and Netmap/VALE).

The video above is a recording of the following sFlow related talk from the conference:
Traffic visibility and control with sFlow (Peter Phaal, InMon)
sFlow instrumentation has been included in Open vSwitch since version 0.99.1 (released 25 Jan 2010). This talk will introduce the sFlow architecture and discuss how it differs from NetFlow/IPFIX, particularly in regards to delivering real-time flow analytics to an SDN controller. The talk will demonstrate that sFlow measurements from Open vSwitch are identical to sFlow measurements made in hardware on bare metal switches, providing unified, end-to-end, measurement across physical and virtual networks. Finally, Open vSwitch / Mininet will be used to demonstrate Elephant flow detection and marking using a combination of sFlow and OpenFlow.
Slides and videos for all the conference talks will soon be available on the Open vSwitch web site.

Sunday, November 10, 2013

UDP packet replication using Open vSwitch

UDP protocols such as sFlow, syslog, NetFlow, IPFIX and SNMP traps, have many advantages for large scale network and system monitoring, see Push vs Pull.  In a typical deployment each managed element is configured to send UDP packets to a designated collector (specified by an IP address and port). For example, in a simple sFlow monitoring system all the switches might be configured to send sFlow data to UDP port 6343 on the host running the sFlow analysis application. Complex deployments may require multiple analysis applications, for example: a first application providing analytics for software defined networking, a second focused on host performance, a third addressing packet capture and security, and the fourth looking at application performance. In addition, a second copy of each application may be required for redundancy. The challenge is getting copies of the data to all the application instances in an efficient manner.

There are a number of approaches to replicating UDP data, each with limitations:
  1. IP Multicast - if the data is sent to an IP multicast address then each application could subscribe to the multicast channel receive a copy of the data. This sounds great in theory, but in practice configuring and maintaining IP multicast connectivity can be a challenge. In addition, all the agents and collectors would need to support IP multicast. In addition, IP multicast also doesn't address the situation where you have multiple applications running on single host and so each application has to be receive the UDP data on a different port.
  2. Replicate at source - each agent could be configured to send a copy of the data to each application. Replicating at source is a configuration challenge (all agents need to be reconfigured if you add an additional application). This approach is also wasteful of bandwidth - multiple copies of the same data are send across the network.
  3. Replicate at destination - a UDP replicator, or "samplicator" application receives the stream of UDP messages, copies them and resends them to each of the applications. This functionality may be deployed as a stand alone application, or be an integrated function within an analysis application. The replicator application is a single point of failure - if it is shut down none of the applications receive data. The replicator adds delay to the measurements and at high data rates can significantly increase UDP loss rate as the datagrams are received, sent, and received again. 
This article will examine a fourth option, using software defined networking (SDN) techniques to replicate and distribute data within the network. The Open vSwitch is implemented in the Linux kernel and includes OpenFlow and network virtualization features that will be used to build the replication network.

First, you will need a server (or virtual machine) running a recent version of Linux. Next download and install Open vSwitch.

Next, configure the Open vSwitch to handle networking for the server:
ovs-vsctl add-br br0
ovs-vsctl add-port br0 eth0
ifconfig eth0 0
ifconfig br0 10.0.0.1/24
Now configure the UDP agents to send their data to 10.0.0.1. You should be able to run a collector application for each service port (e.g. sFlow 6343, syslog 514, etc.).

The first case to consider is replicating the datagrams to a second port on the server (sending packets to App 1 and App 2 in the diagram). First, use the ovs-vsctl command to list the OpenFlow port numbers on the virtual switch:
% ovs-vsctl --format json --columns name,ofport list Interface
{"data":[["eth0",1],["br1",65534]],"headings":["name","ofport"]}
We are interested in replicating packets received on eth0 and output shows that the corresponding OpenFlow port is 1.

The Open vSwitch provides a command line utility ovs-ofctl that uses the OpenFlow protocol to configure forwarding rules in the vSwitch. The following OpenFlow rule will replicate sFlow datagrams:
in_port=1 dl_type=0x0800 nw_proto=17 tp_dst=6343 actions=LOCAL,mod_tp_dst:7343,normal
The match part of the rule looks for packets received on port 1 (in_port=1), where the Ethernet type is IPv4 (dl_type=0x0800), the IP protocol is UDP (nw_protocol=17), and the destination UDP port is 6343 (tp_dst=6343). The actions section of the rule is the key to building the replication function. The LOCAL action delivers the original packet as intended. The destination port is then changed to 7343 (mod_tp_dst:7343) and the modified packet is sent through the normal processing path to be delivered to the application.

Save this rule to a file, say replicate.txt, and then use ovs-ofctl to apply the rule to br0:
ovs-ofctl add-flows br0 replicate.txt
At this point a second sFlow analysis application listening for sFlow datagrams on port 7343 should start receiving data - sflowtool is a convenient way to verify that the packets are being received:
sflowtool -p 7343
The second case to consider is replicating the datagrams to a remote host (sending packets to App 3 in the diagram).
in_port=1 dl_type=0x0800 nw_proto=17 tp_dst=6343 actions=LOCAL,mod_tp_dst:7343,normal,mod_nw_src:10.0.0.1,mod_nw_dst:10.0.0.2,normal
The extended rule includes additional actions that modify the source address of the packets (mod_nw_src:10.0.0.1) and the destination IP address (mod_nw_dst:10.0.0.2) and sends the packet through the normal processing path. Since we are relying on the routing functionality in the Linux stack to deliver the packet, make sure that routing in enabled - see How to Enable IP Forwarding in Linux.
Unicast reverse path filtering (uRPF) is mechanism that routers use to drop spoofed packets (i.e. packets where the source address doesn't belong to the subnet on the access port the packet was received on). uRPF should be enabled wherever practical because spoofing is used in a variety of security and denial of service attacks, e.g. DNS amplification attacks. By modifying the IP source address to be the address of the forwarding host (10.0.0.1) rather than the original source IP address the OpenFlow rule ensures that the packet will pass through uRPF filters, both on the host and on the access router. Rewriting the sFlow source address does not cause any problems because the sFlow protocol identifies the original source of the data within its payload and doesn't rely on the IP source address. However, other UDP protocols (for example, NetFlow/IPFIX) rely on the IP source address to identify the source of the data. In this case, removing the mod_nw_src action will leave the IP source address unchanged, but the packet may well be dropped by uRPF filters. Newer Linux distributions implement strict uRPF by default, however it can be disabled if necessary, see Reverse Path Filtering.
This article has only scratched the surface of capabilities of the Open vSwitch. In situations where passing the raw packets across the network isn't feasible the Open vSwitch can be configured to send the packets over a tunnel (sending packets to App 4 in the diagram). Tunnels, in conjunction with OpenFlow, can be used to create a virtual UDP distribution overlay network with its own addressing scheme and topology - Open vSwitch is used by a number of network virtualization vendors (e.g. VMware NSX). In addition, more complex filters can also be implemented, forwarding datagrams based on source subnet to different collectors etc.

The replication functions don't need to be performed in software in the virtual switch. OpenFlow rules can be pushed to OpenFlow capable hardware switches which can perform the replication, or source based forwarding functions at wire speed. A full blown controller based solution isn't necessarily required, the ovs-ofctl command can be used to push OpenFlow rules to physical switches.

More generally, building flexible UDP datagram distribution and replication networks is an interesting use case for software defined networking. The power of software defined networking is that you can adapt the network behavior to suit the needs of the application - in this case overcoming the limitations of existing UDP distribution solutions by modifying the behavior of the network.

Wednesday, August 28, 2013

RESTflow

Figure 1: Embedded, on-switch flow cache with flow record export
This article describes RESTflow™, a new method for exporting flow records that has significant advantages over current approaches to flow export.

A flow record summarizes a set of packets that share common attributes - for example, a typical flow record includes ingress interface, source IP address, destination IP address, IP protocol, source TCP/UDP port, destination TCP/UDP port, IP ToS, start time, end time, packet count and byte count.

Figure 1 shows the steps performed by the switch in order to construct flow records. First the stream of packets is likely to be sampled (particularly in high-speed switches). Next, the sampled packet header is decoded to extract key fields. A hash function is computed over the keys in order to look up the flow record in the flow cache. If an existing record is found, its values are updated, otherwise a record is created for the new flow. Records are flushed from the cache based on protocol information (e.g. if a FIN flag is seen in a TCP packet), a timeout, inactivity, or when the cache is full. The flushed records are finally sent to the traffic analysis application using one of the many formats that switches use to export flow records (e.g. NetFlow, IPFIX, J-Flow, NetStream, etc.).
Figure 2: External software flow cache with flow record export
Figure 2 shows the relationship between the widely supported sFlow standard for packet export and flow export. With sFlow monitoring, the decode, hash, flow cache and flush functionality are no longer implemented on the switch. Instead, sampled packet headers are immediately sent to the traffic analysis application which decodes the packets and analyzes the data. In typical deployments, large numbers of switches stream sFlow data to a central sFlow analyzer. In addition, sFlow provides a polling function; switches periodically send standard interface counters to the traffic analysis applications, eliminating the need for SNMP polling, see Link utilization.
There are significant advantages to moving the flow cache to external software: the article, Superlinear, discusses some of the scaleability implications of on device flow caches and, Rapidly detecting large flows, sFlow vs. NetFlow/IPFIX, describes how on device flow caches delay measurements and makes them less useful for software defined networking (SDN) applications.
The following example uses the sFlow-RT analyzer to demonstrate flow record export based on sFlow packet data received from a network of switches.
Figure 3: Performance aware software defined networking
Figure 3 from Performance aware software defined networking shows how sFlow-RT exposes the active flow cache to applications that address important use cases, such as DDoS mitigation, large flow load balancing, multi-tenant performance isolation, traffic engineering, and packet capture.

The recent extension of the REST API to support flow record export provides a useful log of network activity that can be incorporated in security, information and event management (SIEM) tools.

Three types of query combine to deliver the RESTflow flexible flow definition and export interface:

1. Define flow cache

The following command instructs the central sFlow-RT analytics engine running on host 10.0.0.162 to build a flow cache for TCP flows and log the completed flows:
curl -H "Content-Type:application/json" -X PUT --data '{"keys":"ipsource,ipdestination,tcpsourceport,tcpdestinationport", "value":"bytes", "log":true}' http://10.0.0.162:8008/flow/tcp/json
What might not be apparent is that the single configuration command to sFlow-RT enabled network wide monitoring of TCP connections, even in a network containing hundreds of physical switches, thousands of virtual switches, different switch models, multiple vendors etc. In contrast, if devices maintain their own flow caches then each switch needs to be re-configured whenever monitoring requirements change - typically a time consuming and complex manual process, see Software defined analytics.

To illustrate the point, the following command defines an additional network wide flow cache for records describing DNS (UDP port 53) requests and log the completed flows:
curl -H "Content-Type:application/json" -X PUT --data '{"keys":"ipsource", "value":"frames", "filter":"udpdestinationport=53", "log":true}' http://10.0.0.162:8008/flow/dns/json

2. Query flow cache definition

The following command retrieves the flow definitions:
$ curl http://10.0.0.162:8008/flow/json
{
 "dns": {
  "filter": "udpdestinationport=53",
  "fs": ",",
  "keys": "ipsource",
  "log": true,
  "n": 5,
  "t": 2,
  "value": "frames"
 },
 "tcp": {
  "fs": ",",
  "keys": "ipsource,ipdestination,tcpsourceport,tcpdestinationport",
  "log": true,
  "n": 5,
  "t": 2,
  "value": "bytes"
 }
}
The definition for a specific flow can also be retrieved:
$ curl http://10.0.0.162:8008/flow/tcp/json
{
 "fs": ",",
 "keys": "ipsource,ipdestination,tcpsourceport,tcpdestinationport",
 "log": true,
 "n": 5,
 "t": 2,
 "value": "bytes"
}

3. Retrieve flow records

The following command retrieves flow records logged by all the flow caches:
curl http://10.0.0.162:8008/flows/json?maxFlows=2
[
 {
  "agent": "10.0.0.20",
  "dataSource": "2",
  "end": 1377658682679,
  "flowID": 250,
  "flowKeys": "10.0.0.162",
  "name": "dns",
  "start": 1377658553679,
  "value": 400
 },
 {
  "agent": "10.0.0.20",
  "dataSource": "5",
  "end": 1377658681678,
  "flowID": 249,
  "flowKeys": "10.0.0.20,10.0.0.236,47571,3260",
  "name": "tcp",
  "start": 1377658613678,
  "value": 1217600
 }
]
And the following command retrieves flow records from a specific cache:
$ curl "http://10.0.0.162:8008/flows/json?name=dns&maxFlows=2"
[
 {
  "agent": "10.0.0.28",
  "dataSource": "53",
  "end": 1377658938378,
  "flowID": 400,
  "flowKeys": "10.0.0.162",
  "name": "dns",
  "start": 1377658398378,
  "value": 400
 },
 {
  "agent": "10.0.0.20",
  "dataSource": "2",
  "end": 1377658682679,
  "flowID": 251,
  "flowKeys": "10.0.0.71",
  "name": "dns",
  "start": 1377658612679,
  "value": 400
 }
]
The JSON encoded text based output is easy to read and widely supported by programming tools.
Transporting large amounts of flow data using a text based protocol might seem inefficient when compared to binary flow record export protocols such as IPFIX, NetFlow etc. However, one of the advantages of a REST API is that it builds on the mature and extensive capabilities built into the HTTP protocol stack. For example, most HTTP clients are capable of handling compression and will set the HTTP Accept-Encoding header to indicate that they are willing to accept compressed data. The sFlow-RT web server responds by compressing the data before sending it, resulting in a 20 times reduction in data volume. Similarly, using a REST API, allows users to leverage the existing infrastructure to load balance, encrypt, authenticate, cache and proxy requests.
The real power of the RESTflow API becomes apparent when it is accessed programmatically. For example, the following Python script defines the TCP flow described earlier and continuously retrieves new flow records:
#!/usr/bin/env python
import requests
import json
import signal

rt = 'http://10.0.0.162:8008'
name = 'tcp'

def sig_handler(signal,frame):
  requests.delete(rt + '/flow/' + name + '/json');
  exit(0)
signal.signal(signal.SIGINT, sig_handler)

flow = {'keys':'ipsource,ipdestination,tcpsourceport,tcpdestinationport',
        'value':'frames',
        'log':True}
r = requests.put(rt + '/flow/' + name + '/json',data=json.dumps(flow))

flowurl = rt + '/flows/json?name=' + name + '&maxFlows=100&timeout=60'
flowID = -1
while 1 == 1:
  r = requests.get(flowurl + "&flowID=" + str(flowID))
  if r.status_code != 200: break
  flows = r.json()
  if len(flows) == 0: continue

  flowID = flows[0]["flowID"]
  flows.reverse()
  for f in flows:
    print str(f['flowKeys']) + ',' + str(int(f['value'])) + ',' + str(f['end'] - f['start']) + ',' + f['agent'] + ',' + str(f['dataSource'])
The following command runs the script, which results in the newly arriving flow records being printed as comma separated text:
$ ./tcp_flows.py 
10.0.0.16,10.0.0.236,38834,3260,4000,98100,10.0.0.16,5
10.0.0.151,10.0.0.152,39046,22,837800,60000,10.0.0.28,2
10.0.0.151,10.0.0.152,39046,22,851433,60399,10.0.0.20,25
10.0.0.20,10.0.0.16,443,48859,12597,64000,10.0.0.253,1
10.0.0.152,10.0.0.151,22,39046,67049,61800,10.0.0.28,19
Instead of simply printing the flow records, the script could easily add them to scale out databases like MongoDB so that they can be combined with other types of information and easily searched.

The sFlow-RT REST API doesn't just provide access to completed flows, access to real-time information on in progress flows is available by querying the central flow cache. For example, the following command searches the flow cache and reports the most active flow in the network (based on current data transfer rate, i.e. bits/second).
$ curl http://10.0.0.162:8008/metric/ALL/tcp/json
[{
 "agent": "10.0.0.28",
 "dataSource": "2",
 "metricN": 9,
 "metricName": "tcp",
 "metricValue": 29958.06882871171,
 "topKeys": [
  {
   "key": "10.0.0.20,10.0.0.28,443,40870",
   "updateTime": 1377664899679,
   "value": 29958.06882871171
  },
  {
   "key": "10.0.0.236,10.0.0.28,3260,56044",
   "updateTime": 1377664888679,
   "value": 23.751630816369214
  }
 ],
 "updateTime": 1377664899679
}]
As well as identifying the most active flow, the query result also identifies the switch and port carrying the traffic (out of potentially tens of thousands of ports being monitored).

While flow records are a useful log of completed flows, the ability to track flows in real time transforms traffic monitoring from a reporting tool to a powerful driver for active control, unlocking the capabilities of software defined networking to dynamically adapt the network to changing demand. Embedded flow caches in networking devices are not easily accessible and even if there were a programmatic way to access the on device cache, polling thousands of devices would take so long that the information would be stale by the time it was retrieved.
Figure 4: Visibility and the software defined data center
Looking at the big picture, flow export is only one of many functions that can be performed by an sFlow analyzer, some of which have been described on this blog. Providing simple, programmatic, access allows these functions to be integrated into the broader orchestration system. REST APIs are the obvious choice since they are already widely used in data center orchestration and monitoring tools.
Embedded flow monitoring solutions typically require CLI access to the network devices to define flow caches and direct flow record export. Access to switch configurations is tightly controlled by the network management team and configuration changes are often limited to maintenance windows. In part this conservatism results because hardware resource limitations on the devices need to be carefully managed - for example, a misconfigured flow cache can destabilize the performance of the switch. In contrast, the central sFlow analyzer is software running on a server with relatively abundant resources that can safely support large numbers of requests without any risk of destabilizing the network.
The REST APIs in sFlow-RT are part of a broader movement to break out of the networking silo and integrate management of network resources with the orchestration tools used to automatically manage compute, storage and application resources. Automation transforms the network from a fragile static resource into a robust and flexible resource that can be adapted to support the changing demands of the applications it supports.

Thursday, November 1, 2012

Finding elephants



The Blind Men and the Elephant
John Godfrey Saxe (1816-1887)

It was six men of Indostan
To learning much inclined,
Who went to see the Elephant
(Though all of them were blind),
That each by observation
Might satisfy his mind.

The First approached the Elephant,
And happening to fall
Against his broad and sturdy side,
At once began to bawl:
"God bless me! but the Elephant
Is very like a wall!"

The Second, feeling of the tusk,
Cried, "Ho, what have we here,
So very round and smooth and sharp?
To me 'tis mighty clear
This wonder of an Elephant
Is very like a spear!"

The Third approached the animal,
And happening to take
The squirming trunk within his hands,
Thus boldly up and spake:
"I see," quoth he, "the Elephant
Is very like a snake!"

The Fourth reached out an eager hand,
And felt about the knee
"What most this wondrous beast is like
Is mighty plain," quoth he:
"'Tis clear enough the Elephant
Is very like a tree!"

The Fifth, who chanced to touch the ear,
Said: "E'en the blindest man
Can tell what this resembles most;
Deny the fact who can,
This marvel of an Elephant
Is very like a fan!"

The Sixth no sooner had begun
About the beast to grope,
Than seizing on the swinging tail
That fell within his scope,
"I see," quoth he, "the Elephant
Is very like a rope!"

And so these men of Indostan
Disputed loud and long,
Each in his own opinion
Exceeding stiff and strong,
Though each was partly in the right,
And all were in the wrong!

MORAL.

So oft in theologic wars,
The disputants, I ween,
Rail on in utter ignorance
Of what each other mean,
And prate about an Elephant
Not one of them has seen!

This poem is just one version of the popular Bind Men and an Elephant story. There are many conclusions that you can draw from the story, but this version captures the idea that you can't properly understand something unless you have seen it in its entirety.

Similar arguments occur among data center operations teams advocating for different performance monitoring technologies such as NetFlow, IPFIX, SNMP, WMI, JMX, libvirt etc. Many of these arguments arise because teams are siloed, with each group arguing from their narrow perspective and dismissing the concerns of other teams. While narrowly focussed tools have their place, they miss the big picture.

Cloud-based architectures tightly couple network, storage and compute resources into a large scale, flexible, platform for delivering application services. The article, System boundary, describes the need for a holistic approach to organizing and managing data center resources. A comprehensive, cloud-oriented, approach to monitoring is essential for troubleshoot performance problems, automating operations and fully exploiting the potential of cloud architectures to increase efficiency by adapting to changing demand.

The sFlow standard addresses the challenge of cloud monitoring by embedding instrumentation throughout the cloud infrastructure in order to provide a comprehensive, real-time, view of the performance of all the individual network, server and application resources as well as the cloud as a whole. The sFlow architecture is designed to see elephants, not just individual body parts. It's only by gaining this comprehensive view of performance that large scale cloud data center environments can be properly understood and managed.

Wednesday, September 12, 2012

Snowflakes, IPFIX, NetFlow and sFlow

Snow flakes by Wilson Bentley
Each snowflake is unique and beautiful. However, while such immense diversity is attractive in nature, variation in data center management standards results in operational complexity, making it difficult to implement the automation and control needed to effectively manage at scale.

The following table examines the approaches taken by the IPFIX and sFlow standards by contrasting how they handle four basic aspects of measurement.

Note: The IPFIX standard is based on Cisco's NetFlow™ version 9 protocol and most of the points of comparison apply equally to NetFlow.

IPFIXsFlow
Packet IPFIX currently defines over 50 fields relating to packet header (see IP Flow Information Export (IPFIX) Entities):
  • protocolIdentifier
  • ipClassOfService
  • tcpControlBits
  • sourceTransportPort
  • sourceIPv4Address
  • destinationTransportPort
  • destinationIPv4Address
  • sourceIPv6Address
  • destinationIPv6Address
  • flowLabelIPv6
  • icmpTypeCodeIPv4
  • igmpType
  • sourceMacAddress
  • vlanId
  • ipVersion
  • ipv6ExtensionHeaders
  • destinationMacAddress
  • icmpTypeCodeIPv6
  • icmpTypeIPv4
  • icmpCodeIPv4
  • icmpTypeIPv6
  • icmpCodeIPv6
  • udpSourcePort
  • udpDestinationPort
  • tcpSourcePort
  • tcpDestinationPort
  • tcpSequenceNumber
  • tcpAcknowledgementNumber
  • tcpWindowSize
  • tcpUrgentPointer
  • tcpHeaderLength
  • ipHeaderLength
  • totalLengthIPv4
  • payloadLengthIPv6
  • ipTTL
  • nextHeaderIPv6
  • ipDiffServCodePoint
  • ipPrecedence
  • fragmentFlags
  • ipPayloadLength
  • udpMessageLength
  • isMulticast
  • ipv4IHL
  • ipv4Options
  • tcpOptions
  • ipTotalLength
  • ethernetHeaderLength
  • ethernetPayloadLength
  • ethernetTotalLength
  • dot1qVlanId
  • dot1qPriority
  • dot1qCustomerVlanId
  • dot1qCustomerPriority
  • ethernetType
The IPFIX standard does not require vendors to support all the fields, each vendor is free to export any combination of fields that they choose, none of the fields are mandatory. The result is that each vendor and each product produces unique and incompatible data.
The sFlow standard specifies a single way to report packet attributes, the packet header, ensuring that every vendor and product produces compatible results.

Every sFlow compatible device deployed since the sFlow standard was published in 2001 provides visibility into every protocol that has ever, or will ever, run over Ethernet. The packet header includes all the protocol fields exported by IPFIX as well as fields associated with emerging protocols such as FCoE, AoE, TRILL, NVGRE and VxLAN that have yet to by defined in IPFIX.
Time IPFIX has over 30 elements that can be used to represent time (see IP Flow Information Export (IPFIX) Entities):
  • flowEndSysUpTime
  • flowStartSysUpTime
  • flowStartSeconds
  • flowEndSeconds
  • flowStartMilliseconds
  • flowEndMilliseconds
  • flowStartMicroseconds
  • flowEndMicroseconds
  • flowStartNanoseconds
  • flowEndNanoseconds
  • flowStartDeltaMicroseconds
  • flowEndDeltaMicroseconds
  • flowDurationMilliseconds
  • flowDurationMicroseconds
  • observationTimeSeconds
  • observationTimeMilliseconds
  • observationTimeMicroseconds
  • observationTimeNanoseconds
  • monitoringIntervalStartMilliSeconds
  • monitoringIntervalEndMilliSeconds
  • collectionTimeMilliseconds
  • maxExportSeconds
  • maxFlowEndSeconds
  • minExportSeconds
  • minFlowStartSeconds
  • maxFlowEndMicroseconds
  • maxFlowEndMilliseconds
  • maxFlowEndNanoseconds
  • minFlowStartMicroseconds
  • minFlowStartMilliseconds
  • minFlowStartNanoseconds
The IPFIX standard allows vendors to report time using these elements in any combination, or to omit timestamps altogether. In order to report time consistently, every agent must have a real-time clock and be time synchronized. Finally, it is left up the vendors to decide how often to export data and so an IPFIX collector must understand each vendor's implementation in order to be certain that it has received all the data and detect data loss.
The sFlow standard requires that data be sent immediately. The stateless nature of the protocol means that data can be combined and timestamps added by the central sFlow collector without any need for timestamps or time synchronization among the agents.

Note: The sFlow datagrams do contain a time stamp, the agent uptime in milliseconds at the time the datagram was sent.
SamplingIPFIX currently defines eight different algorithms for packet sampling (see IANA Packet Sampling Parameters):
  • Systematic count-based Sampling
  • Systematic time-based Sampling
  • Random n-out-of-N Sampling
  • Uniform probabilistic Sampling
  • Property match Filtering
  • Hash based Filtering using BOB
  • Hash based Filtering using IPSX
  • Hash based Filtering using CRC
Vendors are not required to implement any of these algorithms and are free to invent their own sampling schemes (see NetFlow-lite). In addition, many of the standard algorithms can be shown to be inaccurate.
The sFlow standard mandates a single, statistically valid, sampling algorithm. All sFlow compliant vendors and products, implement the same algorithm and produce accurate, interoperable results.
URLThere is no-standard IPFIX element for exporting a URL. However, IPFIX does allow vendor extensions, resulting in multiple schemes for exporting URL data. Examples include:
  • nProbe URLs are additional fields that can be included as flow keys when configuring the probe.
  • Dell SonicWall URLs are included in an HTTP specific table and link to flow records.
  • Citrix AppFlow URLs are included in an HTTP request table with links to additional HTTP response and ingress/egress connection tables. 
In each case, in addition to the URL element itself being vendor specific, the information model associated with the exported URLs is also unique, reflecting the internal architecture of the exporting device.
The sFlow standard mandates a set of HTTP counters and transaction attributes that ensures consistent reporting from HTTP aware entities such as web servers (Apache, Tomcat, NGINX etc.) and load balancers (F5 etc.), irrespective of vendor or internal architecture.

Each URL is exported as part of the standard transaction record that includes: client IP, server IP, referrer, authuser, user-agent, mime-type, status, request-bytes, response-bytes, response time. In addition, the sFlow standard defines a unified data model that links measurements from network devices, servers and application instances to provide a comprehensive, data center wide, view of performance.

From the examples in the table, it is apparent that IPFIX and sFlow standards take two very different approaches. The IPFIX standard is descriptive, defining a standard set of attributes that vendors can use to describe the information that they choose to export. The result is that vendors use IPFIX to differentiate each product, reporting a unique and inconsistent set of measurements based on its internal architecture and product features. In contrast, the sFlow standard is prescriptive, defining a set of measurements that every vendor must implement. While IPFIX provides a way to describe each "snowflake", the sFlow standard results from vendors working together to identifying common measurements and implement them in an interoperable way.

Henry Ford transformed the auto industry by moving from hand-made, custom parts to standardized components and processes that allowed for mass production. The data center is undergoing a similar transformation, from small, static, custom environments to large scale, commoditized, flexible, cloud architectures. The sFlow standard delivers the universal performance measurements needed for automation, enjoys broad vendor support, and along with other disruptive technologies like 10G Ethernet, merchant silicon, Software Defined Networking (SDN), OpenFlow, networked storage and virtualization is enabling this transformation.

Tuesday, May 1, 2012

Software defined networking

Figure 1: Software Defined Networking architecture
The elements of the Software Defined Networking (SDN) architecture are shown in Figure 1. The Data Plane comprises switches connected together to form a network. However, instead of relying on proprietary software running on each switch to control its forwarding behavior, switches in a SDN architecture are controlled by a Network OS (NOS) that interacts with the switches to provide an abstract model of the network topology to Applications running on the NOS. Applications can adapt the network behavior to suite specialized requirements, for example, providing network virtualization services that allow multiple logical networks to share a single physical network - similar to the way in which a hypervisor allows multiple virtual machines to share a single physical machine.

Note: While to diagram shows the NOS as a single logical entity, it is likely to be implemented as a distributed cluster of controllers in order to provide scalability and fault tolerance.

Open APIs for programatic access to the switches is an essential pre-requisite to building a software defined network:
  1. Forwarding The OpenFlow protocol was originally developed so that academic researchers could experiment with external control of switch packet forwarding. OpenFlow quickly gained support, leading to the formation of the Open Networking Foundation (ONF) to develop and promote the OpenFlow standard.
  2. Configuration  It was quickly realized that OpenFlow alone isn't sufficient - a configuration protocol is needed to assign switches to controllers, configure port settings and provision queues. Recently, the Open Networking Foundation released the initial OF-Config version 1.0 specification for configuring OpenFlow switches. OF-Config is defined as an extension of the NETCONF protocol, which provides an programmatic XML/RPC API that is well suited to SDN.
  3. Visibility Current efforts in the SDN community are focused on provisioning of network services.  Going beyond merely providing connectivity to creating a NOS that is aware of network performance requires an API providing visibility into switch traffic. A performance aware NOS allows applications to manage resource allocation, balance loads and ensure quality of service.
The rest of this article examines different approaches to monitoring switch performance and their strengths and weaknesses within the context of software defined networking.
Figure 2: NetFlow/IPFIX and sFlow
The most common APIs for monitoring switch performance, NetFlow/IPFIX and sFlow, are shown in Figure 2:
  1. NetFlow/IPFIX Cisco NetFlow and IPFIX (the IETF standard based on NetFlow) define a protocol for exporting flow records. A flow record summarizes a set of packets that share common atributes - for example, a typical flow record includes ingress interface, source IP address, destination IP address, IP protocol, source TCP/UDP port, destination TCP/UDP port, IP ToS, start time, end time, packet count and byte count. Figure 2 shows the steps performed by the switch in order to construct flow records. First the stream of packets is likely to be sampled (particularly in high-speed switches). Next, the sampled packet header is decoded to extract key fields. A hash function is computed over the keys in order to look up the flow record in the flow cache. If an existing record is found, its values are updated, otherwise a record is created for the new flow. Records are flushed from the cache based on protocol information (e.g. if a FIN flag is seen in a TCP packet), a timeout, inactivity, or when the cache is full. The flushed records are finally sent to the traffic analysis application.
  2. sFlow With sFlow monitoring, the decode, hash, flow cache and flush functionality are no longer implemented on the switch. Instead, sampled packet headers are sent to the traffic analysis application which decodes the packets and aggregates the data. In addition, sFlow provides a polling function, periodically sending standard interface counters to the traffic analysis applications, eliminating the need for SNMP polling, see Link utilization.
Given the differences between the two technologies, the following attributes highlight features of sFlow that make it an attractive option for providing visibility in SDN environments:
  1. Software Defined With sFlow, decoding and analysis of network traffic is performed by external software, allowing the NOS to tailor measurements for consistency with its internal network model. In contrast, the decode, hash, flush, pipeline needed to generate flow records is typically implemented in hardware. Hardware-based measurements are inflexible, requiring switch vendor hardware/firmare upgrades to implement new measurements.  Hardware differences mean that measurements are inconsistent between vendors, or even between different products from the same vendor.
  2. Lightweight The sFlow architecture minimized resources consumed on the switch. In contrast, flow-based measurements consumes scarce TCAM resources that would be better used to increase the number of packet forwarding entries.
  3. Stateless The sFlow architecture is stateless, measurements are not stored on the switch, but immediately exported to the traffic analyzer (NOS) where they can be acted on. Flow-based measurements rely on a stateful flow cache that delays export, making the data less suitable for implementing dynamic control functionality in the NOS, see Delay and stability. 
  4. Easy to configure Very few parameters required to configure sFlow monitoring: the address of the traffic analyzer, a counter polling interval and a packet sampling rate. Flow-based technologies require additional configuration parameters: identifying packet attributes to decode, specifying how packet attributes and values should be used to update the flow cache, flow cache size, flow cache timeouts, etc. Operational complexity acts as a significant barrier to large scale deployment, see  Complexity kills.
  5. Scalable The sFlow architecture scales better than flow-based architectures, see Superlinear. 
  6. Multi-Vendor The sFlow standard is supported by most switch vendors, including virtually all OpenFlow capable switches. 
  7. Merchant Silicon Hardware support for sFlow is widely implemented in merchant silicon.
In the data center, SDN is being used as a component of cloud orchestration systems like OpenStack and CloudStack that manage network, server and application resources in order to deliver Infrastructure as a Service (IaaS), Platform as a Service (PaaS) and Software as a Service (SaaS).  The sFlow standard supports cloud orchestration by providing unified, cloud-scale, visibility that links network, system and application performance in a single integrated system.

Sunday, May 22, 2011

Delay and stability


Stability and control used the recent Amazon EC2 outage as an example to demonstrate the critical role that measurement plays in automating data center operations. This article looks at one important aspect of the automation system, the effect that delay has on the stability.

The diagram, from Electric Drives - Motor Controllers and Control Systems, illustrates how short, moderate and long delays in responding to disturbances affects the stability of the response. A short delay between a disturbance and corrective action ensures that the response is appropriate and results in a stable automation system. Introducing delay results in responses that are out of sync since the system state may have changed during the delay. As delay increases, the responses become increasingly out of step with the system, resulting in instability and failure. The effect of delay on the stability of automation systems is a general phenomenon that applies equally to all types of automation system, including network and system management systems.

There are are a number of steps where delay is introduced into network and system management:
  1. Measurement delay How quickly can changes be detected? Measurement systems often introduce significant delays:
    • Polling Polling systems periodically retrieving performance metrics from each of the systems being managed. Polling intervals can be short for small numbers of systems, but as the number of systems increases, the work involved in polling increases and each system is polled less frequently. Assigning additional servers to monitoring increases polling capacity, but creates an extra layer of management that adds delay to the monitoring system.  The end result is that measurement delay increases as the number of systems gets larger, reducing the stability of the management system.
    • Flows Network monitoring technologies such as NetFlow, jFlow, NetStream and IPFIX report on traffic flows seen by a network device. With flow monitoring, the network device keeps track of connections flowing through it and exports measurements when each connection ends. Delay is intrinsic to this type of measurement; the delay being proportional to the connection duration. While timeouts can be used to limit the measurement delay, reducing delay increases the computational load on the devices and the amount of measurement traffic each device generates.
  2. Planning delay How long does it take the controller to react to a new measurement? Many aspects of data center management are currently left to human operators, resulting in significant delays. Increased automation reduces delays and improves stability.
  3. Configuration delay How long does it take for the control to be implemented? Implementing a configuration change takes time. The controller needs to connect to the systems and communicate configuration changes. If these steps are carried out manually then they introduce significant delay. Automated configuration management significantly improves the speed and accuracy of configuration changes. For example, the OpenFlow protocol allows a controller to rapidly reconfigure networking devices in response to changing network demands.
  4. Response delay How long does it take for the configuration change to take effect? There may be additional delay as the change takes effect, for example move a virtual machine takes an appreciable amount of time.
Each of these sources of delay is cumulative, delay introduced at any stage affects that overall stability of the control policy.

Measurement is the foundation for automation; without measurement there is no information to react to and performance problems go undetected until they result in serious failures. To be effective for automation, the measurement system must have the scalability to report on all critical network, server and application performance metrics (see concept of observability in Stability and control) while ensuring that measurement delays are low.

The sFlow standard is well suited to automation, providing a unified measurement system that includes network, system and application performance. Devices implementing sFlow do not maintain state, instead measurements are sent immediately to a central collector, resulting in extremely low measurement delay and the scalability needed to centrally monitor all resources in the data center. For additional information, the Data center convergence, visibility and control presentation describes the critical role that measurement plays in managing costs and optimizing performance.

Tuesday, November 16, 2010

Complexity kills


The July 2010 presentation NetFlow/IPFIX Various Thoughts from the IETF 3rd NMRG Workshop on NetFlow/IPFIX Usage in Network Management describes some of the challenges that still need to be addressed in NetFlow/IPFIX. In particular the slide above describes how increased flexibility has resulted in greater complexity when trying to configure and deploy NetFlow/IPFIX monitoring systems.

In contrast, sFlow implementations have very few configuration options. While there are superficial similarities between sFlow and NetFlow/IPFIX, the two approaches to network performance management reflect profound differences between the design goals of the two standards (see Standards).

NetFlow/IPFIX was developed to export WAN traffic measurements and is typically deployed in IP routers. Configuring routers is a complex task, requiring configuration of subnets, routing protocols, WAN interfaces etc. Many of the functions in a router are implemented in software, providing a flexible platform that permits complex measurements to be made. Over time, options have been added to NetFlow/IPFIX in order to export increasingly complex measurements used to characterize WAN traffic.

sFlow evolved to provide end-to-end monitoring of high-speed layer 2/3 Ethernet switches. Ethernet switches offer plug-and-play connectivity and require very little configuration. Unlike routers, switches perform most of their functions in hardware, relying on software only to perform simple management tasks. The need to embed measurement in hardware resulted in a standard that is very simple with minimal configuration options. However, the basic sFlow measurements, while simple to configure and implement in switches, provide a rich source of information about the performance of switched networks. Instead of relying on the switches to analyze the traffic, raw data is sent to a central sFlow analyzer (see Choosing an sFlow analyzer). The sFlow architecture results in a highly scalable system that can monitor the large numbers of high-speed switch ports found in layer 2 networks (see Superlinear).

The goal of convergence is to simplify data centers, creating flexible pools of storage and computing running over a flat, high bandwidth, low latency, Ethernet fabric (see Convergence). Eliminating complexity is essential if the scalability and flexibility of a converged infrastructure is to be realized.

Microsoft's Chief Software Architect, Ray Ozzie, eloquently describes the dangers of complexity in Dawn of a New Day, "Complexity kills. Complexity sucks the life out of users, developers and IT. Complexity makes products difficult to plan, build, test and use.  Complexity introduces security challenges. Complexity causes administrator frustration."

Maintaining visibility in large scale, dynamic data center environments requires a measurement technology that is designed for the task. sFlow is a mature, multi-vendor standard supported by most switch vendors that delivers the scalable plug and play visibility needed to manage performance in converged data center environments.

Finally, the end to end visibility that sFlow provides is a critical element in building scalable systems. Measurement eliminates uncertainty and reduces the complexity of managing large systems  (see Scalability). An effective monitoring system is the foundation for automation: reducing costs, improving efficiency and optimizing performance in the data center.

Wednesday, September 8, 2010

Superlinear

(diagram from Going Superlinear)

The article, Going Superlinear, describes factors that affect the scalability of parallel processing systems. Ideally a parallel system will show a linear improvement in performance as the number of processors increases. However, typical parallel systems show sublinear performance since memory or I/O eventually becomes a bottleneck. Finally, the article states that it is sometimes possible to achieve superlinear scalability where adding processors creates a disproportionate increase in performance.

Applying the concepts from parallel processing to analyze the scalability of network performance monitoring systems is an interesting exercise. The parallel processing that allows a network monitoring system to scale is embedded in the network switches and operates as part of the packet forwarding function in each switch port. Therefore, it makes sense to measure system size in number of switch ports, not number of processors. The switch resources consumed by the monitoring system are a limiting factor. If an increase in network size requires a disproportionately increase in switch resources then the monitoring system will have limited scalability.
    The following diagram plots the relationship between network size and parallel speedup. The sublinear (limited scalability) region is shown in red and the superlinear (scalable) region is shown in green. The speedup associated with the sFlow and NetFlow network monitoring systems are plotted on the chart.


    In a NetFlow monitoring system, each switch maintains a flow cache containing active connections. As the network size increases, the amount of memory allocated to each flow cache must be increased to accommodate the increased number of active connections associated with a larger network. This disproportionate increase in switch resource needed to implement NetFlow results in sublinear scalability.

    The article states that superlinear scalability is possible if the monitoring system can:
    • Do disproportionately less work.
    • Harness disproportionately more resources.
    The sFlow monitoring system harnesses both techniques to increase scalability:
    • Less work. Packet sampling allows the switch to perform disproportionately less work (see Scalability and accuracy of packet sampling). Packet samples are not stored on the switch, but are immediately sent to a central collector for analysis. The elimination of the flow cache further reduces the switch resources consumed by monitoring.
    • More resources. Moving analysis to a central collector harnesses the disproportionately larger memory and computational resources available on a server compared to the limited resources available on a switch (see Choosing an sFlow analyzer).
    Moving the flow cache from the switches to the central analyzer greatly improves memory efficiency:
    • NetFlow caches must be sized to handle worst case loads. Most switches in the network will have excess flow cache capacity while a few busy switches might have insufficient flow cache memory. Centralizing the cache allows memory to be pooled and flexibly re-assigned  from idle to busy switches as traffic patterns change.
    • While some NetFlow implementations support packet sampling, the use of sampling has no impact on memory overhead since on-board flow caches are sized for worst case (unsampled) configurations. Thus, if packet sampling is used with NetFlow, the result is poor cache utilization. However, with a centralized cache the memory savings associated with sampling increase with network size.
    • Traffic paths often traverse multiple switches. With NetFlow, each switch in the path will keep a copy the flow in its cache.  A centralized cache reduces memory requirements by eliminating the redundant copies.
    • Centralizing the cache replaces scarce, expensive switch memory with abundant, inexpensive server memory.
      Memory efficiency is just one aspect of sFlow's scalability. Other aspects of the sFlow architecture that contribute to its scalability have been discussed in previous articles: Link utilization, Measurement traffic and Measurement overhead.

      The differences in scalability between NetFlow and sFlow reflect differences in the size and type of network targeted by the two technologies. Typically, NetFlow monitoring is selectively applied to a relatively small number of routed links, whereas sFlow is used to monitor all the links in large switched networks (see LAN and WAN).

      The sFlow standard is widely supported by switch vendors, delivering the scalability needed to manage the performance of large, converged networks. The scalability of sFlow allows performance monitoring to extend beyond the network to include virtualization, server and cloud computing in a single integrated system.