Wednesday, September 12, 2012

Snowflakes, IPFIX, NetFlow and sFlow

Snow flakes by Wilson Bentley
Each snowflake is unique and beautiful. However, while such immense diversity is attractive in nature, variation in data center management standards results in operational complexity, making it difficult to implement the automation and control needed to effectively manage at scale.

The following table examines the approaches taken by the IPFIX and sFlow standards by contrasting how they handle four basic aspects of measurement.

Note: The IPFIX standard is based on Cisco's NetFlow™ version 9 protocol and most of the points of comparison apply equally to NetFlow.

Packet IPFIX currently defines over 50 fields relating to packet header (see IP Flow Information Export (IPFIX) Entities):
  • protocolIdentifier
  • ipClassOfService
  • tcpControlBits
  • sourceTransportPort
  • sourceIPv4Address
  • destinationTransportPort
  • destinationIPv4Address
  • sourceIPv6Address
  • destinationIPv6Address
  • flowLabelIPv6
  • icmpTypeCodeIPv4
  • igmpType
  • sourceMacAddress
  • vlanId
  • ipVersion
  • ipv6ExtensionHeaders
  • destinationMacAddress
  • icmpTypeCodeIPv6
  • icmpTypeIPv4
  • icmpCodeIPv4
  • icmpTypeIPv6
  • icmpCodeIPv6
  • udpSourcePort
  • udpDestinationPort
  • tcpSourcePort
  • tcpDestinationPort
  • tcpSequenceNumber
  • tcpAcknowledgementNumber
  • tcpWindowSize
  • tcpUrgentPointer
  • tcpHeaderLength
  • ipHeaderLength
  • totalLengthIPv4
  • payloadLengthIPv6
  • ipTTL
  • nextHeaderIPv6
  • ipDiffServCodePoint
  • ipPrecedence
  • fragmentFlags
  • ipPayloadLength
  • udpMessageLength
  • isMulticast
  • ipv4IHL
  • ipv4Options
  • tcpOptions
  • ipTotalLength
  • ethernetHeaderLength
  • ethernetPayloadLength
  • ethernetTotalLength
  • dot1qVlanId
  • dot1qPriority
  • dot1qCustomerVlanId
  • dot1qCustomerPriority
  • ethernetType
The IPFIX standard does not require vendors to support all the fields, each vendor is free to export any combination of fields that they choose, none of the fields are mandatory. The result is that each vendor and each product produces unique and incompatible data.
The sFlow standard specifies a single way to report packet attributes, the packet header, ensuring that every vendor and product produces compatible results.

Every sFlow compatible device deployed since the sFlow standard was published in 2001 provides visibility into every protocol that has ever, or will ever, run over Ethernet. The packet header includes all the protocol fields exported by IPFIX as well as fields associated with emerging protocols such as FCoE, AoE, TRILL, NVGRE and VxLAN that have yet to by defined in IPFIX.
Time IPFIX has over 30 elements that can be used to represent time (see IP Flow Information Export (IPFIX) Entities):
  • flowEndSysUpTime
  • flowStartSysUpTime
  • flowStartSeconds
  • flowEndSeconds
  • flowStartMilliseconds
  • flowEndMilliseconds
  • flowStartMicroseconds
  • flowEndMicroseconds
  • flowStartNanoseconds
  • flowEndNanoseconds
  • flowStartDeltaMicroseconds
  • flowEndDeltaMicroseconds
  • flowDurationMilliseconds
  • flowDurationMicroseconds
  • observationTimeSeconds
  • observationTimeMilliseconds
  • observationTimeMicroseconds
  • observationTimeNanoseconds
  • monitoringIntervalStartMilliSeconds
  • monitoringIntervalEndMilliSeconds
  • collectionTimeMilliseconds
  • maxExportSeconds
  • maxFlowEndSeconds
  • minExportSeconds
  • minFlowStartSeconds
  • maxFlowEndMicroseconds
  • maxFlowEndMilliseconds
  • maxFlowEndNanoseconds
  • minFlowStartMicroseconds
  • minFlowStartMilliseconds
  • minFlowStartNanoseconds
The IPFIX standard allows vendors to report time using these elements in any combination, or to omit timestamps altogether. In order to report time consistently, every agent must have a real-time clock and be time synchronized. Finally, it is left up the vendors to decide how often to export data and so an IPFIX collector must understand each vendor's implementation in order to be certain that it has received all the data and detect data loss.
The sFlow standard requires that data be sent immediately. The stateless nature of the protocol means that data can be combined and timestamps added by the central sFlow collector without any need for timestamps or time synchronization among the agents.

Note: The sFlow datagrams do contain a time stamp, the agent uptime in milliseconds at the time the datagram was sent.
SamplingIPFIX currently defines eight different algorithms for packet sampling (see IANA Packet Sampling Parameters):
  • Systematic count-based Sampling
  • Systematic time-based Sampling
  • Random n-out-of-N Sampling
  • Uniform probabilistic Sampling
  • Property match Filtering
  • Hash based Filtering using BOB
  • Hash based Filtering using IPSX
  • Hash based Filtering using CRC
Vendors are not required to implement any of these algorithms and are free to invent their own sampling schemes (see NetFlow-lite). In addition, many of the standard algorithms can be shown to be inaccurate.
The sFlow standard mandates a single, statistically valid, sampling algorithm. All sFlow compliant vendors and products, implement the same algorithm and produce accurate, interoperable results.
URLThere is no-standard IPFIX element for exporting a URL. However, IPFIX does allow vendor extensions, resulting in multiple schemes for exporting URL data. Examples include:
  • nProbe URLs are additional fields that can be included as flow keys when configuring the probe.
  • Dell SonicWall URLs are included in an HTTP specific table and link to flow records.
  • Citrix AppFlow URLs are included in an HTTP request table with links to additional HTTP response and ingress/egress connection tables. 
In each case, in addition to the URL element itself being vendor specific, the information model associated with the exported URLs is also unique, reflecting the internal architecture of the exporting device.
The sFlow standard mandates a set of HTTP counters and transaction attributes that ensures consistent reporting from HTTP aware entities such as web servers (Apache, Tomcat, NGINX etc.) and load balancers (F5 etc.), irrespective of vendor or internal architecture.

Each URL is exported as part of the standard transaction record that includes: client IP, server IP, referrer, authuser, user-agent, mime-type, status, request-bytes, response-bytes, response time. In addition, the sFlow standard defines a unified data model that links measurements from network devices, servers and application instances to provide a comprehensive, data center wide, view of performance.

From the examples in the table, it is apparent that IPFIX and sFlow standards take two very different approaches. The IPFIX standard is descriptive, defining a standard set of attributes that vendors can use to describe the information that they choose to export. The result is that vendors use IPFIX to differentiate each product, reporting a unique and inconsistent set of measurements based on its internal architecture and product features. In contrast, the sFlow standard is prescriptive, defining a set of measurements that every vendor must implement. While IPFIX provides a way to describe each "snowflake", the sFlow standard results from vendors working together to identifying common measurements and implement them in an interoperable way.

Henry Ford transformed the auto industry by moving from hand-made, custom parts to standardized components and processes that allowed for mass production. The data center is undergoing a similar transformation, from small, static, custom environments to large scale, commoditized, flexible, cloud architectures. The sFlow standard delivers the universal performance measurements needed for automation, enjoys broad vendor support, and along with other disruptive technologies like 10G Ethernet, merchant silicon, Software Defined Networking (SDN), OpenFlow, networked storage and virtualization is enabling this transformation.


  1. IPFIX can be used to sample packets similar to sFlow. An example is NetFlow-Lite. My experience with sFlow is that when I go look for traffic from a specific host, sFlow didn't sample it. IPFIX can be used for accurate IP accounting and sFlow can't be.

    Regarding "The result is that each vendor and each product produces unique and incompatible data." This point allows IPFIX vendors to not be limited to a standard. Similar to SNMP there is MIB2 and then there are the Enterprise MIBs.

    1. Jake, the article describes the many ways that IPFIX can perform packet sampling (and Uniform Probalistic Sampling is similar to sFlow's mechanism). However, there is no requirement in IPFIX that vendors implement that algorithm, they are free to implement any sampling algorithm they choose with no requirement that the algorithm produce statistically valid results.

      I like your analogy to SNMP, the problem with the IPFIX standard is that it is like SNMP without MIB2. The major contribution of SNMP was not the protocol itself (I don't think many people think the SNMP protocol is great) but the huge contribution that MIB2 made in simplifying multi-vendor management by defining a standard set of measurements that every vendor was required to implement. The ifTable in particular mandates a set of interface counters that every SNMP compatible device is required to maintain (FYI sFlow defines it's counter export in terms of MIB2, ensuring compatibility between sFlow and SNMP monitoring, see Link utilization).

      There was an attempt in 2005 to define a standard set of IPFIX tables and measurements (the equivalent of MIB2 for IPFIX), but this effort by service providers didn't go anywhere, however, the proposal, IPFIX templates for common ISP usages, makes interesting reading. In this aspect, IPFIX is a regressive step. Cisco's NetFlow version 5 defined the flow record that everyone is familiar with and NetFlow version 5 equivalents were pretty consistently implemented by a number of other vendors.

      As to sFlow's applicability to IP accounting, the document sFlow accuracy and billing describes strategies for accounting based on sFlow and there are many service providers using sFlow for billing.

      If your goal is to monitor specific packets or packet from a specific host, then SPAN/RSPAN is a better approach. Brocade has implemented an ACL based packet capture mechanism as an sFlow vendor extension allowing for detailed protocol following of specific hosts for troubleshooting.

    2. I think you miss understood my analogy to SNMP. IPFIX has options. There are currently over 300 standard IPFIX information elements. If you can't find everything you want from MIB2 in those elements there is currently a draft that seeks to leverage not just all of MIB2, but any MIB. I don't think it is as easy to do with sFlow.

      Pretty much all vendors and I know of very few exceptions, follow these same exact elements. And by the way, although Cisco is free to define information elements above the 300 the way they want, they generally keep them compatible with IPFIX.

      IP accounting based on sFlow samples is still not going to be as accurate as NetFlow or IPFIX. I challenge any consumer to run their own test as we did: and compare these technologies. When you step away from theory and enter the business world, sFlow OK but, not comparable to IPFIX and NetFlow.

    3. I agree that IPFIX has defines hundreds of options - this article listed many of them. The challenge for device vendors is in deciding exactly which measurements they should make and how they should choose to export those measurements. The challenge for network operators is making sense of the different and often incompatible measurements each device exports. The document I referenced earlier, IPFIX templates for common ISP usages, was an attempt to ensure that basic flow records would be compatible between IPFIX and NetFlow implementations across all vendors. Interoperability, standard templates are what is currently missing from IPFIX, i.e. IPFIX has no analog to SNMP's standard MIBs (like MIB2).

      You refer to the export of SNMP data using IPFIX as proposed in the draft, Exporting MIB Variables using the IPFIX Protocol. Again, this proposal provides many options, but does not require vendors to export any particular values or templates. In contrast, sFlow mandates that every vendor export the standard MIB-2 ifTable values for each interface on the device, i.e. ifIndex, ifType, ifSpeed, ifAdminStatus, ifOperStatus, ifInOctets, ifInUcastPkts, ifInMulticastPkts, ifInBroadcastPkts, ifInDiscards, ifInErrors, ifInUnknownProtos, ifOutOctets, ifOutUcastPkts, ifOutMulticastPkts, ifOutBroadcastPkts, ifOutDiscards, ifOutErrors.

      The Network World article you reference does demonstrate some important differences, sFlow reports on IP and non-IP traffic, NetFlow is IP only. The time smearing effect of NetFlow monitoring and its effect in delaying measurements and distorting utilization trends is also demonstrated.

      Unfortunately, problems with the experimental setup make it hard to draw any conclusions about accuracy from the results. Problems with the methodology, some of which were identified in the article, include: the lack of an accurate reference (how can you compare measurement technologies if you don't know the correct answer?), problems with the NetFlow and sFlow configuration in the switches (flow timeouts, sampling rates) and finally problems with scaling of sampled data. The comparison also ignores the MIB-2 interface counters exported by sFlow since they would provide a 100% accurate view of link traffic totals.

      Finally, IPFIX/NetFlow and sFlow are largely complementary and most enterprises will likely use both, see LAN and WAN.

  2. Somehow the URL got cut off in my last post. Here is the correct URL when testing NetFlow Vs. sFlow:

    1. You might be interested in the other articles on this blog that discuss differences between sFlow and NetFlow,

    2. The Network World article was informative, but it doesn't look as though the collector software was correctly scaling the sFlow data. For example, the factor-of-4 discrepancy in the sFlow from the Extreme switch was probably caused by the switch automatically backing off the sampling rate from 1-in-1 to 1-in-4 (which is allowed by the sFlow standard). I would have expected correctly-scaled sFlow to report slightly more traffic than NetFlow in this test. Partly because sFlow reports on the non-IP protocols, and partly because NetFlow does not usually count layer2 header bytes.