Configuring AlaxalA switches

The following commands configure an AlaxalA switch, sampling at 1-in-1024, polling counters every 30 seconds and sending sFlow to an analyzer (10.0.0.50) over UDP using the default sFlow port (6343):

(config)# sflow destination 10.0.0.50 6343
(config)# sflow extended-information-type switch
(config)# sflow sample 1024
(config)# sflow polling-interval 30

For each port:

(config)# interface gigabitethernet 0/4
(config-if)# sflow forward ingress

A previous posting discussed the selection of sampling rates. Additional information can be found on the AlaxalA web site.

See Trying out sFlow for suggestions on getting started with sFlow monitoring and reporting.

Snowflakes, IPFIX, NetFlow and sFlow

Snow flakes by Wilson Bentley

Each snowflake is unique and beautiful. However, while such immense diversity is attractive in nature, variation in data center management standards results in operational complexity, making it difficult to implement the automation and control needed to effectively manage at scale.

The following table examines the approaches taken by the IPFIX and sFlow standards by contrasting how they handle four basic aspects of measurement.

Note: The IPFIX standard is based on Cisco's NetFlow™ version 9 protocol and most of the points of comparison apply equally to NetFlow.

	IPFIX	sFlow
Packet	IPFIX currently defines over 50 fields relating to packet header (see IP Flow Information Export (IPFIX) Entities): protocolIdentifier ipClassOfService tcpControlBits sourceTransportPort sourceIPv4Address destinationTransportPort destinationIPv4Address sourceIPv6Address destinationIPv6Address flowLabelIPv6 icmpTypeCodeIPv4 igmpType sourceMacAddress vlanId ipVersion ipv6ExtensionHeaders destinationMacAddress icmpTypeCodeIPv6 icmpTypeIPv4 icmpCodeIPv4 icmpTypeIPv6 icmpCodeIPv6 udpSourcePort udpDestinationPort tcpSourcePort tcpDestinationPort tcpSequenceNumber tcpAcknowledgementNumber tcpWindowSize tcpUrgentPointer tcpHeaderLength ipHeaderLength totalLengthIPv4 payloadLengthIPv6 ipTTL nextHeaderIPv6 ipDiffServCodePoint ipPrecedence fragmentFlags ipPayloadLength udpMessageLength isMulticast ipv4IHL ipv4Options tcpOptions ipTotalLength ethernetHeaderLength ethernetPayloadLength ethernetTotalLength dot1qVlanId dot1qPriority dot1qCustomerVlanId dot1qCustomerPriority ethernetType The IPFIX standard does not require vendors to support all the fields, each vendor is free to export any combination of fields that they choose, none of the fields are mandatory. The result is that each vendor and each product produces unique and incompatible data.	The sFlow standard specifies a single way to report packet attributes, the packet header, ensuring that every vendor and product produces compatible results. Every sFlow compatible device deployed since the sFlow standard was published in 2001 provides visibility into every protocol that has ever, or will ever, run over Ethernet. The packet header includes all the protocol fields exported by IPFIX as well as fields associated with emerging protocols such as FCoE, AoE, TRILL, NVGRE and VxLAN that have yet to by defined in IPFIX.
Time	IPFIX has over 30 elements that can be used to represent time (see IP Flow Information Export (IPFIX) Entities): flowEndSysUpTime flowStartSysUpTime flowStartSeconds flowEndSeconds flowStartMilliseconds flowEndMilliseconds flowStartMicroseconds flowEndMicroseconds flowStartNanoseconds flowEndNanoseconds flowStartDeltaMicroseconds flowEndDeltaMicroseconds flowDurationMilliseconds flowDurationMicroseconds observationTimeSeconds observationTimeMilliseconds observationTimeMicroseconds observationTimeNanoseconds monitoringIntervalStartMilliSeconds monitoringIntervalEndMilliSeconds collectionTimeMilliseconds maxExportSeconds maxFlowEndSeconds minExportSeconds minFlowStartSeconds maxFlowEndMicroseconds maxFlowEndMilliseconds maxFlowEndNanoseconds minFlowStartMicroseconds minFlowStartMilliseconds minFlowStartNanoseconds The IPFIX standard allows vendors to report time using these elements in any combination, or to omit timestamps altogether. In order to report time consistently, every agent must have a real-time clock and be time synchronized. Finally, it is left up the vendors to decide how often to export data and so an IPFIX collector must understand each vendor's implementation in order to be certain that it has received all the data and detect data loss.	The sFlow standard requires that data be sent immediately. The stateless nature of the protocol means that data can be combined and timestamps added by the central sFlow collector without any need for timestamps or time synchronization among the agents. Note: The sFlow datagrams do contain a time stamp, the agent uptime in milliseconds at the time the datagram was sent.
Sampling	IPFIX currently defines eight different algorithms for packet sampling (see IANA Packet Sampling Parameters): Systematic count-based Sampling Systematic time-based Sampling Random n-out-of-N Sampling Uniform probabilistic Sampling Property match Filtering Hash based Filtering using BOB Hash based Filtering using IPSX Hash based Filtering using CRC Vendors are not required to implement any of these algorithms and are free to invent their own sampling schemes (see NetFlow-lite). In addition, many of the standard algorithms can be shown to be inaccurate.	The sFlow standard mandates a single, statistically valid, sampling algorithm. All sFlow compliant vendors and products, implement the same algorithm and produce accurate, interoperable results.
URL	There is no-standard IPFIX element for exporting a URL. However, IPFIX does allow vendor extensions, resulting in multiple schemes for exporting URL data. Examples include: nProbe URLs are additional fields that can be included as flow keys when configuring the probe. Dell SonicWall URLs are included in an HTTP specific table and link to flow records. Citrix AppFlow URLs are included in an HTTP request table with links to additional HTTP response and ingress/egress connection tables.  In each case, in addition to the URL element itself being vendor specific, the information model associated with the exported URLs is also unique, reflecting the internal architecture of the exporting device.	The sFlow standard mandates a set of HTTP counters and transaction attributes that ensures consistent reporting from HTTP aware entities such as web servers (Apache, Tomcat, NGINX etc.) and load balancers (F5 etc.), irrespective of vendor or internal architecture. Each URL is exported as part of the standard transaction record that includes: client IP, server IP, referrer, authuser, user-agent, mime-type, status, request-bytes, response-bytes, response time. In addition, the sFlow standard defines a unified data model that links measurements from network devices, servers and application instances to provide a comprehensive, data center wide, view of performance.

From the examples in the table, it is apparent that IPFIX and sFlow standards take two very different approaches. The IPFIX standard is descriptive, defining a standard set of attributes that vendors can use to describe the information that they choose to export. The result is that vendors use IPFIX to differentiate each product, reporting a unique and inconsistent set of measurements based on its internal architecture and product features. In contrast, the sFlow standard is prescriptive, defining a set of measurements that every vendor must implement. While IPFIX provides a way to describe each "snowflake", the sFlow standard results from vendors working together to identifying common measurements and implement them in an interoperable way.

Henry Ford transformed the auto industry by moving from hand-made, custom parts to standardized components and processes that allowed for mass production. The data center is undergoing a similar transformation, from small, static, custom environments to large scale, commoditized, flexible, cloud architectures. The sFlow standard delivers the universal performance measurements needed for automation, enjoys broad vendor support, and along with other disruptive technologies like 10G Ethernet, merchant silicon, Software Defined Networking (SDN), OpenFlow, networked storage and virtualization is enabling this transformation.

Vendor support

Cisco's recent support for the sFlow standard should come as no surprise. The graph trends the rapid growth in vendor support for sFlow over the last decade. Today, in addition to Cisco, virtually every other major vendor ships products with sFlow, including: HP, IBM, Dell, Juniper, Brocade, Arista, Huawei, Hitachi, AlaxalA, NEC, Alcatel-Lucent, Fortinet, D-Link, NETGEAR, Extreme Networks, Allied Telesis, ZTE, ZyXEL and LG-ERICCSON.

Growth would have been even faster but industry consolidation has combined a number of sFlow vendors; 3Com and H3C are now combined with ProCurve in Hewlett-Packard, Blade Network Technologies is now part of IBM and Force10 joins PowerConnect as part of Dell. However, this consolidation of US vendors is more than offset by adoption of the sFlow standard among emerging Asian vendors, including: Huawei, ZTE and Edge-Core Networks. Additionally, the graph doesn't count merchant silicon vendors, including Broadcom, Marvell and Intel, that implement sFlow support in the ASICs used by many of the switch vendors.

The rise in vendor support for sFlow was initially driven adoption of 1G Ethernet and more recent growth has been driven by the accelerating deployment of 10G Ethernet. Looking forward, the growth in number of vendors will slow down - there are very few vendors left that do not support sFlow. However, expect vendors to expand the range of products that support sFlow as new 10G, 40G and 100G Ethernet switches are developed to address increasing demand for bandwidth. Also expect to see increased support for sFlow in wireless networks.

Finally, the sFlow standard provides the end-to-end, multi-vendor visibility needed for effective control of resources in the data center and new technologies like OpenFlow and Software Defined Networking (SDN) are unlocking this potential by allowing networks to automatically adapt to the changing real-time traffic patterns reported by sFlow.

Configuring IBM RackSwitch switches

The following commands configure an IBM RackSwitch, sampling packets at 1-in-1024, polling counters every 30 seconds and sending sFlow to an analyzer (10.0.0.50) over UDP using the default sFlow port (6343):

RS(config)# sflow server 10.0.0.50
RS(config)# sflow port 6343
RS(config)# sflow enable

For each port:

RS(config)# interface port 1
RS(config-if)# sflow polling 30
RS(config-if)# sflow sampling 1024

A previous posting discussed the selection of sampling rates. Additional information can be found on the IBM web site.

See Trying out sFlow for suggestions on getting started with sFlow monitoring and reporting.

GPU performance monitoring

NVIDIA's Compute Unified Device Architecture (CUDA™) dramatically increases computing performance by harnessing the power of the graphics processing unit (GPU). Recently, NVIDIA published the sFlow NVML GPU Structures specification, defining a standard set of metrics for reporting GPU health and performance, and extended the Host sFlow agent to export the GPU metrics.

The following displays the sFlow metrics using sflowtool, the GPU metrics are highlighted:

[pp@test] /usr/local/bin/sflowtool
startDatagram =================================
datagramSourceIP 10.0.0.150
datagramSize 512
unixSecondsUTC 1346360234
datagramVersion 5
agentSubId 100000
agent 10.0.0.150
packetSequenceNo 1
sysUpTime 3000
samplesInPacket 1
startSample ----------------------
sampleType_tag 0:2
sampleType COUNTERSSAMPLE
sampleSequenceNo 1
sourceId 2:1
counterBlock_tag 0:2001
adaptor_0_ifIndex 1
adaptor_0_MACs 1
adaptor_0_MAC_0 000000000000
adaptor_1_ifIndex 2
adaptor_1_MACs 1
adaptor_1_MAC_0 e0cb4e98f891
adaptor_2_ifIndex 3
adaptor_2_MACs 1
adaptor_2_MAC_0 e0cb4e98f890
counterBlock_tag 0:2005
disk_total 145102770176
disk_free 46691696640
disk_partition_max_used 76.06
disk_reads 477615
disk_bytes_read 13102692352
disk_read_time 2227298
disk_writes 2370522
disk_bytes_written 193176428544
disk_write_time 445531146
counterBlock_tag 0:2004
mem_total 12618829824
mem_free 2484174848
mem_shared 0
mem_buffers 971259904
mem_cached 8214761472
swap_total 12580810752
swap_free 12580810752
page_in 6400433
page_out 94324428
swap_in 0
swap_out 0
counterBlock_tag 5703:1
nvml_device_count 1
nvml_processes 0
nvml_gpu_mS 0
nvml_mem_mS 0
nvml_mem_bytes_total 6441598976
nvml_mem_bytes_free 6429614080
nvml_ecc_errors 0
nvml_energy_mJ 74569
nvml_temperature_C 54
nvml_fan_speed_pc 30
counterBlock_tag 0:2003
cpu_load_one 0.040
cpu_load_five 0.240
cpu_load_fifteen 0.350
cpu_proc_run 0
cpu_proc_total 229
cpu_num 8
cpu_speed 1600
cpu_uptime 896187
cpu_user 21731800
cpu_nice 120230
cpu_system 5686620
cpu_idle 2844149774
cpu_wio 2992230
cpuintr 570
cpu_sintr 222180
cpuinterrupts 166594944
cpu_contexts 266986130
counterBlock_tag 0:2006
nio_bytes_in 0
nio_pkts_in 0
nio_errs_in 0
nio_drops_in 0
nio_bytes_out 0
nio_pkts_out 0
nio_errs_out 0
nio_drops_out 0
counterBlock_tag 0:2000
hostname test0
UUID 00000000000000000000000000000000
machine_type 3
os_name 2
os_release 2.6.35.14-106.fc14.x86_64
endSample   ----------------------
endDatagram   =================================

Note: Currently only the Linux version of Host sFlow includes the GPU support and the agent needs to be compiled from sources on a system that includes the NVML library.

The inclusion of GPU metrics in Host sFlow offers an extremely scaleable, lightweight solution for monitoring compute cluster performance. In addition to exporting a comprehensive set of standard performance metrics, the Host sFlow agent also offers a convenient API for exporting custom application metrics.

The sFlow standard isn't limited to monitoring compute resources; most network switch vendors include sFlow support, providing detailed visibility into cluster communication patterns and network utilization. Combining sFlow from switches, servers and applications delivers a comprehensive view of cluster performance.

Configuring ZyXEL switches

The following commands configure a ZyXEL switch, sampling packets at 1-in-1024, polling counters every 30 seconds and sending sFlow to an analyzer  (10.0.0.50) over UDP using the default sFlow port (6343):

sysname(config)# sflow
sysname(config)# sflow collector 10.0.0.50 udp-port 6343
sysname(config)# interface port-channel 1-24
sysname(config-interface)# sflow collector 10.0.0.50 poll-interval 30 sampling-rate 1024
sysname(config-interface)# exit
sysname(config)# exit

A previous posting discussed the selection of sampling rates. Additional information can be found on the ZyXEL web site.

See Trying out sFlow for suggestions on getting started with sFlow monitoring and reporting.

Cisco adds sFlow support

Cisco Nexus 3000 series switches

Cisco added support for the sFlow standard in the latest NX-OS 5.0(3)U4(1) release for Nexus 3000 series switches. The Nexus 3000 series are the first Cisco switches based on merchant silicon, which includes hardware support for sFlow, offering scalable, wire-speed, monitoring of all traffic flowing throughout entire networks of Nexus 3000 series switches.

Example: sFlowTrend Top connections chart

The article, 10 Gigabit Ethernet, describes the trend toward 10 Gigabit networking and the critical role that top of rack switches play in next generation data center architectures. Most organisations are predicted to upgrade to 10 Gigabit top of rack switches within the next two years in order to support the demands of virtualization and cloud computing. With the addition of Cisco, all leading switch vendors now have 10 Gigabit top of rack switches that support the sFlow standard, making sFlow the obvious choice when selecting a vendor neutral performance monitoring solution for large scale cloud environments.

Since the Nexus 3000 series switches are the first Cisco products with sFlow, the rest of this article is addressed to Cisco network administrators who are likely to be unfamiliar with sFlow technology. As a Cisco network administrator, you are likely to have experience with using Cisco's Switched Port Analyzer (SPAN) technology to selectively monitor traffic in Cisco edge switches and with Cisco's Netflow technology for monitoring TCP/IP traffic in Cisco routers.

By adding sFlow support to the Nexus 3000 series, Cisco eliminates the need for probes, providing wire-speed 10 Gigabit monitoring of all switch ports - the functional equivalent of forty-eight 10 Gigabit probes and four 40 Gigabit probes in a Nexus 3064 - embedded in the switch hardware at no extra cost. If you are familiar with RMON probes, sFlow is functionally equivalent to deploying an RMON probe for each switch port.

Based on the name, you might think that sFlow is just another version of Cisco Netflow. However, this is not the case - sFlow differs significantly from NetFlow and understanding these differences is important if you want to get the most out of sFlow:

1. sFlow exports interface counters, eliminating the need for SNMP polling - extremely useful when you have tens of thousands of edge switch ports to monitor.
2. sFlow exports packet headers not flow records. By exporting packet headers, sFlow is able to provide full layer 2 - 7 visibility into all types of traffic flowing at the network edge, including: MAC addresses, VLANs, TRILL, tunnels (GRE, VXLAN etc.), Ethernet SAN traffic (FCoE and AoE), IPv6 in addition to the TCP/IP information typically reported by NetFlow. You can even use sFlow with Wireshark for remote packet capture.
3. sFlow is highly scalable. Unlike NetFlow, which is typically enabled on selected links at the core, sFlow is enabled on every port, on every switch, for full end-to-end network visibility. The sFlow measurements are implemented in silicon and won't impact switch CPU. The scalability of sFlow allows tens of thousands of 10G switch ports in the top of rack switches, as well as their 40 Gigabit uplink ports, to be centrally monitored. In addition, sFlow is available in 100 Gigabit switches, ensuring visibility as higher speed interconnects are deployed to support the growing 10 Gigabit edge.
4. sFlow is easy to configure and manage. Eliminating complexity is essential for large scale web 2.0, big data, virtualization and cloud deployments.
5. sFlow is a multi-vendor standard supported by almost every network equipment vendor. You can mix and match Cisco Nexus 3000 series switches with best in class solutions from other vendors and still maintain comprehensive, interoperable, data center wide visibility.
6. sFlow is not just for switches. The sFlow standard also provides visibility into server, storage, virtual machine and application performance, helping to break down management silos by providing a consistent view of performance to operations and development teams (see DevOps).
7. sFlow functionality is determined by the choice of sFlow analyzer. With Flexible NetFlow, much of the analysis is performed on the network device, limiting the functionality of NetFlow collectors to simply recording the data and generating reports. As a result, NetFlow collectors end up being fairly generic in functionality. In contrast, sFlow shifts analysis from the switches to a central sFlow analyzer which determines how to process the data and present the results, see Choosing an sFlow analyzer. The result is a greater diversity of solutions and there is likely to be an sFlow analyzer that is particularly well adapted to your requirements. While many NetFlow collectors claim sFlow support, their support tends to be limited, ignoring sFlow specific features and treating sFlow as if it were basic NetFlow version 5.

Trying out sFlow is easy, just upgrade to the latest NX-OS release, configure sFlow export, and install the free sFlowTrend analyzer to gain real-time visibility - providing immediate answers to the Who, What, Where, When, Why and How questions that are the key to effective management.