Monday, December 5, 2011

sflowtool


The sflowtool command line utility is used to convert standard sFlow records into a variety of different formats. While there are a large number of native sFlow analysis applications, familiarity with sflowtool is worthwhile since it allows a wide variety of additional tools to analyze sFlow data as well as opening up the data to custom scripting.

First download, compile and install sflowtool using the following commands:

[root@xenvm4 ~]# wget http://www.inmon.com/bin/sflowtool-3.22.tar.gz
[root@xenvm4 ~]# tar -xvzf sflowtool-3.22.tar.gz
[root@xenvm4 ~]# cd sflowtool-3.22
[root@xenvm4 sflowtool-3.22]# ./configure
[root@xenvm4 sflowtool-3.22]# make
[root@xenvm4 sflowtool-3.22]# make install

Update 14 August 2015: Download the latest version of sflowtool from GitHub, https://github.com/sflow/sflowtool/archive/master.zip

The default behavior of sflowtool is to convert sFlow into ASCII text:

[root@xenvm4 ~]# sflowtool
startDatagram =================================
datagramSourceIP 10.0.0.111
datagramSize 144
unixSecondsUTC 1321922602
datagramVersion 5
agentSubId 0
agent 10.0.0.20
packetSequenceNo 3535127
sysUpTime 270660704
samplesInPacket 1
startSample ----------------------
sampleType_tag 0:2
sampleType COUNTERSSAMPLE
sampleSequenceNo 228282
sourceId 0:14
counterBlock_tag 0:1
ifIndex 14
networkType 6
ifSpeed 100000000
ifDirection 0
ifStatus 3
ifInOctets 4839078
ifInUcastPkts 15205
ifInMulticastPkts 0
ifInBroadcastPkts 4294967295
ifInDiscards 0
ifInErrors 0
ifInUnknownProtos 4294967295
ifOutOctets 149581962744
ifOutUcastPkts 158884229
ifOutMulticastPkts 4294967295
ifOutBroadcastPkts 4294967295
ifOutDiscards 101
ifOutErrors 0
ifPromiscuousMode 0
endSample   ----------------------
endDatagram   =================================

The text output of flowtool is easily processed using scripts. The following example provides a basic skeleton for processing the output of sflowtool in Perl:

#!/usr/bin/perl -w
use strict;
use POSIX;

open(PS, "/usr/local/bin/sflowtool|") || die "Failed: $!\n";
while( <PS> ) {  
  my ($attr,$value) = split;
 
  # process attribute  
}

close(PS);

Examples of scripts using sflowtool on this blog include Memcached hot keys and Memcached missed keys. Other examples include converting sFlow for Graphite and RRDtool.

The sFlow standard extends to application layer monitoring, including visibility into HTTP performance. Implementations of sFlow for popular web servers, including Apache, NGINX, Tomcat and node.js offer real-time visibility into large web farms.

The -H option causes sflowtool to output the HTTP request samples using the combined log format, making the data accessible to most log analyzers.

[root@xenvm4 ~]# sflowtool -H
10.0.0.70 - - [22/Nov/2011:12:36:32 -0800] "GET http://sflow.org/images/h-photo.jpg HTTP/1.1" 304 0 "http://sflow.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.120 Safari/535.2"
10.0.0.70 - - [22/Nov/2011:12:36:32 -0800] "GET http://sflow.org/inc/nav.js HTTP/1.1" 304 0 "http://sflow.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.120 Safari/535.2"
10.0.0.70 - - [22/Nov/2011:12:36:32 -0800] "GET http://sflow.org/images/participant-foundry.gif HTTP/1.1" 304 0 "http://sflow.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.120 Safari/535.2"

For example, the following commands use sflowtool and webalizer to create reports:

/usr/local/bin/sflowtool -H | rotatelogs log/http_log &
webalizer -o report log/*

The resulting webalizer report shows top URLs:


The sFlow standard operates by randomly sampling packet headers. The sflowtool -t option allows sFlow to be used for remote packet capture, converting packet header information from sFlow to standard pcap format that can be used with packet analysis applications.

The following example uses sflowtool and tcpdump to display a packet trace:

[root@xenvm4 ~]# sflowtool -t | tcpdump -r - -vv
reading from file -, link-type EN10MB (Ethernet)
10:30:01.000000 arp who-has 10.0.0.66 tell 10.0.0.220
10:30:07.000000 IP (tos 0x0, ttl  64, id 49952, offset 0, flags [DF], proto: TCP (6), length: 1500) xenserver1.sf.inmon.com.39120 > openfiler.sf.inmon.com.iscsi-target: . 2757963136:2757964584(1448) ack 4136690254 win 3050 
10:30:07.000000 IP (tos 0x0, ttl  64, id 49953, offset 0, flags [DF], proto: TCP (6), length: 1500) xenserver1.sf.inmon.com.39120 > openfiler.sf.inmon.com.iscsi-target: . 1448:2896(1448) ack 1 win 3050 
10:30:07.000000 IP (tos 0x0, ttl  64, id 49954, offset 0, flags [DF], proto: TCP (6), length: 1500) xenserver1.sf.inmon.com.39120 > openfiler.sf.inmon.com.iscsi-target: . 2896:4344(1448) ack 1 win 3050

The Wireshark article describes how to use sflowtool and Wireshark to graphically display packet information.


sflowtool can also be used to convert sFlow to NetFlow version 5. The following command converts sFlow records into NetFlow records and sends them to UDP port 9991 on netflow.inmon.com:

[root@xenvm4 ~]# sflowtool -c netflow.inmon.com -d 9991

Converting sFlow to NetFlow provides compatibility with NetFlow analyzers.  However, converting sFlow to NetFlow results in a significant loss of information and it is better to use a native sFlow analyzer to get the full value of sFlow. In many cases traffic analysis software supports both sFlow and NetFlow, so conversion is unnecessary.

Finally, sFlow provides information on network, server, virtual machine and application performance and the sflowtool source code offers developers a useful starting point for adding sFlow support to network, server and application performance monitoring software - see Developer resources for additional information.

10 comments:

  1. Hi,
    Thanks for the Info, where can I get the DataPath Miss,DataPath Loss, DataPath Hits and CPU Utilization .
    The metrics (ifSpeed, ifDirection, ifStatus, ifInOctets, etc) on all port in OVS? How can I get these metrics per port?

    ReplyDelete
    Replies
    1. You need to start the sflowovsd daemon to enable sFlow in Open vSwitch. Open vSwitch will report per interface counters, see Host sFlow distributed agent

      Delete
  2. Hello,
    I enabled hsflowd and ovs-sflow agents on my server and run the sflowtool to collect sflow metrics but I am seeing following errors in the output of sflowtool. Could you help suggest what is wrong?
    From hsflowd agent:
    cpu_contexts 1301450780
    counters_sample_element length error (expected 80, found 68)
    caught exception: 3
    endDatagram =================================

    From ovs-sflow agent:
    sampleSequenceNo 256
    sourceId 0:39
    counterBlock_tag 0:1004
    skipping unknown counters_sample_element: 0:1004 len=12
    counterBlock_tag 0:1005
    skipping unknown counters_sample_element: 0:1005 len=20
    counterBlock_tag 0:1



    ReplyDelete
    Replies
    1. What version of sflowtool are you using (the latest is 3.39)? You can download and build the latest version from https://github.com/sflow/sflowtool.

      You can also run the latest version sflowtool using docker, https://hub.docker.com/r/sflow/sflowtool/.

      You might also want to consider sFlow-RT for generating flow metrics.

      Delete
  3. Thanks for your information, Peter. I was using v 3.22. After I changed to v3.39, counters_sample_element length error is disappeared. But skipping unknown counters_sample_element message is still remain.

    ReplyDelete
    Replies
    1. What is the counterBlock_tag number for the unknown element that is being skipped?

      Delete
    2. I missed the numbers is your previous message, tags 1004 and 1005 correspond to OpenFlow port number and ifName structures:
      http://sflow.org/developers/structures.php

      Delete
    3. I just tested sflowtool using a recent version of OVS (using mininet 2.2.1 on Ubuntu 16) and the ifname structures are being decoded correctly, but the openflow port numbers aren't being decoded (counters_sample_element: 0:1004).

      Delete
    4. Hello Peter, Thanks for your detailed information. Yes, I also seeing only OpenFlow port number decoding skip message. For ifname structure tag 1005, I can the the message correctly. Is there any way to configure to solve that OF port number decoding issue?
      sampleType COUNTERSSAMPLE
      sampleSequenceNo 88034
      sourceId 0:6
      counterBlock_tag 0:1004
      skipping unknown counters_sample_element: 0:1004 len=12
      counterBlock_tag 0:1005
      ifName p2p4
      counterBlock_tag 0:1

      Delete
    5. OpenFlow Port decoding issue resolved in version 3.40. Thank you for fixing issue.
      counterBlock_tag 0:1004
      openflow_datapath_id 1111111111111102
      openflow_port 65534

      Delete