Vector Packet Processor (VPP) dropped packet notifications

Vector Packet Processor (VPP) release 25.10 extends the sFlow implementation to include support for dropped packet notifications, providing detailed, low overhead, visibility into traffic flowing through a VPP router, see Vector Packet Processor (VPP) for performance information.

sflow sampling-rate 10000
sflow polling-interval 20
sflow header-bytes 128
sflow direction both
sflow drop-monitoring enable
sflow enable GigabitEthernet0/8/0
sflow enable GigabitEthernet0/9/0
sflow enable GigabitEthernet0/a/0

The above VPP configuration commands enable sFlow monitoring of the VPP dataplane, randomly sampling packets, periodically polling counters, and capturing dropped packets and reason codes. The measurements are send via Linux netlink messages to an instance of the open source Host sFlow agent (hsflowd) which combines the measurements and streams standard sFlow telemetry to a remote collector.

sflow {
  collector { ip=192.0.2.1 udpport=6343 }
  psample { group=1 egress=on }
  dropmon { start=on limit=50 }
  vpp { }
}

The /etc/hsflowd.conf file above enables the modules needed to receive netlink messages from VPP and send the resulting sFlow telemetry to a collector at 192.0.2.1. See vpp-sflow for detailed instructions.

docker run -p 6343:6343/udp sflow/sflowtool

Run sflowtool on the sFlow collector host to verify verify that the data is being received and to see the information available in the sFlow telemetry stream. The pre-built sflow/sflowtool Docker image is the fastest way to run sflowtool.

Docker also makes it easy to try out other sFlow analytics tools, for example, Deploy real-time network dashboards using Docker compose, describes how to quickly set up measurement stack consisting of the sFlow-RT real-time analytics engine, Prometheus time series database, and Grafana dashboard builder.

The sFlow VPP module delivers the same industry standard network performance monitoring available in switches and routers from leading vendors, including Arista, Cisco, Dell, Juniper, NVIDIA, VyOS, etc. to provide comprehensive, network-wide, visibility.

Capture to pcap file using sflowtool

Replay pcap files using sflowtool describes how to capture sFlow datagrams using tcpdump and replay them in real time using sflowtool. However, using tcpdump for the capture has the downside of requiring root privileges. A recent update to sflowtool now makes it possible to use sflowtool to capture sFlow datagrams in tcpdump pcap format without the need for root access.

docker run --rm -p 6343:6343/udp sflow/sflowtool -M > sflow.pcap

Either compile the latest version of sflowtool or, as shown above, use Docker to run the pre-built sflow/sflowtool image. The -M option outputs whole UDP datagrams received to standard output. In either case, type CNTRL + C to end the capture.

Replay pcap files using sflowtool

It can be very useful to capture sFlow telemetry from production networks so that it can be replayed later to perform off-line analysis, or to develop or evaluate sFlow collection tools.

sudo tcpdump -i any -s 0 -w sflow.pcap udp port 6343

Run the command above on the system you are using to collect sFlow data (if you aren't yet collecting sFlow, see Agents for suggested configuration settings). Type Control-C to end the capture after 5 to 10 minutes.  Copy the resulting sflow.pcap file to your laptop.

docker run --rm -it -v $PWD/sflow.pcap:/sflow.pcap sflow/sflowtool \
  -r /sflow.pcap -P 1

Either compile the latest version of sflowtool or, as shown above, use Docker to run the pre-built sflow/sflowtool image. The -P (Playback) option replays the trace in real-time and displays the contents of each sFlow message. Running sflowtool using Docker provides additional examples, including converting the sFlow messages into JSON format for processing by a Python script. 

docker run --rm -it -v $PWD/sflow.pcap:/sflow.pcap sflow/sflowtool \
  -r /sflow.pcap -f 192.168.4.198/6343 -P 1

The -f (forwarding) option takes an IP address and UDP port number as arguments, in this case the laptop's address, 192.168.4.198, and the standard sFlow port, 6343. Use this option to send the sFlow stream to sFlow analytics software.

For example, Deploy real-time network dashboards using Docker compose, describes how to quickly stand up an sFlow-RT, Prometheus, and Grafana analytics stack.

VyOS 1.4 LTS released

Protectli Vault - 4 Port

The VyOS 1.4.0 (Sagitta) LTS release announcement is exciting news! VyOS is an open source router operating system based on Linux that can be installed on commodity PC hardware - for optimal performance at least 1GB RAM and 4GB of storage space is recommended.

The new 1.4 LTS release includes a significantly enhanced implementation of industry standard sFlow telemetry based on the open source Host sFlow agent.

set system sflow interface eth0
set system sflow interface eth1
set system sflow interface eth2
set system sflow interface eth3
set system sflow polling 30
set system sflow sampling-rate 1000
set system sflow drop-monitor-limit 50
set system sflow server 192.0.2.100

Enter the commands above to enable sFlow monitoring on interfaces eth0, eth1, eth2, and eth3. Interface counters will be exported every 30 seconds, packets will be sampled with probability 1/1000, and up to 50 packet headers (and drop reasons) per second will collected from packets dropped by the router. The sFlow telemetry stream will be sent to an sFlow collector at 192.0.2.100.

Running Docker on the sFlow collector makes it easy to run a variety of sFlow analytics tools.

docker run --rm -p 6343:6343/udp sflow/sflowtool

Run the sflow/sflowtool image to decode and print the contents of the sFlow telemetry stream and verify receipt of data.

docker run --rm -p 6343:6343/udp sflow/tcpdump tcp port 80

Run the sflow/tcpdump image to decode and filter sampled packet headers. For more complex packet analysis tasks, try the sflow/tshark image.

Run the sflow/sflowtrend image to trend interface counters and top flows.

Deploy real-time network dashboards using Docker compose describes how to configure Prometheus and Grafana to capture time series data and create custom dashboards.

Dropped packet reason codes in VyOS describes how the new Linux kernel in VyOS 1.4 provides detailed visibility into every dropped packet (including the reason it was dropped). This cabability is used by the new sFlow agent implement the sFlow Dropped Packet Notification Structures extension to provide network-wide visibility into dropped packets.

Download VyOS today to try out the new features. Pre-built LTS images are available with paid support, but anyone can build an image from sources or download the latest rolling release.

Transit delay and queueing

The recently finalized sFlow Transit Delay Structures extension provides visibility into the performance of packet forwarding in a switch or router using the industry standard sFlow protocol.

The diagram provides a logical representation of packet forwarding. A packet is received at an Ingress Port, the packet header is examined and a forwarding decision is made to add the packet to one of the queues associated with an Egress Port, finally the packet is removed from the queue and sent out the Egress Port to be received by the next device in the chain.

The time between sending and receiving a packet is the packet's transit delay. The transit delay is affected by the time it takes to make the forwarding decision and the time the packet spends in the queue. Identifying the specific queue selected and the number of bytes already in the queue fills out the set of performance metrics for the forwarding decision. The sFlow Transit Delay Structures extension adds these performance metrics to the metadata associated with each packet sample. 

The following output from sflowtool shows that data contained in a packet sample:

startSample ----------------------
sampleType_tag 0:1
sampleType FLOWSAMPLE
sampleSequenceNo 91159
sourceId 0:2216
meanSkipCount 400
samplePool 36463600
dropEvents 0
inputPort 2215
outputPort 2216
flowBlock_tag 0:1036
extendedType egress_queue
egress_queue_id 7
flowBlock_tag 0:1040
extendedType queue_depth
queue_depth_bytes 11354112
flowBlock_tag 0:1039
extendedType transit_delay
transit_delay_nS 839660224
flowBlock_tag 0:1
flowSampleType HEADER
headerProtocol 1
sampledPacketSize 1446
strippedBytes 4
headerLen 128
headerBytes 98-03-9B-8F-B5-CC-98-03-9B-94-C7-D5-08-00-45-16-05-94-12-C7-00-00-FE-11-B8-43-C0-00-02-02-C6-33-64-02-30-39-D4-31-05-80-D7-1D-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42
dstMAC 98039b8fb5cc
srcMAC 98039b94c7d5
IPSize 1428
ip.tot_len 1428
srcIP 192.0.2.2
dstIP 198.51.100.2
IPProtocol 17
IPTOS 22
IPTTL 254
IPID 50962
UDPSrcPort 12345
UDPDstPort 54321
UDPBytes 1408
endSample   ----------------------

The forwarding performance information is highlighted. The inputPort, outputPort, egress_queue_id, queue_depth_bytes, and transit_delay_nS values describe the performance observed by the sampled packet. The sampled packet header allows performance to be reported by specific hosts, protocols, ports, connections, etc.

Linux 4.11 kernel extends packet sampling support describes the Linux PSAMPLE interface used by the Host sFlow agent to receive packet samples. PSAMPLE has been extended in the Linux 5.13 kernel to add the performance metrics, PSAMPLE_ATTR_OUT_TC (egress queue), PSAMPLE_ATTR_OUT_TC_OCC (egress queue depth), and PSAMPLE_ATTR_LATENCY (transit delay) needed to populate the sFlow Transit Delay Structures and the Host sFlow agent now exports the performance data when it is available. The decoded sFlow record, above, was generated by Host sFlow running on a hardware switch and shows measurements made by the switch ASIC.

PSAMPLE and the Host sFlow agent are becoming the standard for sFlow monitoring of Linux based operating systems such as Cumulus Linux, DENT, and SONiC.  As ASIC vendors include the measurements in their device driver PSAMPLE support, they will automatically be included in the sFlow telemetry.

Support for the new extensions has also been added to the sFlow-RT real-time analytics engine. The open source sFlow-RT Flow Browser application shown in the screen shot above displays a real-time, up to the second, view of traffic based on the packet sample telemetry streaming from network devices (switches, routers, and hosts).

In the chart above, the value being plotted has been changed from Bits per Second and is now displaying flows with the highest transit delay (in nanoseconds). The specific device, ingress port, egress port, and egress queue are also identified. 

In the chart above, queue depth (in bytes) is displayed, showing that the nearly 12 Mbytes queue depth is responsible for the transit delay seen in the previous chart.

If the queue is full and the packet is dropped, the sFlow Dropped Packet Notification Structures extension allows the sFlow agent to report details of the dropped packet. Using sFlow to monitor dropped packets describes how the Host sFlow agent uses the Linux drop_monitor interface to implement the extension.

In the final chart above, the open source sFlow-RT Discard Browser application displays a sequence of packets being dropped by a switch as a host attempts, and fails, to establish a TCP connection. The reason for dropping the packets (an access control list) as well as device and ingress port where the packets were dropped are captured.

Transit delay and dropped packet monitoring leverage advanced instrumentation in the latest generation of network ASICs to provide valuable insight into network performance. Integration with industry standard streaming sFlow telemetry provides real-time network-wide visibility into traffic, performance, and errors.

Cumulus Linux 4.2

Cumulus Linux is a network operating system for open networking hardware. Cumulus VX is a free virtual appliance that allows network engineers to experiment with Cumulus Linux and verify configurations before deploying into production. 

The Cumulus VX documentation describes how to build network topologies in KVM, VirtualBox, using VMWare hypervisors. If you want to run virtual machines locally, Cumulus in the Cloud is a free service that will allow you to access pre-built networks in the public cloud.

A key feature of Cumulus Linux is the use of the Linux kernel as the authoritative repository of network state. A result of this approach is that the behavior of a Cumulus Linux VX virtual appliance is the same as Cumulus Linux running on a hardware switch. For example, the open source FRR routing daemon shipped with Cumulus Linux uses the Linux netlink API to push routes to the kernel, which forwards packets in the virtual appliance. On a physical switch, routes are still pushed to the kernel, but kernel routing configuration is then offloaded to the switch ASIC so that packets bypass the kernel and are routed by hardware.

Cumulus Linux includes the open source Host sFlow agent. Here again, standard Linux APIs are used to implement sFlow packet sampling (see Linux 4.11 kernel extends packet sampling support). On the virtual appliance, packet sampling is performed by the Linux kernel. On a hardware switch packet sampling is offloaded to the switch ASIC. In both cases streaming sFlow telemetry provides visibility into packet forwarding.

This article demonstrates how to configure and enable sFlow Cumulus Linux (see Monitoring System Statistics and Network Traffic with sFlow). If you don't have access to a switch, download Cumulus VX to follow the example.

First log into the switch.

ssh cumulus@leaf01

Next, edit configuration file:

sudo vi /etc/hsflowd.conf

Change the following configuration setting to send sFlow to a collector at address 10.0.0.30:

sflow {
...
  collector { ip=10.0.0.30 }
...
}

Start the hsflowd daemon:

sudo systemctl enable hsflowd@mgmt.service
sudo systemctl start hsflowd@mgmt.service

In this case the collector is on the out of band management network and so the daemon needs to run in the management VRF (see Management VRF).

Use the following commands instead to send sFlow to an in-band collector on the default VRF:

sudo systemctl enable hsflowd.service
sudo systemctl start hsflowd.service

Note: Automating sFlow configuration is straightforward since the configurations are not switch specific so that every switch in the network can be given the same configuration.

Docker Desktop provides a convenient method of running sFlow analytics software on the collector machine (10.0.0.30).

docker run --rm -p 6343:6343/udp sflow/sflowtool

Use the sflow/sflowtool image to verify that sFlow telemetry is being received.

docker run --rm -p 8008:8008 -p 6343:6343/udp sflow/prometheus

Use the sflow/prometheus image to run the sFlow-RT real-time analyzer packaged with tools to browse flows and metrics and export data to Prometheus and Grafana (see sFlow-RT Network Interfaces, sFlow-RT Countries and Networks, and sFlow-RT Health dashboards for examples). Open the URL http://localhost:8008/ to access the web interface.

RESTful control of Cumulus Linux ACLs describes an open source extension to the Cumulus Linux REST API that has been used to automate control actions based on real-time traffic analytics: Triggered remote packet capture using filtered ERSPAN and DDoS mitigation with Cumulus Linux.

Cumulus VX provides a convenient platform for developing and testing monitoring and control strategies before deploying them into a production network. The consistency between the behavior of the Cumulus VX virtual appliance and Cumulus Linux running on a physical switch assures a seamless transition.

Using sFlow to monitor dropped packets

Visibility into dropped packets describes instrumentation, recently added to the Linux kernel, that provides visibility into packets dropped by the kernel data path on a host, or dropped by a switch ASIC when packets are forwarded in hardware. This article describes integration of drop monitoring in the open source Host sFlow agent and inclusion of drop reporting as part of industry standard sFlow telemetry.

Extending sFlow to provide visibility into dropped packets offers significant benefits for network troubleshooting, providing real-time network-wide visibility into the specific packets that were dropped as well the reason the packet was dropped. This visibility instantly reveals the root cause of drops and the impacted connections.

Packet discard monitoring complements sFlow's existing counter polling and packet sampling mechanisms and shares a common data model so that all three sources of data can be correlated.  For example, if packets are being discarded because of buffer exhaustion, the discard records don't necessarily tell the whole story. The discarded packets may represent mice flows that are victims of an elephant flow. Packet samples will reveal the traffic that isn't being dropped and provide a more complete picture. Counter data adds additional information such as CPU load, interface speed, link utilization, packet and discard rates that further completes the picture.

The following steps build the Host sFlow agent with drop monitoring on an Ubuntu 20 system (a Linux 5.4 kernel or newer is required):

git clone https://github.com/sflow/host-sflow
cd host-sflow
make FEATURES="HOST PCAP DROPMON"
sudo make install
sudo make schedule

Next, edit the /etc/hsflowd.conf file, in this example, directing sFlow to be sent to a collector at 192.168.1.242, enabling packet sampling on host adapter enp0s3, and enabling drop monitoring:

sflow {
  collector { ip=192.168.1.242 }
  pcap { dev = enp0s3 }
  dropmon { group = 1 start = on }
}

Start the agent:

sudu systemctl enable hsflowd
sudo systemctl start hsflowd

Build the latest version of sflowtool on the collector host (192.168.1.242):

git clone https://github.com/sflow/sflowtool
cd sflowtool
./boot.sh
./configure
make
sudo make install

Now run sflowtool to receive and decode the sFlow telemetry stream:

sflowtool

The following example shows the output for a discarded TCP packet:

startSample ----------------------
sampleType_tag 0:5
sampleType DISCARD
sampleSequenceNo 20
sourceId 0:1
dropEvents 0
inputPort 1
outputPort 0
discardCode 289
discardReason unknown_l4
discarded_flowBlock_tag 0:1
discarded_flowSampleType HEADER
discarded_headerProtocol 1
discarded_sampledPacketSize 54
discarded_strippedBytes 0
discarded_headerLen 54
discarded_headerBytes 00-00-00-00-00-00-00-00-00-00-00-00-08-00-45-00-00-28-00-00-40-00-40-06-3C-CE-7F-00-00-01-7F-00-00-01-04-05-04-39-00-00-00-00-14-51-E2-8A-50-14-00-00-B2-B4-00-00
discarded_dstMAC 000000000000
discarded_srcMAC 000000000000
discarded_IPSize 40
discarded_ip.tot_len 40
discarded_srcIP 127.0.0.1
discarded_dstIP 127.0.0.1
discarded_IPProtocol 6
discarded_IPTOS 0
discarded_IPTTL 64
discarded_IPID 0
discarded_TCPSrcPort 1029
discarded_TCPDstPort 1081
discarded_TCPFlags 20
endSample   ----------------------

The sflowtool -T option converts the discarded packet records into PCAP format so that they can be decoded by packet analysis tools such as Wireshark and tcpdump:

sflowtool -T | tshark -r -
   12  22.000000 192.168.1.242 → 192.168.1.87 TCP 78 65527 → 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=64 TSval=1324841769 TSecr=0 SACK_PERM=1

The article sFlow to JSON uses Python examples to demonstrate how sflowtool's ability to convert sFlow records into JSON can be used for further analysis.

SONiC

SONiC is part of the Open Compute Project (OCP), creating "an open source network operating system based on Linux that runs on switches from multiple vendors and ASICs." The latest SONiC.201911 release of the open source SONiC network operating system adds sFlow support.

SONiC: sFlow High Level Design

The diagram shows the elements of the implementation.

1. The open source Host sFlow agent running in the sFlow container monitors the Redis database (in the Database container) for sFlow related configuration changes.
2. The syncd container monitors the configuration database and pushes hardware settings (packet sampling) to the ASIC using the SAI (Switch Abstraction Inteface) driver (see SAI 1.5).
3. The ASIC driver hands sampled packet headers and associated metadata captured by the ASIC to user space via the Linux PSAMPLE netlink channel (see Linux 4.11 kernel extends packet sampling support).
4. The Host sFlow agent receives the PSAMPLE messages and forwards them to configured sFlow collector(s) as standard sFlow packet samples.
5. In addition, the Host sFlow agent streams telemetry (interface counters and host metrics gathered from the Redis database and Linux kernel) to the collector(s) as standard sFlow counter records.

The following CLI commands enable sFlow on all switch ports and send to sFlow collector 10.0.0.70:

sflow collector add 10.0.0.70
sflow polling-interval 30
sflow interface enable all
sflow enable

The open source sflowtool command line utility is a simple way of confirming that sFlow is being received at the collector host (10.0.0.70). The Docker sflow/sflowtool image provides a convenient method of running sflowtool:

docker run -p 6343:6343/udp sflow/sflowtool

If sFlow is arriving at the host, you will see the contents of the sFlow records printed in the console.

Running the Docker sflow/prometheus image exposes network-wide telemetry and flow data in the Prometheus export format so that it can be imported into a time series database:

docker run -p 8008:8008 -p 6343:6343/udp -d sflow/prometheus

See InfluxDB 2.0, Prometheus exporter and Flow metrics with Prometheus and Grafana for examples.

The Prometheus exporter is an application running on the sFlow-RT real-time analytics engine, which has the scaleability to handle large SONiC data center deployments. In addition to delivering metrics, sFlow-RT applications are available that address trouble-shooting, traffic engineering, DDoS mitigation, and security challenges of managing the large scale leaf and spine networks where SONiC is typically deployed.

Running sflowtool using Docker

The sflowtool command line utility is used to convert standard sFlow records into a variety of different formats. While there are a large number of native sFlow analysis applications, familiarity with sflowtool is worthwhile since it provides a simple way to verify receipt of sFlow, understand the contents of the sFlow telemetry stream, and build simple applications through custom scripting.

The sflow/sflowtool Docker image provides a simple way to run sflowtool. Run the following command to print the contents of sFlow packets:

$ docker run -p 6343:6343/udp sflow/sflowtool
startDatagram =================================
datagramSourceIP 10.0.0.111
datagramSize 144
unixSecondsUTC 1321922602
datagramVersion 5
agentSubId 0
agent 10.0.0.20
packetSequenceNo 3535127
sysUpTime 270660704
samplesInPacket 1
startSample ----------------------
sampleType_tag 0:2
sampleType COUNTERSSAMPLE
sampleSequenceNo 228282
sourceId 0:14
counterBlock_tag 0:1
ifIndex 14
networkType 6
ifSpeed 100000000
ifDirection 0
ifStatus 3
ifInOctets 4839078
ifInUcastPkts 15205
ifInMulticastPkts 0
ifInBroadcastPkts 4294967295
ifInDiscards 0
ifInErrors 0
ifInUnknownProtos 4294967295
ifOutOctets 149581962744
ifOutUcastPkts 158884229
ifOutMulticastPkts 4294967295
ifOutBroadcastPkts 4294967295
ifOutDiscards 101
ifOutErrors 0
ifPromiscuousMode 0
endSample   ----------------------
endDatagram   =================================

The -g option flattens the output so that it is more easily filtered using grep:

$ docker run -p 6343:6343/udp sflow/sflowtool -g | grep ifInOctets
2019-09-03T22:37:21+0000 10.0.0.231 0 3203000 0:6 0:2 0:1 ifInOctets 0
2019-09-03T22:37:23+0000 10.0.0.232 0 7242462 0:5 0:2 0:1 ifInOctets 53791415069
2019-09-03T22:37:23+0000 10.0.0.253 0 8178007 0:7 0:2 0:1 ifInOctets 31663763747
2019-09-03T22:37:23+0000 10.0.0.253 0 8178007 0:3 0:2 0:1 ifInOctets 1333603780050
2019-09-03T22:37:26+0000 10.0.0.253 0 8178008 0:1 0:2 0:1 ifInOctets 9116481296

The -L option prints out CSV records with the selected fields:

$ docker run -p 6343:6343/udp sflow/sflowtool -L agent,ifIndex,ifInOctets
10.0.0.253,23,432680126074
10.0.0.30,2,54056144719
10.0.0.253,21,3860664000830
10.0.0.253,3,1345269893416
10.0.0.253,2,1910370790761

The -J option prints out the decoded sFlow datagrams as JSON (with a blank line between each datagram):

$ docker run -p 6343:6343/udp sflow/sflowtool -J
{
 "datagramSourceIP":"172.17.0.1",
 "datagramSize":"1388",
 "unixSecondsUTC":"1567707952",
 "localtime":"2019-09-05T18:25:52+0000",
 "datagramVersion":"5",
 "agentSubId":"0",
 "agent":"10.0.0.253",
 "packetSequenceNo":"8254753",
 "sysUpTime":"165436226",
 "samplesInPacket":"8",
 "samples":[{
   "sampleType_tag":"0:1",
   "sampleType":"FLOWSAMPLE",
   "sampleSequenceNo":"2594544",
   "sourceId":"0:3",
   "meanSkipCount":"500",
   "samplePool":"1622164761",
   "dropEvents":"584479",
   "inputPort":"21",
   "outputPort":"3",
   "elements":[{
     "flowBlock_tag":"0:1",
     "flowSampleType":"HEADER",
     "headerProtocol":"1",
     "sampledPacketSize":"118",
     "strippedBytes":"4",
     "headerLen":"116",
...

The -j option formats the JSON output as a single line per datagram making the output easy to parse in scripts. For example, the following emerging.py script downloads the Emerging Threats compromised IP address database, parses the JSON records, checks to see if source and destination addresses can be found in the database, and prints out information on any matches:

#!/usr/bin/env python

from sys import stdin
from json import loads
from requests import get

blacklist = set()
r = get('https://rules.emergingthreats.net/blockrules/compromised-ips.txt')
for line in r.iter_lines():
  blacklist.add(line)

for line in stdin:
  datagram = loads(line)
  localtime = datagram["localtime"]
  samples = datagram["samples"]
  for sample in samples:
    sampleType = sample["sampleType"]
    elements = sample["elements"]
    if sampleType == "FLOWSAMPLE":
      for element in elements:
        tag = element["flowBlock_tag"]
        if tag == "0:1":
          try:
            src = element["srcIP"]
            dst = element["dstIP"]
            if src in blacklist or dst in blacklist:
              print "%s %s %s" % (localtime,src,dst)
          except KeyError:
            pass

Run the command:

docker run -p 6343:6343/udp sflow/sflowtool -j | ./emerging.py

These were just a few examples, see the sflowtool home page for additional information.

Forwarding using sFlow-RT describes how to set up and tear down sFlow streams using the sFlow-RT analytics engine. This is a simple way to direct a stream of sFlow to a desktop running sflowtool. For example, suppose sflowtool is running on host 10.0.0.30 and sFlow-RT is running on host 10.0.0.1, the following command would start a session:

curl -H "Content-Type:application/json" -X PUT --data '{"address":"10.0.0.30"}' \
http://10.0.0.1:8008/forwarding/sflowtool/json

and the following command would end the session:

curl -X DELETE http://10.0.0.1:8008/forwarding/sflowtool/json

Note: The sflow/sflow-rt Docker image is a convenient way to run sFlow-RT:

docker run -p 8008:8008 -p 6343:6343/udp sflow/sflow-rt

Secure forwarding of sFlow using ssh

Typically sFlow datagrams are sent unencrypted from agents embedded in switches and routers to a local collector/analyzer. Sending sFlow datagrams over the management VLAN or out of band management network generally provides adequate isolation and security within the site. Inter-site traffic within an organization is typically carried over a virtual private network (VPN) which encrypts the data and protects it from eavesdropping.

This article describes a simple method of carrying sFlow datagrams over an encrypted ssh connection which can be useful in situations where a VPN is not available, for example, sending sFlow to an analyzer in the public cloud, or to an external consultant.

The diagram shows the elements of the solution. A collector on the site receives sFlow datagrams from the network devices and uses the sflow_fwd.py script to convert the datagrams into line delimited hexadecimal strings that are sent over an ssh connection to another instance of sflow_fwd.py running on the analyzer that converts the hexadecimal strings back to sFlow datagrams.

The following sflow_fwd.py Python script accomplishes the task:

#!/usr/bin/python

import socket
import sys
import argparse

parser = argparse.ArgumentParser(description='Serialize/deserialize sFlow')
parser.add_argument('-c', '--collector', default='')
parser.add_argument('-s', '--server')
parser.add_argument('-p', '--port', type=int, default=6343)
args = parser.parse_args()

sock=socket.socket(socket.AF_INET,socket.SOCK_DGRAM)

if(args.server != None):
  while True:
    line = sys.stdin.readline()
    if not line:
      break
    buf = bytearray.fromhex(line[:-1])
    sock.sendto(buf, (args.server, args.port))
else: 
  sock.bind((args.collector,args.port))
  while True:
    buf = sock.recv(2048)
    if not buf:
      break
    print buf.encode('hex')
    sys.stdout.flush()

Create a user account on both the collector and analyzer machines, in this example the user is pp. Next copy the script to both machines.

If you log into the collector machine, following command will send sFlow to the analyzer machine:

./sflow_fwd.py | ssh pp@analyzer './sflow_fwd.py -s 127.0.0.1'

If you log into the analyzer machine, the following command will retrieve sFlow from the collector machine:

ssh pp@collector './sflow_fwd.py' | ./sflow_fwd.py -s 127.0.0.1

If a permanent connection is required, it is relatively straightforward to create a daemon using systemd. In this example, the service is being installed on the collector machine by performing the following steps:
First log into the collector generate an ssh key:

ssh-keygen

Next, install the key on the analyzer system:

ssh-copy-id pp@analyzer

Now create the systemd service file, /etc/systemd/system/sflow-tunnel.service:

[Unit]
Description=sFlow tunnel
After=network.target

[Service]
Type=simple
User=pp
ExecStart=/bin/sh -c "/home/pp/sflow_fwd.py | /usr/bin/ssh pp@analyzer './sflow_fwd.py -s 127.0.0.1'"
Restart=on-failure
RestartSec=30

[Install]
WantedBy=multi-user.target

Finally, use the systemctl command to enable and start the daemon:

sudo systemctl enable sflow-tunnel.service
sudo systemctl start sflow-tunnel.service

A simple way to confirm that sFlow is arriving on the analyzer machine is to use sflowtool.

There are numerous articles on this blog describing how the sFlow-RT analytics software can be used to integrate sFlow telemetry with popular metrics and SIEM (security information and event management) tools.

sFlow to JSON

The latest version of sflowtool can convert sFlow datagrams into JSON, making it easy to write scripts to process the standard sFlow telemetry streaming from devices in the network.

Download and compile the latest version of sflowtool:

git clone https://github.com/sflow/sflowtool.git
cd sflowtool/
./boot.sh 
./configure 
make
sudo make install

The -J option formats the JSON output to be human readable:

$ sflowtool -J
{
 "datagramSourceIP":"10.0.0.162",
 "datagramSize":"396",
 "unixSecondsUTC":"1544241239",
 "localtime":"2018-12-07T19:53:59-0800",
 "datagramVersion":"5",
 "agentSubId":"0",
 "agent":"10.0.0.231",
 "packetSequenceNo":"1068783",
 "sysUpTime":"1338417874",
 "samplesInPacket":"2",
 "samples":[
  {
   "sampleType_tag":"0:2",
   "sampleType":"COUNTERSSAMPLE",
   "sampleSequenceNo":"148239",
   "sourceId":"0:3",
   "elements":[
    {
     "counterBlock_tag":"0:1",
     "ifIndex":"3",
     "networkType":"6",
     "ifSpeed":"1000000000",
     "ifDirection":"1",
     "ifStatus":"3",
     "ifInOctets":"4162076356",
     "ifInUcastPkts":"16312256",
     "ifInMulticastPkts":"187789",
     "ifInBroadcastPkts":"2566",
     "ifInDiscards":"0",
     "ifInErrors":"0",
     "ifInUnknownProtos":"0",
     "ifOutOctets":"2115351089",
     "ifOutUcastPkts":"7087570",
     "ifOutMulticastPkts":"4453258",
     "ifOutBroadcastPkts":"6141715",
     "ifOutDiscards":"0",
     "ifOutErrors":"0",
     "ifPromiscuousMode":"0"
    },
    {
     "counterBlock_tag":"0:2",
     "dot3StatsAlignmentErrors":"0",
     "dot3StatsFCSErrors":"0",
     "dot3StatsSingleCollisionFrames":"0",
     "dot3StatsMultipleCollisionFrames":"0",
     "dot3StatsSQETestErrors":"0",
     "dot3StatsDeferredTransmissions":"0",
     "dot3StatsLateCollisions":"0",
     "dot3StatsExcessiveCollisions":"0",
     "dot3StatsInternalMacTransmitErrors":"0",
     "dot3StatsCarrierSenseErrors":"0",
     "dot3StatsFrameTooLongs":"0",
     "dot3StatsInternalMacReceiveErrors":"0",
     "dot3StatsSymbolErrors":"0"
    }
   ]
  },
  {
   "sampleType_tag":"0:1",
   "sampleType":"FLOWSAMPLE",
   "sampleSequenceNo":"11791",
   "sourceId":"0:3",
   "meanSkipCount":"2000",
   "samplePool":"34185160",
   "dropEvents":"0",
   "inputPort":"3",
   "outputPort":"10",
   "elements":[
    {
     "flowBlock_tag":"0:1",
     "flowSampleType":"HEADER",
     "headerProtocol":"1",
     "sampledPacketSize":"102",
     "strippedBytes":"0",
     "headerLen":"104",
     "headerBytes":"0C-AE-4E-98-0B-89-05-B6-D8-D9-A2-66-80-00-54-00-00-45-08-12-04-00-04-10-4A-FB-A0-00-00-BC-A0-00-00-EF-80-00-DE-B1-E7-26-00-20-75-04-B0-C5-00-00-00-00-96-01-20-00-00-00-00-00-01-11-21-31-41-51-61-71-81-91-A1-B1-C1-D1-E1-F1-02-12-22-32-42-52-62-72-82-92-A2-B2-C2-D2-E2-F2-03-13-23-33-43-53-63-73-1A-1D-4D-76-00-00",
     "dstMAC":"0cae4e980b89",
     "srcMAC":"05b6d8d9a266",
     "IPSize":"88",
     "ip.tot_len":"84",
     "srcIP":"10.0.0.203",
     "dstIP":"10.0.0.254",
     "IPProtocol":"1",
     "IPTOS":"0",
     "IPTTL":"64",
     "IPID":"8576",
     "ICMPType":"8",
     "ICMPCode":"0"
    },
    {
     "flowBlock_tag":"0:1001",
     "extendedType":"SWITCH",
     "in_vlan":"1",
     "in_priority":"0",
     "out_vlan":"1",
     "out_priority":"0"
    }
   ]
  }
 ]
}

The output shows the JSON representation of a single sFlow datagram containing one counter sample and one flow sample.

The -j option output formats the JSON output as a single line per datagram making the output easy to parse in scripts. For example, the following Python script, flow.py, runs sflowtool and parses the JSON output:

#!/usr/bin/env python

import subprocess
from json import loads

p = subprocess.Popen(
  ['/usr/local/bin/sflowtool','-j'],
  stdout=subprocess.PIPE,
  stderr=subprocess.STDOUT
)
lines = iter(p.stdout.readline,'')
for line in lines:
  datagram = loads(line)
  localtime = datagram["localtime"]
  samples = datagram["samples"]
  for sample in samples:
    sampleType = sample["sampleType"]
    elements = sample["elements"]
    if sampleType == "FLOWSAMPLE":
      for element in elements:
        tag = element["flowBlock_tag"]
        if tag == "0:1":
          try:
            src = element["srcIP"]
            dst = element["dstIP"]
            pktsize = element["sampledPacketSize"]
            print "%s %s %s %s" % (localtime,src,dst,pktsize)
          except KeyError:
            pass

Running the script prints flow records showing time, source, destination and number of bytes:

$ ./flow.py 
2018-12-07T20:53:06-0800 10.0.0.70 10.0.0.238 110
2018-12-07T20:53:06-0800 10.0.0.70 10.0.0.238 70
2018-12-07T20:53:06-0800 10.0.0.70 10.0.0.238 70
2018-12-07T20:53:06-0800 10.0.0.238 10.0.0.70 90

The script can easily be modified to add additional fields, push data into an SIEM tool (e.g. Logstash), push counter data into a time series database (e.g. InfluxDB), or perform additional analysis in Python. For example, the following script builds on the example, downloading the Emerging Threats compromised address list and logging any flows that match the list:

#!/usr/bin/env python

import subprocess
from json import loads
from requests import get

blacklist = set()
r = get('https://rules.emergingthreats.net/blockrules/compromised-ips.txt')
for line in r.iter_lines():
  blacklist.add(line)

p = subprocess.Popen(
  ['/usr/local/bin/sflowtool','-j'],
  stdout=subprocess.PIPE,
  stderr=subprocess.STDOUT
)
lines = iter(p.stdout.readline,'')
for line in lines:
  datagram = loads(line)
  localtime = datagram["localtime"]
  samples = datagram["samples"]
  for sample in samples:
    sampleType = sample["sampleType"]
    elements = sample["elements"]
    if sampleType == "FLOWSAMPLE":
      for element in elements:
        tag = element["flowBlock_tag"]
        if tag == "0:1":
          try:
            src = element["srcIP"]
            dst = element["dstIP"]
            if src in blacklist or dst in blacklist:
              print "%s %s %s" % (localtime,src,dst)
          except KeyError:
            pass

The open source Host sFlow agent provides a convenient means of experimenting with sFlow if you don't have access to network devices. The Host sFlow agent is also a simple way to gather real-time telemetry from public cloud virtual machine instances where access to the physical network infrastructure is not permitted.

Finally, for advanced sFlow analytics, try sFlow-RT, a real-time analytics engine that exposes a REST API.

Northbound Networks Zodiac GX

Mininet is widely used to emulate software defined networks (SDNs). Mininet flow analytics describes how standard sFlow telemetry, from Open vSwitch used by Mininet emulate the network, provides feedback to an SDN controller, allowing the controller to adapt the network to changing traffic, for example, to mitigate a distributed denial of service (DDoS) attack.

Northbound Networks Zodiac GX is an inexpensive open source software based switch that is ideal for experimenting with software defined networking (SDN) in a physical network setting. The small fanless package makes the switch an attractive option for desktop use. The Zodiac GX is also based on Open vSwitch, making it easy to take SDN control strategies developed on Mininet.

Enabling sFlow on the Zodiac GX is easy, navigate to the System>Startup page and add the following line to the end of the startup script (before the exit 0 line):

ovs-vsctl -- --id=@sflow create sflow agent=$OVS_BR target=$IP_CONTROLLER_1 sampling=100 polling=10 -- set bridge $OVS_BR sflow=@sflow

Reboot the switch for the changed to take effect.

Use sflowtool to verify that sFlow is arriving at the controller host and to examine the contents of the telemetry stream. Running sflowtool using Docker is a simple alternative to building the software from sources:

docker run --rm -p 8008:8008 -p 6343:6343/udp sflow/sflowtool

The text output from sflowtool can be piped into scripts to perform basic sFlow analysis.

A graphical sFlow analyzer performs the analysis tasks for you. The screen shot above shows sFlowTrend, a free sFlow analyzer that displays traffic trends. The software can be downloaded and installed or run using Docker:

docker run --rm -p 6343:6343/udp -p 8087:8087 -p 8443:8443 sflow/sflowtrend

The sFlowTrend charts update every minute. This is generally fast enough for human consumption, but real-time, up to the second, visibility is critical for SDN use cases.

The screen shot from Flow Trend shows an up to the second view of traffic. The spike in traffic is due to a 4K video being streamed from YouTube. The following command runs the software:

docker run --rm -p 6343:6343/udp -p 8008:8008 sflow/flow-trend

Flow Trend is an application running on the sFlow-RT real-time analytics platform.

Applications running on the sFlow-RT platform deliver real-time visibility to SDN, DevOps and Orchestration stacks, enabling new classes of performance aware application such as load balancing, DDoS mitigation, and workload placement.

RYU provides a framework that can be used to develop SDN applications in Python. For example, the following command runs the simple learning bridge application that ships with RYU:

docker run -it --rm -p 6633:6633 osrg/ryu ryu-manager --verbose ryu/ryu/app/simple_switch_13.py

As soon as the switch connects to the controller, you should see a flurry of events as the controller programs flows on the Zodiac GX switch.

Faucet is an SDN controller for production networks implemented using RYU. Before we can use Faucet, we need to gather basic OpenFlow information from the switch.

$ ssh -t admin@10.0.0.230 "sudo ovs-ofctl show ovslan"
admin@10.0.0.230's password: 
Password: 
OFPT_FEATURES_REPLY (xid=0x2): dpid:000044d1fa6291b2
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
 1(eth0.1): addr:44:d1:fa:62:91:b2
     config:     0
     state:      STP_FORWARD
     current:    1GB-FD AUTO_NEG
     speed: 1000 Mbps now, 0 Mbps max
 2(eth0.2): addr:44:d1:fa:62:91:b2
     config:     0
     state:      STP_FORWARD
     current:    1GB-FD AUTO_NEG
     speed: 1000 Mbps now, 0 Mbps max
 3(eth0.3): addr:44:d1:fa:62:91:b2
     config:     0
     state:      STP_FORWARD
     current:    1GB-FD AUTO_NEG
     speed: 1000 Mbps now, 0 Mbps max
 4(eth0.4): addr:44:d1:fa:62:91:b2
     config:     0
     state:      STP_FORWARD
     current:    1GB-FD AUTO_NEG
     speed: 1000 Mbps now, 0 Mbps max
 5(eth0.5): addr:44:d1:fa:62:91:b2
     config:     0
     state:      STP_FORWARD
     current:    1GB-FD AUTO_NEG
     speed: 1000 Mbps now, 0 Mbps max
 LOCAL(ovslan): addr:44:d1:fa:62:91:b2
     config:     0
     state:      0
     current:    1GB-FD AUTO_NEG
     speed: 1000 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
Connection to 10.0.0.230 closed.

Update August 30, 2018: The only piece of information needed to construct the faucet config file below is the switch dpid. The Zodiac GX uses the MAC address of the switch as the dpid, so you can simply read the MAC address printed on a label on the bottom of the switch.

Now create a directory called faucet that contains the initial Faucet configuration file, faucet.yaml:

vlans:
    office:
        vid: 100
        description: "office network"

dps:
    zodiac:
        dp_id: 0x000044d1fa6291b2
        hardware: "ZodiacGX"
        interfaces:
            1:
                name: "eth0.1"
                description: "port1"
                native_vlan: office
            2:
                name: "eth0.2"
                description: "port2"
                native_vlan: office
            3:
                name: "eth0.3"
                description: "port3"
                native_vlan: office
            4:
                name: "eth0.4"
                description: "port4"
                native_vlan: office
            5:
                name: "eth0.5"
                description: "port5"
                native_vlan: office
            0xfffffffe:
                name: "ovslan"
                description: "local"
                native_vlan: office

Now run Faucet:

docker run -it --rm -v $PWD/faucet/:/etc/faucet/ -v $PWD/faucet/:/var/log/faucet/ -p 6633:6653 -p 9302:9302 faucet/faucet

As soon as the switch connects to the controller, you should see events logged to the faucet.log file in the same directory as the faucet.yaml configuration file.

Faucet Documentation describes how to extend the configuration to include firewall, routing, segmentation, and network function virtualization (NFV) rules to the configuration.

The next step is integrating sFlow analytics with the controller. Writing Applications describes how write sFlow-RT applications use REST API and embedded JavaScript API. The document includes Python examples that could be embedded in RYU controller applications. Alternatively, sFlow-RT's embedded HTTP client can be used to push control actions to an SDN controller, see ONOS measurement based control for an example.

The sFlow telemetry stream contains detailed Open vSwitch performance metrics in addition to flow and interface counter data. The sFlow-RT analytics pipeline can be programmed to generate and push statistics to time series databases and dashboards, see Prometheus and Grafana and InfluxDB and Grafana.

Exporting events using syslog describes how sFlow-RT can be programmed to detect and report on traffic anomalies, sending events to Security Information and Event Management (SIEM) tools, using Splunk and Logstash as examples.

The sflow/sflow-rt Docker image provides a convenient means of developing and deploying sFlow-RT applications alongside the SDN controllers demonstrated in this article.

An important benefit of sFlow telemetry is that it decouples monitoring from the control plane. You are free to change SDN controllers, use distributed routing / switching protocols, move between network operating systems, or build your own control plane while maintaining the same level of visibility. Industry standard sFlow is widely supported by vendors, including: A10, Aerohive, ALUe, Allied Telesis, Arista, Aruba, Big Switch, Broadcom, Cisco, Cumulus, Dell, D-Link, Edge-Core, Extreme, F5, Fortinet, Huawei, IP Infusion, Juniper, Mellanox, Netgear, OpenSwitch, Pica8, Proxim, Quanta, SMC, ZTE, and ZyXEL.

Berkeley Packet Filter (BPF)

Linux bridge, macvlan, ipvlan, adapters discusses how industry standard sFlow technology, widely supported by data center switch vendors, has been extended to provide network visibility into the Linux data plane. This article explores how sFlow's lightweight packet sampling mechanism has been implemented on Linux network adapters.

Linux Socket Filtering aka Berkeley Packet Filter (BPF) describes the recently added prandom_u32() function that allows packets to be randomly sampled in the Linux kernel for efficient monitoring of production traffic.

Background: Enhancing Network Intrusion Detection With Integrated Sampling and Filtering, Jose M. Gonzalez and Vern Paxson, International Computer Science Institute Berkeley, discusses the motivation for adding random sampling BPF and the email thread [PATCH] filter: added BPF random opcode describes the Linux implementation and includes an interesting discussion of the motivation for the patch.

The following code shows how the open source Host sFlow agent implements random 1-in-256 packet sampling as a BPF program:

ld rand
mod #256
jneq #1, drop
ret #-1
drop: ret #0

A JIT for packet filters discusses the Linux Just In Time (JIT) compiler for BFP programs, delivering native machine code performance for compiled filters.

Minimizing cost of visibility describes why low overhead monitoring is an essential component for increasing efficiency in cloud infrastructure. The combination of BPF packet sampling with standard sFlow export provides a low overhead method of delivering real-time network visibility into large scale cloud infrastructure.

Podcast with Nick Buraglio and Brent Salisbury

"Have you seen sFlow options in your router configuration or flow collector? Are you looking for alternatives to SNMP or NetFlow? Have you been curious about the instrumentation of your new white box or virtual switch? Yes? Then you will probably enjoy learning more about sFlow!"

Non-Blocking #1: SFlow With Peter Phaal Of InMon And SFlow.Org is a discussion between Brent Salisbury (networkstatic.net), Nick Buraglio (forwardingplane.net), and Peter Phaal (blog.sflow.com).

Web sites and tools mentioned in the podcast:

1. sFlow.org
2. Devices that support sFlow
3. Software to analyze sFlow
4. sFlow.org mailing list
5. sFlow structures
6. blog.sflow.com (incorrectly referenced as blog.sflow.org in the podcast)
7. Host sFlow
8. sflowtool

The podcast touches on a number of topics that have been explored in greater detail on this blog. The topics are listed in roughly the order they are mentioned in the podcast:

1. Widespread support for sFlow among switch vendors
2. Disaggregated flow cache
3. ULOG
4. Push vs Pull
5. sFlow vs SNMP for interface counters
6. Broadcom ASIC table utilization metrics, DevOps, and SDN
7. Broadcom BroadView Instrumentation
8. Rapidly detecting large flows, sFlow vs. NetFlow/IPFIX
9. SDN and large flows
10. Probes
11. Packet headers
12. Network virtualization visibility demo
13. History of sFlow
14. Standards
15. Open vSwitch performance monitoring
16. Wireless
17. Prescriptive vs descriptive standards (sFlow / IPFIX)
18. RMON (4 groups)
19. Observability
20. Host sFlow distributed agent
21. Host sFlow data model
22. Multi-tenant traffic in virtualized network environments
23. Workload placement
24. SDN router using merchant silicon top of rack switch
25. White box Internet router PoC
26. Active Route Manager
27. Leaf and spine traffic engineering using segment routing and SDN
28. CORD: Open-source spine-leaf Fabric (demo from 2015 Open Networking Summit)
29. sflowtool
30. sflowtool for packet caputure
31. sflowtool with Wireshark

Custom events

Measuring Page Load Speed with Navigation Timing describes the standard instrumentation built into web browsers. This article will use navigation timing as an example to demonstrate how custom sFlow events augment standard sFlow instrumentation embedded in network devices, load balancers, hosts and web servers.

The JQuery script can be embedded in a web page to provide timing information:

$(window).load(function(){
 var samplingRate = 10;
 if(samplingRate !== 1 && Math.random() > (1/samplingRate)) return;

 setTimeout(function(){
   if(window.performance) {
     var t = window.performance.timing;
     var msg = {
       sampling_rate : samplingRate,
       t_url         : {type:"string",value:window.location.href},
       t_useragent   : {type:"string",value:navigator.userAgent},
       t_loadtime    : {type:"int32",value:t.loadEventEnd-t.navigationStart},
       t_connecttime : {type:"int32",value:t.responseEnd-t.requestStart} 
     };
     $.ajax({
       url:"/navtiming.php",
       method:"PUT",
       contentType:"application/json",
       data:JSON.stringify(msg) 
     });
    }
  }, 0);
});

The script supports random sampling. In this case a samplingRate of 10 means that, on average, 1-in-10 page hits will generate a measurement record. Measurement records are sent back to the server where the navtiming.php script acts as a gateway, augmenting the measurements and sending them as custom sFlow events.

<?php
$rawInput = file_get_contents("php://input");
$rec = json_decode($rawInput);
$rec->datasource = "navtime";
$rec->t_ip = array("type" => "ip", "value" => $_SERVER['REMOTE_ADDR']);

$msg=array("rtflow"=>$rec);
$sock = fsockopen("udp://localhost",36343,$errno,$errstr);
if(! $sock) { return; }
fwrite($sock, json_encode($msg));
fclose($sock);
?>

In this case the remote IP address associated with the client browser is added to the measurement before it is formatted as a JSON rtflow message and sent to the Host sFlow agent (hsflowd) running on the web server host. The Host sFlow agent encodes the data as an sFlow structure and sends it to the sFlow collector as part of the telemetry stream.

The following sflowtool output verifies that the metrics are being received at the sFlow Analyzer:

startSample ----------------------
sampleType_tag 4300:1003
sampleType RTFLOW
rtflow_datasource_name navtime
rtflow_sampling_rate 1
rtflow_sample_pool 0
rtflow t_url = (string) "http://10.0.0.84/index.html"
rtflow t_useragent = (string) "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36"
rtflow t_loadtime = (int32) 115
rtflow t_connecttime = (int32) 27
rtflow t_ip = (ip) 10.1.1.63
endSample   ----------------------

A more interesting way to consume the data is to use sFlow-RT. For example, the following REST API call programs the sFlow-RT analytics pipeline to create a metric that tracks average t_loadtime by URL:

curl -H "Content-Type:application/json" -X PUT --data '{keys:"t_url",value:"avg:t_loadtime",t:15}' http://localhost:8008/flow/urlloadtime/json

The following query can be used to retrieves the resulting metric value:

curl http://localhost:8008/metric/ALL/urlloadtime/json
[{
 "agent": "10.0.0.84",
 "dataSource": "navtime",
 "lastUpdate": 1807,
 "lastUpdateMax": 1807,
 "lastUpdateMin": 1807,
 "metricN": 1,
 "metricName": "urlloadtime",
 "metricValue": 11.8125,
 "topKeys": [{
  "key": "http://10.0.0.84/index.html",
  "lastUpdate": 1807,
  "value": 11.8125
 }]
}]

RESTflow describes the sFlow-RT REST API used to create flow definitions and access flow based metrics and Defining Flows provides reference material.

Installing mod-sflow provides a rich set of transaction and counter metrics from the Apache web server, including information on worker threads, see Thread pools.

Telemetry from Apache, Host sFlow and the custom events are all combined at the sFlow analyzer. For example, the following query pulls together the load average on the server, with Apache thread pool utilization and URL load times:

curl http://localhost:8008/metric/10.0.0.84/load_one,workers_utilization,urlloadtime/json
[
 {
  "agent": "10.0.0.84",
  "dataSource": "2.1",
  "lastUpdate": 23871,
  "lastUpdateMax": 23871,
  "lastUpdateMin": 23871,
  "metricN": 1,
  "metricName": "load_one",
  "metricValue": 0
 },
 {
  "agent": "10.0.0.84",
  "dataSource": "3.80",
  "lastUpdate": 4123,
  "lastUpdateMax": 4123,
  "lastUpdateMin": 4123,
  "metricN": 1,
  "metricName": "workers_utilization",
  "metricValue": 0.390625
 },
 {
  "agent": "10.0.0.84",
  "dataSource": "navtime",
  "lastUpdate": 8821,
  "lastUpdateMax": 8821,
  "lastUpdateMin": 8821,
  "metricN": 1,
  "metricName": "urlloadtime",
  "metricValue": 91.81072992715491,
  "topKeys": [{
   "key": "http://10.0.0.84/index.html",
   "lastUpdate": 8821,
   "value": 91.81072992715491
  }]
 }
]

The article, Cluster performance metrics, describes the metric API in more detail. Additional sFlow-RT APIs can be used to send data to a variety of DevOps tools, including: Ganglia, Graphite, InfluxDB and Grafana, Logstash, Splunk, cloud analytics services.

Finally, standard sFlow instrumentation is also widely implemented by physical and virtual network devices. Combining data from all these sources provides a comprehensive real-time view of applications and the compute, storage and networking resources that the applications depend on.

Custom metrics with Cumulus Linux

Cumulus Networks, sFlow and data center automation describes how Cumulus Linux is monitored using the open source Host sFlow agent that supports Linux, Windows, FreeBSD, Solaris, and AIX operating systems and KVM, Xen, XCP, XenServer, and Hyper-V hypervisors, delivering a standard set of performance metrics from switches, servers, hypervisors, virtual switches, and virtual machines.

Host sFlow version 1.28.3 adds support for Custom Metrics. This article demonstrates how the extensive set of standard sFlow measurements can be augmented using custom metrics.

Recent releases of Cumulus Linux simplify the task by making machine readable JSON a supported output in command line tools. For example, the cl-bgp tool can be used to dump BGP summary statistics:

cumulus@leaf1$ sudo cl-bgp summary show json
{ "router-id": "192.168.0.80", "as": 65080, "table-version": 5, "rib-count": 9, "rib-memory": 1080, "peer-count": 2, "peer-memory": 34240, "peer-group-count": 1, "peer-group-memory": 56, "peers": { "swp1": { "remote-as": 65082, "version": 4, "msgrcvd": 52082, "msgsent": 52084, "table-version": 0, "outq": 0, "inq": 0, "uptime": "05w1d04h", "prefix-received-count": 2, "prefix-advertised-count": 5, "state": "Established", "id-type": "interface" }, "swp2": { "remote-as": 65083, "version": 4, "msgrcvd": 52082, "msgsent": 52083, "table-version": 0, "outq": 0, "inq": 0, "uptime": "05w1d04h", "prefix-received-count": 2, "prefix-advertised-count": 5, "state": "Established", "id-type": "interface" } }, "total-peers": 2, "dynamic-peers": 0 }

The following Python script, bgp_sflow.py, invokes the command, parses the output, and posts a set of custom sFlow metrics:

#!/usr/bin/env python
import json
import socket
from subprocess import check_output

res = check_output(["/usr/bin/cl-bgp","summary","show","json"])
bgp = json.loads(res)
metrics = {
  "datasource":"bgp",
  "bgp-router-id"    : {"type":"string", "value":bgp["router-id"]},
  "bgp-as"           : {"type":"string", "value":str(bgp["as"])},
  "bgp-total-peers"  : {"type":"gauge32", "value":bgp["total-peers"]},
  "bgp-peer-count"   : {"type":"gauge32", "value":bgp["peer-count"]},
  "bgp-dynamic-peers": {"type":"gauge32", "value":bgp["dynamic-peers"]},
  "bgp-rib-memory"   : {"type":"gauge32", "value":bgp["rib-memory"]},
  "bgp-rib-count"    : {"type":"gauge32", "value":bgp["rib-count"]},
  "bgp-peer-memory"  : {"type":"gauge32", "value":bgp["peer-memory"]},
  "bgp-msgsent"      : {"type":"counter32", "value":sum(bgp["peers"][c]["msgsent"] for c in bgp["peers"])},
  "bgp-msgrcvd"      : {"type":"counter32", "value":sum(bgp["peers"][c]["msgrcvd"] for c in bgp["peers"])}
}
msg = {"rtmetric":metrics}
sock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
sock.sendto(json.dumps(msg),("127.0.0.1",36343))

Adding the following cron entry runs the script every minute:

* * * * * /home/cumulus/bgp_sflow.py > /dev/null 2>&1

The new metrics will now arrive at the sFlow collector. The following sflowtool output verifies that the metrics are being received:

startSample ----------------------
sampleType_tag 4300:1002
sampleType RTMETRIC
rtmetric_datasource_name bgp
rtmetric bgp-as = (string) "65080"
rtmetric bgp-rib-count = (gauge32) 9
rtmetric bgp-dynamic-peers = (gauge32) 0
rtmetric bgp-rib-memory = (gauge32) 1080
rtmetric bgp-peer-count = (gauge32) 2
rtmetric bgp-router-id = (string) "192.168.0.80"
rtmetric bgp-total-peers = (gauge32) 2
rtmetric bgp-msgrcvd = (counter32) 104648
rtmetric bgp-msgsent = (counter32) 104651
rtmetric bgp-peer-memory = (gauge32) 34240
endSample   ----------------------

A more interesting way to consume this data is using sFlow-RT. The diagram above shows a leaf and spine network built using CumuluxVX virtual machines that was used for a Network virtualization visibility demo. Installing the bgp_sflow.py script on each switch adds centralized visibility into fabric wide BGP statistics.

For example, the following sFlow-RT REST API command returns the total bgp messages sent and received summed across all switches:

$ curl http://10.0.0.86:8008/metric/ALL/sum:bgp-msgrcvd,sum:bgp-msgsent/json
[
 {
  "lastUpdateMax": 20498,
  "lastUpdateMin": 20359,
  "metricN": 4,
  "metricName": "sum:bgp-msgrcvd",
  "metricValue": 0.10000302901465385
 },
 {
  "lastUpdateMax": 20498,
  "lastUpdateMin": 20359,
  "metricN": 4,
  "metricName": "sum:bgp-msgsent",
  "metricValue": 0.10000302901465385
 }
]

The custom metrics are fully integrated with all the other sFlow metrics, for example, the following query returns the host_name, bgp-as and load_one metrics associated with bgp-router-id 192.168.0.80:

$ curl http://10.0.0.86:8008/metric/ALL/host_name,bgp-as,load_one/json?bgp-router-id=192.168.0.80
[
 {
  "agent": "10.0.0.80",
  "lastUpdate": 12194,
  "lastUpdateMax": 12194,
  "lastUpdateMin": 12194,
  "metricN": 1,
  "metricName": "host_name",
  "metricValue": "leaf1"
 },
 {
  "agent": "10.0.0.80",
  "dataSource": "bgp",
  "lastUpdate": 22232,
  "lastUpdateMax": 22232,
  "lastUpdateMin": 22232,
  "metricN": 1,
  "metricName": "bgp-as",
  "metricValue": "65080"
 },
 {
  "agent": "10.0.0.80",
  "lastUpdate": 12194,
  "lastUpdateMax": 12194,
  "lastUpdateMin": 12194,
  "metricN": 1,
  "metricName": "load_one",
  "metricValue": 0
 }
]

The article, Cluster performance metrics, describes the metric API in more detail. Additional sFlow-RT APIs can be used to send data to a variety of DevOps tools, including: Ganglia, Graphite, InfluxDB and Grafana, Logstash, Splunk, cloud analytics services.

Software, documentation, applications, and community support is available on sFlow-RT.com. For example, the sFlow-RT Fabric View application shown in the screen capture calculates and displays fabric wide traffic analytics.

Broadcom ASIC table utilization metrics, DevOps, and SDN

Figure 1: Two-Level Folded CLOS Network Topology Example

Figure 1 from the Broadcom white paper, Engineered Elephant Flows for Boosting Application Performance in Large-Scale CLOS Networks, shows a data center leaf and spine topology. Leaf and spine networks are seeing rapid adoption since they provide the scaleability needed to cost effectively deliver the low latency, high bandwidth interconnect for cloud, big data, and high performance computing workloads.

Broadcom Trident ASICs are popular in white box, brite-box and branded data center switches from a wide range of vendors, including: Accton, Agema, Alcatel-Lucent, Arista, Cisco, Dell, Edge-Core, Extreme, Hewlett-Packard, IBM, Juniper, Penguin Computing, and Quanta.

Figure 2: OF-DPA Programming Pipeline for ECMP

Figure 2 shows the packet processing pipeline of a Broadcom ASIC. The pipeline consists of a number of linked hardware tables providing bridging, routing, access control list (ACL), and ECMP forwarding group functions. Operations teams need to be able to proactively monitor table utilizations in order to avoid performance problems associated with table exhaustion.

Broadcom's recently released sFlow specification, sFlow Broadcom Switch ASIC Table Utilization Structures, leverages the industry standard sFlow protocol to offer scaleable, multi-vendor, network wide visibility into the utilization of these hardware tables.

Support for the new extension has just been added to the open source Host sFlow agent, which runs on Cumulus Linux, a Debian based Linux distribution that supports open switch hardware from Agema, Dell, Edge-Core, Penguin Computing, Quanta. Hewlett-Packard recently announced that they will soon be selling a new line of open network switches built by Accton Technologies and supporting Cumulus Linux.

The speed with which this new features can be delivered on hardware from the wide range of vendors supporting Cumulus Linux is a powerful illustration of the power of open networking. While support for the Broadcom ASIC table extension has been checking into the Host sFlow trunk it hasn't yet made it into the Cumulus Networks binary repositories. However, Cumulus Linux is an open platform, so users are free to download sources, compile and install the latest software version direct from SourceForge.

The following output from the open source sflowtool command line utility shows the raw table measurements (this is in addition to the extensive set of sFlow measurements already exported via sFlow on Cumulus Linux):

bcm_asic_host_entries 4
bcm_host_entries_max 8192
bcm_ipv4_entries 0
bcm_ipv4_entries_max 0
bcm_ipv6_entries 0
bcm_ipv6_entries_max 0
bcm_ipv4_ipv6_entries 9
bcm_ipv4_ipv6_entries_max 16284
bcm_long_ipv6_entries 3
bcm_long_ipv6_entries_max 256
bcm_total_routes 10
bcm_total_routes_max 32768
bcm_ecmp_nexthops 0
bcm_ecmp_nexthops_max 2016
bcm_mac_entries 3
bcm_mac_entries_max 32768
bcm_ipv4_neighbors 4
bcm_ipv6_neighbors 0
bcm_ipv4_routes 0
bcm_ipv6_routes 0
bcm_acl_ingress_entries 842
bcm_acl_ingress_entries_max 4096
bcm_acl_ingress_counters 68
bcm_acl_ingress_counters_max 4096
bcm_acl_ingress_meters 18
bcm_acl_ingress_meters_max 8192
bcm_acl_ingress_slices 3
bcm_acl_ingress_slices_max 8
bcm_acl_egress_entries 36
bcm_acl_egress_entries_max 512
bcm_acl_egress_counters 36
bcm_acl_egress_counters_max 1024
bcm_acl_egress_meters 18
bcm_acl_egress_meters_max 512
bcm_acl_egress_slices 2
bcm_acl_egress_slices_max 2

The sflowtool output is useful for troubleshooting and is easy to parse with scripts.

DevOps

The diagram shows how the sFlow-RT analytics engine is used to deliver metrics and events to cloud based and on-site DevOps tools, see: Cloud analytics,  InfluxDB and Grafana, Cloud Analytics, Metric export to Graphite, and Exporting events using syslog.

For example, the following sFlow-RT application simplifies monitoring of the leaf and spine network by combining measurements from all the switches, identifying the switch with the maximum utilization of each table, pushing the summaries to operations dashboard every 15 seconds, and sending syslog events immediately when any table exceeds 80% utilization:

var network_wide_metrics = [
  'max:bcm_host_utilization',
  'max:bcm_mac_utilization',
  'max:bcm_ipv4_ipv6_utilization',
  'max:bcm_total_routes_utilization',
  'max:bcm_ecmp_nexthops_utilization',
  'max:bcm_acl_ingress_utilization',
  'max:bcm_acl_ingress_meters_utilization',
  'max:bcm_acl_ingress_counters_utilization',
  'max:bcm_acl_egress_utilization',
  'max:bcm_acl_egress_meters_utilization',
  'max:bcm_acl_egress_counters_utilization'
];

var max_utilization = 80;

setIntervalHandler(function() {
  var vals = metric('ALL',network_wide_metrics);
  var graphite_metrics = {};
  for each (var val in vals) {
    if(!val.hasOwnProperty('metricValue')) continue;

    // generate syslog events for over utilized tables
    if(val.metricValue >= max_utilization) {
       var event = {
         "asic_table":val.metricName,
         "utilization":val.metricValue,
         "switchIP":val.agent
       };
       try {
         syslog(
           '10.0.0.1', // syslog collector: splunk>, logstash, etc.
           514,        // syslog port
           16,         // facility = local0
           5,          // severity = notice
           event
        );
      } catch(e) { logWarning("syslog() failed " + e); }
    }

    // add metric to graphite set
    graphite_metrics["network.podA."+val.metricName] = val.metricValue;
  }

  // sent metrics to graphite
  try {
    graphite(
      '10.0.0.151',  // graphite server
      2003,          // graphite carbon UDP port
      graphite_metrics
    );
  } catch(e) { logWarning("graphite() failed " + e); }
},15);

The following screen capture shows the graphs starting to appear in Graphite:

Real-time traffic analytics

The table utilization metrics are only a part of the visibility that sFlow provides into the performance of a leaf and spine network.

A leaf and spine fabric is challenging to monitor. The fabric spreads traffic across all the switches and links in order to maximize bandwidth. Unlike traditional hierarchical network designs, where a small number of links can be monitored to provide visibility, a leaf and spine network has no special links or switches where running CLI commands or attaching a probe would provide visibility. Even if it were possible to attach probes, the effective bandwidth of a leaf and spine network can be as high as a Petabit/second, well beyond the capabilities of current generation monitoring tools.

Scaleable traffic measurement is possible because Broadcom ASICs implement hardware support for sFlow monitoring, providing cost effective, line rate visibility that is build into the switches and scales to all port speeds (1G, 10G, 25G, 40G, 50G, 100G, ...) and the high port counts found in large leaf and spine networks.

The 2 minute video provides an overview of some of the performance challenges with leaf and spine fabrics and demonstrates Fabric View - a monitoring solution that leverages industry standard sFlow instrumentation in commodity data center switches to provide real-time visibility into fabric performance. Fabric visibility with Cumulus Linux describes how to set up Fabric View to monitor a Cumulus Linux leaf and spine network.

SDN

Real-time network analytics are a fundamental driver for a number of important SDN use cases, allowing the SDN controller to rapidly detect changes in traffic and respond by applying active controls. SDN fabric controller for commodity data center switches describes how control of the ACL table is the key feature needed to to build scaleable SDN solutions.

REST API for Cumulus Linux ACLs describes open source software to allow an SDN controller to centrally manage the ACL tables on a large scale network of switches running Cumulus Linux.

The ability to install software on the switches is transformative, allowing third party developers and network operators transparent access to the full capabilities of the switch and build solutions that efficiently handle automation challenges.

A number of SDN use cases have been demonstrated that build on Cumulus Linux to leverage the real-time visibility and control capabilities of the switch ASIC:

• DDoS flood attack mitigation
• Large "Elephant" flow marking
• Large flow load balancing

Visit the sFlow.com web site to learn more about SDN control of leaf and spine networks.

Finally, the SDN use cases make extensive use of the ACL table and so this brings us full circle to the importance of the Broadcom sFlow extension providing visibility into the utilization of table resources.