Telemetry, analytics, and control with sFlow® standard
switch(config)# sflow collector 10.0.0.50 switch(config)# sflow sampling 4096 switch(config)# sflow polling 20 switch(config)# sflow enableA previous posting discussed the selection of sampling rates. Additional information can be found in the OpenSwitch sFlow User Guide.
How can sFlow provide real-time visibility at scale and consume so few resources? Shrink ray describes how advanced statistical techniques are used to select and analyze measurements that capture the essential features of network and system performance. A statistical approach yields fast, accurate answers, while minimizing the resources required to measure, transport and analyze the data.The sFlow-RT analytics platform was selected as an example because of the overlap in capabilities with the Cisco Tetration analytics platform. However, sFlow is non-proprietary and there are many other open source and commercial sFlow analytics solutions listed on sFlow.org.
table sflow_ing_take_sample {
/* take_sample > MAX_VAL_31 and valid sflow_session_id => take the sample */
reads {
ingress_metadata.sflow_take_sample : ternary;
sflow_metadata.sflow_session_id : exact;
}
actions {
nop;
sflow_ing_pkt_to_cpu;
}
}
Network visibility is one of the major use cases for P4 based switches. Improving Network Monitoring and Management with Programmable Data Planes describes how P4 can be used to collect information about latency and queueing in the switch forwarding pipeline.
An edge router is a very pricey box indeed, often costing anywhere from $100,000 to $200,000 per 100 Gb/sec port, depending on features in the router and not including optical cables that are also terribly expensive. Moreover, these routers might only be able to cram 80 ports into a half rack or full rack of space. The 7500R universal spine and 7280R universal leaf switches cost on the order of $3,000 per 100 Gb/sec port, and they are considerably denser and less expensive. - Leaving Fixed Function Switches Behind For Universal LeafsBroadcom Jericho ASICs are currently available in Arista 7500R/7280R routers and in Cisco NCS 5000 series routers. Expect further disruption to the router market when white box versions of the 1U router hardware enter the market.
There was general enthusiasm for Broadcom Jericho based routers in a recent discussion on the North American Network Operators' Group (NANOG) mailing list, Arista Routing Solutions, so merchant silicon based routers should be expected to sell well.The Broadcom Jericho ASICs also include hardware instrumentation to support industry standard sFlow traffic monitoring and streaming telemetry. For example, the following commands enable sFlow on all ports on an Arista router:
sflow source-interface Management1 sflow destination 170.1.1.11 sflow polling-interval 30 sflow sample 65535 sflow runSee EOS System Configuration Guide for details.
RP/0/RP0/CPU0:router#config RP/0/RP0/CPU0:router(config)#flow exporter-map exp1 RP/0/RP0/CPU0:router(config-fem)#version v9 RP/0/RP0/CPU0:router(config-fem-ver)#options interface-table timeout 300 RP/0/RP0/CPU0:router(config-fem-ver)#options sampler-table timeout 300 RP/0/RP0/CPU0:router(config-fem-ver)#template data timeout 300 RP/0/RP0/CPU0:router(config-fem-ver)#template options timeout 300 RP/0/RP0/CPU0:router(config-fem-ver)#exit RP/0/RP0/CPU0:router(config-fem)#transport udp 12515 RP/0/RP0/CPU0:router(config-fem)#source Loopback0 RP/0/RP0/CPU0:router(config-fem)#destination 170.1.1.11 RP/0/RP0/CPU0:router(config-fmm)#exit RP/0/RP0/CPU0:router(config)#flow monitor-map MPLS-IPv6-fmm RP/0/RP0/CPU0:router(config-fmm)#record mpls ipv6-fields labels 3 RP/0/RP0/CPU0:router(config-fmm)#exporter exp1 RP/0/RP0/CPU0:router(config-fmm)#cache entries 10000 RP/0/RP0/CPU0:router(config-fmm)#cache permanent RP/0/RP0/CPU0:router(config-fmm)#exit RP/0/RP0/CPU0:router(config)#sampler-map FSM RP/0/RP0/CPU0:router(config-sm)#random 1 out-of 65535 RP/0/RP0/CPU0:router(config-sm)# exitAnd further commands are needed to enable monitoring on each interface (and there can be a large number of interfaces given the high port density of these routers):
RP/0/RP0/CPU0:router(config)#interface HundredGigE 0/3/0/0 RP/0/RP0/CPU0:router(config-if)#flow mpls monitor MPLS-IPv6-fmm sampler FSM ingressSee Netflow Configuration Guide for Cisco NCS 5500 Series Routers, IOS XR Release 6.0.x for configuration details and limitations.
{
"Name": "Test",
"Metadata": {
"Version": 25,
"Description": "This is a sample policy",
"Comment": "This is the first draft",
"Identifier": "data that may be sent by the encoder to the mgmt stn"
},
"CollectionGroups": {
"FirstGroup": {
"Period": 30,
"Paths": [
"RootOper.InfraStatistics.Interface(*).Latest.GenericCounters"
]
}
}
}
Copy the policy file to router:$ scp Test.policy cisco@170.1.1.1:/telemetry/policiesFinally, configure the JSON encoder:
Router# configure Router(config)#telemetry encoder json Router(config-telemetry-json)#policy group FirstGroup Router(config-policy-group)#policy Test Router(config-policy-group)#destination ipv4 170.1.1.11 port 5555 Router(config-policy-group)#commitSee Cisco IOS XR Telemetry Configuration Guide for details.
IPVlan will require routes to be distributed to each endpoint. The driver only builds the Ipvlan L3 mode port and attaches the container to the interface. Route distribution throughout a cluster is beyond the initial implementation of this single host scoped driver. In L3 mode, the Docker host is very similar to a router starting new networks in the container. They are on networks that the upstream network will not know about without route distribution.Cumulus Networks has been working to simplify routing in the ECMP leaf and spine networks and the white paper Routing on the Host: An Introduction shows how the routing configuration used on Cumulus Linux can be extended to the hosts.
interface swp1 ipv6 nd ra-interval 10 link-detect ! interface swp2 ipv6 nd ra-interval 10 link-detect ! interface swp3 ipv6 nd ra-interval 10 link-detect ! router bgp 65130 bgp router-id 192.168.0.130 bgp bestpath as-path multipath-relax neighbor EBGP peer-group neighbor EBGP remote-as external neighbor swp1 interface peer-group EBGP neighbor swp1 capability extended-nexthop neighbor swp2 interface peer-group EBGP neighbor swp2 capability extended-nexthop neighbor swp3 interface peer-group EBGP neighbor swp3 capability extended-nexthopAuto-configured IPv6 link local addresses dramatically simplify the configuration of equal cost multi-path (ECMP) routing, eliminating the need to assign IP addresses and subnets to the routed switch ports. The simplified configuration is easy to template, all the switches have a very similar configuration, and an orchestration tool like Puppet, Chef, Ansible, Salt, etc. can be used to automate the process of configuring the switches.
Why the strange Ethernet interface names (e.g. enp0s3 instead of eth0)? They are the result of the predictable network interface names mechanism that is the default in Ubuntu 16.04. I can't say I am a fan, predictable interface names are difficult to read, push the problem of device naming up the stack, and make it difficult to write portable orchestration scripts.Quagga is used for BGP routing on the Cumulus Linux switches and on the Ubuntu hosts. The host BGP configurations are virtually identical to the switch configurations, but with an added redistribute stanza to automatically advertise locally attached addresses and subnets:
redistribute connected route-map IPVLANThe IPVLAN route-map is used to control the routes that are advertised, limiting them to the range of addresses that have been allocated to the IPVLAN orchestration system. Route filtering is an example of the flexibility that BGP brings as a routing protocol: top of rack switches can filter routes advertised by the hosts to protect the fabric from misconfigured hosts, and hosts can be configured to selectively advertise routes.
user@host1:~$ sudo ip address add 172.16.134.1/24 dev enp0s8And a route immediately appears on host2:
user@host2:~$ ip route
default via 10.0.0.254 dev enp0s3 onlink
10.0.0.0/24 dev enp0s3 proto kernel scope link src 10.0.0.135
172.16.134.0/24 via 169.254.0.1 dev enp0s8 proto zebra metric 20 onlink
Add a network to host2:
user@host2:~$ sudo ip address add 172.16.135.1/24 dev enp0s8And it appears on host1:
cumulus@host1:~$ ip route
default via 10.0.0.254 dev enp0s3 onlink
10.0.0.0/24 dev enp0s3 proto kernel scope link src 10.0.0.134
172.16.134.0/24 dev enp0s8 proto kernel scope link src 172.16.134.1
172.16.135.0/24 via 169.254.0.1 dev enp0s8 proto zebra metric 20 onlink
Connectivity across the leaf and spine fabric can be verified with a ping test:user@host2:~$ ping 172.16.134.1 PING 172.16.134.1 (172.16.134.1) 56(84) bytes of data. 64 bytes from 172.16.134.1: icmp_seq=1 ttl=61 time=2.60 ms 64 bytes from 172.16.134.1: icmp_seq=2 ttl=61 time=2.59 msUbuntu 16.04 was selected as the host operating system since it has built-in IPVLAN support. Docker Experimental Features distribution, which includes the IPVLAN Docker networking plugin, is installed on the two hosts.
user@host1:~$ docker network create -d ipvlan --subnet=172.16.134.0/24 \ -o parent=enp0s8 -o ipvlan_mode=l3 ipv3Note: Route(s) to the IPVLAN network would be automatically distributed if the above command had an option to attach the subnets to the parent interface.
user@host1:~$ docker run --net=ipv3 -it --rm alpine /bin/sh
/ # ifconfig
eth0 Link encap:Ethernet HWaddr 08:00:27:70:FA:B5
inet addr:172.16.134.2 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe70:fab5%32582/64 Scope:Link
UP BROADCAST RUNNING NOARP MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:2 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1%32582/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
/ # ip route
default dev eth0
172.16.134.0/24 dev eth0 src 172.16.134.2
Connectivity between containers attached to the same IPVLAN can be verified by starting a second container and performing a ping test:
user@host1:~$ docker run --net=ipv3 -it --rm alpine /bin/sh / # ping 172.16.134.2 PING 172.16.134.2 (172.16.134.2): 56 data bytes 64 bytes from 172.16.134.2: seq=0 ttl=64 time=0.085 ms 64 bytes from 172.16.134.2: seq=1 ttl=64 time=0.077 msUnfortunately, connecting containers to the IPVLAN network interferes with the auto-configured IPv6 link local BGP connection between host1 and leaf1, which results in host1 being disconnected from the leaf and spine fabric, and losing connectivity to host2. Assigning static IPv4 addresses to leaf1 and host1 complicates the configuration but solves the problem. For example, here is the Quagga BGP configuration on host1:
router bgp 65134 bgp router-id 192.168.0.134 redistribute connected route-map NON-MGMT neighbor 192.168.1.130 remote-as 65130Now it is possible to connect to the containers remotely from host2:
user@host2:~$ ping 172.16.134.2 PING 172.16.134.2 (172.16.134.2) 56(84) bytes of data. 64 bytes from 172.16.134.2: icmp_seq=1 ttl=61 time=2.72 ms 64 bytes from 172.16.134.2: icmp_seq=2 ttl=61 time=2.79 msNow that we have end to end connectivity, how can we monitor traffic?
FROM centos:centos6 RUN yum install -y wget RUN yum install -y java-1.7.0-openjdk RUN wget http://www.inmon.com/products/sFlow-RT/sflow-rt.tar.gz RUN tar -xzf sflow-rt.tar.gz EXPOSE 8008 6343/udp CMD ./sflow-rt/start.shUpdate July 22, 2016: sFlow-RT can now be run from Docker Hub, see sflow/sflow-rt for instructions.
#!/usr/bin/env python
import requests
import json
rt = 'http://127.0.0.1:8008'
flow = {'keys':'node:inputifindex,ipsource,ipdestination,ipttl','value':'bytes'}
requests.put(rt+'/flow/elephant/json',data=json.dumps(flow))
threshold = {'metric':'elephant','value':100000000/8,'byFlow':True,'timeout':600}
requests.put(rt+'/threshold/elephant/json',data=json.dumps(threshold))
eventurl = rt+'/events/json?thresholdID=elephant&maxEvents=10&timeout=60'
eventID = -1
while 1 == 1:
r = requests.get(eventurl + "&eventID=" + str(eventID))
if r.status_code != 200: break
events = r.json()
if len(events) == 0: continue
eventID = events[0]["eventID"]
events.reverse()
for e in events:
print e['flowKey']
Running the script and generating a large flow from host1 (172.16.135.1) to an IPVLAN connected container on host2 (172.16.134.2) gives the following output:$ ./elephant.py spine2,172.16.135.1,172.16.134.2,62 leaf1,172.16.135.1,172.16.134.2,61 host2,172.16.135.1,172.16.134.2,64 leaf2,172.16.135.1,172.16.134.2,63 host1,172.16.135.1,172.16.134.2,61The last field is the IP TTL (time to live). Ordering the events by TTL shows the path across the network (since the TTL is decremented by each switch along the path):
| sFlow | OpenConfig Telemetry | SNMP | |
|---|---|---|---|
| Organization | sFlow.org | OpenConfig.net | IETF |
| Users | General Purpose | Large Service Providers | General Purpose |
| Scope | Data Plane, Control Plane | Management Plane, Control Plane | Control Plane |
| Vendor Support | 40+ (see sFlow.org) | 1 (Cisco IOS XR ) | Near universal |
| Models | structure definitions | YANG models | Management Information Base |
| Encoding | XDR (RFC 4506) | protobufs, JSON, NetConf | ASN.1 |
| Transport | UDP | UDP, HTTP | UDP |
| Mode | Push | Push | Pull |
The article, Management, control and data planes in network devices and systems, by Ivan Pepelnjak, provides background on data, control, and management plane terminology.Gathering data plane telemetry requires hardware support and merchant silicon vendors (Broadcom, Cavium/XPliant, Intel/Fulcrum, Marvell etc.) include sFlow instrumentation in their switching/routing ASICs. Embedded hardware support allows sFlow to efficiently stream standardized data plane telemetry from all devices in a large multi-vendor network.
sudo apt-get update sudo apt-get install libpcap-dev git clone https://github.com/sflow/host-sflow sudo make installThe resulting Host sFlow binary is extremely small (only 163,300 bytes in this case):
pi@raspberrypi:~ $ ls -l /usr/sbin/hsflowd -rwx------ 1 root root 163300 Jun 1 17:18 /usr/sbin/hsflowdNext, specify /etc/hsflowd.conf file for the device:
sflow {
agent = eth0
agent.cidr=::/0
DNSSD = on
DNSSD_domain = .sf.inmon.com
jsonPort = 36343
pcap { dev = eth0 }
}
There are a number of important points to note about this configuration:sudo /etc/init.d/hsflowd startNow add an entry to the sf.inmon.com.zone file on the DNS server:
_sflow._udp 30 SRV 0 0 6343 collector.sf.inmon.com.In this case, the SRV record specifies that sFlow records should be sent via UDP to collector.sf.inmon.com on port 6343. The TTL is set to 30 seconds so that agents will pick up any changes within 30 seconds. A larger TTL should be used to improve scaleability if there are large numbers of devices.
#!/usr/bin/env python
import json
import socket
tempC = int(open('/sys/class/thermal/thermal_zone0/temp').read()) / 1e3
msg = {
"rtmetric": {
"datasource": "sensors",
"tempC": { "type": "gaugeFloat", "value": tempC }
}
}
sock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
sock.sendto(json.dumps(msg),("127.0.0.1",36343))
The following crontab entry runs the script every minute:* * * * * /home/pi/temp.pyThe Host sFlow agent will automatically pick up the configuration via a DNS request, start making measurements, which are immediately send in standard sFlow UDP datagrams to the designated sFlow collector collector.sf.inmon.com. sFlow's immediate transmission of measurements minimizes the memory requirements on the agent (since data doesn't have to be stored for later retrieval) and minimizes the latency before measurements are accessible on the collector (and can be acted on).
pi@raspberrypi:~ $ curl http://localhost:8008/metric/2001:470:67:27d:d811:aa7e:9e54:30e9/json
{
"2.1.bytes_in": 3211.5520193372945,
"2.1.bytes_out": 462.2822036458858,
"2.1.bytes_read": 0,
"2.1.bytes_written": 4537.818511431161,
"2.1.contexts": 7006.546480008057,
"2.1.cpu_guest": 0,
"2.1.cpu_guest_nice": 0,
"2.1.cpu_idle": 99.17638114546376,
"2.1.cpu_intr": 0,
"2.1.cpu_nice": 0,
"2.1.cpu_num": 4,
"2.1.cpu_sintr": 0.025342118601115054,
"2.1.cpu_speed": 0,
"2.1.cpu_steal": 0,
"2.1.cpu_system": 0.456158134820071,
"2.1.cpu_user": 0.3294475418144957,
"2.1.cpu_utilization": 0.8236188545362393,
"2.1.cpu_wio": 0.012671059300557527,
"2.1.disk_free": 24435570688,
"2.1.disk_total": 29627484160,
"2.1.disk_utilization": 17.523977160453796,
"2.1.drops_in": 0,
"2.1.drops_out": 0,
"2.1.errs_in": 0,
"2.1.errs_out": 0,
"2.1.host_name": "raspberrypi",
"2.1.icmp_inaddrmaskreps": 0,
"2.1.icmp_inaddrmasks": 0,
"2.1.icmp_indestunreachs": 0,
"2.1.icmp_inechoreps": 0,
"2.1.icmp_inechos": 0,
"2.1.icmp_inerrors": 0,
"2.1.icmp_inmsgs": 0,
"2.1.icmp_inparamprobs": 0,
"2.1.icmp_inredirects": 0,
"2.1.icmp_insrcquenchs": 0,
"2.1.icmp_intimeexcds": 0,
"2.1.icmp_intimestamps": 0,
"2.1.icmp_outaddrmaskreps": 0,
"2.1.icmp_outaddrmasks": 0,
"2.1.icmp_outdestunreachs": 0,
"2.1.icmp_outechoreps": 0,
"2.1.icmp_outechos": 0,
"2.1.icmp_outerrors": 0,
"2.1.icmp_outmsgs": 0,
"2.1.icmp_outparamprobs": 0,
"2.1.icmp_outredirects": 0,
"2.1.icmp_outsrcquenchs": 0,
"2.1.icmp_outtimeexcds": 0,
"2.1.icmp_outtimestampreps": 0,
"2.1.icmp_outtimestamps": 0,
"2.1.interrupts": 4438.56380300131,
"2.1.ip_defaultttl": 64,
"2.1.ip_forwarding": 2,
"2.1.ip_forwdatagrams": 0,
"2.1.ip_fragcreates": 0,
"2.1.ip_fragfails": 0,
"2.1.ip_fragoks": 0,
"2.1.ip_inaddrerrors": 0,
"2.1.ip_indelivers": 9.165072011280088,
"2.1.ip_indiscards": 0,
"2.1.ip_inhdrerrors": 0,
"2.1.ip_inreceives": 9.215429549803606,
"2.1.ip_inunknownprotos": 0,
"2.1.ip_outdiscards": 0,
"2.1.ip_outnoroutes": 0,
"2.1.ip_outrequests": 2.1653741565112297,
"2.1.ip_reasmfails": 0,
"2.1.ip_reasmoks": 0,
"2.1.ip_reasmreqds": 0,
"2.1.ip_reasmtimeout": 0,
"2.1.load_fifteen": 0.05,
"2.1.load_fifteen_per_cpu": 0.0125,
"2.1.load_five": 0.02,
"2.1.load_five_per_cpu": 0.005,
"2.1.load_one": 0,
"2.1.load_one_per_cpu": 0,
"2.1.machine_type": "arm",
"2.1.mem_buffers": 52133888,
"2.1.mem_cached": 383287296,
"2.1.mem_free": 238026752,
"2.1.mem_shared": 0,
"2.1.mem_total": 970506240,
"2.1.mem_used": 297058304,
"2.1.mem_utilization": 30.608591437339783,
"2.1.os_name": "linux",
"2.1.os_release": "4.4.9-v7+",
"2.1.page_in": 0,
"2.1.page_out": 2.2157316950347465,
"2.1.part_max_used": 31.74,
"2.1.pkts_in": 11.582233860408904,
"2.1.pkts_out": 2.266089233558264,
"2.1.proc_run": 1,
"2.1.proc_total": 237,
"2.1.read_time": 0,
"2.1.reads": 0,
"2.1.swap_free": 104853504,
"2.1.swap_in": 0,
"2.1.swap_out": 0,
"2.1.swap_total": 104853504,
"2.1.tcp_activeopens": 0,
"2.1.tcp_attemptfails": 0,
"2.1.tcp_currestab": 6,
"2.1.tcp_estabresets": 0,
"2.1.tcp_incsumerrs": 0,
"2.1.tcp_inerrs": 0,
"2.1.tcp_insegs": 2.568234464699365,
"2.1.tcp_maxconn": 4294967295,
"2.1.tcp_outrsts": 0,
"2.1.tcp_outsegs": 1.913586463893645,
"2.1.tcp_passiveopens": 0,
"2.1.tcp_retranssegs": 0,
"2.1.tcp_rtoalgorithm": 1,
"2.1.tcp_rtomax": 120000,
"2.1.tcp_rtomin": 200,
"2.1.udp_incsumerrors": 0,
"2.1.udp_indatagrams": 6.596837546580723,
"2.1.udp_inerrors": 0,
"2.1.udp_noports": 0,
"2.1.udp_outdatagrams": 0.2517876926175849,
"2.1.udp_rcvbuferrors": 0,
"2.1.udp_sndbuferrors": 0,
"2.1.uptime": 46603,
"2.1.uuid": "22c4ce8c-067e-4517-8c00-8d822efc4897",
"2.1.write_time": 8.333333333333334,
"2.1.writes": 0.6042904622822036,
"2.ifadminstatus": "up",
"2.ifdirection": "full-duplex",
"2.ifindex": "2",
"2.ifindiscards": 0,
"2.ifinerrors": 0,
"2.ifinoctets": 3241.1101037574294,
"2.ifinpkts": 11.483831973405863,
"2.ifinucastpkts": 11.483831973405863,
"2.ifinutilization": 0.025928880830059432,
"2.ifname": "eth0",
"2.ifoperstatus": "up",
"2.ifoutdiscards": 0,
"2.ifouterrors": 0,
"2.ifoutoctets": 406.6686813740304,
"2.ifoutpkts": 1.8636043114737586,
"2.ifoutucastpkts": 1.8636043114737586,
"2.ifoututilization": 0.0032533494509922435,
"2.ifspeed": 100000000,
"2.iftype": "ethernetCsmacd",
"sensors.tempC": 49.388
}
Note the custom temperature metric at the end of the list.
A future article will examine how the Host sFlow agent can be used to efficiently stream measurements from large numbers of inexpensive Rasberry Pi devices ($5 for model Zero) to the sFlow-RT collector to monitor and control the "Internet of Things" (IoT).The following instructions show how to install sFlow-RT on Raspbian Jesse (the Debian Linux based Raspberry Pi operating system).
wget http://www.inmon.com/products/sFlow-RT/sflow-rt_2.0-1092.deb sudo dpkg -i --ignore-depends=openjdk-7-jre-headless sflow-rt_2.0-1092.debWe are ignoring the dependency on openjdk and will use the default Raspbian Java 1.8 version instead.
-Xms600M -Xmx600M -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalModeThese new settings reduce the requested memory to fit within the 1G on the Raspberry Pi and leave some memory for system tasks. The G1GC garbage collector not available on ARM Java implementation, so we will use incremental concurrent mark and sweep instead.
sudo service sflow-rt startThe sFlow-RT web interface should now be accessible at http://<raspberrypi_ip>:8008/
#!/usr/bin/env python
import requests
import json
rt = 'http://127.0.0.1:8008'
flow = {'keys':'ipsource,ipdestination','value':'bytes'}
requests.put(rt+'/flow/pair/json',data=json.dumps(flow))
threshold = {'metric':'pair','value':100000000/8,'byFlow':True,'timeout':1}
requests.put(rt+'/threshold/elephant/json',data=json.dumps(threshold))
eventurl = rt+'/events/json?thresholdID=elephant&maxEvents=10&timeout=60'
eventID = -1
while 1 == 1:
r = requests.get(eventurl + "&eventID=" + str(eventID))
if r.status_code != 200: break
events = r.json()
if len(events) == 0: continue
eventID = events[0]["eventID"]
events.reverse()
for e in events:
print e['flowKey']
In addition, there are a number of open source sFlow-RT Applications available on the Downloads page (e.g. Mininet dashboard, DDoS mitigation, etc.) and articles describing use cases for sFlow-RT on this blog.