Wednesday, September 25, 2019

Host, Docker, Swarm and Kubernetes monitoring

The open source Host sFlow agent incorporates technologies that address the challenges of microservice monitoring; leveraging recent enhancements to Berkeley Packet Filter (BPF) in the Linux kernel to randomly sample packets, and  Asynchronous Docker metrics to track rapidly changing workloads. The continuous stream of real-time telemetry from all compute nodes, transported using the industry standard sFlow protocol, provides comprehensive real-time cluster-wide visibility into all services and the traffic flowing between them.

The Host sFlow agent is available as pre-packaged rpm/deb files that can be downloaded and installed on each node in a cluster.
sflow {
  collector { ip=10.0.0.70 }
  docker { }
  pcap { dev=docker0 }
  pcap { dev=docker_gwbridge } 
}
The above /etc/hsflowd.conf file, see Configuring Host sFlow for Linux via /etc/hsflowd.conf, enables the docker {} and pcap {} modules for detailed visibility into container metrics and network traffic flows, and streams telemetry to an sFlow collector (10.0.0.70). The configuration is the same for every node making it simple to install and configure Host sFlow on all nodes using orchestration software such as Puppet, Chef, Ansible, etc.

The agent is also available as the pre-build sflow/host-sflow image, providing a simple method of instrumenting nodes running container workloads.
docker run \
--detach \
--name=host-sflow \
--env "COLLECTOR=10.0.0.70" \
--net=host \
--volume /var/run/docker.sock:/var/run/docker.sock:ro \
sflow/host-sflow
Execute above command to install and run the Host sFlow agent on a Docker node.
docker service create \
--mode global \
--name host-sflow \
--env "COLLECTOR=10.0.0.70" \
--network host \
--mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock,readonly \
sflow/host-sflow
Install and run an instance of the Host sFlow agent on each node in a Docker Swarm cluster.

Deploying Host sFlow under Kubernetes is a little more complicated.
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: host-sflow
spec:
  selector:
    matchLabels:
      name: host-sflow
  template:
    metadata:
      labels:
        name: host-sflow
    spec:
      hostNetwork: true
      containers:
      - name: host-sflow
        image: sflow/host-sflow:latest
        env:
          - name: COLLECTOR
            value: "10.0.0.70"
          - name: NET
            value: "host"
        volumeMounts:
          - mountPath: /var/run/docker.sock
            name: docker-sock
            readOnly: true
      volumes:
        - name: docker-sock
          hostPath:
            path: /var/run/docker.sock
First, create a deployment description file like the host-sflow.yml file above.
kubectl apply -f host-sflow.yml
Install and run an instance of the Host sFlow agent on each node in the Kubernetes cluster.
docker run -p 6343:6343/udp sflow/sflowtool
Run the command above on the collector (10.0.0.70) to verify that sFlow is arriving, see Running sflowtool using Docker.
docker run -p 6343:6343/udp -p 8008:8008 sflow/sflow-rt
Run the sflow/sflow-rt image to access real-time cluster performance metrics and network traffic flows through a REST API. Forwarding using sFlow-RT describes how to copy sFlow telemetry streams for additional tools.
Install sFlow-RT applications to export metrics to Prometheus, block DDoS attacks, visualize flows, etc. Writing Applications describes how to use APIs to build your own applications to integrate analytics with automation and monitoring tools.

Monday, September 9, 2019

Packet analysis using Docker

Why use sFlow for packet analysis? To rephrase the Heineken slogan, sFlow reaches the parts of the network that other technologies cannot reach. Industry standard sFlow is widely supported by switch vendors, embedding wire-speed packet monitoring throughout the network. With sFlow, any link or group of links can be remotely monitored. The alternative approach of physically attaching a probe to a SPAN/Mirror port is becoming much less feasible with increasing network sizes (10's of thousands of switch ports) and link speeds (10, 100, and 400 Gigabits). Using sFlow for packet capture doesn't replace traditional packet analysis, instead sFlow extends the capabilities of existing packet capture tools into the high speed switched network.

This article describes the sflow/tcpdump  and sflow/tshark Docker images, which provide a convenient way to analyze packets captured using sFlow.

Run the following command to analyze packets using tcpdump:
$ docker run -p 6343:6343/udp -p 8008:8008 sflow/tcpdump

19:06:42.000000 ARP, Reply 10.0.0.254 is-at c0:ea:e4:89:b0:98 (oui Unknown), length 64
19:06:42.000000 IP 10.0.0.236.548 > 10.0.0.70.61719: Flags [P.], seq 3380015689:3380015713, ack 515038158, win 41992, options [nop,nop,TS val 1720029042 ecr 904769627], length 24
19:06:42.000000 IP 10.0.0.236.548 > 10.0.0.70.61719: Flags [P.], seq 149816:149832, ack 510628, win 41992, options [nop,nop,TS val 1720029087 ecr 904770068], length 16
19:06:42.000000 IP 10.0.0.236.548 > 10.0.0.70.61719: Flags [P.], seq 149816:149832, ack 510628, win 41992, options [nop,nop,TS val 1720029087 ecr 904770068], length 16
The normal tcpdump options can be used. For example, to select DNS packets:
$ docker run -p 6343:6343/udp -p 8008:8008 sflow/tcpdump -vv port 53
reading from file -, link-type EN10MB (Ethernet)
19:08:49.000000 IP (tos 0x0, ttl 64, id 22316, offset 0, flags [none], proto UDP (17), length 65)
    10.0.0.70.43801 > dns.google.53: [udp sum ok] 35941+ A? clients2.google.com. (37)
19:09:00.000000 IP (tos 0x0, ttl 255, id 16813, offset 0, flags [none], proto UDP (17), length 66)
    10.0.0.64.50675 > 10.0.0.1.53: [udp sum ok] 57874+ AAAA? p49-imap.mail.me.com. (38)
The following command selects TCP SYN packets:
$ docker run -p 6343:6343/udp sflow/tcpdump 'tcp[tcpflags] == tcp-syn'
reading from file -, link-type EN10MB (Ethernet)
19:10:37.000000 IP 10.0.0.30.46786 > 10.0.0.162.1179: Flags [S], seq 2993962362, win 29200, options [mss 1460,sackOK,TS val 20531427 ecr 0,nop,wscale 9], length 0
Capture 10 packets to a file and then exit:
$ docker run -v $PWD:/pcap -p 6343:6343/udp sflow/tcpdump -w /pcap/packets.pcap -c 10
reading from file -, link-type EN10MB (Ethernet)
A tcpdump Tutorial with Examples — 50 Ways to Isolate Traffic provides an overview of the capabilities of tcpdump with useful examples.

Run the following command to analyze packets using tshark - a terminal based version of Wireshark:
$ docker run -p 6343:6343/udp -p 8008:8008 sflow/tshark
Capturing on '-'
    1   0.000000   10.0.0.236 → 10.0.0.70    AFP 1518 [Reply without query?]
    2   0.000000   10.0.0.236 → 10.0.0.70    AFP 1518 [Reply without query?]
    3   0.000000   10.0.0.114 → 10.0.0.72    SSH 1518 Server: Encrypted packet (len=1448)
Packets can be filtered using Display Filters. For example, the following command selects DNS traffic:
$ docker run -p 6343:6343/udp -p 8008:8008 sflow/tshark -Y 'dns'
Capturing on '-'
  328  22.000000      8.8.8.8 → 10.0.0.70    DNS 136 Standard query response 0xfce4 AAAA img.youtube.com CNAME ytimg.l.google.com AAAA
  472  36.000000    10.0.0.52 → 10.0.0.1     DNS 79 Standard query 0x173e AAAA www.nytimes.com
Print ip source, destination, protocol and packet lengths:
$ docker run -p 6343:6343/udp -p 8008:8008 sflow/tshark -T fields -e ip.src -e ip.dst -e ip.proto -e ip.len
Capturing on '-'
10.0.0.70 10.0.0.236 6 1500
10.0.0.236 10.0.0.70 6 52
10.0.0.70 10.0.0.236 6 1500
10.0.0.236 10.0.0.70 6 52
10.0.0.70 10.0.0.236 6 1500
Capture 100 packets and print summary of the protocols:
$ docker run -p 6343:6343/udp -p 8008:8008 sflow/tshark -q -z io,phs -c 100
Capturing on '-'
100 packets captured

===================================================================
Protocol Hierarchy Statistics
Filter: 

eth                                      frames:100 bytes:85721
  ip                                     frames:99 bytes:85657
    tcp                                  frames:97 bytes:85119
      dsi                                frames:61 bytes:82122
        _ws.short                        frames:54 bytes:77180
        afp                              frames:6 bytes:4856
          _ws.short                      frames:5 bytes:4766
      _ws.short                          frames:15 bytes:1050
      http                               frames:1 bytes:499
        _ws.short                        frames:1 bytes:499
      iscsi                              frames:1 bytes:118
        iscsi.flags                      frames:1 bytes:118
          scsi                           frames:1 bytes:118
            _ws.short                    frames:1 bytes:118
    ipv6                                 frames:2 bytes:538
      tcp                                frames:2 bytes:538
        tls                              frames:2 bytes:538
          _ws.short                      frames:2 bytes:538
  arp                                    frames:1 bytes:64
    _ws.short                            frames:1 bytes:64
===================================================================
Capture 100 packets and print a summary of the IP traffic by address:
$ docker run -p 6343:6343/udp -p 8008:8008 sflow/tshark -q -z endpoints,ip -c 100
Capturing on '-'
100 packets captured

================================================================================
IPv4 Endpoints
Filter:
                       |  Packets  | |  Bytes  | | Tx Packets | | Tx Bytes | | Rx Packets | | Rx Bytes |
10.0.0.70                     95         81713         44           25507          51           56206   
10.0.0.236                    91         80820         50           55956          41           24864   
10.0.0.30                      6          2369          2            1508           4             861   
10.0.0.16                      1           587          1             587           0               0   
10.0.0.28                      1           587          0               0           1             587   
10.0.0.160                     1          1258          0               0           1            1258   
10.0.0.172                     1           218          1             218           0               0   
================================================================================
The following command prints packet decodes as JSON:
$ docker run -p 6343:6343/udp -p 8008:8008 sflow/tshark -T json
Capturing on '-'
[
  {
    "_index": "packets-2019-09-06",
    "_type": "pcap_file",
    "_score": null,
    "_source": {
      "layers": {
        "frame": {
          "frame.interface_id": "0",
          "frame.interface_id_tree": {
            "frame.interface_name": "-"
          },
          "frame.encap_type": "1",
          "frame.time": "Sep  6, 2019 19:41:12.000000000 UTC",
          "frame.offset_shift": "0.000000000",
          "frame.time_epoch": "1567798872.000000000",
          "frame.time_delta": "0.000000000",
          "frame.time_delta_displayed": "0.000000000",
          "frame.time_relative": "0.000000000",
          "frame.number": "1",
          "frame.len": "64",
          "frame.cap_len": "60",
          "frame.marked": "0",
          "frame.ignored": "0",
          "frame.protocols": "eth:ethertype:arp"
        },
        "eth": {
          "eth.dst": "70:10:6f:d8:13:30",
          "eth.dst_tree": {
            "eth.dst_resolved": "HewlettP_d8:13:30",
            "eth.addr": "70:10:6f:d8:13:30",
            "eth.addr_resolved": "HewlettP_d8:13:30",
            "eth.lg": "0",
            "eth.ig": "0"
          },
          "eth.src": "98:4b:e1:03:4a:61",
          "eth.src_tree": {
            "eth.src_resolved": "HewlettP_03:4a:61",
            "eth.addr": "98:4b:e1:03:4a:61",
            "eth.addr_resolved": "HewlettP_03:4a:61",
            "eth.lg": "0",
            "eth.ig": "0"
          },
          "eth.type": "0x00000806",
          "eth.padding": "00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00"
        },
        "arp": {
          "arp.hw.type": "1",
          "arp.proto.type": "0x00000800",
          "arp.hw.size": "6",
          "arp.proto.size": "4",
          "arp.opcode": "1",
          "arp.src.hw_mac": "98:4b:e1:03:4a:61",
          "arp.src.proto_ipv4": "10.0.0.30",
          "arp.dst.hw_mac": "00:00:00:00:00:00",
          "arp.dst.proto_ipv4": "10.0.0.232"
        },
        "_ws.short": "[Packet size limited during capture: Ethertype truncated]"
      }
    }
  },
The tshark -T ek option formats the JSON output as a single line per packet making the output easy to parse in scripts. For example, the following emerging.py script downloads the Emerging Threats compromised IP address database, parses the JSON records, checks to see if source and destination addresses can be found in the database, and prints out information on any matches:
#!/usr/bin/env python

from sys import stdin
from json import loads
from requests import get

blacklist = set()
r = get('https://rules.emergingthreats.net/blockrules/compromised-ips.txt')
for line in r.iter_lines():
  blacklist.add(line)

for line in stdin:
  msg = loads(line)
  try:
    time = msg['timestamp']
    layers = msg['layers']
    ip = layers["ip"]
    src = ip["ip_ip_src"]
    dst = ip["ip_ip_dst"]
    if src in blacklist or dst in blacklist:
      print "%s %s %s" % (time,src,dst)
  except KeyError:
    pass
The following command runs the script:
$ docker run -p 6343:6343/udp -p 8008:8008 sflow/tshark -T ek | ./tshark.py
See the TShark man page for more options.

Forwarding using sFlow-RT describes how to set up and tear down sFlow streams using the sFlow-RT analytics engine. This is a simple way to direct a stream of sFlow to a desktop running sflowtool. For example, suppose sflowtool is running on host 10.0.0.30 and sFlow-RT is running on host 10.0.0.1, the following command would start a session:
curl -H "Content-Type:application/json" -X PUT --data '{"address":"10.0.0.30"}' \
http://10.0.0.1:8008/forwarding/tcpdump/json
and the following command would end the session:
curl -X DELETE http://10.0.0.1:8008/forwarding/tcpdump/json
Note: The sflow/sflow-rt Docker image is a convenient way to run sFlow-RT:
docker run -p 8008:8008 -p 6343:6343/udp sflow/sflow-rt
Finally, Triggered remote packet capture using filtered ERSPAN, shows how the broad visibility provided by sFlow can be combined with hardware filtering to trigger full packet capture of selected traffic.

Friday, September 6, 2019

Running sflowtool using Docker

The sflowtool command line utility is used to convert standard sFlow records into a variety of different formats. While there are a large number of native sFlow analysis applications, familiarity with sflowtool is worthwhile since it provides a simple way to verify receipt of sFlow, understand the contents of the sFlow telemetry stream, and build simple applications through custom scripting.

The sflow/sflowtool Docker image provides a simple way to run sflowtool. Run the following command to print the contents of sFlow packets:
$ docker run -p 6343:6343/udp sflow/sflowtool
startDatagram =================================
datagramSourceIP 10.0.0.111
datagramSize 144
unixSecondsUTC 1321922602
datagramVersion 5
agentSubId 0
agent 10.0.0.20
packetSequenceNo 3535127
sysUpTime 270660704
samplesInPacket 1
startSample ----------------------
sampleType_tag 0:2
sampleType COUNTERSSAMPLE
sampleSequenceNo 228282
sourceId 0:14
counterBlock_tag 0:1
ifIndex 14
networkType 6
ifSpeed 100000000
ifDirection 0
ifStatus 3
ifInOctets 4839078
ifInUcastPkts 15205
ifInMulticastPkts 0
ifInBroadcastPkts 4294967295
ifInDiscards 0
ifInErrors 0
ifInUnknownProtos 4294967295
ifOutOctets 149581962744
ifOutUcastPkts 158884229
ifOutMulticastPkts 4294967295
ifOutBroadcastPkts 4294967295
ifOutDiscards 101
ifOutErrors 0
ifPromiscuousMode 0
endSample   ----------------------
endDatagram   =================================
The -g option flattens the output so that it is more easily filtered using grep:
$ docker run -p 6343:6343/udp sflow/sflowtool -g | grep ifInOctets
2019-09-03T22:37:21+0000 10.0.0.231 0 3203000 0:6 0:2 0:1 ifInOctets 0
2019-09-03T22:37:23+0000 10.0.0.232 0 7242462 0:5 0:2 0:1 ifInOctets 53791415069
2019-09-03T22:37:23+0000 10.0.0.253 0 8178007 0:7 0:2 0:1 ifInOctets 31663763747
2019-09-03T22:37:23+0000 10.0.0.253 0 8178007 0:3 0:2 0:1 ifInOctets 1333603780050
2019-09-03T22:37:26+0000 10.0.0.253 0 8178008 0:1 0:2 0:1 ifInOctets 9116481296
The -L option prints out CSV records with the selected fields:
$ docker run -p 6343:6343/udp sflow/sflowtool -L agent,ifIndex,ifInOctets
10.0.0.253,23,432680126074
10.0.0.30,2,54056144719
10.0.0.253,21,3860664000830
10.0.0.253,3,1345269893416
10.0.0.253,2,1910370790761
The -J option prints out the decoded sFlow datagrams as JSON (with a blank line between each datagram):
$ docker run -p 6343:6343/udp sflow/sflowtool -J
{
 "datagramSourceIP":"172.17.0.1",
 "datagramSize":"1388",
 "unixSecondsUTC":"1567707952",
 "localtime":"2019-09-05T18:25:52+0000",
 "datagramVersion":"5",
 "agentSubId":"0",
 "agent":"10.0.0.253",
 "packetSequenceNo":"8254753",
 "sysUpTime":"165436226",
 "samplesInPacket":"8",
 "samples":[{
   "sampleType_tag":"0:1",
   "sampleType":"FLOWSAMPLE",
   "sampleSequenceNo":"2594544",
   "sourceId":"0:3",
   "meanSkipCount":"500",
   "samplePool":"1622164761",
   "dropEvents":"584479",
   "inputPort":"21",
   "outputPort":"3",
   "elements":[{
     "flowBlock_tag":"0:1",
     "flowSampleType":"HEADER",
     "headerProtocol":"1",
     "sampledPacketSize":"118",
     "strippedBytes":"4",
     "headerLen":"116",
...
The -j option formats the JSON output as a single line per datagram making the output easy to parse in scripts. For example, the following emerging.py script downloads the Emerging Threats compromised IP address database, parses the JSON records, checks to see if source and destination addresses can be found in the database, and prints out information on any matches:
#!/usr/bin/env python

from sys import stdin
from json import loads
from requests import get

blacklist = set()
r = get('https://rules.emergingthreats.net/blockrules/compromised-ips.txt')
for line in r.iter_lines():
  blacklist.add(line)

for line in stdin:
  datagram = loads(line)
  localtime = datagram["localtime"]
  samples = datagram["samples"]
  for sample in samples:
    sampleType = sample["sampleType"]
    elements = sample["elements"]
    if sampleType == "FLOWSAMPLE":
      for element in elements:
        tag = element["flowBlock_tag"]
        if tag == "0:1":
          try:
            src = element["srcIP"]
            dst = element["dstIP"]
            if src in blacklist or dst in blacklist:
              print "%s %s %s" % (localtime,src,dst)
          except KeyError:
            pass
Run the command:
docker run -p 6343:6343/udp sflow/sflowtool -j | ./emerging.py
These were just a few examples, see the sflowtool home page for additional information.

Forwarding using sFlow-RT describes how to set up and tear down sFlow streams using the sFlow-RT analytics engine. This is a simple way to direct a stream of sFlow to a desktop running sflowtool. For example, suppose sflowtool is running on host 10.0.0.30 and sFlow-RT is running on host 10.0.0.1, the following command would start a session:
curl -H "Content-Type:application/json" -X PUT --data '{"address":"10.0.0.30"}' \
http://10.0.0.1:8008/forwarding/sflowtool/json
and the following command would end the session:
curl -X DELETE http://10.0.0.1:8008/forwarding/sflowtool/json
Note: The sflow/sflow-rt Docker image is a convenient way to run sFlow-RT:
docker run -p 8008:8008 -p 6343:6343/udp sflow/sflow-rt

Tuesday, September 3, 2019

Forwarding using sFlow-RT

The diagrams show different two different configurations for sFlow monitoring:
  1. Without Forwarding Each sFlow agent is configured to stream sFlow telemetry to each of the analysis applications. This configuration is appropriate when a small number of applications is being used to continuously monitor performance. However, the overhead on the network and agents increases as additional analyzers are added. Often it is not possible to increase the number of analyzers since many embedded sFlow agents have limited resources and only support a small number of sFlow streams. In addition, the complexity of configuring each agent to add or remove an analysis application can be significant since agents may reside in Ethernet switches, routers, servers, hypervisors and applications on many different platforms from a variety of vendors.
  2. With Forwarding In this case all the agents are configured to send sFlow to a forwarding module which resends the data to the analysis applications. In this case analyzers can be added and removed simply by reconfiguring the forwarder without any changes required to the agent configurations.
There are many variations between these two extremes. Typically there will be one or two analyzers used for continuous monitoring and additional tools, like Wireshark, might be deployed for troubleshooting when the continuous monitoring tools detect anomalies.

This article will demonstrate how to forward sFlow using sFlow-RT.

Download and install and install the software and configure the sFlow agents to stream telemetry to the sFlow-RT instance.
The sFlow-RT status page, accessible on HTTP port 8008, can be used to verify that sFlow is being received from the agents. Click on the API option then click on the Open REST API Explorer button to access documentation on the sFlow-RT REST API.
The following REST API call creates a forwarding session, SessionA, directing a stream of sFlow to analyzer 10.0.0.30:
curl -H "Content-Type:application/json" -X PUT --data '{"address":"10.0.0.30"}' \
http://127.0.0.1:8008/forwarding/SessionA/json
Create a second session, SessionB, to a non-standard port, 7343:
curl -H "Content-Type:application/json" \
-X PUT --data '{"address":"10.0.0.30","port":7343}' \
http://127.0.0.1:8008/forwarding/SessionB/json
Create a third session, SessionC, to forward sFlow from selected agent, 10.0.0.254:
curl -H "Content-Type:application/json" \
-X PUT --data '{"address":"10.0.0.30","port":8343,"agents":["10.0.0.254"]}' \
http://127.0.0.1:8008/forwarding/SessionC/json
See the all forwarding sessions:
curl http://127.0.0.1:8008/forwarding/json
Delete forwarding session, SessionB:
curl -X DELETE http://127.0.0.1:8008/forwarding/SessionB/json
In addition, sFlow-RT supports the complex filtering and forwarding operations needed stream per-tenant views of the sFlow telemetry in a shared network, see Multi-tenant sFlow.
Finally, the streaming analytics capabilities of sFlow-RT can be used to simultaneously deliver metrics to time series databases (e.g. Prometheus and Grafana), send events to SIEM tools like Splunk or Logstash (e.g. Exporting events using syslog), and export flow data (e.g. sFlow to IPFIX/NetFlow) while also running embedded applications to visualize data, mitigate DDoS attacks, and optimize routing.