Thursday, October 13, 2016

Real-time domain name lookups

Reverse DNS requests request the domain name associated with an IP address, for example providing the name for IP address  This article demonstrates how the sFlow-RT engine incorporates domain name lookups in real-time flow analytics.

First, use the dns.servers System Property is used to specify one or more DNS servers to handle the reverse lookup requests. For example, the following command uses Docker to run sFlow-RT with DNS lookups directed to server
docker run -e "RTPROP=-Ddns.servers=" \
-p 8008:8008 -p 6343:6343/udp -d sflow/sflow-rt
The following Python script uses the sFlow-RT REST API to define a flow and log the resulting flow records:
#!/usr/bin/env python
import requests
import json

flow = {'keys':'dns:ipsource,dns:ipdestination',
flowurl = 'http://localhost:8008/flows/json?name=dnspair&maxFlows=10&timeout=60'
flowID = -1
while 1 == 1:
  r = requests.get(flowurl + "&flowID=" + str(flowID))
  if r.status_code != 200: break
  flows = r.json()
  if len(flows) == 0: continue

  flowID = flows[0]["flowID"]
  for f in flows:
    print json.dumps(f,indent=1)
Running the script generates the following output:
$ ./
 "value": 233370.92322668363, 
 "end": 1476234478177, 
 "name": "dnspair", 
 "flowID": 1523, 
 "agent": "", 
 "start": 1476234466195, 
 "dataSource": "10", 
 "flowKeys": ","
 "value": 39692.88754760739, 
 "end": 1476234478177, 
 "name": "dnspair", 
 "flowID": 1524, 
 "agent": "", 
 "start": 1476234466195, 
 "dataSource": "10", 
 "flowKeys": ","
The token dns:ipsource in the flow definition is an example of a Key Function. Functions can be combined to define flow keys or in filters.
Returns a dns name if available, otherwise the original IP address is returned
Returns the last 2 parts of the DNS name, e.g. becomes

DNS results are cached by the dns: function in order to provide real-time lookups and reduce the load on the backend name server(s). Cache size and timeout settings are tune-able using System Properties.

No comments:

Post a Comment