Wednesday, October 9, 2019

InfluxDB 2.0

Introducing the Next-Generation InfluxDB 2.0 Platform mentions that InfluxDB 2.0 will be able to scrape Prometheus exporters. Get started with InfluxDB provides instructions for running an alpha version of the new software using Docker:
docker run --name influxdb -p 9999:9999 quay.io/influxdb/influxdb:2.0.0-alpha
Prometheus exporter describes an application that runs on the sFlow-RT analytics platform that converts real-time streaming telemetry from industry standard sFlow agents. Host, Docker, Swarm and Kubernetes monitoring describes how to deploy agents on popular container orchestration platforms.
The screen capture above shows three scrapers configured in InfluxDB 2.0:
  1. sflow-rt-analyzer,
    URL: http://10.0.0.70:8008/app/prometheus/scripts/export.js/analyzer/txt
  2. sflow-rt-dump,
    URL: http://10.0.0.70:8008/app/prometheus/scripts/export.js/dump/ALL/ALL/txt
  3. sflow-rt-flow-src-dst,
    URL: http://10.0.0.70:8008/app/prometheus/scripts/export.js/flows/ALL/txt?metric=flow_src_dst_bps&key=ipsource,ipdestination&value=bytes&aggMode=max&maxFlows=100&minValue=1000&scale=8
The first collects metrics about the performance of the sFlow-RT analytics engine, the second, all the metrics exported by the sFlow agents, and the third, is a flow metric (see Flow metrics with Prometheus and Grafana).
InfluxDB 2.0 now includes the data exploration and dashboard building capabilities that were previously in the separate Chronograf application. The screen capture above shows a simple chart trending ifinoctets across a number of switch ports.

Note: There are a number of articles on this blog that demonstrate how to push metrics from sFlow-RT into InfluxDB 1.0 using its REST API. The ability to scrape metrics from a Prometheus exporter simplifies the integration.

Tuesday, October 1, 2019

Flow metrics with Prometheus and Grafana

The Grafana dashboard above shows real-time network traffic flow metrics. This article describes how to define and collect flow metrics using the Prometheus time series database and build Grafana dashboards using those metrics.
Prometheus exporter describes an application that runs on the sFlow-RT analytics platform that converts real-time streaming telemetry from industry standard sFlow agents. Host, Docker, Swarm and Kubernetes monitoring describes how to deploy agents on popular container orchestration platforms.

The latest version of the Prometheus exporter application adds flow export.
global:
  scrape_interval:     15s
  evaluation_interval: 15s

rule_files:
  # - "first.rules"
  # - "second.rules"

scrape_configs:
  - job_name: 'sflow-rt-metrics'
    metrics_path: /app/prometheus/scripts/export.js/dump/ALL/ALL/txt
    static_configs:
      - targets: ['10.0.0.70:8008']
  - job_name: 'sflow-rt-src-dst-bps'
    metrics_path: /app/prometheus/scripts/export.js/flows/ALL/txt
    static_configs:
      - targets: ['10.0.0.70:8008']
    params:
      metric: ['ip_src_dst_bps']
      key: ['ipsource','ipdestination']
      label: ['src','dst']
      value: ['bytes']
      scale: ['8']
      minValue: ['1000']
      maxFlows: ['100']
  - job_name: 'sflow-rt-countries-bps'
    metrics_path: /app/prometheus/scripts/export.js/flows/ALL/txt
    static_configs:
      - targets: ['10.0.0.70:8008']
    params:
      metric: ['ip_countries_bps']
      key: ['null:[country:ipsource]:unknown','null:[country:ipdestination]:unknown']
      label: ['src','dst']
      value: ['bytes']
      scale: ['8']
      aggMode: ['sum']
      minValue: ['1000']
      maxFlows: ['100']
The above prometheus.yml file extends the previous example to add two additional scrape jobs, sflow-rt-src-dst-bps and sflow-rt-countries-bps, that return flow metrics. Defining flows describes the attributes and settings available to build a flow definition. The metric: setting names the Prometheus metric and the label: setting is used to map corresponding sFlow-RT flow keys into Prometheus labels.
The first step in building a Grafana dashboard panel to display flow data is to construct a query:
topk(10, sum(ip_src_dst_bps) by (src))
In this case, the query sums the flows by source address and return the top 10 values for each interval in the graph.

The query for the Top Source Countries chart is a little more complex:
topk(10,sum(ip_countries_bps{src!="unknown"}) by (src))
In this case unknown source country values (the value set in the prometheus.yml file to use when a country lookup fails on an ipsource address) are excluded in the query.
In the visualization settings, Null value: null as zeroTooltip Mode: Single, label the Left Y axis, and Legend Show disabled.
Finally, give the chart a title.
The Prometheus exporter application on sFlow-RT (accessible on port 8008) has a REST API explorer, above, that can be used to experiment with flow settings before configuring a Prometheus scraper job. When testing the settings, the first query will not return any data since the flow hasn't been programmed. Click the Execute button a second time to see data. Also consider using the sflow/flow-trend application as a way to gain familiarity with sFlow-RT's flow analytics engine.