Thursday, September 15, 2022

Low latency flow analytics


Real-time analytics on network flow data with Apache Pinot describes LinkedIn's flow ingestion and analytics pipeline for sFlow and IPFIX exports from network devices. The solution uses Apache Kafka message queues to connect LinkedIn's InFlow flow analyzer with the Apache Pinot datastore to support low latency queries. The article describes the scale of the monitoring system, InFlow receives 50k flows per second from over 100 different network devices on the LinkedIn backbone and edge devices and states InFlow requires storage of tens of TBs of data with a retention of 30 days. The article concludes, Following the successful onboarding of flow data to a real-time table on Pinot, freshness of data improved from 15 mins to 1 minute and query latencies were reduced by as much as 95%.
The sFlow-RT real-time analytics engine provides a faster, simpler, more scaleable, alternative for flow monitoring. sFlow-RT  radically simplifies the measurement pipeline, combining flow collection, enrichment, and analytics in a single programmable stage. Removing pipeline stages improves data freshness — flow measurements represent an up to the second view of traffic flowing through the monitored network devices. The improvement from minute to sub-second data freshness enhances automation use cases such as automated DDoS mitigation, load balancing, and auto-scaling (see Delay and stability).

An essential step in improving data freshness is eliminating IPFIX and migrating to an sFlow only network monitoring system. Rapidly detecting large flows, sFlow vs. NetFlow/IPFIX describes how the on-device flow cache component of IPFIX/NetFlow measurements adds an additional stage (and additional latency, anywhere from 1 to 30 minutes) to the flow analytics pipeline. On the other hand, sFlow is stateless, eliminates the flow cache, and guarantees sub-second freshness. Moving from IPFIX/NetFlow is straightforward as most of the leading router vendors now include sFlow support, see Real-time flow telemetry for routers. In addition, sFlow is widely supported in switches, making it possible to efficiently monitor every device in the data center.

Removing pipeline stages increases scaleability and reduces the cost of the solution. A single instance of sFlow-RT can monitor 10's of thousands of network devices and over a million flows per second, replacing large numbers of scale-out pipeline instances. Removing the need for distributed message queues improves resilience by decoupling the flow monitoring system from the network so that it can reliably deliver the visibility needed to manage network congestion "meltdown" events — see Who monitors the monitoring systems?

Introducing a programmable flow analytics stage before storing the data significantly reduces storage requirements (and improves query latency) since only metrics of interest will be computed and stored (see Flow metrics with Prometheus and Grafana).

The LinkedIn paper mentions that they are developing an eBPF Skyfall agent for monitoring hosts. The open source Host sFlow agent extends sFlow to hosts and can be deployed at scale today, see Real-time Kubernetes cluster monitoring example.

docker run -p 8008:8008 -p 6343:6343/udp sflow/prometheus
Trying out sFlow-RT is easy. For example, run the above command to start the analyzer using the pre-built sflow/prometheus Docker image. Configure sFlow agents to stream telemetry to the analyzer and access the through the web interface on port 8008 (see Getting Started). The default settings should work for most small to moderately sized networks. See Tuning Performance for tips on optimizing performance for larger sites.

Wednesday, August 31, 2022

DDoS Sonification

Sonification presents data as sounds instead of visual charts. One of the best known examples of sonification is the representation of radiation level as a click rate in a Geiger counter. This article describes ddos-sonify, an experiment to see if sound can be usefully employed to represent information about Distributed Denial of Service (DDoS) attacks. The DDoS attacks and BGP Flowspec responses testbed was used to create the video demonstration at the top of this page in which a series of simulated DDoS attacks are detected and mitigated. Play the video to hear the results.

The software uses the Tone.js library to control Web Audio sound generation functionality in a web browser.

var voices = {};
var loop;
var loopInterval = '4n';
$('#sonify').click(function() {
  if($(this).prop("checked")) {
    voices.synth = new Tone.PolySynth(Tone.Synth).toDestination();
    voices.metal = new Tone.PolySynth(Tone.MetalSynth).toDestination();
    voices.pluck = new Tone.PolySynth(Tone.PluckSynth).toDestination();
    voices.membrane = new Tone.PolySynth(Tone.MembraneSynth).toDestination();
    voices.am = new Tone.PolySynth(Tone.AMSynth).toDestination();
    voices.fm = new Tone.PolySynth(Tone.FMSynth).toDestination();
    voices.duo = new Tone.PolySynth(Tone.DuoSynth).toDestination();
    Tone.Transport.bpm.value=80;
    loop = new Tone.Loop((now) => {
      sonify(now);
    },loopInterval).start(0);
    Tone.Transport.start();
  } else {
    loop.stop();
    loop.dispose();
    Tone.Transport.stop();
  }
});
Clicking on the Convert charts to sound checkbox on the web page initializes the different sound synthesizers that will be used to create sounds and starts a timed loop that will periodically call the sonify() function convert current values of each of the metrics into sounds.
var metrics = [
  {name:'top-5-ip-flood', threshold:'threshold_ip_flood', voice:'synth'},
  {name:'top-5-ip-fragmentation', threshold:'threshold_ip_fragmentation', voice:'duo'},
  {name:'top-5-icmp-flood', threshold:'threshold_icmp_flood', voice:'pluck'},
  {name:'top-5-udp-flood', threshold:'threshold_udp_flood', voice:'membrane'},
  {name:'top-5-udp-amplification', threshold:'threshold_udp_amplification', voice:'metal'},
  {name:'top-5-tcp-flood', threshold:'threshold_tcp_flood', voice:'am'},
  {name:'top-5-tcp-amplification', threshold:'threshold_tcp_amplification', voice:'fm'}
];
var notes = ['C4','D4','E4','F4','G4','A4','B4','C5'];
function sonify(now) {
  var sounds = {};
  var max = {};
  metrics.forEach(function(metric) {
    let vals = db.trend.trends[metric.name];
    let topn = vals[vals.length - 1];
    let thresh = db.trend.values[metric.threshold];
    let chord = sounds[metric.voice];
    if(!chord) {
      chord = {};
      sounds[metric.voice] = chord;
    }
    for(var key in topn) {
      let [tgt,group,port] = key.split(',');
      let note = notes[port % notes.length];
      chord[note] = Math.max(chord[note] || 0, Math.min(1,topn[key] / thresh));
      max[metric.voice] = Math.max(max[metric.voice] || 0, chord[note]);
    };
  });
  var interval = Tone.Time(loopInterval).toSeconds();
  var delay = 0;
  for(let voice in sounds) {
    let synth = voices[voice];
    let chord = sounds[voice];
    let maxval = max[voice];
    if(maxval) {
      let volume = Math.min(0,(maxval - 1) * 20);
      synth.volume.value=volume;
      let note_array = [];
      for(let note in chord) {
        let val = chord[note];
        if((val / maxval) < 0.7) continue;
        note_array.push(note);
      }
      let duration = Tone.Time(maxval*interval).quantize('64n');
      if(duration > 0) synth.triggerAttackRelease(note_array,duration,now+delay);
    }
    delay += Tone.Time('16n').toSeconds();
  }
}
The metrics array identifies individual DDoS metrics and their related thresholds and associates them with a sound (voice). The sonify() function retrieves current values of each of the metrics and scales them by their respective threshold. Each metric value is mapped to a musical note based on the TCP/UDP port used in the attack. Different attack types are mapped to different voices, for example, a udp_amplification attack will have a metallic sound while a udp_flood attack will have a percussive sound. Volume and duration of notes are proportional to the intensity of the attack.

The net effect in a production network is of a quiet rythm of instruments. When a DDoS attack occurs, the notes associated with the particular attack become much louder and drown out the background sounds. Over time it is possible to recoginize the distinct sounds on each type of DDoS attack.

Tuesday, August 23, 2022

NVIDIA ConnectX SmartNICs

NVIDIA ConnectX SmartNICs offer best-in-class network performance, serving low-latency, high-throughput applications with one, two, or four ports at 10, 25, 40, 50, 100, 200, and up to 400 gigabits per second (Gb/s) Ethernet speeds.

This article describes how use the instrumentation built into ConnectX SmartNICs for data center wide network visibility. Real-time network telemetry for automation provides some background, giving an overview of the sFlow industry standard with an example of troubleshooting a high performance GPU compute cluster.

Linux as a network operating system describes how standard Linux APIs are used in NVIDIA Spectrum switches to monitor data center network performance. Linux Kernel Upstream Release Notes v5.19 describes recent driver enhancements for ConnectX SmartNICs that extend visibility to servers for end-to-end visibility into the performance of high performance distributed compute infrastructure.

The open source Host sFlow agent uses standard Linux APIs to configure instrumentation in switches and hosts, streaming the resulting measurements to analytics software in real-time for comprehensive data center wide visibility.

Packet sampling provides detailed visibility into traffic flowing across the network. Hardware packet sampling makes it possible to monitor 400 gigabits per second interfaces on the server at line rate with minimal CPU/memory overhead.
psample { group=1 egress=on }
dent { sw=off switchport=^eth[0-9]+$ }
The above Host sFlow configuration entries enable packet sampling on the host. Linux 4.11 kernel extends packet sampling support describes the Linux PSAMPLE netlink channel used by the network adapter to send packet samples to the Host sFlow agent. The dent module automatically configures hardware packet sampling on network interfaces matching the switchport pattern (eth0, eth1, .. in this example) using the Linux tc-sample API, directing the packet samples to be sent to the specified psample group
Visibility into dropped packets offers significant benefits for network troubleshooting, providing real-time network-wide visibility into the specific packets that were dropped as well the reason the packet was dropped. This visibility instantly reveals the root cause of drops and the impacted connections.
dropmon { group=1 start=on sw=on hw=on }
The above Host sFlow configuration entry enables hardware dropped packet monitoring on the network adapter hardware.
sflow {
  collector { ip=10.0.0.1 }
  psample { group=1 egress=on }
  dropmon { group=1 start=on sw=on hw=on }
  dent { sw=off switchport=^eth[0-9]+$ }
}
The above /etc/hsflowd.conf file shows a complete configuration. The centralized collector, 10.0.0.1, receives sFlow from all the servers and switches in the network to provide comprehensive end-to-end visibility.
The sFlow-RT analytics engine receives and analyzes the sFlow telemetry stream, providing real-time analytics to visibility and automation systems (e.g. Flow metrics with Prometheus and Grafana).

Tuesday, August 9, 2022

DDoS detection with advanced real-time flow analytics

The diagram shows two high bandwidth flows of traffic to the Customer Network, the first (shown in blue) is a bulk transfer of data to a big data application, and the second (shown in red) is a distributed denial of service (DDoS) attack in which large numbers of compromised hosts attempt to flood the link connecting the Customer Network to the upstream Transit Provider. Industry standard sFlow telemetry from the customer router streams to an instance of the sFlow-RT real-time analytics engine which is programmed to detect (and mitigate) the DDoS attack.

This article builds on the Docker testbed to demonstrate how advanced flow analytics can be used to separate the two types of traffic and detect the DDoS attack.

docker run --rm -d -e "COLLECTOR=host.docker.internal" -e "SAMPLING=100" \
--net=host -v /var/run/docker.sock:/var/run/docker.sock:ro \
--name=host-sflow sflow/host-sflow
First, start a Host sFlow agent using the pre-built sflow/host-sflow image to generate the sFlow telemetry that would stream from the switches and routers in a production deployment. 
setFlow('ddos_amplification', {
  keys:'ipdestination,udpsourceport',
  value: 'frames',
  values: ['count:ipsource']
});
setThreshold('ddos_amplification', {
  metric:'ddos_amplification',
  value: 10000,
  byFlow:true,
  timeout: 2
});
setEventHandler(function(event) {
  var [ipdestination,udpsourceport] = event.flowKey.split(',');
  var [sourcecount] = event.values;
  if(sourcecount === 1) {
    logInfo("bulk transfer to " + ipdestination);
  } else {
    logInfo("DDoS port " + udpsourceport + " against " + ipdestination);
  }
},['ddos_amplification']);
The ddos.js script above provides a simple demonstration of sFlow-RT's advanced flow analytics. The setFlow() function defines the a flow signature for detecting UDP amplification attacks, identifying the targetted IP address and the amplification protocol. In addition to the primary value of frames per second, a secondary value counting the number of ipsource addresses has been included. The setThreshold() function causes an event to by generated whenever a flow exceeds 10,000 frames per second. Finally, the setEventHandler() function defines how the events will be processed. See Writing Applications for more information on developing sFlow-RT applications.
docker run --rm -v $PWD/ddos.js:/sflow-rt/ddos.js \
-p 8008:8008 -p 6343:6343/udp --name sflow-rt \
sflow/prometheus -Dscript.file=ddos.js
Start sFlow-RT using pre-built sflow/prometheus image.
docker run --rm -it sflow/hping3 --flood --udp -k \
-p 443 host.docker.internal
In a separate window, simulate a bulk tranfer using pre-built sflow/hping3 image (use CTL+C to stop the attack).
2022-08-09T00:03:20Z INFO: bulk transfer to 192.168.65.2
The transfer will be immediately detected and logged in the sFlow-RT window.
docker run --rm -it sflow/hping3 --flood --udp -k \
--rand-source -s 53 host.docker.internal
Simulate a UDP amplification attack.
2022-08-09T00:05:19Z INFO: DDoS port 53 against 192.168.65.2
The attack will be immmediately detected and logged in the sFlow-RT window.

The open source sFlow-RT ddos-protect application is a full featured DDoS mitigation solution that uses the advanced flow analytics features described in this article to detect a wide range of volumetric attacks. In addition, ddos-protect can automatically mitigate attacks using BGP remotely triggered blackhole (RTBH) or BGP Flowspec actions. DDoS protection quickstart guide describes how to test, deploy, and monitor the DDoS mitigation solution with examples using Arista, Cisco, and Juniper routers.

Friday, July 1, 2022

SR Linux in Containerlab

This article uses Containerlab to emulate a simple network and experiment with Nokia SR Linux and sFlow telemetry. Containerlab provides a convenient method of emulating network topologies and configurations before deploying into production on physical switches.

curl -O https://raw.githubusercontent.com/sflow-rt/containerlab/master/srlinux.yml

Download the Containerlab topology file.

containerlab deploy -t srlinux.yml

Deploy the topology.

docker exec -it clab-srlinux-h1 traceroute 172.16.2.2

Run traceroute on h1 to verify path to h2.

traceroute to 172.16.2.2 (172.16.2.2), 30 hops max, 46 byte packets
 1  172.16.1.1 (172.16.1.1)  2.234 ms  *  1.673 ms
 2  172.16.2.2 (172.16.2.2)  0.944 ms  0.253 ms  0.152 ms

Results show path to h2 (172.16.2.2) via router interface (172.16.1.1).

docker exec -it clab-srlinux-switch sr_cli

Access SR Linux command line on switch.

Using configuration file(s): []
Welcome to the srlinux CLI.
Type 'help' (and press <ENTER>) if you need any help using this.
--{ + running }--[  ]--                                                                                                                                                              
A:switch#

SR Linux CLI describes how to use the interface.

A:switch# show system sflow status

Get status of sFlow telemetry.

-------------------------------------------------------------------------
Admin State            : enable
Sample Rate            : 10
Sample Size            : 256
Total Samples          : <Unknown>
Total Collector Packets: 0
-------------------------------------------------------------------------
  collector-id     : 1
  collector-address: 172.100.100.5
  network-instance : default
  source-address   : 172.100.100.6
  port             : 6343
  next-hop         : <Unresolved>
-------------------------------------------------------------------------
-------------------------------------------------------------------------
--{ + running }--[  ]--

The output shows settings and operational status for sFlow configured on the switch.

Connect to the sFlow-RT Metric Browser application, http://localhost:8008/app/browse-metrics/html/index.html?metric=ifinoctets.

docker exec -it clab-srlinux-h1 iperf3 -c 172.16.2.2

Run an iperf3 throughput test between h1 and h2.

Connecting to host 172.16.2.2, port 5201
[  5] local 172.16.1.2 port 38522 connected to 172.16.2.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   675 KBytes  5.52 Mbits/sec   54   22.6 KBytes       
[  5]   1.00-2.00   sec   192 KBytes  1.57 Mbits/sec   18   12.7 KBytes       
[  5]   2.00-3.00   sec   255 KBytes  2.09 Mbits/sec   16   1.41 KBytes       
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    9   1.41 KBytes       
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec   18   1.41 KBytes       
[  5]   5.00-6.00   sec   255 KBytes  2.08 Mbits/sec   17   1.41 KBytes       
[  5]   6.00-7.00   sec   191 KBytes  1.56 Mbits/sec   10   8.48 KBytes       
[  5]   7.00-8.00   sec   191 KBytes  1.56 Mbits/sec    7   7.07 KBytes       
[  5]   8.00-9.00   sec   191 KBytes  1.56 Mbits/sec    7   8.48 KBytes       
[  5]   9.00-10.00  sec   191 KBytes  1.56 Mbits/sec   10   7.07 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.09 MBytes  1.75 Mbits/sec  166             sender
[  5]   0.00-10.27  sec  1.59 MBytes  1.30 Mbits/sec                  receiver

iperf Done.

The iperf3 test results show that the SR Linux dataplane implementation in the Docker container has severely limited forwarding performance, making the image unsuitable for emulating network traffic. In addition, the emulated forwarding plane does not support the packet sampling mechanism available on hardware switches that allows sFlow to provide real-time visibility into traffic flowing through the switch.

For readers interested in performance monitoring, the https://github.com/sflow-rt/containerlab repository provides examples showing performance monitoring of data center leaf / spine fabrics, EVPN visibility, and DDoS mitigation using Containerlab. These examples use Linux as a network operating system. In this case, the containers running on Containerlab use the Linux dataplane for maximumum performance. On physical switch hardware the Linux kernel can offload dataplane forwarding to the switch ASIC for line rate performance. 

containerlab destroy -t srlinux.yml

When you are finished, run the above command to stop the containers and free the resources associated with the emulation.

Thursday, June 2, 2022

Using Ixia-c to test RTBH DDoS mitigation

Remote Triggered Black Hole Scenario describes how to use the Ixia-c traffic generator to simulate a DDoS flood attack. Ixia-c supports the Open Traffic Generator API that is used in the article to program two traffic flows: the first representing normal user traffic (shown in blue) and the second representing attack traffic (show in red).

The article goes on to demonstrate the use of remotely triggered black hole (RTBH) routing to automatically mitigate the simulated attack. The chart above shows traffic levels during two simulated attacks. The DDoS mitigation controller is disabled during the first attack. Enabling the controller for the second attack causes to attack traffic to be dropped the instant it crosses the threshold.

The diagram shows the Containerlab topology used in the Remote Triggered Black Hole Scenario lab (which can run on a laptop). The Ixia traffic generator's eth1 interface represents the Internet and its eth2 interface represents the Customer Network being attacked. Industry standard sFlow telemetry from the Customer router, ce-router, streams to the DDoS mitigation controller (running an instance of DDoS Protect). When the controller detects a denial of service attack it pushed a control via BGP to the ce-router, which in turn pushes the control upstream to the service provider router, pe-router, which drops the attack traffic to prevent flooding of the ISP Circuit that would otherwise disrupt access to the Customer Network.

Arista, Cisco, and Juniper have added sFlow support to their BGP routers, see Real-time flow telemetry for routers, making it straightforward to take this solution from the lab to production. Support for Open Traffic Generator API across a range of platforms makes it possible to develop automated tests in the lab environment and apply them to production hardware.

Tuesday, April 26, 2022

BGP Remotely Triggered Blackhole (RTBH)

DDoS attacks and BGP Flowspec responses describes how to simulate and mitigate common DDoS attacks. This article builds on the previous examples to show how BGP Remotely Triggered Blackhole (RTBH) controls can be applied in situations where BGP Flowpsec is not available, or is unsuitable as a mitigation response.
docker run --rm -it --privileged --network host --pid="host" \
  -v /var/run/docker.sock:/var/run/docker.sock -v /run/netns:/run/netns \
  -v ~/clab:/home/clab -w /home/clab \
  ghcr.io/srl-labs/clab bash
Start Containerlab.
curl -O https://raw.githubusercontent.com/sflow-rt/containerlab/master/ddos.yml
Download the Containerlab topology file.
sed -i "s/\\.ip_flood\\.action=filter/\\.ip_flood\\.action=drop/g" ddos.yml
Change mitigation policy for IP Flood attacks from Flowspec filter to RTBH.
containerlab deploy -t ddos.yml
Deploy the topology.
Access the DDoS Protect screen at http://localhost:8008/app/ddos-protect/html/
docker exec -it clab-ddos-attacker hping3 \
--flood --rawip -H 47 192.0.2.129
Launch an IP Flood attack. The DDoS Protect dashboard shows that as soon as the ip_flood attack traffic reaches the threshold a control is implemented and the attack traffic is immediately dropped. The entire process between the attack being launched, detected, and mitigated happens within a second, ensuring minimal impact on network capacity and services.
docker exec -it clab-ddos-sp-router vtysh -c "show running-config"
See sp-router configuration.
Building configuration...

Current configuration:
!
frr version 8.2.2_git
frr defaults datacenter
hostname sp-router
no ipv6 forwarding
log stdout
!
ip route 203.0.113.2/32 Null0
!
interface eth2
 ip address 198.51.100.1/24
exit
!
router bgp 64496
 bgp bestpath as-path multipath-relax
 bgp bestpath compare-routerid
 neighbor fabric peer-group
 neighbor fabric remote-as external
 neighbor fabric description Internal Fabric Network
 neighbor fabric ebgp-multihop 255
 neighbor fabric capability extended-nexthop
 neighbor eth1 interface peer-group fabric
 no neighbor eth1 capability extended-nexthop
 !
 address-family ipv4 unicast
  redistribute connected route-map HOST_ROUTES
  neighbor fabric route-map RTBH in
 exit-address-family
 !
 address-family ipv4 flowspec
  neighbor fabric activate
 exit-address-family
exit
!
bgp community-list standard BLACKHOLE seq 5 permit blackhole
!
route-map HOST_ROUTES permit 10
 match interface eth2
exit
!
route-map RTBH permit 10
 match community BLACKHOLE
 set ip next-hop 203.0.113.2
exit
!
route-map RTBH permit 20
exit
!
ip nht resolve-via-default
!
end
The configuration creates null route for 203.0.113.2/32 and rewrites the next-hop address to 203.0.113.2 for routes that are marked with the BGP blackhole community.
docker exec -it clab-ddos-sp-router vtysh -c "show ip route"
Show the forwarding state on sp-router.
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

K>* 0.0.0.0/0 [0/0] via 172.100.100.1, eth0, 12:36:08
C>* 172.100.100.0/24 is directly connected, eth0, 12:36:08
B>* 192.0.2.0/24 [20/0] via fe80::a8c1:abff:fe32:b21e, eth1, weight 1, 12:36:03
B>  192.0.2.129/32 [20/0] via 203.0.113.2 (recursive), weight 1, 00:00:04
  *                         unreachable (blackhole), weight 1, 00:00:04
C>* 198.51.100.0/24 is directly connected, eth2, 12:36:08
S>* 203.0.113.2/32 [1/0] unreachable (blackhole), weight 1, 12:36:08
Traffic to the victim IP address, 192.0.2.129, is directed to the 203.0.113.2 next-hop, where it is discarded before it can saturate the link to the customer router, ce-router.