Tuesday, October 20, 2020

Docker DDoS testbed


Docker testbed describes how to use Docker Desktop to build a test network to experiment with real-time sFlow streaming telemetry and analytics. This article extends the testbed to experiment with distributed denial of service (DDoS) detection and mitigation techniques described in Real-time DDoS mitigation using BGP RTBH and FlowSpec.

Start a Host sFlow agent using the pre-built sflow/host-sflow image:
docker run --rm -d -e "COLLECTOR=host.docker.internal" -e "SAMPLING=10" \
--net=host -v /var/run/docker.sock:/var/run/docker.sock:ro \
--name=host-sflow sflow/host-sflow
Start ExaBGP using the pre-built sflow/exabgp image. ExaBGP connects to the sFlow-RT analytics software and displays BGP RTBH / Flowspec controls sent by sFlow-RT:
docker run --rm sflow/exabgp
In a second terminal window, start an instance of the sFlow-RT analytics software using the pre-built sflow/ddos-protect image:
GW=`docker network inspect bridge -f '{{range .IPAM.Config}}{{.Gateway}}{{end}}'`

SUBNET=`docker network inspect bridge -f '{{range .IPAM.Config}}{{.Subnet}}{{end}}'`

docker run --rm -p 6343:6343/udp -p 8008:8008 -p 1179:1179 --name=sflow-rt \
sflow/ddos-protect -Dddos_protect.router=$GW -Dddos_protect.as=65001 \
-Dddos_protect.enable.flowspec=yes -Dddos_protect.group.local=$SUBNET \
-Dddos_protect.mode=automatic \
-Dddos_protect.udp_amplification.action=filter \
-Dddos_protect.udp_amplification.threshold=5000
Open the sFlow-RT dashboard at http://localhost:8008/
The sFlow Agents gauge confirms that sFlow is being received from the Host sFlow agent. Now access the DDoS Protect application at http://localhost:8008/app/ddos-protect/html/index.html
The BGP chart at the bottom right verifies that BGP connection has been established so that controls can be sent to ExaBGP, which will display them in the terminal window.

Finally, hping3 can be used to generate simulated DDoS attacks. Start a simulated DNS amplification attack using the pre-built sflow/hping3 image:
GW=`docker network inspect bridge -f '{{range .IPAM.Config}}{{.Gateway}}{{end}}'`

docker run --rm sflow/hping3 --flood --udp -k -a 198.51.100.1 -s 53 $GW
The attack shows up immediately in the DDoS protect dashboard, http://localhost:8008/app/ddos-protect/html/index.html
The udp_amplification chart shows the traffic rising to cross the threshold and trigger the control shown in the Controls chart.
The exabgp log shows the Flowspec rule that was sent to block the attack, filtering traffic to 172.17.0.1/32 with UDP source port 53. Type CNTRL+C in the hping3 window to end the attack.

This testbed provides a convenient way to become familiar with the tools to automatically mitigate DDoS attacks. The following articles provide additional information on moving the solution into production: Real-time DDoS mitigation using BGP RTBH and FlowSpecPushing BGP Flowspec rules to multiple routers, and Monitoring DDoS mitigationReal-time network and system metrics as a service provides background on the sFlow-RT analytics platform running the DDoS Protect application.

Wednesday, October 7, 2020

Broadcom Mirror on Drop (MoD)

Networking Field Day 23 included a presentation by Bhaskar Chinni describing Broadcom's Mirror-on-Drop (MOD) capability. MOD capable hardware can generate a notification whenever a packet is dropped by the ASIC, reporting the packet header and the reason that the packet was dropped. MOD is supported by Trident 3, Tomahawk 3,  and Jericho 2 or later ASICs that are included in popular data center switches and widely deployed in data centers.

The recently published sFlow Dropped Packet Notification Structures specification adds drop notifications to industry standard sFlow telemetry export, complementing the existing push based counter and packet sampling measurements. The inclusion of drop monitoring in sFlow will allow the benefits of MOD to be fully realized, ensuring consistent end-to-end visibility into dropped packets across multiple vendors and network operating systems.

Using Advanced Telemetry to Correlate GPU and Network Performance Issues demonstrates how packet drop notifications from NVIDA Mellanox switches forms part of an integrated sFlow telemetry stream that provides the system wide observability needed to drive automation.

MOD instrumentation on Broadcom based switches provides the foundation needed for network vendors to integrate the functionality with sFlow agents to add dropped packet notifications. 

Tuesday, October 6, 2020

Using Advanced Telemetry to Correlate GPU and Network Performance Issues


The image above was captured from the recent talk Using Advanced Telemetry to Correlate GPU and Network Performance Issues [A21870] presented at the NVIDIA GTC conference. The talk includes a demonstration of monitoring a high performance GPU compute cluster in real-time. The real-time dashboard provides an up to the second view of key performance metrics for the cluster.

This diagram shows the elements of the GPU compute cluster that was demonstrated. Cumulus Linux running on the switches reduces operational complexity by allowing you to run the same Linux operating system on the network devices as is run on the compute servers. sFlow telemetry is generated by the open source Host sFlow agent that runs on the servers and the switches, using standard Linux APIs to enable instrumentation and gather measurements. On switches, the measurements are offloaded to the ASIC to provide line rate monitoring.

Telemetry from all the switches and servers in the cluster is streamed to an sFlow-RT analyzer, which builds a real-time view of performance that can be used to drive operational dashboards and automation.

The Real-time GPU and network telemetry dashboard combines measurements from all the devices to provide view of cluster performance. Each of the three charts demonstrated a different type of measurement in the sFlow telemetry stream:
  1. GPU Utilization is based on sFlow's counter push mechanism, exporting NVIDIA Management Library (NVML) counters. This chart trends buffer, memory, and execution utilization of the GPUs in the cluster.
  2. Network Traffic is based on sFlow's random packet sampling mechanism, supported by the Linux kernel on servers, and offloaded to the Mellanox ASIC on the switches. This chart trends the top network flows crossing the network.
  3. Network Drops is based on sFlow's recently added dropped packet notification mechanism, see Using sFlow to monitor dropped packets. This chart trends dropped packet source and destination addresses and the reason the packet was dropped.
The cluster is running a video transcoding workload in which video is streamed across the network to a GPU where it is transcoded, and the result returned. A normal transcoding task is shown on left, where the charts show an increase in GPU and network activity and zero dropped packets. A failed transcoding task is shown in the middle. Here the GPU activity is low, there is no network activity, and there is a sequence of packets dropped by an access control list (ACL). Removing the ACL fixes the problem, which is confirmed by the new data shown on the right of the trend charts.

The sFlow data model integrates the three telemetry streams: counters, packet samples, and drop notifications. Each type of data is useful on its own, but together they provide the system wide observability needed to drive automation.

Tuesday, September 8, 2020

Cumulus Linux 4.2

Cumulus Linux is a network operating system for open networking hardware. Cumulus VX is a free virtual appliance that allows network engineers to experiment with Cumulus Linux and verify configurations before deploying into production. 
The Cumulus VX documentation describes how to build network topologies in KVM, VirtualBox, using VMWare hypervisors. If you want to run virtual machines locally, Cumulus in the Cloud is a free service that will allow you to access pre-built networks in the public cloud.

A key feature of Cumulus Linux is the use of the Linux kernel as the authoritative repository of network state. A result of this approach is that the behavior of a Cumulus Linux VX virtual appliance is the same as Cumulus Linux running on a hardware switch. For example, the open source FRR routing daemon shipped with Cumulus Linux uses the Linux netlink API to push routes to the kernel, which forwards packets in the virtual appliance. On a physical switch, routes are still pushed to the kernel, but kernel routing configuration is then offloaded to the switch ASIC so that packets bypass the kernel and are routed by hardware.

Cumulus Linux includes the open source Host sFlow agent. Here again, standard Linux APIs are used to implement sFlow packet sampling (see Linux 4.11 kernel extends packet sampling support). On the virtual appliance, packet sampling is performed by the Linux kernel. On a hardware switch packet sampling is offloaded to the switch ASIC. In both cases streaming sFlow telemetry provides visibility into packet forwarding.

This article demonstrates how to configure and enable sFlow Cumulus Linux (see Monitoring System Statistics and Network Traffic with sFlow). If you don't have access to a switch, download Cumulus VX to follow the example.

First log into the switch.
ssh cumulus@leaf01
Next, edit configuration file:
sudo vi /etc/hsflowd.conf
Change the following configuration setting to send sFlow to a collector at address 10.0.0.30:
sflow {
...
  collector { ip=10.0.0.30 }
...
}
Start the hsflowd daemon:
sudo systemctl enable hsflowd@mgmt.service
sudo systemctl start hsflowd@mgmt.service
In this case the collector is on the out of band management network and so the daemon needs to run in the management VRF (see Management VRF).

Use the following commands instead to send sFlow to an in-band collector on the default VRF:
sudo systemctl enable hsflowd.service
sudo systemctl start hsflowd.service
Note: Automating sFlow configuration is straightforward since the configurations are not switch specific so that every switch in the network can be given the same configuration.

Docker Desktop provides a convenient method of running sFlow analytics software on the collector machine (10.0.0.30).
docker run --rm -p 6343:6343/udp sflow/sflowtool
Use the sflow/sflowtool image to verify that sFlow telemetry is being received.
docker run --rm -p 8008:8008 -p 6343:6343/udp sflow/prometheus
Use the sflow/prometheus image to run the sFlow-RT real-time analyzer packaged with tools to browse flows and metrics and export data to Prometheus and Grafana (see sFlow-RT Network Interfaces, sFlow-RT Countries and Networks, and sFlow-RT Health dashboards for examples). Open the URL http://localhost:8008/ to access the web interface.

RESTful control of Cumulus Linux ACLs describes an open source extension to the Cumulus Linux REST API that has been used to automate control actions based on real-time traffic analytics: Triggered remote packet capture using filtered ERSPAN and DDoS mitigation with Cumulus Linux.

Cumulus VX provides a convenient platform for developing and testing monitoring and control strategies before deploying them into a production network. The consistency between the behavior of the Cumulus VX virtual appliance and Cumulus Linux running on a physical switch assures a seamless transition.

Tuesday, August 11, 2020

Real-time trending of dropped packets

Discard Browser is a recently released open source application running on the sFlow-RT real-time analytics engine. The software uses streaming analytics to trend dropped packets.
Using sFlow to monitor dropped packets describes the recently added packet drop monitoring functionality added to the open source Host sFlow agent. The article describes how to install and configure the agent on Linux-based platforms and stream industry standard sFlow telemetry to an sFlow collector.

Visibility into dropped packets describes instrumentation, recently added to the Linux kernel, that provides visibility into packets dropped by the kernel data path on a host, or dropped by a switch ASIC when packets are forwarded in hardware.  Extending sFlow to provide visibility into dropped packets offers significant benefits for network troubleshooting, providing real-time network-wide visibility into the specific packets that were dropped as well the reason the packet was dropped. This visibility instantly reveals the root cause of drops and the impacted connections.

Packet discard monitoring complements sFlow's existing counter polling and packet sampling mechanisms and shares a common data model so that all three sources of data can be correlated.  For example, if packets are being discarded because of buffer exhaustion, the discard records don't necessarily tell the whole story. The discarded packets may represent mice flows that are victims of an elephant flow. Packet samples will reveal the traffic that isn't being dropped and provide a more complete picture. Counter data adds additional information such as CPU load, interface speed, link utilization, packet and discard rates that further completes the picture.

A simple way to run Discard Browser, pre-packaged with sFlow-RT collector, is to use the Docker sflow/prometheus image:
docker run --rm -p 6343:6343/udp -p 8008:8008 sflow/prometheus
Access the web interface at http://localhost:8008/.

The screen shot at the top of the page shows dropped connection attempts to the Linux host running the sFlow agent. The connection attempt were generated using curl:
curl http://192.168.1.87:8080
The Docker image also includes Metric Browser and Flow Browser applications that allow the full range of sFlow telemetry to be explored - see Real-time network and system metrics as a service for examples.

The included Prometheus exporter allows data to be pulled into a time series database to be included in dashboards. The Grafana sFlow-RT Network InterfacessFlow-RT Countries and NetworkssFlow-RT DDoS Protect, and sFlow-RT Health dashboards provide examples.

Friday, July 31, 2020

Using sFlow to monitor dropped packets

Visibility into dropped packets describes instrumentation, recently added to the Linux kernel, that provides visibility into packets dropped by the kernel data path on a host, or dropped by a switch ASIC when packets are forwarded in hardware. This article describes integration of drop monitoring in the open source Host sFlow agent and inclusion of drop reporting as part of industry standard sFlow telemetry.

Extending sFlow to provide visibility into dropped packets offers significant benefits for network troubleshooting, providing real-time network-wide visibility into the specific packets that were dropped as well the reason the packet was dropped. This visibility instantly reveals the root cause of drops and the impacted connections.

Packet discard monitoring complements sFlow's existing counter polling and packet sampling mechanisms and shares a common data model so that all three sources of data can be correlated.  For example, if packets are being discarded because of buffer exhaustion, the discard records don't necessarily tell the whole story. The discarded packets may represent mice flows that are victims of an elephant flow. Packet samples will reveal the traffic that isn't being dropped and provide a more complete picture. Counter data adds additional information such as CPU load, interface speed, link utilization, packet and discard rates that further completes the picture.

The following steps build the Host sFlow agent with drop monitoring on an Ubuntu 20 system (a Linux 5.4 kernel or newer is required):
git clone https://github.com/sflow/host-sflow
cd host-sflow
make FEATURES="HOST PCAP DROPMON"
sudo make install
sudo make schedule
Next, edit the /etc/hsflowd.conf file, in this example, directing sFlow to be sent to a collector at 192.168.1.242, enabling packet sampling on host adapter enp0s3, and enabling drop monitoring:
sflow {
  collector { ip=192.168.1.242 }
  pcap { dev = enp0s3 }
  dropmon { group = 1 start = on }
}
Start the agent:
sudu systemctl enable hsflowd
sudo systemctl start hsflowd
Build the latest version of sflowtool on the collector host (192.168.1.242):
git clone https://github.com/sflow/sflowtool
cd sflowtool
./boot.sh
./configure
make
sudo make install
Now run sflowtool to receive and decode the sFlow telemetry stream:
sflowtool
The following example shows the output for a discarded TCP packet:
startSample ----------------------
sampleType_tag 0:5
sampleType DISCARD
sampleSequenceNo 20
sourceId 0:1
dropEvents 0
inputPort 1
outputPort 0
discardCode 289
discardReason unknown_l4
discarded_flowBlock_tag 0:1
discarded_flowSampleType HEADER
discarded_headerProtocol 1
discarded_sampledPacketSize 54
discarded_strippedBytes 0
discarded_headerLen 54
discarded_headerBytes 00-00-00-00-00-00-00-00-00-00-00-00-08-00-45-00-00-28-00-00-40-00-40-06-3C-CE-7F-00-00-01-7F-00-00-01-04-05-04-39-00-00-00-00-14-51-E2-8A-50-14-00-00-B2-B4-00-00
discarded_dstMAC 000000000000
discarded_srcMAC 000000000000
discarded_IPSize 40
discarded_ip.tot_len 40
discarded_srcIP 127.0.0.1
discarded_dstIP 127.0.0.1
discarded_IPProtocol 6
discarded_IPTOS 0
discarded_IPTTL 64
discarded_IPID 0
discarded_TCPSrcPort 1029
discarded_TCPDstPort 1081
discarded_TCPFlags 20
endSample   ----------------------
The sflowtool -T option converts the discarded packet records into PCAP format so that they can be decoded by packet analysis tools such as Wireshark and tcpdump:
sflowtool -T | tshark -r -
   12  22.000000 192.168.1.242 → 192.168.1.87 TCP 78 65527 → 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=64 TSval=1324841769 TSecr=0 SACK_PERM=1
The article sFlow to JSON uses Python examples to demonstrate how sflowtool's ability to convert sFlow records into JSON can be used for further analysis.

Thursday, July 16, 2020

Visibility into dropped packets

Dropped packets have a profound impact on network performance and availability. Packet discards due to congestion can significantly impact application performance. Dropped packets due to black hole routes, expired TTLs, MTU mismatches, etc. can result in insidious connection failures that are time consuming and difficult to diagnose.

Devlink Trap describes recent changes to the Linux drop monitor service that provide visibility into packets dropped by switch ASIC hardware. When a packet is dropped by the ASIC, an event is generated that includes the header of the dropped packet and the reason why it was dropped. A hardware policer is used to limit the number of events generated by the ASIC to a rate that can be handled by the Linux kernel. The events are delivered to userspace applications using the Linux netlink service.

Running the dropwatch command line tool on an Ubuntu 20 system demonstrates the instrumentation:
pp@ubuntu20:~$ sudo dropwatch
Initializing null lookup method
dropwatch> set alertmode packet
Setting alert mode
Alert mode successfully set
dropwatch> start
Enabling monitoring...
Kernel monitoring activated.
Issue Ctrl-C to stop monitoring
drop at: __udp4_lib_rcv+0xae5/0xbb0 (0xffffffffb05ead95)
origin: software
input port ifindex: 2
timestamp: Wed Jul 15 23:57:36 2020 223253465 nsec
protocol: 0x800
length: 128
original length: 243

drop at: __netif_receive_skb_core+0x14f/0xc90 (0xffffffffb05351af)
origin: software
input port ifindex: 2
timestamp: Wed Jul 15 23:57:36 2020 530212717 nsec
protocol: 0x4
length: 52
original length: 52
The dropwatch package also includes the dwdump command, allowing the dropped packets to be analyzed using packet decoding tools such as tshark:
pp@ubuntu20:~$ sudo dwdump | tshark -r -
   12   0.777386 BelkinIn_a8:8a:7d → Spanning-tree-(for-bridges)_00 STP 196 Conf. Root = 61440/4095/24:f5:a2:a8:8a:7b  Cost = 0  Port = 0x8003
   13   2.825301 BelkinIn_a8:8a:7d → Spanning-tree-(for-bridges)_00 STP 196 Conf. Root = 61440/4095/24:f5:a2:a8:8a:7b  Cost = 0  Port = 0x8003
Linux is the base for many open source and commercial network operating systems used today (e.g. SONiC, DENT, Cumulus, EOS, NX-OS, and IOS-XR). A standard interface for monitoring packet drops is an exciting development, allowing Linux based network monitoring agents to collect, export, and analyze drop events to rapidly identify the root cause of dropped packets.