Monday, March 1, 2021

DDoS Mitigation with Juniper, sFlow, and BGP Flowspec

Real-time DDoS mitigation using BGP RTBH and FlowSpec, DDoS protection of local address space, Pushing BGP Flowspec rules to multiple routersMonitoring DDoS mitigation, and Docker DDoS testbed demonstrate how sFlow and BGP Flowspec are combined by the DDoS Protect application running on the sFlow-RT real-time analytics engine to automatically detect and block DDoS attacks.

This article discusses how to deploy the DDoS Protect application in a Juniper Networks environment. Juniper has a long history of supporting BGP Flowspec on their routing platforms and Juniper has added support for sFlow to their entire product range, see sFlow available on Juniper MX series routers.

First, Junos doesn't provide a way to connect to the non-standard BGP port (1179) that sFlow-RT uses by default. Allowing sFlow-RT to open the standard BGP port (179) requires that the service be given additional Linux capabilities. 

docker run --rm --net=host --sysctl net.ipv4.ip_unprivileged_port_start=0 \
sflow/ddos-protect -Dbgp.port=179

The above command launches the prebuilt sflow/ddos-protect Docker image. Alternatively, if sFlow-RT has been installed as a deb / rpm package, then the required permissions can be added to the service.

sudo systemctl edit sflow-rt.service
Type the above command to edit the service configuration and add the following lines:
[Service]
AmbientCapabilities=CAP_NET_BIND_SERVICE
Next, edit the sFlow-RT configuration file for the DDoS Protect application:
sudo vi /usr/local/sflow-rt/conf.d/ddos-protect.conf
and add the line:
bgp.port=179
Finally, restart sFlow-RT:
sudo systemctl restart sflow-rt
The application is now listening for BGP connections on TCP port 179.

Now configure the router to send sFlow telemetry to sFlow-RT - see Junos: sFlow Monitoring Technology
set protocols sflow collector 192.168.65.2 udp-port 6343
set protocols sflow polling-interval 20
set protocols sflow sample-rate ingress 1000
set protocols sflow interfaces ge-0/0/0
set protocols sflow interfaces ge-0/0/1
...
For example, the above commands enable sFlow monitoring on a Juniper MX router. See sFlow-RT Agents for recommended sFlow configuration settings.

Also configure a BGP Flowspec session with sFlow-RT - see Junos: Multiprotocol BGP.
policy-options {
    policy-statement ACCEPT_ALL {
        from protocol bgp;
        then accept;
    }
}
routing-options {
    router-id 1.1.1.1;
    autonomous-system 65000;
}
protocols {
    bgp {
        group sflow-rt {
            type internal;
            local-address 172.17.0.2;
            family inet {
                unicast;
                flow {
                    no-validate ACCEPT_ALL;
                }
            }
            family inet6 {
                unicast;
                flow {
                    no-validate ACCEPT_ALL;
                }
            }
            neighbor 192.168.65.2 {
                import ACCEPT_ALL;
                peer-as 65000;
            }
        }
    }
}
The above configuration establishes the BGP Flowspec session with sFlow-RT.

Real-time DDoS mitigation using BGP RTBH and FlowSpec describes how to simulate a DDoS UDP amplification attack in order to test the automated detection and control functionality.  
root@07358a106c21> show route table inetflow.0 detail    

inetflow.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
192.0.2.129,*,proto=17,srcport=53/term:N/A (1 entry, 0 announced)
        *BGP    Preference: 170/-101
                Next hop type: Fictitious, Next hop index: 0
                Address: 0x55653aae979c
                Next-hop reference count: 1
                Next hop: 
                State: <Active Int Ext SendNhToPFE>
                Local AS: 65000 Peer AS: 65000
                Age: 6 
                Validation State: unverified 
                Task: BGP_65000.192.168.65.2
                AS path: I 
                Communities: traffic-rate:0:0
                Accepted
                Localpref: 100
                Router ID: 0.6.6.6
Command line output from the router shown above verifies that a Flowspec control blocking the amplification attack has been received. The control will remain in place for 60 minutes (the configured timeout), after which it will be automatically withdrawn. If the attack is still in progress it will be immediately detected and the control reapplied.

DDoS Protect can mitigate a wide range of common attacks, including: NTP, DNS, Memcached, SNMP, and SSDP amplification attacks; IP, UDP, ICMP and TCP flood attacks; and IP fragmentation attacks. Mitigation options include: remote triggered black hole (RTBH), filtering, rate limiting, DSCP marking, and redirection. IPv6 is fully supported in detection and mitigation of each of these attack types.

Monday, January 25, 2021

Topology


Real-time network and system metrics as a service describes how to use data captured from the network shown above to explore the functionality of sFlow-RT real-time analytics software. This article builds on the previous article to show how knowledge of network topology can be used to enhance analytics, see Topology for documentation.

First, follow the instructions in the previous example and start an instance of sFlow-RT using the captured sFlow.  
curl -O https://raw.githubusercontent.com/sflow-rt/fabric-view/master/demo/topology.json
Then, download the topology file for the example.
curl -X PUT -H "Content-Type: application/json" -d @topology.json \
http://localhost:8008/topology/json
Install the topology using the sFlow-RT REST API.
curl http://localhost:8008/topology/json
Retrieve the topology.
{
 "version": 0,
 "links": {
  "L1": {
   "node2": "spine1",
   "node1": "leaf1",
   "port1": "swp1s0",
   "port2": "swp49"
  },
  "L2": {
   "node2": "spine1",
   "node1": "leaf1",
   "port1": "swp1s1",
   "port2": "swp50"
  },
  "L3": {
   "node2": "spine2",
   "node1": "leaf1",
   "port1": "swp1s2",
   "port2": "swp51"
  },
  "L4": {
   "node2": "spine2",
   "node1": "leaf1",
   "port1": "swp1s3",
   "port2": "swp52"
  },
  "L5": {
   "node2": "spine2",
   "node1": "leaf2",
   "port1": "swp1s0",
   "port2": "swp49"
  },
  "L6": {
   "node2": "spine2",
   "node1": "leaf2",
   "port1": "swp1s1",
   "port2": "swp50"
  },
  "L7": {
   "node2": "spine1",
   "node1": "leaf2",
   "port1": "swp1s2",
   "port2": "swp51"
  },
  "L8": {
   "node2": "spine1",
   "node1": "leaf2",
   "port1": "swp1s3",
   "port2": "swp52"
  }
 }
}
The JSON topology structure contains the eight links connecting the leaf and spine switches in the diagram, identifying the nodes and ports associated with each link.
curl -H "Content-Type:application/json" -X PUT \
--data '{"keys":"ipsource,ipdestination",value:"bytes"}' \
http://localhost:8008/flow/srcdst/json
Now define the srcdst flow metric described in the previous article.
curl "http://localhost:8008/activeflows/TOPOLOGY/srcdst/json?aggMode=edge"
Knowledge of topology opens up additional options when querying for flows. For example, the above command only considers devices that are part of the topology and sums flows entering edge device access ports, i.e. traffic entering the the leaf switches from the servers.
[
 {
  "flowN": 1,
  "value": 248800.14028768288,
  "key": "10.4.3.2,10.4.4.2"
 },
 {
  "flowN": 1,
  "value": 176879.3798722214,
  "key": "10.4.1.2,10.4.2.2"
 },
 {
  "flowN": 1,
  "value": 526.0366052656848,
  "key": "10.4.4.2,10.4.3.2"
 },
 {
  "flowN": 1,
  "value": 375.06686598182193,
  "key": "10.4.2.2,10.4.1.2"
 }
]
The result accurately reports the amount of traffic being exchanged between the servers, discarding duplicate data reported as traffic flows traverse the links between switches.
-Dbrowse-flows.agents=TOPOLOGY -Dbrowse-flows.aggMode=edge
Adding the above arguments to the end of the command line used to start sFlow-RT configures the Flow Browser application to use the topology de-duplication method.
Click on the link below to plot a graph of the top IP Protocols using the browse-flows application (screen capture shown above):
http://localhost:8008/app/browse-flows/html/index.html?keys=ipprotocol&value=bps
Note: No data will be shown until the topology is posted to sFlow-RT.
function print(label,obj) {
  logInfo(label+"="+JSON.stringify(obj));
}

setFlow('protocol',{keys:'ipprotocol',value:'bytes'});

setIntervalHandler(function() {
  print("locate_mac",topologyLocateHostMac('000AF725C062'));
  print("locate_ip",topologyLocateHostIP('10.4.3.2'));
  print("flow_max",activeFlows('ALL','protocol',5,0,'max'));
  print("flow_sum",activeFlows('ALL','protocol',5,0,'sum'));
  print("flow_edge",activeFlows('TOPOLOGY','protocol',5,0,'edge'));
});
The demo.js script shown above uses sFlow-RT's embedded scripting API, see Writing Applications. The script defines the flow called protocol that tracks top IP Protocols and prints out the top flows with different aggregation methods. The script also demonstrates an additional capability made possible when topology is known. The topologyLocateHostMac() and topologyLocateHostIP() function locates an addresses to the edge port connecting them to the network.
-Dscript.file=$PWD/demo.js
Run the script by adding the above argument to the end of the command line used to run sFlow-RT.
2021-01-22T17:08:35-08:00 INFO: locate_mac=[{"ipaddress":"10.4.3.2","node":"leaf1","agent":"192.168.0.11","ifindex":"38","port":"swp32s1","mac":"000AF725C062"}]
2021-01-22T17:08:35-08:00 INFO: locate_ip=[{"ipaddress":"10.4.3.2","node":"leaf1","agent":"192.168.0.11","ifindex":"38","port":"swp32s1","mac":"000AF725C062"}]
2021-01-22T17:08:35-08:00 INFO: flow_max=[{"flowN":16,"agent":"192.168.0.14","value":1208583376.782055,"dataSource":"54","key":"6"}]
2021-01-22T17:08:35-08:00 INFO: flow_sum=[{"flowN":16,"value":6615963204.827695,"key":"6"}]
2021-01-22T17:08:35-08:00 INFO: flow_edge=[{"flowN":4,"value":2104039983.2917378,"key":"6"}]
The output from the script shows that the addresses were located to leaf1 port swp32s1. The flow_max and flow_sum queries don't use the topology and combine data from all 16 data sources (switch ports) that are reporting traffic. The sum mode returns the largest value since traffic is added for every data source. The max mode finds the data source reporting the largest value for the flow and reports that value (agent: 192.168.0.14, dataSource: 54). The edge mode is equivalent to the REST query used earlier.

Mininet is a network emulator that you can run on your laptop in a virtual machine (e.g. using Multipass) that provides a useful platform for building virtual topologies and exploring topology related analytics. Mininet dashboardMininet weathermap, and Mininet, ONOS, and segment routing provide examples.


Ideally the network configuration and topology will be available in a centralized repository that can be queried to generate the information required by sFlow-RT. Alternatively, Link Layer Discovery Protocol (LLDP) data retrieved from network devices can be used to construct the topology. Fabric Visibility, Arista EOS CloudVision, and Fabric visibility with Cumulus Linux provide examples.

Thursday, December 10, 2020

FRR

FRRouting (FRR) is an IP routing protocol suite for Linux and Unix platforms which includes protocol daemons for BGP, IS-IS, LDP, OSPF, PIM, and RIP.

This article uses the Docker DDoS testbed to explore FRRouting's BGP Flowspec capability. Follow the steps in the Docker DDoS testbed article, but instead of running ExaBGP, run the following command to start the pre-built sflow/frr image:
docker % docker run --rm -it --privileged --name frr sflow/frr
Run the following command to show the router configuration:
docker exec frr vtysh -c "show running-config"
The results will be displayed:
Building configuration...

Current configuration:
!
frr version 7.5_git
frr defaults traditional
hostname ed9e435c6f3d
no ipv6 forwarding
log stdout
!
router bgp 65000
 bgp router-id 0.0.0.1
 neighbor 192.168.65.2 remote-as 65001
 neighbor 192.168.65.2 port 1179
 neighbor 192.168.65.2 ebgp-multihop 255
 neighbor 192.168.65.2 timers connect 10
 !
 address-family ipv4 unicast
  neighbor 192.168.65.2 route-map ALLOW-ALL in
  neighbor 192.168.65.2 route-map ALLOW-ALL out
 exit-address-family
 !
 address-family ipv4 flowspec
  neighbor 192.168.65.2 activate
  neighbor 192.168.65.2 route-map ALLOW-ALL in
  neighbor 192.168.65.2 route-map ALLOW-ALL out
 exit-address-family
 !
 address-family ipv6 unicast
  neighbor 192.168.65.2 activate
  neighbor 192.168.65.2 route-map ALLOW-ALL in
  neighbor 192.168.65.2 route-map ALLOW-ALL out
 exit-address-family
 !
 address-family ipv6 flowspec
  neighbor 192.168.65.2 activate
  neighbor 192.168.65.2 route-map ALLOW-ALL in
  neighbor 192.168.65.2 route-map ALLOW-ALL out
 exit-address-family
!
route-map ALLOW-ALL permit 100
!
ip nht resolve-via-default
!
line vty
!
end

Simulate a DDoS attack as describes in Docker DDoS testbed.

Now run the following command to see the Flowspec rule:

docker exec frr vtysh -c "show bgp ipv4 flowspec detail"

The following rule was sent to the router:

BGP flowspec entry: (flags 0x418)
	Destination Address 172.17.0.1/32
	IP Protocol = 17 
	Source Port = 53 
	FS:rate 0.000000
	received for 00:00:48
	not installed in PBR

Displayed  1 flowspec entries

The rule isn't installed in the policy based routing (PBR) table since the current version of FRRouting doesn't include the necessary Netfilter driver.

FRRouting software is widely used in open source network operating systems such as SONiC, OpenSwitch, and DENT. Once the Flowspec driver integration is complete, the white box switch hardware supported by these network operation systems will provide a cost effective method of mitigating DDoS attacks - combining the real-time visibility of sFlow with the real-time control capabilities of Flowspec.

Thursday, November 19, 2020

Multipass

Multipass is a command line tool for running Ubuntu virtual machines on Mac or Windows. Multipass uses the native virtualization capabilities of the host operating system to simplify the creation of virtual machines.

Docker testbed and Docker DDoS testbed describe how to use containers to experiment with network visibility and control. However, not all software is amenable to running in containers, and so the ability to quickly create and configure virtual machines is a useful complement. This article demonstrates how to use Multipass to quickly build a virtual machine to run Mininet network emulation software.
multipass launch --name=mininet bionic
multipass exec mininet -- sudo apt update
multipass exec mininet -- sudo apt -y install mininet python-ryu
multipass exec mininet -- sudo apt -y install default-jre python-requests hping3
multipass exec mininet -- wget https://inmon.com/products/sFlow-RT/sflow-rt.tar.gz
multipass exec mininet -- tar -xzf sflow-rt.tar.gz
multipass exec mininet -- ./sflow-rt/get-app.sh sflow-rt mininet-dashboard

Run the above commands in a terminal to create the virtual machine. Multipass commands can easily be scripted to automate the creation and configuration of virtual machines.

multipass list
List the virtual machines.
Name                    State             IPv4             Image
test                    Running           192.168.64.2     Ubuntu 18.04 LTS

Find the IP address of the mininet virtual machine we just created (192.168.64.2).

multipass exec mininet -- ./sflow-rt/start.sh

Start sFlow-RT.

Use a web browser to connect to the VM and access the Mininet Dashboad application running on sFlow-RT, in this case http://192.168.64.2:8008/app/mininet-dashboard/html/
multipass exec mininet -- sudo mn --custom sflow-rt/extras/sflow.py \
--link tc,bw=10 --topo tree,depth=2,fanout=2 --test iperf
In a separate terminal, run a test using Mininet.
The Mininet Dashboard shows the test traffic, the links in the emulated network carrying the traffic, and the diameter of the emulated network.
The Topology tab shows a Mininet weathermap, displaying a dynamic view of topology in which link widths update in real-time to reflect traffic flowing over the virtual network.

An external SDN controller can be used to control packet forwarding in the network.
multipass exec mininet -- ryu-manager ryu.app.simple_switch,ryu.app.ofctl_rest
Start the Ryu SDN controller in a terminal window.
multipass exec mininet -- sudo mn --custom sflow-rt/extras/sflow.py \
--link tc,bw=10 --topo tree,depth=2,fanout=2 \
--controller=remote,ip=127.0.0.1 --test iperf
Run the same test as before, but this time connecting to the SDN controller. For a more interesting example, Ryu measurement based control describes how to detect and block denial of service attacks using the controller.

There are additional multipass commands available to manage the virtual machine.
multipass shell mininet
Connect to the virtual machine and access a command shell.
multipass stop mininet
Stop the virtual machine.
multipass start mininet
Start the virtual machine.
multipass delete mininet
multipass purge
Delete the virtual machine.

There are a number of Mininet articles on this blog describing projects that can be run using the virtual machine that demonstrate performance aware software defined networking concepts.

Friday, November 13, 2020

sFlow Monitoring for AI


A Proposal towards sFlow Monitoring Dashboards for AI-controlled NRENs is a recent talk by Mariam Kiran (Esnet) presented at the recent GÉANT Telemetry and Big Data Workshop.
In the talk, Miram describes the set open source tools (Netdata, Prometheus, Zabbix, Ntopng, and PerfSONAR) that they attempted to synthesize a complete picture of the network.
A number of tools were combined since each tool provides a different subset of the measurements needed to drive the AI controller. 
Integrating the data from the different sources was a challenge, but they were able to pull the data together into a single Grafana dashboard. Unfortunately, there was a lot of noise in legacy measurement schemes, making the data set unsuitable for training the AI controller.
The team decided to go toward sFlow, replacing the legacy monitoring tools with sFlow enabled devices, in order to generate the very clean data needed for machine learning.

For background, the talk, Real-time network telemetry for automation, describes why sFlow is uniquely suited to automation, providing the comprehensive, real-time, system-wide, visibility needed to make networked systems observable.

Thursday, November 12, 2020

Real-time network telemetry for automation


The video discusses telemetry and requirements for network automation, providing an overview of sFlow measurement architecture and a discussion of recently added packet drop monitoring functionality, and ending with a live demonstration of GPU compute cluster analytics. The slides from the video are available here.

The video is part of recent talk Using Advanced Telemetry to Correlate GPU and Network Performance Issues [A21870] presented at the NVIDIA GTC conference

Wednesday, November 11, 2020

DENT


Introducing DENT OS, switchdev NOS for the rest of us talk, presented at the recent Open Source Summit, describes the Linux Foundation DENT project. The slides from the presentation are available.

Linux switchdev is an in-kernel driver model for switch devices which offload the forwarding (data) plane from the kernel. Integrating switch ASIC drivers in the Linux kernel makes switch ports appear as additional Linux network interfaces that can be configured and managed using standard Linux tools. 

DENT is an Ubuntu based Linux distribution that packages the drivers for switch hardware (fans, temperature sensors, ASIC, etc) along with the open source FRRouting routing protocol suite which includes protocol daemons for BGP, IS-IS, LDP, OSPF, PIM, and RIP. The FRRouting software uses the Linux netlink API to program Linux kernel packet forwarding, which on a hardware switch platform is offloaded by the switchdev driver to the ASIC for line rate forwarding.

A major benefit of DENT's approach to making Linux into the network operating system is that the same tools used to configure, manage and monitor Linux servers can also be used to manage network switches. In addition, a DENT virtual machine behaves in exactly the same way as DENT running on a physical switch, allowing configurations to be validated in virtual environments before being pushed into the production network.

The open source Host sFlow agent streams standard sFlow telemetry from Linux hosts. On a switchdev based network switch, hardware counters are gathered using standard Linux API's for monitoring network interfaces. The Host sFlow agent also supports Linux psample and drop_monitor APIs that provide visibility into packet flows and dropped packets. On hardware switches, the switchdev driver offloads these measurements to the switch ASIC which provides line rate visibility into packet forwarding. 
Installing Host sFlow agents on DENT OS based switches activates the standard sFlow hardware instrumentation in switch ASICs to stream network telemetry to an sFlow collector, for example sFlow-RT, for the real-time network-wide view of performance needed to drive automation.