Monday, June 19, 2017

Remotely Triggered Black Hole (RTBH) Routing

The screen shot demonstrates real-time distributed denial of service (DDoS) mitigation. Automatic mitigation was disabled for the first simulated attack (shown on the left of the chart).  The attack reaches a sustained packet rate of 1000 packets per second for a period of 60 seconds. Next, automatic mitigation was enabled and a second attack launched. This time, as soon as the traffic crosses the threshold (the horizontal red line), a BGP remote trigger message is sent to router, which immediately drops the traffic.
The diagram shows the test setup. The network was built out of freely available components: CumulusVX switches and Ubuntu 16.04 servers running under VirtualBox.

The following configuration is installed on the ce-router:
router bgp 65140
 bgp router-id 0.0.0.140
 neighbor 10.0.0.70 remote-as 65140
 neighbor 10.0.0.70 port 1179
 neighbor 172.16.141.2 remote-as 65141
 !
 address-family ipv4 unicast
  neighbor 10.0.0.70 allowas-in
  neighbor 10.0.0.70 route-map blackhole-in in
 exit-address-family
!
ip community-list standard blackhole permit 65535:666
!
route-map blackhole-in permit 20
 match community blackhole
 match ip address prefix-len 32
 set ip next-hop 192.0.2.1
The ce-router peers with the upstream service provider router (sp-router 172.16.141.2) as well as with sFlow-RT.  A route-map is used to filter updates from sFlow-RT (10.0.0.70), matching the well-known blackhole community 65535:666, and setting the null route next-hop 192.0.2.1. The route-map also ensures that only /32 prefixes are accepted. In production, the route-map should also filter out the addresses of critical infrastructure (router IP addresses etc.).

An additional route-map will typically be required to select the blackhole routes to propagate upstream, re-mapping the community to meet the upstream service provider's policy, e.g. Hurricane Electric uses community 6939:666.

In addition, the ce-router is configured to send sFlow to the controller (10.0.0.70), see Switch configurations.

Run the DDoS mitigation application on server 10.0.0.70 using docker:
docker run --net=host sflow/ddos-blackhole -Dddos_blackhole.router=10.0.0.140 -Dddos_blackhole.as=65140
Alternatively, an RPM or DEB packaged version of sFlow-RT can be downloaded and installed on the server and the ddos-blackhole application can be installed. For example, on Ubuntu:
wget https://inmon.com/products/sFlow-RT/sflow-rt_3.0-1460.deb
dpkg -i sflow-rt_3.0-1460.deb
/usr/local/sflow-rt/get-app.sh sflow-rt ddos-blackhole
Edit the configuration file, /usr/local/sflow-rt/conf.d/sflow-rt.conf:
bgp.start=yes
ddos_blackhole.router=10.0.0.140
ddos_blackhole.as=65141
Next, start the daemon:
service sflow-rt start
In either case, the remainder of the configuration is handled through the web interface, accessible via http://10.0.0.70:8008/

Click on the Settings tab and upload the following IP Address Groups file groups.json:
{
 "external": [
  "0.0.0.0/0"
 ],
 "private": [
  "10.0.0.0/8",
  "192.168.0.0/16"
 ],
 "multicast": [
  "224.0.0.0/4"
 ],
 "exclude":
  "172.16.0.0/12"
 "web": [
  "172.16.140.0/24"
 ]
}
The groups identify addresses that are external (possible attackers) and local (possible targets). By default, traffic to the externalprivate, multicast and exclude groups will not trigger actions. Any additional group names, in this case web, are blackhole candidates.

Note: Only traffic from the external group to backhole candidate groups (web) is shown on the Charts tab (and considered for DDoS detection and mitigation).

The following command on sp-host simulates an ICMP flood attack on ce-host:
ping -f 172.16.140.1
The following messages should appear in the sFlow-RT logs:
2017-06-17T00:17:39+0000 INFO: Listening, BGP port 1179
2017-06-17T00:17:40+0000 INFO: Listening, sFlow port 6343
2017-06-17T00:17:40+0000 INFO: Listening, HTTP port 8008
2017-06-17T00:17:40+0000 INFO: app/ddos-blackhole/scripts/ddos.js started
2017-06-17T00:17:40+0000 INFO: app/ddos-blackhole/scripts/stats.js started
2017-06-17T00:17:46+0000 INFO: BGP open 10.0.0.140 41692
2017-06-17T00:18:13+0000 INFO: DDoS blocking 172.16.140.1
2017-06-17T00:20:25+0000 INFO: DDoS allowing 172.16.140.1
The screen capture at the top of this article shows that the time between the attack being launched and successfully blocked is just a few of seconds.

14 comments:

  1. Great work Peter! Im just giving this a test run, but I am at the stage where I am trying to start the daemon with 'service sflow-rt start' and a prompt, comes up requesting for a passowrd. May you please tell me what the password is? much thanks

    ReplyDelete
    Replies
    1. You need root permission to restart a Linux service, so this is either the root password, or your own password if you are using sudo.

      If you are running on a recent Linux system with systemd, then it is better to use the systemd commands to start the service:
      systemctl start sflow-rt.service
      use the following command so that sFlow-RT automatically starts on a reboot:
      systemctl enable sflow-rt.service

      Delete
  2. nice share Peter,
    but how change bgp remote port on cumulus vx?

    "neighbor 10.0.0.70 port 1179"

    i can't find BGP port parameter configuration on cumulus VX. thx before.

    ReplyDelete
    Replies
    1. It looks like the port setting isn't available using the NCLU "net add bgp ..." commands. You need to use vtysh to set the port, e.g.
      sudo vtysh
      configure terminal
      router bgp 65140
      neighbor 10.0.0.70 port 1179
      exit
      exit
      write mem

      Delete
  3. Peter, here in the diagram, in step 3, is the "sflow-RT" acting as a "BGP Flowspec peer router" sending an 'ACL' to block the traffic ? Basically I'm trying to compare the "BGP Flowspec" approach with this (or if Flowspec plays any role here).

    thanks.

    ReplyDelete
    Replies
    1. Are you referring to the article, Real-time DDoS mitigation using sFlow and BGP FlowSpec? The two setups are similar, in this case a null route is relayed by the ce-router to the sp-router so that all traffic to the blackholed address is dropped. In the FlowSpec case, a more specific ACL filter is relayed, only dropping traffic from a specified UDP port to the targeted address.

      Delete
  4. Hi Peter, thanks you for the nice articles!

    How do you get Cumulus VX to send out flow samples? According to Cumulus the flow samples originate from the ASIC, and thus doesn't work with VX.

    I managed to recompile the sFlow daemon with the PCAP option, and now it is sending out flow samples. However, they don't show up in the chart.

    See https://gitlab.com/yoeri/ddos-workshop for my setup.

    ReplyDelete
    Replies
    1. With the default Cumulus VX sflow agent (hsflowd), samples are received using the nflog{} module. You can configure sampling in iptables using the command on the Cumlux VX switch:

      sudo iptables -I FORWARD -j NFLOG --nflog-group 1 --nflog-prefix SFLOW

      Delete
  5. such a nice article, informations are well crafted here. Goodjob!!!

    1.is it okay? if i place the sflow-RT controller in the same BGP network as customer BGP network AS65140?

    2.just an assumption:
    ISP -------------My BGP
    already peer established between us.

    2.whats the additional/new configurations that i have to configure in my BGP router ?

    3. do i have to let my upstream provider to know about my balckhole setup?

    ReplyDelete
    Replies
    1. 1. is it okay? if i place the sflow-RT controller in the same BGP network as customer BGP network AS65140?
      I don't think an iBGP session would work since iBGP routes won't be propagated by the ce-router to the upstream sp-router. You should be able to use a private AS for sFlow-RT (like 65141 used in this example).

      2.just an assumption:
      ISP -------------My BGP
      already peer established between us.

      Yes.

      3. whats the additional/new configurations that i have to configure in my BGP router ?

      a) You need to configure the ce-router to add the additional eBGP session to sFlow-RT (via port 1179). The router needs to be configured to initiate the connection, sFlow-RT operates passively and waits for the router to connect. b) You also need to configure the route-map to control how black-hole routes advertised by sFlow-RT are filtered/rewritten before being passed up to the sp-router. c) You need to configure the router to send sFlow to sFlow-RT (or configure sFlow on the core switches connected to the router if the router doesn't support sFlow).

      4. do i have to let my upstream provider to know about my blackhole setup?

      Remotely triggered black hole routing must be a service offered by the service provider in order for this scheme to work. They will have a particularly BGP community that you need to tag the routes with in order for them to drop the traffic, e.g. Customer Blackhole Community. You can set the community with the -Dddos_blackhole.community=xxxx:xxx option when starting sFlow-RT, or rewrite the community in the route-map.

      Delete
  6. Hello Mr.Peter,

    1.I am using a Huawei router NE40-X3/X8 series modeils for BGP . Router supports Netsream protocol not sflow protocol. is there any way around to configure this router to connect with sflow-rt?

    2.do you have any suggestions for monitoring the netsream based routers ?


    3. I have some questions in the configurations section?
    My topology:
    I peered with 2 ISP's.
    ISPA(AS4XXX) -------------MY BGP AS6344X---100.100.100.X(Customer Network)
    ISPB (AS5XXX) ------------MY BGP AS6344X---100.100.101.X (Customer Network)
    |
    |
    100.100.100.X
    (Sflow-RT) AS 65530

    If i establish a peer to my BGP AS6344X from sflow-RT using a privae AS65530 will it work ?


    4. Lets assume, there is a BGP in Tokyo and another BGP in Hong Kong.
    belongs to the same APNIC regions.
    sflow-rt can handle the two BGP blackhole settings?


    Thank-you!

    i really appreciate your time on this matter!

    ReplyDelete
    Replies
    1. sFlow-RT is a real-time flow analyzer - see Rapidly detecting large flows, sFlow vs. NetFlow/IPFIX. NetFlow/IPFIX/NetStream aren't supported since the don't provide real-time information.

      You might try FastNetMon - I believe it supports NetFlow based protocols - it won't be as fast, but it should work.

      Delete
  7. Hi!

    We're wisp and we need protection from DDoS. What are hardware requirements for 20G of traffic?

    ReplyDelete
    Replies
    1. You might want to look at the newer ddos-protect application. It supports BGP Flowspec filters in addition to RTBH. The solution uses your existing routers (provided they support sFlow and BGP). The sFlow-RT analytics software requires modest compute resources, 4 CPU cores and 2G RAM should be sufficient for most use cases.

      Delete