The following traceroute run on Peer 1 shows the set of router hops from 192.168.250.1 to 192.168.251.1
[root@peer1 ~]# traceroute -s 192.168.250.1 192.168.251.1 traceroute to 192.168.251.1 (192.168.251.1), 30 hops max, 40 byte packets 1 192.168.152.2 (192.168.152.2) 3.090 ms 3.014 ms 2.927 ms 2 192.168.150.3 (192.168.150.3) 3.377 ms 3.161 ms 3.099 ms 3 192.168.251.1 (192.168.251.1) 6.440 ms 6.228 ms 3.217 msEnsuring that packets are first forwarded by the default route on the Hardware Switch is the key to accelerating forwarding decisions for the Software Router. Any specific routes added to Hardware Switch will override the default route and matching packets will bypass the Software Router and will be forwarded by the Hardware Switch.
In this test bench, routing is performed using Quagga instances running on Peer 1, Peer 2, Hardware Switch and Software Router.
Peer 1
router bgp 150 bgp router-id 192.168.150.1 network 192.168.250.0/24 neighbor 192.168.152.3 remote-as 152 neighbor 192.168.152.3 ebgp-multihop 2
Peer 2
router bgp 151 bgp router-id 192.168.151.1 network 192.168.251.0/24 neighbor 192.168.152.3 remote-as 152 neighbor 192.168.152.3 ebgp-multihop 2
Software Router
interface lo ip address 192.168.152.3/32 router bgp 152 bgp router-id 192.168.152.3 neighbor 192.168.150.1 remote-as 150 neighbor 192.168.150.1 update-source 192.168.152.3 neighbor 192.168.150.1 passive neighbor 192.168.151.1 remote-as 151 neighbor 192.168.151.1 update-source 192.168.152.3 neighbor 192.168.151.1 passive neighbor 10.0.0.162 remote-as 152 neighbor 10.0.0.162 port 1179 neighbor 10.0.0.162 timers connect 30 neighbor 10.0.0.162 route-reflector-client
Hardware Switch
router bgp 65000 bgp router-id 0.0.0.1 neighbor 10.0.0.162 remote-as 65000 neighbor 10.0.0.162 port 1179 neighbor 10.0.0.162 timers connect 30In addition, the following lines in /etc/network/interfaces configure the bridge:
auto br_xen iface br_xen bridge-ports swp1 swp2 swp3 address 192.168.150.2/24 address 192.168.151.2/24 address 192.168.152.2/24Cumulus Networks, sFlow and data center automation describes how to configure sFlow monitoring on Cumulus Linux switches. The switch is configured to send sFlow to 10.0.0.162 (the host running the SDN controller).
SDN Routing Application
SDN router using merchant silicon top of rack switch describes how to install sFlow-RT and provides an application for pushing active routes to an accelerator. The application has been modified for this setup and is running on host 10.0.0.162:
bgpAddNeighbor('10.0.0.152',152);
bgpAddNeighbor('10.0.0.233',65000);
bgpAddSource('10.0.0.233','10.0.0.152',10);
var installed = {};
setIntervalHandler(function() {
let now = Date.now();
let top = bgpTopPrefixes('10.0.0.152',20000,1);
if(!top || !top.hasOwnProperty('topPrefixes')) return;
let tgt = bgpTopPrefixes('10.0.0.233',0);
if(!tgt || 'established' != tgt.state) return;
for(let i = 0; i < top.topPrefixes.length; i++) {
let entry = top.topPrefixes[i];
if(bgpAddRoute('10.0.0.233',entry)) {
installed[entry.prefix] = now;
}
}
for(let prefix in installed) {
let time = installed[prefix];
if(time === now) continue;
if(bgpRemoveRoute('10.0.0.233',prefix)) {
delete installed[prefix];
}
}
}, 1);
Start the application:
$ ./start.sh 2015-07-21T19:36:52-0700 INFO: Listening, BGP port 1179 2015-07-21T19:36:52-0700 INFO: Listening, sFlow port 6343 2015-07-21T19:36:53-0700 INFO: Starting the Jetty [HTTP/1.1] server on port 8008 2015-07-21T19:36:53-0700 INFO: Starting com.sflow.rt.rest.SFlowApplication application 2015-07-21T19:36:53-0700 INFO: Listening, http://localhost:8008 2015-07-21T19:36:53-0700 INFO: bgp.js started 2015-07-21T19:36:57-0700 INFO: BGP open /10.0.0.152:50010 2015-07-21T19:37:23-0700 INFO: BGP open /10.0.0.233:55097Next, examine the routing table on the Hardware Switch:
cumulus@cumulus$ ip route default via 192.168.152.1 dev br_xen 10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.233 192.168.150.0/24 dev br_xen proto kernel scope link src 192.168.150.2 192.168.151.0/24 dev br_xen proto kernel scope link src 192.168.151.2 192.168.152.0/24 dev br_xen proto kernel scope link src 192.168.152.2This is the default set of routes configured to pass traffic to and from the Software Router.
To generate traffic using iperf, run the following command on Peer 2:
iperf -s -B 192.168.251.1And generate traffic with the following command on Peer 1:
iperf -c 192.168.251.1 -B 192.168.250.1Now check the routing table on the Hardware Switch again:
cumulus@cumulus$ ip route
default via 192.168.152.1 dev br_xen
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.233
192.168.150.0/24 dev br_xen proto kernel scope link src 192.168.150.2
192.168.151.0/24 dev br_xen proto kernel scope link src 192.168.151.2
192.168.152.0/24 dev br_xen proto kernel scope link src 192.168.152.2
192.168.250.0/24 via 192.168.150.1 dev br_xen proto zebra metric 20
192.168.251.0/24 via 192.168.151.1 dev br_xen proto zebra metric 20
Note the two hardware routes that have been added by the SDN controller. The route override can be verified by repeating the traceroute test:
[root@peer1 ~]# traceroute -s 192.168.250.1 192.168.251.1 traceroute to 192.168.251.1 (192.168.251.1), 30 hops max, 40 byte packets 1 192.168.150.2 (192.168.150.2) 3.260 ms 3.151 ms 3.014 ms 2 192.168.251.1 (192.168.251.1) 4.418 ms 4.351 ms 4.260 msComparing with the original traceroute, notice that packets bypass the Software Router interface (192.168.150.3) and are forwarded entirely in hardware.
The traffic analytics driving the forwarding decisions can be viewed through the sFlow-RT REST API:
$ curl http://10.0.0.162:8008/bgp/topprefixes/10.0.0.52/json
{
"as": 152,
"direction": "destination",
"id": "192.168.152.3",
"learnedPrefixesAdded": 2,
"learnedPrefixesRemoved": 0,
"nPrefixes": 2,
"pushedPrefixesAdded": 0,
"pushedPrefixesRemoved": 0,
"startTime": 1437535255553,
"state": "established",
"topPrefixes": [
{
"aspath": "150",
"localpref": 100,
"med": 0,
"nexthop": "192.168.150.1",
"origin": "IGP",
"prefix": "192.168.250.0/24",
"value": 1.4462334178258518E7
},
{
"aspath": "151",
"localpref": 100,
"med": 0,
"nexthop": "192.168.151.1",
"origin": "IGP",
"prefix": "192.168.251.0/24",
"value": 391390.33359066787
}
],
"valuePercentCoverage": 100,
"valueTopPrefixes": 1.4853724511849185E7,
"valueTotal": 1.4853724511849185E7
}
The SDN application automatically removes routes from the hardware once they become idle, or to make room for more active routes if the hardware routing table exceeds the set limit of 20,000 routes, or if they are withdrawn. This switch has a maximum capacity of 32,768 routes and standard sFlow analytics can be used to monitor hardware table utilizations - Broadcom ASIC table utilization metrics, DevOps, and SDN.The test setup was configured to quickly test the concept using limited hardware at hand and can be improved for production deployment (using smaller CIDRs and VLANs to separate peer traffic).This proof of concept demonstrates that it is possible to use SDN analytics and control to combine standard sFlow and BGP capabilities of commodity hardware and deliver Terabit routing capacity with just a few thousand dollars of hardware.
