Thursday, August 30, 2018

Northbound Networks Zodiac GX

Mininet is widely used to emulate software defined networks (SDNs). Mininet flow analytics describes how standard sFlow telemetry, from Open vSwitch used by Mininet emulate the network, provides feedback to an SDN controller, allowing the controller to adapt the network to changing traffic, for example, to mitigate a distributed denial of service (DDoS) attack.

Northbound Networks Zodiac GX is an inexpensive open source software based switch that is ideal for experimenting with software defined networking (SDN) in a physical network setting. The small fanless package makes the switch an attractive option for desktop use. The Zodiac GX is also based on Open vSwitch, making it easy to take SDN control strategies developed on Mininet.
Enabling sFlow on the Zodiac GX is easy, navigate to the System>Startup page and add the following line to the end of the startup script (before the exit 0 line):
ovs-vsctl -- --id=@sflow create sflow agent=$OVS_BR target=$IP_CONTROLLER_1 sampling=100 polling=10 -- set bridge $OVS_BR sflow=@sflow
Reboot the switch for the changed to take effect.

Use sflowtool to verify that sFlow is arriving at the controller host and to examine the contents of the telemetry stream. Running sflowtool using Docker is a simple alternative to building the software from sources:
docker run --rm -p 8008:8008 -p 6343:6343/udp sflow/sflowtool
The text output from sflowtool can be piped into scripts to perform basic sFlow analysis.
A graphical sFlow analyzer performs the analysis tasks for you. The screen shot above shows sFlowTrend, a free sFlow analyzer that displays traffic trends. The software can be downloaded and installed or run using Docker:
docker run --rm -p 6343:6343/udp -p 8087:8087 -p 8443:8443 sflow/sflowtrend
The sFlowTrend charts update every minute. This is generally fast enough for human consumption, but real-time, up to the second, visibility is critical for SDN use cases.
The screen shot from Flow Trend shows an up to the second view of traffic. The spike in traffic is due to a 4K video being streamed from YouTube. The following command runs the software:
docker run --rm -p 6343:6343/udp -p 8008:8008 sflow/flow-trend
Flow Trend is an application running on the sFlow-RT real-time analytics platform.
Applications running on the sFlow-RT platform deliver real-time visibility to SDN, DevOps and Orchestration stacks, enabling new classes of performance aware application such as load balancing, DDoS mitigation, and workload placement.

RYU provides a framework that can be used to develop SDN applications in Python. For example, the following command runs the simple learning bridge application that ships with RYU:
docker run -it --rm -p 6633:6633 osrg/ryu ryu-manager --verbose ryu/ryu/app/simple_switch_13.py
As soon as the switch connects to the controller, you should see a flurry of events as the controller programs flows on the Zodiac GX switch.

Faucet is an SDN controller for production networks implemented using RYU. Before we can use Faucet, we need to gather basic OpenFlow information from the switch.
$ ssh -t admin@10.0.0.230 "sudo ovs-ofctl show ovslan"
admin@10.0.0.230's password: 
Password: 
OFPT_FEATURES_REPLY (xid=0x2): dpid:000044d1fa6291b2
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
 1(eth0.1): addr:44:d1:fa:62:91:b2
     config:     0
     state:      STP_FORWARD
     current:    1GB-FD AUTO_NEG
     speed: 1000 Mbps now, 0 Mbps max
 2(eth0.2): addr:44:d1:fa:62:91:b2
     config:     0
     state:      STP_FORWARD
     current:    1GB-FD AUTO_NEG
     speed: 1000 Mbps now, 0 Mbps max
 3(eth0.3): addr:44:d1:fa:62:91:b2
     config:     0
     state:      STP_FORWARD
     current:    1GB-FD AUTO_NEG
     speed: 1000 Mbps now, 0 Mbps max
 4(eth0.4): addr:44:d1:fa:62:91:b2
     config:     0
     state:      STP_FORWARD
     current:    1GB-FD AUTO_NEG
     speed: 1000 Mbps now, 0 Mbps max
 5(eth0.5): addr:44:d1:fa:62:91:b2
     config:     0
     state:      STP_FORWARD
     current:    1GB-FD AUTO_NEG
     speed: 1000 Mbps now, 0 Mbps max
 LOCAL(ovslan): addr:44:d1:fa:62:91:b2
     config:     0
     state:      0
     current:    1GB-FD AUTO_NEG
     speed: 1000 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
Connection to 10.0.0.230 closed.
Update August 30, 2018: The only piece of information needed to construct the faucet config file below is the switch dpid. The Zodiac GX uses the MAC address of the switch as the dpid, so you can simply read the MAC address printed on a label on the bottom of the switch.
Now create a directory called faucet that contains the initial Faucet configuration file, faucet.yaml:
vlans:
    office:
        vid: 100
        description: "office network"

dps:
    zodiac:
        dp_id: 0x000044d1fa6291b2
        hardware: "ZodiacGX"
        interfaces:
            1:
                name: "eth0.1"
                description: "port1"
                native_vlan: office
            2:
                name: "eth0.2"
                description: "port2"
                native_vlan: office
            3:
                name: "eth0.3"
                description: "port3"
                native_vlan: office
            4:
                name: "eth0.4"
                description: "port4"
                native_vlan: office
            5:
                name: "eth0.5"
                description: "port5"
                native_vlan: office
            0xfffffffe:
                name: "ovslan"
                description: "local"
                native_vlan: office
Now run Faucet:
docker run -it --rm -v $PWD/faucet/:/etc/faucet/ -v $PWD/faucet/:/var/log/faucet/ -p 6633:6653 -p 9302:9302 faucet/faucet
As soon as the switch connects to the controller, you should see events logged to the faucet.log file in the same directory as the faucet.yaml configuration file.

Faucet Documentation describes how to extend the configuration to include firewall, routing, segmentation, and network function virtualization (NFV) rules to the configuration.

The next step is integrating sFlow analytics with the controller. Writing Applications describes how write sFlow-RT applications use REST API and embedded JavaScript API. The document includes Python examples that could be embedded in RYU controller applications. Alternatively, sFlow-RT's embedded HTTP client can be used to push control actions to an SDN controller, see ONOS measurement based control for an example.

The sFlow telemetry stream contains detailed Open vSwitch performance metrics in addition to flow and interface counter data. The sFlow-RT analytics pipeline can be programmed to generate and push statistics to time series databases and dashboards, see Prometheus and Grafana and InfluxDB and Grafana.

Exporting events using syslog describes how sFlow-RT can be programmed to detect and report on traffic anomalies, sending events to Security Information and Event Management (SIEM) tools, using Splunk and Logstash as examples.

The sflow/sflow-rt Docker image provides a convenient means of developing and deploying sFlow-RT applications alongside the SDN controllers demonstrated in this article.

An important benefit of sFlow telemetry is that it decouples monitoring from the control plane. You are free to change SDN controllers, use distributed routing / switching protocols, move between network operating systems, or build your own control plane while maintaining the same level of visibility. Industry standard sFlow is widely supported by vendors, including: A10, Aerohive, ALUe, Allied Telesis, Arista, Aruba, Big Switch, Broadcom, Cisco, Cumulus, Dell, D-Link, Edge-Core, Extreme, F5, Fortinet, Huawei, IP Infusion, Juniper, Mellanox, Netgear, OpenSwitch, Pica8, Proxim, Quanta, SMC, ZTE, and ZyXEL.

Monday, August 20, 2018

RDMA over Converged Ethernet (RoCE)

RDMA over Converged Ethernet is a network protocol that allows remote direct memory access (RDMA) over an Ethernet network. One of the benefits running RDMA over Ethernet is the visibility provided by standard sFlow instrumentation embedded in the commodity Ethernet switches used to build data center leaf and spine networks where RDMA is most prevalent.

The sFlow telemetry stream includes packet headers, sampled at line rate by the switch hardware. Hardware packet sampling allows the switch to monitor traffic at line rate on all ports, keeping up with the high speed data transfers associated with RoCE.

The diagram above shows the packet headers associated with RoCEv1 and RoCEv2 packets. Decoding the InfiniBand Global Routing Header (IB GRH) and InfiniBand Base Transport Header (IB BTH) allows an sFlow analyzer to report in detail on RoCE traffic.
The sFlow-RT real-time analytics engine recently added support for RoCE by decoding InfiniBand Global Routing and InfiniBand Base Transport fields. The screen capture of the sFlow-RT Flow-Trend application shows traffic associated with an RoCEv2 connection between two hosts, 10.10.2.22 and 10.10.2.52. The traffic consists of SEND and ACK messages exchanged as part of a reliable connection (RC).

The standard sFlow instrumentation provides comprehensive network wide visibility into RoCE and all other applications sharing the network resources. Real-time visibility is an essential part of automating networks, providing the feedback needed to ensure that resources are efficiently allocated and rapidly identifying overloaded resources so that remediation action can be taken before significant service degradation occurs.