Programmable hardware: Barefoot Networks, PISA, and P4

Barefoot Networks recently came out of stealth to reveal their  Tofino 6.5Tbit/second (65 X 100GE or 260 X 25GE) fully user-programmable switch. The diagram above, from the talk Programming The Network Data Plane by Changhoon Kim of Barefoot Networks, shows the Protocol Independent Switch Architecture (PISA) of the programmable switch silicon.

A logical switch data-plane described in the P4 language is compiled to program the general purpose PISA hardware. For example, the following P4 code snippet is part of a P4 sFlow implementation:

table sflow_ing_take_sample {
    /* take_sample > MAX_VAL_31 and valid sflow_session_id => take the sample */
    reads {
        ingress_metadata.sflow_take_sample : ternary;
        sflow_metadata.sflow_session_id : exact;
    }
    actions {
        nop;
        sflow_ing_pkt_to_cpu;
    }
}

Network visibility is one of the major use cases for P4 based switches. Improving Network Monitoring and Management with Programmable Data Planes describes how P4 can be used to collect information about latency and queueing in the switch forwarding pipeline.

The document also describes an architecture for In-band Network Telemetry (INT) in which the ingress switch is programmed to insert a header containing measurements to packets entering the network. Each switch in the path is programmed to append additional measurements to the packet header. The egress switch is programmed to remove the header so that the packet can be delivered to its destination. The egress switch is responsible for processing the measurements or sending them on to analytics software.

In-band telemetry is an interesting example of the flexibility provided by P4 programmable hardware and the detailed information that can be gathered about latency and queueing from the hardware forwarding pipeline. However, there are practical issues that should be considered with this approach:

1. Transporting measurement headers is complex with different encapsulations for each transport protocol:  Geneve, VxLAN, etc.
2. Addition of headers increases the size of packets and risks causing traffic to be dropped downstream due to maximum transmission unit (MTU) restrictions.
3. The number of measurements that can be added by each switch and the number of switches adding measurements in the path needs to be limited.
4. In-band telemetry cannot be incrementally deployed. Ideally, all devices need to participate, or at a minimum, the ingress and egress devices need to be in-band telemetry aware.
5. In-band telemetry transports data from the data plane to the control/management planes, providing a potential attack surface that could be exploited by crafting malicious packets with fake measurement headers.

The sFlow architecture provides an out of band alternative for transporting the per packet forwarding plane measurements defined by INT. Instead of adding the measurements to the egress packet, measurements can be attached as metadata to the sampled packets that are handed to the switch CPU. The sFlow agent immediately forwards the additional packet metadata as part of the standard sFlow telemetry stream to a centralized collector. Using sFlow as the telemetry transport has a number of benefits:

1. Simple to deploy since there is no modification of packets (no issues with encapsulations, MTU, number of measurements, path length, incremental deployment, etc.)
2. Extensibility of sFlow protocol allows additional forwarding plane measurements to augment standard sFlow measurements, fully integrating the new measurements with sFlow data exported from other switches in the network (sFlow is supported by over 40 switch vendors and is a standard feature of switch ASICs).
3. sFlow's is a unidirectional telemetry transport protocol originates from the device management plane, can be sent out of band, limiting possible attack surfaces.

The great thing about programmable hardware is that behavior can be modified by changing the software. Implementing out of band telemetry is a matter of combining measurements from the P4 INT code with the P4 sFlow agent code. Compiling and installing out of band sFlow telemetry code reprograms the hardware to implement the new scheme.

The advent of P4 and programmable hardware opens up exciting possibilities for defining additional packet sample metadata, counters, and gauges to augment the sFlow telemetry stream and gain additional insight into the performance of production traffic in large scale, high capacity, networks.

The real-time sFlow streaming telemetry can be used to drive automated controls, for example, to block DDoS attacks, or load balance large "Elephant" flows across multiple paths. Here again P4 combined with programmable hardware makes it possible to create additional control capabilities, for example, to be able to block the large numbers of source addresses involved in a DDoS attack, where sFlow analytics would be used to identify the attackers and their points of ingress and program each of the switches with filters based on their location in the network. The ability to customize the hardware to address specific tasks makes more efficient use of hardware resources than is possible with fixed function device. In this case, defining a specialized DDoS drop table would allow for a much larger number of filters than would be possible with a general purpose ACL table.