Tracing network packets with eBPF and pwru

pwru (packet, where are you?) is an open source tool from Cilium that used eBPF instrumentation in recent Linux kernels to trace network packets through the kernel.

In this article we will use Multipass to create a virtual machine to experiment with pwru. Multipass is a command line tool for running Ubuntu virtual machines on Mac or Windows. Multipass uses the native virtualization capabilities of the host operating system to simplify the creation of virtual machines.

multipass launch --name=ebpf noble
multipass exec ebpf -- sudo apt update
multipass exec ebpf -- sudo apt -y install git clang llvm make libbpf-dev flex bison golang
multipass exec ebpf -- git clone https://github.com/cilium/pwru.git
multipass exec ebpf --working-directory pwru -- make
multipass exec ebpf -- sudo ./pwru/pwru -h

Run the commands above to create the virtual machine and build pwru from sources.

multipass exec ebpf -- sudo ./pwru/pwru port https

Run pwru to trace https traffic on the virtual machine.

multipass exec ebpf -- curl https://sflow-rt.com

In a second window, run the above command to generate an https request from the virtual machine.

SKB                CPU PROCESS          NETNS      MARK/x        IFACE       PROTO  MTU   LEN   TUPLE FUNC
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0               0         0x0000 1500  60    192.168.66.3:47460->54.190.130.38:443(tcp) ip_local_out
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0               0         0x0000 1500  60    192.168.66.3:47460->54.190.130.38:443(tcp) __ip_local_out
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0               0         0x0800 1500  60    192.168.66.3:47460->54.190.130.38:443(tcp) ip_output
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  60    192.168.66.3:47460->54.190.130.38:443(tcp) nf_hook_slow
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  60    192.168.66.3:47460->54.190.130.38:443(tcp) apparmor_ip_postroute
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  60    192.168.66.3:47460->54.190.130.38:443(tcp) ip_finish_output
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  60    192.168.66.3:47460->54.190.130.38:443(tcp) __ip_finish_output
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  60    192.168.66.3:47460->54.190.130.38:443(tcp) ip_finish_output2
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  60    192.168.66.3:47460->54.190.130.38:443(tcp) neigh_resolve_output
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  60    192.168.66.3:47460->54.190.130.38:443(tcp) eth_header
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  60    192.168.66.3:47460->54.190.130.38:443(tcp) skb_push
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) __dev_queue_xmit
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) qdisc_pkt_len_init
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) netdev_core_pick_tx
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) sch_direct_xmit
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) validate_xmit_skb_list
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) validate_xmit_skb
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) netif_skb_features
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) passthru_features_check
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) skb_network_protocol
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) skb_csum_hwoffload_help
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) skb_checksum_help
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) skb_ensure_writable
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) validate_xmit_xfrm
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) dev_hard_start_xmit
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) start_xmit
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) skb_clone_tx_timestamp
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) xmit_skb
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  86    192.168.66.3:47460->54.190.130.38:443(tcp) skb_to_sgvec
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  86    192.168.66.3:47460->54.190.130.38:443(tcp) __skb_to_sgvec

The pwru output provides a detailed trace of each packet through the Linux stack, identifying each application, namespace, interface and kernel module traversed by the packet. Type Ctrl+C to stop the trace.

multipass exec ebpf -- sudo ufw default allow
multipass exec ebpf -- sudo ufw deny out to any port https
multipass exec ebpf -- sudo ufw enable

Now create a firewall rule to block outgoing https traffic and repeat the test

SKB                CPU PROCESS          NETNS      MARK/x        IFACE       PROTO  MTU   LEN   TUPLE FUNC
0xffff970bc5f568e8 0   ~r/bin/curl:7138 4026531840 0               0         0x0000 1500  60    192.168.67.5:38932->54.190.130.38:443(tcp) ip_local_out
0xffff970bc5f568e8 0   ~r/bin/curl:7138 4026531840 0               0         0x0000 1500  60    192.168.67.5:38932->54.190.130.38:443(tcp) __ip_local_out
0xffff970bc5f568e8 0   ~r/bin/curl:7138 4026531840 0               0         0x0800 1500  60    192.168.67.5:38932->54.190.130.38:443(tcp) nf_hook_slow
0xffff970bc5f568e8 0   ~r/bin/curl:7138 4026531840 0               0         0x0800 1500  60    192.168.67.5:38932->54.190.130.38:443(tcp) kfree_skb_reason(SKB_DROP_REASON_NETFILTER_DROP)
0xffff970bc5f568e8 0   ~r/bin/curl:7138 4026531840 0               0         0x0800 1500  60    192.168.67.5:38932->54.190.130.38:443(tcp) skb_release_head_state
0xffff970bc5f568e8 0   ~r/bin/curl:7138 4026531840 0               0         0x0800 0     60    192.168.67.5:38932->54.190.130.38:443(tcp) tcp_wfree
0xffff970bc5f568e8 0   ~r/bin/curl:7138 4026531840 0               0         0x0800 0     60    192.168.67.5:38932->54.190.130.38:443(tcp) skb_release_data
0xffff970bc5f568e8 0   ~r/bin/curl:7138 4026531840 0               0         0x0800 0     60    192.168.67.5:38932->54.190.130.38:443(tcp) kfree_skbmem

This time, the curl command hangs and the pwru trace shows that packets are being dropped in the Linux firewall.

multipass exec ebpf -- sudo ufw disable

Disable the firewall.

There are additional multipass commands available to manage the virtual machine.

multipass shell ebpf

Connect to the virtual machine and access a command shell.

multipass stop ebpf

Stop the virtual machine.

multipass start ebpf

Start the virtual machine.

multipass delete ebpf
multipass purge

Delete the virtual machine.

CONTAINERlab

CONTAINERlab is a Docker orchestration tool for creating virtual network topologies. This article describes how to build and monitor the leaf and spine topology shown above.

Note: Docker testbed describes a simple testbed for experimenting with sFlow analytics using Docker Desktop, but it doesn't have the ability to construct complex topologies. 

multipass launch --cpus 2 --mem 4G --name containerlab
multipass shell containerlab

The above commands use the multipass command line tool to create an Ubuntu virtual machine and open shell access.

sudo apt update
sudo apt -y install docker.io
bash -c "$(curl -sL https://get-clab.srlinux.dev)"

Type the above commands into the shell to install CONTAINERlab.

Note: Multipass describes how to build a Mininet network emulator to experiment with software defined networking.

name: test
topology:
  nodes:
    leaf1:
      kind: linux
      image: sflow/frr
    leaf2:
      kind: linux
      image: sflow/frr
    spine1:
      kind: linux
      image: sflow/frr
    spine2:
      kind: linux
      image: sflow/frr
    h1:
      kind: linux
      image: alpine:latest
    h2:
      kind: linux
      image: alpine:latest
  links: 
    - endpoints: ["leaf1:eth1","spine1:eth1"]
    - endpoints: ["leaf1:eth2","spine2:eth1"]
    - endpoints: ["leaf2:eth1","spine1:eth2"]
    - endpoints: ["leaf2:eth2","spine2:eth2"]
    - endpoints: ["h1:eth1","leaf1:eth3"]
    - endpoints: ["h2:eth1","leaf2:eth3"]

The test.yml file shown above specifies the topology. In this case we are using FRRouting (FRR) containers for the leaf and spine switches and Alpine Linux containers for the two hosts.

sudo containerlab deploy --topo test.yml

The above command creates the virtual network and starts containers for each of the network nodes.

sudo containerlab inspect --topo test.yml

Type the command above to list the container instances in the topology.

The table shows each of the containers and the assigned IP addresses.

sudo docker exec -it clab-test-leaf1 vtysh

Type the command above to run the FRR VTY shell so that the switch can be configured.

leaf1# show running-config 
Building configuration...

Current configuration:
!
frr version 7.5_git
frr defaults datacenter
hostname leaf1
log stdout
!
interface eth3
 ip address 172.16.1.1/24
!
router bgp 65006
 bgp router-id 172.20.20.6
 bgp bestpath as-path multipath-relax
 bgp bestpath compare-routerid
 neighbor fabric peer-group
 neighbor fabric remote-as external
 neighbor fabric description Internal Fabric Network
 neighbor fabric capability extended-nexthop
 neighbor eth1 interface peer-group fabric
 neighbor eth2 interface peer-group fabric
 !
 address-family ipv4 unicast
  network 172.16.1.0/24
 exit-address-family
!
route-map ALLOW-ALL permit 100
!
ip nht resolve-via-default
!
line vty
!
end

The BGP configuration for leaf1 is shown above.

Note: We are using BGP unnumbered to simplify the configuration so peers are automatically discovered.

The other switches, leaf2, spine1, and spine2 have similar configurations.

Next we need to configure the hosts.

sudo docker exec -it clab-test-h1 sh

Open a shell on h1

ip addr add 172.16.1.2/24 dev eth1
ip route add 172.16.2.0/24 via 172.16.1.1

Configure networking on h1. The other host, h2, has a similar configuration.

sudo docker exec -it clab-test-h1 ping 172.16.2.2
PING 172.16.2.2 (172.16.2.2): 56 data bytes
64 bytes from 172.16.2.2: seq=0 ttl=61 time=0.928 ms
64 bytes from 172.16.2.2: seq=1 ttl=61 time=0.160 ms
64 bytes from 172.16.2.2: seq=2 ttl=61 time=0.201 ms

Use ping to verify that there is connectivity between h1 and h2.

apk add iperf3

Install iperf3 on h1 and h2

iperf3 -s --bind 172.16.2.2

Run an iperf3 server on h2

iperf3 -c 172.16.2.2
Connecting to host 172.16.2.2, port 5201
[  5] local 172.16.1.2 port 52066 connected to 172.16.2.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.41 GBytes  12.1 Gbits/sec    0   1.36 MBytes       
[  5]   1.00-2.00   sec  1.41 GBytes  12.1 Gbits/sec    0   1.55 MBytes       
[  5]   2.00-3.00   sec  1.44 GBytes  12.4 Gbits/sec    0   1.55 MBytes       
[  5]   3.00-4.00   sec  1.44 GBytes  12.3 Gbits/sec    0   2.42 MBytes       
[  5]   4.00-5.00   sec  1.46 GBytes  12.6 Gbits/sec    0   3.28 MBytes       
[  5]   5.00-6.00   sec  1.42 GBytes  12.2 Gbits/sec    0   3.28 MBytes       
[  5]   6.00-7.00   sec  1.44 GBytes  12.4 Gbits/sec    0   3.28 MBytes       
[  5]   7.00-8.00   sec  1.28 GBytes  11.0 Gbits/sec    0   3.28 MBytes       
[  5]   8.00-9.00   sec  1.40 GBytes  12.0 Gbits/sec    0   3.28 MBytes       
[  5]   9.00-10.00  sec  1.25 GBytes  10.7 Gbits/sec    0   3.28 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  13.9 GBytes  12.0 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  13.9 GBytes  12.0 Gbits/sec                  receiver

Run an iperf3 test on h1

Now that we have a working test network, it's time to add some monitoring.

We will be installing sFlow agents on the switches and hosts that will stream telemetry to sFlow-RT analytics software which will provide a real-time network-wide view of performance.

sudo docker exec -it clab-test-leaf1 sh

Open a shell on leaf1

apk --update add libpcap-dev build-base linux-headers gcc git
git clone https://github.com/sflow/host-sflow.git
cd host-sflow/
make FEATURES="PCAP"
make install

Install Host sFlow agent on leaf1.

Note: The steps above could be included in a Dockerfile in order to create an image with built-in instrumentation.

vi /etc/hsflowd.conf

Edit the Host sFlow configuration file.

sflow {
  polling = 30
  sampling = 400
  collector { ip = 172.20.20.1 }
  pcap { dev = eth1 }
  pcap { dev = eth2 }
  pcap { dev = eth3 }
}

The above settings enable packet sampling on interfaces eth1, eth2 and eth3

sudo docker exec -d clab-test-leaf1 /usr/sbin/hsflowd -d

Start the Host sFlow agent on leaf1.

Install and run Host sFlow agents on the remaining switches and hosts, leaf2, spine1, spine2, h1, and h2.

sudo docker run --rm -d -p 6343:6343/udp -p 8008:8008 --name sflow-rt sflow/prometheus

Use the pre-built sflow/prometheus container to start an instance of sFlow-RT to collect and analyze the telemetry.

multipass list

List the multipass virtual machines.

containerlab            Running           192.168.64.7     Ubuntu 20.04 LTS
                                          172.17.0.1
                                          172.20.20.1

Use a web browser to connect to the sFlow-RT web interface. In this case at http://192.168.64.7:8008 

The sFlow-RT dashboard verifies that telemetry is being received from 6 agents (the four switches and two hosts).

The screen capture shows a real-time view of traffic flowing across the network during an iperf3 test. 

The chart shows that the traffic flows via spine2. Repeated tests showed that traffic traffic was never taking the path via spine1, indicating that the ECMP hash function was not taking into account the TCP ports.

sudo docker exec clab-test-leaf1 sysctl -w net.ipv4.fib_multipath_hash_policy=1
sudo docker exec clab-test-leaf2 sysctl -w net.ipv4.fib_multipath_hash_policy=1

We are using a newer Linux kernel, so running the above commands changes the hashing algorithm to include the layer 4 headers, see Celebrating ECMP in Linux — part one and Celebrating ECMP in Linux — part two.

Topology describes how knowledge of network topology can be used to enhance the analytics capabilities of sFlow-RT.

{
  "links": {
    "link1": { "node1":"leaf1","port1":"eth1","node2":"spine1","port2":"eth1"},
    "link2": { "node1":"leaf1","port1":"eth2","node2":"spine2","port2":"eth1"},
    "link3": { "node1":"leaf2","port1":"eth1","node2":"spine1","port2":"eth2"},
    "link4": { "node1":"leaf2","port1":"eth2","node2":"spine2","port2":"eth2"}
  }
}

The links specification in the test.yml file can easily be converted into sFlow-RT's JSON format.

CONTAINERlab is a very promising tool for efficiently emulating complex networks. CONTAINERlab supports NokiaSR-Linux, Juniper vMX, Cisco IOS XRv9k and Arista vEOS, as well as Linux containers. Many of the proprietary network operating systems are only delivered as virtual machines and Vrnetlab integration makes it possible for CONTAINERlab to run these virtual machines. However, virtual machine nodes require considerably more resources than simple containers.

Linux with open source routing software (FRRouting) is an accessible alternative to vendor routing stacks (no registration / license required, no restriction on copying means you can share images on Docker Hub, no need for virtual machines).  FRRouting is popular in production network operating systems (e.g. Cumulus Linux, SONiC, DENT, etc.) and the VTY shell provides an industry standard CLI for configuration, so labs built around FRR allow realistic network configurations to be explored.

Multipass

Multipass is a command line tool for running Ubuntu virtual machines on Mac or Windows. Multipass uses the native virtualization capabilities of the host operating system to simplify the creation of virtual machines.

Docker testbed and Docker DDoS testbed describe how to use containers to experiment with network visibility and control. However, not all software is amenable to running in containers, and so the ability to quickly create and configure virtual machines is a useful complement. This article demonstrates how to use Multipass to quickly build a virtual machine to run Mininet network emulation software.

multipass launch --name=mininet bionic
multipass exec mininet -- sudo apt update
multipass exec mininet -- sudo apt -y install mininet python-ryu
multipass exec mininet -- sudo apt -y install default-jre python-requests hping3
multipass exec mininet -- wget https://inmon.com/products/sFlow-RT/java11/sflow-rt.tar.gz
multipass exec mininet -- tar -xzf sflow-rt.tar.gz
multipass exec mininet -- ./sflow-rt/get-app.sh sflow-rt mininet-dashboard

Run the above commands in a terminal to create the virtual machine. Multipass commands can easily be scripted to automate the creation and configuration of virtual machines.

multipass list

List the virtual machines.

Name                    State             IPv4             Image
test                    Running           192.168.64.2     Ubuntu 18.04 LTS

Find the IP address of the mininet virtual machine we just created (192.168.64.2).

multipass exec mininet -- ./sflow-rt/start.sh

Start sFlow-RT.

Use a web browser to connect to the VM and access the Mininet Dashboard application running on sFlow-RT, in this case http://192.168.64.2:8008/app/mininet-dashboard/html/

multipass exec mininet -- sudo mn --custom sflow-rt/extras/sflow.py \
--link tc,bw=10 --topo tree,depth=2,fanout=2 --test iperf

In a separate terminal, run a test using Mininet.

The Mininet Dashboard shows the test traffic, the links in the emulated network carrying the traffic, and the diameter of the emulated network.

The Topology tab shows a Mininet weathermap, displaying a dynamic view of topology in which link widths update in real-time to reflect traffic flowing over the virtual network.

An external SDN controller can be used to control packet forwarding in the network.

multipass exec mininet -- ryu-manager ryu.app.simple_switch,ryu.app.ofctl_rest

Start the Ryu SDN controller in a terminal window.

multipass exec mininet -- sudo mn --custom sflow-rt/extras/sflow.py \
--link tc,bw=10 --topo tree,depth=2,fanout=2 \
--controller=remote,ip=127.0.0.1 --test iperf

Run the same test as before, but this time connecting to the SDN controller. For a more interesting example, Ryu measurement based control describes how to detect and block denial of service attacks using the controller.

There are additional multipass commands available to manage the virtual machine.

multipass shell mininet

Connect to the virtual machine and access a command shell.

multipass stop mininet

Stop the virtual machine.

multipass start mininet

Start the virtual machine.

multipass delete mininet
multipass purge

Delete the virtual machine.

There are a number of Mininet articles on this blog describing projects that can be run using the virtual machine that demonstrate performance aware software defined networking concepts.