Tracing network packets with eBPF and pwru

pwru (packet, where are you?) is an open source tool from Cilium that used eBPF instrumentation in recent Linux kernels to trace network packets through the kernel.

In this article we will use Multipass to create a virtual machine to experiment with pwru. Multipass is a command line tool for running Ubuntu virtual machines on Mac or Windows. Multipass uses the native virtualization capabilities of the host operating system to simplify the creation of virtual machines.

multipass launch --name=ebpf noble
multipass exec ebpf -- sudo apt update
multipass exec ebpf -- sudo apt -y install git clang llvm make libbpf-dev flex bison golang
multipass exec ebpf -- git clone https://github.com/cilium/pwru.git
multipass exec ebpf --working-directory pwru -- make
multipass exec ebpf -- sudo ./pwru/pwru -h

Run the commands above to create the virtual machine and build pwru from sources.

multipass exec ebpf -- sudo ./pwru/pwru port https

Run pwru to trace https traffic on the virtual machine.

multipass exec ebpf -- curl https://sflow-rt.com

In a second window, run the above command to generate an https request from the virtual machine.

SKB                CPU PROCESS          NETNS      MARK/x        IFACE       PROTO  MTU   LEN   TUPLE FUNC
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0               0         0x0000 1500  60    192.168.66.3:47460->54.190.130.38:443(tcp) ip_local_out
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0               0         0x0000 1500  60    192.168.66.3:47460->54.190.130.38:443(tcp) __ip_local_out
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0               0         0x0800 1500  60    192.168.66.3:47460->54.190.130.38:443(tcp) ip_output
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  60    192.168.66.3:47460->54.190.130.38:443(tcp) nf_hook_slow
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  60    192.168.66.3:47460->54.190.130.38:443(tcp) apparmor_ip_postroute
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  60    192.168.66.3:47460->54.190.130.38:443(tcp) ip_finish_output
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  60    192.168.66.3:47460->54.190.130.38:443(tcp) __ip_finish_output
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  60    192.168.66.3:47460->54.190.130.38:443(tcp) ip_finish_output2
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  60    192.168.66.3:47460->54.190.130.38:443(tcp) neigh_resolve_output
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  60    192.168.66.3:47460->54.190.130.38:443(tcp) eth_header
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  60    192.168.66.3:47460->54.190.130.38:443(tcp) skb_push
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) __dev_queue_xmit
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) qdisc_pkt_len_init
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) netdev_core_pick_tx
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) sch_direct_xmit
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) validate_xmit_skb_list
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) validate_xmit_skb
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) netif_skb_features
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) passthru_features_check
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) skb_network_protocol
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) skb_csum_hwoffload_help
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) skb_checksum_help
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) skb_ensure_writable
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) validate_xmit_xfrm
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) dev_hard_start_xmit
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) start_xmit
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) skb_clone_tx_timestamp
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  74    192.168.66.3:47460->54.190.130.38:443(tcp) xmit_skb
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  86    192.168.66.3:47460->54.190.130.38:443(tcp) skb_to_sgvec
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0             ens3:2      0x0800 1500  86    192.168.66.3:47460->54.190.130.38:443(tcp) __skb_to_sgvec

The pwru output provides a detailed trace of each packet through the Linux stack, identifying each application, namespace, interface and kernel module traversed by the packet. Type Ctrl+C to stop the trace.

multipass exec ebpf -- sudo ufw default allow
multipass exec ebpf -- sudo ufw deny out to any port https
multipass exec ebpf -- sudo ufw enable

Now create a firewall rule to block outgoing https traffic and repeat the test

SKB                CPU PROCESS          NETNS      MARK/x        IFACE       PROTO  MTU   LEN   TUPLE FUNC
0xffff970bc5f568e8 0   ~r/bin/curl:7138 4026531840 0               0         0x0000 1500  60    192.168.67.5:38932->54.190.130.38:443(tcp) ip_local_out
0xffff970bc5f568e8 0   ~r/bin/curl:7138 4026531840 0               0         0x0000 1500  60    192.168.67.5:38932->54.190.130.38:443(tcp) __ip_local_out
0xffff970bc5f568e8 0   ~r/bin/curl:7138 4026531840 0               0         0x0800 1500  60    192.168.67.5:38932->54.190.130.38:443(tcp) nf_hook_slow
0xffff970bc5f568e8 0   ~r/bin/curl:7138 4026531840 0               0         0x0800 1500  60    192.168.67.5:38932->54.190.130.38:443(tcp) kfree_skb_reason(SKB_DROP_REASON_NETFILTER_DROP)
0xffff970bc5f568e8 0   ~r/bin/curl:7138 4026531840 0               0         0x0800 1500  60    192.168.67.5:38932->54.190.130.38:443(tcp) skb_release_head_state
0xffff970bc5f568e8 0   ~r/bin/curl:7138 4026531840 0               0         0x0800 0     60    192.168.67.5:38932->54.190.130.38:443(tcp) tcp_wfree
0xffff970bc5f568e8 0   ~r/bin/curl:7138 4026531840 0               0         0x0800 0     60    192.168.67.5:38932->54.190.130.38:443(tcp) skb_release_data
0xffff970bc5f568e8 0   ~r/bin/curl:7138 4026531840 0               0         0x0800 0     60    192.168.67.5:38932->54.190.130.38:443(tcp) kfree_skbmem

This time, the curl command hangs and the pwru trace shows that packets are being dropped in the Linux firewall.

multipass exec ebpf -- sudo ufw disable

Disable the firewall.

There are additional multipass commands available to manage the virtual machine.

multipass shell ebpf

Connect to the virtual machine and access a command shell.

multipass stop ebpf

Stop the virtual machine.

multipass start ebpf

Start the virtual machine.

multipass delete ebpf
multipass purge

Delete the virtual machine.