Systemd traffic marking

Monitoring Linux services describes how the open source Host sFlow agent exports metrics from services launched using systemd, the default service manager on most recent Linux distributions. In addition, the Host sFlow agent efficiently samples network traffic using Linux kernel capabilities: PCAP/BPF, nflog, and ulog.

This article describes a recent extension to the Host sFlow systemd module, mapping sampled traffic to the individual services the generate or consume them. The ability to color traffic by application greatly simplifies service discovery and service dependency mapping; making it easy to see how services communicate in a multi-tier application architecture.

The following /etc/hsflowd.conf file configures the Host sFlow agent, hsflowd, to sampling packets on interface eth0, monitor systemd services and mark the packet samples, and track tcp performance:

sflow {
  collector { ip = 10.0.0.70 }
  pcap { dev = eth0 }
  systemd { markTraffic = on }
  tcp { }
}

The diagram above illustrates how the Host sFlow agent is able to efficiently monitor and classify traffic. In this case both the Host sFlow agent and an Apache web server are are running as services managed by systemd. A network connection , shown in red, to the HTTP service. In this case, configuring the pcap module to monitor interface eth0 on the server programs a Berkeley Packet Filter (BPF) that randomly samples packets in the Linux kernel and provides copies (shown as the dotted red line) to the Host sFlow agent. In addition, the Host sFlow agent queries systemd to obtain a list of running services and the resources allocated to them. Further Linux kernel tables are queried to identify the network sockets that were opened by each service.

The Host sFlow then attaches an additional record to exported packet samples to indicate the services generating or consuming the packets:

/* Traffic source/sink entity reference */
/* opaque = flow_data; enterprise = 0; format = 2210 */
/* Set Data source to all zeroes if unknown
struct extended_entities {
 sflow_data_source_expanded src_ds;    /* Data Source associated with
                                          packet source */
 sflow_data_source_expanded dst_ds;    /* Data Source associated with
                                          packet destination */
}

Note: The data source references point to the performance metrics exported by the systemd module, see Monitoring Linux services.

Finally, enabling the tcp module adds delay, retransmit, loss, and reordering information to the sampled packet, see Network performance monitoring.

The screen capture above shows network traffic colored by service name. The chart colors traffic associated with the httpd.service in blue, remote login traffic associated with the sshd.service in red, BGP traffic associated with the bird.service in gold, and traffic to the inmsfd.service in green.

The chart was generated using the open source Flow Trend application running on the sFlow-RT real-time analytics platform. The chart is the result of the following flow definition:

host:[or:dssource:dsdestination]:vir_host_name

The host: function is used to join information from the sampled flow with telemetry reported for each of the services. Additional keys can be added to the flow definition to break out the traffic by network addresses, quality of service, or any of the many properties reported by sFlow, see Defining Flows for additional information.

Networking on the host has been referred to as the "Goldilocks Zone" because the host provides context that is unavailable in network switches and routers. The sFlow standard defines measurements that network, host, and application entities send in a continuous telemetry stream to analytics software that can combine the data to provide a comprehensive end-to-end view of activity, see sFlow Host Structures.