Wednesday, March 17, 2021

Transit delay and queueing


The recently finalized sFlow Transit Delay Structures extension provides visibility into the performance of packet forwarding in a switch or router using the industry standard sFlow protocol.

The diagram provides a logical representation of packet forwarding. A packet is received at an Ingress Port, the packet header is examined and a forwarding decision is made to add the packet to one of the queues associated with an Egress Port, finally the packet is removed from the queue and sent out the Egress Port to be received by the next device in the chain.

The time between sending and receiving a packet is the packet's transit delay. The transit delay is affected by the time it takes to make the forwarding decision and the time the packet spends in the queue. Identifying the specific queue selected and the number of bytes already in the queue fills out the set of performance metrics for the forwarding decision. The sFlow Transit Delay Structures extension adds these performance metrics to the metadata associated with each packet sample. 

The following output from sflowtool shows that data contained in a packet sample:

startSample ----------------------
sampleType_tag 0:1
sampleType FLOWSAMPLE
sampleSequenceNo 91159
sourceId 0:2216
meanSkipCount 400
samplePool 36463600
dropEvents 0
inputPort 2215
outputPort 2216
flowBlock_tag 0:1036
extendedType egress_queue
egress_queue_id 7
flowBlock_tag 0:1040
extendedType queue_depth
queue_depth_bytes 11354112
flowBlock_tag 0:1039
extendedType transit_delay
transit_delay_nS 839660224
flowBlock_tag 0:1
flowSampleType HEADER
headerProtocol 1
sampledPacketSize 1446
strippedBytes 4
headerLen 128
headerBytes 98-03-9B-8F-B5-CC-98-03-9B-94-C7-D5-08-00-45-16-05-94-12-C7-00-00-FE-11-B8-43-C0-00-02-02-C6-33-64-02-30-39-D4-31-05-80-D7-1D-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42-42
dstMAC 98039b8fb5cc
srcMAC 98039b94c7d5
IPSize 1428
ip.tot_len 1428
srcIP 192.0.2.2
dstIP 198.51.100.2
IPProtocol 17
IPTOS 22
IPTTL 254
IPID 50962
UDPSrcPort 12345
UDPDstPort 54321
UDPBytes 1408
endSample   ----------------------

The forwarding performance information is highlighted. The inputPort, outputPort, egress_queue_id, queue_depth_bytes, and transit_delay_nS values describe the performance observed by the sampled packet. The sampled packet header allows performance to be reported by specific hosts, protocols, ports, connections, etc.

Linux 4.11 kernel extends packet sampling support describes the Linux PSAMPLE interface used by the Host sFlow agent to receive packet samples. PSAMPLE has been extended in the Linux 5.13 kernel to add the performance metrics, PSAMPLE_ATTR_OUT_TC (egress queue), PSAMPLE_ATTR_OUT_TC_OCC (egress queue depth), and PSAMPLE_ATTR_LATENCY (transit delay) needed to populate the sFlow Transit Delay Structures and the Host sFlow agent now exports the performance data when it is available. The decoded sFlow record, above, was generated by Host sFlow running on a hardware switch and shows measurements made by the switch ASIC.

PSAMPLE and the Host sFlow agent are becoming the standard for sFlow monitoring of Linux based operating systems such as Cumulus Linux, DENT, and SONiC.  As ASIC vendors include the measurements in their device driver PSAMPLE support, they will automatically be included in the sFlow telemetry.

Support for the new extensions has also been added to the sFlow-RT real-time analytics engine. The open source sFlow-RT Flow Browser application shown in the screen shot above displays a real-time, up to the second, view of traffic based on the packet sample telemetry streaming from network devices (switches, routers, and hosts).

In the chart above, the value being plotted has been changed from Bits per Second and is now displaying flows with the highest transit delay (in nanoseconds). The specific device, ingress port, egress port, and egress queue are also identified. 

In the chart above, queue depth (in bytes) is displayed, showing that the nearly 12 Mbytes queue depth is responsible for the transit delay seen in the previous chart.

If the queue is full and the packet is dropped, the sFlow Dropped Packet Notification Structures extension allows the sFlow agent to report details of the dropped packet. Using sFlow to monitor dropped packets describes how the Host sFlow agent uses the Linux drop_monitor interface to implement the extension.

In the final chart above, the open source sFlow-RT Discard Browser application displays a sequence of packets being dropped by a switch as a host attempts, and fails, to establish a TCP connection. The reason for dropping the packets (an access control list) as well as device and ingress port where the packets were dropped are captured.

Transit delay and dropped packet monitoring leverage advanced instrumentation in the latest generation of network ASICs to provide valuable insight into network performance. Integration with industry standard streaming sFlow telemetry provides real-time network-wide visibility into traffic, performance, and errors.

Tuesday, March 9, 2021

InfluxDB 2.0 released


InfluxData advances possibilities of time series data with general availability of InfluxDB 2.0 announced the production release of InfluxDB 2.0. This article demonstrates how to import sFlow data into InfluxDB 2.0 using sFlow-RT in order to provide visibility into network traffic.

Real-time network and system metrics as a service describes how to use Docker Desktop to replay previously captured sFlow data. Follow the instructions in the article to start an instance of sFlow-RT.

Create a directory for InfluxDB to use to store data and configuration settings:
mkdir data
Now start InfluxDB using the pre-built influxdb image:
docker run --rm --name=influxdb -p 8086:8086 \
-v  $PWD/data:/var/lib/influxdb2 influxdb:alpine \
--nats-max-payload-bytes=10000000

Note: sFlow-RT is collecting metrics for all the sFlow agents embedded in switches, routers, and servers. The default value of nats-max-payload-bytes (1048576) may be too small to hold all the metrics returned when sFlow-RT is queried. The error,  nats: maximum payload exceeded, in InfluxDB logs indicates that the limit needs to be increased. In this example, the value has been increased to 10000000.

Now access the InfluxDB web interface at http://localhost:8086/

The screen capture above shows three scrapers configured in InfluxDB 2.0:
  1. sflow-analyzer
    URL: http://host.docker.internal:8008/prometheus/analyzer/txt
  2. sflow-metrics
    URL: http://host.docker.internal:8008/prometheus/metrics/ALL/ALL/txt
  3. sflow-flow-src-dst
    URL: http://host.docker.internal:8008/app/prometheus/scripts/export.js/flows/ALL/txt?metric=flow_src_dst_bps&key=ipsource,ipdestination&value=bytes&aggMode=max&maxFlows=100&minValue=1000&scale=8
The first collects metrics about the performance of the sFlow-RT analytics engine, the second, all the metrics exported by the sFlow agents, and the third, is a flow metric.
InfluxDB 2.0 now includes the data exploration and dashboard building capabilities that were previously in the separate Chronograf application. The screen capture above shows a simple chart trending the flow metric.

Monday, March 1, 2021

DDoS Mitigation with Juniper, sFlow, and BGP Flowspec

Real-time DDoS mitigation using BGP RTBH and FlowSpec, DDoS protection of local address space, Pushing BGP Flowspec rules to multiple routers, Monitoring DDoS mitigation, and Docker DDoS testbed demonstrate how sFlow and BGP Flowspec are combined by the DDoS Protect application running on the sFlow-RT real-time analytics engine to automatically detect and block DDoS attacks.

This article discusses how to deploy the DDoS Protect application in a Juniper Networks environment. Juniper has a long history of supporting BGP Flowspec on their routing platforms and Juniper has added support for sFlow to their entire product range, see sFlow available on Juniper MX series routers.

First, Junos doesn't provide a way to connect to the non-standard BGP port (1179) that sFlow-RT uses by default. Allowing sFlow-RT to open the standard BGP port (179) requires that the service be given additional Linux capabilities. 

docker run --rm --net=host --sysctl net.ipv4.ip_unprivileged_port_start=0 \
sflow/ddos-protect -Dbgp.port=179

The above command launches the prebuilt sflow/ddos-protect Docker image. Alternatively, if sFlow-RT has been installed as a deb / rpm package, then the required permissions can be added to the service.

sudo systemctl edit sflow-rt.service
Type the above command to edit the service configuration and add the following lines:
[Service]
AmbientCapabilities=CAP_NET_BIND_SERVICE
Next, edit the sFlow-RT configuration file for the DDoS Protect application:
sudo vi /usr/local/sflow-rt/conf.d/ddos-protect.conf
and add the line:
bgp.port=179
Finally, restart sFlow-RT:
sudo systemctl restart sflow-rt
The application is now listening for BGP connections on TCP port 179.

Now configure the router to send sFlow telemetry to sFlow-RT - see Junos: sFlow Monitoring Technology
set protocols sflow collector 192.168.65.2 udp-port 6343
set protocols sflow polling-interval 20
set protocols sflow sample-rate ingress 1000
set protocols sflow interfaces ge-0/0/0
set protocols sflow interfaces ge-0/0/1
...
For example, the above commands enable sFlow monitoring on a Juniper MX router. See sFlow-RT Agents for recommended sFlow configuration settings.

Also configure a BGP Flowspec session with sFlow-RT - see Junos: Multiprotocol BGP.
policy-options {
    policy-statement ACCEPT_ALL {
        from protocol bgp;
        then accept;
    }
}
routing-options {
    router-id 1.1.1.1;
    autonomous-system 65000;
}
protocols {
    bgp {
        group sflow-rt {
            type internal;
            local-address 172.17.0.2;
            family inet {
                unicast;
                flow {
                    no-validate ACCEPT_ALL;
                }
            }
            family inet6 {
                unicast;
                flow {
                    no-validate ACCEPT_ALL;
                }
            }
            neighbor 192.168.65.2 {
                import ACCEPT_ALL;
                peer-as 65000;
            }
        }
    }
}
The above configuration establishes the BGP Flowspec session with sFlow-RT.

Real-time DDoS mitigation using BGP RTBH and FlowSpec describes how to simulate a DDoS UDP amplification attack in order to test the automated detection and control functionality.  
root@07358a106c21> show route table inetflow.0 detail    

inetflow.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
192.0.2.129,*,proto=17,srcport=53/term:N/A (1 entry, 0 announced)
        *BGP    Preference: 170/-101
                Next hop type: Fictitious, Next hop index: 0
                Address: 0x55653aae979c
                Next-hop reference count: 1
                Next hop: 
                State: <Active Int Ext SendNhToPFE>
                Local AS: 65000 Peer AS: 65000
                Age: 6 
                Validation State: unverified 
                Task: BGP_65000.192.168.65.2
                AS path: I 
                Communities: traffic-rate:0:0
                Accepted
                Localpref: 100
                Router ID: 0.6.6.6
Command line output from the router shown above verifies that a Flowspec control blocking the amplification attack has been received. The control will remain in place for 60 minutes (the configured timeout), after which it will be automatically withdrawn. If the attack is still in progress it will be immediately detected and the control reapplied.

DDoS Protect can mitigate a wide range of common attacks, including: NTP, DNS, Memcached, SNMP, and SSDP amplification attacks; IP, UDP, ICMP and TCP flood attacks; and IP fragmentation attacks. Mitigation options include: remote triggered black hole (RTBH), filtering, rate limiting, DSCP marking, and redirection. IPv6 is fully supported in detection and mitigation of each of these attack types.

Monday, January 25, 2021

Topology


Real-time network and system metrics as a service describes how to use data captured from the network shown above to explore the functionality of sFlow-RT real-time analytics software. This article builds on the previous article to show how knowledge of network topology can be used to enhance analytics, see Topology for documentation.

First, follow the instructions in the previous example and start an instance of sFlow-RT using the captured sFlow.  
curl -O https://raw.githubusercontent.com/sflow-rt/fabric-view/master/demo/topology.json
Then, download the topology file for the example.
curl -X PUT -H "Content-Type: application/json" -d @topology.json \
http://localhost:8008/topology/json
Install the topology using the sFlow-RT REST API.
curl http://localhost:8008/topology/json
Retrieve the topology.
{
 "version": 0,
 "links": {
  "L1": {
   "node2": "spine1",
   "node1": "leaf1",
   "port1": "swp1s0",
   "port2": "swp49"
  },
  "L2": {
   "node2": "spine1",
   "node1": "leaf1",
   "port1": "swp1s1",
   "port2": "swp50"
  },
  "L3": {
   "node2": "spine2",
   "node1": "leaf1",
   "port1": "swp1s2",
   "port2": "swp51"
  },
  "L4": {
   "node2": "spine2",
   "node1": "leaf1",
   "port1": "swp1s3",
   "port2": "swp52"
  },
  "L5": {
   "node2": "spine2",
   "node1": "leaf2",
   "port1": "swp1s0",
   "port2": "swp49"
  },
  "L6": {
   "node2": "spine2",
   "node1": "leaf2",
   "port1": "swp1s1",
   "port2": "swp50"
  },
  "L7": {
   "node2": "spine1",
   "node1": "leaf2",
   "port1": "swp1s2",
   "port2": "swp51"
  },
  "L8": {
   "node2": "spine1",
   "node1": "leaf2",
   "port1": "swp1s3",
   "port2": "swp52"
  }
 }
}
The JSON topology structure contains the eight links connecting the leaf and spine switches in the diagram, identifying the nodes and ports associated with each link.
curl -H "Content-Type:application/json" -X PUT \
--data '{"keys":"ipsource,ipdestination",value:"bytes"}' \
http://localhost:8008/flow/srcdst/json
Now define the srcdst flow metric described in the previous article.
curl "http://localhost:8008/activeflows/TOPOLOGY/srcdst/json?aggMode=edge"
Knowledge of topology opens up additional options when querying for flows. For example, the above command only considers devices that are part of the topology and sums flows entering edge device access ports, i.e. traffic entering the the leaf switches from the servers.
[
 {
  "flowN": 1,
  "value": 248800.14028768288,
  "key": "10.4.3.2,10.4.4.2"
 },
 {
  "flowN": 1,
  "value": 176879.3798722214,
  "key": "10.4.1.2,10.4.2.2"
 },
 {
  "flowN": 1,
  "value": 526.0366052656848,
  "key": "10.4.4.2,10.4.3.2"
 },
 {
  "flowN": 1,
  "value": 375.06686598182193,
  "key": "10.4.2.2,10.4.1.2"
 }
]
The result accurately reports the amount of traffic being exchanged between the servers, discarding duplicate data reported as traffic flows traverse the links between switches.
-Dbrowse-flows.agents=TOPOLOGY -Dbrowse-flows.aggMode=edge
Adding the above arguments to the end of the command line used to start sFlow-RT configures the Flow Browser application to use the topology de-duplication method.
Click on the link below to plot a graph of the top IP Protocols using the browse-flows application (screen capture shown above):
http://localhost:8008/app/browse-flows/html/index.html?keys=ipprotocol&value=bps
Note: No data will be shown until the topology is posted to sFlow-RT.
function print(label,obj) {
  logInfo(label+"="+JSON.stringify(obj));
}

setFlow('protocol',{keys:'ipprotocol',value:'bytes'});

setIntervalHandler(function() {
  print("locate_mac",topologyLocateHostMac('000AF725C062'));
  print("locate_ip",topologyLocateHostIP('10.4.3.2'));
  print("flow_max",activeFlows('ALL','protocol',5,0,'max'));
  print("flow_sum",activeFlows('ALL','protocol',5,0,'sum'));
  print("flow_edge",activeFlows('TOPOLOGY','protocol',5,0,'edge'));
});
The demo.js script shown above uses sFlow-RT's embedded scripting API, see Writing Applications. The script defines the flow called protocol that tracks top IP Protocols and prints out the top flows with different aggregation methods. The script also demonstrates an additional capability made possible when topology is known. The topologyLocateHostMac() and topologyLocateHostIP() function locates an addresses to the edge port connecting them to the network.
-Dscript.file=$PWD/demo.js
Run the script by adding the above argument to the end of the command line used to run sFlow-RT.
2021-01-22T17:08:35-08:00 INFO: locate_mac=[{"ipaddress":"10.4.3.2","node":"leaf1","agent":"192.168.0.11","ifindex":"38","port":"swp32s1","mac":"000AF725C062"}]
2021-01-22T17:08:35-08:00 INFO: locate_ip=[{"ipaddress":"10.4.3.2","node":"leaf1","agent":"192.168.0.11","ifindex":"38","port":"swp32s1","mac":"000AF725C062"}]
2021-01-22T17:08:35-08:00 INFO: flow_max=[{"flowN":16,"agent":"192.168.0.14","value":1208583376.782055,"dataSource":"54","key":"6"}]
2021-01-22T17:08:35-08:00 INFO: flow_sum=[{"flowN":16,"value":6615963204.827695,"key":"6"}]
2021-01-22T17:08:35-08:00 INFO: flow_edge=[{"flowN":4,"value":2104039983.2917378,"key":"6"}]
The output from the script shows that the addresses were located to leaf1 port swp32s1. The flow_max and flow_sum queries don't use the topology and combine data from all 16 data sources (switch ports) that are reporting traffic. The sum mode returns the largest value since traffic is added for every data source. The max mode finds the data source reporting the largest value for the flow and reports that value (agent: 192.168.0.14, dataSource: 54). The edge mode is equivalent to the REST query used earlier.

Mininet is a network emulator that you can run on your laptop in a virtual machine (e.g. using Multipass) that provides a useful platform for building virtual topologies and exploring topology related analytics. Mininet dashboard, Mininet weathermap, and Mininet, ONOS, and segment routing provide examples.


Ideally the network configuration and topology will be available in a centralized repository that can be queried to generate the information required by sFlow-RT. Alternatively, Link Layer Discovery Protocol (LLDP) data retrieved from network devices can be used to construct the topology. Fabric Visibility, Arista EOS CloudVision, and Fabric visibility with Cumulus Linux provide examples.

Thursday, December 10, 2020

FRR

FRRouting (FRR) is an IP routing protocol suite for Linux and Unix platforms which includes protocol daemons for BGP, IS-IS, LDP, OSPF, PIM, and RIP.

This article uses the Docker DDoS testbed to explore FRRouting's BGP Flowspec capability. Follow the steps in the Docker DDoS testbed article, but instead of running ExaBGP, run the following command to start the pre-built sflow/frr image:
docker run --rm -d --privileged --name frr sflow/frr
Run the following command to show the router configuration:
docker exec frr vtysh -c "show running-config"
The results will be displayed:
Building configuration...

Current configuration:
!
frr version 7.5_git
frr defaults traditional
hostname ed9e435c6f3d
no ipv6 forwarding
log stdout
!
router bgp 65000
 bgp router-id 0.0.0.1
 neighbor 192.168.65.2 remote-as 65001
 neighbor 192.168.65.2 port 1179
 neighbor 192.168.65.2 ebgp-multihop 255
 neighbor 192.168.65.2 timers connect 10
 !
 address-family ipv4 unicast
  neighbor 192.168.65.2 route-map ALLOW-ALL in
  neighbor 192.168.65.2 route-map ALLOW-ALL out
 exit-address-family
 !
 address-family ipv4 flowspec
  neighbor 192.168.65.2 activate
  neighbor 192.168.65.2 route-map ALLOW-ALL in
  neighbor 192.168.65.2 route-map ALLOW-ALL out
 exit-address-family
 !
 address-family ipv6 unicast
  neighbor 192.168.65.2 activate
  neighbor 192.168.65.2 route-map ALLOW-ALL in
  neighbor 192.168.65.2 route-map ALLOW-ALL out
 exit-address-family
 !
 address-family ipv6 flowspec
  neighbor 192.168.65.2 activate
  neighbor 192.168.65.2 route-map ALLOW-ALL in
  neighbor 192.168.65.2 route-map ALLOW-ALL out
 exit-address-family
!
route-map ALLOW-ALL permit 100
!
ip nht resolve-via-default
!
line vty
!
end

Simulate a DDoS attack as describes in Docker DDoS testbed.

Now run the following command to see the Flowspec rule:

docker exec frr vtysh -c "show bgp ipv4 flowspec detail"

The following rule was sent to the router:

BGP flowspec entry: (flags 0x418)
	Destination Address 172.17.0.1/32
	IP Protocol = 17 
	Source Port = 53 
	FS:rate 0.000000
	received for 00:00:48
	not installed in PBR

Displayed  1 flowspec entries

The rule isn't installed in the policy based routing (PBR) table since the current version of FRRouting doesn't include the necessary Netfilter driver.

FRRouting software is widely used in open source network operating systems such as SONiC, OpenSwitch, and DENT. Once the Flowspec driver integration is complete, the white box switch hardware supported by these network operation systems will provide a cost effective method of mitigating DDoS attacks - combining the real-time visibility of sFlow with the real-time control capabilities of Flowspec.

Thursday, November 19, 2020

Multipass

Multipass is a command line tool for running Ubuntu virtual machines on Mac or Windows. Multipass uses the native virtualization capabilities of the host operating system to simplify the creation of virtual machines.

Docker testbed and Docker DDoS testbed describe how to use containers to experiment with network visibility and control. However, not all software is amenable to running in containers, and so the ability to quickly create and configure virtual machines is a useful complement. This article demonstrates how to use Multipass to quickly build a virtual machine to run Mininet network emulation software.
multipass launch --name=mininet bionic
multipass exec mininet -- sudo apt update
multipass exec mininet -- sudo apt -y install mininet python-ryu
multipass exec mininet -- sudo apt -y install default-jre python-requests hping3
multipass exec mininet -- wget https://inmon.com/products/sFlow-RT/java11/sflow-rt.tar.gz
multipass exec mininet -- tar -xzf sflow-rt.tar.gz
multipass exec mininet -- ./sflow-rt/get-app.sh sflow-rt mininet-dashboard

Run the above commands in a terminal to create the virtual machine. Multipass commands can easily be scripted to automate the creation and configuration of virtual machines.

multipass list
List the virtual machines.
Name                    State             IPv4             Image
test                    Running           192.168.64.2     Ubuntu 18.04 LTS

Find the IP address of the mininet virtual machine we just created (192.168.64.2).

multipass exec mininet -- ./sflow-rt/start.sh

Start sFlow-RT.

Use a web browser to connect to the VM and access the Mininet Dashboard application running on sFlow-RT, in this case http://192.168.64.2:8008/app/mininet-dashboard/html/
multipass exec mininet -- sudo mn --custom sflow-rt/extras/sflow.py \
--link tc,bw=10 --topo tree,depth=2,fanout=2 --test iperf
In a separate terminal, run a test using Mininet.
The Mininet Dashboard shows the test traffic, the links in the emulated network carrying the traffic, and the diameter of the emulated network.
The Topology tab shows a Mininet weathermap, displaying a dynamic view of topology in which link widths update in real-time to reflect traffic flowing over the virtual network.

An external SDN controller can be used to control packet forwarding in the network.
multipass exec mininet -- ryu-manager ryu.app.simple_switch,ryu.app.ofctl_rest
Start the Ryu SDN controller in a terminal window.
multipass exec mininet -- sudo mn --custom sflow-rt/extras/sflow.py \
--link tc,bw=10 --topo tree,depth=2,fanout=2 \
--controller=remote,ip=127.0.0.1 --test iperf
Run the same test as before, but this time connecting to the SDN controller. For a more interesting example, Ryu measurement based control describes how to detect and block denial of service attacks using the controller.

There are additional multipass commands available to manage the virtual machine.
multipass shell mininet
Connect to the virtual machine and access a command shell.
multipass stop mininet
Stop the virtual machine.
multipass start mininet
Start the virtual machine.
multipass delete mininet
multipass purge
Delete the virtual machine.

There are a number of Mininet articles on this blog describing projects that can be run using the virtual machine that demonstrate performance aware software defined networking concepts.

Friday, November 13, 2020

sFlow Monitoring for AI


A Proposal towards sFlow Monitoring Dashboards for AI-controlled NRENs is a recent talk by Mariam Kiran (Esnet) presented at the recent GÉANT Telemetry and Big Data Workshop.
In the talk, Miram describes the set open source tools (Netdata, Prometheus, Zabbix, Ntopng, and PerfSONAR) that they attempted to synthesize a complete picture of the network.
A number of tools were combined since each tool provides a different subset of the measurements needed to drive the AI controller. 
Integrating the data from the different sources was a challenge, but they were able to pull the data together into a single Grafana dashboard. Unfortunately, there was a lot of noise in legacy measurement schemes, making the data set unsuitable for training the AI controller.
The team decided to go toward sFlow, replacing the legacy monitoring tools with sFlow enabled devices, in order to generate the very clean data needed for machine learning.

For background, the talk, Real-time network telemetry for automation, describes why sFlow is uniquely suited to automation, providing the comprehensive, real-time, system-wide, visibility needed to make networked systems observable.