tag:blogger.com,1999:blog-19786529798408290132024-03-12T19:54:33.755-07:00sFlowTelemetry, analytics, and control with sFlow® standardPeterhttp://www.blogger.com/profile/00856599914190257147noreply@blogger.comBlogger490125tag:blogger.com,1999:blog-1978652979840829013.post-59006327188146263112024-02-22T16:15:00.000-08:002024-02-22T16:15:13.913-08:00VyOS 1.4 LTS released<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvg9cZ6Tmn4sc2Klq67WYBwMscAA-niyve2EBW8XRoXipn7UvRc-1gNn3dsaSDAS7WgghepzeCGtyYn2Q4hIbibduIwziG4NemtQmJtIro_B99IfCnrsoELm8LhI2OG6V6Frd2TNpeZj2T3tmvR7-D5g-KWgmnE_dVlHcCQ-gwtaUV6DVR8uy0dv9sMkHX/s679/protectl4.jpg" style="display: block; margin-left: auto; margin-right: auto; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="275" data-original-width="679" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvg9cZ6Tmn4sc2Klq67WYBwMscAA-niyve2EBW8XRoXipn7UvRc-1gNn3dsaSDAS7WgghepzeCGtyYn2Q4hIbibduIwziG4NemtQmJtIro_B99IfCnrsoELm8LhI2OG6V6Frd2TNpeZj2T3tmvR7-D5g-KWgmnE_dVlHcCQ-gwtaUV6DVR8uy0dv9sMkHX/s600/protectl4.jpg" width="600" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://protectli.com/vault-4-port/">Protectli Vault - 4 Port</a></td></tr></tbody></table>
<p>The <a href="https://blog.vyos.io/vyos-1.4.0-lts-release">VyOS 1.4.0 (Sagitta) LTS release</a> announcement is exciting news! <a href="https://vyos.io/">VyOS</a> is an open source router operating system based on Linux that can be installed on commodity PC hardware - for optimal performance at least 1GB RAM and 4GB of storage space is recommended.</p>
<p>The new 1.4 LTS release includes a significantly enhanced implementation of industry standard <a href="https://sflow.org">sFlow</a> telemetry based on the open source <a href="https://sflow.net/">Host sFlow</a> agent.</p>
<pre>set system sflow interface eth0
set system sflow interface eth1
set system sflow interface eth2
set system sflow interface eth3
set system sflow polling 30
set system sflow sampling-rate 1000
set system sflow drop-monitor-limit 50
set system sflow server 192.0.2.100</pre>
Enter the commands above to enable sFlow monitoring on interfaces <i>eth0</i>, <i>eth1</i>, <i>eth2</i>, and <i>eth3</i>. Interface counters will be exported every 30 seconds, packets will be sampled with probability 1/1000, and up to 50 packet headers (and drop reasons) per second will collected from packets dropped by the router. The sFlow telemetry stream will be sent to an sFlow collector at <i>192.0.2.100</i>.
<p>Running <a href="https://www.docker.com/">Docker</a> on the sFlow collector makes it easy to run a variety of sFlow analytics tools.</p>
<pre>docker run --rm -p 6343:6343/udp sflow/sflowtool</pre>
Run the <a href="https://hub.docker.com/r/sflow/sflowtool">sflow/sflowtool</a> image to decode and print the contents of the sFlow telemetry stream and verify receipt of data.
<pre>docker run --rm -p 6343:6343/udp sflow/tcpdump tcp port 80</pre>
Run the <a href="https://hub.docker.com/r/sflow/tcpdump">sflow/tcpdump</a> image to decode and filter sampled packet headers. For more complex packet analysis tasks, try the <a href="https://hub.docker.com/r/sflow/tshark">sflow/tshark</a> image.<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxtS_jY6Er-Lcql7gbG8rRyLoh3NCwCWlF1DLJl_0nNtR5IJeu5EjbGF5lisK5WThzkXZZPRHuci9nY3K5cHiCYfDOREuE2corpzevtXdbDuRayjNuE8wXonwpWupgrGrTMztlQ6-0zW9FiAxYuWnFQ80EBzNgh_Y4JrsZuACR7WbmUF8T4nwNntfcEsaR/s2320/vyos-sflowtrend.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1776" data-original-width="2320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxtS_jY6Er-Lcql7gbG8rRyLoh3NCwCWlF1DLJl_0nNtR5IJeu5EjbGF5lisK5WThzkXZZPRHuci9nY3K5cHiCYfDOREuE2corpzevtXdbDuRayjNuE8wXonwpWupgrGrTMztlQ6-0zW9FiAxYuWnFQ80EBzNgh_Y4JrsZuACR7WbmUF8T4nwNntfcEsaR/s600/vyos-sflowtrend.png" width="600" /></a></div>
Run the <a href="https://hub.docker.com/r/sflow/sflowtrend">sflow/sflowtrend</a> image to trend interface counters and top flows.<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFMvUe1kUGuhc1YN4twGqY4zFP8IjeE-cenghXJiqMJlwTEHYHDExP_NVuCrkDqa7O_0vCezd-uMU_Oz6ZpMuGeliWCdLcbdsWyrpTSXoIXDVKhTHrlBe2s5sLxXMYbTIBuShFKMuQh6v_4gB7_12RVPBXU1gvpZsxNoaPztlHE2DhESnUg1Obpx0Vrg-p/s2152/grafana-rt-country.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="2152" data-original-width="1906" height="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFMvUe1kUGuhc1YN4twGqY4zFP8IjeE-cenghXJiqMJlwTEHYHDExP_NVuCrkDqa7O_0vCezd-uMU_Oz6ZpMuGeliWCdLcbdsWyrpTSXoIXDVKhTHrlBe2s5sLxXMYbTIBuShFKMuQh6v_4gB7_12RVPBXU1gvpZsxNoaPztlHE2DhESnUg1Obpx0Vrg-p/s600/grafana-rt-country.png" /></a></div>
<a href="https://blog.sflow.com/2023/07/deploy-real-time-network-dashboards.html">Deploy real-time network dashboards using Docker compose</a> describes how to configure <a href="https://prometheus.io/">Prometheus</a> and <a href="https://grafana.com/">Grafana</a> to capture time series data and create custom dashboards.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWBAdK17A3KbXAT6EycVSyvn9_zvdhpbnydhypjS_Zb6IiMz6L9CJJdI5FQdl97z3sEwueb1IV_okJXitdZwLSyEQf4ZkCoy1RfHD2N5kMyMtmdYZFy-IjfBO03iTchbOjlVBQW4AQKKKGCqKM4z_yPy7vlKwEkc7sRjpLF-U5uB-NGPThMu1QMFfZoAzg/s2214/vyos-linux-reasons.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1308" data-original-width="2214" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWBAdK17A3KbXAT6EycVSyvn9_zvdhpbnydhypjS_Zb6IiMz6L9CJJdI5FQdl97z3sEwueb1IV_okJXitdZwLSyEQf4ZkCoy1RfHD2N5kMyMtmdYZFy-IjfBO03iTchbOjlVBQW4AQKKKGCqKM4z_yPy7vlKwEkc7sRjpLF-U5uB-NGPThMu1QMFfZoAzg/s600/vyos-linux-reasons.png" width="600" /></a></div><a href="https://blog.sflow.com/2023/04/dropped-packet-reason-codes-in-vyos.html">Dropped packet reason codes in VyOS</a> describes how the new Linux kernel in VyOS 1.4 provides detailed visibility into every dropped packet (including the reason it was dropped). This cabability is used by the new sFlow agent implement the <a href="https://sflow.org/sflow_drops.txt">sFlow Dropped Packet Notification Structures</a> extension to provide network-wide visibility into dropped packets.
<p>Download VyOS today to try out the new features. <a href="https://vyos.io/subscriptions/software">Pre-built LTS images</a> are available with paid support, but anyone can <a href="https://github.com/vyos/">build an image from sources</a> or <a href="https://vyos.net/get/nightly-builds/">download the latest rolling release</a>.</p>Peterhttp://www.blogger.com/profile/00856599914190257147noreply@blogger.com0tag:blogger.com,1999:blog-1978652979840829013.post-64345829459834962222024-01-15T08:41:00.000-08:002024-01-15T08:41:23.303-08:00Raspberry Pi 5 network emulation with Containerlab<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhW6MdPvWOM-5BWC1ZtD38q0ycMLp3pL4HFH7CTj6jOetqbABF_un1VkSsfxcW1lbiMr5Kk5RTqiziyxMkiV_SjBAX3iXiKNnbzxj7eP88MJiTJ-tGZRzyvZd5G9t4lhH4UN25_Qqdg5mZLvizoCMvDkrFobtWTcA6zHykV7lYrebI8lcp_sbtNsxuFTU2c/s640/clos5.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="270" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhW6MdPvWOM-5BWC1ZtD38q0ycMLp3pL4HFH7CTj6jOetqbABF_un1VkSsfxcW1lbiMr5Kk5RTqiziyxMkiV_SjBAX3iXiKNnbzxj7eP88MJiTJ-tGZRzyvZd5G9t4lhH4UN25_Qqdg5mZLvizoCMvDkrFobtWTcA6zHykV7lYrebI8lcp_sbtNsxuFTU2c/s600/clos5.png" width="600" /></a></div>
The GitHub <a href="https://github.com/sflow-rt/containerlab">sflow-rt/containerlab</a> project contains example network topologies for the <a href="https://containerlab.dev/">Containerlab</a> network emulation tool that demonstrate real-time streaming telemetry in realistic data center topologies and network configurations. The examples use the same <a href="https://frrouting.org/">FRRouting (FRR) engine</a> that is part of <a href="https://sonicfoundation.dev/">SONiC</a>, <a href="https://www.nvidia.com/en-us/networking/ethernet-switching/cumulus-linux/">NVIDIA Cumulus Linux</a>, and <a href="https://dent.dev/">DENT</a> network operating systems. Containerlab can be used to experiment before deploying solutions into production. Examples include: tracing ECMP flows in leaf and spine topologies, EVPN visibility, and automated DDoS mitigation using BGP Flowspec and RTBH controls.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjkbkmAcbTjm2vfUzKpKZUWVYAsb_AHZzoyba8lJhoQCndOUCHKQ2zbzU3dys0aoepdF-PgcUcpDOZn8bsFFLad8F6aeCzbSOCeVr93NNCD2HZU4Fqa9HlTmauEu9eTrxfb2N1FEcfU1DntOwTLUZ7IMMuQFLPGrW7GHfxr1LWnC4HteKn8sVXkAo4O0j7Z/s4032/IMG_4133.jpg" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="3024" data-original-width="4032" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjkbkmAcbTjm2vfUzKpKZUWVYAsb_AHZzoyba8lJhoQCndOUCHKQ2zbzU3dys0aoepdF-PgcUcpDOZn8bsFFLad8F6aeCzbSOCeVr93NNCD2HZU4Fqa9HlTmauEu9eTrxfb2N1FEcfU1DntOwTLUZ7IMMuQFLPGrW7GHfxr1LWnC4HteKn8sVXkAo4O0j7Z/s600/IMG_4133.jpg" width="600" /></a></div><a href="https://blog.sflow.com/2024/01/raspberry-pi-5-real-time-network.html">Raspberry Pi 5 real-time network analytics</a> describes how to install Docker on a Raspberry Pi 5.
<pre>docker run hello-world</pre>
Run the <i>hello-world</i> container to verify that Docker in properly installed and running before proceeding.
<pre>git clone https://github.com/sflow-rt/containerlab.git</pre>
Download the <a href="https://github.com/sflow-rt/containerlab">sflow-rt/containerlab</a> project from GitHub.
<pre>cd containerlab
./run-clab</pre>
Start Containerlab.
<pre>containerlab deploy -t clos5.yml</pre>
Start the 5 stage leaf and spine topology shown at the top of this page. The initial launch may take a couple of minutes as the container images are downloaded for the first time. Once the images are downloaded, the topology deploys in around 10 seconds.
<pre>./topo.py clab-clos5</pre>
Push the topology to the sFlow-RT analytics software.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwRWoxzQGQ3FHFnkZ3s2KSTtd4RlueQmWd3CLfxaHhyr5cJc-gr2zJtTZYLZSWjD1l9xCI3kVwgsmWBkNukVRF2inwiuR_RHA8_kO6UM-xWe5Kf-Jlh7RGTYxzXJA0_ANYaoeaLeSYuTSbYNkR62sDUXxeq682fSDJB6YUPkS9PcCgo6UY5Jf-GaDG2DzP/s1472/rt-ecosystem.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="740" data-original-width="1472" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwRWoxzQGQ3FHFnkZ3s2KSTtd4RlueQmWd3CLfxaHhyr5cJc-gr2zJtTZYLZSWjD1l9xCI3kVwgsmWBkNukVRF2inwiuR_RHA8_kO6UM-xWe5Kf-Jlh7RGTYxzXJA0_ANYaoeaLeSYuTSbYNkR62sDUXxeq682fSDJB6YUPkS9PcCgo6UY5Jf-GaDG2DzP/s600/rt-ecosystem.png" width="600" /></a></div>An instance of the <a href="https://sflow-rt.com/">sFlow-RT</a> real-time analytics engine receives industry standard <a href="https://sflow.org/">sFlow</a> telemetry from all the switches in the network. All of the switches in the topology are configured to send sFlow to the sFlow-RT instance. In this case, Containerlab is running the pre-built <a href="https://hub.docker.com/r/sflow/clab-sflow-rt">sflow/clab-sflow-rt</a> image which packages sFlow-RT with useful applications for exploring the data.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjg3BMhHOY5OoeYB_rW-z-6DnS9ofMH6YZ_pkNwFumvvAmH6icu6JIzPytFX-toM79h8YWfpq53BY1ktQaGm_doduSIegVbfSyhTTrrIfStKPSQhaeL2L9zqGetw3FNUif0oI1xjqwq8lTlGkTR89ZIdI9VycOudKKPCOJrApeeilXa80VrM5QZE2t5DUs6/s1776/clab-rt-dash-pi5.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1588" data-original-width="1776" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjg3BMhHOY5OoeYB_rW-z-6DnS9ofMH6YZ_pkNwFumvvAmH6icu6JIzPytFX-toM79h8YWfpq53BY1ktQaGm_doduSIegVbfSyhTTrrIfStKPSQhaeL2L9zqGetw3FNUif0oI1xjqwq8lTlGkTR89ZIdI9VycOudKKPCOJrApeeilXa80VrM5QZE2t5DUs6/s600/clab-rt-dash-pi5.png" width="600" /></a></div>
Connect to the web interface on port 8008. The sFlow-RT dashboard verifies that telemetry is being received from 10 agents (the 10 switches in the Clos fabric). See the <a href="https://sflow-rt.com/intro.php">sFlow-RT Quickstart</a> guide for more information.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVijiqjZfjXm5L0kUVMAAfgAABQZp5Z5CMFQgebtb-px3I_AhfwT0wuc08OK3ofdywUf1-zowbPUD-7mNdxSOgGrCOaxxYPXt5lyxV_GbFSsP4GEkHu2PKOTXTFIAjLsnSp6QGQN2mjIpbpiG4Vh0ky4JLEdOK5_bCFg0IBggrSfd-bJ6PB73h6VOpQR-H/s2264/clab-dash-pi5.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="2264" data-original-width="1776" height="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVijiqjZfjXm5L0kUVMAAfgAABQZp5Z5CMFQgebtb-px3I_AhfwT0wuc08OK3ofdywUf1-zowbPUD-7mNdxSOgGrCOaxxYPXt5lyxV_GbFSsP4GEkHu2PKOTXTFIAjLsnSp6QGQN2mjIpbpiG4Vh0ky4JLEdOK5_bCFg0IBggrSfd-bJ6PB73h6VOpQR-H/s600/clab-dash-pi5.png" /></a></div>
The <i>Containerlab Dashboard</i> (click on sFlow-RT <i>Apps</i> tab and <i>containerlab-dashboard</i> button) shows real-time dashboard displaying up to the second traffic.
<pre>docker exec -it clab-clos5-h1 iperf3 -c 172.16.4.2</pre>
Each of the hosts in the network has an <i>iperf3</i> server, so running the above command will test bandwidth between h1 and h4.
<pre>docker exec -it clab-clos5-h1 iperf3 -c 2001:172:16:4::2</pre>
Generate a large IPv6 flow between <i>h1</i> and <i>h4</i>. The traffic flows should immediately appear in the <i>Top Flows</i> chart. You can check the accuracy by comparing the values reported by <i>iperf3</i> with those shown in the chart.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxWdIU5g3gC_nfoxEb_q_5WzyAZWMG0ulkE0IHr-21lmcaEa7G1jhZR8Bf7BV-jWU9r8sldTWtD_-L_yshAXKLOd6QaXZDAWLukGq9uoYV7G1QFwO_EeZGthzdIIN5CvA_HvOkqLhDDogrxallzGZH58mlFJISvGE1dUzy-HvIAw9ehICFlHMslWOenMzh/s1776/clab-topo-pi5.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1766" data-original-width="1776" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxWdIU5g3gC_nfoxEb_q_5WzyAZWMG0ulkE0IHr-21lmcaEa7G1jhZR8Bf7BV-jWU9r8sldTWtD_-L_yshAXKLOd6QaXZDAWLukGq9uoYV7G1QFwO_EeZGthzdIIN5CvA_HvOkqLhDDogrxallzGZH58mlFJISvGE1dUzy-HvIAw9ehICFlHMslWOenMzh/s600/clab-topo-pi5.png" width="600" /></a></div>
Click on the <i>Topology</i> tab to see a real-time weathermap of traffic flowing over the topology. See how repeated iperf3 tests take different ECMP (equal-cost multi-path) routes across the network.
<pre>docker exec -it clab-clos5-leaf1 vtysh</pre>
Linux with open source routing software (FRRouting) is an accessible alternative to vendor routing stacks (no registration / license required, no restriction on copying means you can share images on Docker Hub, no need for virtual machines). FRRouting is popular in production network operating systems (e.g. Cumulus Linux, SONiC, DENT, etc.) and the <a href="https://docs.frrouting.org/en/latest/vtysh.html">VTY shell</a> provides an industry standard CLI for configuration, so labs built around FRR allow realistic network configurations to be explored.
<pre>docker exec -it clab-clos5-leaf1 vtysh -c "show running-config"</pre>
Use <i>vtysh</i> to show the running configuration on <i>leaf1</i>.
<pre>containerlab destroy -t clos5.yml</pre>
When you are finished, run the above command to stop the containers and free the resources associated with the emulation. Try out <a href="https://github.com/sflow-rt/containerlab#readme">other topologies</a> from the project to explore topics such as DDoS mitigation, BGP Flowspec, and EVPN.
<p><b>Note:</b> If you are building your own topologies, the Raspberry Pi 5 8G can comfortably handle topologies with up to 50 FRR/Alpine Linux nodes.</p>
<p><a href="https://sflow-rt.com/intro.php">Getting Started</a> provides an introduction to sFlow-RT analytics and APIs. Containerlab provides a useful environment for developing and testing monitoring applications for sFlow-RT before moving them into production.</p>
<p>Moving monitoring solutions from Containerlab to production is straightforward since sFlow is widely implemented in datacenter equipment from vendors including: A10, Arista, Aruba, Cisco, Edge-Core, Extreme, Huawei, Juniper, NEC, Netgear, Nokia, NVIDIA, Quanta, and ZTE. In addition, the open source <a href="https://sflow.net/">Host sFlow</a> agent makes it easy to extend visibility beyond the physical network into the compute infrastructure.</p>
<p><a href="https://blog.sflow.com/2024/01/raspberry-pi-5-real-time-network.html">Raspberry Pi 5 real-time network analytics</a> describes how to deploy an sFlow-RT, Prometheus, and Grafana monitoring stack to monitor live network traffic.</p>Peterhttp://www.blogger.com/profile/00856599914190257147noreply@blogger.com0tag:blogger.com,1999:blog-1978652979840829013.post-45445242998518687142024-01-09T07:31:00.000-08:002024-01-22T12:10:27.832-08:00Raspberry Pi 5 real-time network analytics<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9uzRCI0jX3snNhoTK_KgDS17Ava3gFDuAMRPgxbsXJ7fp450MzXvWoRkpiW8VmqXTmRZswNYH9NJngZ9sHhTCBx8quq9J30FCxY-zaqRusDb9lLAtamy-7f_eQSafJAW4uIbjCLz0hb1SUUZhrlkN1_JEnwXYCIp8HT3fKuzLCzIdhLqrkI2xkE1HNsFK/s4032/IMG_4133.jpg" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="3024" data-original-width="4032" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9uzRCI0jX3snNhoTK_KgDS17Ava3gFDuAMRPgxbsXJ7fp450MzXvWoRkpiW8VmqXTmRZswNYH9NJngZ9sHhTCBx8quq9J30FCxY-zaqRusDb9lLAtamy-7f_eQSafJAW4uIbjCLz0hb1SUUZhrlkN1_JEnwXYCIp8HT3fKuzLCzIdhLqrkI2xkE1HNsFK/w640-h480/IMG_4133.jpg" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://www.canakit.com/canakit-raspberry-pi-5-starter-kit-aluminum.html">CanaKit Raspberry Pi 5 Starter Kit - Aluminum</a></td></tr></tbody></table>
This article describes how build an inexpensive Raspberry Pi 5 based server for real-time flow analytics using industry standard <a href="https://sflow.org">sFlow</a> streaming telemetry. Support for sFlow is widely implemented in datacenter equipment from vendors including: A10, Arista, Aruba, Cisco, Edge-Core, Extreme, Huawei, Juniper, NEC, Netgear, Nokia, NVIDIA, Quanta, and ZTE.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhVyRaVV32S9J3yAqZA2BbFm3k2IGBqh80yUR6p3BGudig_HBWzWTvTUEieq-1wZeFtlcaT_mv3anNC8xexGkl7GEPr8kZDrCViRJzYWNzdwOy1s9BThsOf2955By8b_Hb6zD0_vBR66HlsO0RBKlZ4XggCGTdLVAJowl66tlo0tWc54rjpTDiK7QkQcYb6/s1584/pi5-imager.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1180" data-original-width="1584" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhVyRaVV32S9J3yAqZA2BbFm3k2IGBqh80yUR6p3BGudig_HBWzWTvTUEieq-1wZeFtlcaT_mv3anNC8xexGkl7GEPr8kZDrCViRJzYWNzdwOy1s9BThsOf2955By8b_Hb6zD0_vBR66HlsO0RBKlZ4XggCGTdLVAJowl66tlo0tWc54rjpTDiK7QkQcYb6/s600/pi5-imager.png" width="600" /></a></div>
In this example, we will use an 8G Raspberry Pi 5 running Raspberry Pi OS Lite (64-bit). The easiest way to format a memory card and install the operating system is to use the <a href="https://www.raspberrypi.com/software/">Raspberry Pi Imager</a> (shown above).
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRe1wI0Hxarxa8e1RZ9syjc6gYtATlk9diYN4gF1v48U8WO5cJujjUCVMKMsHavdMyCIIqyWd7RjCYmGnEHXlvyzz2Ylt9WBIIKkQry8q-V2xusLfp_dyua47RZBGEHgyBgmpgUFfGQUmyC5JOtNJzovMofRirs_yUUrlxmHAxLnvqOQyFKiLi9iKB75hN/s1584/pi5-imager-options.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1180" data-original-width="1584" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRe1wI0Hxarxa8e1RZ9syjc6gYtATlk9diYN4gF1v48U8WO5cJujjUCVMKMsHavdMyCIIqyWd7RjCYmGnEHXlvyzz2Ylt9WBIIKkQry8q-V2xusLfp_dyua47RZBGEHgyBgmpgUFfGQUmyC5JOtNJzovMofRirs_yUUrlxmHAxLnvqOQyFKiLi9iKB75hN/s600/pi5-imager-options.png" width="600" /></a></div>
Click on <i>EDIT SETTINGS</i> button to customize the installation.<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5wFzUqMmJ4Zjqr-qgwKs0CFzhnrQO8RLOgwVwdvf-bq020e-GxqfERYcPaCgfvGkKfeb0xkh2U_Mqh94B2o3u798SNO3m6OX7mAnquGiw-IhXCPdKMtbSIW7Hs6ejAkLi7w9R0BGRW2qE38a8XihTAy5t5yZH3E3agdYFvnnIq5aUUmjnaHP6Jiv73X2g/s1532/pi5-imager-custom.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1532" data-original-width="1302" height="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5wFzUqMmJ4Zjqr-qgwKs0CFzhnrQO8RLOgwVwdvf-bq020e-GxqfERYcPaCgfvGkKfeb0xkh2U_Mqh94B2o3u798SNO3m6OX7mAnquGiw-IhXCPdKMtbSIW7Hs6ejAkLi7w9R0BGRW2qE38a8XihTAy5t5yZH3E3agdYFvnnIq5aUUmjnaHP6Jiv73X2g/s600/pi5-imager-custom.png" /></a></div>Set a <i>hostname</i>, <i>username</i>, and <i>password</i>.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3Lu51Uf61Npet5tiYjTtSUHVZvbI-h7HHobMdAQIKVWY1XbAid3TFVEhKcr5ejDlUAodSU8ChOsuzDscPMnBZ-5uItmzqc5go8slELEMYwAi9QWKUgGKbWURJm_JoraHsVMl4Q1w1C5KpHyZrH90MFrBogYf1qpxQSRnsxgzrcyOlw3Kck-kyEV3wJL6_/s1532/pi5-imager-services.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1532" data-original-width="1302" height="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3Lu51Uf61Npet5tiYjTtSUHVZvbI-h7HHobMdAQIKVWY1XbAid3TFVEhKcr5ejDlUAodSU8ChOsuzDscPMnBZ-5uItmzqc5go8slELEMYwAi9QWKUgGKbWURJm_JoraHsVMl4Q1w1C5KpHyZrH90MFrBogYf1qpxQSRnsxgzrcyOlw3Kck-kyEV3wJL6_/s600/pi5-imager-services.png" /></a></div>Click on the <i>SERVICES</i> tab and select <i>Enable SSH</i>. Click <i>SAVE</i> to save the settings and then <i>YES</i> to apply the settings and create a bootable micro SD card. These initial settings allow the Rasberry Pi to be accessed over the network without having to attach a screen, keyboard, and mouse.<pre>ssh pp@192.168.4.170</pre>
Use ssh to log into Raspberry Pi (having installled the micro SD card).<pre>sudo apt-get update && sudo apt-get -y upgrade</pre>
Update packages and OS to latest version.<pre>curl -sSL https://get.docker.com | sh</pre>
Install Docker.
<pre>sudo usermod -aG docker $USER</pre>
Give permission to run Docker without sudo command. Exit ssh session and log in again to pick up the new settings.
<pre>docker run hello-world</pre>
Run the <i>hello-world</i> container to verify that docker in properly installed and running.
<pre>git clone https://github.com/sflow-rt/prometheus-grafana.git
cd prometheus-grafana
./start.sh</pre>
Start <a href="https://sflow-rt.com">sFlow-RT</a>, <a href="https://prometheus.io/">Prometheus</a>, and <a href="https://grafana.com/">Grafana</a> using Docker compose.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgpM92xbfSQt8X1Hm5vkHv-E141ki-Jwymp1mUsun1OouxAxxtcIZRjqCN0T6NfJiYivoxKRQd-7lL8dl-zzK6Dojs4ny53IhjQHnGcS5q_wdC9XlQ5S-a0xgTF2lF-WIVQnZtVfQWrMvPqo7nIG_6qW-V4Wo69ZhRwPPepAhXfT31I51Mr-C2kyq5t74lR/s1472/rt-ecosystem.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="740" data-original-width="1472" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgpM92xbfSQt8X1Hm5vkHv-E141ki-Jwymp1mUsun1OouxAxxtcIZRjqCN0T6NfJiYivoxKRQd-7lL8dl-zzK6Dojs4ny53IhjQHnGcS5q_wdC9XlQ5S-a0xgTF2lF-WIVQnZtVfQWrMvPqo7nIG_6qW-V4Wo69ZhRwPPepAhXfT31I51Mr-C2kyq5t74lR/s600/rt-ecosystem.png" width="600" /></a></div>
Configure <a href="https://sflow-rt.com/agents.php">sFlow Agents</a> embedded in switches, routers and servers to stream sFlow telemetry to the Raspberry Pi. The sFlow-RT <a href="https://sflow-rt.com/intro.php">Getting Started</a> guide shows how to verify that sFlow is being received and includes tools flow and counter based analytics.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEE3atjZZ79TbAICqw4YVsMyTSWGKeECM0D1GtRkV5d6UY9GrJSraPiWc7IRHLmaO2ch2GNzGScVnFSx7kawKs44GH-zvQ1PlCrNAth0GODMPYlINcmUe9ex5krqUqmpRjPIeTpm3t6z2qi5n92E6gCKkPN0Uhjk4bOY9Uc_gv5QBk2VHtL-t5sRUPEyBC/s2002/getting-started-flows.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1796" data-original-width="2002" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEE3atjZZ79TbAICqw4YVsMyTSWGKeECM0D1GtRkV5d6UY9GrJSraPiWc7IRHLmaO2ch2GNzGScVnFSx7kawKs44GH-zvQ1PlCrNAth0GODMPYlINcmUe9ex5krqUqmpRjPIeTpm3t6z2qi5n92E6gCKkPN0Uhjk4bOY9Uc_gv5QBk2VHtL-t5sRUPEyBC/s600/getting-started-flows.png" width="600" /></a></div>
For example, the <i>Flow Browser</i> application lets you list attributes of network traffic that you are interested in and trend top flows with the attributes in real-time (up to the second). <a href="https://sflow-rt.com/define_flow.php">Defining Flows</a> describes the flow analytics capability of sFlow-RT that can be explored.<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjX3Lmk3Dx1S-fXDgnBjnJGYhULJLaOF363r71QwXmR3s-PbEgMbGvO2vF08qz9meY6nk85X2aPoSaN2zaAMv-YZUPyqpT1QSTCT_tkw2HKU4d5cNbUlkaQiqmalFTZMbL_oZsz6L0nJz8yQKP8wsdZ9x33G0S_3nKvvdbH7fKXN2MyTni3se9eLVCAncPt/s2152/grafana-rt-country.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="2152" data-original-width="1906" height="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjX3Lmk3Dx1S-fXDgnBjnJGYhULJLaOF363r71QwXmR3s-PbEgMbGvO2vF08qz9meY6nk85X2aPoSaN2zaAMv-YZUPyqpT1QSTCT_tkw2HKU4d5cNbUlkaQiqmalFTZMbL_oZsz6L0nJz8yQKP8wsdZ9x33G0S_3nKvvdbH7fKXN2MyTni3se9eLVCAncPt/s600/grafana-rt-country.png" /></a></div>
<a href="https://blog.sflow.com/2023/07/deploy-real-time-network-dashboards.html">Deploy real-time network dashboards using Docker compose</a> describes how to configure Prometheus and Grafana to capture time series data and create custom dashboards.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPU7TwIC5rrun-cd_xts9KsH3SZsf9EUNV6S4cJpmLI0q1jHLsUHxsoMxU6sBQs0nIdzW5pDaC6MDE-kxAp-g6ijhTKKrJshDWPEmPDf_P82l_adZSHrt_1btJZJ22UaaW_i_pWqfm24Iow2FhduuNyXKdeE4CjIsedTcXJslmM0ttvoIGZSfnLUTx_MBo/s1768/pi5-dash.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1582" data-original-width="1768" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPU7TwIC5rrun-cd_xts9KsH3SZsf9EUNV6S4cJpmLI0q1jHLsUHxsoMxU6sBQs0nIdzW5pDaC6MDE-kxAp-g6ijhTKKrJshDWPEmPDf_P82l_adZSHrt_1btJZJ22UaaW_i_pWqfm24Iow2FhduuNyXKdeE4CjIsedTcXJslmM0ttvoIGZSfnLUTx_MBo/s600/pi5-dash.png" width="600" /></a></div>
The Raspberry Pi 5 is surprisingly capable, this pocket-sized server can easily monitor thousands of high speed (100G+) links, providing up to the second visibility into network flows. In this example, sFlow telemetry from 100 switches, each with 48 active 100G ports, was easily handled by the Raspberry Pi 5. Performance of the Prometheus database is likely to be the limiting factor given the relatively slow disk performance of the micro SD card, but could be improved adding an M.2 PCIe disk.Peterhttp://www.blogger.com/profile/00856599914190257147noreply@blogger.com3tag:blogger.com,1999:blog-1978652979840829013.post-26798871361824577932023-11-17T07:44:00.000-08:002023-11-17T07:44:46.540-08:00SC23 Over 6 Terabits per Second of WAN Traffic<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpEdA6Dh7_TbJoY7QDhVPWFr08aN87W_iBg4k3DcjhSXGlpeO7Y0-H0JyxeIpDYWZjRls974WWMbKlpFVIgNNrxwELAbx88fSot0xedbcXL_weA74ZDUpnXMUshw5eKLJ-LoOBipkEIOJa6K3U-Gb5mmayHeFkH0fkEfH_9qynPvF4ad5dVXld9pHLcCx_/s2458/sc23-wan-stress.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="2184" data-original-width="2458" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpEdA6Dh7_TbJoY7QDhVPWFr08aN87W_iBg4k3DcjhSXGlpeO7Y0-H0JyxeIpDYWZjRls974WWMbKlpFVIgNNrxwELAbx88fSot0xedbcXL_weA74ZDUpnXMUshw5eKLJ-LoOBipkEIOJa6K3U-Gb5mmayHeFkH0fkEfH_9qynPvF4ad5dVXld9pHLcCx_/s600/sc23-wan-stress.png" width="600" /></a></div>
<a href="https://coloradosun.com/2023/11/13/fastest-internet-service-terabits-denver-sc23/">The world’s fastest temporary internet service gets turned on in Denver for one week only</a> describes the SCinet temporary network built to support the <a href="https://sc23.supercomputing.org/">The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC23)</a> this week in Denver. The <i>SC23 WAN Stress Test</i> chart demonstrates that the provisioned 6.71 terabits bits per second capacity was pushed to the limits.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghTd4DGWRrXoZ0Gh3wSEJXfJdtfY2wMw5zs91p61zwzRNmJT2vGHqyx8vj-gjrGsUg7gmDzY8QUECZJ70xJvE-N8CirbWkT5Sb9IEkov1cWj5RTDYQb_wzd9l956EB8AdQ0WtdCYsH6Cfmue0WVUcRO8D_JVJYUpW6IOIdrNnEe9HclEJm5DRzyb_FFhVc/s2232/sc23-total-bytes.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1696" data-original-width="2232" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghTd4DGWRrXoZ0Gh3wSEJXfJdtfY2wMw5zs91p61zwzRNmJT2vGHqyx8vj-gjrGsUg7gmDzY8QUECZJ70xJvE-N8CirbWkT5Sb9IEkov1cWj5RTDYQb_wzd9l956EB8AdQ0WtdCYsH6Cfmue0WVUcRO8D_JVJYUpW6IOIdrNnEe9HclEJm5DRzyb_FFhVc/s600/sc23-total-bytes.png" width="600" /></a></div>
<a href="https://blog.sflow.com/2023/11/sc23-scinet-traffic.html">SC23 SCinet traffic</a> describes the architecture of the real-time monitoring system used to comprehensively monitor the SCinet network and generate these charts. This chart shows that over 175 Petabytes of data were transfered during the show.<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqJyfh8lr8TqwsDnwfsm_Ybif-fIMZazbqtgYDqdAQm6jBVZEEjAFBHahSPtwMuhhVUkEcFz62_UMQraX7VyuWDSb1qtdOIRU96GQKLrOfNPK0L9tbJG_7Y_CtlDSoON9luRejfZtHKzAu3BMZG4Us3uIq2jYr2_K9QPd5HvG9N59n-lWA5KTII6ffMMZt/s2790/sc23-drop-demo.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="2270" data-original-width="2790" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqJyfh8lr8TqwsDnwfsm_Ybif-fIMZazbqtgYDqdAQm6jBVZEEjAFBHahSPtwMuhhVUkEcFz62_UMQraX7VyuWDSb1qtdOIRU96GQKLrOfNPK0L9tbJG_7Y_CtlDSoON9luRejfZtHKzAu3BMZG4Us3uIq2jYr2_K9QPd5HvG9N59n-lWA5KTII6ffMMZt/s600/sc23-drop-demo.png" width="600" /></a></div><a href="https://blog.sflow.com/2023/11/sc23-dropped-packet-visibility.html">SC23 Dropped packet visibility demonstration</a> describes a joint demonstration by InMon Corp and Arista Networks of one of newest developments in sFlow telemetry, identifying every dropped packet, the reason it was dropped, and the location it was dropped across all the switches in real-time.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjV9EZBiJjsQ43ZGvjkd4R3Tv2AeuJndYa2LHI-e2Rz7puEz4shDJ-sQCvUdDQ1mzJUhh0Qde7_enTJAZ97PaEN7ZXVAZPaP_PevgOMYcHbHwLqKyd-mLNLf9liRNHKeXBGh0zzpvTMS4HFDkPSSfYD5n1QQMSnLFME3C_wgOIRPiVFmtqcMiIuWKEMvY14/s1822/sc23-heatmap.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1294" data-original-width="1822" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjV9EZBiJjsQ43ZGvjkd4R3Tv2AeuJndYa2LHI-e2Rz7puEz4shDJ-sQCvUdDQ1mzJUhh0Qde7_enTJAZ97PaEN7ZXVAZPaP_PevgOMYcHbHwLqKyd-mLNLf9liRNHKeXBGh0zzpvTMS4HFDkPSSfYD5n1QQMSnLFME3C_wgOIRPiVFmtqcMiIuWKEMvY14/s600/sc23-heatmap.png" width="600" /></a></div><a href="https://blog.sflow.com/2023/11/sc23-wifi-traffic-heatmap.html">SC23 WiFi Traffic Heatmap</a> shows a real-time view of WiFi usage at the conference displayed on a conference floorplan.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2AFBHWb8TiytQJv7vIshCnRIVWPEire48BhFRObShKe2ZMUJS0Xl1K6GHjizYnsoj8nWJL_wFK75sCiHQuIWDsS201XQRdWGDAzMamgoCMAK5K9SJWid4ZbDQb24p7ixkX4VjmJ5kPSArgsdl_wkGfcdN-S2h_Omw613YjHy9hSFi1roLmBkzryM5qJXJ/s3496/sc23-dtn4.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="2962" data-original-width="3496" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2AFBHWb8TiytQJv7vIshCnRIVWPEire48BhFRObShKe2ZMUJS0Xl1K6GHjizYnsoj8nWJL_wFK75sCiHQuIWDsS201XQRdWGDAzMamgoCMAK5K9SJWid4ZbDQb24p7ixkX4VjmJ5kPSArgsdl_wkGfcdN-S2h_Omw613YjHy9hSFi1roLmBkzryM5qJXJ/s600/sc23-dtn4.png" width="600" /></a></div>
Finally, <a href="https://blog.sflow.com/2023/11/sc23-data-transfer-node-tcp-metrics.html">SC23 Data Transfer Node TCP Metrics</a> demonstrates how standard metrics maintained by the Linux kernel can be used to augment sFlow telemetry and track the performance of large science data transfers.Peterhttp://www.blogger.com/profile/00856599914190257147noreply@blogger.com0tag:blogger.com,1999:blog-1978652979840829013.post-9393081307262426922023-11-16T14:34:00.000-08:002023-11-16T14:34:03.994-08:00SC23 Data Transfer Node TCP Metrics<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYkf_jht6S5l1FI-e0RHC2uXwrBakDIF_sJq2umUvrMQJbaeepS-G_uVTL_zW1FXMPivU4UWYAMUDYFSmCArC_FIdLxp7Tya9FmNk7Iiaj6te0YKUm0aLHPvmGTX1SIRymogmA1ARU9JwM802qYXbOi16aZpBwOnL9m_DSqyqyZqgJ0QbTGzD-dSRp4OXn/s3496/sc23-dtn4.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="2962" data-original-width="3496" height="542" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYkf_jht6S5l1FI-e0RHC2uXwrBakDIF_sJq2umUvrMQJbaeepS-G_uVTL_zW1FXMPivU4UWYAMUDYFSmCArC_FIdLxp7Tya9FmNk7Iiaj6te0YKUm0aLHPvmGTX1SIRymogmA1ARU9JwM802qYXbOi16aZpBwOnL9m_DSqyqyZqgJ0QbTGzD-dSRp4OXn/w640-h542/sc23-dtn4.png" width="640" /></a></div>
The dashboard shown above is based on the open source <a href="https://github.com/sflow-rt/dtn">sflow-rt/dtn</a> project. The dashboard shows data captured from <a href="https://sc23.supercomputing.org/">The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC23)</a> being held this week in Denver.
<p>The dashboard displays data gathered from open source <a href="https://sflow.net">Host sFlow</a> agents installed on Data Transfer Nodes (DTNs) run by the <a href="https://www.caltech.edu/">Caltech</a> High Energy Physics Department and used for handling transfer of large scientific data sets (for example, accessing experiment data from the <a href="https://home.cern/">CERN</a> particle accelerator). <a href="https://blog.sflow.com/2016/10/network-performance-monitoring.html">Network performance monitoring</a> describes how the Host sFlow agents augment standard <a href="https://sflow.org">sFlow</a> telemetry with measurements that the Linux kernel maintains as part of the normal operation of the TCP protocol stack.</p><p>The dashboard shows 5 large flows (greater than 50 Gigabits per Second). For each large flow being tracked, additional TCP performance metrics are displayed:</p><p></p><ul style="text-align: left;"><li><b>RTT</b> The round trip time observed between DTNs</li><li><b>RTT Wait</b> The amount of time that data waits on sender before it can be sent.</li><li><b>RTT Sdev</b> The standard deviation on observed RTT. This variation is a measure of jitter.</li><li><b>Avg. Packet Size</b> The average packet size used to send data.</li><li><b>Packets in Flight</b> The number of unacknowledged packets.</li></ul><p></p>
<p>See <a href="https://sflow-rt.com/define_flow.php#tcpinfo">Defining Flows</a> for full range of attributes that can be used to create flow metrics.</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_FTJTpoT-obB2UJsfX13YWt4NGNifrFvYmpZm8NE59JdJHhjs0WNSY4DEHBpeD9yGKl849t_aW9XzIGCUUP3NiUCqqjOwEFPfQaR8Nzf4JT17h8ywAjP7kj3qxPPu71sNcfV3BHugw6E28WYt942CZH2A0kZ5wieTsldgA3lgC4JbjQFIYVdkcI-lFaIO/s1024/SC23%20v10.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="579" data-original-width="1024" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_FTJTpoT-obB2UJsfX13YWt4NGNifrFvYmpZm8NE59JdJHhjs0WNSY4DEHBpeD9yGKl849t_aW9XzIGCUUP3NiUCqqjOwEFPfQaR8Nzf4JT17h8ywAjP7kj3qxPPu71sNcfV3BHugw6E28WYt942CZH2A0kZ5wieTsldgA3lgC4JbjQFIYVdkcI-lFaIO/s600/SC23%20v10.png" width="600" /></a></div>
The conference network used in the demonstration, <a href="https://sc23.supercomputing.org/scinet/">SCinet</a>, is described as <i>the most powerful and advanced network on Earth, connecting the SC community to the world</i>.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgR6J29rZEjqSPCxCovTqwLwByS3gOojhFYoDaNXxdFuwRnWUXlOBvE-qHiLWt_JZhUJuSXqQWvl3jBbo95rel99uj12EXAQYId44ZkLVCe4sc1IOPkq_HTUHqqcHPm0K13XQwxWZ_LYJ26bDPuM7Um2RdbNoDfd047VxcDznPtXsCPx6iHmSHNafS-nm-J/s1472/rt-ecosystem.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="740" data-original-width="1472" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgR6J29rZEjqSPCxCovTqwLwByS3gOojhFYoDaNXxdFuwRnWUXlOBvE-qHiLWt_JZhUJuSXqQWvl3jBbo95rel99uj12EXAQYId44ZkLVCe4sc1IOPkq_HTUHqqcHPm0K13XQwxWZ_LYJ26bDPuM7Um2RdbNoDfd047VxcDznPtXsCPx6iHmSHNafS-nm-J/s600/rt-ecosystem.png" width="600" /></a></div>
In this example, the <a href="https://sflow-rt.com/">sFlow-RT</a> real-time analytics engine receives <a href="https://sflow-rt.com/">sFlow</a> telemetry from switches, routers, and servers in the SCinet network and creates metrics to drive the real-time charts in the dashboard. <a href="https://sflow-rt.com/intro.php">Getting Started</a> provides a quick introduction to deploying and using sFlow-RT for real-time network-wide flow analytics.
<p>Finally, check out the <a href="https://blog.sflow.com/2023/11/sc23-dropped-packet-visibility.html">SC23 Dropped packet visibility demonstration</a>, <a href="https://blog.sflow.com/2023/11/sc23-scinet-traffic.html">SC23 SCinet traffic</a>, and <a href="https://blog.sflow.com/2023/11/sc23-wifi-traffic-heatmap.html">SC23 WiFi Traffic Heatmap</a> for additional network visibility demonstrations from the show.</p>
Peterhttp://www.blogger.com/profile/00856599914190257147noreply@blogger.com0tag:blogger.com,1999:blog-1978652979840829013.post-72531646700084609982023-11-15T13:31:00.000-08:002023-11-15T13:31:12.616-08:00SC23 WiFi Traffic Heatmap<div class="separator" style="clear: both; text-align: center;">
<iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/EgeUJmJShI8?si=2XwjK8rdyTxRwjpF&controls=0" title="YouTube video player" width="560"></iframe>
</div>
Real-time WiFi-Traffic Heatmap (source code GitHub: <a href="https://github.com/cod3monk/showfloor-heatmap">cod3monk/showfloor-heatmap</a>) displays real-time WiFi traffic from <a href="https://sc23.supercomputing.org/">The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC23)</a> being held this week in Denver.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjoBdMAjVxfTueIxxvNoFpcUjSoLUOt4re7HRk6T4lrzERt8DfpVDkUKhYYK3_t1fe9uGRyBPCC0h3olJzS1DG-0334v48GbJCXtQnVZmGVFaPpZWZRpFqchSW_nOSJyaQTE9ZS-czcrduyftvobdiN9FwktiqHfXZWQ9cCH_lQuDX-CfE0r-DxIKH-2hW-/s1024/SC23%20v10.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="579" data-original-width="1024" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjoBdMAjVxfTueIxxvNoFpcUjSoLUOt4re7HRk6T4lrzERt8DfpVDkUKhYYK3_t1fe9uGRyBPCC0h3olJzS1DG-0334v48GbJCXtQnVZmGVFaPpZWZRpFqchSW_nOSJyaQTE9ZS-czcrduyftvobdiN9FwktiqHfXZWQ9cCH_lQuDX-CfE0r-DxIKH-2hW-/s600/SC23%20v10.png" width="600" /></a></div>
The conference network used in the demonstration, <a href="https://sc23.supercomputing.org/scinet/">SCinet</a>, is described as <i>the most powerful and advanced network on Earth, connecting the SC community to the world.
</i><div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9-bzN0dPdCa5Rhjin7IS9ZsUd-_jjLMSZE0kSd14guT1y-aTWFbf-6FeJcwZPhxVSZRNH7fbOa2O3Xj8OnvnYX4tio_PQ-7xjiFvoaXMtZpAI8agcY-QL5Jb9w4uwqWdEDAi4xtEqph59msvcK4maPkzSL-3WwngdCkauS8w8rKp5GFukg2XDUOXSL8aY/s1472/rt-ecosystem.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="740" data-original-width="1472" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9-bzN0dPdCa5Rhjin7IS9ZsUd-_jjLMSZE0kSd14guT1y-aTWFbf-6FeJcwZPhxVSZRNH7fbOa2O3Xj8OnvnYX4tio_PQ-7xjiFvoaXMtZpAI8agcY-QL5Jb9w4uwqWdEDAi4xtEqph59msvcK4maPkzSL-3WwngdCkauS8w8rKp5GFukg2XDUOXSL8aY/s600/rt-ecosystem.png" width="600" /></a></div>
In this example, the <a href="https://sflow-rt.com">sFlow-RT</a> real-time analytics engine receives <a href="https://sflow.org">sFlow</a> telemetry from switches, routers, and servers in the SCinet network and creates metrics to drive the real-time heatmap. <a href="https://sflow-rt.com/intro.php">Getting Started</a> provides a quick introduction to deploying and using sFlow-RT for real-time network-wide flow analytics.
<p>Additional use cases being demonstrated this week include, <a href="https://blog.sflow.com/2023/11/sc23-dropped-packet-visibility.html">SC23 Dropped packet visibility demonstration</a> and <a href=" https://blog.sflow.com/2023/11/sc23-scinet-traffic.html">SC23 SCinet traffic</a>.</p>Peterhttp://www.blogger.com/profile/00856599914190257147noreply@blogger.com0tag:blogger.com,1999:blog-1978652979840829013.post-51471691612182416382023-11-13T06:56:00.000-08:002023-11-17T16:27:21.407-08:00SC23 SCinet traffic <div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh03oUOxLECpoEyzYGaCJNHGOnTVRI2Tp2M-tCUV58x8mu66ZOuHGA2j3qFaL574D5UMzi7mat0yXtEJsk9ndnwACzDcHgSCbdh6eGcoHkHDNqtwXpqV4ohhdLL1zaBf8AateSc4edxl0wCUlrXBdzEkLOT10UTsfTaxoX_fUsXSJojeIIdaWA-24hb5Cnc/s2210/sc23-traffic-rt.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1354" data-original-width="2210" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh03oUOxLECpoEyzYGaCJNHGOnTVRI2Tp2M-tCUV58x8mu66ZOuHGA2j3qFaL574D5UMzi7mat0yXtEJsk9ndnwACzDcHgSCbdh6eGcoHkHDNqtwXpqV4ohhdLL1zaBf8AateSc4edxl0wCUlrXBdzEkLOT10UTsfTaxoX_fUsXSJojeIIdaWA-24hb5Cnc/s600/sc23-traffic-rt.png" width="600" /></a></div>
The real-time dashboard shows total network traffic at <a href="https://sc23.supercomputing.org/">The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC23)</a> conference being held this week in Denver. The dashboard shows that 31 Petabytes of data have been transferred already and the conference hasn't even started.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhhEm_w3SL7ufG4aPdkSsieKiMEs1kPtMW8mi-yiq4dnMA83VVxxItsbmtw0QL7DizQFqQPGL5XMl32rMVTDZgP_-yFOc4uUtvkOhV5ijD2mY_W2oKG1QjJ07cftyNY0M7NEX6kSrYW21FMa-IcGogYYVM0ru0vEJR8tfoVYmLWZiyjO7iQ6yTdJLOpxna5/s1024/SC23%20v10.png" style="display: block; padding: 1em 0px; text-align: center;"><img border="0" data-original-height="579" data-original-width="1024" height="363" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhhEm_w3SL7ufG4aPdkSsieKiMEs1kPtMW8mi-yiq4dnMA83VVxxItsbmtw0QL7DizQFqQPGL5XMl32rMVTDZgP_-yFOc4uUtvkOhV5ijD2mY_W2oKG1QjJ07cftyNY0M7NEX6kSrYW21FMa-IcGogYYVM0ru0vEJR8tfoVYmLWZiyjO7iQ6yTdJLOpxna5/w640-h363/SC23%20v10.png" width="640" /></a></div>
The conference network used in the demonstration, <a href="https://sc23.supercomputing.org/scinet/">SCinet</a>, is described as <i>the most powerful and advanced network on Earth, connecting the SC community to the world</i>.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjf4q_tJABIngJqdakDvQeCvwwosexIr8p9j_-opZIQjGrSOJoKlfWxsJD9huxQUUqi4VBKLrP2pg0mN_PFBjbSE5u6eXrIiLcHxeQRlXYj5gKj2QESymM0KApN8MHMMJZtaTu8gqWr48u-aTHjetEA6SXFa_sbHW1fhvV-vH8uAShif2B8A91523Y7Af8S/s1472/rt-ecosystem.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="740" data-original-width="1472" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjf4q_tJABIngJqdakDvQeCvwwosexIr8p9j_-opZIQjGrSOJoKlfWxsJD9huxQUUqi4VBKLrP2pg0mN_PFBjbSE5u6eXrIiLcHxeQRlXYj5gKj2QESymM0KApN8MHMMJZtaTu8gqWr48u-aTHjetEA6SXFa_sbHW1fhvV-vH8uAShif2B8A91523Y7Af8S/s600/rt-ecosystem.png" width="600" /></a></div>
In this example, the <a href="https://sflow-rt.com">sFlow-RT</a> real-time analytics engine receives <a href="https://sflow.org">sFlow</a> telemetry from switches, routers, and servers in the SCinet network and creates metrics to drive the real-time charts in the dashboard. <a href="https://sflow-rt.com/intro.php">Getting Started</a> provides a quick introduction to deploying and using sFlow-RT for real-time network-wide flow analytics.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEil7ZLvmvvfeOQPZWFe7TCVM02rKv8qnWLR7Uae5QnCNgyorM0maSgZ1Ee_yc9Pd2xDHeGPfZAPK8fvPRN3ngOHzXJBrEZtuPtwfn_I0bh5sSgFK_UOL3GfW9GTcHv0c6SWcm47WXsow-feDfKaN7KmY1e2D75jcB88zO8VA5RX6UhyphenhyphenPTWwAETqjPZuqoNS/s2210/sc23-traffic-grafana.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1360" data-original-width="2210" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEil7ZLvmvvfeOQPZWFe7TCVM02rKv8qnWLR7Uae5QnCNgyorM0maSgZ1Ee_yc9Pd2xDHeGPfZAPK8fvPRN3ngOHzXJBrEZtuPtwfn_I0bh5sSgFK_UOL3GfW9GTcHv0c6SWcm47WXsow-feDfKaN7KmY1e2D75jcB88zO8VA5RX6UhyphenhyphenPTWwAETqjPZuqoNS/s600/sc23-traffic-grafana.png" width="600" /></a></div>
The dashboard above trends <i>SC23 Total Traffic</i>. The dashboard was constructed using the Prometheus time series database to store metrics retrieved from sFlow-RT and Grafana to build the dashboard. <a href="https://blog.sflow.com/2023/07/deploy-real-time-network-dashboards.html">Deploy real-time network dashboards using Docker compose</a> demonstrates how to deploy and configure these tools to create custom dashboards like the one shown here.
<p>Finally, check out the <a href="https://blog.sflow.com/2023/11/sc23-dropped-packet-visibility.html">SC23 Dropped packet visibility demonstration</a> to learn about one of newest developments in sFlow monitoring and see a live demonstration.</p>Peterhttp://www.blogger.com/profile/00856599914190257147noreply@blogger.com0tag:blogger.com,1999:blog-1978652979840829013.post-44688525692910160282023-11-10T13:46:00.000-08:002023-11-21T10:43:42.270-08:00SC23 Dropped packet visibility demonstration<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmXVxEMCGWW1nhLEmeo6TT2zgbc_Qk1WeuiMrMHtNBq77e6GFwFR0wtL0mST6hWrqQ0eq9DayQ9dC_aX9ZkmLrju5oRTuOr-v7k4wy7FP9iYYBTvthmaaiQoYOEd8yNRYjW0msxlBPO38Z90d_QRgon0WuK9v_-YL9jGqkcv4awj6GX1RAM4-djD6yyonP/s2376/sc23-rt-drops.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="2262" data-original-width="2376" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmXVxEMCGWW1nhLEmeo6TT2zgbc_Qk1WeuiMrMHtNBq77e6GFwFR0wtL0mST6hWrqQ0eq9DayQ9dC_aX9ZkmLrju5oRTuOr-v7k4wy7FP9iYYBTvthmaaiQoYOEd8yNRYjW0msxlBPO38Z90d_QRgon0WuK9v_-YL9jGqkcv4awj6GX1RAM4-djD6yyonP/s600/sc23-rt-drops.png" width="600" /></a></div>
The real-time dashboard is a joint InMon / Arista <a href="https://sc23.supercomputing.org/scinet/network-research-exhibition/">Network Research Exhibition</a>, <i>SC23-NRE-026 Standard Packet Drop Monitoring In High Performance Networks</i>. a part of <a href="https://sc23.supercomputing.org/">The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC23)</a> conference being held this week in Denver.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTV8kW_O4iOy1LA8ZLogELIIBIyx6M1XqIe7d0ST1b2jSAN0hrGvt9OB3MWsWeUTSjzWE0PGPceO5wqy341i9vv54tGrarbZ8h2oJwMozdHWRvksV58LMMfK7Z1ez13NbZbDCGSF3o2AKaqbVbtGohLC6riJtaeyyh6Wf5CXkXKVs24CYJOACBpCjzii7L/s1024/SC23%20v10.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="579" data-original-width="1024" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTV8kW_O4iOy1LA8ZLogELIIBIyx6M1XqIe7d0ST1b2jSAN0hrGvt9OB3MWsWeUTSjzWE0PGPceO5wqy341i9vv54tGrarbZ8h2oJwMozdHWRvksV58LMMfK7Z1ez13NbZbDCGSF3o2AKaqbVbtGohLC6riJtaeyyh6Wf5CXkXKVs24CYJOACBpCjzii7L/s600/SC23%20v10.png" width="600" /></a></div>
The conference network used in the demonstration, <a href="https://sc23.supercomputing.org/scinet/">SCinet</a>, is described as <i>the most powerful and advanced network on Earth, connecting the SC community to the world.</i>
<p>The <i>SC23-NRE-026 Standard Packet Drop Monitoring In High Performance Networks</i> dashboard combines telemetry from all the Arista switches in the SCinet network to provide real-time network-wide view of performance. Each of the three charts demonstrate a different type of measurement in the <a href="https://sflow.org">sFlow</a> telemetry stream:</p>
<ul style="text-align: left;"><li><b>Counters: Total Traffic</b> shows total traffic calculated from interface counters streamed from all interfaces. Counters provide a useful way of accurately reporting byte, frame, error and discard counters for each network interface. In this case, the chart rolls up data from all interfaces to trend total traffic on the network.</li><li><b>Samples: Top Flows</b> shows the top 5 largest traffic flows traversing the network. The chart is based on sFlow's random packet sampling mechanism, providing a scaleable method of determining the hosts and services responsible for the traffic reported by the counters. Visibility into top flows is essential if one wants to take action to manage network usage and capacity: immediately identifying DDoS attacks, elephant flows, and tracking changing service demands. <br /><b>Note:</b> Network addresses have been masked for privacy.</li><li><b>Notifications: Dropped Packets</b> shows each dropped packet, the device that dropped it, and the reason it was dropped. Dropped packets have a profound impact on network performance and availability. Packet discards due to congestion can significantly impact application performance. Dropped packets due to black hole routes, expired TTLs, MTU mismatches, etc can result in insidious connection failures that are time consuming and difficult to diagnose. <br /><b>Note:</b> Network addresses have been masked for privacy.</li></ul>
The sFlow data model integrates the three telemetry streams: counters, packet samples, and drop notifications. Each type of data is useful on its own, but together they provide the system wide observability needed to drive automation.
<pre>sflow sampling 50000
sflow polling-interval 20
sflow vrf mgmt destination 2001:XXX:XXX:XXXX::XXX
sflow vrf mgmt source-interface Management0
sflow extension bgp
sflow run</pre>
The above Arista EOS commands enable sFlow counter polling and packet sampling on all ports, sending the sFlow telemetry to the sFlow analyzer at 2001:XXX:XXX:XXXX::XXX (IPv6 address masked for privacy).
<pre>flow tracking mirror-on-drop
sample limit 100 pps
!
tracker SC23
exporter SC23
format sflow
collector sflow
local interface Management0
no shutdown</pre>
The above commands add <a href="https://sflow.org/sflow_drops.txt">sFlow Dropped Packet Notification Structures</a> to the sFlow telemetry feed. EOS 4.30.1f on Jericho 2 platforms (e.g. Arista 7804r3 at the core of SCinet diagram) is required since the implementation is based on <a href="https://blog.sflow.com/2020/10/broadcom-mirror-on-drop-mod.html">Broadcom Mirror on Drop (MoD)</a> instrumentation. Broadcom implements mirror-on-drop in Jericho 2, Trident 3, and Tomahawk 3, or later ASICs so it should be possible for Arista to release broad support across products incorporating these ASICs.
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcpsXMiRtVs6zpNW36nUSxZCXc6OWwu-v5t87Svbwy4KF9mLOFQz_a9tzx1HOZPnLQmFeKzvmblPjNMqoIjzy7GfHyug3O7wPzZ4jluuK7_nnoiTOFME_ybWvXT7qgLgwteDTm4-SximOOSvLpKd4aQTGI1RI09Ljrhb7xFkizbMImgjAz2kupgQkA-BSV/s1472/rt-ecosystem.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="740" data-original-width="1472" height="322" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcpsXMiRtVs6zpNW36nUSxZCXc6OWwu-v5t87Svbwy4KF9mLOFQz_a9tzx1HOZPnLQmFeKzvmblPjNMqoIjzy7GfHyug3O7wPzZ4jluuK7_nnoiTOFME_ybWvXT7qgLgwteDTm4-SximOOSvLpKd4aQTGI1RI09Ljrhb7xFkizbMImgjAz2kupgQkA-BSV/w640-h322/rt-ecosystem.png" width="640" /></a></div>
In this example, the <a href="https://sflow-rt.com">sFlow-RT</a> real-time analytics engine receives sFlow telemetry from switches, routers, and servers in the SCinet network and create metrics to drive the real-time charts in the dashboard. <a href="https://sflow-rt.com/intro.php">Getting Started</a> provides a quick introduction to deploying and using sFlow-RT for real-time network-wide flow analytics. The demonstration dashboard only scratches the surface of the detailed visibility that is possible analyzing the <a href="https://blog.sflow.com/2009/05/packet-headers.html">packet headers</a> exported in sFlow packet samples and dropped packet notifications - see <a href="https://sflow-rt.com/define_flow.php">Defining Flows</a>.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZ6ejlJ_MTms9Bb3JWwW2KGhAUnDdjUCTYw1f80XqcjVh1eW82g-rIJ_aVzK4CkzLDVbF0VIX1jFVLSaxdGDklZYPXOKcjOUJuJ_rjyaOYDlVWV8tlD4p10Qtlql4R28wmBMnD4ol0xpMLdUUSoDoFK5kEYDgxZPlzXt4Bi5zTJKJEs4MxFr21nDDs0EeI/s2612/grafana-sc23-drops.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1970" data-original-width="2612" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZ6ejlJ_MTms9Bb3JWwW2KGhAUnDdjUCTYw1f80XqcjVh1eW82g-rIJ_aVzK4CkzLDVbF0VIX1jFVLSaxdGDklZYPXOKcjOUJuJ_rjyaOYDlVWV8tlD4p10Qtlql4R28wmBMnD4ol0xpMLdUUSoDoFK5kEYDgxZPlzXt4Bi5zTJKJEs4MxFr21nDDs0EeI/s600/grafana-sc23-drops.png" width="600" /></a></div>
The dashboard above trends <i>Total Packet Rate</i> and <i>Dropped Packet Rate by Reason</i>. The dashboard was constructed using the Prometheus time series database to store metrics retrieved from sFlow-RT and Grafana to build the dashboard. <a href="https://blog.sflow.com/2023/07/deploy-real-time-network-dashboards.html">Deploy real-time network dashboards using Docker compose</a> demonstrates how to deploy and configure these tools to create custom dashboards like the one shown here.
<p>Industry standard sFlow telemetry is widely supported by data center switch vendors and provides the scaleable real-time visibility needed to understand and manage traffic in high performance networks. The open source <a href="https://sflow.net">Host sFlow</a> agent extends visibility onto servers to ensure end-to-end visibility.</p><p>Visibility into dropped packets is essential for Artificial Intelligence/Machine Learning (AI/ML) workloads, where a single dropped packet can stall large scale computational tasks, idling millions of dollars worth of GPU/CPU resources, and delaying the completion of business critical workloads. Enable real-time sFlow telemetry to provide the observability needed to effectively manage these networks.</p>Peterhttp://www.blogger.com/profile/00856599914190257147noreply@blogger.com0tag:blogger.com,1999:blog-1978652979840829013.post-42128165897244518932023-10-05T07:56:00.001-07:002023-10-05T07:56:57.151-07:00Internet eXchange Provider (IXP) Metrics<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg2SdW9QUGypAHUoKCt4X8U0pmJr4hleC0Dz6bBHsKugjMqOdAEgzPaFo0Xb0lvyQCW6eQgq52kSfqDA6wtsPMpNQcgfLK1BaaMNRvum3GhaqZ5NpcVC51oPQWOAZx3dqNBJH23ooUkDjQVUS4cx_VeAXscXnsymZMyk902EjghV4hwYfD2nbB8x9btNDJS/s2250/ixp-traffic.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="2250" data-original-width="2036" height="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg2SdW9QUGypAHUoKCt4X8U0pmJr4hleC0Dz6bBHsKugjMqOdAEgzPaFo0Xb0lvyQCW6eQgq52kSfqDA6wtsPMpNQcgfLK1BaaMNRvum3GhaqZ5NpcVC51oPQWOAZx3dqNBJH23ooUkDjQVUS4cx_VeAXscXnsymZMyk902EjghV4hwYfD2nbB8x9btNDJS/s600/ixp-traffic.png" /></a></div><p><a href="https://github.com/sflow-rt/ixp-metrics">IXP Metrics</a> is available on Github. The application provides real-time monitoring of traffic between members of an Internet eXchange Provider (IXP) network.</p><p>This article will use Arista switches as an example to illustrate the steps needed to deploy the monitoring solution, however, these steps should work for other network equipment vendors (provided you modify the vendor specific elements in this example).</p>
<pre>git clone https://github.com/sflow-rt/prometheus-grafana.git
cd prometheus-grafana
env RT_IMAGE=ixp-metrics ./start.sh</pre>
<p>The easiest way to get started is to use Docker, see <a href="https://blog.sflow.com/2023/07/deploy-real-time-network-dashboards.html">Deploy real-time network dashboards using Docker compose</a>, and deploy the <a href="https://hub.docker.com/r/sflow/ixp-metrics">sflow/ixp-metrics</a> image bundling the IXP Metrics application.
</p><pre>scrape_configs:
- job_name: sflow-rt-ixp-metrics
metrics_path: /app/ixp-metrics/scripts/metrics.js/prometheus/txt
static_configs:
- targets: ['sflow-rt:8008']</pre>
Follow the directions in <a href="https://blog.sflow.com/2023/07/deploy-real-time-network-dashboards.html">the article</a> to add a Prometheus scrape task to retrieve the metrics.
<pre>sflow source-interface management 1
sflow destination 10.0.0.50
sflow polling-interval 20
sflow sample 50000
sflow run</pre>
<p>Enable <a href="https://sflow.org">sFlow</a> on all exchange switches, directing sFlow telemetry to the Docker host (in this case 10.0.0.50).</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEguIBcHwYG6gK2cMmwvY1lhu26x-mmxyZxx31zrBItqq4w-dGm9cu9VKiSsQsVf6duPUfyIuR9hiOtAJn3dvGtX7DXoMQNF8XEPcdEDm2UTxsaFwAmLDY96QBwq8NXov8V6Dz4Q8kx4l0BZISMU-NX0061ywqGRGNlafhDzDnmsOLqXeTWJBYq-EFcv_bNH/s2124/ixp-rt-dash.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1516" data-original-width="2124" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEguIBcHwYG6gK2cMmwvY1lhu26x-mmxyZxx31zrBItqq4w-dGm9cu9VKiSsQsVf6duPUfyIuR9hiOtAJn3dvGtX7DXoMQNF8XEPcdEDm2UTxsaFwAmLDY96QBwq8NXov8V6Dz4Q8kx4l0BZISMU-NX0061ywqGRGNlafhDzDnmsOLqXeTWJBYq-EFcv_bNH/s600/ixp-rt-dash.png" width="600" /></a></div>
Use the <a href="https://sflow-rt.com">sFlow-RT</a> <i>Status</i> page to confirm that sFlow is being received from the switches. In this case 286 sFlow datagrams per second are being received from 9 switches.<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhuPaoAIAQ31Lm5pR6RL2L3SczCm87Oj6aOW-zjslMGiqhS8nDsD9ZS5kTjWPjds09FjGoUS6YCoM9JKQJ_aOxZDRLGr0dfHx7hhCxUtUJQ4Zg1dF3KJQxk5OM6rdLfkC6G249TdeYkud3b5JKyPuqWT23_9_SZfe-aviiaDIPZwxocOd4MBjuMxWMVTvsT/s2268/ixp-metrics-members.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1206" data-original-width="2268" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhuPaoAIAQ31Lm5pR6RL2L3SczCm87Oj6aOW-zjslMGiqhS8nDsD9ZS5kTjWPjds09FjGoUS6YCoM9JKQJ_aOxZDRLGr0dfHx7hhCxUtUJQ4Zg1dF3KJQxk5OM6rdLfkC6G249TdeYkud3b5JKyPuqWT23_9_SZfe-aviiaDIPZwxocOd4MBjuMxWMVTvsT/s600/ixp-metrics-members.png" width="600" /></a></div>
The <a href="https://github.com/euro-ix/json-schemas/">IX-F Member Export JSON Schema V1.0</a> is used to identify exchange members and their assigned MAC addresses. Upload the member data to the IXP Metrics <i>Settings</i> tab. Additional tabs are provided to locate members and MAC addresses to switch ports, query for unauthorized traffic, see real-time charts, etc.<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqQyiEjHcwIersa5rk2GpQ2bMiFTLDM478Je2A1lHWRoVxZ1cImjjKUqptX9NIuvFCHhXOBH9CZsQx9mc6Jc5q3_SH8TCiF5t3sqDQRhTcaRBAaqg3hO1WljVypVM7OPS6XOpDH-8N-rwf0-ZdsN9SUYnr-tnYmnIpnlq7hgLg7XyLeKWj83XzT13XzMau/s2268/topology-form.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1206" data-original-width="2268" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqQyiEjHcwIersa5rk2GpQ2bMiFTLDM478Je2A1lHWRoVxZ1cImjjKUqptX9NIuvFCHhXOBH9CZsQx9mc6Jc5q3_SH8TCiF5t3sqDQRhTcaRBAaqg3hO1WljVypVM7OPS6XOpDH-8N-rwf0-ZdsN9SUYnr-tnYmnIpnlq7hgLg7XyLeKWj83XzT13XzMau/s600/topology-form.png" width="600" /></a></div>
Upload an sFlow-RT <a href="https://sflow-rt.com/topology.php">Topology</a>. In this example, <a href="https://sflow-rt.com/topology.php#eapi">Arista eAPI</a> can be used to query Arista switches and discover the network topology.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiiS48F_UpLyo4j6sgbelCICHO8OGAaVUXWhKfkVNplRlXUCg8xiMul2F7O3mk3xPsa6nzF0qHigdxkVz46T5wIHDBy7YEPYs3SoFXeNjXXg72PY4vtFnBJ3y86GkGnsRixs8oK0ufOT1spbkr4CMdEhLUM6H2JLYdJ0UfhUy4ZvC4RK3vQBOmOa23JhxzQ/s2124/topology-status.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1300" data-original-width="2124" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiiS48F_UpLyo4j6sgbelCICHO8OGAaVUXWhKfkVNplRlXUCg8xiMul2F7O3mk3xPsa6nzF0qHigdxkVz46T5wIHDBy7YEPYs3SoFXeNjXXg72PY4vtFnBJ3y86GkGnsRixs8oK0ufOT1spbkr4CMdEhLUM6H2JLYdJ0UfhUy4ZvC4RK3vQBOmOa23JhxzQ/s600/topology-status.png" width="600" /></a></div>
Use the Topology <i>Status</i> page to verify that sFlow telemetry is being received for all the switches and links in the topology.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYS-1bkWsDKrZx86jc5mLPijZfpM9Mm3pjpwJBr2-19tUfwlm5Ene17wyg6b0NH4u1N-L5AqMZVGwj_TZDn1tj9Ji1YzhZYVoOZGbQYtNvkCFfHIBIWX15zHfY8NYzEoRRcMJAU74XPsvNwpZ_g7cre3TDwOCXV_1rFkef2bOQu7YwRNELtQyaKM9kCXa0/s2672/ixp-overall.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1678" data-original-width="2672" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYS-1bkWsDKrZx86jc5mLPijZfpM9Mm3pjpwJBr2-19tUfwlm5Ene17wyg6b0NH4u1N-L5AqMZVGwj_TZDn1tj9Ji1YzhZYVoOZGbQYtNvkCFfHIBIWX15zHfY8NYzEoRRcMJAU74XPsvNwpZ_g7cre3TDwOCXV_1rFkef2bOQu7YwRNELtQyaKM9kCXa0/s600/ixp-overall.png" width="600" /></a></div>
<a href="https://grafana.com/grafana/dashboards/19706-overall-traffic/">sFlow-RT IXP Overall Traffic</a> dashboard (ID: 19706) shows overall traffic in and out of exchange, breakdown of IPv4, IPv6 and ARP traffic, packet size distribution, and total number of BGP peering connections across exchange.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfPy0z76qbi1wBMGiZGjsscQb0fNTzIW_O9Y5HSyw-wyY_meX3BwtYN6z084xnATDxz3V9JC16Mad8VTS2xvOTK2xjck4zaw46PFi4crEpGlEgjqZ-jw9A5tg3Qyi2UtGWkW9XcfxO873p7sSrWrRjCDj8z3__1_CO1SSlCQcAVapPH0kXwd5nh7Wg97E4/s2672/ixp-member-top-n.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="2296" data-original-width="2672" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfPy0z76qbi1wBMGiZGjsscQb0fNTzIW_O9Y5HSyw-wyY_meX3BwtYN6z084xnATDxz3V9JC16Mad8VTS2xvOTK2xjck4zaw46PFi4crEpGlEgjqZ-jw9A5tg3Qyi2UtGWkW9XcfxO873p7sSrWrRjCDj8z3__1_CO1SSlCQcAVapPH0kXwd5nh7Wg97E4/s600/ixp-member-top-n.png" width="600" /></a></div>
<a href="https://grafana.com/grafana/dashboards/19707-member-traffic-top-n/">sFlow-RT IXP Member Traffic Top N</a> dashboard (ID: 19707) shows peering traffic. Select a member and see trends of traffic to / from other members of the exchange.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOtWLkRIZXLU7XQdF66vuzHAtwaGk3gi_TTSl74Cr0xWE28jiKeI89l-6UOffUxp3JSgZa38gRWtHrQRzI25BILHLRayLiNWm5TYPBrCVgPCbezTSZhKGx_ZHIZrKNxDeqfEhI4-Ziuwnbr-Ebef4oRRzx_ZxRjJUKo_yLZyTTheACS47sBx8Rf22wKls7/s2350/ixp-matrix.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="2350" data-original-width="2240" height="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOtWLkRIZXLU7XQdF66vuzHAtwaGk3gi_TTSl74Cr0xWE28jiKeI89l-6UOffUxp3JSgZa38gRWtHrQRzI25BILHLRayLiNWm5TYPBrCVgPCbezTSZhKGx_ZHIZrKNxDeqfEhI4-Ziuwnbr-Ebef4oRRzx_ZxRjJUKo_yLZyTTheACS47sBx8Rf22wKls7/s600/ixp-matrix.png" /></a></div>
<a href="https://grafana.com/grafana/dashboards/19708-traffic-matrix/">sFlow-RT IXP Traffic Matrix</a> dashboard displays a grid view of the traffic exchanged between members across the exchange.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGwR9xqzq3Upass4z_4fghWjENLjLK6koexJbmH9WuoqwLhUUN31MkixP8ssfSymoHv7W0FxvQyFdSMz-37fZm0LM5cxSBuykVzvLwrHluqifkDfn4H0hVb2bzt4B20CtbnSMooekuW-A-Sg128XlB8NLvEHp_4bfkueKDSu4dpxFLM8HFv1pqkBt5G_1d/s2264/grafana-weather-trend.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1518" data-original-width="2264" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGwR9xqzq3Upass4z_4fghWjENLjLK6koexJbmH9WuoqwLhUUN31MkixP8ssfSymoHv7W0FxvQyFdSMz-37fZm0LM5cxSBuykVzvLwrHluqifkDfn4H0hVb2bzt4B20CtbnSMooekuW-A-Sg128XlB8NLvEHp_4bfkueKDSu4dpxFLM8HFv1pqkBt5G_1d/s600/grafana-weather-trend.png" width="600" /></a></div>
<a href="https://blog.sflow.com/2023/08/grafana-network-weathermap.html">Grafana Network Weathermap</a> describes how to construct a real-time dashboard showing network topology and link utilizations.
<p>Support for sFlow is a standard in switches used to construct Internet Exchanges. The combination of <a href="https://www.docker.com/">Docker</a>, <a href="https://sflow-rt.com">sFlow-RT</a>, <a href="https://prometheus.io/">Prometheus</a>, and <a href="https://grafana.com/">Grafana</a> provide a scaleable, cost effective, and flexible method of monitoring traffic and generating real-time dashboards.</p>Peterhttp://www.blogger.com/profile/00856599914190257147noreply@blogger.com0tag:blogger.com,1999:blog-1978652979840829013.post-68900900748007692042023-08-14T09:06:00.000-07:002023-08-14T09:06:07.039-07:00Containerlab dashboard<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFSauXRjTM1ogr1wUJQXa-wmrf7OoI-tBfc7NAduJRa0HJiXJtlIFA3ZpOWiLUbmVwjWSAgSXnMQJu_AA7ZL4vM3DOZ0HOO83u8aba7NsCzh_gnE6XYs0bbUk0ejUGxjY-bUMgxlCz-xDMINYbVC4EwVSYQQSrfWBoRbkvoPFsAn1Jhm_rYWFHWQC5G64L/s2496/clab-dash-charts.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="2202" data-original-width="2496" height="564" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFSauXRjTM1ogr1wUJQXa-wmrf7OoI-tBfc7NAduJRa0HJiXJtlIFA3ZpOWiLUbmVwjWSAgSXnMQJu_AA7ZL4vM3DOZ0HOO83u8aba7NsCzh_gnE6XYs0bbUk0ejUGxjY-bUMgxlCz-xDMINYbVC4EwVSYQQSrfWBoRbkvoPFsAn1Jhm_rYWFHWQC5G64L/w640-h564/clab-dash-charts.png" width="640" /></a></div>
The GitHub <a href="https://github.com/sflow-rt/containerlab">sflow-rt/containerlab</a> project contains example network topologies for the <a href="https://containerlab.dev/">Containerlab</a> network emulation tool that demonstrate real-time streaming telemetry in realistic data center topologies and network configurations. The examples use the same <a href="https://frrouting.org/">FRRouting (FRR) engine</a> that is part of <a href="https://sonicfoundation.dev/">SONiC</a>, <a href="https://www.nvidia.com/en-us/networking/ethernet-switching/cumulus-linux/">NVIDIA Cumulus Linux</a>, and <a href="https://dent.dev/">DENT</a> network operating systems. Containerlab can be used to experiment before deploying solutions into production. Examples include: tracing ECMP flows in leaf and spine topologies, EVPN visibility, and automated DDoS mitigation using BGP Flowspec and RTBH controls.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhnLfQ1mIszf3svhNz-6-J58BI_3gxqrnHcHm8KrR8DgriNNm0CPEwklzE0S1osSDVb4zY7ocjUGA7hsVrt6krBAmyuHnoRkVmCKmP3z450Wc1qIGztP9u8ogk3raOI7hibJZ1AaJsAl8u1tWItXgOryg29EEuf8c0K27EKQKXrsJGCRzxHKxgLbzhsDoBH/s640/clos5.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="270" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhnLfQ1mIszf3svhNz-6-J58BI_3gxqrnHcHm8KrR8DgriNNm0CPEwklzE0S1osSDVb4zY7ocjUGA7hsVrt6krBAmyuHnoRkVmCKmP3z450Wc1qIGztP9u8ogk3raOI7hibJZ1AaJsAl8u1tWItXgOryg29EEuf8c0K27EKQKXrsJGCRzxHKxgLbzhsDoBH/s600/clos5.png" width="600" /></a></div>
The screen capture at the top of this article shows a real-time dashboard displaying up to the second traffic analytics gathered from the 5 stage Clos fabric shown above. This article walks through the steps needed to run the example.
<pre>git clone https://github.com/sflow-rt/containerlab.git
cd containerlab
./run-clab</pre>
Run the above commands to download the project and run Containerlab on a system with <a href="https://www.docker.com/">Docker</a> installed. <a href="https://www.docker.com/products/docker-desktop/">Docker Desktop</a> is a conventient way to run the labs on a laptop.<pre>containerlab deploy -t clos5.yml</pre>
Start the emulation.
<pre>./topo.py clab-clos5</pre>
Post topology to sFlow-RT REST API. Connect to <a href="http://localhost:8008/app/containerlab-dashboard/html/">http://localhost:8008/app/containerlab-dashboard/html/</a> to access the Dashboard shown at the top of this article.
<pre>docker exec -it clab-clos5-h1 iperf3 -c 172.16.4.2</pre>
Each of the hosts in the network has an <i>iperf3</i> server, so running the above command will test bandwidth between <i>h1</i> and <i>h4</i>.
<pre>docker exec -it clab-clos5-h1 iperf3 -c 2001:172:16:4::2</pre>
Generate a large IPv6 flow between <i>h1</i> and <i>h4</i>.
The traffic flows should immediately appear in the <i>Top Flows</i> chart. You can check the accuracy by comparing the values reported by <i>iperf3</i> with those shown in the chart.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7GMEWeIMnyGuhl8MVNgD6n2ciThw3eLHsuvKmqNp9k8dmw0HA9md2mJL2AUDVGxvQBT6RKsWHT02SdTvWXDA6HEVTkcx6gro0G4wpU0JD-P2AtWvCU0N5ZoGMSOjkJZpp3fNlJ_f_mFq2McGjcIVxiqHpwKkDFg7kNQ6xqmAoSc-WHIgjRWSIxROHTyzk/s2496/clab-dash-topo.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1702" data-original-width="2496" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7GMEWeIMnyGuhl8MVNgD6n2ciThw3eLHsuvKmqNp9k8dmw0HA9md2mJL2AUDVGxvQBT6RKsWHT02SdTvWXDA6HEVTkcx6gro0G4wpU0JD-P2AtWvCU0N5ZoGMSOjkJZpp3fNlJ_f_mFq2McGjcIVxiqHpwKkDFg7kNQ6xqmAoSc-WHIgjRWSIxROHTyzk/s600/clab-dash-topo.png" width="600" /></a></div>
Click on the <i>Topology</i> tab to see a real-time weathermap of traffic flowing over the topology. See how repeated <i>iperf3</i> tests take different ECMP (equal-cost multi-path) routes across the network.
<pre>docker exec -it clab-clos5-leaf1 vtysh</pre>
Linux with open source routing software (FRRouting) is an accessible alternative to vendor routing stacks (no registration / license required, no restriction on copying means you can share images on Docker Hub, no need for virtual machines). FRRouting is popular in production network operating systems (e.g. Cumulus Linux, SONiC, DENT, etc.) and the <a href="http://docs.frrouting.org/en/latest/vtysh.html">VTY shell</a> provides an industry standard CLI for configuration, so labs built around FRR allow realistic network configurations to be explored.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinaAU-7g5Q2OCCxRbefJ7pzs8WgJu45oIjCGsz1WiKQ2kr61SI1tKkFSGXLIOuyGvTRAupDQqbq1HZs6BRGqLGkmgTHO2jbuStQEAwECfMs_oNIPuBOw8_v2h9YNrofgkHKb9XgvTJirvZi0AcU7QcncHPbUmzr3QqBRg9VHI-VdTRvQVv7Q41N_iYtPf0/s2496/clab-dash-rt.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1834" data-original-width="2496" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinaAU-7g5Q2OCCxRbefJ7pzs8WgJu45oIjCGsz1WiKQ2kr61SI1tKkFSGXLIOuyGvTRAupDQqbq1HZs6BRGqLGkmgTHO2jbuStQEAwECfMs_oNIPuBOw8_v2h9YNrofgkHKb9XgvTJirvZi0AcU7QcncHPbUmzr3QqBRg9VHI-VdTRvQVv7Q41N_iYtPf0/s600/clab-dash-rt.png" width="600" /></a></div>
Connect to <a href="http://localhost:8008/">http://localhost:8008/</a> to access the main sFlow-RT status page, additional applications, and the REST API. See <a href="https://sflow-rt.com/intro.php">Getting Started</a> for more information.<pre>containerlab destroy -t clos5.yml</pre>
When you are finished, run the above command to stop the containers and free the resources associated with the emulation. Try out <a href="https://github.com/sflow-rt/containerlab#readme">other topologies</a> from the project to explore topics such as DDoS mitigation, BGP Flowspec, and EVPN.
<p>Moving the monitoring solution from Containerlab to production is straightforward since sFlow is widely implemented in datacenter equipment from vendors including: A10, Arista, Aruba, Cisco, Edge-Core, Extreme, Huawei, Juniper, NEC, Netgear, Nokia, NVIDIA, Quanta, and ZTE. In addition, the open source <a href="https://sflow.net">Host sFlow</a> agent makes it easy to extend visibility beyond the physical network into the compute infrastructure.</p>Peterhttp://www.blogger.com/profile/00856599914190257147noreply@blogger.com0tag:blogger.com,1999:blog-1978652979840829013.post-36505363918430044932023-08-08T07:04:00.000-07:002023-08-08T07:04:54.189-07:00Grafana Network Weathermap<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEga_PNjCfnkQhE-W8Tq3UjGd-rKtH596hy7PtrFzWYvnLj8yFXGztqny-HpweS9ZxM4P3OYbEKhaSRXc8tcUh5oOSjRDh5iEvxVV8tCmdyq2DzlPJRu1fan4S6NXVIScjnAIxFrxRQCCUe-LeLC3B6IdlK2Pb3-sZDFxru6Xx1H0GXq7a3N1nwnfYGnje6m/s2176/grafana-weather.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1430" data-original-width="2176" height="420" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEga_PNjCfnkQhE-W8Tq3UjGd-rKtH596hy7PtrFzWYvnLj8yFXGztqny-HpweS9ZxM4P3OYbEKhaSRXc8tcUh5oOSjRDh5iEvxVV8tCmdyq2DzlPJRu1fan4S6NXVIScjnAIxFrxRQCCUe-LeLC3B6IdlK2Pb3-sZDFxru6Xx1H0GXq7a3N1nwnfYGnje6m/w640-h420/grafana-weather.png" width="640" /></a></div>
The screen capture above shows a simple network weathermap, displaying a network topology with links animated by real-time network analytics.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLjPBGq8F5XnGn3rpxlLf_yf-Xahhz9zY7i0GVduD3q9hMXzBPYKDn-PQzxfzIAZfR8Gn-6eZov5f4APUfFfi3zgKO0bbaH8i9Lm8lYiUTFsgTjQomgRlbgsWDh9WZ7zuRtfCQoGGzmaFvscZL7RqmNieevcX4fUjvSKNShPCtDIo9FyyJVMwgDy0qMCzo/s2264/grafana-weather-trend.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1518" data-original-width="2264" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLjPBGq8F5XnGn3rpxlLf_yf-Xahhz9zY7i0GVduD3q9hMXzBPYKDn-PQzxfzIAZfR8Gn-6eZov5f4APUfFfi3zgKO0bbaH8i9Lm8lYiUTFsgTjQomgRlbgsWDh9WZ7zuRtfCQoGGzmaFvscZL7RqmNieevcX4fUjvSKNShPCtDIo9FyyJVMwgDy0qMCzo/s600/grafana-weather-trend.png" width="600" /></a></div>Hovering over a link in the weathermap pops up a trend chart showing traffic on the link over the last 30 minutes.
<p><a href="https://blog.sflow.com/2023/07/deploy-real-time-network-dashboards.html">Deploy real-time network dashboards using Docker compose</a>, describes how to quickly deploy a real-time network analytics stack that includes the <a href="https://sflow-rt.com">sFlow-RT</a> analytics engine, <a href="https://prometheus.io/">Prometheus</a> time series database, and <a href="https://grafana.com/">Grafana</a> to create dashboards. This article describes how to extend the example using the <a href="https://grafana.com/grafana/plugins/knightss27-weathermap-panel/">Grafana Network Weathermap Plugin</a> to display network topologies like the ones shown here.</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhT7EZdA3li7IUsjZr-vAPipBN870O_BQ5m5rxLU0wec8t7tOd5bDBj6PHEouEQHrFWbRtsXXumoLcO1jDyU7G2yXUkY6mLN7ba0SeWbPY1xqWs9N_BoUimzr2tj1ds_kfIG-0pfpmBByv6Lg1gSLE4c9-TyZGUwxUIgHvLOzZD0lY-dv6B_yjlB1chRB_e/s1856/grafana-weather-queries.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1242" data-original-width="1856" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhT7EZdA3li7IUsjZr-vAPipBN870O_BQ5m5rxLU0wec8t7tOd5bDBj6PHEouEQHrFWbRtsXXumoLcO1jDyU7G2yXUkY6mLN7ba0SeWbPY1xqWs9N_BoUimzr2tj1ds_kfIG-0pfpmBByv6Lg1gSLE4c9-TyZGUwxUIgHvLOzZD0lY-dv6B_yjlB1chRB_e/s600/grafana-weather-queries.png" width="600" /></a></div>
First, add a dashboard panel and select the <i>Network Weathermap</i> visualization. Next define the three metrics shown above. The <i>ifinoctets</i> and <i>ifoutoctets</i> need to be scaled by 8 to convert from bytes per second to bits per second. Creating a custom legend entry makes it easier to select metrics to associate metric instances with weathermap links.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvovuxEOGMcUK7o9EBDEkk2aGpFvUpWUCBtsl3yEkJUPvSUxpGig30mWRpybRNj0yDGOkn_-5Ac6DlCYzJj2A-i84oY3aTIuK7lo4x4g0Rie4j2oqpBrkqE_WOCynKqGY9pApVt0RQx0Eka2ARrVWnOcFruOT_q_0UGO2go3OlkbFlQ0vjT0OmmqVAqaNv/s726/grafana-weathermap-scale.png" style="display: block; padding: 1em 0px; text-align: center;"><img border="0" data-original-height="590" data-original-width="726" height="260" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvovuxEOGMcUK7o9EBDEkk2aGpFvUpWUCBtsl3yEkJUPvSUxpGig30mWRpybRNj0yDGOkn_-5Ac6DlCYzJj2A-i84oY3aTIuK7lo4x4g0Rie4j2oqpBrkqE_WOCynKqGY9pApVt0RQx0Eka2ARrVWnOcFruOT_q_0UGO2go3OlkbFlQ0vjT0OmmqVAqaNv/w320-h260/grafana-weathermap-scale.png" width="320" /></a></div>
Add a color scale that will be used to color links by link utilization. Defining the scale first ensures that links will be displayed correctly when they are added later.<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhQx0e9YJ3JG4kq5FvtpRy8fkuKk3GVAoqbBK1OtIgDNy310DhmsufCLsJF8XdKg9i0qtgTqmofyPEGexmOUzo1xdByGLkvdNFZrOK69RpBxaggtBKX52Z04umaW6Av4jQIydDQYvnyJRuK3__LmGPEJf7wp64DL65FL4p_ZjVpSIjQulM-tTvjrSuhvigJ/s1598/grafana-weathermap-node.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1598" data-original-width="764" height="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhQx0e9YJ3JG4kq5FvtpRy8fkuKk3GVAoqbBK1OtIgDNy310DhmsufCLsJF8XdKg9i0qtgTqmofyPEGexmOUzo1xdByGLkvdNFZrOK69RpBxaggtBKX52Z04umaW6Av4jQIydDQYvnyJRuK3__LmGPEJf7wp64DL65FL4p_ZjVpSIjQulM-tTvjrSuhvigJ/s600/grafana-weathermap-node.png" /></a></div>
Add the nodes to the canvas and drag them to their desired locations. There is a large library of icons that can be used to indicate the node types. The <i>Enable Node Grid Snapping</i> makes it easier to line up nodes.<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0O8kx_DrLSmSC7zlsNoCLlfv9LlKIEwdmlJiU8SnrgaoOCbE4_6jcnAm9BsXvbe8MkxmkJAnFUJ2USHzGpfFJzKEIWYQ_FJJH7lUspGySRpNqbX6W7RzY3UDh0dxlPkOg6uCsgSWqTxR0Fp2KqQ5w89GwlYDoWnr760tT-Z86sgA8J-ufYVCgneb4qmTn/s1884/grafana-weathermap-link.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1884" data-original-width="748" height="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0O8kx_DrLSmSC7zlsNoCLlfv9LlKIEwdmlJiU8SnrgaoOCbE4_6jcnAm9BsXvbe8MkxmkJAnFUJ2USHzGpfFJzKEIWYQ_FJJH7lUspGySRpNqbX6W7RzY3UDh0dxlPkOg6uCsgSWqTxR0Fp2KqQ5w89GwlYDoWnr760tT-Z86sgA8J-ufYVCgneb4qmTn/s600/grafana-weathermap-link.png" /></a></div>
Add links to connect the nodes. Each link needs to be associated with in/out metrics and and a link speed. Setting the <i>Side Anchor Point</i> values correctly ensures a clean layout.<p>Network weathermaps are only one method of displaying network telemetry - work through the examples in <a href="https://blog.sflow.com/2023/07/deploy-real-time-network-dashboards.html">Deploy real-time network dashboards using Docker compose</a> to learn how to construct dashboards of trend charts and analyze traffic flows.</p>Peterhttp://www.blogger.com/profile/00856599914190257147noreply@blogger.com0tag:blogger.com,1999:blog-1978652979840829013.post-5239166704070906942023-07-13T10:03:00.002-07:002023-08-04T16:50:30.208-07:00Deploy real-time network dashboards using Docker compose<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcpsXMiRtVs6zpNW36nUSxZCXc6OWwu-v5t87Svbwy4KF9mLOFQz_a9tzx1HOZPnLQmFeKzvmblPjNMqoIjzy7GfHyug3O7wPzZ4jluuK7_nnoiTOFME_ybWvXT7qgLgwteDTm4-SximOOSvLpKd4aQTGI1RI09Ljrhb7xFkizbMImgjAz2kupgQkA-BSV/s1472/rt-ecosystem.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="740" data-original-width="1472" height="322" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcpsXMiRtVs6zpNW36nUSxZCXc6OWwu-v5t87Svbwy4KF9mLOFQz_a9tzx1HOZPnLQmFeKzvmblPjNMqoIjzy7GfHyug3O7wPzZ4jluuK7_nnoiTOFME_ybWvXT7qgLgwteDTm4-SximOOSvLpKd4aQTGI1RI09Ljrhb7xFkizbMImgjAz2kupgQkA-BSV/w640-h322/rt-ecosystem.png" width="640" /></a></div><br />This article demonstrates how to use docker compose to quickly deploy a real-time network analytics stack that includes the <a href="https://sflow-rt.com">sFlow-RT</a> analytics engine, <a href="https://prometheus.io/">Prometheus</a> time series database, and <a href="https://grafana.com/">Grafana</a> to create dashboards.
<pre>git clone https://github.com/sflow-rt/prometheus-grafana.git
cd prometheus-grafana
./start.sh</pre>Download the <a href="https://github.com/sflow-rt/prometheus-grafana">sflow-rt/prometheus-grafana</a> project from GitHub on a system with <a href="https://www.docker.com/">Docker</a> installed and start the containers. The <i>start.sh</i> script runs <a href="https://docs.docker.com/compose/">docker compose</a> to bring up the containers specified in the <a href="https://github.com/sflow-rt/prometheus-grafana/blob/master/compose.yml">compose.yml</a> file, passing in user information so that the containers have correct permission to write data files in the <i>prometheus</i> and <i>grafana</i> directories.<div><blockquote>All the Docker images in this example are available for both x86 and ARM processors, so this stack can be deployed on Intel/AMD platforms as well as Apple M1/M2 or Raspberry Pi. <a href="https://blog.sflow.com/2023/06/raspberry-pi-4-real-time-network.html">Raspberry Pi 4 real-time network analytics</a> describes how to configure a Raspberry Pi 4 to run Docker and perform real-time network analytics and is a simple way to run this stack for smaller networks.</blockquote></div>
<p><a href="https://sflow-rt.com/agents.php">Configure sFlow Agents</a> in network devices to stream sFlow telemetry to the host running the analytics stack. See <a href="https://sflow-rt.com/intro.php">Getting Started</a> for information on how to verify that sFlow telemetry is being received.</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQFus-8-tXAuFxbKRUk4gSXEEO8UXO0k1Jw1qhJ4xYu3_1IWSs1h1mR6_-q_5I6uWjusvjNQzaSPPbMggcxGSZyhIIpeABJ5dIHfbQnP_GlSDGM2X0GawLwS-d_Z_eO9GwguE82w9PH4e5-9LvmDKYe_YezWz9qnh7Qzl_Yw1rAd0fFqnU4nRVZtRQFu1j/s1990/grafana-login.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1636" data-original-width="1990" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQFus-8-tXAuFxbKRUk4gSXEEO8UXO0k1Jw1qhJ4xYu3_1IWSs1h1mR6_-q_5I6uWjusvjNQzaSPPbMggcxGSZyhIIpeABJ5dIHfbQnP_GlSDGM2X0GawLwS-d_Z_eO9GwguE82w9PH4e5-9LvmDKYe_YezWz9qnh7Qzl_Yw1rAd0fFqnU4nRVZtRQFu1j/s600/grafana-login.png" width="600" /></a></div>
Connect to the Grafana web interface on port 3000 using default user name and password (admin/admin). You will be promted to change the password.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgq_6PbxW2gfPHbays8aYkjTxslHN-PSPt83sq_ul8o0yMSRsY6BGvS0DLmLkVx0YKM3brbHnI4NfG1umHC5P-k3FbhpXpCPIO2LQyHzThYoDAFZCo2whlWvZPu37yFTeB9V2e9_FXwkLTCmQvTP3aePZIklCKXkp-xhyuStUu_sSBTk6zQPRwSYAVeHBGC/s1990/grafana-import.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1634" data-original-width="1990" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgq_6PbxW2gfPHbays8aYkjTxslHN-PSPt83sq_ul8o0yMSRsY6BGvS0DLmLkVx0YKM3brbHnI4NfG1umHC5P-k3FbhpXpCPIO2LQyHzThYoDAFZCo2whlWvZPu37yFTeB9V2e9_FXwkLTCmQvTP3aePZIklCKXkp-xhyuStUu_sSBTk6zQPRwSYAVeHBGC/s600/grafana-import.png" width="600" /></a></div>
Select the option to Import a new Dashboard.<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqnJJCUdyoOiqoZoY2g8o0BjL8oyXYPmbZsdmNRPOICwDdjoWpAqMBKJgmWhplxQxfC2se73hDbtyNbJD1s4cg07LkJ_I5lt7l601uJ_wlD5x0Om3p5hvAvqW4Y-aABTDV9P58iFfznVrTs2TSmbqGwV29Leyu2M7VUqeprvxeoIPuRGOHj-gUGgjsWMXw/s1990/grafana-load.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1216" data-original-width="1990" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqnJJCUdyoOiqoZoY2g8o0BjL8oyXYPmbZsdmNRPOICwDdjoWpAqMBKJgmWhplxQxfC2se73hDbtyNbJD1s4cg07LkJ_I5lt7l601uJ_wlD5x0Om3p5hvAvqW4Y-aABTDV9P58iFfznVrTs2TSmbqGwV29Leyu2M7VUqeprvxeoIPuRGOHj-gUGgjsWMXw/s600/grafana-load.png" width="600" /></a></div>
Enter the code <i>11201</i> to import <a href="https://grafana.com/grafana/dashboards/11201-sflow-rt-network-interfaces/">sFlow-RT Network Interfaces</a> dashboard from Grafana.com and click on the <i>Load</i> button.<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhVmIdu2FtMDlbxHsdN99MTO--VcffyYt5ESFevFlSJWU6D8nHtMy3psiSGaPNX4jaFEadtgD11U2_WkoigPg8DKw1-tIfGOtB8yLHwLLnTfSw9CZuCcyu_QTzXGDy2y3cHI6wSKGeIsCdLAZawkDlibki5m8hGLrfPsBgjAeMLZN9b29Z_KPmhOXpLK25V/s1990/grafana-data.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1886" data-original-width="1990" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhVmIdu2FtMDlbxHsdN99MTO--VcffyYt5ESFevFlSJWU6D8nHtMy3psiSGaPNX4jaFEadtgD11U2_WkoigPg8DKw1-tIfGOtB8yLHwLLnTfSw9CZuCcyu_QTzXGDy2y3cHI6wSKGeIsCdLAZawkDlibki5m8hGLrfPsBgjAeMLZN9b29Z_KPmhOXpLK25V/s600/grafana-data.png" width="600" /></a></div>
Select the <i>sflow_rt_data</i> Prometheus database and click on the <i>Import</i> button.<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFQ9Gs7PEkSB3y8mFdwoIT_yO1CtoU9RSXVIOT2JZuxy_gq5xQzlW6VVo2VVM-FO2Qdg5BeN21IeSSRfIQ0VsbMSVUJ_qLu1UA-lSTk0aayDPKBeDbMHhA7ZtLaLGCPqUMRSKnQkgcuMtVnnuVmOV89jcRJS6pvQTYGAuawEoermF2n2JiFsIH_NESHo34/s2414/grafana-dash.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="2414" data-original-width="1990" height="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFQ9Gs7PEkSB3y8mFdwoIT_yO1CtoU9RSXVIOT2JZuxy_gq5xQzlW6VVo2VVM-FO2Qdg5BeN21IeSSRfIQ0VsbMSVUJ_qLu1UA-lSTk0aayDPKBeDbMHhA7ZtLaLGCPqUMRSKnQkgcuMtVnnuVmOV89jcRJS6pvQTYGAuawEoermF2n2JiFsIH_NESHo34/s600/grafana-dash.png" /></a></div>
The dashboard should appear showing top interfaces by Utilization, Discards and Errors.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyKuV32QygKnvy0bNZLVFX5IF5b3bZgw4j0MtpwKLg9wkSw0bK-6fgBtBI_ysYpWyHKT9u7F7ep46o4-wGbe2j8wbPGr_w_j79En0OUxG4Djuex0lbucLU-djPExA-87lhb9zmEUnhbwybzxmoD7acDOWVkpPuC0poVZQKvd36OPyqWoRGV6fb89q5GAT5/s2364/grafana-rt-dash.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="2364" data-original-width="1990" height="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyKuV32QygKnvy0bNZLVFX5IF5b3bZgw4j0MtpwKLg9wkSw0bK-6fgBtBI_ysYpWyHKT9u7F7ep46o4-wGbe2j8wbPGr_w_j79En0OUxG4Djuex0lbucLU-djPExA-87lhb9zmEUnhbwybzxmoD7acDOWVkpPuC0poVZQKvd36OPyqWoRGV6fb89q5GAT5/s600/grafana-rt-dash.png" /></a></div>
Repeat the steps to add the <a href="https://grafana.com/grafana/dashboards/11096-sflow-rt/">sFlow-RT Health dashboard</a>, code <i>11096</i>.<p>The <a href="https://grafana.com/grafana/dashboards/11146-sflow-rt-countries-and-networks/">sFlow-RT Countries and Networks</a> dashboard is an example of a flow based metric, plotting information about source and destination countries and provider networks based on traffic analytics.</p><p>Prometheus has already been programmed to gather metrics for the previous two example, but to run this third example, we need to modify the Prometheus configuration to gather the flow based metrics needed for the dashboard.</p>
<pre> - job_name: 'sflow-rt-countries'
metrics_path: /app/prometheus/scripts/export.js/flows/ALL/txt
static_configs:
- targets: ['sflow-rt:8008']
params:
metric: ['sflow_country_bps']
key:
- 'null:[country:ipsource:both]:unknown'
- 'null:[country:ipdestination:both]:unknown'
label: ['src','dst']
value: ['bytes']
scale: ['8']
aggMode: ['sum']
minValue: ['1000']
maxFlows: ['100']
- job_name: 'sflow-rt-asns'
metrics_path: /app/prometheus/scripts/export.js/flows/ALL/txt
static_configs:
- targets: ['sflow-rt:8008']
params:
metric: ['sflow_asn_bps']
key:
- 'null:[asn:ipsource:both]:unknown'
- 'null:[asn:ipdestination:both]:unknown'
label: ['src','dst']
value: ['bytes']
scale: ['8']
aggMode: ['sum']
minValue: ['1000']
maxFlows: ['100']</pre>
Edit the <i>prometheus/prometheus.yml</i> file and add the above lines to the end of the file.
<pre>docker restart prometheus</pre>
Restart the prometheus container to pick up the new configuration and start collecting the data.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEik0E45rwjIvaZc07BzPcZXZNpbvBVvMLU6kqj6OFmu1jr3U0e-qabn0HojRpLrdp-vr_HmL6kSZKcpcMO9q08ZTNzEV3OWlvz48wfXLMKVnJWkd--5pEV750M6K70-vy3n8IWFO6FdcQJFC-i9H5NKgaIkeM8VTjHPTeS3dVNvn8A3cFb9w_Xx0ErodYXE/s2152/grafana-rt-country.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="2152" data-original-width="1906" height="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEik0E45rwjIvaZc07BzPcZXZNpbvBVvMLU6kqj6OFmu1jr3U0e-qabn0HojRpLrdp-vr_HmL6kSZKcpcMO9q08ZTNzEV3OWlvz48wfXLMKVnJWkd--5pEV750M6K70-vy3n8IWFO6FdcQJFC-i9H5NKgaIkeM8VTjHPTeS3dVNvn8A3cFb9w_Xx0ErodYXE/s600/grafana-rt-country.png" /></a></div>
Add dashboard <i>11146</i> to load the <i>sFlow-RT Countries and Networks</i> dashboard.
<p><a href="https://sflow-rt.com/intro.php">Getting Started</a> describes how to use the sFlow-RT <i>Flow Browser</i> and <i>Metrics Browser</i> applications to explore the data that is available (the sFlow-RT web interface is exposed on port 8008). Once you have found a useful metric, add it to the set of metrics for Prometheus (the Prometheus web interface is exposed on port 9090) to collect and use Grafana to build dashboards that incorporate the new metrics. <a href="https://blog.sflow.com/2019/10/flow-metrics-with-prometheus-and-grafana.html">Flow metrics with Prometheus and Grafana</a> describes how Prometheus can use sFlow-RT's REST API to define and retrieve traffic flow based metrics like the ones in the <i>Countries and Networks</i> dashboard. </p>Peterhttp://www.blogger.com/profile/00856599914190257147noreply@blogger.com0tag:blogger.com,1999:blog-1978652979840829013.post-9489381792189120712023-06-11T13:17:00.002-07:002023-07-21T10:23:57.362-07:00Raspberry Pi 4 real-time network analytics<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhaGPklLxF88NiIeMbEZqdF-sZqYzqKPLyiernWnIAuCD4Bkl_p88fpttrri8s-Moj_o6ZJjlGjrf_-ah7_dSegAV6HKMcWQdalfZtLl5xHnu9gxiQPj70Fdc55jfUo0hayz0W7tHEVbQmCm2_EnEqZBQbDcHw9jHDnX5TJ4ASLnOrXii-pRXoBqXX_yA/s4032/IMG_3718.HEIC" style="display: block; margin-left: auto; margin-right: auto; padding: 1em 0px; text-align: center;"><img border="0" data-original-height="3024" data-original-width="4032" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhaGPklLxF88NiIeMbEZqdF-sZqYzqKPLyiernWnIAuCD4Bkl_p88fpttrri8s-Moj_o6ZJjlGjrf_-ah7_dSegAV6HKMcWQdalfZtLl5xHnu9gxiQPj70Fdc55jfUo0hayz0W7tHEVbQmCm2_EnEqZBQbDcHw9jHDnX5TJ4ASLnOrXii-pRXoBqXX_yA/w640-h480/IMG_3718.HEIC" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://www.canakit.com/raspberry-pi-4-extreme-aluminum-case-kit.html" style="text-align: left;">CanaKit Raspberry Pi 4 EXTREME Kit - Aluminum</a></td></tr></tbody></table>This article describes how build an inexpensive Raspberry Pi 4 based server for real-time flow analytics of industry standard <a href="https://sflow.org">sFlow</a> streaming telemetry. Support for sFlow is widely implemented in datacenter equipment from vendors including: A10, Arista, Aruba, Cisco, Edge-Core, Extreme, Huawei, Juniper, NEC, Netgear, Nokia, NVIDIA, Quanta, and ZTE.<p></p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKPA5sgDOXBypVIL6cppXaeZsiRH1KkRVvJ_bcBc26ZH0O97--dAgpWqD9OFM6Nn6UKAf1taaUMW5BxANB2tYqx-Ggr-FtXR1NOjImLMILEQD1T4adoMEuhoQQl1egZYfsOPOpPh5gi-dwZH1M5f0Wj3qvvmDUcqZ-AISpxwtxfqspDTmDWoCMWYOGkw/s1584/pi-imager.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1120" data-original-width="1584" height="452" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKPA5sgDOXBypVIL6cppXaeZsiRH1KkRVvJ_bcBc26ZH0O97--dAgpWqD9OFM6Nn6UKAf1taaUMW5BxANB2tYqx-Ggr-FtXR1NOjImLMILEQD1T4adoMEuhoQQl1egZYfsOPOpPh5gi-dwZH1M5f0Wj3qvvmDUcqZ-AISpxwtxfqspDTmDWoCMWYOGkw/w640-h452/pi-imager.png" width="640" /></a></div>
In this example, we will use an 8G Raspberry Pi 4 running <i>Raspberry Pi OS Lite (64-bit)</i>. The easiest way to format a memory card and install the operating system is to use the <a href="https://www.raspberrypi.com/software/">Raspberry Pi Imager</a> (shown above).
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5hF0Q0HKJHScZOzQj3UC6-hTxTCS0nwBv-eWrVjhICj601XgqitWT7qrV3pDnkzRqZZAGtUWkxeRqrTJP-XrKirDYUdXyNTzP-QoSLThOpoGfNgtRPl16_KIMJQmDtT_jdr1ZoIKQFcaWaE4dmeLgqSyEZfFPWPs9IyS-Un0ESeqre8J7T3xhJr-HzQ/s1584/pi-imager-settings.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1120" data-original-width="1584" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5hF0Q0HKJHScZOzQj3UC6-hTxTCS0nwBv-eWrVjhICj601XgqitWT7qrV3pDnkzRqZZAGtUWkxeRqrTJP-XrKirDYUdXyNTzP-QoSLThOpoGfNgtRPl16_KIMJQmDtT_jdr1ZoIKQFcaWaE4dmeLgqSyEZfFPWPs9IyS-Un0ESeqre8J7T3xhJr-HzQ/s600/pi-imager-settings.png" width="600" /></a></div>
Click on the gear icon to set a user and password and enable ssh access. These initial settings allow the Rasberry Pi to be accessed over the network without having to attach a screen, keyboard, and mouse.
<p>Next, follow instruction for installing <a href="https://docs.docker.com/engine/install/debian/">Docker Engine</a> (Raspberry Pi OS Lite is based on Debian 11).</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiiH9lu23JoYgF6vTdD2ZJUjEdsIX1NfnUO91NhjS0Y8gr3k_5ZnrvI_37UpEwLFwAb-_yBQVo76wWTkzeDMXj7-8BI977O6ZhWW6TytiyqhqZ5i3U3nlY6wn2J_DOLamJ7KMQTnV3bkrlWQN2RLc7rzz-uOYNGe8t_Ic7XDoXlcqRshZWP_W_McgoVKQ/s1472/rt-ecosystem.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="740" data-original-width="1472" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiiH9lu23JoYgF6vTdD2ZJUjEdsIX1NfnUO91NhjS0Y8gr3k_5ZnrvI_37UpEwLFwAb-_yBQVo76wWTkzeDMXj7-8BI977O6ZhWW6TytiyqhqZ5i3U3nlY6wn2J_DOLamJ7KMQTnV3bkrlWQN2RLc7rzz-uOYNGe8t_Ic7XDoXlcqRshZWP_W_McgoVKQ/s600/rt-ecosystem.png" width="600" /></a></div>
The diagram shows how the <a href="https://sflow-rt.com">sFlow-RT</a> real-time analytics engine receives a continuous telemetry stream from industry standard sFlow instrumentation build into network, server and application infrastructure and delivers analytics through APIs and can easily be integrated with a wide variety of on-site and cloud, orchestration, DevOps and Software Defined Networking (SDN) tools.
<pre>docker run -p 6343:6343/udp -p 127.0.0.1:8008:8008 \
--name sflow-rt -d --restart unless-stopped sflow/prometheus</pre>
Run the pre-built <a href="https://hub.docker.com/r/sflow/prometheus">sflow/prometheus</a> Docker image. In this example access to the user interface is limited to local host in order prevent unauthorized access since no access controls are provided by sFlow-RT.<pre>ssh -L 8008:127.0.0.1:8008 pp@192.168.4.163</pre>
Use ssh to connect to the Raspberry Pi (192.168.4.163) and tunnel port 8008 to your laptop.<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0Z6Xa1U14CXg4PZfzxKuC6iLl2eS8aslu0Nw1Iv1Oe7pNz9Qy4tE81dZ_8sUnazfMzdxuwKpLtIxX8nz8GCio9PHeBzTJvTYrQgP4a66F6Wvu0jMhQpyFXl3usPR0v_3SuXHRY0nCPMWre0rNxweKF85prI5YlVmGjOpjTWm_AzJ9e9WzhHm1ZBzg7g/s1986/rt-dash.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1526" data-original-width="1986" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0Z6Xa1U14CXg4PZfzxKuC6iLl2eS8aslu0Nw1Iv1Oe7pNz9Qy4tE81dZ_8sUnazfMzdxuwKpLtIxX8nz8GCio9PHeBzTJvTYrQgP4a66F6Wvu0jMhQpyFXl3usPR0v_3SuXHRY0nCPMWre0rNxweKF85prI5YlVmGjOpjTWm_AzJ9e9WzhHm1ZBzg7g/s600/rt-dash.png" width="600" /></a></div>Access the web interface at <a href="http://127.0.0.1:8008/">http://127.0.0.1:8008/</a>. See <a href="https://sflow-rt.com/intro.php">Getting Started</a> for instructions for enabling monitoring and browsing metrics. Python is installed by default on Raspberry Pi OS, making it convenient to experiment with the sFlow-RT REST API, see <a href="https://sflow-rt.com/writing_applications.php">Writing Applications</a>.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPmNaBnj9RjaZgNvlD0XJpUfRCd6osh3jAIcqp-HQBMIpZtn-fLOtANfLLYrduiebLYQXUBwwFRzWe796iZcffsLKxWoHT9vO4ZlCChAi5M2suzl112WprVZwXACaWrEj698PBSPNXKX_m_zmpQg06s7uQhyQPIPe8sHNs3c8wbTm-gpfLWUIqIAn_dQ/s640/clos5.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="270" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPmNaBnj9RjaZgNvlD0XJpUfRCd6osh3jAIcqp-HQBMIpZtn-fLOtANfLLYrduiebLYQXUBwwFRzWe796iZcffsLKxWoHT9vO4ZlCChAi5M2suzl112WprVZwXACaWrEj698PBSPNXKX_m_zmpQg06s7uQhyQPIPe8sHNs3c8wbTm-gpfLWUIqIAn_dQ/s600/clos5.png" width="600" /></a></div>If you don't have immediate access to a network and want to experiment, follow the instructions in <a href="https://blog.sflow.com/2023/05/leaf-and-spine-network-emulation-on-mac.html">Leaf and spine network emulation on Mac OS M1/M2 systems</a> to emulate the 5 stage leaf and spine network shown above using <a href="https://containerlab.dev/">Containerlab</a>.
<pre>docker stop sflow-rt</pre><b>
Note:</b> If you are going to try the examples, first run the command above to stop the sflow-rt image to avoid port contention when Containerlab starts an instance of sFlow-RT.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqu-nLMTLi1s2PahoCaB5Xli4pU6M6C0iy74Jz217OEjSVSNxBjZvgq3av-kBKy4cfYFySf-aCrF6Lwocs5O9UBYobKvQyJMEaOqfLHBgCTVyNScYzFVgGD63ffKVKEw3BkarNi-h1f78zOcHy7Ph7op2z8JIOJn8j_OA0Jt4_waELU7P4UrKG37L9fA/s901/clos5-flows.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="874" data-original-width="901" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqu-nLMTLi1s2PahoCaB5Xli4pU6M6C0iy74Jz217OEjSVSNxBjZvgq3av-kBKy4cfYFySf-aCrF6Lwocs5O9UBYobKvQyJMEaOqfLHBgCTVyNScYzFVgGD63ffKVKEw3BkarNi-h1f78zOcHy7Ph7op2z8JIOJn8j_OA0Jt4_waELU7P4UrKG37L9fA/s600/clos5-flows.png" width="600" /></a></div>
The screen capture shows a real-time view of traffic flowing across the the emulated leaf and spine network during a series iperf3 tests. The emulated results are very close to those you can expect when monitoring production traffic on a physical network.<p>The Raspberry Pi 4 is surprisingly capable, this pocket-sized server can easily monitor hundreds of high speed (100G+) links, providing up to the second visibility into network flows.</p>Peterhttp://www.blogger.com/profile/00856599914190257147noreply@blogger.com0tag:blogger.com,1999:blog-1978652979840829013.post-82083450361518856562023-05-23T16:06:00.000-07:002023-05-23T16:06:18.242-07:00Leaf and spine network emulation on Mac OS M1/M2 systems<p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9_BwEf7Ee5lgR7Vzg22kGB8v0ahXncJIVsltkYvVD6DXAfdt2EFj_Xm2STbzmfJ9QgLozEeZd6ZFnavZ1wKaWkEjijbg2zJ1yQw9Ka0vCDR1CWVVJFarOJLo_k1_zeJxwLdI7hCT6D6eUgnv0XPfw-OcpXIj2nG98DHqR8XJjhC7JogR30brbsjpDyA/s640/clos5.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="270" data-original-width="640" height="270" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9_BwEf7Ee5lgR7Vzg22kGB8v0ahXncJIVsltkYvVD6DXAfdt2EFj_Xm2STbzmfJ9QgLozEeZd6ZFnavZ1wKaWkEjijbg2zJ1yQw9Ka0vCDR1CWVVJFarOJLo_k1_zeJxwLdI7hCT6D6eUgnv0XPfw-OcpXIj2nG98DHqR8XJjhC7JogR30brbsjpDyA/w640-h270/clos5.png" width="640" /></a></div><br />The GitHub <a href="https://github.com/sflow-rt/containerlab">sflow-rt/containerlab</a> project contains example network topologies for the <a href="https://containerlab.dev/">Containerlab</a> network emulation tool that demonstrate real-time streaming telemetry in realistic data center topologies and network configurations. The examples use the same <a href="https://frrouting.org/">FRRouting (FRR) engine</a> that is part of <a href="https://sonicfoundation.dev/">SONiC</a>, <a href="https://www.nvidia.com/en-us/networking/ethernet-switching/cumulus-linux/">NVIDIA Cumulus Linux</a>, and <a href="https://dent.dev/">DENT</a> network operating systems. Containerlab can be used to experiment before deploying solutions into production. Examples include: tracing ECMP flows in leaf and spine topologies, EVPN visibility, and automated DDoS mitigation using BGP Flowspec and RTBH controls.<p></p><p>The Containerlab project currently has limited support for Mac OS, stating <i>"ARM-based Macs (M1/2) are not supported, and no binaries are generated for this platform. This is mainly due to the lack of network images built for arm64 architecture as of now."</i> However, this argument doesn't apply to the Linux based images used in these examples.</p><p>First install <a href="https://www.docker.com/products/docker-desktop/">Docker Desktop</a> on your Apple silicon based Mac (select the Apple Chip option).</p>
<pre>mkdir clab
cd clab
docker run --rm -it --privileged \
--network host --pid="host" \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /run/netns:/run/netns \
-v $(pwd):$(pwd) -w $(pwd) \
sflow/clab bash</pre>
<p>Run Containerlab by typing the above commands in a terminal. This command uses a pre-built multi-architecture <a href="https://hub.docker.com/r/sflow/clab">sflow/clab</a> image. If you are running on an x86 platform, follow the official <a href="https://containerlab.dev/install/">Containerlab Installation</a> instructions.</p>
<pre>git clone https://github.com/sflow-rt/containerlab.git</pre>
<p>Download the Containerlab topologies from the <a href="https://github.com/sflow-rt/containerlab">sflow-rt/containerlab</a> project.</p>
<pre>containerlab deploy -t containerlab/clos5.yml</pre>
<p>Start the 5 stage leaf and spine topology shown at the top of this page. The initial launch may take a couple of minutes as the container images are downloaded for the first time. Once the images are downloaded, the topology deploys in around 10 seconds.</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHMztAegNBQplGOfzRkV0Zek7BTDNqJPQAGThk_U5VnhpduFNBXS0Phb4KWaRFRec-TOxZxP4sxe1mZU-DrQwwqQ9v1YWNvOo5JjQ0h5Q9LghwNYCMLqQh2kOcLO9cz6K2j16PkDtKS4-lc8yuofLbcnigr_1Jg9mBaX8EvveOat8ih5HA1-wlSav7XQ/s1472/rt-ecosystem.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="740" data-original-width="1472" height="322" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHMztAegNBQplGOfzRkV0Zek7BTDNqJPQAGThk_U5VnhpduFNBXS0Phb4KWaRFRec-TOxZxP4sxe1mZU-DrQwwqQ9v1YWNvOo5JjQ0h5Q9LghwNYCMLqQh2kOcLO9cz6K2j16PkDtKS4-lc8yuofLbcnigr_1Jg9mBaX8EvveOat8ih5HA1-wlSav7XQ/w640-h322/rt-ecosystem.png" width="640" /></a></div>An instance of the <a href="https://sflow-rt.com/">sFlow-RT</a> real-time analytics engine receives industry standard <a href="https://sflow.org/">sFlow</a> telemetry from all the switches in the network. All of the switches in the topology are configured to send sFlow to the sFlow-RT instance. In this case, Containerlab is running the pre-built <a href="https://hub.docker.com/r/sflow/prometheus">sflow/prometheus</a> image which packages sFlow-RT with useful applications for exploring the data.<p></p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_X5IMPa6q4eOwt4LGcB38GvJkZbl-OcB19QWnlGerSSiEIhmv0y7Pj79o65XAduVAgO7Rf8npCPrf1Qn9B6Vkt_Ay7mg6Oh8KJjMpAlkMwBoYGYOIWIQ0EVRgDwYVBG-xaZmI4Qt4kdEgsfM8XaEvsJ7ad2bc-L_0c1LIOBbl0vOpZivR8mS08Dkcdg/s901/clos5-dash.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="755" data-original-width="901" height="536" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_X5IMPa6q4eOwt4LGcB38GvJkZbl-OcB19QWnlGerSSiEIhmv0y7Pj79o65XAduVAgO7Rf8npCPrf1Qn9B6Vkt_Ay7mg6Oh8KJjMpAlkMwBoYGYOIWIQ0EVRgDwYVBG-xaZmI4Qt4kdEgsfM8XaEvsJ7ad2bc-L_0c1LIOBbl0vOpZivR8mS08Dkcdg/w640-h536/clos5-dash.png" width="640" /></a></div>Connect to the web interface, <a href="http://localhost:8008">http://localhost:8008</a>. The sFlow-RT dashboard verifies that telemetry is being received from 10 agents (the 10 switches in the Clos fabric). See the <a href="https://sflow-rt.com/intro.php">sFlow-RT Quickstart</a> guide for more information.<p></p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrRbi0-nl3IhD0cK-48q3t9FCpaLvrpUA2hTu2t9eds7z1eQVaobRD2wdQ4nFwVuhjgbyZquHvTrZU80Y1vo_TIR15mqP07kEsnCGwkuJIWEdqfH5NskAGIcDApfOn403j0kPFn9NyQUPf_R0bgfDq0GdWwXQzbn3R63vw9MJLQGxL7KpATJqDkf3sEw/s901/clos5-flows.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="874" data-original-width="901" height="620" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrRbi0-nl3IhD0cK-48q3t9FCpaLvrpUA2hTu2t9eds7z1eQVaobRD2wdQ4nFwVuhjgbyZquHvTrZU80Y1vo_TIR15mqP07kEsnCGwkuJIWEdqfH5NskAGIcDApfOn403j0kPFn9NyQUPf_R0bgfDq0GdWwXQzbn3R63vw9MJLQGxL7KpATJqDkf3sEw/w640-h620/clos5-flows.png" width="640" /></a></div>The screen capture shows a real-time view of traffic flowing across the network during a series <a href="https://github.com/esnet/iperf">iperf3</a> tests. Click on the sFlow-RT <i>Apps</i> menu and select the <i>browse-flows</i> application, or <a href="http://localhost:8008/app/browse-flows/html/index.html?keys=ipsource%2Cipdestination%2Cnode%3Ainputifindex%2Cifname%3Ainputifindex%2Cipttl&value=bps">click here</a> for a direct link to a chart with the settings shown above.
<pre>docker exec -it clab-clos5-h1 iperf3 -c 172.16.4.2</pre>
<p>Each of the hosts in the network has an iperf3 server, so running the above command will test bandwidth between h1 and h4.</p>
<pre>docker exec -it clab-clos5-leaf1 vtysh</pre>
<p>Linux with open source routing software (FRRouting) is an accessible alternative to vendor routing stacks (no registration / license required, no restriction on copying means you can share images on Docker Hub, no need for virtual machines). FRRouting is popular in production network operating systems (e.g. Cumulus Linux, SONiC, DENT, etc.) and the <a href="http://docs.frrouting.org/en/latest/vtysh.html">VTY shell</a> provides an industry standard CLI for configuration, so labs built around FRR allow realistic network configurations to be explored.</p>
<pre>containerlab destroy -t containerlab/clos5.yml</pre>
<p>When you are finished, run the above command to stop the containers and free the resources associated with the emulation. Try out <a href="https://github.com/sflow-rt/containerlab#readme">other topologies</a> from the project to explore topics such as DDoS mitigation, BGP Flowspec, and EVPN.</p><p>Moving the monitoring solution from Containerlab to production is straightforward since sFlow is widely implemented in datacenter equipment from vendors including: A10, Arista, Aruba, Cisco, Edge-Core, Extreme, Huawei, Juniper, NEC, Netgear, Nokia, NVIDIA, Quanta, and ZTE. In addition, the open source <a href="https://sflow.net/">Host sFlow</a> agent makes it easy to extend visibility beyond the physical network into the compute infrastructure.</p>Peterhttp://www.blogger.com/profile/00856599914190257147noreply@blogger.com0tag:blogger.com,1999:blog-1978652979840829013.post-59462315183653318782023-04-10T06:57:00.001-07:002023-04-10T13:56:46.200-07:00VyOS DDoS mitigation<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgs-L7Thh6bUgY8j0HZgReCpfKPkOJFjK20e9PUtcqMYp2a1pMtcvtpxcss5GYTz_RySN3s-gRwkaTr1oabfkq6XvRaPT3eg_ITh5_zwUANugJSDIvafNG2QgoyVtPkoHARFend5L_HWUfSUpk7CGqb3z1A_qx2wlniN2FYkSk64jXD32c4PWOH6S97vA/s1411/vyos-ddos-diagram.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="378" data-original-width="1411" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgs-L7Thh6bUgY8j0HZgReCpfKPkOJFjK20e9PUtcqMYp2a1pMtcvtpxcss5GYTz_RySN3s-gRwkaTr1oabfkq6XvRaPT3eg_ITh5_zwUANugJSDIvafNG2QgoyVtPkoHARFend5L_HWUfSUpk7CGqb3z1A_qx2wlniN2FYkSk64jXD32c4PWOH6S97vA/s600/vyos-ddos-diagram.png" width="600" /></a></div>
<a href="https://blog.sflow.com/2023/04/real-time-flow-analytics-on-vyos.html">Real-time flow analytics on VyOS</a> describes how to install real-time analytics based on sFlow and the sFlow-RT analytics engine. This article extends the example to show how to automatically mitigate DDoS attacks using flow analytics combined with BGP Remotely Triggered Black Hole (RTBH) / Flowspec.
<pre>vyos@vyos:~$ add container image sflow/ddos-protect</pre>
First, download the <a href="https://hub.docker.com/r/sflow/ddos-protect">sflow/ddos-protect</a> image.
<pre>vyos@vyos:~$ mkdir -m 777 /config/sflow-rt</pre>
Create a directory to store persistent container state.
<pre>set container network sflowrt prefix 192.168.1.0/24</pre>
Define an internal network to connect to container. Currently VyOS BGP does not allow direct connections to local addresses (e.g. 127.0.0.1), so we need to put controller on its own network so the router can connect and receive DDoS mitigation BGP RTBH / Flowspec controls.
<pre>set container name sflow-rt image sflow/ddos-protect
set container name sflow-rt host-name sflow-rt
set container name sflow-rt arguments '-Dddos_protect.router=192.168.1.1 -Dddos_protect.enable.flowspec=yes'
set container name sflow-rt environment RTMEM value 200M
set container name sflow-rt memory 0
set container name sflow-rt volume store source /config/sflow-rt
set container name sflow-rt volume store destination /sflow-rt/store
set container name sflow-rt network sflowrt address 192.168.1.2</pre>
<p>Configure a container to run the image. The <i>-Dddos_protect.router</i> argument sets the BGP neighbor address, 192.168.1.1.</p>
<pre>vyos@vyos:~$ ifconfig podman-sflowrt
podman-sflowrt: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.1 netmask 255.255.255.0 broadcast 192.168.1.255
ether be:9e:69:f4:d0:4e txqueuelen 1000 (Ethernet)
RX packets 28 bytes 2662 (2.5 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 27 bytes 8032 (7.8 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0</pre>
Connections to containers on <i>sflowrt</i> container network appear to originate from 192.168.1.1, the address assigned to VyOS interface <i>podman-sflowrt</i>.
<pre>set system sflow interface eth0
set system sflow interface eth1
set system sflow interface eth2
set system sflow polling 30
set system sflow sampling-rate 1000
set system sflow drop-monitor-limit 50
set system sflow server 192.168.1.2</pre>
Configure sFlow and send to <i>sflow-rt</i> container address 192.168.1.2.<pre>set protocols bgp system-as 64500
set protocols bgp neighbor 192.168.1.2 port 1179
set protocols bgp neighbor 192.168.1.2 remote-as 65000
set protocols bgp neighbor 192.168.1.2 address-family ipv4-unicast
set protocols bgp neighbor 192.168.1.2 address-family ipv4-flowspec</pre>
Configure <i>sflow-rt</i> as BGP neighbor. Documentation ASN 64500 should be replaced by your ASN. The private ASN 65000 is a DDoS Protect default and can be changed with the <i>-Dddos_protect.as</i> argument.
<pre>ssh -L 8008:192.168.1.2:8008 vyos@router.example</pre>
Use ssh tunnel to connect to the container network and access web interface at <a href="http://localhost:8008">http://localhost:8008</a>.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6dTaxkenQ339gtD7oPRRqVNVBDS7dgdzVNL7igbLw320tTKC_5oQz7EWKtEQnfl7_PpolDnWZNO5o3zjC37_C8UsU_-p71Ur6I8ElnKiX7nBVHQxNBgjSGMY3bYIfrHz1mFSg-3KXgHYFP4VU5MaYOm6CFlLR7DLMUGtTg_UNYssdZKYe--UBlL-d_g/s2308/vyos-ddos-dash.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="2204" data-original-width="2308" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6dTaxkenQ339gtD7oPRRqVNVBDS7dgdzVNL7igbLw320tTKC_5oQz7EWKtEQnfl7_PpolDnWZNO5o3zjC37_C8UsU_-p71Ur6I8ElnKiX7nBVHQxNBgjSGMY3bYIfrHz1mFSg-3KXgHYFP4VU5MaYOm6CFlLR7DLMUGtTg_UNYssdZKYe--UBlL-d_g/s600/vyos-ddos-dash.png" width="600" /></a></div>
<a href="https://blog.sflow.com/2020/02/real-time-ddos-mitigation-using-bgp.html">Real-time DDoS mitigation using BGP RTBH and FlowSpec</a> describes how to configure the DDoS protect application. The screen capture above shows the <i>Charts</i> page after a couple of simulated DDoS attacks on an address, 198.51.100.129, protected by the VyOS router. The charts show two <b>ip_flood</b> and a single <b>udp_amplification</b> attack - see <a href="https://blog.sflow.com/2022/03/ddos-attacks-and-bgp-flowspec-responses.html">DDoS attacks and BGP Flowspec responses</a> for information on simulating different types of DDoS attack to test mitigation responses.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjrvZphwwMZzKEDUaTEmcyVKDetl9haYwa1Q5IrD48pU4nXggVkKN93Q1NZsF9UATJLbtUAmrZL2gkA_dy13S-RyRUazh7i0wF-42zrJut4SyGfdQQqDSZTfYkpXKIZwXO_oR1FgkxDAPB08WeLQ_0RabYCl5n0jsFioKF5BbGZAlH0vRPXRYgg8eTtA/s2308/vyos-ddos-actions.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1200" data-original-width="2308" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjrvZphwwMZzKEDUaTEmcyVKDetl9haYwa1Q5IrD48pU4nXggVkKN93Q1NZsF9UATJLbtUAmrZL2gkA_dy13S-RyRUazh7i0wF-42zrJut4SyGfdQQqDSZTfYkpXKIZwXO_oR1FgkxDAPB08WeLQ_0RabYCl5n0jsFioKF5BbGZAlH0vRPXRYgg8eTtA/s600/vyos-ddos-actions.png" width="600" /></a></div>
The <i>Controls</i> page shows three active controls. The table shows the targeted address, administrative address group, attack type, protocol, detection time, mitigation action and status of each active DDoS attack.<pre>vyos@vyos:~$ show bgp ipv4
BGP table version is 0, local router ID is 192.168.1.1, vrf id 0
Default local pref 100, local AS 64500
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
198.51.100.129/32
192.0.2.1 0 65000 i
Displayed 1 routes and 1 total paths</pre>
The show command verifies that a Remotely Triggered Black Hole (RTBH) rule has been received for the <i>drop</i> mitigation actions. Advertising a black hole route risks collateral damage since it drops all traffic to the targetted host in order to protect network bandwidth and services provided by other hosts. <pre>vyos@vyos:~$ show bgp ipv4 flowspec detail
BGP flowspec entry: (flags 0x418)
Destination Address 198.51.100.129/32
IP Protocol = 17
Source Port = 53
FS:rate 0.000000
received for 00:00:12
not installed in PBR</pre>
The show command verifies that a Flowspec rule has been received for the <i>filter</i> mitigation action. Using Flowspec to filter traffic is more targetted than using black hole routes. In this case only UDP traffic (IP Protocol 17) with Source Port 53 (DNS) is dropped, all other services provided by the targetted host are still accessible.<pre>vyos@vyos:~$ show log container sflow-rt
2023-04-08T00:24:14Z INFO: Starting sFlow-RT 3.0-1681
2023-04-08T00:24:16Z INFO: Version check, running latest
2023-04-08T00:24:17Z INFO: Listening, BGP port 1179
2023-04-08T00:24:18Z INFO: Listening, sFlow port 6343
2023-04-08T00:24:19Z INFO: Listening, HTTP port 8008
2023-04-08T00:24:19Z INFO: DNS server 1.1.1.1
2023-04-08T00:24:19Z INFO: app/ddos-protect/scripts/ddos.js started
2023-04-08T00:24:19Z INFO: app/prometheus/scripts/export.js started
2023-04-08T00:24:19Z INFO: app/browse-drops/scripts/top.js started
2023-04-08T00:24:19Z INFO: app/browse-flows/scripts/top.js started
2023-04-08T00:26:11Z INFO: BGP open 192.168.1.1 51252
2023-04-08T14:37:36Z INFO: DDoS drop ip_flood 198.51.100.129 local 47
2023-04-08T14:38:19Z INFO: DDoS filter udp_amplification 198.51.100.129 local 53
2023-04-08T14:38:19Z INFO: DDoS drop ip_flood 198.51.100.129 local 17</pre>Attacks are recorded in the container log. <a href="https://blog.sflow.com/2020/04/monitoring-ddos-mitigation.html">Monitoring DDoS mitigation</a> describes how to use Prometheus / Elasticsearch / Grafana to monitor DDoS activity and build dashboards.<p>This is only a partial configuration. Peering sessions with upstream routers need to be configured to propagate controls so that DDoS attack traffic can be blocked before it saturates the upstream link. The limited scrubbing capacity of the VyOS software router isn't a factor since traffic will be dropped in hardware upstream. The flexibility of the VyOS router is an advantage in providing visibility and analytics to quickly trigger mitigation actions.</p>Peterhttp://www.blogger.com/profile/00856599914190257147noreply@blogger.com0tag:blogger.com,1999:blog-1978652979840829013.post-74888062037106679592023-04-04T07:11:00.003-07:002023-04-05T10:24:41.375-07:00Real-time flow analytics on VyOS<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDRBht-UM93aTJWKJ4ub6i3Op0147eFaXnfjENFdXvSlV5aiJKZCEZBK9TIdJ_bQvYWqb_6cLCQd0CFJ9e9SOXeReON95sgUBSah0rW_lkJzoJ5lItbUnTXn7MoDY7ChKmtlyGxoathwpdXL32qHRyimWvuZyZJ-2Tu1kVE0dmpRWDI7nGzZ5IB9I9lw/s1388/vyos-analytics.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="444" data-original-width="1388" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDRBht-UM93aTJWKJ4ub6i3Op0147eFaXnfjENFdXvSlV5aiJKZCEZBK9TIdJ_bQvYWqb_6cLCQd0CFJ9e9SOXeReON95sgUBSah0rW_lkJzoJ5lItbUnTXn7MoDY7ChKmtlyGxoathwpdXL32qHRyimWvuZyZJ-2Tu1kVE0dmpRWDI7nGzZ5IB9I9lw/s600/vyos-analytics.png" width="600" /></a></div><a href="https://blog.sflow.com/2023/03/vyos-with-host-sflow-agent.html">VyOS with Host sFlow agent</a> describes support for streaming sFlow telemetry added to the open source <a href="https://vyos.net/">VyOS router operating system</a>. This article describes how to install analytics software on a VyOS router by configuring a container.
<pre>vyos@vyos:~$ add container image sflow/ddos-protect</pre>
First, download the <a href="https://hub.docker.com/r/sflow/ddos-protect">sflow/ddos-protect</a> image.
<pre>vyos@vyos:~$ mkdir -m 777 /config/sflow-rt</pre>
Create a directory to store persistent container state.
<pre>set container name sflow-rt image sflow/ddos-protect
set container name sflow-rt allow-host-networks
set container name sflow-rt arguments '-Dhttp.hostname=10.0.0.240'
set container name sflow-rt environment RTMEM value 200M
set container name sflow-rt memory 0
set container name sflow-rt volume store source /config/sflow-rt
set container name sflow-rt volume store destination /sflow-rt/store</pre>Configure a container to run the image. The <i>RMEM</i> environment variable setting limits the amount of memory that the container will use to <i>200M</i> bytes. The <i>-Dhttp.hostname</i> argument sets the internal web server to listen on management address, <i>10.0.0.240</i>, assigned to <i>eth0</i> on this router. The container has is no built-in authentication, so access needs to be limited using an ACL or through a reverse proxy - see <a href="https://sflow-rt.com/download.php">Download and install</a>.<pre>set system sflow interface eth0
set system sflow interface eth1
set system sflow interface eth2
set system sflow polling 30
set system sflow sampling-rate 1000
set system sflow drop-monitor-limit 50
set system sflow server 127.0.0.1</pre>
Next, configure sFlow agent to send to localhost (127.0.0.1).
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmhoHgkhaGuIu05iyL779QAp5FIGHx5Bwy4X6A2I6SdFL_dDL7WCOm0tqa7aiVruoVBmYgDiu3ivA0SN0yTlF-dCtn328l_BFEpvnFowu343uj4AgNisSkzfZmxl4KWMdrZEGThmbTMexDyg3MS-1uEu7A656kPinE8dMsRe9a-NfzboyX_PODQMJBtA/s1810/vyos-sflow-rt.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1528" data-original-width="1810" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmhoHgkhaGuIu05iyL779QAp5FIGHx5Bwy4X6A2I6SdFL_dDL7WCOm0tqa7aiVruoVBmYgDiu3ivA0SN0yTlF-dCtn328l_BFEpvnFowu343uj4AgNisSkzfZmxl4KWMdrZEGThmbTMexDyg3MS-1uEu7A656kPinE8dMsRe9a-NfzboyX_PODQMJBtA/s600/vyos-sflow-rt.png" width="600" /></a></div>
Finally connect to the web interface on the router at port 8008. The status page verifies that the sFlow-RT analytics engine is receiving sFlow from 1 <i>sFlow Agent</i> (the VyOS router). See <a href="https://sflow-rt.com/intro.php">Getting started</a> for more information.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgEiC4XSmoBAhozxWBSszuFamgevOhuTf9RkCjF_QxIMhLAdq5XUGfsd3KbxpXpCN88FWPcu38VL8SyuK5irXwc0dBO9qig4tcgreRIbEHLjbDQzBFcu305VyThn_KkrMthYwmmfVGMWVlIoNjUVGnreEgAkjaDvCTsNZe4I3-2jd9OFRokPBePyN70w/s2282/vyos-flows.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1330" data-original-width="2282" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgEiC4XSmoBAhozxWBSszuFamgevOhuTf9RkCjF_QxIMhLAdq5XUGfsd3KbxpXpCN88FWPcu38VL8SyuK5irXwc0dBO9qig4tcgreRIbEHLjbDQzBFcu305VyThn_KkrMthYwmmfVGMWVlIoNjUVGnreEgAkjaDvCTsNZe4I3-2jd9OFRokPBePyN70w/s600/vyos-flows.png" width="600" /></a></div>
The included <a href="https://github.com/sflow-rt/browse-flows">Flow Browser</a> application provides an up to the second view traffic flows. <a href="https://sflow-rt.com/define_flow.php">Defining Flows</a> describes the fields that can be used to break out traffic.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh95iK7menK-YSASa-t_8l-QiYhkezDS-gk9yaIe0CxBDJMYFlrgyZ3CnTIzqn6uyFnQxq4KdFW1zPwZ1KS3OPH_fyHh-Rd1Bzr-StCbwejorg56ISEiCjI9TyfH6ojK8tsovlMAdYzXndyX5L2BkIh0M3oD2eU0ApIGkosHIsbpvsmJH4PniwBz0gb0Q/s2282/vyos-drops.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1318" data-original-width="2282" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh95iK7menK-YSASa-t_8l-QiYhkezDS-gk9yaIe0CxBDJMYFlrgyZ3CnTIzqn6uyFnQxq4KdFW1zPwZ1KS3OPH_fyHh-Rd1Bzr-StCbwejorg56ISEiCjI9TyfH6ojK8tsovlMAdYzXndyX5L2BkIh0M3oD2eU0ApIGkosHIsbpvsmJH4PniwBz0gb0Q/s600/vyos-drops.png" width="600" /></a></div><a href="https://blog.sflow.com/2023/03/vyos-dropped-packet-notifications.html">VyOS dropped packet notifications</a> describes how to configure and monitor sFlow dropped packet notifications. The included <a href="https://github.com/sflow-rt/browse-drops">Discard Browser</a> provides an up to the second view of dropped packets.<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1timiQLJ9V4wOgjfmwR4Z8p_5lUn-YiV4-2b7hlvFH47d8aWgti0krlQpjvZcAF0hUagddeS67sdsx2L5LEl1a5ZjMsDCp-DWm4FNsPL5-BKBehxoMcHTXc1UeE6n4k8oe6oPVNB5npFaTCojWUQJ9_UsufJXrVS66wSTtDQ5t8n0eCaho5VzJt9TIQ/s2380/vyos-metrics.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1690" data-original-width="2380" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1timiQLJ9V4wOgjfmwR4Z8p_5lUn-YiV4-2b7hlvFH47d8aWgti0krlQpjvZcAF0hUagddeS67sdsx2L5LEl1a5ZjMsDCp-DWm4FNsPL5-BKBehxoMcHTXc1UeE6n4k8oe6oPVNB5npFaTCojWUQJ9_UsufJXrVS66wSTtDQ5t8n0eCaho5VzJt9TIQ/s600/vyos-metrics.png" width="600" /></a></div>
The included <a href="https://github.com/sflow-rt/browse-metrics">Metric Browser</a> application lets you explore the metrics that are being streamed. The chart updates in real-time as data arrives and in this case shows CPU utilization on the VyOS router. The standard set of metrics exported by the Host sFlow agent include interface counters as well as host cpu, memory, network and disk performance metrics. <a href="https://sflow-rt.com/metrics.php">Metrics</a> lists the set of available metrics.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipopFgQoEvoHUCqbZ4HjCYCV86ARUiFk3KJLZ2mUHAfUaZmKbZwOuV4wpLiizvuIJOTYDwRZg8nQgl23aXEguUCyJ8KOcLYnv2keFxNzZH1DNeSSrgGDXAPyJrsv0-Ue1jsGCGesQuWten14XpHlpBoeg4Sfcs7BBrYTDjc8783Y8Qd8UxpBkXhPRIHA/s2048/grafana-traffic.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1378" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipopFgQoEvoHUCqbZ4HjCYCV86ARUiFk3KJLZ2mUHAfUaZmKbZwOuV4wpLiizvuIJOTYDwRZg8nQgl23aXEguUCyJ8KOcLYnv2keFxNzZH1DNeSSrgGDXAPyJrsv0-Ue1jsGCGesQuWten14XpHlpBoeg4Sfcs7BBrYTDjc8783Y8Qd8UxpBkXhPRIHA/s600/grafana-traffic.png" width="600" /></a></div><a href="https://blog.sflow.com/2019/10/flow-metrics-with-prometheus-and-grafana.html">Flow metrics with Prometheus and Grafana</a> describes how integrate flow analytics into operational dashboards. The included <a href="https://github.com/sflow-rt/prometheus">Prometheus</a> application exposes flow analytics in the standard Prometheus scrape format so that they can be logged in time series databases.<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0nf1GVXw0QcJKs7Dq0nSDboIx3JLGqEZuKI7ga8PrNKXl0xrrTajYXRmIA2Jl6EaooqrL8ji8tv1hpWTfhirMTos-s9hbwv5xUz2Z1kyZdX9Phe5-obWp5XB6oDQAlubc7MW4C13sjSKznGtAKvYIsyv_c7LCzHb24xg0kmaucdu0Oh4naCbMXy7Tqg/s2366/vyos-ddos.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="2196" data-original-width="2366" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0nf1GVXw0QcJKs7Dq0nSDboIx3JLGqEZuKI7ga8PrNKXl0xrrTajYXRmIA2Jl6EaooqrL8ji8tv1hpWTfhirMTos-s9hbwv5xUz2Z1kyZdX9Phe5-obWp5XB6oDQAlubc7MW4C13sjSKznGtAKvYIsyv_c7LCzHb24xg0kmaucdu0Oh4naCbMXy7Tqg/s600/vyos-ddos.png" width="600" /></a></div>
<a href="https://blog.sflow.com/2021/10/ddos-protection-quickstart-guide.html">DDoS protection quickstart guide</a> describes how to use real-time sFlow analytics with BGP Flowspec / RTBH to automatically mitigate DDoS attacks. The included <a href="https://github.com/sflow-rt/ddos-protect">DDoS Protect</a> application detects common volumetric attacks and can apply automated responses. The screen capture shows traffic associated with a series of simulated DDoS attacks against hosts behind the VyOS router, see <a href="https://blog.sflow.com/2022/03/ddos-attacks-and-bgp-flowspec-responses.html">DDoS attacks and BGP Flowspec responses</a>.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhB8qDBJHz9LxzzTYdWKIxUG7ku5nYJy9JrhEnPFk88qygzk6IZFRPfT-jeTIdrblQwVDYdGHsSbG5H4y8RgXFni5bQlpVNJMbzKsLyHNx5OtHRStGqs2Mv57tLYdlN-25_c9ZTconJS71wcqcriOyevBLxSeHVIUAqQnfYOAfHmxcRRwMNYd0Ct2pPlg/s2366/vyos-rt-rest.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="2196" data-original-width="2366" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhB8qDBJHz9LxzzTYdWKIxUG7ku5nYJy9JrhEnPFk88qygzk6IZFRPfT-jeTIdrblQwVDYdGHsSbG5H4y8RgXFni5bQlpVNJMbzKsLyHNx5OtHRStGqs2Mv57tLYdlN-25_c9ZTconJS71wcqcriOyevBLxSeHVIUAqQnfYOAfHmxcRRwMNYd0Ct2pPlg/s600/vyos-rt-rest.png" width="600" /></a></div>
The embedded sFlow-RT analytics engine exposes a REST API that can be used to program flow analytics, set thresholds, monitor events, and gather statistics. In addition, the applications shown in this article were all written using sFlow-RT's embedded scripting API. See <a href="https://sflow-rt.com/writing_applications.php">Writing Applications</a> for more information.Peterhttp://www.blogger.com/profile/00856599914190257147noreply@blogger.com0tag:blogger.com,1999:blog-1978652979840829013.post-67261030184241072542023-04-03T07:02:00.000-07:002023-04-03T07:02:12.914-07:00Dropped packet reason codes in VyOS<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCKzQ3NgNkd0F1EJ7Ss-vT34RrIxk0FKOjnuk06bxBQcXYxMsmX6kCFWIqBOleEjCSwen3TvPNVCzmzGxWi6SsAn1EAaLoWUan3RYOcS8xP9NU_aOIcmh3b6Wvky-lu5g1Wfc4__JxMqLtIia42ZxmDW7Py5Ro-PCOIRHgZp6bBbyj8Zh_tLsyicicjA/s2214/vyos-linux-reasons.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1308" data-original-width="2214" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCKzQ3NgNkd0F1EJ7Ss-vT34RrIxk0FKOjnuk06bxBQcXYxMsmX6kCFWIqBOleEjCSwen3TvPNVCzmzGxWi6SsAn1EAaLoWUan3RYOcS8xP9NU_aOIcmh3b6Wvky-lu5g1Wfc4__JxMqLtIia42ZxmDW7Py5Ro-PCOIRHgZp6bBbyj8Zh_tLsyicicjA/s600/vyos-linux-reasons.png" width="600" /></a></div>
The article <a href="https://blog.sflow.com/2023/03/vyos-with-host-sflow-agent.html">VyOS with Host sFlow agent</a> describes how to use industry standard <a href="https://sflow.org">sFlow</a> telemetry to monitor network traffic flows and statistics in the latest VyOS rolling releases. <a href="https://blog.sflow.com/2023/03/vyos-dropped-packet-notifications.html">VyOS dropped packet notifications</a> describes how sFlow also provides visibility into network packet drops and <a href="https://blog.sflow.com/2023/03/dropped-packet-reason-codes-in-linux-6.html">Dropped packet reason codes in Linux 6+ kernels</a> describes how newer kernels are able to provide specific reasons for dropping packets. <pre>vyos@vyos:~$ uname -r
6.1.22-amd64-vyos</pre>
<p>The latest VyOS rolling release runs on a Linux 6.1 kernel and the latest release of VyOS now provides enhanced visibility into dropped packets using kernel reason codes.</p>
<pre>vyos@vyos:~$ show version
Version: VyOS 1.4-rolling-202303310716
Release train: current
Built by: autobuild@vyos.net
Built on: Fri 31 Mar 2023 07:16 UTC
Build UUID: 1a7448d9-d53c-48a0-8644-ed1970c1abb8
Build commit ID: 75c9311fba375e
Architecture: x86_64
Boot via: installed image
System type: guest
Hardware vendor: innotek GmbH
Hardware model: VirtualBox
Hardware S/N: 0
Hardware UUID: da75808d-ff60-1d4c-babd-84a7fa341053
Copyright: VyOS maintainers and contributors</pre>
Verify that the version of of VyOS is VyOS 1.4-rolling-202303310716 or later.
<p>In the previous article, <a href="https://blog.sflow.com/2023/03/vyos-dropped-packet-notifications.html">VyOS dropped packet notifications</a>, two tests were performed, the first a failed attempt to connect to the VyOS router using telnet (telnet has been disabled in the router config), and the second a traceroute test between two hosts connected to the router. The sFlow drop reason codes reported for these two tests were <i>unknown_l4</i> and <i>unknown_l3</i> respectively. The Linux kernel functional names weren't much more specific, <i>tcp_v4_rcv</i> and <i>ip_forward</i> respectively. However, in this case, the Linux 6.1 kernel instrumentation allows more specific sFlow drop reasons to be reported, as shown in the chart at the top of this article.</p>
<ul>
<li><b>port_unreachable</b> This sFlow drop reason code is defined by reference to <a href="https://www.rfc-editor.org/rfc/rfc1812#section-5.2.7.1">RFC 1812 section 5.2.7.1</a> and is defined as <i>"Port Unreachable - generated if the designated transport protocol (e.g., UDP) is unable to demultiplex the datagram in the transport layer of the final destination but has no protocol mechanism to inform the sender"</i></li>
<li><b>ip_1_parsing</b> This sFlow drop reason code is defined by reference to <a href="https://www.kernel.org/doc/html/latest/networking/devlink/devlink-trap.html">Devlink Trap</a> and is defined as <i>"Traps packets dropped due to an error in the first IP header parsing. This packet trap could include packets which do not pass an IP checksum check, a header length check (a minimum of 20 bytes), which might suffer from packet truncation thus the total length field exceeds the received packet length etc."</i></li>
</ul>
The detailed reasons make it easier to identify the root causes of packet drops, particularly when combined with information from the dropped packet's header that is also included in the <a href="https://sflow.org/sflow_drops.txt">sFlow Dropped Packet Notification</a> messages.Peterhttp://www.blogger.com/profile/00856599914190257147noreply@blogger.com0tag:blogger.com,1999:blog-1978652979840829013.post-15525075550782765492023-03-30T07:07:00.000-07:002023-03-30T07:07:34.368-07:00Dropped packet reason codes in Linux 6+ kernels<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxFEzstH21O86-Z1pP_gSvhfG1IWwj2edrvz-xkBkMXQyteBO4VqoqHYCitXk1PD7c8SN-brFtV17d5SdTgbTRoBlX_4NKKz7OGU9KWNH9eW1fgV3BYTwuNvrciC4AHNdf-l89bPVMBxdrH1igPUV6XpT0kS6ccozi_ISPyIFYEP8lBpccXDnWqx2rbw/s2240/linux_drop_reason.png" style="display: block; padding: 1em 0px; text-align: center;"><img border="0" data-original-height="1314" data-original-width="2240" height="375" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxFEzstH21O86-Z1pP_gSvhfG1IWwj2edrvz-xkBkMXQyteBO4VqoqHYCitXk1PD7c8SN-brFtV17d5SdTgbTRoBlX_4NKKz7OGU9KWNH9eW1fgV3BYTwuNvrciC4AHNdf-l89bPVMBxdrH1igPUV6XpT0kS6ccozi_ISPyIFYEP8lBpccXDnWqx2rbw/w640-h375/linux_drop_reason.png" width="640" /></a></div><a href="https://blog.sflow.com/2020/07/using-sflow-to-monitor-dropped-packets.html">Using sFlow to monitor dropped packets</a> describes support for standard <a href="https://sflow.org/sflow_drops.txt">sFlow Dropped Packet Notications</a> in the open source <a href="https://sflow.net">Host sFlow</a> agent. This article describes additional capabilities in Linux 6+ kernels that clarify reasons why packets are dropped in the kernel.<p>The recent addition of <a href="https://github.com/torvalds/linux/blob/master/include/net/dropreason.h">dropreason.h</a> in Linux 6+ kernels provides detailed reasons for packet drops. The netlink drop_monitor API has been extended to include the <span style="font-family: courier;">NET_DM_ATTR_REASON</span> attribute to report the drop reason, see <a href="https://github.com/torvalds/linux/blob/master/include/uapi/linux/net_dropmon.h#L71-L100">net_dropmon.h</a>.</p><p>The following example illustrates the value of the reason code in explaining Linux packet drops.</p>
<pre>tcp_v4_rcv+0x7c/0xef0</pre>
The value of <span style="font-family: courier;">NET_DM_ATTR_SYMBOL</span> shown above indicates that the packet was dropped in the <span style="font-family: courier;">tcp_v4_rcv</span> function in Linux kernel at memory location <span style="font-family: courier;">0x7c/0xef0</span>. While this information is helpful, there are many reasons why a TCP packet may be dropped.
<pre>NO_SOCKET</pre>
In this case, the value of <span style="font-family: courier;">NET_DM_ATTR_REASON</span> shown above indicates that the TCP packet was dropped because no application had opened a socket and so there was nowhere to deliver the packet.
<blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;">In the case of <a href="https://blog.sflow.com/2021/05/linux-as-network-operating-system.html">Linux-based hardware switches</a> or <a href="https://blog.sflow.com/2022/08/nvidia-connectx-smartnics.html">smart network adapters</a>, where packet processing is offloaded to hardware, the netlink drop_monitor events include <span style="font-family: courier;">NET_DM_ATTR_HW_TRAP_GROUP_NAME</span> and <span style="font-family: courier;">NET_DM_ATTR_HW_TRAP_NAME</span> attributes and packet header information supplied by the hardware driver, see <a href="https://www.kernel.org/doc/html/latest/networking/devlink/devlink-trap.html">Devlink Trap</a>.</p></blockquote>
<p>The latest version of the open source <a href="https://sflow.net">Host sFlow</a> agent includes adds support for the <span style="font-family: courier;">NET_DM_ATTR_REASON</span> attribute to improve the accuracy of the sFlow drop_reason.</p>
<pre>port_unreachable</pre>
In our example, the Host sFlow is now able to report <span style="font-family: courier;">port_unreachable</span> as the reason for the dropped packet, rather than a generic <span style="font-family: courier;">unknown_l4</span> reason reported for older kernels.
<p>The screen capture at the top of this article shows dropped packet information displayed in real-time using the <a href="https://github.com/sflow-rt/browse-drops">Discard Browser</a> application running on the <a href="https://sflow-rt.com">sFlow-RT</a> analytics engine. The chart demonstrates how the combination of information from the header of the dropped packet along with the reason for dropping the packet quickly gets to the root cause of the packet drop. In this case an attempt has been made from <i>172.16.1.174</i> to connect to <i>172.16.1.1</i> via <a href="https://en.wikipedia.org/wiki/Telnet">telnet</a> (tcp port 23) and telnet has not been enabled on the server so the packet was dropped - as it should be since telnet is not a secure method of connecting.</p>
<pre>docker run --name sflow-rt -p 8008:8008 -p 6343:6343/udp -d sflow/prometheus</pre>
<p>A quick way to experiment with sFlow is to run the pre-built <a href="https://hub.docker.com/r/sflow/prometheus">sflow/prometheus</a> image using Docker. The bundled Discard Browser with the settings shown in the screen capture can be launched by <a href="http://localhost:8008/app/browse-drops/html/index.html?keys=ipsource%2Cipdestination%2Ctcpdestinationport%2Cfunction_full%2Clinux_drop_reason%2Creason&value=fps">clicking here</a>.</p>Peterhttp://www.blogger.com/profile/00856599914190257147noreply@blogger.com0tag:blogger.com,1999:blog-1978652979840829013.post-17404028435000842562023-03-27T07:03:00.000-07:002023-03-27T07:03:49.960-07:00VyOS dropped packet notifications<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhb0r_TBCYfQRdvhzrZ8CvdQgisbFxcn2tnty_51PyQSaBzmBwwIHAIf-VbBllyTc-F4LESQrIXAHfqAi_IUoIk1l70EjZrMDXQ4xt1uoqxqLYdf29JkXyHrWdaAkTi0Oa_gPx7kcLaU1WiVV3cUfRyjeZv3syOxIT0qLTEirRaBEg_iGyzmvxhFjkyMg/s2236/vyos-dropmon.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1318" data-original-width="2236" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhb0r_TBCYfQRdvhzrZ8CvdQgisbFxcn2tnty_51PyQSaBzmBwwIHAIf-VbBllyTc-F4LESQrIXAHfqAi_IUoIk1l70EjZrMDXQ4xt1uoqxqLYdf29JkXyHrWdaAkTi0Oa_gPx7kcLaU1WiVV3cUfRyjeZv3syOxIT0qLTEirRaBEg_iGyzmvxhFjkyMg/s600/vyos-dropmon.png" width="600" /></a></div>
<a href="https://blog.sflow.com/2023/03/vyos-with-host-sflow-agent.html">VyOS with Host sFlow agent</a> describes how to configure and analyze industry standard <a href="https://sflow.org">sFlow</a> telemetry recently added to the VyOS open source router platform. This article discusses sFlow <a href="https://sflow.org/sflow_drops.txt">dropped packet notifications</a> support added to the latest release.
<p>Dropped packets have a profound impact on network performance and availability. Packet discards due to congestion can significantly impact application performance. Dropped packets due to black hole routes, expired TTLs, MTU mismatches, etc. can result in insidious connection failures that are time consuming and difficult to diagnose. Visibility into dropped packets offers significant benefits for network troubleshooting, providing real-time network-wide visibility into the specific packets that were dropped as well the reason the packet was dropped. This visibility instantly reveals the root cause of drops and the impacted connections.</p>
<pre>vyos@vyos:~$ show version
Version: VyOS 1.4-rolling-202303260914
Release train: current
Built by: autobuild@vyos.net
Built on: Sun 26 Mar 2023 09:14 UTC
Build UUID: 72b34f74-bfcd-4b51-9b95-544319c2dac5
Build commit ID: d68bda6a295ba9
Architecture: x86_64
Boot via: installed image
System type: guest
Hardware vendor: innotek GmbH
Hardware model: VirtualBox
Hardware S/N: 0
Hardware UUID: df0a2b79-b8c4-8342-a27f-76aa3e52ad6d
Copyright: VyOS maintainers and contributors</pre>
<p>Verify that the version of of VyOS is VyOS 1.4-rolling-202303260914 or later.</p>
On VyOS dropped packet monitoring relies on instrumentation built into recent Linux kernels and exposed through the netlink drop_monitor API. Enabling drop_monitor in VyOS kernel configuration allows the Host sFlow agent to capture and export information on dropped packets.
<pre>set system sflow interface eth0
set system sflow interface eth1
set system sflow interface eth2
set system sflow polling 30
set system sflow sampling-rate 1000
set system sflow drop-monitor-limit 50
set system sflow server 10.0.0.30 port 6343</pre>
The <i>drop-monitor-limit</i> configuration entry enables dropped packet monitoring and sets a rate limit of 50 dropped packets notifications per second.
<pre>docker run --name sflow-rt -p 8008:8008 -p 6343:6343/udp -d sflow/prometheus</pre>
<p>A quick way to experiment with sFlow is to run the pre-built <a href="https://hub.docker.com/r/sflow/prometheus">sflow/prometheus</a> image using Docker on the sFlow server (in this case on 10.0.0.30). The chart at the top of the page uses the <i>Discard Browser</i> application to display an up to the second view of packets dropped by the VyOS router, click on <a href="http://localhost:8008/app/browse-drops/html/index.html?keys=ipsource%2Cipdestination%2Cor%3Atcpdestinationport%3Audpdestinationport%2Cstack%2Cipttl%2Creason%2Cfunction&value=fps&filter=isbroadcast!%3Dtrue">this link</a> to open the application with the settings shown.</p><p>The chart shows the results of two tests, the first a failed attempt to connect to the VyOS router using telnet (telnet has been disabled in the router config), and the second a traceroute test between two hosts connected to the router. The <i>reason</i> field reports the sFlow drop reason code and the <i>function</i> reports the linux kernel function that dropped the packet. With the telnet test, the packet was dropped in the <i>tcp_v4_rcv</i> function and is reported as an <i>unknown_l4</i> sFlow reason. In the case of the traceroute test, 3 packets were dropped in the <i>ip_forward</i> function and are reported as <i>unknown_l3</i> reason.</p>
<p>Enabling sFlow dropped packet notifications on all switches, routers, and hosts provides end-to-end visibility into dropped packets, rapidly identifying the location and reason for packet drops as well as identifying the impacted services.</p>
<p>Dropped packet monitoring complements sFlow's existing counter polling and packet sampling mechanisms and shares a common data model so that all three sources of data can be correlated. For example, if packets are being discarded because of buffer exhaustion, the discard records don't necessarily tell the whole story. The discarded packets may represent mice flows that are victims of an elephant flow. Packet samples will reveal the traffic that isn't being dropped and provide a more complete picture. Counter data adds additional information such as CPU load, interface speed, link utilization, packet and discard rates that further completes the picture.</p>Peterhttp://www.blogger.com/profile/00856599914190257147noreply@blogger.com0tag:blogger.com,1999:blog-1978652979840829013.post-21366210132904993822023-03-17T12:25:00.001-07:002023-03-17T12:25:39.079-07:00VyOS with Host sFlow agent<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhc4VZje3pBLICBEMrwtdOGLfSSs0KAfFQg03eKnfnztJme_2P5kcNApdW8ae3hTyv86WWxmtsqZNFqx9XCRMrIKeNieht6DLt10BfkE2JowK57zvYkhfFxV86CqDjbN9NIv4jMWxoITPCT87ZAIu279bcr62mQcu6KvTiQVmZJWvvRuWgGGC7rgkGrmQ/s2934/vyos-browse-flows.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1318" data-original-width="2934" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhc4VZje3pBLICBEMrwtdOGLfSSs0KAfFQg03eKnfnztJme_2P5kcNApdW8ae3hTyv86WWxmtsqZNFqx9XCRMrIKeNieht6DLt10BfkE2JowK57zvYkhfFxV86CqDjbN9NIv4jMWxoITPCT87ZAIu279bcr62mQcu6KvTiQVmZJWvvRuWgGGC7rgkGrmQ/s600/vyos-browse-flows.png" width="600" /></a></div><a href="https://blog.sflow.com/2023/03/vyos.html">VyOS</a> described deficiencies with the embedded sFlow implementation in the open source <a href="https://vyos.net/">VyOS router operating system</a> and suggested that the open source <a href="https://sflow.net">Host sFlow</a> agent be installed as an alternative. The VyOS developer community embraced the suggestion and has been incredibly responsive, integrating, and releasing a version of VyOS with Host sFlow support within a week.<pre>vyos@vyos:~$ show version
Version: VyOS 1.4-rolling-202303170317
Release train: current
Built by: autobuild@vyos.net
Built on: Fri 17 Mar 2023 03:17 UTC
Build UUID: 45391302-1240-4cc7-95a8-da8ee6390765
Build commit ID: e887f582cfd7de
Architecture: x86_64
Boot via: installed image
System type: guest
Hardware vendor: innotek GmbH
Hardware model: VirtualBox
Hardware S/N: 0
Hardware UUID: 871dd0f0-c4ec-f147-b1a7-ed536511f141
Copyright: VyOS maintainers and contributors</pre>
Verify that the version of of VyOS is <i>VyOS 1.4-rolling-202303170317</i> or later
<pre>set system sflow interface eth0
set system sflow interface eth1
set system sflow interface eth2
set system sflow polling 30
set system sflow sampling-rate 1000
set system sflow server 10.0.0.30 port 6343</pre>
The above commands configure sFlow export in the VyOS CLI using the embedded Host sFlow agent.
<pre>docker run --name sflow-rt -p 8008:8008 -p 6343:6343/udp -d sflow/prometheus</pre>A quick way to experiment with sFlow is to run the pre-built <a href="https://hub.docker.com/r/sflow/prometheus">sflow/prometheus</a> image using Docker on the sFlow server (in this case on 10.0.0.30). The chart at the top of the page uses the <i>Flow Browser</i> application to display an up to the second view of the largest tcp flows through the VyOS router, click on <a href="http://localhost:8008/app/browse-flows/html/index.html?keys=ipsource%2Cipdestination%2Ctcpsourceport%2Ctcpdestinationport&value=bps">this link</a> to open the application with the settings shown.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipopFgQoEvoHUCqbZ4HjCYCV86ARUiFk3KJLZ2mUHAfUaZmKbZwOuV4wpLiizvuIJOTYDwRZg8nQgl23aXEguUCyJ8KOcLYnv2keFxNzZH1DNeSSrgGDXAPyJrsv0-Ue1jsGCGesQuWten14XpHlpBoeg4Sfcs7BBrYTDjc8783Y8Qd8UxpBkXhPRIHA/s2048/grafana-traffic.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1378" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipopFgQoEvoHUCqbZ4HjCYCV86ARUiFk3KJLZ2mUHAfUaZmKbZwOuV4wpLiizvuIJOTYDwRZg8nQgl23aXEguUCyJ8KOcLYnv2keFxNzZH1DNeSSrgGDXAPyJrsv0-Ue1jsGCGesQuWten14XpHlpBoeg4Sfcs7BBrYTDjc8783Y8Qd8UxpBkXhPRIHA/s600/grafana-traffic.png" width="600" /></a></div>
<a href="https://blog.sflow.com/2019/10/flow-metrics-with-prometheus-and-grafana.html">Flow metrics with Prometheus and Grafana</a> describes how integrate flow analytics into operational dashboards.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgllQZgzXrTtb_AsDN-sg5CMVdCpCAuwMYRSo2N-VNXtMJdeyRkfSUwl8zLlFyJFCKZisMgW3yzZVnUI35jKFJQFboX25z_qa6tMd-W0lgR_BnwiIlpmfViHhan147b2tTOw0WXxCWvMtMpjoXRHfinuY7GtrmRAkTXo0z-l1PbFNb1uH2vM4Xy-hex3w/s1682/monitoring-ddos.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="354" data-original-width="1682" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgllQZgzXrTtb_AsDN-sg5CMVdCpCAuwMYRSo2N-VNXtMJdeyRkfSUwl8zLlFyJFCKZisMgW3yzZVnUI35jKFJQFboX25z_qa6tMd-W0lgR_BnwiIlpmfViHhan147b2tTOw0WXxCWvMtMpjoXRHfinuY7GtrmRAkTXo0z-l1PbFNb1uH2vM4Xy-hex3w/s600/monitoring-ddos.png" width="600" /></a></div><a href="https://blog.sflow.com/2021/10/ddos-protection-quickstart-guide.html">DDoS protection quickstart guide</a> describes how to use real-time sFlow analytics with BGP Flowspec / RTBH to automatically mitigate DDoS attacks.Peterhttp://www.blogger.com/profile/00856599914190257147noreply@blogger.com0tag:blogger.com,1999:blog-1978652979840829013.post-6634165149379767162023-03-11T12:52:00.005-08:002023-03-17T12:18:56.009-07:00VyOS<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOP0X8I37Jwp8iR1t6xFgGtqvYNGwv7513Yzn1b0bC8gZLqWQ1CaRm4TInqMlBRssvqrwT1fwMqmNzMOPhJhHl5uqeRo3FqliSOfZonaEg8lDuFLJZgMzVOkKxGEr6Tyep6S2T0fHUWuNJCzxswwx5aQLGI2Ybf5KyNyhV2Fu4UFfCdzQ3yxqI_NcseA/s466/vyatta-diagram.png" style="display: block; padding: 1em 0px; text-align: center;"><img border="0" data-original-height="416" data-original-width="466" height="286" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOP0X8I37Jwp8iR1t6xFgGtqvYNGwv7513Yzn1b0bC8gZLqWQ1CaRm4TInqMlBRssvqrwT1fwMqmNzMOPhJhHl5uqeRo3FqliSOfZonaEg8lDuFLJZgMzVOkKxGEr6Tyep6S2T0fHUWuNJCzxswwx5aQLGI2Ybf5KyNyhV2Fu4UFfCdzQ3yxqI_NcseA/w320-h286/vyatta-diagram.png" width="320" /></a></div><a href="https://vyos.net/">VyOS</a> is an open source router operating system based on Linux. This article discusses how to improve network traffic visibility on VyOS based routers using the open source <a href="https://sflow.net">Host sFlow</a> agent.
<p>VyOS claims <a href="https://sflow.org">sFlow</a> support, so why is it necessary to install an alternative sFlow agent? The following experiment demonstrates that there are significant issues with the VyOS sFlow implementation.
</p><pre>vyos@vyos:~$ show version
Version: VyOS 1.4-rolling-202301260317
Release train: current
Built by: autobuild@vyos.net
Built on: Thu 26 Jan 2023 03:17 UTC
Build UUID: a95385b7-12f9-438d-b49c-b91f47ea7ab7
Build commit ID: d5ea780295ef8e
Architecture: x86_64
Boot via: installed image
System type: KVM guest
Hardware vendor: innotek GmbH
Hardware model: VirtualBox
Hardware S/N: 0
Hardware UUID: 6988d219-49a6-0a4a-9413-756b0395a73d
Copyright: VyOS maintainers and contributors</pre>
Install a recent version of VyOS under <a href="https://www.virtualbox.org/">VirtualBox</a> and configure routing between two Linux virtual machines connected to <i>eth1</i> and <i>eth2</i> on the router. Out of band management is configured on <i>eth0</i>.<pre>set system flow-accounting disable-imt
set system flow-accounting sflow agent-address 10.0.0.50
set system flow-accounting sflow sampling-rate 1000
set system flow-accounting sflow server 10.0.0.30 port 6343
set system flow-accounting interface eth0
set system flow-accounting interface eth1
set system flow-accounting interface eth2</pre>
The above commands configure sFlow monitoring on VyOS using the native sFlow agent.<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_gP74S6swPqeZjIaLrDcsIILUhWZg7aXXSbUgJZaAJFY6itYXuKGXBWcxLfJ_w2kRfyhHECUf6TNKGP5GpHZYlUiMWKBcVE_wXEiWAo204nJ96X_omu31LM-lgXR8SAQTsVPt7Tk98U9pRQKCBbHOAXf9NDgH2_IO40qSiFZv9DrAoSUHWym8iBbXcw/s2892/vyatta-test.png" style="display: block; padding: 1em 0px; text-align: center;"><img border="0" data-original-height="2892" data-original-width="1838" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_gP74S6swPqeZjIaLrDcsIILUhWZg7aXXSbUgJZaAJFY6itYXuKGXBWcxLfJ_w2kRfyhHECUf6TNKGP5GpHZYlUiMWKBcVE_wXEiWAo204nJ96X_omu31LM-lgXR8SAQTsVPt7Tk98U9pRQKCBbHOAXf9NDgH2_IO40qSiFZv9DrAoSUHWym8iBbXcw/w406-h640/vyatta-test.png" width="406" /></a></div>
The <a href="https://hub.docker.com/r/sflow/sflow-test">sflow/sflow-test</a> tool is used to test the sFlow implementation while generating traffic consisting of a series of <a href="https://github.com/esnet/iperf">iperf3</a> tests (each generating approximately 50Mbps). The test fails in a number of significant ways:<div><ol style="text-align: left;"><li>The implementation of sFlow is incomplete, omitting required interface counter export</li><li>The peak traffic reported (3Mbps) is a fraction of the traffic generated by iperf3</li><li>There is an inconsistency in the packet size reported in the sFlow messages</li><li>Tests comparing counters and flow data fail because of missing counter export (1)</li></ol><div>Fortunately, VyOS is a Linux based operating system, so we can install the <a href="https://sflow.net/">Host sFlow</a> agent as an alternative to the native sFlow implementation to provide traffic visibility.</div>
<pre>delete system flow-accounting</pre>
First, disable the native VyOS sFlow agent.<pre>wget https://github.com/sflow/host-sflow/releases/download/v2.0.38-1/hsflowd-ubuntu20_2.0.38-1_amd64.deb
sudo dpkg -i hsflowd-ubuntu20_2.0.38-1_amd64.deb</pre>
Next, download and install the Host sFlow agent by typing the above commands in VyOS shell.<pre># hsflowd configuration file
# http://sflow.net/host-sflow-linux-config.php
sflow {
collector { ip=10.0.0.30 }
pcap { dev = eth0 }
pcap { dev = eth1 }
pcap { dev = eth2 }
}</pre>
Edit the <b>/etc/hsflowd.conf</b> file.
<pre>systemctl restart hsflowd</pre>
Restart the sFlow agent to pick up the new configuration.<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhdoC30GydHkTS2m4umPIrzrBevy0vNtgTGBjjhkUsM-dVdvJGZfPhlFpsT-jr9fozy8UAaooG2ALDyllgYAIKO94BAyztDh0tHS-S4n_pHZrFBLZXjFzunm6B-7p9hOZlnonIzELr61Ea5imyG5Qprmb6FJKaR8NPW5HbypVIipVZEe7aUQI0WPTCQNg/s2892/vyatta-hsflowd-test.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="2892" data-original-width="1838" height="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhdoC30GydHkTS2m4umPIrzrBevy0vNtgTGBjjhkUsM-dVdvJGZfPhlFpsT-jr9fozy8UAaooG2ALDyllgYAIKO94BAyztDh0tHS-S4n_pHZrFBLZXjFzunm6B-7p9hOZlnonIzELr61Ea5imyG5Qprmb6FJKaR8NPW5HbypVIipVZEe7aUQI0WPTCQNg/s600/vyatta-hsflowd-test.png" /></a></div>Rerunnig <i>sflow-test</i> shows that the implementation now passes. The peaks shown in the trend graph are consistent with the traffic generated by <i>iperf3</i> and with traffic levels reported in interface counters.<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhIu9NDzbMvUOBRLI5hZcBvb3C9l_ggrvgVhfUcxNNZ2BE2c620-eXzadKgSRz08SKxwff9bcTn8x_DcI2PSZxXh9XyfBcxVV9JdEZn6dfB98Wpzopa5myKKfu240PEWcJcg7dl1By1omBl7fyoSltxMphMF4S6Pcr5DKHuMsMCzGPJmzhONiWUTNdl-g/s2210/vyatta-browse-flows.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1310" data-original-width="2210" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhIu9NDzbMvUOBRLI5hZcBvb3C9l_ggrvgVhfUcxNNZ2BE2c620-eXzadKgSRz08SKxwff9bcTn8x_DcI2PSZxXh9XyfBcxVV9JdEZn6dfB98Wpzopa5myKKfu240PEWcJcg7dl1By1omBl7fyoSltxMphMF4S6Pcr5DKHuMsMCzGPJmzhONiWUTNdl-g/s600/vyatta-browse-flows.png" width="600" /></a></div></div>
The <i>sflow/sflow-test</i> Docker image also includes the <i>Flow Browser</i> application that can be used to monitor traffic flows in real-time. The screen shot above shows traffic from a single <i>iperf3</i> test.<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhn2dmcRAfMj2mDO6ng3v_hUIg9iukCAK3Ib-hE1qYyWoEOv3lL30Nt_1lhTqwB1cAw0QtBwXWX6NbBOZRvGeEULRW6WO6NbvarG4dJ7oC2iAyfkE9KsBLKTAoXdNjQCfgTgAQepKqZHjPE5TfJKQLePIBlNMNMfhqbfOnji2pv-Lx7zmykYsFQ5LhhUQ/s2380/vyatta-cpu.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1696" data-original-width="2380" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhn2dmcRAfMj2mDO6ng3v_hUIg9iukCAK3Ib-hE1qYyWoEOv3lL30Nt_1lhTqwB1cAw0QtBwXWX6NbBOZRvGeEULRW6WO6NbvarG4dJ7oC2iAyfkE9KsBLKTAoXdNjQCfgTgAQepKqZHjPE5TfJKQLePIBlNMNMfhqbfOnji2pv-Lx7zmykYsFQ5LhhUQ/s600/vyatta-cpu.png" width="600" /></a></div>
The <i>sflow/sflow-test</i> Docker image also includes the <i>Metric Browser</i> application that can be used to monitor counters in real-time. The screen shot above shows <i>cpu_utilization</i>.
<p>The <i>sFlow Test</i>, <i>Browse Flows</i> and <i>Browse Metrics</i> applications run on the <a href="https://sflow-rt.com">sFlow-RT</a> analytics engine. Additional examples include <a href="https://blog.sflow.com/2019/10/flow-metrics-with-prometheus-and-grafana.html">Flow metrics with Prometheus and Grafana</a> and <a href="https://blog.sflow.com/2021/10/ddos-protection-quickstart-guide.html">DDoS protection quickstart guide</a>.</p>Peterhttp://www.blogger.com/profile/00856599914190257147noreply@blogger.com0tag:blogger.com,1999:blog-1978652979840829013.post-89849031250047294792023-02-14T09:33:00.000-08:002023-02-14T09:33:15.825-08:00Real-time flow analytics with Containerlab templates<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_2Oqx4iBE9Ve0ZwXvFPapKROutAiwRHnpA7ypcaZYRLJmXBkUIBVCQg62xfrEeAqzizzLIIms5RrNMD1u3rXMQsp4tk5x0gIxtRcx571bXhJxuf1uoppbytqNmYxH7YkGJZGgYcB7dPC11ILUiRECA4I7qV0EO384WPemLzlmNP2VuKqBDQKkBkShoQ/s600/clos3.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="347" data-original-width="600" height="370" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_2Oqx4iBE9Ve0ZwXvFPapKROutAiwRHnpA7ypcaZYRLJmXBkUIBVCQg62xfrEeAqzizzLIIms5RrNMD1u3rXMQsp4tk5x0gIxtRcx571bXhJxuf1uoppbytqNmYxH7YkGJZGgYcB7dPC11ILUiRECA4I7qV0EO384WPemLzlmNP2VuKqBDQKkBkShoQ/w640-h370/clos3.png" width="640" /></a></div>
The GitHub <a href="https://github.com/sflow-rt/containerlab">sflow-rt/containerlab</a> project contains example network topologies for the <a href="https://containerlab.dev/">Containerlab</a> network emulation tool that demonstrate real-time streaming telemetry in realistic data center topologies and network configurations. The examples use the same <a href="https://frrouting.org/">FRRouting (FRR)</a> engine that is part of <a href="https://sonic-net.github.io/SONiC/">SONiC</a>, <a href="https://www.nvidia.com/en-us/networking/ethernet-switching/cumulus-linux/">NVIDIA Cumulus Linux</a>, and <a href="https://dent.dev/">DENT</a>. Containerlab can be used to experiment before deploying solutions into production. Examples include: tracing ECMP flows in leaf and spine topologies, EVPN visibility, and automated DDoS mitigation using BGP Flowspec and RTBH controls.
<p>This article describes an experiment with Containerlab's advanced <a href="https://containerlab.dev/manual/topo-def-file/#generated-topologies">Generated topologies</a> capability, taking the 3 stage Clos topology shown above and creating a template that can be used to generate topologies with any number of leaf and spine switches.
</p>
<p>The <a href="https://github.com/sflow-rt/containerlab/blob/master/clos3.yml">clos3.yml</a> topology file specifies the 2 leaf 2 spine topology shown above:</p>
<pre>name: clos3
mgmt:
network: fixedips
ipv4_subnet: 172.100.100.0/24
ipv6_subnet: 2001:172:100:100::/80
topology:
defaults:
env:
COLLECTOR: 172.100.100.8
nodes:
leaf1:
kind: linux
image: sflow/clab-frr
mgmt_ipv4: 172.100.100.2
mgmt_ipv6: 2001:172:100:100::2
env:
LOCAL_AS: 65001
NEIGHBORS: eth1 eth2
HOSTPORT: eth3
HOSTNET: "172.16.1.1/24"
HOSTNET6: "2001:172:16:1::1/64"
exec:
- touch /tmp/initialized
leaf2:
kind: linux
image: sflow/clab-frr
mgmt_ipv4: 172.100.100.3
mgmt_ipv6: 2001:172:100:100::3
env:
LOCAL_AS: 65002
NEIGHBORS: eth1 eth2
HOSTPORT: eth3
HOSTNET: "172.16.2.1/24"
HOSTNET6: "2001:172:16:2::1/64"
exec:
- touch /tmp/initialized
spine1:
kind: linux
image: sflow/clab-frr
mgmt_ipv4: 172.100.100.4
mgmt_ipv6: 2001:172:100:100::4
env:
LOCAL_AS: 65003
NEIGHBORS: eth1 eth2
exec:
- touch /tmp/initialized
spine2:
kind: linux
image: sflow/clab-frr
mgmt_ipv4: 172.100.100.5
mgmt_ipv6: 2001:172:100:100::5
env:
LOCAL_AS: 65003
NEIGHBORS: eth1 eth2
exec:
- touch /tmp/initialized
h1:
kind: linux
image: sflow/clab-iperf3
mgmt_ipv4: 172.100.100.6
mgmt_ipv6: 2001:172:100:100::6
exec:
- ip addr add 172.16.1.2/24 dev eth1
- ip route add 172.16.2.0/24 via 172.16.1.1
- ip addr add 2001:172:16:1::2/64 dev eth1
- ip route add 2001:172:16:2::/64 via 2001:172:16:1::1
h2:
kind: linux
image: sflow/clab-iperf3
mgmt_ipv4: 172.100.100.7
mgmt_ipv6: 2001:172:100:100::7
exec:
- ip addr add 172.16.2.2/24 dev eth1
- ip route add 172.16.1.0/24 via 172.16.2.1
- ip addr add 2001:172:16:2::2/64 dev eth1
- ip route add 2001:172:16:1::/64 via 2001:172:16:2::1
sflow-rt:
kind: linux
image: sflow/prometheus
mgmt_ipv4: 172.100.100.8
mgmt_ipv6: 2001:172:100:100::8
ports:
- 8008:8008
links:
- endpoints: ["leaf1:eth1","spine1:eth1"]
- endpoints: ["leaf1:eth2","spine2:eth1"]
- endpoints: ["leaf2:eth1","spine1:eth2"]
- endpoints: ["leaf2:eth2","spine2:eth2"]
- endpoints: ["h1:eth1","leaf1:eth3"]
- endpoints: ["h2:eth1","leaf2:eth3"]</pre>
<p>The new <a href="https://github.com/sflow-rt/containerlab/blob/master/clos3.clab.gotmpl">clos3.clab.gotmpl</a> file is a templated version of the topology:</p>
<pre>name: clos3
mgmt:
network: fixedips
ipv4_subnet: 172.100.100.0/24
ipv6_subnet: 2001:172:100:100::/80
topology:
defaults:
kind: linux
env:
COLLECTOR: 172.100.100.{{ add $.spines.num $.leaves.num $.leaves.num 2 }}
nodes:
{{- range $leafIndex := seq 1 $.leaves.num }}
leaf{{ $leafIndex }}:
image: sflow/clab-frr
mgmt_ipv4: 172.100.100.{{ add $leafIndex 1 }}
mgmt_ipv6: 2001:172:100:100::{{ add $leafIndex 1 }}
env:
LOCAL_AS: {{ add 65000 $leafIndex }}
NEIGHBORS:{{- range $spineIndex := seq 1 $.spines.num }} eth{{ $spineIndex}}{{- end }}
HOSTPORT: eth{{ add $.spines.num 1 }}
HOSTNET: 172.16.{{ $leafIndex }}.1/24
HOSTNET6: 2001:172:16:{{ $leafIndex }}::1/64
exec:
- touch /tmp/initialized
{{- end }}
{{- range $spineIndex := seq 1 $.spines.num }}
spine{{ $spineIndex }}:
image: sflow/clab-frr
mgmt_ipv4: 172.100.100.{{ add $.leaves.num $spineIndex 1 }}
mgmt_ipv6: 2001:172:100:100::{{ add $.leaves.num $spineIndex 1 }}
env:
LOCAL_AS: {{ add 65000 $.leaves.num 1 }}
NEIGHBORS:{{- range $leafIndex := seq 1 $.leaves.num }} eth{{ $leafIndex }}{{- end }}
exec:
- touch /tmp/initialized
{{- end }}
{{- range $leafIndex := seq 1 $.leaves.num }}
h{{ $leafIndex }}:
image: sflow/clab-iperf3
mgmt_ipv4: 172.100.100.{{ add $.spines.num $.leaves.num $leafIndex 1 }}
mgmt_ipv6: 2001:172:100:100::{{ add $.spines.num $.leaves.num $leafIndex 1 }}
exec:
- ip addr add 172.16.{{ $leafIndex }}.2/24 dev eth1
- ip route add 172.16.0.0/16 via 172.16.{{ $leafIndex }}.1
- ip addr add 2001:172:16:{{ $leafIndex }}::2/64 dev eth1
- ip route add 2001:172:16::/48 via 2001:172:16:{{ $leafIndex }}::1
{{- end }}
sflow-rt:
image: sflow/prometheus
mgmt_ipv4: 172.100.100.{{ add $.spines.num $.leaves.num $.leaves.num 2 }}
mgmt_ipv6: 2001:172:100:100::{{ add $.spines.num $.leaves.num $.leaves.num 2 }}
ports:
- 8008:8008
links:
{{- range $spineIndex := seq 1 $.spines.num }}
{{- range $leafIndex := seq 1 $.leaves.num }}
- endpoints: ["spine{{ $spineIndex }}:eth{{ $leafIndex }}", "leaf{{ $leafIndex }}:eth{{ $spineIndex }}"]
{{- end }}
{{- end }}
{{- range $leafIndex := seq 1 $.leaves.num }}
- endpoints: ["leaf{{ $leafIndex }}:eth{{ add $.spines.num 1 }}", "h{{ $leafIndex }}:eth1"]
{{- end }}</pre>
The template makes uses of settings in the corresponsing <a href="https://github.com/sflow-rt/containerlab/blob/master/clos3.clab_vars.yml">clos3.clab_vars.yml</a> file:
<pre>spines:
num: 2
leaves:
num: 2</pre>
While creating a template involves some work, the result is a more compact representation of the configuration since repetative leaf and spine configurations are captures in iterative blocks. The advantage becomes clear with larger topologies since a 4 leaf 4 spine explicit configuration would be twice as large, but the tempate remains unchanged.
<pre>docker run --rm -it --privileged --network host --pid="host" \
-v /var/run/docker.sock:/var/run/docker.sock -v /run/netns:/run/netns \
-v $(pwd):$(pwd) -w $(pwd) \
ghcr.io/srl-labs/clab bash</pre>
Run the above command to start Containerlab.
<pre>wget https://raw.githubusercontent.com/sflow-rt/containerlab/master/clos3.clab.gotmpl
wget https://raw.githubusercontent.com/sflow-rt/containerlab/master/clos3.clab_vars.yml</pre>
Download the template and settings files.
<pre>containerlab deploy -t clos3.clab.gotmpl</pre>
Create the emulated network.
<pre>docker exec -it clab-clos3-leaf1 vtysh -c "show running-config"</pre>
See configuration of <i>leaf1</i> router.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj93dRDqG4m3z8BzrKV2jgNjENUg2OM1eb-DFztdeth3502Fd8Eq4zBfmrKF0XvvfEfdgYNPeeIbdwuTZBraYugWVGNUqo27GLWGWYSfV88YLnd5isZJZGVwybwwL3hPhSTq7p6TzSupgkEA1d-sM-M26xQUhxy6rYPj5jh8NquNiRwEQLrPcDQtiaEuA/s1784/clab-tmpl-dash.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1518" data-original-width="1784" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj93dRDqG4m3z8BzrKV2jgNjENUg2OM1eb-DFztdeth3502Fd8Eq4zBfmrKF0XvvfEfdgYNPeeIbdwuTZBraYugWVGNUqo27GLWGWYSfV88YLnd5isZJZGVwybwwL3hPhSTq7p6TzSupgkEA1d-sM-M26xQUhxy6rYPj5jh8NquNiRwEQLrPcDQtiaEuA/s600/clab-tmpl-dash.png" width="600" /></a></div>
Connect to the web interface, <a href="http://localhost:8008">http://localhost:8008</a>. The <a href="https://sflow-rt.com">sFlow-RT</a> dashboard verifies that telemetry is being received from the four (two leaf and two spine) switches in the topology. See the <a href="https://sflow-rt.com/intro.php">sFlow-RT Quickstart</a> guide for more information.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqcNP8FzRX3wHlEmp6tAuJxozQ-GhuUWlund_LrcywMSP2yCiIXfW_rItHqwkFUrbdoTt-QGIPxr2dCAyvzS0W4Lep21Y4eMzID3D-FiEg1NlDZneyHOAMjT0E3BguTVcUsNpvocrh1iXuovdvoyPCt1ZfZlo1agb_2ArF-BDdJZ9cp-QWZUvtji0k_A/s2210/clab-tmpl-topn.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1316" data-original-width="2210" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqcNP8FzRX3wHlEmp6tAuJxozQ-GhuUWlund_LrcywMSP2yCiIXfW_rItHqwkFUrbdoTt-QGIPxr2dCAyvzS0W4Lep21Y4eMzID3D-FiEg1NlDZneyHOAMjT0E3BguTVcUsNpvocrh1iXuovdvoyPCt1ZfZlo1agb_2ArF-BDdJZ9cp-QWZUvtji0k_A/s600/clab-tmpl-topn.png" width="600" /></a></div>
The screen capture shows a real-time view of traffic flowing across the network during an iperf3 tests. Click on the sFlow-RT <i>Apps</i> menu and select the <i>browse-flows</i> application, or <a href="http://localhost:8008/app/browse-flows/html/index.html?keys=ipsource%2Cipdestination&value=bps">click here</a> for a direct link to a chart with the settings shown above.
<pre>docker exec -it clab-clos3-h1 iperf3 -c 172.16.2.2</pre>
Each of the hosts in the network has an iperf3 server, so running the above command will test bandwidth between <i>h1</i> and <i>h2</i>.
<pre>containerlab destroy -t clos3.clab.gotmpl</pre>
<p>When you are finished, run the above command to stop the containers and free the resources associated with the emulation.</p><p>Finally, try editing the <i>clos3.clab_vars.yml</i> file and increase the number of leaf switches to 12 and the number of spine switches to 5 and repeat the tests with a more realistic topology. A big advantage of using containers to emulate network devices is that they are extremely lightweight, allowing realistic production networks to be emulated on a laptop. Try the other <a href="https://github.com/sflow-rt/containerlab">sflow-rt/containerlab</a> examples to experiment with DDoS mitigation, EVPN monitoring, and flow tracing.</p>Peterhttp://www.blogger.com/profile/00856599914190257147noreply@blogger.com0tag:blogger.com,1999:blog-1978652979840829013.post-34117129743408723742022-12-01T07:01:00.000-08:002022-12-01T07:01:46.898-08:00IPv6 flow analytics with Containerlab<p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhM0MfjJKQqWc5MDuAsZSE02FiYxC6KFZhrzqp-xERZRsYEGy2KTpVvLO6XeJUlxvSN3BvNXjPyXI-5hp3WSDoys6MotUk2nr4376ILN0G9Y1EIAM9RqVStBbVz_iWgwhODxrohz6CU1zF7fr3eFnkgGFMPUXN0F27fio0ec0RxYaLNVr9SugIWsXfIAg/s640/clos5.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="270" data-original-width="640" height="270" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhM0MfjJKQqWc5MDuAsZSE02FiYxC6KFZhrzqp-xERZRsYEGy2KTpVvLO6XeJUlxvSN3BvNXjPyXI-5hp3WSDoys6MotUk2nr4376ILN0G9Y1EIAM9RqVStBbVz_iWgwhODxrohz6CU1zF7fr3eFnkgGFMPUXN0F27fio0ec0RxYaLNVr9SugIWsXfIAg/w640-h270/clos5.png" width="640" /></a></div>
<p><a href="https://containerlab.dev/">CONTAINERlab</a> is a <a href="https://www.docker.com/">Docker</a> orchestration tool for creating virtual network topologies. The <a href="https://github.com/sflow-rt/containerlab">sflow-rt/containerlab</a> project contains a number of topologies demonstrating industry standard streaming <a href="https://sflow.org">sFlow</a> telemetry with realistic data center topologies. This article extends the examples in <a href="https://blog.sflow.com/2022/02/real-time-telemetry-from-5-stage-clos.html">Real-time telemetry from a 5 stage Clos fabric</a> and <a href="https://blog.sflow.com/2022/03/real-time-evpn-fabric-visibility.html">Real-time EVPN fabric visibility</a> to demonstrate visibility into IPv6 traffic flows.</p>
<pre>docker run --rm -it --privileged --network host --pid="host" \
-v /var/run/docker.sock:/var/run/docker.sock -v /run/netns:/run/netns \
-v $(pwd):$(pwd) -w $(pwd) \
ghcr.io/srl-labs/clab bash</pre>
<p>Run the above command to start Containerlab if you already have Docker installed. Otherwise, <a href="https://containerlab.srlinux.dev/install/">Installation</a> provides detailed instructions for a variety of platforms.</p>
<pre>curl -O <a href="https://raw.githubusercontent.com/sflow-rt/containerlab/master/clos5.yml">https://raw.githubusercontent.com/sflow-rt/containerlab/master/clos5.yml</a></pre>
<p>Download the topology file for the 5 stage Clos fabric shown above.</p>
<pre>containerlab deploy -t clos5.yml</pre>
<p>Finally, deploy the topology.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjmNh-Zv8s95i9Z10HdFAi9Ho-LcHE1O99UR4Qua7gfHNtJs6DWvrbk5ah0uEKGgQE5P2KRRXi6KSpQ7LZq3IgMcYkQGArz4D9_sM4K2DwU8iqJRSBUHfezMj5XGIxnbW9jgqaGrsPlH1rAWw1vjtLqsMjqkTIihKGHpBTWGFdpsgyU3k02R7xhUX_Zg/s1852/browseflows-ipv6.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1756" data-original-width="1852" height="606" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjmNh-Zv8s95i9Z10HdFAi9Ho-LcHE1O99UR4Qua7gfHNtJs6DWvrbk5ah0uEKGgQE5P2KRRXi6KSpQ7LZq3IgMcYkQGArz4D9_sM4K2DwU8iqJRSBUHfezMj5XGIxnbW9jgqaGrsPlH1rAWw1vjtLqsMjqkTIihKGHpBTWGFdpsgyU3k02R7xhUX_Zg/w640-h606/browseflows-ipv6.png" width="640" /></a></div>The screen capture shows a real-time view of traffic flowing across the network during an iperf3 test. Click on the <a href="http://localhost:8008/html/index.html#apps">sFlow-RT Apps</a> menu and select the <i>browse-flows</i> application, or click <a href="http://localhost:8008/app/browse-flows/html/index.html?keys=ip6source%2Cip6destination%2Cnode%3Ainputifindex%2Cifname%3Ainputifindex%2Cip6ttl&value=bps">here</a> for a direct link to a chart with the settings shown above.
<pre>docker exec -it clab-clos5-h1 iperf3 -c 2001:172:16:4::2</pre>
<p>Each of the hosts in the network has an iperf3 server, so running the above command will test bandwidth between <i>h1</i> and <i>h4</i>.</p>
<pre>containerlab destroy -t clos5.yml</pre>
<p>When you are finished, run the above command to stop the containers and free the resources associated with the emulation.</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzwlALYoc--COEwMlfiOLLpS38RJf-G6vC0nTzGCITIyIA2xYWhTrOob8nnpwBe8BxMETgR9yYog8KpF1MNoYHlMGoOBX4a5VnsGUGGuB6XjDdAiy6FipcjshTvBka5_9cdBFoRcKP-fb4Q5SBt8t8f-sBIZ2AGow368JICAhhS-ioJvtFzMfK4Hz-AQ/s1222/evpn3.png" style="display: block; padding: 1em 0px; text-align: center;"><img border="0" data-original-height="686" data-original-width="1222" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzwlALYoc--COEwMlfiOLLpS38RJf-G6vC0nTzGCITIyIA2xYWhTrOob8nnpwBe8BxMETgR9yYog8KpF1MNoYHlMGoOBX4a5VnsGUGGuB6XjDdAiy6FipcjshTvBka5_9cdBFoRcKP-fb4Q5SBt8t8f-sBIZ2AGow368JICAhhS-ioJvtFzMfK4Hz-AQ/w640-h360/evpn3.png" width="640" /></a></div>
<a href="https://blog.sflow.com/2022/03/real-time-evpn-fabric-visibility.html">Real-time EVPN fabric visibility</a> describes the EVPN configuration above in detail. The following steps extend the example to demonstrate visibility into IPv6 flows.
<pre>curl -O <a href="https://raw.githubusercontent.com/sflow-rt/containerlab/master/evpn3.yml">https://raw.githubusercontent.com/sflow-rt/containerlab/master/evpn3.yml</a></pre>
Download the topology file.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgpD54XaWNZA_U3tuq88Z2ALCR5E9HZ5Wq7_nFGg456iucfwDXf2vZSu9-s1SRtLqW740eGVLUwhJtIT46KXl3-mwpZAoVfAi2iEfWtLjV3Pyhz8M9pT-HsjqXs8vbdzqRAxZW_RKICM5xTReAL7EHmvxCRqI6A5yXuyu8rCTefMzs3aYH5zrDCvzGvDQ/s1852/evpn-ipv6.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1756" data-original-width="1852" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgpD54XaWNZA_U3tuq88Z2ALCR5E9HZ5Wq7_nFGg456iucfwDXf2vZSu9-s1SRtLqW740eGVLUwhJtIT46KXl3-mwpZAoVfAi2iEfWtLjV3Pyhz8M9pT-HsjqXs8vbdzqRAxZW_RKICM5xTReAL7EHmvxCRqI6A5yXuyu8rCTefMzs3aYH5zrDCvzGvDQ/s600/evpn-ipv6.png" width="600" /></a></div>
The screen capture shows a real-time view of traffic flowing across the network during an iperf3 test. Connect to the sFlow-RT Flow Browser application, or click <a href="http://localhost:8008/app/browse-flows/html/index.html?keys=stack%2Cnull%3Aipsource%2Cnull%3Aipdestination%2Cip6source%2Cip6destination%2Cnode%3Ainputifindex%2Cip6ttl&value=bps">here</a> for a direct link to a chart with the settings shown above.
<pre>docker exec -it clab-evpn3-h1 iperf3 -c 2001:172:16:10::2</pre>
<p>Each of the hosts in the network has an iperf3 server, so running the above command will test bandwidth between h1 and h2. The flow should immediately appear in the Flow Browser chart.</p>
<pre>containerlab destroy -t evpn3.yml</pre>
<p>When you are finished, run the above command to stop the containers and free the resources associated with the emulation.</p><p>The following articles describe additionial examples based on <a href="https://github.com/sflow-rt/containerlab">sflow-rt/containerlab</a> topologies:</p>
<ul style="text-align: left;">
<li><a href="https://blog.sflow.com/2022/02/topology-aware-fabric-analytics.html">Topology aware fabric analytics</a></li>
<li><a href="https://blog.sflow.com/2022/03/containerlab-ddos-testbed.html">Containerlab DDoS testbed</a></li><li><a href="https://blog.sflow.com/2022/03/ddos-attacks-and-bgp-flowspec-responses.html">DDoS attacks and BGP Flowspec responses</a></li>
<li><a href="https://blog.sflow.com/2022/04/bgp-remotely-triggered-blackhole-rtbh.html">BGP Remotely Triggered Blackhole (RTBH)</a></li>
</ul>
Moving the monitoring solution from Containerlab to production is straightforward since sFlow is <a href="https://sflow.org/products/network.php">widely implemented</a> in datacenter equipment from vendors including: A10, Arista, Aruba, Cisco, Edge-Core, Extreme, Huawei, Juniper, NEC, Netgear, Nokia, NVIDIA, Quanta, and ZTE. In addition, the open source <a href="https://sflow.net/">Host sFlow</a> agent makes it easy to extend visibility beyond the physical network into the compute infrastructure.Peterhttp://www.blogger.com/profile/00856599914190257147noreply@blogger.com0tag:blogger.com,1999:blog-1978652979840829013.post-56704088236097990112022-11-17T11:47:00.000-08:002022-11-17T11:47:57.769-08:00SC22 SCinet network monitoring<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRY1ls4W2DeUBOJMkbcCoR3OLOFRz4NCbQo5lifbYj58vd8oaQtjPTrKInd_B_enfGWAUiKHGeA7k_joYZ2427MZigw1VMiB2dICBGZKfD5UbLtbp8KlpXTP9uDPmSi28_HL9X042w5-XLP_LTDeyl0xDtZM7mMOQCe-eNBs8KYQ6bQWVzg2E90RQL-Q/s2220/sc22-dash.png" style="display: block; padding: 1em 0px; text-align: center;"><img border="0" data-original-height="2176" data-original-width="2220" height="627" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRY1ls4W2DeUBOJMkbcCoR3OLOFRz4NCbQo5lifbYj58vd8oaQtjPTrKInd_B_enfGWAUiKHGeA7k_joYZ2427MZigw1VMiB2dICBGZKfD5UbLtbp8KlpXTP9uDPmSi28_HL9X042w5-XLP_LTDeyl0xDtZM7mMOQCe-eNBs8KYQ6bQWVzg2E90RQL-Q/w640-h627/sc22-dash.png" width="640" /></a></div>
The data shown in the chart was gathered from <a href="https://sc22.supercomputing.org/">The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC22)</a> being held this week in Dallas. The conference network, <a href="https://sc22.supercomputing.org/scinet/">SCinet</a>, is described as <i>the fastest and most powerful network on Earth, connecting the SC community to the world</i>. The chart provides an up to the second view of overall SCinet traffic, the lower chart showing total traffic hitting a sustained 8Tbps.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgv-6KS8PYTAptQ71l3oDIuR86zObNtlcosjeBbWBZ6JlRLrbLK84QrprbmagFD-TOC07C2YmlFRq-fCQ290NQE-BWxHTB1ddbR-cyMzt-fVCCsws6OUOYKkJy0eW_0gr8JPgkKgGszn4JTYt1Bo9TzioEToxp0NN5qVclvaBhE1LPEKjy4ZaQ1J3UfTA/s3781/scinet2022.jpeg" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="2268" data-original-width="3781" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgv-6KS8PYTAptQ71l3oDIuR86zObNtlcosjeBbWBZ6JlRLrbLK84QrprbmagFD-TOC07C2YmlFRq-fCQ290NQE-BWxHTB1ddbR-cyMzt-fVCCsws6OUOYKkJy0eW_0gr8JPgkKgGszn4JTYt1Bo9TzioEToxp0NN5qVclvaBhE1LPEKjy4ZaQ1J3UfTA/s600/scinet2022.jpeg" width="600" /></a></div>
The poster shows the topology of the SCinet network. Monitoring flow data from 5,852 switch/router ports with 162Tbps total bandwith with sub-second latency is required to construct the charts.<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiiVbhQfTJYKidQKebHJVno__H-ebb_8VsjoQc8siABHNYBWSvv3_HHPH_jgjReGxHPA72twC9dWj8dwl1JzRx5qizE6dWcwlmwK7TLOkDmzO0tYKzutgClrBLvlb-mHYgXIS3oBfWvgiA4mdl3OS_4Fw4JxoPh0xAXQ1wAFYLgaOD4vEq3pOTZT0pSNg/s1472/sflow-rt.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="740" data-original-width="1472" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiiVbhQfTJYKidQKebHJVno__H-ebb_8VsjoQc8siABHNYBWSvv3_HHPH_jgjReGxHPA72twC9dWj8dwl1JzRx5qizE6dWcwlmwK7TLOkDmzO0tYKzutgClrBLvlb-mHYgXIS3oBfWvgiA4mdl3OS_4Fw4JxoPh0xAXQ1wAFYLgaOD4vEq3pOTZT0pSNg/s600/sflow-rt.png" width="600" /></a></div>
The chart was generated using industry standard streaming <a href="https://sflow.org">sFlow</a> telemetry from switches and routers in the SCinet network. An instance of the <a href="https://sflow-rt.com">sFlow-RT</a> real-time analytics engine computes the flow metrics shown in the charts.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEDBBMQbvW6lCEAPCdEKZD6kY47j1i75nXpeVROs9GEAaPH3FVMV98rtnMd02yjZDsYusq9bFsxgSSjqsKZa2ttQPE-RJ4tjbbds3tfQk7hbUWzm0BEk6-R2jAFIO73puvrz-abW1YAoIlbiUWkWRE2ypimQrGhunkb2E60vJl0NKuyw6AazvZrjhIgQ/s2238/sc22-elephants.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1318" data-original-width="2238" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEDBBMQbvW6lCEAPCdEKZD6kY47j1i75nXpeVROs9GEAaPH3FVMV98rtnMd02yjZDsYusq9bFsxgSSjqsKZa2ttQPE-RJ4tjbbds3tfQk7hbUWzm0BEk6-R2jAFIO73puvrz-abW1YAoIlbiUWkWRE2ypimQrGhunkb2E60vJl0NKuyw6AazvZrjhIgQ/s600/sc22-elephants.png" width="600" /></a></div>
Most of the load was due to large 400Gbit/s, 200Gbit/s and 100Gbit/s flows that were part of the <a href="https://sc22.supercomputing.org/scinet/network-research-exhibition/">Network Research Exhibition</a>. The chart above shows that 10 large flows are responsible for 1.5Tbps of traffic.<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgzqSgG0mkHimiO5JO-sohshlllH33I7hLWXDTsI6J1myV-gUhChWtAZri8SPWyK1o9lWuUNFiW2pOJPLjZk3Blg3RZeGhA65slTGJxXld9Ced5W4CpwSKAjq3uh6D89oYtE5H-N03im9iyXA46Gg9p3PcgzPYR2pJ5IaG1X14ZnuLjM5MKSE1DUS0-Vg/s2490/scitags-demo-sc22.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="2194" data-original-width="2490" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgzqSgG0mkHimiO5JO-sohshlllH33I7hLWXDTsI6J1myV-gUhChWtAZri8SPWyK1o9lWuUNFiW2pOJPLjZk3Blg3RZeGhA65slTGJxXld9Ced5W4CpwSKAjq3uh6D89oYtE5H-N03im9iyXA46Gg9p3PcgzPYR2pJ5IaG1X14ZnuLjM5MKSE1DUS0-Vg/s600/scitags-demo-sc22.png" width="600" /></a></div><a href="https://blog.sflow.com/2022/11/scientific-network-tags-scitags.html">Scientific network tags (scitags)</a> describes how IPv6 flowlabels allow network flow analytics to identify network traffic associated with bulk scientific data transfers.<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgAki6sZvhp8PsNqacnKGn6OOpI3d5o4eMJLLXZmk7j80xJ9b9GTBL6J9My_Sqt61Z7zMx79eSqj4y1ZItIortz2L0QZrAr63w_tTsaZNSmYUATJwjOvKD90UzqEMTl-4rCHRrK1U5fUxIHa_AnI0vatIDaYqufRKOKXeoAcVLN0clcqEqLu1XWroctVw/s2212/rdma-sc2022.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1600" data-original-width="2212" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgAki6sZvhp8PsNqacnKGn6OOpI3d5o4eMJLLXZmk7j80xJ9b9GTBL6J9My_Sqt61Z7zMx79eSqj4y1ZItIortz2L0QZrAr63w_tTsaZNSmYUATJwjOvKD90UzqEMTl-4rCHRrK1U5fUxIHa_AnI0vatIDaYqufRKOKXeoAcVLN0clcqEqLu1XWroctVw/s600/rdma-sc2022.png" width="600" /></a></div><a href="https://blog.sflow.com/2022/11/rdma-network-visibility.html">RDMA network visibility</a> shows how bulk data transfers using <a href="https://en.wikipedia.org/wiki/Remote_direct_memory_access">Remote Direct Memory Access (RDMA)</a>.Peterhttp://www.blogger.com/profile/00856599914190257147noreply@blogger.com0tag:blogger.com,1999:blog-1978652979840829013.post-83642350162830544382022-11-16T07:03:00.001-08:002022-11-17T10:54:04.663-08:00RDMA network visibility<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi6eWv40eap7g_trP4gIRN-3wczLxFnKVwLg7ViPJengvcCno3kEpwKxRKbxOh-TBs6wxjH96DStQDGeZnhVX4DI9X0KsfDG9rkquBWefpkfe0DbJt-Wnjk9Le1Oq6J2nNkah_eAXWA5XIbjuwP9Vv2cnQilAKuYGiwVAkavvhFn4BGu2fD8z4mkEif3Q/s2212/rdma-sc2022.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1600" data-original-width="2212" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi6eWv40eap7g_trP4gIRN-3wczLxFnKVwLg7ViPJengvcCno3kEpwKxRKbxOh-TBs6wxjH96DStQDGeZnhVX4DI9X0KsfDG9rkquBWefpkfe0DbJt-Wnjk9Le1Oq6J2nNkah_eAXWA5XIbjuwP9Vv2cnQilAKuYGiwVAkavvhFn4BGu2fD8z4mkEif3Q/s600/rdma-sc2022.png" width="600" /></a></div>
The <a href="https://en.wikipedia.org/wiki/Remote_direct_memory_access">Remote Direct Memory Access (RDMA)</a> data shown in the chart was gathered from <a href="https://sc22.supercomputing.org/">The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC22)</a> being held this week in Dallas. The conference network, <a href="https://sc22.supercomputing.org/scinet/">SCinet</a>, is described as <i>the fastest and most powerful network on Earth, connecting the SC community to the world</i>.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0ZP55CfYVna5NaCP2nean6kbytiy216wo73LxtFw9pYN57ee1zi2K8ySTHIOMGX1IRylZwvaHS6QSPJ9c1KIuPSlAzvOTb4F8AC3Xg3TrXQ6Pj-38p2irtEbYQWWF6-4Ssx2SnBOSM1Qwkybe-elNSVQAw7z_69vHjcYrh38Nf78MAUk0Mw0w7oIrog/s1362/sc22-rdma.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1028" data-original-width="1362" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0ZP55CfYVna5NaCP2nean6kbytiy216wo73LxtFw9pYN57ee1zi2K8ySTHIOMGX1IRylZwvaHS6QSPJ9c1KIuPSlAzvOTb4F8AC3Xg3TrXQ6Pj-38p2irtEbYQWWF6-4Ssx2SnBOSM1Qwkybe-elNSVQAw7z_69vHjcYrh38Nf78MAUk0Mw0w7oIrog/s600/sc22-rdma.png" width="600" /></a></div>
<a href="https://sc22.supercomputing.org/wp-content/uploads/2022/11/SC22-NRE-012-Linden_Mercer-Resilient_Distributed_Processing-1.pdf">Resilient Distributed Processing and Reconfigurable Networks</a> is one of the demonstrations using SCinet - Location: Booth 2847 (StarLight). Planned SC22 focus is on RDMA enabled data movement and dynamic network control.
<ol style="text-align: left;"><li>RDMA Tbps performance over global distance for timely Terabyte bulk data transfers (goal << 1 min Tbyte transfer on N by 400G network).</li><li>Dynamic shifting of processing and network resources from on location/path/system to another (in response to demand and availability).</li></ol>
The real-time chart at the top of this page shows an up to the second view of RDMA traffic (broken out by source, destination, and RDMA operation).
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh28tV4UGgD8MYFP604J9tiDpRb7eNT6qTFpi8DwG9TYJJH6z5Mc1Bo-nIpyjTI5GlM91tIyCyviH4XMpUGY5IaBElzN5NL0pnM1qWeVj0lyqoNZx42YSgYkdpXGXNBl_H4HULZRvYFeiw1PzG5YYtQ75EWBV5VZ7F43tEQMveiGCe-fxEqDNc2nUKPxA/s1032/rt-ecosystem.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="520" data-original-width="1032" height="322" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh28tV4UGgD8MYFP604J9tiDpRb7eNT6qTFpi8DwG9TYJJH6z5Mc1Bo-nIpyjTI5GlM91tIyCyviH4XMpUGY5IaBElzN5NL0pnM1qWeVj0lyqoNZx42YSgYkdpXGXNBl_H4HULZRvYFeiw1PzG5YYtQ75EWBV5VZ7F43tEQMveiGCe-fxEqDNc2nUKPxA/w640-h322/rt-ecosystem.png" width="640" /></a></div>
The chart was generated using industry standard streaming sFlow telemetry from switches and routers in the SCinet network. An instance of the <a href="https://sflow-rt.com">sFlow-RT</a> analytics engine computes the RDMA flow metrics shown in the chart. <a href="https://blog.sflow.com/2013/08/restflow.html">RESTflow</a> describes how sFlow disaggregates the traditional NetFlow / IPFIX analytics pipeline to offer flexible, scaleable, low latency flow measurements. <a href="https://blog.sflow.com/2019/10/flow-metrics-with-prometheus-and-grafana.html">Flow metrics with Prometheus and Grafana</a> describes how metrics can be stored in a time series database for use in operational dashboards.
<p>Real-time traffic analytics transforms network monitoring from reporting on the past to observing and acting on the present to automate troubleshooting and traffic engineering, e.g. <a href="https://blog.sflow.com/2015/06/leaf-and-spine-traffic-engineering.html">Leaf and spine traffic engineering using segment routing and SDN</a> and <a href="https://blog.sflow.com/2021/10/ddos-protection-quickstart-guide.html">DDoS protection quickstart guide</a>.</p>Peterhttp://www.blogger.com/profile/00856599914190257147noreply@blogger.com0