Arista EOS telemetry

Arista EOS switches support industry standard sFlow telemetry, enabling hardware instrumentation supported by merchant silicon to export hardware interface counters and flow data. The latest release of the open source Host sFlow agent has been ported to EOS, augmenting the telemetry with standard host CPU, memory, and disk IO metrics.

Linux as a Switch Operating System: Five Lessons Learned identifies benefits of using Linux as the basis for EOS. In this context, the Linux operating system made it easy to port the Host sFlow agent, use standard Linux package management (RPM Package Manager), and gather metrics using standard Linux APIs. A new eAPI module automatically synchronizes the Host sFlow daemon with the EOS sFlow configuration.

The following sflowtool output shows the additional metrics contributed by a Host sFlow agent installed on an Arista switch:

startDatagram =================================
datagramSourceIP 172.17.0.1
datagramSize 704
unixSecondsUTC 1490843418
datagramVersion 5
agentSubId 100000
agent 10.0.0.90
packetSequenceNo 714
sysUpTime 0
samplesInPacket 1
startSample ----------------------
sampleType_tag 0:2
sampleType COUNTERSSAMPLE
sampleSequenceNo 714
sourceId 2:1
counterBlock_tag 0:2001
counterBlock_tag 0:2010
udpInDatagrams 1459
udpNoPorts 16
udpInErrors 0
udpOutDatagrams 4765
udpRcvbufErrors 0
udpSndbufErrors 0
udpInCsumErrors 0
counterBlock_tag 0:2009
tcpRtoAlgorithm 1
tcpRtoMin 200
tcpRtoMax 120000
tcpMaxConn 4294967295
tcpActiveOpens 102
tcpPassiveOpens 100
tcpAttemptFails 0
tcpEstabResets 0
tcpCurrEstab 8
tcpInSegs 19930
tcpOutSegs 19804
tcpRetransSegs 0
tcpInErrs 0
tcpOutRsts 2
tcpInCsumErrors 0
counterBlock_tag 0:2008
icmpInMsgs 1606
icmpInErrors 0
icmpInDestUnreachs 16
icmpInTimeExcds 0
icmpInParamProbs 0
icmpInSrcQuenchs 0
icmpInRedirects 0
icmpInEchos 1590
icmpInEchoReps 0
icmpInTimestamps 0
icmpInAddrMasks 0
icmpInAddrMaskReps 0
icmpOutMsgs 0
icmpOutErrors 1606
icmpOutDestUnreachs 0
icmpOutTimeExcds 16
icmpOutParamProbs 0
icmpOutSrcQuenchs 0
icmpOutRedirects 0
icmpOutEchos 0
icmpOutEchoReps 0
icmpOutTimestamps 1590
icmpOutTimestampReps 0
icmpOutAddrMasks 0
icmpOutAddrMaskReps 0
counterBlock_tag 0:2007
ipForwarding 2
ipDefaultTTL 64
ipInReceives 24685
ipInHdrErrors 0
ipInAddrErrors 42
ipForwDatagrams 0
ipInUnknownProtos 0
ipInDiscards 0
ipInDelivers 23025
ipOutRequests 26170
ipOutDiscards 0
ipOutNoRoutes 0
ipReasmTimeout 0
ipReasmReqds 0
ipReasmOKs 0
ipReasmFails 0
ipFragOKs 4
ipFragFails 0
ipFragCreates 8
counterBlock_tag 0:2005
disk_total 1907843072
disk_free 1083969536
disk_partition_max_used 43.18
disk_reads 16549
disk_bytes_read 1337825280
disk_read_time 7420
disk_writes 412
disk_bytes_written 1159168
disk_write_time 216
counterBlock_tag 0:2004
mem_total 1938849792
mem_free 85483520
mem_shared 0
mem_buffers 106614784
mem_cached 735801344
swap_total 0
swap_free 0
page_in 830716
page_out 566
swap_in 0
swap_out 0
counterBlock_tag 0:2003
cpu_load_one 0.070
cpu_load_five 0.060
cpu_load_fifteen 0.050
cpu_proc_run 0
cpu_proc_total 221
cpu_num 1
cpu_speed 2698
cpu_uptime 17265
cpu_user 272510
cpu_nice 50
cpu_system 178050
cpu_idle 16279880
cpu_wio 550
cpuintr 461060
cpu_sintr 41840
cpuinterrupts 5458397
cpu_contexts 5338141
cpu_steal 0
cpu_guest 0
cpu_guest_nice 0
counterBlock_tag 0:2006
nio_bytes_in 8149749
nio_pkts_in 115730
nio_errs_in 0
nio_drops_in 0
nio_bytes_out 4996846
nio_pkts_out 28451
nio_errs_out 0
nio_drops_out 0
counterBlock_tag 0:2000
hostname leaf1
UUID 33-28-66-a5-82-27-43-49-a5-f1-c1-ba-cc-6c-1d-d3
machine_type 2
os_name 2
os_release 3.4.43.Ar-4170906.4180F
endSample   ----------------------
endDatagram   =================================

There are a number of additional open source and commercial sFlow collectors available.

For example, the diagram shows how new and existing cloud based or locally hosted orchestration, operations, and security tools can leverage the sFlow-RT analytics service to gain real-time visibility.

Installing Host sFlow agent on an Arista switch

The following steps download and install the Host sFlow agent on an Arista switch and direct the telemetry stream to collector 10.0.0.50:

1. Install the Host sFlow agent (hsflowd)

eos# copy https://github.com/sflow/host-sflow/releases/download/v2.0.9-1/hsflowd-eos-2.0.9-1.i686.rpm extension:
eos# extension hsflowd-eos-2.0.9-1.i686.rpm
eos# bash sudo service hsflowd start
eos# copy installed-extensions boot-extensions

2. Enable eAPI, see eAPI and Unix Domain Socket

eos(config)# management api http-commands
eos(config-mgmt-api-http-cmds)# protocol unix-socket
eos(config-mgmt-api-http-cmds)# no shutdown

3. Configure switch to run hsflowd on startup:

eos(config)# event-handler hsflowd
eos(config-handler-hsflowd)# trigger on-boot
eos(config-handler-hsflowd)# action bash sudo service hsflowd start
eos(config-handler-hsflowd)# delay 60
eos(config-handler-hsflowd)# asynchronous

4. Configure sFlow Introduction to Managing EOS Devices – Setting up Management

eos(config)# sflow source-interface Management1
eos(config)# sflow destination 10.0.0.50
eos(config)# sflow run

The host metrics should immediately begin to be received at the sFlow collector.

Nutanix

Maximum Performance from Acropolis Hypervisor and Open vSwitch describes the network architecture within a Nutanix converged infrastructure appliance - see diagram above. This article will explore how the Host sFlow agent can be deployed to enable sFlow instrumentation in the Open vSwitch (OVS)  and deliver streaming network and system telemetry from nodes in a Nutanix cluster.

This article is based on a single hardware node running Nutanix Community Edition (CE), built following the instruction in Part I: How to setup a three-node NUC Nutanix CE cluster. If you don't have hardware readily available, the article, 6 Nested Virtualization Resources To Get You Started With Community Edition, describes how to run Nutanix CE as a virtual machine.

The sFlow standard is widely supported by network equipment vendors, which combined with sFlow from each Nutanix appliance, delivers end to end visibility in the Nutanix cluster. The following screen captures from the free sFlowTrend tool are representative examples of the data available from the Nutanix appliance.

The Network > Top N chart displays the top flows traversing OVS. In this case an HTTP connection is responsible for most of the traffic. Inter-VM and external traffic flows traverse OVS and are efficiently monitored by the embedded sFlow instrumentation.

The Hosts > CPU utilization chart shows an increase in CPU utilization due to the increased traffic.

The Hosts > Disk IO shows the Write operations associated with connection.

Installing Host sFlow agent on Nutanix appliance

The following steps install Host sFlow on a Nutanix device:

First log into the Nutanix host as root.

Update June 19, 2019: When you log in as root you will see a warning that installing software on the hypervisor is not supported. See comment below.

Next, find the latest version of the Centos 7 RPM on sFlow.net and use the following commands to download and install the software:

wget https://github.com/sflow/host-sflow/releases/download/v2.0.8-1/hsflowd-centos7-2.0.8-1.x86_64.rpm
rpm -ivh hsflowd-centos7-2.0.8-1.x86_64.rpm
rm hsflowd-centos7-2.0.8-1.x86_64.rpm

Edit the /etc/hsflowd.conf file to direct sFlow telemetry to collector 10.0.0.50, enable KVM monitoring (virtual machine stats), and push sFlow configuration to OVS (network stats):

sflow {
  ...
  # collectors:
  collector { ip=10.0.0.50 udpport=6343 }
  ...
  # Open vSwitch sFlow configuration:
  ovs { }
  # KVM (libvirt) hypervisor and VM monitoring:
  kvm { }
  ...
}

Now start the Host sFlow daemon:

systemctl enable hsflowd.service
systemctl start hsflowd.service

Data will immediately start to appear in sFlowTrend.