Monday, March 20, 2017

Nutanix

Maximum Performance from Acropolis Hypervisor and Open vSwitch describes the network architecture within a Nutanix converged infrastructure appliance - see diagram above. This article will explore how the Host sFlow agent can be deployed to enable sFlow instrumentation in the Open vSwitch (OVS)  and deliver streaming network and system telemetry from nodes in a Nutanix cluster.
This article is based on a single hardware node running Nutanix Community Edition (CE), built following the instruction in Part I: How to setup a three-node NUC Nutanix CE cluster. If you don't have hardware readily available, the article, 6 Nested Virtualization Resources To Get You Started With Community Edition, describes how to run Nutanix CE as a virtual machine.
The sFlow standard is widely supported by network equipment vendors, which combined with sFlow from each Nutanix appliance, delivers end to end visibility in the Nutanix cluster. The following screen captures from the free sFlowTrend tool are representative examples of the data available from the Nutanix appliance.
The Network > Top N chart displays the top flows traversing OVS. In this case an HTTP connection is responsible for most of the traffic. Inter-VM and external traffic flows traverse OVS and are efficiently monitored by the embedded sFlow instrumentation.
The Hosts > CPU utilization chart shows an increase in CPU utilization due to the increased traffic.
The Hosts > Disk IO shows the Write operations associated with connection.

Installing Host sFlow agent on Nutanix appliance

The following steps install Host sFlow on a Nutanix device:

First log into the Nutanix host as root.
Update June 19, 2019: When you log in as root you will see a warning that installing software on the hypervisor is not supported. See comment below.
Next, find the latest version of the Centos 7 RPM on sFlow.net and use the following commands to download and install the software:
wget https://github.com/sflow/host-sflow/releases/download/v2.0.8-1/hsflowd-centos7-2.0.8-1.x86_64.rpm
rpm -ivh hsflowd-centos7-2.0.8-1.x86_64.rpm
rm hsflowd-centos7-2.0.8-1.x86_64.rpm
Edit the /etc/hsflowd.conf file to direct sFlow telemetry to collector 10.0.0.50, enable KVM monitoring (virtual machine stats), and push sFlow configuration to OVS (network stats):
sflow {
  ...
  # collectors:
  collector { ip=10.0.0.50 udpport=6343 }
  ...
  # Open vSwitch sFlow configuration:
  ovs { }
  # KVM (libvirt) hypervisor and VM monitoring:
  kvm { }
  ...
}
Now start the Host sFlow daemon:
systemctl enable hsflowd.service
systemctl start hsflowd.service
Data will immediately start to appear in sFlowTrend.

6 comments:

  1. Peter, I came across this blog and wanted to ask that you add a warning that this configuration is not supported or approved by Nutanix. When the user logs into the AHV console as root, there is a large warning that says no additional packages should be installed.

    Nutanix AHV is a cluster-optimized hypervisor appliance.

    Alteration of the hypervisor appliance (unless advised by Nutanix
    Technical Support) is unsupported and may result in the hypervisor or
    VMs functioning incorrectly.

    Unsupported alterations include (but are not limited to):

    - Configuration changes.
    - Installation of third-party software not approved by Nutanix.
    - Installation or upgrade of software packages from non-Nutanix
    sources (using yum, rpm, or similar).

    ReplyDelete
    Replies
    1. Jason, I have published your warning.

      I hope Nutanix is considering including the open source Host sFlow agent as a supported component in AHV. The agent unlocks the instrumentation latent in OVS to provide detailed flow analytics with no impact on performance.

      I am aware of very large Nutanix AHV clusters that have been running sFlow telemetry in production without any problems for years. This is a very simple solution that provides comprehensive cluster wide visibility that is not available through any other mechanism. These customers clearly feel that the benefits outweigh the risk of running an unsupported configuration, but a supported solution to the requirement for network visibility would clearly be preferable.

      Delete
    2. Peter, it's been a little while but I haven't forgotten about this! With the release of AOS 6.6 and Prism Central 2022.9, Nutanix has support for configuring up to 5 IPFix export destinations. You can find the APIs for adding IPFix Export at developers.nutanix.com in the Networking namespace.

      Delete
    3. Jason, I hope you will consider also supporting sFlow. The sFlow packet samples provide detailed visibility into tunneled traffic (VxLAN, GRE, etc) as well as streaming interface counters. OVS has had sFlow support since version 1.0.

      The Host sFlow agent mentioned in this article is widely deployed on open source NOS platforms and is easily integrated - most recently included in latest version of VyOS, see VyOS with Host sFlow agent

      Delete
  2. This is good if they provide these agents as part of regular license.

    ReplyDelete
    Replies
    1. I'm happy to note that the IPFix Export configuration available in PC 2022.9 and AOS 6.6 does not require any additional license!

      Delete