Friday, October 15, 2010

KVM


Performance of the KVM (Kernel-based Virtual Machine) virtualization system can be monitored using sFlow through a combination of Host sFlow and Open vSwitch software (see Cloud-scale performance monitoring).

The Host sFlow agent provides visibility into the performance of the physical and virtual servers (see sFlow Host Structures). Download the Host sFlow agent and install it on your KVM server using the following commands:

rpm -Uvh hsflowd_KVM-XXX.rpm
service hsflowd start

Note: The virtual machine statistics exported by the Host sFlow agent are equivalent to statistics that can be obtained using the libvirt based tools (virsh, virt-manager) typically used in KVM environments (see libvirt). However, the Host sFlow agent also exports detailed statistics from the physical server as well as coordinating traffic measurements made by the vSwitch.

The Open vSwitch natively supports sFlow, providing visibility into network traffic. To include traffic monitoring, download the Open vSwitch and follow the instructions in How to Install Open vSwitch on Linux and How to Use Open vSwitch with KVM.

Note: There are many good reasons to use the Open vSwitch as your default switch. In addition to sFlow support, the Open vSwitch implements the OpenFlow protocol, providing distributed switching and network virtualization functionality.

The article, Configuring Open vSwitch, describes the steps to manually configure sFlow monitoring. However, manual configuration is not recommended since additional configuration is required every time a bridge is added to the vSwitch, or the system is rebooted. Instead, the Host sFlow agent can automatically configure the sFlow settings in the Open vSwitch, just run the following command to enable this feature:

service sflowovsd start

Note: While the Open vSwitch currently provides the best option for monitoring traffic between virtual machines, the combination of the Host sFlow agent and an sFlow capable switch offers a good alternative (see Hybrid server monitoring). The emerging IEEE 802.1Qbg Edge Virtual Bridging (EVB) standard will make it possible for the switch to monitor traffic between virtual machines (see VEPA).

The following steps configure all the sFlow agents to sample packets at 1-in-512, poll counters every 20 seconds and send sFlow to an analyzer (10.0.0.50) over UDP using the default sFlow port (6343).

Note: A previous posting discussed the selection of sampling rates.

The default configuration method used for sFlow is DNS-SD; enter the following DNS settings in the site DNS server:

analyzer A 10.0.0.50

_sflow._udp SRV 0 0 6343 analyzer
_sflow._udp TXT (
"txtvers=1"
"polling=20"
"sampling=512"
)

Note: These changes must be made to the DNS zone file corresponding to the search domain in the KVM server's /etc/resolv.conf file.

Once the sFlow settings are added to the DNS server, they will be automatically picked up by the Host sFlow agents. If you need to change the sFlow settings, simply change them on the DNS server and the change will automatically be applied to all the KVM systems in the data center.

Manual configuration is an option if you do not want to use DNS-SD. Edit the Host sFlow agent configuration file, /etc/hsflowd.conf, on each KVM server:

sflow{
  DNSSD = off
  polling = 20
  sampling = 512
  collector{
    ip = 10.0.0.50
    udpport = 6343
  }
}

After editing the configuration file you will need to restart the Host sFlow agent:

service hsflowd restart

An sFlow analyzer is needed to receive the sFlow data and report on performance (see Choosing an sFlow analyzer). The free sFlowTrend analyzer is a great way to get started, see sFlowTrend adds server performance monitoring to see examples.

18 comments:

  1. Can sFlow monitor the performance of KVM if Open vSwitch is not used? If yes are there documents? Will it be able to collect the VM statistics similar to the libvert?

    Thanks.
    Yuantai

    ReplyDelete
  2. Yes, Host sFlow will export libvirt VM statistics - this functionality does not depend on Open vSwitch. Either build Host sFlow from sources (the build script will automatically detect and link to libvirt - if it is present), or you can download the pre-built RPM (hsflowd_KVM-1.22.2-1.x86_64.rpm).

    Note: Open vSwitch is required if you want to see inter-VM traffic flows.

    ReplyDelete
    Replies
    1. I only see the "Red Hat KVM". Does that mean there is no Host sFlow for Debian\Ubuntu KVM?

      Thanks.
      Yuantai

      Delete
  3. Hi Peter,
    Thanks for responding. I am going to test out the Host sFlow on the KVM. We don't use Open vSwitch and won't use DNS-SD. Do you have a document on installing\configuring the Host sFlow under those conditions? We will use the .tar.gz instead of RPM package.

    Yuantai

    ReplyDelete
  4. Yes, if you compile from sources it should notice that libvirt is present and include the VM statistics. If you look in trunk/src/Makefile you will see that it sets LIBVIRT=yes if it finds anything matching /usr/lib/libvirt.*, but you can force that when you compile, like this:

    make LIBVIRT=yes

    If you don't get vdisk statistics, then try this:

    make LIBVIRT=yes VRTDSKPATH=yes

    If you have feedback or just want to share your experience with this, please comment on the host-sflow mailing list:
    https://lists.sourceforge.net/lists/listinfo/host-sflow-discuss

    Neil

    ReplyDelete
  5. Neil,
    Thanks for the instruction.

    If we intend to send the sFlow data to Ganglia (gmond), do we still need the sFlow Analyzer? in gmond configuration file, there is an option sFlow session. If I include the session in the gmond.config, can I assume the sFlow agent (including the one on KVM) will send data to gmond instead of sFlow Analyzer?

    --Yuantai

    ReplyDelete
    Replies
    1. If you are using Ganglia, then you will configure a single instance of gmond as an sFlow collector and configure the Host sFlow agents to send sFlow to the gmond instance.

      Ganglia / Host sFlow configuration

      You also need to configure gmond to accept the virtual machine metrics:

      Using Ganglia to monitor virtual machine pools

      There are a number of articles on Ganglia on this blog, click on the Ganglia label to see them all.

      Delete
  6. We installed the KVM RPM on a KVM host and the non-KVM RPM on a non-KVM host. If you query the RPM database it shows the same on both hosts. Here is the output on the KVM host:

    [root@vmhost ~]# rpm -qa | grep -i hsflowd
    hsflowd-1.22.2-1.x86_64

    We get the same output on the non-KVM host:
    [root@non-vmhost ~]# rpm -qa | grep -i hsflowd
    hsflowd-1.22.2-1

    This is an issue when we put both RPMs on our internal repo as it recognizes them as the same when they are not. Can this be updated or a source RPM made available?

    Thanks!

    Matthew

    ReplyDelete
  7. Good point. We should probably have the KVM (libvirt) binary appear with a different rpm package name.

    In the short term you can probably do this yourself. Just download the source tarball edit the trunk/hsflowd.spec file to change the name, and then run "make rpm". I haven't tried this myself so let me know if you run into any problems.

    Neil


    ReplyDelete
  8. Neil,

    I was able to get a KVM version built once I performed the "make rpm" on a KVM host and renamed the tarball appropriately. Thank you for the help.

    We are going to move forward with rolling out hsflowd to all our KVM and non-KVM hosts tomorrow. I appreciate you responding so quickly today.

    Matthew

    ReplyDelete
  9. Hi Neil and Peter,
    We converted the KVM RPM package to a debian package and installed on a KVM. We setup an ganglia instance and were able to send the sFlow data to ganglia and displayed by gweb. However, most VM disk related data metrics have no value and the syslog has entries ".... hsflowd: virStorageLookupByPath(/var/lib/nova/instances/instance-00000053/disk) failed".

    Any suggestions in fixing the issue?

    Thanks.
    Yuantai

    ReplyDelete
    Replies
    1. Yuantai,

      There is another way to get disk stats. Did you try compiling with:

      make VRTDSKPATH=yes

      ?

      Although this still doesn't fill in all the fields. See the source code in hsflowd.c. Some of the counters we want just didn't seem to be accessible via libvirt.

      Neil

      Delete
    2. Neil,
      Thanks for your quick response.

      I was hoping if we could get this to work with the package.

      Will thy the source code.

      Have a good weekend.
      Yuantai

      Delete
  10. hello,
    I try to get the openvswitch ports metrics and sflow don't sent any metric to ganglia, what can I do to monitor the traffic for each port of openvswitch
    thanks

    ReplyDelete
  11. Hey Aymen,
    were you able to use ganglia to receive ovs metrics?

    ReplyDelete
    Replies
    1. Ganglia doesn't display the per Open vSwitch port counters (or packet samples), it just displays the virtual machine and hypervisor data.

      Did you start sflowovsd? You should be receiving sFlow data from Open vSwitch. You can verify that the data is being send using sflowtool. There are a number of sFlow analyzers that can handle the Open vSwitch sFlow, but fewer that will handle virtual machine metrics and virtual switch metrics, try: sFlow-RT, or sFlowTrend

      Delete
  12. Hi Guys,

    i try to install host sflow send to graphite server. i manage to get the physical stats being write to graphite but not for instance under KVM using libvirt. did i do something wrong in the configuration? os is there any specific things that i need to enable?

    ReplyDelete
    Replies
    1. Graphite doesn't natively understand sFlow. How are you converting sFlow to Graphite metrics?

      The sflow2graphite script needs to be modified to include virtual machine metrics.

      Alternatively, you could use sFlow-RT, but again, you need to decide which metrics you want to send to graphite, see Metric export to Graphite

      Delete