Xen.org mascot
A previous article talked about Xen Cloud Platform (XCP). The latest XCP 0.5 release includes the Open vSwitch, providing native support for sFlow monitoring of network traffic. This article will describe the steps needed to enable sFlow on the XCP 0.5 release.
First, download and install XCP 0.5 on a server. The simplest way to build an XCP server is use the Base ISO from the Xen Cloud Platform Source page.
The article, Configuring Open vSwitch, describes the steps to manually configure sFlow monitoring. However, manual configuration is not recommended since additional configuration is required every time a bridge is added to the vSwitch, or the system is rebooted.
The Host sFlow agent automatically configures the Open vSwitch and provides additional performance information from the physical and virtual machines (see Cloud-scale performance monitoring).
Download the Host sFlow agent and install it on your XCP server using the following commands:
rpm -Uvh hsflowd_XCP_05-XXX.rpm service hsflowd start service sflowovsd start
The following steps configure all the sFlow agents to sample packets at 1-in-512, poll counters every 20 seconds and send sFlow to an analyzer (10.0.0.50) over UDP using the default sFlow port (6343).
Note: A previous posting discussed the selection of sampling rates.
The default configuration method used for sFlow is DNS-SD; enter the following DNS settings in the site DNS server:
analyzer A 10.0.0.50 _sflow._udp SRV 0 0 6343 analyzer _sflow._udp TXT ( "txtvers=1" "polling=20" "sampling=512" )
Note: These changes must be made to the DNS zone file corresponding to the search domain in the XCP server's /etc/resolv.conf file.
Once the sFlow settings are added to the DNS server, they will be automatically picked up by the Host sFlow agents. If you need to change the sFlow settings, simply change them on the DNS server and the change will automatically be applied to all the XenServer systems in the data center.
Manual configuration is an option if you do not want to use DNS-SD. Edit the Host sFlow agent configuration file, /etc/hsflowd.conf, on each XCP server:
sflow{
DNSSD = off
polling = 20
sampling = 512
collector{
ip = 10.0.0.50
udpport = 6343
}
}
After editing the configuration file you will need to restart the Host sFlow agent:
service hsflowd restart
An sFlow analyzer is needed to receive the sFlow data and report on performance (see Choosing an sFlow analyzer). The free sFlowTrend analyzer is a great way to get started, see sFlowTrend adds server performance monitoring to see examples.
Hello Peter,
ReplyDeleteI installed the free sFlowTrend analyzer but there is the option of adding an sFlow agent as host, only as a switch.
Any help will be appreciated,
Thanks
Deborah,
ReplyDeleteThe description is a little confusing, but there is only one list of agents, including both switches and hosts. If you are using Host sFlow along with the Open vSwitch then each XCP server acts as both a switch and a host.
Once you configure sFlow on the XCP server, you should see the new agent automatically appear in the list of agents in sFlowTrend. sFlowTrend has a flashing green status box in the bottom right corner of the window that indicates that it is receiving sFlow. If you don't see the new agent, it's likely that there is a firewall somewhere between the XCP server and sFlowTrend blocking UDP port 6343.
Peter
Hi again. Peter.
ReplyDeleteI was testing sFlow and realized something. It says that the total host memory is 300MB. The physical machine has 1.5GB. but when on the XCP console and typing free -m. I can see the 300MB. So. am I doing something wrong? Is there a way to see the physical machine info and not the just the dom0?
In addition to exporting detailed statistics for dom0, the sFlow agent also exports statistics for the hypervisor and each virtual machine. These statistics are very similar to what you would obtain from xentop/libxenstat. To answer your specific question, the sFlow vir_node.memory and virt_node.memory_free metrics should give you the information you are looking for.
ReplyDeleteHi Peter,
ReplyDeleteI've looked the sFlow vir_node.memory and virt_node.memory_free metrics and they really provide the information I was looking for.
Thanks a lot for your help
Hi Peter,
ReplyDeletesorry if I'm disturbing you.
I am monitoring the virtual memory utilization through the vMemoryUtil variable and it is always at 100%, no matter which VM.
Using 'free -m' on the VM I can see that there is some free memory left, so I don't think that 100% from sFlow is correct.
Thanks a lot for your help
The sFlow per-VM memory utilization numbers should match the the numbers that you see when running xentop on dom0.
ReplyDeleteThe hypervisor sees 100% memory utilization for Linux VMs. Linux utilizes all the memory available, assigning any otherwise unused memory to the disk cache. If you want a detailed view of the memory assignment within the VM then you should install Host sFlow on the VM.
Peter,
ReplyDeleteI'm confused. First, the numbers from xentop on dom0 don't match with the numbers that i see on SFlow per-VM memory utilization. Second, i installed the Host sFlow on the VM and the numbers collected again don't match with the values available for the "free" command and xentop. The values collected are on link as follow:
http://pastebin.com/52mSHMmi
Thanks for your help!!!
Peter,
ReplyDeletecan i generate reports from sFlowTrend automatically at intervals of time defined previously or it is necessary develop something with the sFlowTool?
Thanks!
Hi Deborah,
ReplyDeleteI'll jump in with a few comments.
- The totals in your example look right, based on how sFlowTrend converts MB to GB. For example, for DOM0 300MB = 300/1024 = 0.29GB. The other totals work out similarly.
- As Peter mentions, the memory utilization of a Linux host, as seen from DOM0, will appear to be 100%.
- What looks strange is the utilization for DOM0 and for the VM with host sFlow installed. When digging into this further, it appears that xentop's reporting of the memory % (eg for DOM0) is the % of physical memory allocated to DOM0 - I would guess that your system has 300M assigned to DOM0, and this is 20% of the physical memory on your system - for a total of 1.5G physical memory. I'm working this out by reverse engineering on my system, does that look right to you? Similarly, 256M at 17% ~ 1.5G. The values reported by sFlowTrend do not relate to physical memory, they relate to how much is used of allocated. Comparing the results of free -m on my DOM0, it seems that the equivalent of 'used' reported by free -m, in sFlowTrend is used+buffers+cache(+shared = 0 in Linux). I think that the sFlowTrend 'used' value corresponds to the number in the '-/+ buffers cache' line in free. The sFlowTrend % numbers are of total (allocated).
In looking at this, we did realise that there is a bug in sFlowTrend - on charts (eg memory for a specific host), the values are not scaled to MB or GB correctly - they are just bytes divided by 1000 etc, rather than 1024 etc, which makes comparing numbers a bit difficult.
How have you been using sFlowTrend? From the summary statistic page, showing all hosts, you can drill down to the chart for a specific host. Once there you can filter on the metrics by clicking on them in the legend. Also, you can use reports (Advanced for each detailed metric) if you would rather get a table of results, or different chart types.
Hope this helps.
Stuart
Hi Deborah,
ReplyDeleteCurrently, sFlowTrend only runs a report when the run report button is pressed, although a future release will include scheduled reports.
If you have any sFlowTrend questions, you can try posting in the sFlowTrend group at http://groups.google.com/group/sflowtrend.
regards,
Stuart
hi,
ReplyDeleteI configure sFlow on the XCP server (The latest XCP 0.5 release), i see the new agent automatically appear in the list of agents in sFlowTrend with green status. The agent is a host.I need to receive traffic from openvswitch (xenbr0) to monitor each virtual machine. I cannot Edit or check sFlow agent address box and Use global SNMP settings box.
In addition I don't see the new agent in Switch selector in the control bar.
The goal is control the rate used by each virtual machine.
Finnaly, what about scheduled reports, does sFlowTrend includ its?
Any help will be appreciated,
Thanks
Manel
To automatically configure sFlow on the underlying Open vSwitch you need to start sflowovsd as well as hsflowd. But maybe you did that already and it still didn't work? sflowovsd has two strategies for configuring Open vSwitch via ovs-vsctl. One that uses the ovs-vsctl "--id=@newsflow" construct to do everything in one command, and one that doesn't. The use of that construct became compulsory somewhere between Open vSwitch version 1.0 and 1.1.0pre1.
ReplyDeleteThe strategy that sflowovsd starts with depends on the output that it gets from "ovs-vsctl --version". Can I ask what you get for that on your hypervisor?
If something goes wrong then it may try to fall back and use the older scheme instead.
A quick way to get to the bottom of this is to stop the daemon and then run it again manually with debugging turned on. Like this:
root> /sbin/service sflowovsd stop
root> sflowovsd -ddd
Please send what you see.
Another possible work-around is just to turn sflowovsd off and issue the ovs-vsctl commands in a startup script. That would be OK if you never want to change the sFlow settings (the advantage of sflowovsd is that it will update the sFlow settings any time they are changed in /etc/hsflowd.conf or via DNS-SD).
Regards,
Neil