Friday, September 2, 2011

Java virtual machine


The jmx-sflow-agent project is an open source implementation of sFlow monitoring for Java Virtual Machines (JVM). Instrumenting Java using sFlow provides scalable, real-time monitoring for applications such as Hadoop, Cassandra and Tomcat that typically involve large numbers of virtual machines.

The sFlow Host Structures extension describes a set of standard metrics for monitoring physical and virtual machines. The jmx-sflow-agent extends the set of physical server metrics exported by a Host sFlow agent which exports the physical server metrics.

The jmx-sflow-agent exports the generic virtual machine structures. In addition, the jmx-sflow-agent uses the Java Management Extensions (JMX) interface to collect performance statistics about java threads, heap/non-heap memory, garbage collection, compilation and class loading. These additional metrics are exported along with the generic virtual machine statistics.

The jmx-sflow-agent software is designed to integrate with the Host sFlow agent to provide a complete picture of server performance. Download, install and configure Host sFlow before proceeding to install the tomcat-sflow-valve - see Installing Host sFlow on a Linux Server. There are a number of options for analyzing cluster performance using Host sFlow, including Ganglia and sFlowTrend.

Next, download the sflowagent.jar file from http://jmx-sflow-agent.googlecode.com. Copy the sflowagent.jar file into the same directory as your Java application. Enabling the sFlow instrumentation involves including -javaagent:sflowagent.jar argument when starting the Java application.

The following example shows how instrumentation is added to MyApp:

java -javaagent:sflowagent.jar\
     -Dsflow.hostname=vm.test\
     -Dsflow.uuid=564dd23c-453d-acd2-fa64-85da86713620\
     -Dsflow.dsindex=80001\
     MyApp

Arguments can be passed to the sFlow module as system properties:

  • sflow.hostname, optionally assign a "hostname" to identify the virtual machine. Choose a naming strategy that is helpful in identifying virtual machine instances, for example: "hadoop.node1" etc. The hostname is exported in the sFlow host_descr structure.
  • sflow.uuid, optionally assign a UUID to the virtual machine so that it can be uniquely identified. The UUID is exported in the sFlow host_descr structure.
  • sflow.dsindex, uniquely identifies the data source associated with this virtual machine on this server. The dsindex number only needs to be set if more than one virtual machine is running on the server. For virtual machines offering network services, use TCP/UDP port number associated with the service as the dsindex, otherwise, use numbers in the range 65536-99999 to avoid clashes with other sFlow agents on the server.

The sFlow module shares Host sFlow's configuration and will automatically send data to the set of sFlow analyzers specified in the Host sFlow configuration file, picking up any configuration changes without the need to restart the Java application.

Finally, the real potential of of the Java virtual machine sFlow agent is as part of a broader sFlow monitoring system providing scaleable, real-time, visibility into applications, servers, storage and networking across the entire data center.

9 comments:

  1. Is it possible to have the jmx-sflow-agent report on the sizes of the different memory pools within the JVM? I notice that it only calls ManagementFactory.getMemoryMXBean() to get the overall utilization rather than getMemoryPoolMXBeans() to get a list of the various memory pools. Would be really handy to use sFlow to monitor Perm Gen space, for example.

    ReplyDelete
  2. The jmx-sflow-agent implements the following specification:

    sFlow Java Virtual Machine Structures

    The document defines a JVM independent set of metrics geared toward large scale multi-vendor operational monitoring rather than debugging a particular JVM. If performance problems are detected with a JVM, other tools such as jconsole can be used to inspect the memory pools specific to that JVM.

    ReplyDelete
  3. Hi,
    I have host sflow agent on a windows server,and ganglia on a ubuntu desktop.
    I used jmx-sflow-agent monitoring the jvm on local ubuntu,then ganglia draw graph of jvm;
    But when I tried it on the windows server, ganglia on ubuntu can't get metrics of jvm(the metrics hasn't been created in the rrds directory as other metrics).
    Does anybody meet the problem?
    Thank you very much!

    ReplyDelete
    Replies
    1. The java agent doesn't support Windows - it needs to be able to pick up registry settings and the functionality hasn't been implemented yet.

      Delete
    2. So it is.
      Thanks for your explanation!

      Delete
  4. I am facing the following problem when trying to checkout the code. Please help.

    svn: E175002: Unable to connect to a repository at URL 'http://jmx-sflow-agent.googlecode.com/svn/trunk'
    svn: E175002: Server sent unexpected return value (413 Request Entity Too Large) in response to OPTIONS request for 'http://jmx-sflow-agent.googlecode.com/svn/trunk'

    ReplyDelete
  5. I just tried the following command on a CentOS 5.8 system using subversion 1.6.11 and it seems to be working:

    svn checkout http://jmx-sflow-agent.googlecode.com/svn/trunk/ jmx-sflow-agent-read-only

    Are you using any additional options on the svn command? A different version of subversion?

    ReplyDelete
  6. hi,

    I am using ganglia with hsflow jmx agent. I could not see metrics in ganglia web interface. I configured evertying according to tutorial. Is there a way to check if jmx-sflow-agent send metrics or not with telnet or tcpdump ?

    ReplyDelete
    Replies
    1. You can run sflowtool on the gmond head node to verify that sFlow is arriving and contains the host and java data you expect.

      Are you running Ganglia 3.3+, see Using Ganglia to monitor Java virtual machines for settings.

      Do you have the Host sFlow agent installed as well? Are the core CPU, memory etc metrics showing up in Ganglia?

      Finally the jmx-agent picks up its configuration from a file written by the Host sFlow agent. This mechanism doesn't work on Windows (where the Host sFlow agent writes configuration into the Windows registry). Currently the java agent doesn't read the windows registry and isn't supported on Windows.

      Delete