Monitor JVM Process Remotely

Monitor JVM from command line locally

You can monitor your target JVM process with many tools, say jstat, jinfo, jstack, jmap and so on.

First find out vimid with

jps -l

and then run

jinfo <vimid>

But for the first time, you may get the following errors

Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can’t attach to the process … not permitted …

After some googling, we can slove it by (it’s due to a bug of Linux kernal)

echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

Mointor JVM with graphical tools remotely

Setup JMX RMI

Besides using commands locally, we can also use graphical tools like Jconsole or VisualVM to monitor a JVM process remotely. The latter one is a replacement for the former one.

To enable such remote monitoring, you need to make your remote JVM process to be ready for remote JMX RMI connections.

Here is the official reference.

I summarize my steps as following.

cp JRE_HOME/lib/management/management.properties to home directory

com.sun.management.jmxremote.port=xxx
com.sun.management.jmxremote.ssl=false
com.sun.management.jmxremote.authenticate=true
com.sun.management.jmxremote.password.file=/home/benson/jmxremote.password
com.sun.management.jmxremote.access.file=/home/benson/jmxremote.access
com.sun.management.jmxremote.rmi.port=xxx // this is very important, let's talk about it later

cp JRE_HOME/lib/management/jmxremote.password.template to home directory as jmxremote.password.

monitorRole xxx
controlRole xxx

For security reason, you have to change it as owner readable/writable only

chmod 600 jmxremote.password

cp JRE_HOME/lib/management/jmxremote.access to home directory.

monitorRole   readonly
controlRole   readwrite \
              create javax.management.monitor.*,javax.management.timer.* \
              unregiste

start your jar with your mangement file specified

java -jar -Dcom.sun.management.config.file=$HOME/management.properties -Djava.rmi.server.hostname=xx.xx.xx.xx xxx.jar

Please note that java.rmi.server.hostname is also important here. According to the offical document

By default, the remote stubs for locally created remote objects that are sent to client contains the IP address of the local host in dotted-quad format. For remote stubs to be associated with a specific interface address, the java.rmi.server.hostname system property must be set to IP address of that interface.

In my case, if I start my remote Java VM without java.rmi.server.hostname being specified to the remote public IP, with Jconsole started with -debug, I got error

java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is: 
	java.net.ConnectException: Connection refused (Connection refused)
	at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)

As I mentioned earilier, if you are in a serious production environment(with restrcit firewall rule), you probably need to setup com.sun.management.jmxremote.rmi.port as well.

According to the offical document

setting this property(com.sun.management.jmxremote.port=portNum) publishes an additional RMI connector in a private read-only registry at the specified port using the name, jmxrmi. The port number to which the RMI connector will be bound using the system property:
com.sun.management.jmxremote.rmi.port

If this property is not specified, it will randomly pick an unused port. As a result, if you are using white list for your firewall rule, connections to your remote JVM process will be blocked.

Lastly, please notice that when adding JMX connection in VisualVM, you need to specify com.sun.management.jmxremote.port rather than com.sun.management.jmxremote.rmi.port. Then former one should be where the RMI registry listen to.

Connect to remote server with jvisualvm

It’s straight forward to setup your jvisualvm client. Just note that if you follow my preivous steps, you need to specify the credentials configed in file jmxremote.password.

Setup Jstatd (Optional)

Alike to JMX, Jstatd also provides you ability to monitor JVM. But it’s much more limited. There is a comparison between JMX and Jstatd here.

In average cases, JMX is enough for you. The reason why I need to setup Jstatd is that Visual GC plugin is based on it. The heap graph coming with raw VisualVM is too simple to do meanful JVM memory monitoring.

Starting Jstatd directly will give you error

Could not create remote object
access denied ("java.util.PropertyPermission" "java.rmi.server.ignoreSubClasses" "write")
java.security.AccessControlException: access denied ("java.util.PropertyPermission" "java.rmi.server.ignoreSubClasses" "write")
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
        ...

You can refer to the offical document Jstatd and Policy File Syntax.

Basically, you need to first setup a policy file

grant codebase "file:${java.home}/../lib/tools.jar" {
   permission java.security.AllPermission;
};

The above config uses Property Expansion. Please note that

If a property can’t be expanded in a grant entry, permission entry, or keystore entry, that entry is ignored.

Then start jstatd with

jstatd -J-Djava.security.policy=/home/benson/jmxremote/jstatd.all.policy  -J-Djava.rmi.server.hostname=xxx

you may also specify -J-Djava.rmi.server.logCalls=true for debugging purpose.
Please note that you need to specify java.rmi.server.hostname as before.

By default, the above command will start jstatd with an internal RMI registry, bound to the default port 1099. Besides, it will also randomly pick a port for the RMI connector.

Note: If your JVM process is behind a firewall, you need to take special care to have this random port open to your monitor client.