The Mule Runtime Manager agent JMX service allows you to track specific JMX metrics and publish them to external services. The JMX service sends all the gathered metrics to publishers, which in turn publish the metrics to external monitoring tools [1].

Datadog is a monitoring service which has a lightweight Java plugin named JMXFetch. This plugin is called by the Datadog Agent to connect to the MBean Server and collect your application metrics. This plugin sends metrics to the Datadog Agent using the DogStatsD server running within the Agent[2].

Full monitoring of your mule runtimes is accomplished when you install the Datadog Mule integration into the agent, enables the Datadog APM and gather your mule runtime JVM metrics. These metrics enable a deep dive infrastructure inspection and correlate your instances/servers with the APM services, delivering a holistic view of your customer-hosted runtime planes, everything in a single platform.

JMX metrics at mule runtimes

To open JMX metrics consumption at your mule servers for the Datadog agent, you must run your mule runtime or add the below arguments [3] to your mule instance conf folder in a file named wrapper-additional.conf:

·-M-Dcom.sun.management.jmxremote

o   To allow the JMX client access to the Mule Java VM.

·-M-Dcom.sun.management.jmxremote.port=portNum

o   To enable JVM monitoring of your instance. In the property above, portNum is the port number through which you want to enable JMX RMI connections.

·-M-Dcom.sun.management.jmxremote.local.only=true

o   If set to false, remote system monitoring can be configured but if you are running the Datadog agent in the same instance where your mule runtime is running this is not neccessary.

·-M-Dcom.sun.management.jmxremote.authenticate=false

o   Password authentication for remote monitoring is enabled by default. To disable it, set the following system property when you start the Mule runtime, moreover all traffic is kept locally and no outbound monitoring is configured.

·-M-Dcom.sun.management.jmxremote.ssl=false

o   To use client SSL authentication then set it true, N/A for local monitoring.

·-M-Djava.rmi.server.hostname=host

o   To enable JVM monitoring of your instance. In the property above, host is the local IP address of your server to enable JMX RMI connections.

Note: Above properties, like the host and port, will be used later in the Datadog agent configuration.

After running your Mule runtime with the above configuration, the agent can collect these metrics and report them into the Datadog platform, if your Mule runtime was running and you added the properties as a wrapper-additional.conf a restart of your instance is required, let’s configure the agent to do so.

Setting up the Datadog Agent for JMX fetch

Prerequisites

1.     A Datadog agent running.

2.     Your Mule applications must be customer hosted.

3.     The Datadog agent must be able to communicate with the Mule server. Ensure that the configured host and port for the Mule runtime JMX service are reachable.

Configuration/Steps

·Configure the depicted file at the following link according to your needs and specs, please follow Datadog documentation:

o   https://docs.datadoghq.com/integrations/java/?tab=host#configuration

·Restart the datadog agent

o   Linux:

§  https://docs.datadoghq.com/agent/guide/agent-commands/?tab=agentv6v7#restart-the-agent

o   Windows:

§  https://docs.datadoghq.com/agent/basic_agent_usage/windows/?tab=gui

·Let’s check the JMXFetch is running and gathering our instance(s):

o   Please follow the below document to verify that JMXFetch check is executing correctly:

§  https://docs.datadoghq.com/agent/troubleshooting/agent_check_status/?tab=agentv6v7

§  When configured and started successfully, it should show you something similar like the below image:

 

Visualizing the JMX metrics

Now, let's confirm you are able to see the metrics in Datadog web page. Login into: https://app.datadoghq.com/account/login?next=%2F

·Once there, go to the left menu, select Dashboards and there, look for JVM Metrics or JVM - Overview dashboard.

o   If the JVM Metrics dashboard is selected, then go up into the dashboard, below the dashboard name, and check for the $host variable, here select your host or “*” to show hosts gathered metrics.

o   If the JVM - Overview dashboard is selected, then go up into the dashboard, below the dashboard name, and check for the $scope variable, here select your host or “*” to show hosts gathered metrics.

o   Please find below an image of what it is depicted above:

 

·And that’s it you can visualize your JMX metrics OOB.

 This link contains the list of the metrics you can obtain from JMX.

References

[1] https://docs.mulesoft.com/runtime-manager/jmx-service

[2] https://docs.datadoghq.com/integrations/java/?tab=host#metric-collection

[3] https://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html