Duplicate metric error when mapping two MBeans to the same metric with different label

[IncandescentChrysalis]

I have been successfully using the following configuration up until 0.19.0:

  - pattern: '^java.lang<type=Memory><HeapMemoryUsage>committed'
    name: jvm_memory_committed_bytes
    labels:
      area: heap
  - pattern: '^java.lang<type=Memory><NonHeapMemoryUsage>committed'
    name: jvm_memory_committed_bytes
    labels:
      area: nonheap

Trying to upgrade to 1.0.1, I now encounter the following error:

An Exception occurred while scraping metrics: java.lang.IllegalStateException: jvm_memory_committed_bytes: duplicate metric name.
        at io.prometheus.metrics.model.registry.PrometheusRegistry.scrape(PrometheusRegistry.java:75)
        at io.prometheus.metrics.exporter.common.PrometheusScrapeHandler.scrape(PrometheusScrapeHandler.java:112)
        at io.prometheus.metrics.exporter.common.PrometheusScrapeHandler.handleRequest(PrometheusScrapeHandler.java:53)
        at io.prometheus.metrics.exporter.httpserver.MetricsHandler.handle(MetricsHandler.java:43)
        at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:95)
        at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82)
        at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:98)
        at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:851)
        at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:95)
        at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:818)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

How do I add several values to the same metric on distinct labels with 1.0.1?

[dhoard]

@IncandescentChrysalis This appears to be a bug. I have written an integration test and reproduced the issue.

Investigation

The exception is generated from the underlying client_java library.

[duplicate metric name.](https://github.com/prometheus/client_java/blob/ac0a930dd213bc598030af417e58478ba29d669e/prometheus-metrics-model/src/main/java/io/prometheus/metrics/model/registry/PrometheusRegistry.java#L60-L80)

In the exporter, we try to reuse the same builder if the rule name matches.

Workaround

To workaround the issue using 1.0.1 until the bug is resolved, you can use different names for each metric.

rules:
  - pattern: '^java.lang<type=Memory><HeapMemoryUsage>committed'
    name: jvm_memory_committed_heap_bytes
  - pattern: '^java.lang<type=Memory><NonHeapMemoryUsage>committed'
    name: jvm_memory_committed_nonheap_bytes

[IncandescentChrysalis]

Thanks for having taken the time to look into the issue and having improved the tests suite!

The circumvention would wreak havoc in out monitoring system.
The most stable path would be to put the upgrade attempt on hold until this is fixed.

[dhoard]

@IncandescentChrysalis are you using the Java agent or the standalone (HTTP) version?

[dhoard]

Investigation

I have created a branch with a test that reproduces the issues.

Branch

https://github.com/dhoard/jmx_exporter/tree/issue-988

Integration test

https://github.com/dhoard/jmx_exporter/blob/issue-988/integration_test_suite/integration_tests/src/test/java/io/prometheus/jmx/test/Issue988Test.java

Integration test configuration

https://github.com/dhoard/jmx_exporter/tree/issue-988/integration_test_suite/integration_tests/src/test/resources/io/prometheus/jmx/test/Issue988Test

[IncandescentChrysalis]

@IncandescentChrysalis are you using the Java agent or the standalone (HTTP) version?

Java agent

[dhoard]

Investigation

JvmMetrics collector

When using the Java agent exporter, the exporter registers a JvmMetrics collector for JVM metrics...

jmx_exporter/jmx_prometheus_javaagent/src/main/java/io/prometheus/jmx/JavaAgent.java

Line 54 in db0cc44

JvmMetrics.builder().register(PrometheusRegistry.defaultRegistry);

The JvmMetrics collector creates various Jvm metrics...

Reference: https://prometheus.github.io/client_java/instrumentation/jvm/

https://github.com/prometheus/client_java/blob/main/prometheus-metrics-instrumentation-jvm/src/main/java/io/prometheus/metrics/instrumentation/jvm/JvmMemoryMetrics.java

Exporter rules that define overlapping metrics named collected by the JvmMetrics collector will result in an exception...

An Exception occurred while scraping metrics: java.lang.IllegalStateException: jvm_memory_committed_bytes: duplicate metric name.
        at io.prometheus.metrics.model.registry.PrometheusRegistry.scrape(PrometheusRegistry.java:75)
        at io.prometheus.metrics.exporter.common.PrometheusScrapeHandler.scrape(PrometheusScrapeHandler.java:112)
        at io.prometheus.metrics.exporter.common.PrometheusScrapeHandler.handleRequest(PrometheusScrapeHandler.java:53)
        at io.prometheus.metrics.exporter.httpserver.MetricsHandler.handle(MetricsHandler.java:43)
        at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:95)
        at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82)
        at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:98)
        at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:851)
        at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:95)
        at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:818)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

Solution

1. Remove exporter configuration rules that define conflicting metric names. (The JvmCollector is already exposing the metrics.)

2. Use an alternate metric names.

[IncandescentChrysalis]

I am unsure if the status: waiting on feedback tag is targeted at the requester.

If that is the case, my original question stands

How do I add several values [from distinct MBeans] to the same metric on distinct labels with 1.0.1?

As of now, there is a functional regression, upgrading a working configuration achieving this from 0.19.0

[dhoard]

@IncandescentChrysalis The code is working as designed/documented in the 1.0.1 release announcement (https://github.com/prometheus/jmx_exporter/tree/release-1.0.1/docs) and client_java documentation https://prometheus.github.io/client_java/instrumentation/jvm/

Solution

• Remove exporter configuration rules that define conflicting metric names. (The JvmCollector is already exposing the metrics.)

• Use a(n) alternate metric name(s).

• Add additional labels such that metric + labels do not conflict.

[IncandescentChrysalis]

I finally understood what is happening.

The JVM metrics, originally managed by the current exporter directly, is now off-handed to your client_java library.
hence, the exporter at hand does not manage JVM metrics anymore and any definition referencing them shall be removed, generating a conflict otherwise, which is what happened to be my case.
This is a major functional change which is not particularly underlined in the 1.* documentation.
The solution for me was to remove any definition to JVM metrics on my JMX exporter configuration. We were lucky enough not to endure any breaking change, as the metrics & labels name we had defined were exactly the same as the ones client_java generates.

Moreover, there is no control over JVM metrics anymore since you hardcode the use of client_java's JvmMetrics.
If someone wished control over which JVM metrics to expose with 0.*, 1.* will have this ability removed.
This could also be underlined.

[dhoard]

@IncandescentChrysalis previous versions of the exporter used an older version of client_java which exposed JVM metrics but didn't correctly enforce metric uniqueness (metric name + labels)

You could have duplicate metrics with potentially different values.

The change was required to support upcoming OpenTelemetry support and is referenced here...

https://github.com/prometheus/jmx_exporter/tree/release-1.0.1/docs#notes
https://github.com/prometheus/jmx_exporter/releases/tag/1.0.1

... but I'm sure it could be improved.