Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Elasticsearch - FATAL ERROR in native method: processing of -javaagent failed, processJavaStart failed #850

Closed
real747-david opened this issue Jul 21, 2023 · 24 comments
Closed

Elasticsearch - FATAL ERROR in native method: processing of -javaagent failed, processJavaStart failed #850

real747-david opened this issue Jul 21, 2023 · 24 comments
Assignees
@dhoard
Labels
theme: configuration theme: third party issue

Comments

@real747-david
Copy link

config.yaml

rules:

  • pattern: ".*"

ES using bundled JDK

openjdk version "18.0.1.1" 2022-04-22
OpenJDK Runtime Environment (build 18.0.1.1+2-6)
OpenJDK 64-Bit Server VM (build 18.0.1.1+2-6, mixed mode, sharing)

add "-javaagent:/elk/jmx_prometheus_javaagent-0.19.0.jar=12345:/elk/config.yaml" in elasticsearch 8.3.2 jvm.options , start ES and got error:

Exception in thread "main" java.lang.reflect.InvocationTargetException
FATAL ERROR in native method: processing of -javaagent failed, processJavaStart failed
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:119)
at java.base/java.lang.reflect.Method.invoke(Method.java:577)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:491)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:503)
Caused by: java.lang.NoClassDefFoundError: com/sun/net/httpserver/Authenticator
at io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:39)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
... 3 more
Caused by: java.lang.ClassNotFoundException: com.sun.net.httpserver.Authenticator
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 5 more
*** java.lang.instrument ASSERTION FAILED ***: "result" with message agent load/premain call failed at open/src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 422
@dhoard
Copy link
Collaborator

dhoard commented Jul 22, 2023

After some testing, I believe this is an issue with Elasticsearch/Docker image.

  1. Testing the integration test suite with a Docker image based on elasticseach:8.3.2 passes.

  2. Setting "ES JAVA_OPTS` doesn't work...

ES_JAVA_OPTIS=-javaagent:/opt/prometheus/jmx_prometheus_javaagent-0.19.0.jar=12345:/opt/prometheus/exporter.yaml
  1. Setting CLI_JAVA_OPTS appears to partially work (I see JVM metrics, but no Elasticsearch MBeans...
CLI_JAVA_OPTIS=-javaagent:/opt/prometheus/jmx_prometheus_javaagent-0.19.0.jar=12345:/opt/prometheus/exporter.yaml

@real747-david
Copy link
Author

I have add env in shell. and ES can startup, but there are no port 12345 listening.
[elk@elk1 ~]$ export CLI_JAVA_OPTIS=-javaagent:/elk/jmx_prometheus_javaagent-0.19.0.jar=12345:/elk/config.yaml

[elk@elk1 ~]$ netstat -nltp
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp6 0 0 192.168.188.121:9200 :::* LISTEN 1827/java
tcp6 0 0 192.168.188.121:9300 :::* LISTEN 1827/java
tcp6 0 0 :::22 :::* LISTEN -
[elk@elk1 ~]$

@dhoard
Copy link
Collaborator

dhoard commented Jul 22, 2023

[elk@elk1 ~]$ export CLI_JAVA_OPTIS=-javaagent:/elk/jmx_prometheus_javaagent-0.19.0.jar=12345:/elk/config.yaml

You have a typo in your variable name CLI_JAVA_OPTIS should be CLI_JAVA_OPTS

@real747-david
Copy link
Author

more testing, should be set env name with ES_JAVA_OPTS, but not ES_JAVA_OPTIS, but I got same errors. and I just use tarball version ES, not docker image. I have also test ES 8.6, same errors. apprecaite you give more clue and more help.

[elk@elk1 ~]$ export ES_JAVA_OPTS="$ES_JAVA_OPTS -javaagent:/elk/jmx_prometheus_javaagent-0.19.0.jar=12345:/elk/config.yaml"
[elk@elk1 ~]$
[elk@elk1 ~]$

[elk@elk1 ~]$ /elk/ELK/elasticsearch-8.3.2/bin/elasticsearch -d
warning: ignoring JAVA_HOME=/elk/ELK/zulu8.70.0.23-ca-jdk8.0.372-linux_x64; using bundled JDK
FATAL ERROR in native method: processing of -javaagent failed, processJavaStart failed
Exception in thread "main" java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:119)
at java.base/java.lang.reflect.Method.invoke(Method.java:577)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:491)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:503)
Caused by: java.lang.NoClassDefFoundError: com/sun/net/httpserver/Authenticator
at io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:39)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
... 3 more
Caused by: java.lang.ClassNotFoundException: com.sun.net.httpserver.Authenticator
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 5 more
*** java.lang.instrument ASSERTION FAILED ***: "result" with message agent load/premain call failed at open/src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 422
[elk@elk1 ~]$

@real747-david
Copy link
Author

interesting test result. CLI_JAVA_OPTS works, but I can temporarily see jvm metrics on port 12345 during the ES starting stage.

export CLI_JAVA_OPTS="$CLI_JAVA_OPTS -javaagent:/elk/jmx_prometheus_javaagent-0.19.0.jar=12345:/elk/config.yaml"

you can see 12345 is listening below:

[root@elk1 ~]# netstat -nltp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1047/sshd
tcp6 0 0 192.168.188.121:9300 :::* LISTEN 1916/java
tcp6 0 0 :::22 :::* LISTEN 1047/sshd
tcp6 0 0 :::12345 :::* LISTEN 1845/java

but 12345 port disappear after port 9200 listening. and I could not access metrics. any clues, Thank you in advance.
[root@elk1 ~]# netstat -nltp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1047/sshd
tcp6 0 0 192.168.188.121:9300 :::* LISTEN 1750/java
tcp6 0 0 :::22 :::* LISTEN 1047/sshd
tcp6 0 0 192.168.188.121:9200 :::* LISTEN 1750/java
[root@elk1 ~]#

@real747-david
Copy link
Author

and I try ES 7.17.11, 12345 port can startup, but I could not access the metrics, no firewall open.

[elk@elk1 ~]$ export ES_JAVA_OPTS="$ES_JAVA_OPTS -javaagent:/elk/jmx_prometheus_javaagent-0.19.0.jar=12345:/elk/config.yaml"
[elk@elk1 ~]$ /elk/ELK/elasticsearch-7.17.11/bin/elasticsearch -d

[elk@elk1 ~]$ netstat -nltp
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp6 0 0 192.168.188.121:9200 :::* LISTEN 6891/java
tcp6 0 0 192.168.188.121:9300 :::* LISTEN 6891/java
tcp6 0 0 :::22 :::* LISTEN -
tcp6 0 0 :::12345 :::* LISTEN 6891/java
[elk@elk1 ~]$

[root@elk1 ~]# curl http://elk1:9200
{
"name" : "elk1",
"cluster_name" : "test-elk",
"cluster_uuid" : "MSc7FO-uR0qrOjIvPUCZIg",
"version" : {
"number" : "7.17.11",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "eeedb98c60326ea3d46caef960fb4c77958fb885",
"build_date" : "2023-06-23T05:33:12.261262042Z",
"build_snapshot" : false,
"lucene_version" : "8.11.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
[root@elk1 ~]#
[root@elk1 ~]#
[root@elk1 ~]# curl http://elk1:12345/metrics
curl: (56) Recv failure: Connection reset by peer
[root@elk1 ~]#

@dhoard
Copy link
Collaborator

dhoard commented Jul 22, 2023

Can create/publish a Docker image or provide a Dockerfile that can be used for testing/reproducing the issue?

@real747-david
Copy link
Author

I do not have docker evnironment. just download the ES from
https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.3.2-linux-x86_64.tar.gz

and jmx_exporter from https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.19.0/jmx_prometheus_javaagent-0.19.0.jar

unzip the ES in somewhere and run below command, you may see the errors same with me (ES_JAVA_OPTS).

export CLI_JAVA_OPTS="$CLI_JAVA_OPTS -javaagent:/elk/jmx_prometheus_javaagent-0.19.0.jar=12345:/elk/config.yaml"
/elk/ELK/elasticsearch-8.3.2/bin/elasticsearch -d

appreciate further help.

@dhoard
Copy link
Collaborator

dhoard commented Jul 22, 2023

I just set up a new CentOS Stream 9 VM and followed your instructions - works as expected/no issues.

@real747-david
Copy link
Author

ok, my OS is redhat 8.4, but I do not think the issue related to OS level. May I have your config.yaml content.
and also result of command :

netstat -nltp

@dhoard
Copy link
Collaborator

dhoard commented Jul 22, 2023 

[administrator@centos-stream-9 ~]$ cat /elk/config.yaml 
rules:
  - pattern: ".*"

[administrator@centos-stream-9 ~]$ netstat -nltp
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -                   
tcp6       0      0 ::1:9300                :::*                    LISTEN      16704/java          
tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN      16704/java          
tcp6       0      0 :::12345                :::*                    LISTEN      16640/java          
tcp6       0      0 :::9200                 :::*                    LISTEN      16704/java          
tcp6       0      0 :::22                   :::*                    LISTEN      -

[administrator@centos-stream-9 ~]$ curl centos-stream-9:12345
# HELP jvm_classes_currently_loaded The number of classes that are currently loaded in the JVM
# TYPE jvm_classes_currently_loaded gauge
jvm_classes_currently_loaded 4152.0
# HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution
# TYPE jvm_classes_loaded_total counter
jvm_classes_loaded_total 4152.0
# HELP jvm_classes_unloaded_total The total number of classes that have been unloaded since the JVM has started execution
# TYPE jvm_classes_unloaded_total counter
jvm_classes_unloaded_total 0.0
# HELP jmx_config_reload_success_total Number of times configuration have successfully been reloaded.
# TYPE jmx_config_reload_success_total counter
jmx_config_reload_success_total 0.0
# HELP jvm_memory_pool_allocated_bytes_total Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.
# TYPE jvm_memory_pool_allocated_bytes_total counter
jvm_memory_pool_allocated_bytes_total{pool="Eden Space",} 7.5694864E7
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'profiled nmethods'",} 5271168.0
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-profiled nmethods'",} 1173248.0
jvm_memory_pool_allocated_bytes_total{pool="Compressed Class Space",} 1904904.0
jvm_memory_pool_allocated_bytes_total{pool="Metaspace",} 1.6362432E7
jvm_memory_pool_allocated_bytes_total{pool="Tenured Gen",} 1.4459224E7
jvm_memory_pool_allocated_bytes_total{pool="Survivor Space",} 2369304.0
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-nmethods'",} 1287936.0

(truncated output)

@real747-david
Copy link
Author

May I know how do you startup the es, Thanks.

@dhoard
Copy link
Collaborator

dhoard commented Jul 22, 2023 

/elk/ELK/elasticsearch-8.3.2/bin/elasticsearch

@real747-david
Copy link
Author

found reason why 12345 port disapper. I run elasticsearch with -d to put es in background, so the agent process gone.

so I run below to solve the issue:
nohup /home/elk/elasticsearch-8.8.2/bin/elasticsearch > /dev/null &

Thank you so much. really appreciate.

@dhoard
Copy link
Collaborator

dhoard commented Jul 22, 2023

The exporter agent is loaded as part of the running JVM - it will stay running as long as the process loading it is running.

I would create an issue/consult the Elasticsearch project for guidance on how to use a Java agent when running Elasticsearch in daemon mode.

@dhoard dhoard closed this as completed Jul 22, 2023
@dhoard dhoard mentioned this issue Aug 1, 2023
Closed
@real747-david
Copy link
Author

found another problem, the jmx exporter do not get jvm metrics from ES, ES GC is G1, but I could not found any G1 pool metrics. any idea, thanks.

[elk@alma elasticsearch-8.8.2]$ ps -ef |grep elastic |grep -i g1
elk 32391 32316 1 13:31 pts/0 00:00:28 /home/elk/elasticsearch-8.8.2/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=allow -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=SPI,COMPAT --add-opens=java.base/java.io=org.elasticsearch.preallocate -XX:+UseG1GC -Djava.io.tmpdir=/tmp/elasticsearch-14104347288580956251 -XX:+HeapDumpOnOutOfMemoryError -XX:+ExitOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m -Dcom.sun.management.jmxremote.port=6543 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Xms3870m -Xmx3870m -XX:MaxDirectMemorySize=2028994560 -XX:G1HeapRegionSize=4m -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=15 -Des.distribution.type=tar --module-path /home/elk/elasticsearch-8.8.2/lib --add-modules=jdk.net --add-modules=org.elasticsearch.preallocate -m org.elasticsearch.server/org.elasticsearch.bootstrap.Elasticsearch
[elk@alma elasticsearch-8.8.2]$

metric on port 12345

HELP jvm_memory_pool_bytes_max Max bytes of a given JVM memory pool.

TYPE jvm_memory_pool_bytes_max gauge

jvm_memory_pool_bytes_max{pool="CodeHeap 'non-nmethods'",} 5828608.0
jvm_memory_pool_bytes_max{pool="Metaspace",} -1.0
jvm_memory_pool_bytes_max{pool="Tenured Gen",} 4.4761088E7
jvm_memory_pool_bytes_max{pool="CodeHeap 'profiled nmethods'",} 1.22912768E8
jvm_memory_pool_bytes_max{pool="Eden Space",} 1.7891328E7
jvm_memory_pool_bytes_max{pool="Survivor Space",} 2228224.0
jvm_memory_pool_bytes_max{pool="Compressed Class Space",} 1.073741824E9
jvm_memory_pool_bytes_max{pool="CodeHeap 'non-profiled nmethods'",} 1.22916864E8

@dhoard
Copy link
Collaborator

dhoard commented Aug 2, 2023 •
edited
Loading

Looking at the command, I don't see the Java agent exporter configured but I do see external JMX configuration values.

Are you using jmx_prometheus_httpserver.jar as a separate process?

@real747-david
Copy link
Author

real747-david commented Aug 3, 2023 •
edited
Loading

I am not using jmx_prometheus_httpserver.jar, external JMX configuration just for learnning MBean.
based on this case solution, I just create a start.sh to run ES

[elk@alma elasticsearch-8.8.2]$ cat start.sh

#!/bin/bash
export CLI_JAVA_OPTS="$CLI_JAVA_OPTS -javaagent:/home/elk/jmx_prometheus_javaagent-0.19.0.jar=12345:/home/elk/config.yaml"
/home/elk/elasticsearch-8.8.2/bin/elasticsearch &

found ES running process as sub process under jmx-exporter one.

[elk@alma ~]$ netstat -nltp
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:8125 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:19999 0.0.0.0:* LISTEN -
tcp6 0 0 :::6543 :::* LISTEN 73152/java
tcp6 0 0 :::9200 :::* LISTEN 73152/java
tcp6 0 0 127.0.0.1:9300 :::* LISTEN 73152/java
tcp6 0 0 ::1:9300 :::* LISTEN 73152/java
tcp6 0 0 :::22 :::* LISTEN -
tcp6 0 0 :::38615 :::* LISTEN 73152/java
tcp6 0 0 :::12345 :::* LISTEN 73074/java
tcp6 0 0 :::34077 :::* LISTEN 73152/java
tcp6 0 0 ::1:8125 :::* LISTEN -
[elk@alma ~]$
[elk@alma ~]$
[elk@alma ~]$ ps -ef |grep 73074
elk 73074 1 0 11:50 pts/0 00:00:04 /home/elk/elasticsearch-8.8.2/jdk/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -javaagent:/home/elk/jmx_prometheus_javaagent-0.19.0.jar=12345:/home/elk/config.yaml -Dcli.name=server -Dcli.script=/home/elk/elasticsearch-8.8.2/bin/elasticsearch -Dcli.libs=lib/tools/server-cli -Des.path.home=/home/elk/elasticsearch-8.8.2 -Des.path.conf=/home/elk/elasticsearch-8.8.2/config -Des.distribution.type=tar -cp /home/elk/elasticsearch-8.8.2/lib/:/home/elk/elasticsearch-8.8.2/lib/cli-launcher/ org.elasticsearch.launcher.CliToolLauncher
elk 73152 73074 1 11:50 pts/0 00:00:29 /home/elk/elasticsearch-8.8.2/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=allow -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=SPI,COMPAT --add-opens=java.base/java.io=org.elasticsearch.preallocate -XX:+UseG1GC -Djava.io.tmpdir=/tmp/elasticsearch-1812014175294261083 -XX:+HeapDumpOnOutOfMemoryError -XX:+ExitOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m -Dcom.sun.management.jmxremote.port=6543 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Xms3870m -Xmx3870m -XX:MaxDirectMemorySize=2028994560 -XX:G1HeapRegionSize=4m -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=15 -Des.distribution.type=tar --module-path /home/elk/elasticsearch-8.8.2/lib --add-modules=jdk.net --add-modules=org.elasticsearch.preallocate -m org.elasticsearch.server/org.elasticsearch.bootstrap.Elasticsearch
elk 75445 75388 0 12:17 pts/3 00:00:00 grep --color=auto 73074
[elk@alma ~]$

And may I have any update in the issue/consult how to use java agent when running in daemon mode.

@dhoard
Copy link
Collaborator

dhoard commented Aug 3, 2023

Interesting... can you confirm the Java version you are using? (I belive that Elasticsearch bundles a JVM.)

@dhoard
Copy link
Collaborator

dhoard commented Aug 3, 2023

Using the JVM bundled with 8.8.2 and using an exporter.yaml with...

rules:
  - pattern: ".*"

The results are...

jvm_memory_pool_bytes_used{pool="G1 Eden Space",} 0.0
jvm_memory_pool_bytes_used{pool="G1 Old Gen",} 4173824.0
jvm_memory_pool_bytes_used{pool="G1 Survivor Space",} 2439864.0
jvm_memory_pool_bytes_committed{pool="G1 Eden Space",} 8388608.0
jvm_memory_pool_bytes_committed{pool="G1 Old Gen",} 1.6777216E7
jvm_memory_pool_bytes_committed{pool="G1 Survivor Space",} 4194304.0
jvm_memory_pool_bytes_max{pool="G1 Eden Space",} -1.0
jvm_memory_pool_bytes_max{pool="G1 Old Gen",} 1.34217728E8
jvm_memory_pool_bytes_max{pool="G1 Survivor Space",} -1.0
jvm_memory_pool_bytes_init{pool="G1 Eden Space",} 4194304.0
jvm_memory_pool_bytes_init{pool="G1 Old Gen",} 1.2582912E7
jvm_memory_pool_bytes_init{pool="G1 Survivor Space",} 0.0
jvm_memory_pool_collection_used_bytes{pool="G1 Eden Space",} 0.0
jvm_memory_pool_collection_used_bytes{pool="G1 Old Gen",} 4173824.0
jvm_memory_pool_collection_used_bytes{pool="G1 Survivor Space",} 2439864.0
jvm_memory_pool_collection_committed_bytes{pool="G1 Eden Space",} 8388608.0
jvm_memory_pool_collection_committed_bytes{pool="G1 Old Gen",} 1.6777216E7
jvm_memory_pool_collection_committed_bytes{pool="G1 Survivor Space",} 4194304.0
jvm_memory_pool_collection_max_bytes{pool="G1 Eden Space",} -1.0
jvm_memory_pool_collection_max_bytes{pool="G1 Old Gen",} 1.34217728E8
jvm_memory_pool_collection_max_bytes{pool="G1 Survivor Space",} -1.0
jvm_memory_pool_collection_init_bytes{pool="G1 Eden Space",} 4194304.0
jvm_memory_pool_collection_init_bytes{pool="G1 Old Gen",} 1.2582912E7
jvm_memory_pool_collection_init_bytes{pool="G1 Survivor Space",} 0.0
jvm_memory_pool_allocated_bytes_total{pool="G1 Old Gen",} 4173824.0
jvm_memory_pool_allocated_bytes_total{pool="G1 Eden Space",} 8388608.0
jvm_memory_pool_allocated_bytes_total{pool="G1 Survivor Space",} 2439864.0
jvm_gc_collection_seconds_count{gc="G1 Young Generation",} 3.0
jvm_gc_collection_seconds_sum{gc="G1 Young Generation",} 0.005
jvm_gc_collection_seconds_count{gc="G1 Concurrent GC",} 2.0
jvm_gc_collection_seconds_sum{gc="G1 Concurrent GC",} 0.0
jvm_gc_collection_seconds_count{gc="G1 Old Generation",} 0.0
jvm_gc_collection_seconds_sum{gc="G1 Old Generation",} 0.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_used{type="GarbageCollector",key="CodeHeap 'profiled nmethods'",} 1418240.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_used{type="GarbageCollector",key="G1 Old Gen",} 4173824.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_used{type="GarbageCollector",key="CodeHeap 'non-profiled nmethods'",} 261632.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_used{type="GarbageCollector",key="G1 Survivor Space",} 2005952.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_used{type="GarbageCollector",key="Compressed Class Space",} 525800.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_used{type="GarbageCollector",key="Metaspace",} 4788840.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_used{type="GarbageCollector",key="G1 Eden Space",} 0.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_used{type="GarbageCollector",key="CodeHeap 'non-nmethods'",} 3311104.0
java_lang_G1_Eden_Space_PeakUsage_init{type="MemoryPool",} 4194304.0
java_lang_G1_Survivor_Space_PeakUsage_max{type="MemoryPool",} -1.0
java_lang_G1_Concurrent_GC_CollectionTime{type="GarbageCollector",} 0.0
java_lang_G1_Survivor_Space_CollectionUsage_init{type="MemoryPool",} 0.0
java_lang_G1_Survivor_Space_CollectionUsageThreshold{type="MemoryPool",} 0.0
java_lang_G1_Survivor_Space_Usage_init{type="MemoryPool",} 0.0
java_lang_G1_Eden_Space_Valid{type="MemoryPool",} 1.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_used{type="GarbageCollector",key="CodeHeap 'profiled nmethods'",} 2295168.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_used{type="GarbageCollector",key="G1 Old Gen",} 4908032.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_used{type="GarbageCollector",key="CodeHeap 'non-profiled nmethods'",} 396672.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_used{type="GarbageCollector",key="G1 Survivor Space",} 3488240.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_used{type="GarbageCollector",key="Compressed Class Space",} 715752.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_used{type="GarbageCollector",key="Metaspace",} 6498144.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_used{type="GarbageCollector",key="G1 Eden Space",} 0.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_used{type="GarbageCollector",key="CodeHeap 'non-nmethods'",} 3313664.0
java_lang_G1_Eden_Space_Usage_max{type="MemoryPool",} -1.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_committed{type="GarbageCollector",key="CodeHeap 'profiled nmethods'",} 2555904.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_committed{type="GarbageCollector",key="G1 Old Gen",} 1.6777216E7
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_committed{type="GarbageCollector",key="CodeHeap 'non-profiled nmethods'",} 2555904.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_committed{type="GarbageCollector",key="G1 Survivor Space",} 4194304.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_committed{type="GarbageCollector",key="Compressed Class Space",} 655360.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_committed{type="GarbageCollector",key="Metaspace",} 5046272.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_committed{type="GarbageCollector",key="G1 Eden Space",} 8388608.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_committed{type="GarbageCollector",key="CodeHeap 'non-nmethods'",} 3342336.0
java_lang_G1_Concurrent_GC_LastGcInfo_GcThreadCount{type="GarbageCollector",} 5.0
java_lang_G1_Survivor_Space_Usage_committed{type="MemoryPool",} 4194304.0
java_lang_G1_Concurrent_GC_LastGcInfo_endTime{type="GarbageCollector",} 102.0
java_lang_G1_Concurrent_GC_LastGcInfo_id{type="GarbageCollector",} 2.0
java_lang_G1_Old_Gen_CollectionUsageThresholdSupported{type="MemoryPool",} 1.0
java_lang_G1_Concurrent_GC_LastGcInfo_duration{type="GarbageCollector",} 0.0
java_lang_G1_Young_Generation_LastGcInfo_GcThreadCount{type="GarbageCollector",} 5.0
java_lang_G1_Survivor_Space_Usage_max{type="MemoryPool",} -1.0
java_lang_G1_Survivor_Space_UsageThresholdSupported{type="MemoryPool",} 0.0
java_lang_G1_Old_Gen_UsageThresholdCount{type="MemoryPool",} 0.0
java_lang_G1_Eden_Space_CollectionUsageThresholdCount{type="MemoryPool",} 0.0
java_lang_G1_Eden_Space_CollectionUsageThresholdExceeded{type="MemoryPool",} 0.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_used{type="GarbageCollector",key="CodeHeap 'profiled nmethods'",} 1418240.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_used{type="GarbageCollector",key="G1 Old Gen",} 4173824.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_used{type="GarbageCollector",key="CodeHeap 'non-profiled nmethods'",} 261632.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_used{type="GarbageCollector",key="G1 Survivor Space",} 2005952.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_used{type="GarbageCollector",key="Compressed Class Space",} 525800.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_used{type="GarbageCollector",key="Metaspace",} 4788840.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_used{type="GarbageCollector",key="G1 Eden Space",} 0.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_used{type="GarbageCollector",key="CodeHeap 'non-nmethods'",} 3311104.0
java_lang_G1_Old_Gen_Usage_used{type="MemoryPool",} 4908032.0
java_lang_G1_Old_Gen_UsageThresholdSupported{type="MemoryPool",} 1.0
java_lang_G1_Eden_Space_CollectionUsage_used{type="MemoryPool",} 0.0
java_lang_G1_Young_Generation_LastGcInfo_duration{type="GarbageCollector",} 2.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_committed{type="GarbageCollector",key="CodeHeap 'profiled nmethods'",} 2555904.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_committed{type="GarbageCollector",key="G1 Old Gen",} 1.6777216E7
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_committed{type="GarbageCollector",key="CodeHeap 'non-profiled nmethods'",} 2555904.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_committed{type="GarbageCollector",key="G1 Survivor Space",} 4194304.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_committed{type="GarbageCollector",key="Compressed Class Space",} 786432.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_committed{type="GarbageCollector",key="Metaspace",} 6684672.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_committed{type="GarbageCollector",key="G1 Eden Space",} 8388608.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_committed{type="GarbageCollector",key="CodeHeap 'non-nmethods'",} 3407872.0
java_lang_G1_Old_Gen_CollectionUsage_max{type="MemoryPool",} 1.34217728E8
java_lang_G1_Old_Gen_Usage_max{type="MemoryPool",} 1.34217728E8
java_lang_G1_Eden_Space_Usage_used{type="MemoryPool",} 0.0
java_lang_G1_Survivor_Space_CollectionUsageThresholdSupported{type="MemoryPool",} 1.0
java_lang_G1_Old_Gen_Usage_committed{type="MemoryPool",} 1.6777216E7
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_init{type="GarbageCollector",key="CodeHeap 'profiled nmethods'",} 2555904.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_init{type="GarbageCollector",key="G1 Old Gen",} 1.2582912E7
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_init{type="GarbageCollector",key="CodeHeap 'non-profiled nmethods'",} 2555904.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_init{type="GarbageCollector",key="G1 Survivor Space",} 0.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_init{type="GarbageCollector",key="Compressed Class Space",} 0.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_init{type="GarbageCollector",key="Metaspace",} 0.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_init{type="GarbageCollector",key="G1 Eden Space",} 4194304.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_init{type="GarbageCollector",key="CodeHeap 'non-nmethods'",} 2555904.0
java_lang_G1_Old_Gen_PeakUsage_max{type="MemoryPool",} 1.34217728E8
java_lang_G1_Survivor_Space_PeakUsage_used{type="MemoryPool",} 3488240.0
java_lang_G1_Old_Gen_CollectionUsageThresholdExceeded{type="MemoryPool",} 0.0
java_lang_G1_Old_Generation_CollectionTime{type="GarbageCollector",} 0.0
java_lang_G1_Eden_Space_UsageThresholdSupported{type="MemoryPool",} 0.0
java_lang_G1_Old_Gen_PeakUsage_committed{type="MemoryPool",} 2.097152E7
java_lang_G1_Survivor_Space_Valid{type="MemoryPool",} 1.0
java_lang_G1_Eden_Space_CollectionUsageThresholdSupported{type="MemoryPool",} 1.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_max{type="GarbageCollector",key="CodeHeap 'profiled nmethods'",} 1.22023936E8
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_max{type="GarbageCollector",key="G1 Old Gen",} 1.34217728E8
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_max{type="GarbageCollector",key="CodeHeap 'non-profiled nmethods'",} 1.22028032E8
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_max{type="GarbageCollector",key="G1 Survivor Space",} -1.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_max{type="GarbageCollector",key="Compressed Class Space",} 1.073741824E9
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_max{type="GarbageCollector",key="Metaspace",} -1.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_max{type="GarbageCollector",key="G1 Eden Space",} -1.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_max{type="GarbageCollector",key="CodeHeap 'non-nmethods'",} 7606272.0
java_lang_G1_Eden_Space_PeakUsage_committed{type="MemoryPool",} 8388608.0
java_lang_G1_Old_Gen_CollectionUsage_init{type="MemoryPool",} 1.2582912E7
java_lang_G1_Old_Gen_PeakUsage_used{type="MemoryPool",} 4908032.0
java_lang_G1_Young_Generation_CollectionTime{type="GarbageCollector",} 8.0
java_lang_G1_Survivor_Space_CollectionUsage_max{type="MemoryPool",} -1.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_max{type="GarbageCollector",key="CodeHeap 'profiled nmethods'",} 1.22023936E8
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_max{type="GarbageCollector",key="G1 Old Gen",} 1.34217728E8
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_max{type="GarbageCollector",key="CodeHeap 'non-profiled nmethods'",} 1.22028032E8
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_max{type="GarbageCollector",key="G1 Survivor Space",} -1.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_max{type="GarbageCollector",key="Compressed Class Space",} 1.073741824E9
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_max{type="GarbageCollector",key="Metaspace",} -1.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_max{type="GarbageCollector",key="G1 Eden Space",} -1.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_max{type="GarbageCollector",key="CodeHeap 'non-nmethods'",} 7606272.0
java_lang_G1_Old_Gen_Valid{type="MemoryPool",} 1.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_max{type="GarbageCollector",key="CodeHeap 'profiled nmethods'",} 1.22023936E8
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_max{type="GarbageCollector",key="G1 Old Gen",} 1.34217728E8
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_max{type="GarbageCollector",key="CodeHeap 'non-profiled nmethods'",} 1.22028032E8
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_max{type="GarbageCollector",key="G1 Survivor Space",} -1.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_max{type="GarbageCollector",key="Compressed Class Space",} 1.073741824E9
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_max{type="GarbageCollector",key="Metaspace",} -1.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_max{type="GarbageCollector",key="G1 Eden Space",} -1.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_max{type="GarbageCollector",key="CodeHeap 'non-nmethods'",} 7606272.0
java_lang_G1_Old_Gen_CollectionUsage_committed{type="MemoryPool",} 1.6777216E7
java_lang_G1_Old_Generation_CollectionCount{type="GarbageCollector",} 0.0
java_lang_G1_Old_Gen_CollectionUsageThresholdCount{type="MemoryPool",} 0.0
java_lang_G1_Eden_Space_CollectionUsage_max{type="MemoryPool",} -1.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_init{type="GarbageCollector",key="CodeHeap 'profiled nmethods'",} 2555904.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_init{type="GarbageCollector",key="G1 Old Gen",} 1.2582912E7
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_init{type="GarbageCollector",key="CodeHeap 'non-profiled nmethods'",} 2555904.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_init{type="GarbageCollector",key="G1 Survivor Space",} 0.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_init{type="GarbageCollector",key="Compressed Class Space",} 0.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_init{type="GarbageCollector",key="Metaspace",} 0.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_init{type="GarbageCollector",key="G1 Eden Space",} 4194304.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_init{type="GarbageCollector",key="CodeHeap 'non-nmethods'",} 2555904.0
java_lang_G1_Survivor_Space_PeakUsage_committed{type="MemoryPool",} 4194304.0
java_lang_G1_Old_Gen_CollectionUsageThreshold{type="MemoryPool",} 0.0
java_lang_G1_Old_Gen_UsageThresholdExceeded{type="MemoryPool",} 0.0
java_lang_G1_Eden_Space_CollectionUsageThreshold{type="MemoryPool",} 0.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_init{type="GarbageCollector",key="CodeHeap 'profiled nmethods'",} 2555904.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_init{type="GarbageCollector",key="G1 Old Gen",} 1.2582912E7
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_init{type="GarbageCollector",key="CodeHeap 'non-profiled nmethods'",} 2555904.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_init{type="GarbageCollector",key="G1 Survivor Space",} 0.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_init{type="GarbageCollector",key="Compressed Class Space",} 0.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_init{type="GarbageCollector",key="Metaspace",} 0.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_init{type="GarbageCollector",key="G1 Eden Space",} 4194304.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_init{type="GarbageCollector",key="CodeHeap 'non-nmethods'",} 2555904.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_committed{type="GarbageCollector",key="CodeHeap 'profiled nmethods'",} 2555904.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_committed{type="GarbageCollector",key="G1 Old Gen",} 1.6777216E7
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_committed{type="GarbageCollector",key="CodeHeap 'non-profiled nmethods'",} 2555904.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_committed{type="GarbageCollector",key="G1 Survivor Space",} 4194304.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_committed{type="GarbageCollector",key="Compressed Class Space",} 655360.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_committed{type="GarbageCollector",key="Metaspace",} 5046272.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_committed{type="GarbageCollector",key="G1 Eden Space",} 8388608.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageBeforeGc_committed{type="GarbageCollector",key="CodeHeap 'non-nmethods'",} 3342336.0
java_lang_G1_Survivor_Space_PeakUsage_init{type="MemoryPool",} 0.0
java_lang_G1_Survivor_Space_CollectionUsageThresholdCount{type="MemoryPool",} 0.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_used{type="GarbageCollector",key="CodeHeap 'profiled nmethods'",} 2295168.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_used{type="GarbageCollector",key="G1 Old Gen",} 4173824.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_used{type="GarbageCollector",key="CodeHeap 'non-profiled nmethods'",} 396672.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_used{type="GarbageCollector",key="G1 Survivor Space",} 2439864.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_used{type="GarbageCollector",key="Compressed Class Space",} 715752.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_used{type="GarbageCollector",key="Metaspace",} 6498144.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_used{type="GarbageCollector",key="G1 Eden Space",} 4194304.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_used{type="GarbageCollector",key="CodeHeap 'non-nmethods'",} 3313664.0
java_lang_G1_Young_Generation_LastGcInfo_startTime{type="GarbageCollector",} 214.0
java_lang_G1_Eden_Space_Usage_init{type="MemoryPool",} 4194304.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_max{type="GarbageCollector",key="CodeHeap 'profiled nmethods'",} 1.22023936E8
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_max{type="GarbageCollector",key="G1 Old Gen",} 1.34217728E8
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_max{type="GarbageCollector",key="CodeHeap 'non-profiled nmethods'",} 1.22028032E8
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_max{type="GarbageCollector",key="G1 Survivor Space",} -1.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_max{type="GarbageCollector",key="Compressed Class Space",} 1.073741824E9
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_max{type="GarbageCollector",key="Metaspace",} -1.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_max{type="GarbageCollector",key="G1 Eden Space",} -1.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_max{type="GarbageCollector",key="CodeHeap 'non-nmethods'",} 7606272.0
java_lang_G1_Young_Generation_Valid{type="GarbageCollector",} 1.0
java_lang_G1_Eden_Space_CollectionUsage_committed{type="MemoryPool",} 8388608.0
java_lang_G1_Eden_Space_PeakUsage_max{type="MemoryPool",} -1.0
java_lang_G1_Young_Generation_LastGcInfo_endTime{type="GarbageCollector",} 216.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_committed{type="GarbageCollector",key="CodeHeap 'profiled nmethods'",} 2555904.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_committed{type="GarbageCollector",key="G1 Old Gen",} 1.6777216E7
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_committed{type="GarbageCollector",key="CodeHeap 'non-profiled nmethods'",} 2555904.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_committed{type="GarbageCollector",key="G1 Survivor Space",} 4194304.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_committed{type="GarbageCollector",key="Compressed Class Space",} 786432.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_committed{type="GarbageCollector",key="Metaspace",} 6684672.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_committed{type="GarbageCollector",key="G1 Eden Space",} 8388608.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageBeforeGc_committed{type="GarbageCollector",key="CodeHeap 'non-nmethods'",} 3407872.0
java_lang_G1_Old_Generation_Valid{type="GarbageCollector",} 1.0
java_lang_G1_Concurrent_GC_Valid{type="GarbageCollector",} 1.0
java_lang_G1_Survivor_Space_CollectionUsageThresholdExceeded{type="MemoryPool",} 0.0
java_lang_G1_Eden_Space_Usage_committed{type="MemoryPool",} 8388608.0
java_lang_G1_Young_Generation_CollectionCount{type="GarbageCollector",} 4.0
java_lang_G1_Concurrent_GC_LastGcInfo_startTime{type="GarbageCollector",} 102.0
java_lang_G1_Survivor_Space_Usage_used{type="MemoryPool",} 3488240.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_init{type="GarbageCollector",key="CodeHeap 'profiled nmethods'",} 2555904.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_init{type="GarbageCollector",key="G1 Old Gen",} 1.2582912E7
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_init{type="GarbageCollector",key="CodeHeap 'non-profiled nmethods'",} 2555904.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_init{type="GarbageCollector",key="G1 Survivor Space",} 0.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_init{type="GarbageCollector",key="Compressed Class Space",} 0.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_init{type="GarbageCollector",key="Metaspace",} 0.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_init{type="GarbageCollector",key="G1 Eden Space",} 4194304.0
java_lang_G1_Concurrent_GC_LastGcInfo_memoryUsageAfterGc_init{type="GarbageCollector",key="CodeHeap 'non-nmethods'",} 2555904.0
java_lang_G1_Eden_Space_CollectionUsage_init{type="MemoryPool",} 4194304.0
java_lang_G1_Concurrent_GC_CollectionCount{type="GarbageCollector",} 2.0
java_lang_G1_Old_Gen_Usage_init{type="MemoryPool",} 1.2582912E7
java_lang_G1_Survivor_Space_CollectionUsage_committed{type="MemoryPool",} 4194304.0
java_lang_G1_Eden_Space_PeakUsage_used{type="MemoryPool",} 4194304.0
java_lang_G1_Survivor_Space_CollectionUsage_used{type="MemoryPool",} 3488240.0
java_lang_G1_Old_Gen_UsageThreshold{type="MemoryPool",} 0.0
java_lang_G1_Young_Generation_LastGcInfo_id{type="GarbageCollector",} 4.0
java_lang_G1_Old_Gen_PeakUsage_init{type="MemoryPool",} 1.2582912E7
java_lang_G1_Old_Gen_CollectionUsage_used{type="MemoryPool",} 4173824.0
jvm_memory_pool_allocated_bytes_created{pool="G1 Old Gen",} 1.691040458451E9
jvm_memory_pool_allocated_bytes_created{pool="G1 Eden Space",} 1.691040458451E9
jvm_memory_pool_allocated_bytes_created{pool="G1 Survivor Space",} 1.691040458451E9

@dhoard dhoard reopened this Aug 3, 2023
@real747-david
Copy link
Author

yes, using bundles JVM, and I have try same config.yaml. but still could not get G1 metrics. any other differency on my side.

@dhoard
Copy link
Collaborator

dhoard commented Aug 3, 2023

Debugging...

  1. The classes do exist in the JDK shipped with Elasticsearch 8.8.2. (Using the JDK in the integration suite shows the metrics)

  2. setting CLI_CLIENT_OPTS only enables the exporter for the CLI process (not the core Elasticsearch process.) It doesn't appear to use G1CG

  3. adding the exporter to the ES_JAVA_OPTS (per Elasticsearch documentation) fails

Exception in thread "main" java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:491)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:503)
Caused by: java.lang.NoClassDefFoundError: com/sun/net/httpserver/Authenticator
	at io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:39)
	... 6 more
  1. Elasticsearch uses modules.

Did you create an issue/consult the Elasticsearch project for guidance on how to use a Java agent when running Elasticsearch?

@dhoard
Copy link
Collaborator

dhoard commented Aug 3, 2023

This is definitely module related.

If I set ES_JAVA_OPTS ...

export ES_JAVA_OPTS="--add-modules jdk.httpserver --add-exports jdk.httpserver/com.sun.net.httpserver=jdk.httpserver -javaagent:/opt/elasticsearch/jmx_prometheus_javaagent.jar=12345:/opt/elasticsearch/exporter.yaml"

The exporter loads and listens on the specified port, accepts the request, then closes the connection without any output.

@dhoard
Copy link
Collaborator

dhoard commented Aug 3, 2023

@real747-david @Stono I have determined configuration changes in Elasticsearch to allow the jmx_prometheus_javaagent to work correctly.

  1. Environment variable
export ES_JAVA_OPTS="--add-modules jdk.httpserver --add-exports jdk.httpserver/com.sun.net.httpserver=jdk.httpserver -javaagent:/opt/elasticsearch/jmx_prometheus_javaagent.jar=12345:/opt/elasticsearch/exporter.yaml"

where the path to the jar and exporter.yaml is absolute.

  1. JVM / Java policy changes

Edit the Elasticsearch JVM java.policy settings (<elasticsearch directory>/jdk/conf/security/java.policy) adding grants

    permission java.net.SocketPermission "*", "listen, accept, resolve";
    permission java.io.FilePermission "/proc/self/status", "read";
    // Path below must match the location of the exporter.yaml
    permission java.io.FilePermission "/opt/elasticsearch/*", "read";
    permission javax.management.MBeanServerPermission "*";
    permission javax.management.MBeanPermission "*", "*";
    permission java.lang.RuntimePermission "*";

@dhoard dhoard removed the status: new label Aug 3, 2023
@dhoard dhoard changed the title FATAL ERROR in native method: processing of -javaagent failed, processJavaStart failed Elasticsearch - FATAL ERROR in native method: processing of -javaagent failed, processJavaStart failed Aug 4, 2023
@dhoard dhoard self-assigned this May 16, 2024
@dhoard dhoard closed this as completed May 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dhoard dhoard

Labels
theme: configuration theme: third party issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dhoard @real747-david