[Bug] Failed to generate the JVM identifier #696
Comments
grzesuav
added
bug
|
listed via I have another application with the same module set and it is working fine, no JMV identifier issue and no UI error when taking recording |
|
I guess the error message refers to this JMX module - https://github.com/openjdk/jmc/blob/jmc9/application/org.openjdk.jmc.console.agent/src/main/java/org/openjdk/jmc/console/agent/AgentJmxHelper.java#L55 ? |
These are referring to the JMC Agent integration feature, which is not really related to the JVM ID problem you're experiencing. If you don't have the JMC Agent installed in your application then seeing messages like this is expected behaviour. As for the JVM ID being missing, this could be a connection timing problem. In 3.0, Cryostat tries to connect to the target JVM immediately when it is discovered so that it can compute the JVM's hash ID. Sometimes this means it is actually connecting prematurely - if using Kubernetes Endpoints discovery for example, it's possible that Kubernetes creates the Endpoints object and informs Cryostat about it before the target JVM is actually ready to accept incoming JMX connections, so when Cryostat sees the target and tries to connect then it fails to do so and the ID gets left blank. Later on the target JVM becomes ready and Cryostat is able to connect to it when you try to start a recording or perform other actions, but the ID never gets updated. #689 should also address this kind of bug, if my guess at the behaviour above is accurate in this particular case.
This looks more like a problem: It's not really clear from this what caused the transaction rollback, just that there was some NPE. Some part of the stack trace seems to point to , but I'm not quite sure how something could be null at this point.Is this "Request failed (500)" error consistently reproducible? |
andrewazores
removed
the
needs-triage
|
yes, for the same group where there is no JVM identfiers I got this. On my staging env I updated cryostat to 3.0.1 , will re-check with this version |
|
I don't think 3.0.1 contained any patches that would affect these specific bugs, but I'd be curious to see if it does help somehow. |
|
yeah, it still happening. It is misleading as recording gets started despite the error message 
|
|
Cryostat creates the remote recording over JMX (or Agent HTTP, if you're using the Cryostat Agent instead of JMX) and then creates a database record for itself to keep track of that recording, so what you're seeing looks like Cryostat is able to connect to the target and perform the JMX operation, but then it fails when trying to create the database entry. So the remote recording does actually get created, but the Cryostat API operation and database transaction failed. There was another somewhat related bug some time ago so now when you try to do various API operations on a target, Cryostat connects to the target and tries to list whatever remote recordings may exist on it and synchronize its database to that data at the later time. So it should look like this:
In this case it looks like step 4 fails somehow and results in step 5 being an HTTP 500 response, but everything else is working. The NPE actually makes a lot of sense now that I step through that, and with your observation that this happens specifically to the targets that don't have the JVM ID set: so the database record creation fails because part of that transaction involves emitting a notification that the record has been created, and that notification references the target's JVM ID and requires it to be non-null. This post-persist notification hook therefore fails, and this rolls back the transaction. So if that analysis is correct, then #689 should fix the bug. This won't land in a release until 4.0 however which is months away. We are planning a 3.0.2 release in the meantime as well, but I will need to strip down that patch to something less invasive before I can backport it. In the meantime, you might be able to work around this issue by creating a dummy Stored Credential. If you go to Security and create a credential with a match expression that matches these targets where the JVM ID is not present (or simply use the expression |
Current Behavior
On certain pods I see message
Failed to generate the JVM identifier-I guess it can be related with underlying error in logs
2024-10-24 08:40:47,211 WARNING [io.cry.cor.jmc.JMCAgentJMXHelper] (executor-thread-1537) Could not retrieve event probes: javax.management.InstanceNotFoundException: org.openjdk.jmc.jfr.agent:type=AgentControllerwhen trying to capture recording I see also console issue -
however recording is created fine -
the error seems to be related to
HTTP Request to /api/v3/targets/31/recordings failed, error id: e8c0c380-fdc9-4961-b7b1-fbb46b44b230-4: jakarta.transaction.RollbackException: ARJUNA016053: Could not commit transaction.Attaching logs -
cryostat-Explore-logs-2024-10-24 10_47_03.txt
Expected Behavior
No issue with JVM identifier generation, no issue when creating recording (or more meaningful one)
Steps To Reproduce
No response
Environment
Anything else?
No response
The text was updated successfully, but these errors were encountered: