Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build(deps): update Quarkus to 3.8 LTS #609

Merged
merged 12 commits into from
Sep 30, 2024

Conversation

andrewazores
Copy link
Member

@andrewazores andrewazores commented Aug 15, 2024

Welcome to Cryostat! 👋

Before contributing, make sure you have:

  • Read the contributing guidelines
  • Linked a relevant issue which this PR resolves
  • Linked any other relevant issues, PR's, or documentation, if any
  • Resolved all conflicts, if any
  • Rebased your branch PR on top of the latest upstream main branch
  • Attached at least one of the following labels to the PR: [chore, ci, docs, feat, fix, test]
  • Signed all commits using a GPG signature

To recreate commits with GPG signature git fetch upstream && git rebase --force --gpg-sign upstream/main


Related to #364
Closes #626

Description of the change:

Update Quarkus to 3.8.

Having some problems: quarkusio/quarkus#42596

Motivation for the change:

This change is helpful because users may want to...

How to manually test:

  1. Run CRYOSTAT_IMAGE=quay.io... bash smoketest.bash...
  2. ...

@andrewazores andrewazores added dependencies Pull requests that update a dependency file chore Refactor, rename, cleanup, etc. safe-to-test labels Aug 15, 2024
@github-actions github-actions bot added the needs-triage Needs thorough attention from code reviewers label Aug 15, 2024
@andrewazores andrewazores added blocked and removed needs-triage Needs thorough attention from code reviewers labels Aug 15, 2024
@andrewazores
Copy link
Member Author

andrewazores commented Sep 12, 2024

Fixed the initial JSON key-value serialization problem, but now more issues:

[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 2024-09-12 16:16:38:761 +0000 [cryostat-agent-worker-2] ERROR io.cryostat.agent.Registration - Failed to generate credentials
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: java.util.concurrent.CompletionException: Could not submit credentials
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at io.cryostat.agent.WebServer.lambda$generateCredentials$1(WebServer.java:125)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1705)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at java.base/java.lang.Thread.run(Thread.java:829)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: Caused by: java.util.concurrent.CompletionException: io.cryostat.agent.RegistrationException: io.cryostat.agent.shaded.org.apache.shaded.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 1,063; received: 1,014)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:645)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	... 8 more
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: Caused by: io.cryostat.agent.RegistrationException: io.cryostat.agent.shaded.org.apache.shaded.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 1,063; received: 1,014)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at io.cryostat.agent.CryostatClient.lambda$queryExistingCredentials$11(CryostatClient.java:233)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	... 8 more
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: Caused by: io.cryostat.agent.shaded.org.apache.shaded.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 1,063; received: 1,014)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at io.cryostat.agent.shaded.org.apache.shaded.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at io.cryostat.agent.shaded.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._loadMore(UTF8StreamJsonParser.java:220)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at io.cryostat.agent.shaded.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._loadMoreGuaranteed(UTF8StreamJsonParser.java:2457)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at io.cryostat.agent.shaded.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2540)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at io.cryostat.agent.shaded.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2520)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at io.cryostat.agent.shaded.com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:294)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at io.cryostat.agent.shaded.com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer._deserializeContainerNoRecursion(JsonNodeDeserializer.java:572)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at io.cryostat.agent.shaded.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer$ObjectDeserializer.deserialize(JsonNodeDeserializer.java:154)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at io.cryostat.agent.shaded.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer$ObjectDeserializer.deserialize(JsonNodeDeserializer.java:126)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at io.cryostat.agent.shaded.com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:342)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at io.cryostat.agent.shaded.com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4905)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at io.cryostat.agent.shaded.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3885)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	at io.cryostat.agent.CryostatClient.lambda$queryExistingCredentials$11(CryostatClient.java:230)
[16:16:38] [cryostat-agent-worker-2/INFO]: [STDERR]: 	... 9 more

on the gameserver JDK 11 and 17 containers. The gameserver JDK 21, vertx, and quarkus agent test application containers seem fine somehow.

@andrewazores
Copy link
Member Author

smoketest.bash -O -t gameserver-jdk17 works fine, so maybe this is not really a JDK version or Agent version problem, but some other kind of conflict arising.

@andrewazores
Copy link
Member Author

$ ./smoketest.bash -O -t gameserver-jdk11,gameserver-jdk17,gameserver-jdk21 is also okay... ?

@andrewazores
Copy link
Member Author

andrewazores commented Sep 12, 2024

Some quick and dirty hacking around to print out the raw response body that the Agent is seeing shows that the message body really does end prematurely:

...
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - _
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - I
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - D
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - \
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - "
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - ]
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient -  
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - =
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - =
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient -  
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - \
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - "
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - 8
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - 3
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - b
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - 5
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - 0
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - b
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - 6
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - a
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - -
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - a
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - f
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - c
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - 9
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - -
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - 4
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - 7
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - d
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - e
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - -
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - 8
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - 2
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - 3
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - 1
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - -
[19:15:20] [cryostat-agent-worker-0/INFO]: [STDERR]: 2024-09-12 19:15:20:752 +0000 [cryostat-agent-worker-0] ERROR io.cryostat.agent.CryostatClient - Unable to parse response as JSON
...

which corresponds to (this is a manually truncated response to illustrate the point - the https response here is really complete and as normal):

$ https --auth=user:pass :8443/api/v2.2/credentials
{
    "data": {
        "result": [
            {
                "id": 3,
                "matchExpression": "target.connectUrl == \"http://vertx-agent-2:8911/\" && target.annotations.platform[\"INSTANCE_ID\"] == \"c7f74de0-e33d-4aef-b805-a04d4bd4564e\"",
                "numMatchingTargets": 1
            },
            {
                "id": 4,
                "matchExpression": "target.connectUrl == \"http://vertx-agent-1:8910/\" && target.annotations.platform[\"INSTANCE_ID\"] == \"beb83ce2-0bd3-450f-bc29-6a02200f995f\"",
                "numMatchingTargets": 1
            },
            {
                "id": 2,
                "matchExpression": "target.connectUrl == \"http://vertx-agent-3:8912/\" && target.annotations.platform[\"INSTANCE_ID\"] == \"e58f1363-3999-470e-89d5-9b3226f9644f\"",
                "numMatchingTargets": 1
            },
            {
                "id": 1,
                "matchExpression": "target.connectUrl == \"https://quarkus-cryostat-agent:9977/\" && target.annotations.platform[\"INSTANCE_ID\"] == \"e7eef7f4-f73c-4c80-8c4a-cb4a47fe75cf\"",
                "numMatchingTargets": 1
            },
            {
                "id": 5,
                "matchExpression": "target.connectUrl == \"http://gameserver-jdk21:9496/\" && target.annotations.platform[\"INSTANCE_ID\"] == \"83b50b6a-afc9-47de-8231-

so the JSON is indeed truncated, for some reason. But the equivalent HTTPie or curl requests get the same Content-Length header and see a full response body. So, maybe this is a bug on the Agent side, closing the response too early?

@andrewazores
Copy link
Member Author

Comparing against the current main Cryostat container (Quarkus 3.2) with curl,

$ curl -vk https://user:pass@localhost:8443/api/v2.2/credentials | wc --bytes
* processing: https://user:pass@localhost:8443/api/v2.2/credentials
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying [::1]:8443...
* Connected to localhost (::1) port 8443
* ALPN: offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [21 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [1493 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [520 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: C=CA; ST=ON; L=Toronto; O=RedHat; OU=JavaMonitoring; CN=Cryostat
*  start date: Sep 12 19:19:14 2024 GMT
*  expire date: Sep 12 19:19:14 2025 GMT
*  issuer: C=CA; ST=ON; L=Toronto; O=RedHat; OU=JavaMonitoring; CN=Cryostat
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
* using HTTP/1.1
* Server auth using Basic with user 'user'
} [5 bytes data]
> GET /api/v2.2/credentials HTTP/1.1
> Host: localhost:8443
> Authorization: Basic dXNlcjpwYXNz
> User-Agent: curl/8.2.1
> Accept: */*
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [122 bytes data]
< HTTP/1.1 200 OK
< Cache-Control: no-cache
< Content-Type: application/json;charset=UTF-8
< Gap-Auth: user
< Date: Thu, 12 Sep 2024 19:21:02 GMT
< Transfer-Encoding: chunked
< 
{ [5 bytes data]
100  1461    0  1461    0     0  70396      0 --:--:-- --:--:-- --:--:-- 73050
* Connection #0 to host localhost left intact
1461

It looks like the current main might just be using chunked encoding instead, even though the response body is relatively short, which could be avoiding whatever content length mishandling happens on the Agent side. Issuing this same curl request with the container built from this PR results in a fixed-length response with a Content-Length header, though that header does actually match the full response body length, so it doesn't appear to be a server-side issue.

@andrewazores
Copy link
Member Author

Here's the simple Agent patch I applied before rebuilding the gameserver containers to get this kind of debug log output:

diff --git a/src/main/java/io/cryostat/agent/CryostatClient.java b/src/main/java/io/cryostat/agent/CryostatClient.java
index 4692bd6..cfc1e66 100644
--- a/src/main/java/io/cryostat/agent/CryostatClient.java
+++ b/src/main/java/io/cryostat/agent/CryostatClient.java
@@ -18,6 +18,7 @@ package io.cryostat.agent;
 import java.io.BufferedInputStream;
 import java.io.IOException;
 import java.io.InputStream;
+import java.io.OutputStream;
 import java.net.URI;
 import java.net.URISyntaxException;
 import java.nio.file.Files;
@@ -53,6 +54,7 @@ import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
 import jdk.jfr.Recording;
 import org.apache.commons.io.FileUtils;
 import org.apache.commons.io.input.CountingInputStream;
+import org.apache.commons.io.input.TeeInputStream;
 import org.apache.http.HttpHeaders;
 import org.apache.http.HttpResponse;
 import org.apache.http.client.HttpClient;
@@ -226,7 +228,15 @@ public class CryostatClient {
                         })
                 .thenApply(
                         res -> {
-                            try (InputStream is = res.getEntity().getContent()) {
+                            try (InputStream is =
+                                    new TeeInputStream(
+                                            res.getEntity().getContent(),
+                                            new OutputStream() {
+                                                @Override
+                                                public void write(int b) throws IOException {
+                                                    log.info(new String(new char[] {(char) b}));
+                                                }
+                                            })) {
                                 return mapper.readValue(is, ObjectNode.class);
                             } catch (IOException e) {
                                 log.error("Unable to parse response as JSON", e);

@andrewazores
Copy link
Member Author

andrewazores commented Sep 12, 2024

I'm thinking this might be the culprit:

https://github.com/cryostatio/cryostat-agent/blob/d95c369d76640acf82265cd2be067e046b0b15a9/src/main/java/io/cryostat/agent/CryostatClient.java#L464

This whenComplete is meant to act something like a try { } finally { req.close() }, but the ordering of the whenComplete vs the other completion stages is not actually well defined. I suspect it could be called prematurely and close the stream that the JSON parser is still reading, leading to this behaviour. I'm not sure why this only seems to manifest with the newer Quarkus version and the potential change in behaviour fromTransfer-Encoding: chunked to Content-Length - if anything, it seems like it should break the other way around instead. This could also explain why the behaviour seems to change with JDK versions - the internal implementation of the CompletionStage/CompletableFuture could be different and affect the call order of the different stage handlers, ie. the thenApply vs whenComplete.

@andrewazores
Copy link
Member Author

Applying this Agent patch seems to solve the problem, implying that the explanation above is correct:

diff --git a/src/main/java/io/cryostat/agent/CryostatClient.java b/src/main/java/io/cryostat/agent/CryostatClient.java
index 4692bd6..5556e71 100644
--- a/src/main/java/io/cryostat/agent/CryostatClient.java
+++ b/src/main/java/io/cryostat/agent/CryostatClient.java
@@ -18,6 +18,7 @@ package io.cryostat.agent;
 import java.io.BufferedInputStream;
 import java.io.IOException;
 import java.io.InputStream;
+import java.io.OutputStream;
 import java.net.URI;
 import java.net.URISyntaxException;
 import java.nio.file.Files;
@@ -53,6 +54,7 @@ import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
 import jdk.jfr.Recording;
 import org.apache.commons.io.FileUtils;
 import org.apache.commons.io.input.CountingInputStream;
+import org.apache.commons.io.input.TeeInputStream;
 import org.apache.http.HttpHeaders;
 import org.apache.http.HttpResponse;
 import org.apache.http.client.HttpClient;
@@ -125,7 +127,9 @@ public class CryostatClient {
                                         + "?token="
                                         + pluginInfo.getToken()));
         log.trace("{}", req);
-        return supply(req, (res) -> logResponse(req, res)).thenApply(this::isOkStatus);
+        return supply(req, (res) -> logResponse(req, res))
+                .thenApply(this::isOkStatus)
+                .whenComplete((v, t) -> req.reset());
     }
 
     public CompletableFuture<PluginInfo> register(
@@ -174,7 +178,8 @@ public class CryostatClient {
                                     log.error("Unable to parse response as JSON", e);
                                     throw new RegistrationException(e);
                                 }
-                            });
+                            })
+                    .whenComplete((v, t) -> req.reset());
         } catch (JsonProcessingException e) {
             return CompletableFuture.failedFuture(e);
         }
@@ -209,7 +214,8 @@ public class CryostatClient {
                                 return CompletableFuture.completedFuture(prevId);
                             }
                             return submitCredentials(prevId, credentials, callback);
-                        });
+                        })
+                .whenComplete((v, t) -> req.reset());
     }
 
     private CompletableFuture<Integer> queryExistingCredentials(URI callback) {
@@ -226,7 +232,15 @@ public class CryostatClient {
                         })
                 .thenApply(
                         res -> {
-                            try (InputStream is = res.getEntity().getContent()) {
+                            try (InputStream is =
+                                    new TeeInputStream(
+                                            res.getEntity().getContent(),
+                                            new OutputStream() {
+                                                @Override
+                                                public void write(int b) throws IOException {
+                                                    log.info(new String(new char[] {(char) b}));
+                                                }
+                                            })) {
                                 return mapper.readValue(is, ObjectNode.class);
                             } catch (IOException e) {
                                 log.error("Unable to parse response as JSON", e);
@@ -254,7 +268,8 @@ public class CryostatClient {
                                                                 selfMatchExpression(callback)))
                                         .map(sc -> sc.id)
                                         .findFirst()
-                                        .orElse(-1));
+                                        .orElse(-1))
+                .whenComplete((v, t) -> req.reset());
     }
 
     private CompletableFuture<Integer> submitCredentials(
@@ -317,7 +332,8 @@ public class CryostatClient {
                                     location.substring(
                                             location.lastIndexOf('/') + 1, location.length());
                             return Integer.valueOf(id);
-                        });
+                        })
+                .whenComplete((v, t) -> req.reset());
     }
 
     public CompletableFuture<Void> deleteCredentials(int id) {
@@ -326,7 +342,9 @@ public class CryostatClient {
         }
         HttpDelete req = new HttpDelete(baseUri.resolve(CREDENTIALS_API_PATH + "/" + id));
         log.trace("{}", req);
-        return supply(req, (res) -> logResponse(req, res)).thenApply(res -> null);
+        return supply(req, (res) -> logResponse(req, res))
+                .whenComplete((v, t) -> req.reset())
+                .thenApply(res -> null);
     }
 
     public CompletableFuture<Void> deregister(PluginInfo pluginInfo) {
@@ -341,6 +359,7 @@ public class CryostatClient {
         log.trace("{}", req);
         return supply(req, (res) -> logResponse(req, res))
                 .thenApply(res -> assertOkStatus(req, res))
+                .whenComplete((v, t) -> req.reset())
                 .thenApply(res -> null);
     }
 
@@ -362,6 +381,7 @@ public class CryostatClient {
             log.trace("{}", req);
             return supply(req, (res) -> logResponse(req, res))
                     .thenApply(res -> assertOkStatus(req, res))
+                    .whenComplete((v, t) -> req.reset())
                     .thenApply(res -> null);
         } catch (JsonProcessingException e) {
             return CompletableFuture.failedFuture(e);
@@ -432,20 +452,21 @@ public class CryostatClient {
                                         .build());
         req.setEntity(entityBuilder.build());
         return supply(
-                req,
-                (res) -> {
-                    Instant finish = Instant.now();
-                    log.trace(
-                            "{} {} ({} -> {}): {}/{}",
-                            req.getMethod(),
-                            res.getStatusLine().getStatusCode(),
-                            fileName,
-                            req.getURI(),
-                            FileUtils.byteCountToDisplaySize(is.getByteCount()),
-                            Duration.between(start, finish));
-                    assertOkStatus(req, res);
-                    return (Void) null;
-                });
+                        req,
+                        (res) -> {
+                            Instant finish = Instant.now();
+                            log.trace(
+                                    "{} {} ({} -> {}): {}/{}",
+                                    req.getMethod(),
+                                    res.getStatusLine().getStatusCode(),
+                                    fileName,
+                                    req.getURI(),
+                                    FileUtils.byteCountToDisplaySize(is.getByteCount()),
+                                    Duration.between(start, finish));
+                            assertOkStatus(req, res);
+                            return (Void) null;
+                        })
+                .whenComplete((v, t) -> req.reset());
     }
 
     private HttpResponse logResponse(HttpRequestBase req, HttpResponse res) {
@@ -460,8 +481,7 @@ public class CryostatClient {
         // it responds with an auth challenge, and then send the auth information we have, and use
         // the client auth cache. This flow is supported for Bearer tokens in httpclient 5.
         authorizationSupplier.get().ifPresent(v -> req.addHeader(HttpHeaders.AUTHORIZATION, v));
-        return CompletableFuture.supplyAsync(() -> fn.apply(executeQuiet(req)), executor)
-                .whenComplete((v, t) -> req.reset());
+        return CompletableFuture.supplyAsync(() -> fn.apply(executeQuiet(req)), executor);
     }
 
     private HttpResponse executeQuiet(HttpUriRequest req) {

@andrewazores
Copy link
Member Author

/build_test

Copy link

Workflow started at 9/26/2024, 10:42:29 AM. View Actions Run.

Copy link

No GraphQL schema changes detected.

Copy link

No OpenAPI schema changes detected.

Copy link

CI build and push: All tests pass ✅
https://github.com/cryostatio/cryostat/actions/runs/11054619371

Clean up labels/annotations internal representation using KeyValue type
for better uniformity across API. Fix up some misuse of Blocking
annotation and some bad transaction handling at startup.
@andrewazores
Copy link
Member Author

/build_test

Copy link

Workflow started at 9/30/2024, 11:26:18 AM. View Actions Run.

Copy link

No GraphQL schema changes detected.

Copy link

No OpenAPI schema changes detected.

Copy link

CI build and push: All tests pass ✅
https://github.com/cryostatio/cryostat/actions/runs/11109252733

@andrewazores andrewazores merged commit 739a65f into cryostatio:main Sep 30, 2024
9 checks passed
@andrewazores andrewazores deleted the quarkus-3.8 branch September 30, 2024 15:42
andrewazores added a commit to andrewazores/cryostat3 that referenced this pull request Oct 8, 2024
* build(deps): update Quarkus to 3.8 LTS

* add hibernate format mapper override

* remove no longer necessary lazy init

* disable openapi management / swagger UI unless in dev mode

* remove duplicate healthcheck override

* reduce healthcheck start periods

* correct healthcheck URL

* use authproxy image which contains wget for healthcheck

* correct healthchecks for vertx agent test applications
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
chore Refactor, rename, cleanup, etc. dependencies Pull requests that update a dependency file safe-to-test
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant