Fix REST client connection handling #19883

tmetzke · 2024-06-28T18:49:35Z

Describe the bug

The following integration tests don't work as expected over REST:

shouldActivateJobsIfBatchIsTruncated in the LongPollingActivateJobsTest
shouldActivateJobForOpenRequest in the LongPollingActivateJobsTest

To Reproduce

Add the option true to the test cases in the booleans array in the ValueSource annotation, basically reverting this commit.
Run the tests

Expected behavior

The tests pass.

Log/Stacktrace

The tests fail with a timeout on the Camunda client

Environment:

OS: all
Zeebe Version: 8.6.0-alpha3
Configuration: long polling enabled in the gateway

Workaround

Switch to using the gRPC protocol for job activation using long polling.

Additional context

This limitation is documented in our docs with camunda/camunda-docs#4375

The text was updated successfully, but these errors were encountered:

Currently, the gateway cannot handle long-lasting requests or connections well enough for all use cases, apparently. This will be tackled with #19883. Until then, those ITs are disabled in the QA module.

tmetzke · 2024-06-28T19:14:34Z

Observations

Copied from #19333 (comment)

@koevskinikola and I investigated the other failing IT (shouldActivateJobForOpenRequest):

If we remove sendActivateRequestsAndClose AFTER the test's relevant client (lines 117/118) is opened (i.e. removing it from line 120), the test passes.

The sendActivateRequestsAndClose before the test's relevant client is not affecting the result in any way.

It looks like the connection of our test client is interrupted if new requests come in.

It doesn't exactly explain why the other test case fails but could have the same root cause (long-running connections not working in all cases on the server side).

Currently, the gateway cannot handle long-lasting requests or connections well enough for all use cases, apparently. This will be tackled with #19883. Until then, those ITs are disabled in the QA module.

tmetzke · 2024-07-01T06:29:46Z

Observations round 2

After another round of manual tests using cURL locally, I strongly assume the issue resides on the client instead of the server.
All IT scenarios are successful using cURL manually. All IT scenarios also fail using the Java client in a "production" setup, i.e. the test setup is not the root cause.

Setups

Setup A

Start the StandaloneBroker.
Send a job activation request via cURL with a requestTimeout of 15 seconds.
Send 5 more job activation requests in parallel via cURL with a client-side timeout of 1 second.

Setup B

Setup A
Before the 15s are up, create a process instance with one service task that has a job type as defined by the initial activation request

Setup C

Start the StandaloneBroker.
Create 10 process instances with one service task with job type "foo" and a variable "bar" with a String value where 'x' is repeated 1,048,576 times (max message size in bytes / 4).
Send a job activation request via cURL for type "foo".

Results

Results A

The initial request comes back successfully with an empty list of jobs.

Results B

The initial request comes back successfully with a job list containing one item.

Results C

The initial request comes back successfully with a job list containing 10 items.

Currently, the gateway cannot handle long-lasting requests or connections well enough for all use cases, apparently. This will be tackled with #19883. Until then, those ITs are disabled in the QA module.

koevskinikola · 2024-09-12T13:40:56Z

ZPA mob-programming session:

We did some additional investigations on the two failing tests using Wireshark. We managed to capture the communication of both successful and failing runs for both tests. The successful runs were made by modifying the tests. The modifications are explained below.

The captures can be found on this (internal) link.

It seems that the root cause for the two failing tests is different:

Test shouldActivateJobsIfBatchIsTruncated:

Failure: we could see a [TCP Window Full] message that broke the transmission of the variable data, which resulted in a [RST, ACK] message from the Gateway, asking for the connection to be reset due to protocol errors.
Success: we reduced the size of the variable, which allowed the test to pass.
Next step: we should investigate how we can adjust the TCP window size from the client to allow for larger messages to be transmitted.

Test shouldActivateJobForOpenRequest:

Failure: Once the job activation request is sent, we see a TCP [ACK] message from the client, followed by a TCP [RST, ACK] message from the Gateway. This seems to indicate a protocol error. It's not clear why this happens.
Success: The test performs 3 additional job activation requests after the open long-polling request. If we remove these requests, the test succeeds, i.e. data is transmitted as expected.
Next step: investigate why a TCP [RST, ACK] may be sent in the context of Apache HttpClient 5 and a Spring Boot REST API.

tmetzke · 2024-09-12T14:01:36Z

Regarding shouldActivateJobsIfBatchIsTruncated:

Looking at this description, it may be that client for HTTP/2 (gRPC) can easily increase the window size. This might not be possible for HTTP/1.1 (REST).
This might simply be a limitation of REST over HTTP/1.1 itself

koevskinikola · 2024-09-13T11:32:00Z

@tmetzke since TCP is a transport layer protocol, we can't control it from the application layer, i.e. an HTTP client. gRPC, which is built on HTTP/2, supports multiplexing and streaming manages to push the TCP window size to larger values more quickly.

I found that we can influence the TCP window size by increasing socket buffer sizes through the following Apache HttpClient 5 configuration options:

final IOReactorConfig ioReactorConfig =
        IOReactorConfig.custom() // the IOReactor manages non-blocking connection
            .setSoTimeout(Timeout.ofSeconds(30)) // Overall socket timeout
            .setSndBufSize(64 * 1024) // Larger send buffer size
            .setRcvBufSize(64 * 1024) // Larger receive buffer size
            .setTcpNoDelay(true) // Disable Nagle's algorithm for lower latency
            .build();

    final HttpAsyncClientBuilder builder =
        HttpAsyncClients.custom()
            .setIOReactorConfig(ioReactorConfig)

However, the truncated test continues to fail. This also doesn't guarantee a stable connection, i.e. there might still be failures depending on the size of the data.

tmetzke · 2024-09-13T11:40:41Z

Thanks for giving this a closer look 👍
I am still wondering why this works with cURL, though 🤔
@megglos and maybe @npepinpe, can we document this as a "REST over HTTP/1.1" protocol limitation of the client?
I don't see much we can do about it right now. It introduces a regression when moving from gRPC to REST communication using the Java client.

npepinpe · 2024-09-16T11:37:00Z

AFAIK [TCP Window Full] is just a Wireshark thing, telling you the sender has sent a full window amount of bytes without any acks. I'm not sure why the sender would then decide to send a RST, I suspect possibly because it expects something to have gone wrong. So the question is why the client doesn't send any ACK at that point 🤔

I'd need to look at the dumps I guess, it's unclear to me. AFAIK this shouldn't be a limitation of HTTP/1.1, we should be able to send arbitrarily large responses - the only thing I could imagine is someone sending a RST due to a time out somewhere if the read/send doesn't finish fast enough. Maybe I misunderstood the issue though 🤔

tmetzke · 2024-09-16T12:13:17Z

Thanks, @npepinpe, that would also tie in better with my observation that cURL is capable of handling the response from the server. It indicates a specific problem with the Apache HTTP client, not the HTTP/1.1 communication. This remains a tough nut to crack 🙈

koevskinikola · 2024-09-16T12:31:38Z

@npepinpe the stack trace of the failure shows that a timeout is triggered. However, no configuration adjustments have impacted the timeout so far. I'll continue debugging.

npepinpe · 2024-09-16T15:13:23Z

So the RST likely comes from the time out, right? The question is why the client stops acknowledging - are we overwhelming it? Maybe we are?

OTOH, things I would check (and instrument with metrics):

Connection pool settings
Threading/Reactor settings
Socket timeouts (not to be confused with application level timeouts, i.e. per request)
Pipelining settings

koevskinikola · 2024-09-17T07:47:05Z

@npepinpe so far I've tried the following configuration adjustments, but no changes in the failures:

in HttpClientFactory:

private HttpAsyncClientBuilder defaultClientBuilder() {
  ...
      final PoolingAsyncClientConnectionManager connectionManager =
        PoolingAsyncClientConnectionManagerBuilder.create()
            .setTlsStrategy(tlsStrategy)
            .setMaxConnPerRoute(10) // <--- tried increasing the max connections per route and total
            .setMaxConnTotal(20)
            .setPoolConcurrencyPolicy(PoolConcurrencyPolicy.LAX) // <--- and made the concurrency policy LAX to allow for more connections
            .setDefaultConnectionConfig(     // <---- I also tried passing along a default connection config
                ConnectionConfig.custom()
                    .setSocketTimeout(Timeout.ofSeconds(40L))
                    .setConnectTimeout(Timeout.ofSeconds(40L))
                    .build())
            .build();

    // the IOReactorConfig is completely new. I added it to control the socket management of async connections
    final IOReactorConfig ioReactorConfig =
        IOReactorConfig.custom()
            .setSoTimeout(Timeout.ofSeconds(30)) // Overall socket timeout
            .setSndBufSize(64 * 1024) // Larger send buffer size (the idea was to influence the TCP window size)
            .setRcvBufSize(64 * 1024) // Larger receive buffer size  (the idea was to influence the TCP window size)
            .setTcpNoDelay(true) // Disable Nagle's algorithm for lower latency  (the idea was to influence the TCP window size)
            .build();

    final HttpAsyncClientBuilder builder =
        HttpAsyncClients.custom()
            .setConnectionManager(connectionManager)
            .setIOReactorConfig(ioReactorConfig)                                        // <--- the IOReacotorConfig is passed here
            .setDefaultHeaders(Collections.singletonList(acceptHeader))
            .setUserAgent("zeebe-client-java/" + VersionUtil.getVersion())
            .evictExpiredConnections()
            .setCharCodingConfig(
                CharCodingConfig.custom().setCharset(StandardCharsets.UTF_8).build())
            .evictIdleConnections(TimeValue.ofSeconds(30))
            .useSystemProperties(); // allow users to customize via system properties
...
}

private Builder defaultClientRequestConfigBuilder() {
    return RequestConfig.custom()
        .setResponseTimeout(Timeout.ofSeconds(30L))
        .setConnectionRequestTimeout(Timeout.ofSeconds(40L)) // <--- Also tried increasing the request timeout
   ... 
}

I think these config properties cover the first three points you listed above. I'm not sure about the last one, I'll do some research on how that can be done.

koevskinikola · 2024-09-17T10:49:45Z

UPDATE:

I missed the request timeout override here. That was overriding the socket timeout of the connections InternalDataChannel.
After adjusting this timeout to 40sec, the following occurs:
- shouldActivateJobsIfBatchIsTruncated - failure remains the same.
- shouldActivateJobForOpenRequest - a server-side error occurs:

java.util.concurrent.CompletionException: io.camunda.zeebe.client.api.command.ProblemException: Failed with code 503: 'Service Unavailable'. Details: 'class ProblemDetail {
    type: about:blank
    title: Service Unavailable
    status: 503
    detail: null
    instance: /v2/jobs/activation
}'

	at java.base/java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:413)
	at java.base/java.util.concurrent.CompletableFuture.join(CompletableFuture.java:2118)
	at io.camunda.zeebe.it.client.command.LongPollingActivateJobsTest.shouldActivateJobForOpenRequest(LongPollingActivateJobsTest.java:130)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:212)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:194)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:212)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:212)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:212)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:212)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:212)
	at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:1024)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:556)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:546)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:611)
	at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:291)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:212)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:212)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:212)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1709)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:556)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:546)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:611)
	at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:291)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1709)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:556)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:546)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:611)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
Caused by: io.camunda.zeebe.client.api.command.ProblemException: Failed with code 503: 'Service Unavailable'. Details: 'class ProblemDetail {
    type: about:blank
    title: Service Unavailable
    status: 503
    detail: null
    instance: /v2/jobs/activation
}'
	at io.camunda.zeebe.client.impl.http.ApiCallback.handleErrorResponse(ApiCallback.java:111)
	at io.camunda.zeebe.client.impl.http.ApiCallback.completed(ApiCallback.java:59)
	at io.camunda.zeebe.client.impl.http.ApiCallback.completed(ApiCallback.java:30)
	at org.apache.hc.core5.concurrent.BasicFuture.completed(BasicFuture.java:123)
	at org.apache.hc.core5.concurrent.ComplexFuture.completed(ComplexFuture.java:72)
	at org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncClient$2$1.completed(InternalAbstractHttpAsyncClient.java:304)
	at org.apache.hc.core5.http.nio.support.AbstractAsyncResponseConsumer$1.completed(AbstractAsyncResponseConsumer.java:101)
	at org.apache.hc.core5.http.nio.entity.AbstractBinAsyncEntityConsumer.completed(AbstractBinAsyncEntityConsumer.java:84)
	at org.apache.hc.core5.http.nio.entity.AbstractBinDataConsumer.streamEnd(AbstractBinDataConsumer.java:81)
	at org.apache.hc.core5.http.nio.support.AbstractAsyncResponseConsumer.streamEnd(AbstractAsyncResponseConsumer.java:142)
	at org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec$1.streamEnd(HttpAsyncMainClientExec.java:251)
	at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamHandler.dataEnd(ClientHttp1StreamHandler.java:270)
	at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamDuplexer.dataEnd(ClientHttp1StreamDuplexer.java:366)
	at org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onInput(AbstractHttp1StreamDuplexer.java:334)
	at org.apache.hc.core5.http.impl.nio.AbstractHttp1IOEventHandler.inputReady(AbstractHttp1IOEventHandler.java:64)
	at org.apache.hc.core5.http.impl.nio.ClientHttp1IOEventHandler.inputReady(ClientHttp1IOEventHandler.java:41)
	at org.apache.hc.core5.reactor.InternalDataChannel.onIOEvent(InternalDataChannel.java:142)
	at org.apache.hc.core5.reactor.InternalChannel.handleIOEvent(InternalChannel.java:51)
	at org.apache.hc.core5.reactor.SingleCoreIOReactor.processEvents(SingleCoreIOReactor.java:178)
	at org.apache.hc.core5.reactor.SingleCoreIOReactor.doExecute(SingleCoreIOReactor.java:127)
	at org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(AbstractSingleCoreIOReactor.java:86)
	at org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.java:44)
	at java.base/java.lang.Thread.run(Thread.java:1570)

koevskinikola · 2024-09-17T11:54:02Z

I pushed my changes to #22302

npepinpe · 2024-09-17T11:57:48Z

Who is producing the 503? The server is returning this? Or is the client producing this error because the response never arrives? From the stack trace, I assume the server, right? Why are we returning 503?

koevskinikola · 2024-09-17T12:07:13Z

The server is producing a 500 which is mapped to an unexpected error.

When debugging locally I see the following debug message from the server side:

Broker is out of disk space. All client requests will be rejected

I also see the following client-side error during the setup phase of the tests:

14:01:54.693 [] [] [httpclient-dispatch-1] ERROR org.apache.hc.client5.http.impl.async - Pool entry is not present in the set of leased entries
java.lang.IllegalStateException: Pool entry is not present in the set of leased entries
	at org.apache.hc.core5.pool.StrictConnPool.release(StrictConnPool.java:268) ~[httpcore5-5.2.5.jar:5.2.5]
	at org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnectionManager.release(PoolingAsyncClientConnectionManager.java:410) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.client5.http.impl.async.InternalHttpAsyncExecRuntime.discardEndpoint(InternalHttpAsyncExecRuntime.java:147) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.client5.http.impl.async.InternalHttpAsyncExecRuntime.discardEndpoint(InternalHttpAsyncExecRuntime.java:170) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncClient$2.failed(InternalAbstractHttpAsyncClient.java:346) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.client5.http.impl.async.AsyncRedirectExec$1.failed(AsyncRedirectExec.java:248) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.client5.http.impl.async.AsyncHttpRequestRetryExec$1.failed(AsyncHttpRequestRetryExec.java:197) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.client5.http.impl.async.AsyncProtocolExec$1.failed(AsyncProtocolExec.java:295) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec$1.failed(HttpAsyncMainClientExec.java:131) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamHandler.failed(ClientHttp1StreamHandler.java:285) ~[httpcore5-5.2.5.jar:5.2.5]
	at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamDuplexer.disconnected(ClientHttp1StreamDuplexer.java:220) ~[httpcore5-5.2.5.jar:5.2.5]
	at org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onDisconnect(AbstractHttp1StreamDuplexer.java:408) ~[httpcore5-5.2.5.jar:5.2.5]
	at org.apache.hc.core5.http.impl.nio.AbstractHttp1IOEventHandler.disconnected(AbstractHttp1IOEventHandler.java:95) ~[httpcore5-5.2.5.jar:5.2.5]
	at org.apache.hc.core5.http.impl.nio.ClientHttp1IOEventHandler.disconnected(ClientHttp1IOEventHandler.java:41) ~[httpcore5-5.2.5.jar:5.2.5]
	at org.apache.hc.core5.reactor.InternalDataChannel.disconnected(InternalDataChannel.java:204) ~[httpcore5-5.2.5.jar:5.2.5]
	at org.apache.hc.core5.reactor.SingleCoreIOReactor.processClosedSessions(SingleCoreIOReactor.java:231) ~[httpcore5-5.2.5.jar:5.2.5]
	at org.apache.hc.core5.reactor.SingleCoreIOReactor.doTerminate(SingleCoreIOReactor.java:106) ~[httpcore5-5.2.5.jar:5.2.5]
	at org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(AbstractSingleCoreIOReactor.java:93) ~[httpcore5-5.2.5.jar:5.2.5]
	at org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.java:44) ~[httpcore5-5.2.5.jar:5.2.5]
	at java.base/java.lang.Thread.run(Thread.java:1570) [?:?]

koevskinikola · 2024-10-01T15:30:32Z

Until this bug is root-caused and fixed, the limitation is documented through camunda/camunda-docs#4375

tmetzke mentioned this issue Jun 28, 2024

[EPIC] Complete Zeebe REST API infrastructure #18254

Closed

tmetzke assigned tmetzke and koevskinikola Jun 28, 2024

tmetzke changed the title ~~Fix REST gateway connection handling~~ Fix REST client connection handling Jul 1, 2024

tmetzke added scope/clients-java Marks an issue or PR to appear in the Java client section of the changelog and removed scope/gateway Marks an issue or PR to appear in the gateway section of the changelog component/gateway labels Jul 1, 2024

tmetzke mentioned this issue Jul 1, 2024

The Java client supports Job activation over REST (including long polling) #18256

Closed

tmetzke removed their assignment Aug 12, 2024

koevskinikola linked a pull request Sep 13, 2024 that will close this issue

Fix java client rest connection for long polling jobs #22302

Draft

3 tasks

koevskinikola mentioned this issue Oct 1, 2024

Document limitation of long-polling job activation via REST API camunda/camunda-docs#4375

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix REST client connection handling #19883

Fix REST client connection handling #19883

tmetzke commented Jun 28, 2024 •

edited by koevskinikola

Loading

tmetzke commented Jun 28, 2024 •

edited

Loading

tmetzke commented Jul 1, 2024

koevskinikola commented Sep 12, 2024 •

edited

Loading

tmetzke commented Sep 12, 2024

koevskinikola commented Sep 13, 2024 •

edited

Loading

tmetzke commented Sep 13, 2024

npepinpe commented Sep 16, 2024 •

edited

Loading

tmetzke commented Sep 16, 2024

koevskinikola commented Sep 16, 2024

npepinpe commented Sep 16, 2024 •

edited

Loading

koevskinikola commented Sep 17, 2024 •

edited

Loading

koevskinikola commented Sep 17, 2024

koevskinikola commented Sep 17, 2024

npepinpe commented Sep 17, 2024 •

edited

Loading

koevskinikola commented Sep 17, 2024

koevskinikola commented Oct 1, 2024

Fix REST client connection handling #19883

Fix REST client connection handling #19883

Comments

tmetzke commented Jun 28, 2024 • edited by koevskinikola Loading

tmetzke commented Jun 28, 2024 • edited Loading

Observations

tmetzke commented Jul 1, 2024

Observations round 2

Setups

Setup A

Setup B

Setup C

Results

Results A

Results B

Results C

koevskinikola commented Sep 12, 2024 • edited Loading

tmetzke commented Sep 12, 2024

koevskinikola commented Sep 13, 2024 • edited Loading

tmetzke commented Sep 13, 2024

npepinpe commented Sep 16, 2024 • edited Loading

tmetzke commented Sep 16, 2024

koevskinikola commented Sep 16, 2024

npepinpe commented Sep 16, 2024 • edited Loading

koevskinikola commented Sep 17, 2024 • edited Loading

koevskinikola commented Sep 17, 2024

koevskinikola commented Sep 17, 2024

npepinpe commented Sep 17, 2024 • edited Loading

koevskinikola commented Sep 17, 2024

koevskinikola commented Oct 1, 2024

tmetzke commented Jun 28, 2024 •

edited by koevskinikola

Loading

tmetzke commented Jun 28, 2024 •

edited

Loading

koevskinikola commented Sep 12, 2024 •

edited

Loading

koevskinikola commented Sep 13, 2024 •

edited

Loading

npepinpe commented Sep 16, 2024 •

edited

Loading

npepinpe commented Sep 16, 2024 •

edited

Loading

koevskinikola commented Sep 17, 2024 •

edited

Loading

npepinpe commented Sep 17, 2024 •

edited

Loading