Agent.queryRaw - requests sometimes hang on without any result

[chinaryov]

Hello,

We sometimes have a strange problem. Once in a few days query request to IC hang on and there are no any result, even exception. Bad thing is that I cannot reproduce it in a test.

Here is the information that I have found so far.

Example code:

    private byte[] query(String canisterId, Request<byte[]> request) {
        final Principal canisterPrincipal = Principal.fromString(canisterId);
        final CompletableFuture<Response<byte[]>> responseCompletableFuture = agent.queryRaw(
                canisterPrincipal, canisterPrincipal, methodName, request, Optional.empty());
        final Response<byte[]> response;
        try {
            response = responseCompletableFuture.get();
        } catch (Throwable e) {
            throw new RuntimeException(e);
        }
        return response.getPayload();
    }

Calling method 'responseCompletableFuture.get();' hangs on forever.

Source code - ReplicaApacheHttpTransport

Line 198

client.execute(httpRequest, new FutureCallback<SimpleHttpResponse>() {
    @Override
    public void completed(SimpleHttpResponse httpResponse) {
        ...
    }

    @Override
    public void failed(Exception ex) {
        ...
    }

    @Override
    public void cancelled() {
        ...
    }
});        

After calling method 'client.execute' methods of interface FutureCallback (complete, failed and cancelled) not called.

Source code - InternalAbstractHttpAsyncClient (httpclient5-5.1.3.jar)

Line 223

...
            executeImmediate(...)
...

    void executeImmediate(
            final HttpRequest request,
            final AsyncEntityProducer entityProducer,
            final AsyncExecChain.Scope scope,
            final AsyncExecCallback asyncExecCallback) throws HttpException, IOException {
        execChain.execute(request, entityProducer, scope, asyncExecCallback);
    }

After method execution 'executeImmediate' no any method was called of 'AsyncExecCallback'.

Best regards,
Alexander

[rdobrik]

Hi Alexander, which version of Java are you using? I noticed some issues that Apache HTTP client has with certain Java versions.

[chinaryov]

Hi Alexander, which version of Java are you using? I noticed some issues that Apache HTTP client has with certain Java versions.

java version "1.8.0_321"

[rdobrik]

Hmm, I am working on new Agent version too now. Maybe there is some unhandled exception. I will add some additional log messages and push beta version out. Give me day or two.

[chinaryov]

I have changed my code to this method:

    /**
     * Waits if necessary for at most the given time for this future
     * to complete, and then returns its result, if available.
     *
     * @param timeout the maximum time to wait
     * @param unit the time unit of the timeout argument
     * @return the result value
     * @throws CancellationException if this future was cancelled
     * @throws ExecutionException if this future completed exceptionally
     * @throws InterruptedException if the current thread was interrupted
     * while waiting
     * @throws TimeoutException if the wait timed out
     */
    public T get(long timeout, TimeUnit unit)

of the class CompletableFuture instead of simple get() without parameters.
That works for me.

[rdobrik]

Thanks Alexander, this is actually great suggestion, especially for update operations, where we already know expiration time. I am using get method in some internal calls as well (just to simplify calls interface for users) but I should set timeout there