Gremlin Java Client Demo no longer works

[jfassad]

Connections through nlb with iam-auth and enable-ssl no longer works

timeouts during handshake.

[iansrobinson]

Hi @jfassad - thanks for reporting this. Can you tell me which region you were running in, and which Neptune engine version you are running? I'll then try and reproduce. Thanks, ian

[jfassad]

Hello @iansrobinson . region is us-east-1 , neptune engine is 1.0.4.1.R2
It appears the problem might be related to something in the AWS NLB .
I've replaced the NLB for a simpleproxy running in an EC2 instance and the problem went away.

Also, the problem is not related to iam-auth, the connection hangs with or without it, during the websocket handshake.
Increasing the handshake timeout to 1 minute didn't help either.

I've been using this setup to allow devs to run development services locally and connect to the neptune cluster inside the VPC for months... the problem started happening yesterday or the day before. ( one dev claims it was working on March 1st ) When we noticed the problem I grabbed the Gremlin Java Client Demo to try to reproduce the problem and eliminate anything changed on our end, even though we had not made any changes for quite some time, and verified that the Gremlin Java Client Demo was not working either.

[jfassad]

@iansrobinson, I made another test. I've created a new NLB with TCP listener and it works. The problem happens only using NLB with a TLS listener.

[andrew-haus]

I was able to reproduce this locally as well.

Neptune engine version - 1.0.5.1
Region - us-west-2

We do not have a NLB and upgrading our gremlin library to 4.3.13 seems to have fixed the issue so far.

Exception below:

Caused by: org.apache.tinkerpop.gremlin.driver.exception.ConnectionException: Could not complete connection setup to the server. Ensure that SSL is correctly configured at both the client and the server. Ensure that client WebSocket handshake protocol matches the server. Ensure that the server is still reachable.
	at org.apache.tinkerpop.gremlin.driver.Channelizer$WebSocketChannelizer.connected(Channelizer.java:268) ~[gremlin-driver-3.4.11.jar:3.4.11]
	at org.apache.tinkerpop.gremlin.driver.Connection.<init>(Connection.java:120) ~[gremlin-driver-3.4.11.jar:3.4.11]
	at org.apache.tinkerpop.gremlin.driver.ConnectionPool.lambda$new$0(ConnectionPool.java:102) ~[gremlin-driver-3.4.11.jar:3.4.11]
	at java.util.concurrent.CompletableFuture$AsyncRun.run$$$capture(CompletableFuture.java:1736) ~[?:?]
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java) ~[?:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
	... 3 more
Caused by: io.netty.handler.ssl.SslHandshakeTimeoutException: handshake timed out after 10000ms
	at io.netty.handler.ssl.SslHandler$7.run(SslHandler.java:2112) ~[netty-handler-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[netty-common-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) ~[netty-common-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:164) ~[netty-common-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java) ~[netty-common-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) ~[netty-common-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) ~[netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.72.Final.jar:4.1.72.Final]
	... 1 more

[vicwolk]

I was able to reproduce this locally as well.

Neptune engine version - 1.0.5.1 Region - us-west-2

We do not have a NLB and upgrading our gremlin library to 4.3.13 seems to have fixed the issue so far.

Exception below:

Caused by: org.apache.tinkerpop.gremlin.driver.exception.ConnectionException: Could not complete connection setup to the server. Ensure that SSL is correctly configured at both the client and the server. Ensure that client WebSocket handshake protocol matches the server. Ensure that the server is still reachable.
	at org.apache.tinkerpop.gremlin.driver.Channelizer$WebSocketChannelizer.connected(Channelizer.java:268) ~[gremlin-driver-3.4.11.jar:3.4.11]
	at org.apache.tinkerpop.gremlin.driver.Connection.<init>(Connection.java:120) ~[gremlin-driver-3.4.11.jar:3.4.11]
	at org.apache.tinkerpop.gremlin.driver.ConnectionPool.lambda$new$0(ConnectionPool.java:102) ~[gremlin-driver-3.4.11.jar:3.4.11]
	at java.util.concurrent.CompletableFuture$AsyncRun.run$$$capture(CompletableFuture.java:1736) ~[?:?]
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java) ~[?:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
	... 3 more
Caused by: io.netty.handler.ssl.SslHandshakeTimeoutException: handshake timed out after 10000ms
	at io.netty.handler.ssl.SslHandler$7.run(SslHandler.java:2112) ~[netty-handler-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[netty-common-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) ~[netty-common-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:164) ~[netty-common-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java) ~[netty-common-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) ~[netty-common-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) ~[netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.72.Final.jar:4.1.72.Final]
	... 1 more

I believe you meant version 3.4.13 instead of 4.3.13 and, either way, the problem still lingers. Plus, the timeout is from netty and is untouchable. Any other timeouts (even the 15s one) don't mean much, because the netty 10s timeout takes place.

Anyway, this problem is still happening now and it doesn't matter if we're using simpleproxy or not. It seems to happen randomly, with a higher rate of not completing the handshake.