Issues migrating dse 6.8.14 to apache 4.0.13

[sinohC]

Hi!

We are planning to migrate some dse 6.8 cluster that are running into docker to apache cassandra 4.0 that will be running into k8s, for that we have created a new 4.0.13 image including your everywherestrategy patch.

Testing the image alone, seems to be working fine as it is able to create the keyspace with the everywherestrategy and it is able to insert rows there.

Problems begin when we try to connect the new cassandra 4.0 datacenter to the existing dse 6.8 cluster. Next errors appear when the first cassandra node tries to connect to dse:

Errors on DSE

ERROR [CoreThread-0] 2024-12-05 09:23:31,298 InboundHandshakeHandler.java:123 - Unexpected error while negotiating internode protocol handshake java.lang.NegativeArraySizeException: null at org.apache.cassandra.net.CompactEndpointSerializationHelper.deserialize(CompactEndpointSerializationHelper.java:58) at org.apache.cassandra.net.async.HandshakeProtocol$ThirdHandshakeMessage.maybeDecode(HandshakeProtocol.java:281) at org.apache.cassandra.net.async.InboundHandshakeHandler.handleMessagingStartResponse(InboundHandshakeHandler.java:266) at org.apache.cassandra.net.async.InboundHandshakeHandler.decode(InboundHandshakeHandler.java:107) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:520) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:289) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:529) at org.apache.cassandra.concurrent.EpollTPCEventLoopGroup$SingleCoreEventLoop.processEpollEvents(EpollTPCEventLoopGroup.java:1020) at org.apache.cassandra.concurrent.EpollTPCEventLoopGroup$SingleCoreEventLoop.processEvents(EpollTPCEventLoopGroup.java:988) at org.apache.cassandra.concurrent.EpollTPCEventLoopGroup$SingleCoreEventLoop.run(EpollTPCEventLoopGroup.java:542) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) at org.apache.cassandra.utils.concurrent.InlinedThreadLocalThread.run(InlinedThreadLocalThread.java:251) Contraer

Errors on Apache Cassandra

ERROR [Messaging-EventLoop-3-1] 2024-12-05 09:23:29,343 NoSpamLogger.java:98 - /XXXXXXXX:7000->/XXXXXXXX:7000-URGENT_MESSAGES-[no-channel] failed to connect java.nio.channels.ClosedChannelException: null at org.apache.cassandra.net.OutboundConnectionInitiator$Handler.channelInactive(OutboundConnectionInitiator.java:248) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:819) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source)

We tried different tests (docker to docker , k8s to k8s, docker to k8s) with same result , so we assume that the issue should be something else...

Is it something else for this migration to be achieved needed?

Thanks in advance!

[sinohC]

@smiklosovic maybe you could give some ideas here?

Thanks so much in advance.

[smiklosovic]

@sinohC can you run your setup without this plugin? Can be even dse 6.8 and 4.0.13 in one cluster? I think this plugin is not hard requirement to do that. If it works without this plugin and the error happens with this plugin, then it it legit to deal with it.

[smiklosovic]

@sinohC there is also test framework in test/ directory by which you can verify what combination of versions work and what not. Also, the plugin for Cassandra 4 is compiled against version 4.0.7 so it might differ in 4.0.13 somehow. Also I do not see this was tested against DSE 6.8, the tests were conducted on 5.x if you look into test results markdown but looking into the sources, I think that the test scripts were written with 6.8 in mind as they explicitly mention that version there too.

Please try to verify that you can use that against 4.0.7 first by executing the testing framework.

[smiklosovic]

Your setup is also very specific because what you try to do is to have 1 dc on DSE and another dc on Apache Cassandra. So you are trying to have two separate datacenters. Maybe it would be worth to test what happens if you are joining Apache Cassandra node to the same DC all your DSE nodes are.

[sinohC]

@sinohC can you run your setup without this plugin? Can be even dse 6.8 and 4.0.13 in one cluster? I think this plugin is not hard requirement to do that. If it works without this plugin and the error happens with this plugin, then it it legit to deal with it.

You mean that the desired migration should work without this plugin? Or you mean to try the connectivity just to test if the error changes? I mean, I never tested without the plugin as I assume it wont work and the purpose of the plugin is to fix the posible problems related to the everywherestrategy

Your setup is also very specific because what you try to do is to have 1 dc on DSE and another dc on Apache Cassandra. So you are trying to have two separate datacenters. Maybe it would be worth to test what happens if you are joining Apache Cassandra node to the same DC all your DSE nodes are.

We will test this option too.