Server side: OutOfDirectMemoryError

[flagshen]

grpc version: 1.52.1
java version: 1.8
Now my client uses 100 threads, each thread sends 1M data, the server directly returns the response, I set the out-of-heap memory size limit to 512M, the client as long as the execution of the server will appear the following error, and the server program will not reduce the memory.

io.grpc.netty.shaded.io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 4194304 byte(s) of direct memory (used: 532676887, max: 536870912)
        at io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:845)
        at io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:774)
        at io.grpc.netty.shaded.io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:649)
        at io.grpc.netty.shaded.io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:624)
        at io.grpc.netty.shaded.io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:203)
        at io.grpc.netty.shaded.io.netty.buffer.PoolArena.tcacheAllocateSmall(PoolArena.java:173)
        at io.grpc.netty.shaded.io.netty.buffer.PoolArena.allocate(PoolArena.java:134)
        at io.grpc.netty.shaded.io.netty.buffer.PoolArena.allocate(PoolArena.java:126)
        at io.grpc.netty.shaded.io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:396)
        at io.grpc.netty.shaded.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
        at io.grpc.netty.shaded.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
        at io.grpc.netty.shaded.io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
        at io.grpc.netty.shaded.io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120)
        at io.grpc.netty.shaded.io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75)
        at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:785)
        at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
        at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385)
        at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:750)

The client code is as follows:

Svs.CommonDataResponse response = clientStub.sOFSignData(Svs.SignDataRequest.newBuilder()
                .setInData(ByteString.copyFrom(inData))
                .build());
        if (response.getRv() == Error.OK.getCode()) {
            return response.getData();
        }

The server code is as follows:

ServerCallStreamObserver<Svs.CommonDataResponse> clientCallStreamObserver = (ServerCallStreamObserver<Svs.CommonDataResponse>) responseObserver;
        clientCallStreamObserver.setOnReadyHandler(()-> {
            if (clientCallStreamObserver.isReady()) {
                Svs.CommonDataResponse resp = Svs.CommonDataResponse.newBuilder()
                        .setRv(ErrorMgr.RET_OK)
                        .setData("signature")
                        .build();
                clientCallStreamObserver.onNext(resp);
                clientCallStreamObserver.onCompleted();
            }
        });

[kannanjgithub]

Can you check if the issue occurs with the latest grpc java version (1.65.0)?

[flagshen]

Can you check if the issue occurs with the latest grpc java version (1.65.0)?

Upgrading to the latest version will still have this problem

[ejona86]

You are getting the OOM on server-side? How many concurrent RPCs is the server handling?

It looks like the servers response is pretty small, so the memory is mostly being consumed by the requests.

[flagshen]

Yes, there is a problem on the server, the server handles the concurrency of 100 RPCs.
I sent it to the client, but the server did not process the request data and directly returned a string.

[Alam-Saran]

Working on this issue and analysis in progress.

[Alam-Saran]

I tried the multiple scenarios/ways to reproduce the issue reported in the github with different memory threshold but unable to reproduce it. Waiting for further updates on this.