Memory leak issue

[twoeths]

• Version:
  v1.1.1

• Platform:
  Linux 5.15.0-79-generic #86-Ubuntu SMP Mon Jul 10 16:07:21 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

• Subsystem:

Severity:

Medium

Description:

Refer to the profile taken by a lodestar beacon-node here

There are > 15k of objects retained belonging to these classes:

• Socket
• ConnectionImpl
• Timeout
• XXHandShake
• MPlexStreamMuxer

Steps to reproduce the error:

Run a lodestar mainnet node and leave it for a long time, then monitor heap size of network thread

[twoeths]

I found all Sockets have a timeout and they're all destroyed

all connection statuses are also "closed"

also there are socket events attached during the MultiaddrConnection.close() function, especially the "drain" one

it seems to me the MultiaddrConnection.close() was called, but somehow it still keep some resources especially the socket listeners

[SgtPooki]

@tuyennhv slightly off-topic but is there any chance you could share the grafana dashboard json you use over at lodestar so we can use it in ipfs/helia-http-gateway ?

[twoeths]

@SgtPooki here's the libp2p dashboard in lodestar https://github.com/ChainSafe/lodestar/blob/unstable/dashboards/lodestar_libp2p.json

[twoeths]

@achingbrain need to reopen until it's confirmed this is fixed in lodestar, waiting for a new release of transport-tcp to test

[achingbrain]

@tuyennhv did you manage to test the new release?

[twoeths]

@achingbrain we tested v9.0.22 and found it does not resolve the memory leak issue. It also introduced some performance impact that caused node to forward less gossip messages to mesh peers, I'm not sure if it links to libp2p-tcp specifically or other libs during the upgrade to have to rollback in our release

see https://github.com/ChainSafe/lodestar/pull/6701/files

[twoeths]

I think we need to upgrade all libp2p dependencies to the latest versions and evaluate again cc @wemeetagain
need to take a new profile with leaked memory for libp2p-tcp >= v9.0.22 to move this forward

[achingbrain]

I think we need to upgrade all libp2p dependencies to the latest versions

Please do, there was a small leak fixed by @libp2p/[email protected] so that's a good idea in general.

[twoeths]

in progress PR by @wemeetagain : ChainSafe/lodestar#6711