enhancement: Initializing conversations using newConversation takes 20-60 seconds

[fabriguespe]

Describe

Steps to Reproduce:

1. Use a wallet with thousands of conversations (around 7,000 to 10,000).
2. Attempt to initialize a new conversation using newConversation.

Expected Behavior:

Initialization should complete in a more reasonable time frame (milliseconds to a couple of seconds).

Actual Behavior:

It takes 20-60 seconds to initialize a new conversation.

Observations:

• Redis caching has improved the time from 5-8 minutes to 20-60 seconds, but it's still not optimal.
• This issue seems to affect wallets with a large number of conversations.

[rygine]

@fabriguespe can you provide a way to reproduce this?

[fabriguespe]

I could simulate thousands of wallet messages and measure conversation creation delays progressively.

is there a way to test this scenario with our benchmarking tools?

cc @daria-github

[seeARMS]

I'm reasonably confident we're running into this issue at paragraph.xyz.

We delivered ~100k XMTP messages to >6k unique wallets this quarter. Lately we've been timing out on newConversation. For some reason this also seems to be blocking the NodeJS event loop? It's preventing other Express requests from being handled when newConversation stalls.