This is a companion report to a report that was already filed against go-libp2p-pubsub here: libp2p/go-libp2p-pubsub#524

The main advice we received from @vyzo working on go-libp2p-pubsub was to utilize libp2p validators for our incoming pubsub messages. Is that something that kubo exposes currently?

Also, even if the libp2p validators are exposed via kubo APIs, it will take time to set up the right ones and get our community of node operators to upgrade to utilize them. In the meantime our network is still in a bad state right now so I'm very interested in any ideas for what we can do in the short term to reach a stable state again.

Thank you!

A possible solution that can solve your current predicament is to provide a default validator in ipfs that uses the message envelope seqno as the nonce.

This works for all peers/topics, and can reject/ignore old messages without needing to look into the message payload itself.

This can be implemented quite easily and provide the bandaid needed to stop the bleeding and buy time to design the proper abstractions for a validator api.

A bit more on the mechanics of this.

The proposed validator relies on message signing and use of the seqno, which are default features used by ipfs.
This means that every message has an origin peer, a signature (validated by pubsub lib itself) and a monotonically incrementing seqno in the envelope (constructed by the pubsub lib, initialized with the current unix timestamp in nanos and incremented thereafter).

Thus we can use this seqno as a per peer nonce and extinguish the kind of floods that is being observed here.
Note that the per peer nonce shouldnt in generally be kept in memory (modulo caching) to avoid creating a dos vector.

The validator can be registered by every topic thats is joined, or we can add an api in pubsub for default validators for all topics.

Thanks for the ideas, @vyzo! This is very helpful.

So we could expose RegisterTopicValidator and UnregisterTopicValidator like we did with WithSeenMessagesTTL (but as a function, of course)?

I plan on working on this tmrw, as it is holiday here today.
Unless you want to pick it up today of course.

I will add a new api to pubsub for default validators, and write a default validator that behaves as described in kubo.
No need to expose anything new.

I plan on working on this tmrw, as it is holiday here today. Unless you want to pick it up today of course.

I will add a new api to pubsub for default validators, and write a default validator that behaves as described in kubo. No need to expose anything new.

Would really appreciate the help, @vyzo 🙏🏼 Our whole team is all-hands-on-deck debugging and testing in preparation for a major product launch at Eth Denver over the next couple of days (hence our concern about Pubsub flooding at this time).

The current plan: kubo maintainers will try to get out a patch release that includes Vyzo's proposed solution this week.

Really appreciate the attention here @vyzo and @p-shahi!!!

So there are a couple of parties involved here:

1. vyzo@ for the go-libp2p-pubsub work
2. ipfs/kubo-maintainers for doing a Kubo release.

This is on the Kubo maintainers' 2023-02-28 standup agenda to discuss to figure out what we can do here. We understand this is time sensitive for you all. We'll update the issue after standup (by 19UTC on 2023-02-28).

Datastore API we want pubsub validator to consume: https://pkg.go.dev/github.com/ipfs/go-datastore#Datastore (with a wrapper in Kubo side).

Notes from 2023-02-28 conversation with @Jorropo and @vyzo :
@vyzo is going to make the go-libp2p-pubsub changes today (2023-02-28). I'll ask @MarcoPolo to review.
@vyzo will make the change in Kubo tomorrow (2023-03-01). He'll do this change against 0.18.1. @Jorropo will review.
@galargh will cut a patch release. This will happen on Thursday, 2023-03-02, European morning at the latest.

Ceramic: a few things:

1. General: Kubo maintainers / PL EngRes is working to help support your launch/announcements this week.
2. Is there a slack channel where you usually engage with PL folks in? (Apologies for not finding it as I know we have had recent conversations here. We can also talk in #ipfs-operators.)
3. Per above we're aiming to get a patch release for you on Thursday European time. You can track this here: Release 0.18.2 #9679
4. After you get through your event, I think it's worth having an architecture conversation as on the surface I'm a bit worried about Kubo the binary being a critical part of your setup. I believe since the past conversation when you all moved from js-ipfs the landscape has changed including:

• Additional investment / changes in js-libp2p including full DHT support and up-and-coming js-ipfs replacement in Helia (not saying though you need to go back to JS world - just wanting to make sure you're aware)
• Ability to more easily customize Kubo with FX plugins
• go-libipfs effort for empowering groups to build the ipfs implementation they want
  I'd like to educate myself and other team members on your usescass/needs and ensure we're steering you in a direction that best supports you.

The pubsub side of things is here: libp2p/go-libp2p-pubsub#525
This provides the necessary API support and an implementation, modulo an interface to the datastore.

Thanks @vyzo @BigLep for all your work driving this forward! Wanted to let you all know that at the end of the day yesterday the pubsub flood resolved, so we are no longer in an active crisis situation at the moment. That said, this has happened before and will likely happen again, so we're still very interested in getting the fixes you all are working on to help protect us going forward. And we'd absolutely love to have an architecture review and discuss how we can improve the way we build on kubo/libp2p to set us up for success in the future.

Also @BigLep, we do have a shared channel in the filecoin slack: https://filecoinproject.slack.com/archives/C01V5AWPF97

Pubsub release has been cut in https://github.com/libp2p/go-libp2p-pubsub/releases/tag/v0.9.2

I will hook it up to ipfs next.

@stbrody : you bet.
Given you're not in an active crisis situation, we are just going to include this in 0.19 (RC is planned for tomorrow). Because of dependency updates, it is unfortunately more challenging to backport to a 0.18.2. We can do it if necessary but want to avoid the legwork if it isn't needed. (Doing a patch release also involves some manual work so we'll save there as well.)

Concerning architecture review, let's connect after Denver. Feel free to reach out on FIL Slack when you're back and settled.

FYI @vyzo @BigLep - this is happening to us on our mainnet again. Seems to have started about 28 hours ago when everyone on our team was traveling back from EthDenver

@stbrody : ACK. @Jorropo is working to get CI to pass so #9684 can be merged.

@stbrody : we've run into some snags here. @Jorropo will get these moved a top level issue tomorrow (2023-03-10). Assuming you're all back from EthDenver, I want to get the ball rolling on understanding your needs and how we unblock you since per above we aren't going to merge the PR fix into master.

Can you please invite @BigLep , @Jorropo , and @lidel into a collaboration channel in Filecoin Slack to help with the coordination? (Or if you want me to create a new channel I can do that.). (I can't currently access https://filecoinproject.slack.com/archives/C01V5AWPF97 - I assume it's private.)

Slack access has been granted and I have moved the conversation there for scheduling.