[optimistic] Per-instance waitgroup inaccurately reflects status of optimistic processing

[michaelneuder]

Describe the bug
As pointed out by Auston and Max of aestus relay, the current waitgroup design for handling optimistic block processing only works on a per-instance basis. In order to work as intended, some cross-process synchronization step is needed. Probably the best way to do this is through redis.

Expected behavior
The waitgroup is incremented up receipt of each optimistic builder submission:

mev-boost-relay/services/api/service.go

Line 609 in c52858a

api.optimisticBlocksWG.Add(1)

It is then waited on in a few places to try to ensure there is no outstanding optimistic blocks being processed:

mev-boost-relay/services/api/service.go

Line 1304 in c52858a

api.optimisticBlocksWG.Wait()

Since there are multiple builder API and proposer API instances, the per-instance waitgroup isn't sufficient to ensure that all the optimistic blocks have been processed.

Additional context
Auston has explored changes to redis to handle this coordination: austonst@f52c977

[metachris]

makes sense to me, code looks good too if you think that's the right approach 👍