bug(zero-bin): Inconsistent async failures on block batches

[Nashtare]

When proving batches of blocks (size 10), I got several issues with output as follows:

Error: Failed to send proof

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
   1: <futures_util::future::future::flatten::Flatten<Fut,<Fut as core::future::future::Future>::Output> as core::future::future::Future>::poll
   2: <futures_util::future::future::Then<Fut1,Fut2,F> as core::future::future::Future>::poll
   3: <S as futures_core::stream::TryStream>::try_poll_next
   4: <futures_util::stream::try_stream::try_collect::TryCollect<St,C> as core::future::future::Future>::poll
   5: prover::ProverInput::prove::{{closure}}
   6: leader::client::client_main::{{closure}}
   7: leader::main::{{closure}}
   8: tokio::runtime::park::CachedParkThread::block_on
   9: tokio::runtime::context::runtime::enter_runtime
  10: tokio::runtime::runtime::Runtime::block_on
  11: leader::main
  12: std::sys_common::backtrace::__rust_begin_short_backtrace
  13: std::rt::lang_start::{{closure}}
  14: std::rt::lang_start_internal
  15: main
  16: <unknown>
  17: __libc_start_main
  18: _start

The errors are inconsistent between runs, but we should add some strengthening mechanism (with possibly some retry-upon-error kind of behavior), to prevent this.

I've been running as follows:

for n in {20175010..20180000..10}
do
    ./prove_rpc.sh $n $((n+9)) endpoint native true 1000 1000 test_only
done

(note that these block ranges are past Cancun upgrade, they have been ran against fix/native_cancun branch)

[atanmarko]

Fixed with increasing the ulimit: ulimit -n 4096

[Nashtare]

Seems there are still some sporadic issues with even ulimit -n 16384.

Ran with for n in {20175180..20180000..10}; do time ./prove_rpc.sh $n $((n+9)) rpc_url native true 1000 1000 test_only; done

[BGluth]

Maybe this is a bit of an aside, but I'll bet that the error:

Error: Failed to send proof

is probably an anyhow Result and that only the most recent context is getting printed and not the other contexts that are added as it calls down the stack. Probably with the rest of the contexts it might have given more of a hint that we were running out of file descriptors.

[atanmarko]

I will look into this in more depth soon

[atanmarko]

Fixed with #369