Cache the reasons why transactions are dropped to support troubleshooting

[marun]

Why

Recent attempts to enable e2e testing of C-Chain workflows (specifically atomic transactions) in avalanchego ran into problems where transactions were dropped with no easy way to diagnose the reason. This was in contrast to X- and P-Chain atomic transactions, where a failed transaction generally returned the reason for the failure.

Issue tracking the flakes that prompted the filing of this issue: ava-labs/avalanchego#1993

Suggested How

Since evm transaction issuance doesn't support returning a reason a transaction was dropped (only the status is returned), maybe it makes sense to maintain a cache mapping the ids of dropped transactions to the reasons the transactions were dropped and allow requests to be made to that cache? e.g. GetDroppedReason(txID) (string, error)

Alternatives

Another approach might be to increase the detail of evm logging around a transaction being dropped. This would allow determining why a transaction was dropped so long as someone had access to the logs. For non-test scenarios, though, a user may not have access to the logs.

Ideally both logging and API access to dropped reasons would be possible, but if only one were possible the logging is likely the lower-hanging fruit.