You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Two weeks ago (let's say approximately since 2024-03-15, when the first 1.37.x resharding happened), we started receiving complaints about transactions needing to be added on the ReadRPC side.
This means that queries like EXPERIMENTAL_tx_status or tx are responded to with UNKNOWN_TRANSACION.
We haven't handled this problem entirely, so the issue is ongoing.
This is related to the transaction_details stored in ScyllaDB
If we run tx-indexer around the blocks with "missing" transactions, they appear
We figured that when a transaction is missing, we actually have it in ScyllaDB. The error we experience is borsh-deserialization error
We have reindexed transaction around those blocks to make them work.
While I can find an explanation for the two of them (more than a few months ago) the majority of the transactions are new. It means they were indexed with this issue even though we have introduced the additional checks to prevent it. It doesn't make sense to me.
We double checked borsh and can confirm it has all necessary checks and guaranteed deterministic ser/de. The only suspect is ScyllaDB. I assume there's something happening during the data distribution on the write. We do only 3-4 checks to prevent them. Perhaps it's worth to:
Add metric to watch for the error rate for transactions saving
Two weeks ago (let's say approximately since 2024-03-15, when the first 1.37.x resharding happened), we started receiving complaints about transactions needing to be added on the ReadRPC side.
This means that queries like
EXPERIMENTAL_tx_status
ortx
are responded to withUNKNOWN_TRANSACION
.We haven't handled this problem entirely, so the issue is ongoing.
transaction_details
stored in ScyllaDBtx-indexer
around the blocks with "missing" transactions, they appearborsh
-deserialization errorThe next updates will go in the comments*
The text was updated successfully, but these errors were encountered: