Consistency of UTxO set in Blockfrost query layer

[klntsky]

We've been working on introducing Blockfrost support to cardano-transaction-lib as part of our Catalyst Fund9 project. The goal of the project is to add support for Blockfrost as an alternative query layer and provide functionality on par with our own Kupo+Ogmios backend.

We are having troubles ensuring that the UTxO set changes are propagated after transaction submission.

In particular, we discovered that:

• If a transaction exists, it is not guaranteed that the UTxOs it created are available to query in /addresses/{address}/utxos, essentially, it is added to the DB before it is confirmed.
• Even if the UTxOs are available in /addresses/{address}/utxos, /utils/txs/evaluate endpoint may still refuse to see them as available, because, AFAIK, Ogmios can lag behind the chain tip of db-sync. This is especially problematic for us, because in scenarios with multiple dependent transactions sent in a row, we constantly hit errors. And we can't check if the UTxOs are seen to Ogmios before we actually use the endpoint.

We implemented a dirty workaround for this problem - an aritificial delay that can be set to a big enough value to make our tests pass.

Would it be possible to improve the situation?

[mmahut]

I assume this problem is due to forks (different nodes seeming different state of the chain at all times) and not due to lagging, as usually db-sync tends to lag behind ogmios usually as it takes time to process and insert data.

Please note, that this will be never consistent due to rollbacks. Even if the transaction makes it into the mempool (we have a /mempool endpoint now BTW) and is visible by one instance of cardano node, it could be dropped from the network entirely during a rollback.

Because of this, it is recommended for applications to rely a transactions being on chain only by looking at the number of confirmations. The number of confirmation here really depends on the use case, practically the chain becomes truly immutable only after 36 hours or so but I do not think I have seen larger rollback than few blocks (not counting the epoch switch where the forks might be much larger).

We're working on providing a new endpoint that would support [additonal UTxOs sets](additonal UTxOs), which should help for testing.

[mmahut]

Sorry for misinformation, the finality is set to 2160 blocks, that means it is about 12 hours, not 36.

Looking at our internal Cardano nodes, the block propagation between them is within one second for 92.1%, so I really think this is due to forks. We relay the transactions to the network, but the SPOs making the next block might be on a totally different version of the chain.

[klntsky]

I think that rollbacks are a tangential issue.

The problem is that Blockfrost does not provide a consistent view of the UTxO set in its own endpoints (the second point of my original post) at any given time, because there are, in fact, two query layers with different chain tips.

This problem absolutely never happened when we used our own query layer (Ogmios).

as usually db-sync tends to lag behind ogmios usually as it takes time to process and insert data.

I forgot to mention that the issue is not deterministically reproducible. I have observed an error from Ogmios (/evaluate) citing unkown inputs after the UTxOs have been added - multiple times, so I believe sometimes it lags behind.

We're working on providing a new endpoint that would support [additonal UTxOs sets](additonal UTxOs), which should help for testing.

That's great! We'll be able to use Tx chaining. But it would not allow us to move forward with this specific problem. We could, of course, employ more hacks, e.g. provide all used UTxOs as additional UTxOs to Ogmios when calling /evaluate, but I think it would be better to do this on your side so that it could benefit all users and not just CTL users.

Looking at our internal Cardano nodes, the block propagation between them is within one second for 92.1%, so I really think this is due to forks.

Does Ogmios look at the same cardano-node the dbsync looks at? If so, the forks are not related, because cardano-node has its own consistent view. Or do you have a particular scenario I fail to consider?

[mmahut]

If so, the forks are not related, because cardano-node has its own consistent view.

They are, because a rollback takes some time until it is propagated and processed within the db-sync database (up to several seconds, deleting and inserting new records). Ogmios is looking at the current acquired snapshot directly as a ledger snapshot, where db-sync is processing the incoming blocks too. During this times, there is a high chance of discrepancy.

As these are two different tools, I do not think there is a way to ensure an absolute consistency between them. I cannot think of one, I welcome any suggestions.

Either, the end-user should ensure they are only counting UTxOs that are considered stable from the chain point of view (by number of confirmations) or they can emulate the evaluation using the additional UTxOs feature taking into account that this given view might not exist in the future due to rollbacks.

[klntsky]

As these are two different tools, I do not think there is a way to ensure an absolute consistency between them. I cannot think of one, I welcome any suggestions.

I think one of the options is to fork Ogmios and replace the part where it interacts with cardano-node to get confirmed UTxOs with a query to the DB where dbsync writes data.