Performance regression in indexer 2.15.1 for transactions endpoint

[onetechnical]

Subject of the issue

With 2.15.1 and newer, there appear to be latencies in requesting e.g. /v2/transactions?min-round=$min_round&max-round=$max_round. Whereas before this was very quick, in later versions on TestNet at least, these may take several seconds to return.

This might be related to the following:

c7b3b78

Your environment

• Software version: algod v3.13.3-stable
• Operating System details: Ubuntu 20.04, i4i.8xlarge instance, local NVMe, local postgresql 13
• Network: testnet

Steps to reproduce

1. Caught up testnet indexer and algod
2. Execute a series of queries for transactions, one round at a time.

Here is an example bash script:

#!/bin/bash

INDEXER_API='localhost:8980'
INDEXER_TOKEN='YOUR_INDEXER_TOKEN'

function get_round {
    curl -s $INDEXER_API/health | jq .round
    return
}

function get_transactions_for_round_range {
    min_round=$1
    max_round=$2
    curl -s -H "X-Indexer-API-Token: $INDEXER_TOKEN" "$INDEXER_API/v2/transactions?min-round=$min_round&max-round=$max_round" > /dev/null 2>/dev/null
    return
}



function main {
    round=$(get_round)
    let start_round=$(( $round - 10000 ))
    while [ $start_round -lt $round ]; do
        echo -n "$start_round   "
        time get_transactions_for_round_range $start_round $start_round
        let start_round=$(( $start_round + 1 ))
    done
}

main

Expected behaviour

Fast responses

Actual behaviour

Responses can take around 8s.

[robdmoore]

I'm unsure if this is the same issue or not, but I've recently found that there is an intermittent, but consistently occurring and significant performance issue with indexer when running the automated tests for https://github.com/algorandfoundation/algokit-subscriber-ts.

There are some tests in that library that use the following function in a series of tests (e.g. filters.spec.ts and events.spec.ts): https://github.com/algorandfoundation/algokit-subscriber-ts/blob/main/tests/filterFixture.ts#L29. Namely the call to waitForIndexerTransaction.

This method is defined in algokit-utils here: https://github.com/algorandfoundation/algokit-utils-ts/blob/main/src/testing/fixtures/algorand-fixture.ts#L122 and consists of these two underlying methods: https://github.com/algorandfoundation/algokit-utils-ts/blob/main/src/testing/indexer.ts#L12 and https://github.com/algorandfoundation/algokit-utils-ts/blob/main/src/indexer-lookup.ts#L20.

The behaviour I'm seeing is that intermittently, but consistently (multiple times every test run) indexer will wait for about 17s (almost exactly) until the transaction that's being waited for appears. The rest of the time it will happily appear sub-second. This makes me think there some sort of deadlock or delay somewhere.

I've looked at logs in algod, conduit, indexer and postgres and attached a profiler of sorts of postgres, but haven't seen anything conclusive yet.

Two things I have found:

• The problem seems a lot worse with recent versions of algod/indexer/conduit
• The problem was even worse still when I used this code (the previous code in this commit) instead: algorandfoundation/algokit-subscriber-ts@0ec7735

[urtho]

@robdmoore
To confirm this is an issue with follower getting stuck on random relays please try setting this in conduit for a source of blocks:

netaddr: "http://mainnet-api.algonode.cloud"
or for testnet:
netaddr: "http://testnet-api.algonode.cloud"

This switches to a cloud based "instant" follower that is synced to any round you need and does not suffer from intermittent pauses.

If the 17sec gaps disappear then we need to start digging into the follower issue.

[robdmoore]

To be clear: I'm running these tests on AlgoKit LocalNet aka with dev mode algod (hence the 17s is really noticeable vs the sub-second near-instant commit and index times).

I could try and run the tests against testnet, but it wouldn't be quite the same test given how different dev mode is. If I get a chance though I'll give it a crack and see what happens.

[gmalouf]

So I'm following, we think this is a dev mode specific issues on an indexer 3.x install?

[robdmoore]

I assume so if there are no performance issues seen in normal indexer. Noting it's not necessarily indexer itself, but could be conduit it's unclear which is the cause of the 17s delay in indexing.

It's easily reproduceable with the above-linked test suite.

[gmalouf]

Closing this in favor of #1613 - the original issue in the description no longer exists (indexer 2.x has been retired and the described behavior has not been observed on mainnet/testnet in 3.x outside of dev mode).