load tests fail obscurely when nix dependencies change

[robx]

The Loadtest (Nix) CI check compares Postgrest versions on the main and PR branches. If the nix dependencies are changed in a PR, this will typically cause build failures of the main branch, causing the check to fail.

E.g., in PRs #2306 and #2292, we've seen the Loadtest (Nix) CI check failing with

Running loadtest on "main"...
2022/06/07 23:54:11 encode: can't detect encoding of "stdin"
Error: Process completed with exit code 1.

(e.g. here: https://github.com/PostgREST/postgrest/runs/6784583539?check_suite_focus=true)

• improve the test output so that it's clear what's happening nix: more explicit logging in load test #2309
• skip the CI check or make it optional when the cabal file changes
• (stretch goal) fix the load test to work even when dependencies change

[wolfgangwalther]

I observed this earlier (didn't bother to look it up when and where) and came to the conclusion that this is unavoidable, basically.

IIRC when the cabal file changes, this will be the consequence - and it's perfectly fine. The idea for the loadtest is to compare code the code we have written and the impact it has on performance. It is not meant to test performance of dependencies or performance in any setting close to reality. It's quite artificial in fact.

I think what we should do is: Run the loadtest conditionally and only when the cabal file does not change - i.e. we don't have a dependency update. We just need to make sure that our PRs are either dependency updates or changes to actual code. Then, it would be absolutely fine to just not run the loadtest for dependency updates.

[robx]

@wolfgangwalther Thanks, yes that makes sense. I'll update the issue accordingly. Also thanks for review on #2309!