Using captureKicksDiagnostics to speed up multiple plugin tests

[komikat]

Extending #4144 to cabal-plugin-tests and merging upstream changes on hlint-plugin-tests
Update: Extending to multiple plugins as noted below.
Update 2: Scope reverted to cabal and hlint plugins. (ref #4371)

TODO

• update the remaining tests which waitForDiagnostics unconditionally
• resolve the one hlint-plugin test failing after the update
• fix pre-commit issues

[komikat]

hey @soulomoon, any suggestions?

[fendor]

@komikat Did you investigate the failing test case?

[komikat]

Yes, I'm looking into it, will hopefully have something by tomorrow.

PS: the failing testcase is ignoreHintGoldenResolveTest.

[fendor]

This looks like... the testcase is wrong? I can't explain it, but changing "Eta reduce" with "Redundant id" works for me locally. The testcase itself is still wrong, you have to accept the differences, but no unknown code actions any more.

[fendor]

Perhaps we lack a .hlint.yaml in the temp directory or something?

[soulomoon]

@komikat since I do not familiar with cabal plugin, I can only give very general suggestions. Take consideration of the following.

1. the time gap the debouncer needed to emit the diagnostic
2. Be sure the number of cabal kicks that would happened in the test logic. Seems like @fendor 's new finding suggest some incorrect logic in the test case?

[komikat]

hlint --hint=test-hlint-config.yaml UnrecognizedPragmasOff.hs
only returns Eta Reduce so it is a hlint config file issue

[komikat]

Setting .hlint.yaml correctly breaks the testcase "provides 3.8 code actions including apply all" while not fixing this one at the same time, seems like the .hlint.yaml might not have been copied correctly for the unrecognized pragma test case.

Will also have to look into "provides 3.8 code actions including apply all" since it's correctly failing after ignoring redundant id, it expects both redundant id and eta reduce currently.

[komikat]

Moved hlint tests to the virtual file system and fixed a config issue with the resolveTests.
hlint-plugin-tests takes 15s now.

[soulomoon]

good work！

[fendor]

Thanks for pushing this over the finish line! Please also make sure to remove dead code, e.g. code that is commented out.

[komikat]

Updated PR scope, please refer TODO.

[fendor]

LGTM, doing a final CI run to make sure the tests didn't get more flaky

[komikat]

seems like the timeout in PluginSimpleTests still happens erratically (unmarked as broken in #4259)