Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integration test improvements and a few bugfixes #2789

Merged
merged 7 commits into from
Dec 2, 2022

Conversation

na--
Copy link
Member

@na-- na-- commented Dec 1, 2022

Split among the first 3 commits are various improvements and additions to the existing integration tests:

  • proper exit code detection in integration tests
  • mocking of the cloud API and checks for run_status and result_status values in k6 run --out cloud tests
  • support for using and testing the k6 REST API in integration tests, without port collisions
  • support for mocking signals (like Ctrl+C) mid-test run

The 4th commit actually fixes #2788, a small bug I noticed while trying to remove the whole core.Engine (#1889). However, because the Engine removal will likely be merged at the start of the v0.43.0 cycle and because the bug was easy to fix even in the current implementation, I decided to fix it now.

This way we'll get the fix released in k6 v0.42.0 and I'll have the test already present in master, in preparation for the bigger PR.

The 5th commit fixes a minor data-race with the progressbars and masks a few others with dangling goroutines after errors in the init phase, see #2789 (comment) for details. And the 7th commit fixes another minor data race in the old Engine, see #2789 (comment).

Before this, an integration test might have specified an expected non-zero exit code, but the test would have succeeded even if k6 finished normally, with a 0 exit code, by just returning from the main function and not calling os.Exit() explicitly.
@na-- na-- added this to the v0.42.0 milestone Dec 1, 2022
@na-- na-- force-pushed the bugfixes-and-test-improvements branch 2 times, most recently from d77c1dc to b5c68fd Compare December 1, 2022 08:46
@na-- na-- force-pushed the bugfixes-and-test-improvements branch from a3b61e6 to a6823b2 Compare December 1, 2022 10:05
@na--
Copy link
Member Author

na-- commented Dec 1, 2022

I added one more commit to fix a data race and mask a few others, where goroutines spawned by go run persist and log things after cmdRun.run() and even newRootCommand().execute() have finished executing... 😞

As I wrote in the FIXME and TODO comments, the proper fix will have to wait until #2790 and #1889 have been implemented, hopefully in the next cycle.

oleiade
oleiade previously approved these changes Dec 1, 2022
Copy link
Member

@oleiade oleiade left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left a couple of non-blocking nitpicks. Really great work, glad to see those improvements 👏🏻

cmd/root_test.go Outdated Show resolved Hide resolved
cmd/root_test.go Outdated Show resolved Hide resolved
cmd/root_test.go Outdated Show resolved Hide resolved
@na-- na-- force-pushed the bugfixes-and-test-improvements branch from b37b0dd to 94bafd6 Compare December 1, 2022 15:15
oleiade
oleiade previously approved these changes Dec 1, 2022
Copy link
Contributor

@imiric imiric left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! 👏 Just left some minor comments.

cmd/integration_test.go Outdated Show resolved Hide resolved
cmd/integration_test.go Show resolved Hide resolved
cmd/integration_test.go Show resolved Hide resolved
cmd/integration_test.go Show resolved Hide resolved
The Engine didn't wait for the run_status to be properly propagated to the outputs before it finished executing, leading to a potential data race.
@na--
Copy link
Member Author

na-- commented Dec 1, 2022

These tests were more useful than expected, found another data race in the current Engine... 😭 The Engine didn't wait for the run_status to be properly propagated to the outputs before it finished executing, leading to a potential data race.

This will also be fixed in the new architecture, it's just one more reason why the current Engine architecture sucks. Still, the fix was relatively easy (129fdb6), so I chose to do it rather than disable the test that triggered it.

@na-- na-- changed the title Integration test improvements and a bugfix Integration test improvements and a few bugfixes Dec 1, 2022
@na-- na-- merged commit 56fcad0 into master Dec 2, 2022
@na-- na-- deleted the bugfixes-and-test-improvements branch December 2, 2022 10:37
@na-- na-- mentioned this pull request Dec 5, 2022
@na-- na-- added the bug label Dec 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Incorect run_status in early-aborted k6 run --out cloud test
4 participants