Remove the Engine and split metrics/VUs handling

[na--]

The current Engine vaguely resembles a Rube Goldberg machine... It has slowly been getting better, for example #1869 improved it slightly by mostly decoupling the outputs from the Engine, but it still has quite a lot of moving parts that can be streamlined and improved. And as #1887, it still has some gotchas and potentially some other hidden bugs or data races....

I'm opening this issue as an epic where we can gather Engine issues and ideas for fixing them. Here are some specific problems that currently exist, though there are quite a few other TODOs sprinkled in the code already:

• the engineWait and engineRun callbacks are more than a bit strange, but they were the easiest way to handle the --linger peculiarities: https://github.com/loadimpact/k6/blob/2036baeaa83d76ce17aae95fef7e78e3ef24c735/cmd/run.go#L255
• stopping and starting outputs doesn't really need to be in the Engine anymore, so it probably shouldn't be: https://github.com/loadimpact/k6/blob/2036baeaa83d76ce17aae95fef7e78e3ef24c735/core/engine.go#L133-L134
• the fix for handleSummary called before all metrics are crunched #1887 is a bit sketchy and it might be a good idea to add a timeout for it (Fix metrics data race in the Engine test run finalization #1888 (comment)), or maybe completely refactor the way metrics are processed, so we don't need such hacks. For example, maybe decouple metric ingestion and metric processing, sort of how outputs do it?

Needless to say, but this refactoring should definitely be done after #1832, and maybe even some of its related issues, given that they will probably substantially change the requirements.

[na--]

I've been thinking on some related issues to this problem recently and I think I was wrong ⬆️, especially with this:

stopping and starting outputs doesn't really need to be in the Engine anymore, so it probably shouldn't be

I think it might actually be the other way around 😅 Instead of refactoring metrics handling or outputs out of the Engine, we should probably refactor the test execution out of it 💡 It is actually the thing that doesn't really fit all that well. Most of it is handled by the ExecutionScheduler anyway, the Engine just gets in the way.

So, my current idea is that we should actually focus the Engine completely on metrics, outputs, thresholds data crunching, etc. We can even rename it to something more appropriate like a MetricsEngine or something like that and move it out of core/ and into metrics/. Maybe even split it apart and tackle the built-in metrics and the outputs separately... 🤔

And we can focus the ExecutionScheduler on the actual test execution and likely also get it out of core/ and into some better place, e.g. /execution (related to #1302).

A big potential benefit of this decoupling would be the fact that we might be able to much more easily support test suites (i.e. multiple test scripts in a single k6 run, see #1342) with it. A single central MetricsEngine and multiple independent ExecutionSchedulers, one for every sub-test 💡

[na--]

A lot of these changes were done in #2426 and already merged. A rough version of the remainder are implemented in #2438, so they need to be split into a separate PR next cycle, after rebasing and refactoring the tests to use the new APIs.