feat(test runner): allow multiple global setups #32955
Conversation
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Oof, my change swaps the ordering of existing |
|
Discussed this with Dima in a call. Reverting the ordering change so we don't break anything. This makes the ordering harder to reason about when folks mix both APIs, but that's discouraged anyways. |
|
|
||
| Path to the global setup file. This file will be required and run before all the tests. It must export a single function that takes a [FullConfig] argument. | ||
| Path to the global setup file. This file will be required and run before all the tests. It must export a single function that takes a [FullConfig] argument. Pass an array for multiple global setup files. |
There was a problem hiding this comment.
I kept this pretty short so we don't overload it. Let me know if you think we need to talk about ordering here.
There was a problem hiding this comment.
We literally overload it here, which is fine. Not sure the wording of your doc is right though, please revise.
There was a problem hiding this comment.
I think it's right - what would you recommend instead?
There was a problem hiding this comment.
You are fine. It felt like a couple of words are missing. "Pass an array of file names to specify multiple global setup files"
There was a problem hiding this comment.
good idea! added the words in c8c483c.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
|
||
| Path to the global setup file. This file will be required and run before all the tests. It must export a single function that takes a [FullConfig] argument. | ||
| Path to the global setup file. This file will be required and run before all the tests. It must export a single function that takes a [FullConfig] argument. Pass an array for multiple global setup files. |
There was a problem hiding this comment.
We literally overload it here, which is fine. Not sure the wording of your doc is right though, please revise.
| this.config = { | ||
| configFile: resolvedConfigFile, | ||
| rootDir: pathResolve(configDir, userConfig.testDir) || configDir, | ||
| forbidOnly: takeFirst(configCLIOverrides.forbidOnly, userConfig.forbidOnly, false), | ||
| fullyParallel: takeFirst(configCLIOverrides.fullyParallel, userConfig.fullyParallel, false), | ||
| globalSetup: takeFirst(resolveScript(userConfig.globalSetup, configDir), null), | ||
| globalTeardown: takeFirst(resolveScript(userConfig.globalTeardown, configDir), null), | ||
| globalSetup: takeFirst(...this.globalSetups, null), |
There was a problem hiding this comment.
nit: there isn't much to take first from, this is pretty much | null
There was a problem hiding this comment.
Would you prefer this?
| globalSetup: takeFirst(...this.globalSetups, null), | |
| globalSetup: this.globalSetups[0] ?? null, |
There was a problem hiding this comment.
nit means "while you are here if you also find it weird, replace it". Your suggestion works.
This comment has been minimized.
This comment has been minimized.
| 'globalSetup3.ts': `module.exports = () => { console.log('%%globalSetup3'); return () => console.log('%%globalSetup3Function'); }`, | ||
| 'globalSetup4.ts': `module.exports = () => console.log('%%globalSetup4');`, | ||
| 'globalTeardown1.ts': `module.exports = () => console.log('%%globalTeardown1')`, | ||
| 'globalTeardown2.ts': `module.exports = () => console.log('%%globalTeardown2');`, |
There was a problem hiding this comment.
I'd also throw here, and check that it did not prevent globalTeardown1 from running.
There was a problem hiding this comment.
Did that in a72ec6e. This behaviour is surprising to me, was somehow expecting a failing teardown to stop further teardowns from being executed.
With that in mind, I don't think that a teardown function returned from globalSetup should behave any differently than a globalTeardown when it throws an error. I've made sure that a failing function doesn't block any globalTeardown script from being executed in 9b78566. Would be lovely if you could give that another look.
I think this could benefit from another refactoring where we pull globalSetup and globalTeardown scripts into separate tasks, but i'd like to get this PR merged first and then refactor in a follow-up.
There was a problem hiding this comment.
On a second note, this could be considered a breaking change. If folks are currently relying on the globalSetup result throwing an error to prevent globalTeardown from being executed, that's broken with 9b78566. So maybe we should just stick with the incongruence.
There was a problem hiding this comment.
I agree, let's not touch the relative behavior of globalSetup returned function vs globalTeardown. The aim here is to support multiple, and our general rule is that all teardowns run even if one of them fails. This applies to fixtures, afterEach and afterAll hooks, and now to global teardown as well.
Co-authored-by: Dmitry Gozman <[email protected]> Signed-off-by: Simon Knott <[email protected]>
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Test results for "tests 1"5 flaky35065 passed, 634 skipped Merge workflow run. |
Closes #32369
The idea behind ordering is that for pairs of setup/teardown, each pair is only torn down after all pairs executed after it are torn down. This allows "stacking" these setup steps onto another, where one setup depends on the previous setup.
For
globalSetup: ['s1', 's2'], globalTeardown: ['t1', 't2'], this results in orders1, s2 ; t2, t1.There's a second API some people use where setup steps return their respective teardown steps. If that API is used exclusively, the same ordering applies. Mixing both APIs is discouraged, as that gives hard-to-reason-about ordering.