SWC-6536: handle timing issues related to using larger runner #5184
Conversation
…fail flaky tests more quickly to reduce cost in CI
…form, confirm login was correct before waiting for redirect
| // This may be the first test to run and trigger SWC compilation | ||
| // ...which may take 15+ seconds. Mark this test as slow, | ||
| // ...so the test will be given triple timeout. | ||
| // https://playwright.dev/docs/api/class-test#test-slow-1 | ||
| setup.slow() |
There was a problem hiding this comment.
Successful auth.setup tests are taking 2-4 seconds on the larger Linux runner, 4-6 seconds locally, and 7-11 seconds on the macOS runner. Generally, this test is much faster than the other e2e tests, so stop marking this test as "slow" -- allow the test to fail faster when flaky, so we spend spend less money on larger runner in CI.
| const usernameInput = page.getByLabel('Username or Email Address') | ||
| await usernameInput.fill(testUserName) | ||
| await expect(usernameInput).toHaveValue(testUserName) | ||
|
|
||
| const passwordInput = page.getByLabel('Password') | ||
| await expect(passwordInput).toBeEmpty() | ||
| await passwordInput.fill(testUserPassword) | ||
| await expect(passwordInput).not.toBeEmpty() |
There was a problem hiding this comment.
Ensure that the username/password values have been set before trying to sign in
| // Ensure that correct username/password were received | ||
| const loadingButton = page.getByRole('button', { name: 'Logging you in' }) | ||
| await expect(loadingButton).toBeVisible() | ||
| await expect(loadingButton).not.toBeVisible() | ||
| await expect( | ||
| page.getByText('The provided username/password combination is incorrect'), | ||
| ).not.toBeVisible() |
There was a problem hiding this comment.
Confirm that the correct username/password were received, hopefully will help us troubleshoot if login fails to redirect as in this run
| @@ -21,7 +21,7 @@ dotenv.config() | |||
| export default defineConfig({ | |||
| testDir: './e2e', | |||
| /* Timeout to allow portal enough time to compile when running locally */ | |||
| timeout: 5 * 60 * 1000, | |||
| timeout: 2 * 60 * 1000, | |||
There was a problem hiding this comment.
Reduce overall test timeout, so flaky tests fail faster -- currently, the longest test (teams.loggedin) takes about 25s locally, 17s on larger runner, and 35-40s on macOS runner, so 2 minutes should be enough time for now. If we add a longer test in the future, we can configure a longer timeout just for that test.
There was a problem hiding this comment.
I like the idea of having these fail faster if making these longer doesn't substantially make these less flaky. Optimizing for resilience is good, but I wouldn't worry much about shaving seconds or minutes off to save money.
No description provided.