Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Snapshot testing] Improve test consistency #1103

Open
wants to merge 11 commits into
base: main
Choose a base branch
from

Conversation

meissadia
Copy link
Contributor

Closes #1099

I explored a few options to reduce the flakiness of our snapshot tests and the approach implemented in this PR seems to provide the most reliable result with sensible tradeoffs.

Local test runs in headless mode have passed 100%, 6x in a row, with 4 workers. Snapshot testing via the UI can still be rather flaky but, since tests pass consistently in headless mode, my interpretation is that UI failures are due to excessive system resource consumption by the Playwright UI causing inconsistent page rendering results.

Changes

  • Reduced the suite-wide config's maxDiffPixels from 25 to 0, allowing us to catch even the smallest unexpected UI change
  • Added snapshotModifications.css to apply snapshot-testing-specific style modifications. This approach allows us to replace dotted borders with solid borders on all links to ensure consistent rendering. Since the dotted style comes from the CFPB Design System, it does not seem necessary to verify it's implementation as part of our app (though it would certainly be preferred that these tests validate exactly what the user is expected to see).
  • Added the page header's USA flag to the suite-wide mask (ignore) list. This element tends to trigger false-positives on tests with very tall screenshots and, since it's implementation comes from the Design System, does not seem to be something we need to validate as part of our app. As a next step, we could break these taller pages down into smaller tests of subsections of the page, but for now this workaround seems acceptable.
    • Screenshot 2025-01-09 at 11 36 49 AM

How to test this PR

  1. Checkout this branch and yarn start
  2. yarn test:e2e:snapshot --headless
  3. Ensure all tests pass

Notes - Alternate approaches not chosen

  • I did experiment with masking the entire page footer, since that was the most frequent trigger of failures due to dotted border issues, but still encountered failures due to links within the page content, so that didn't feel like the most helpful approach.
  • I considered breaking full-page tests into smaller element tests, but found that with the masking of .u-usa-flag and modifying the link border style, we were able to avoid the effort needed to significantly refactor how these tests are implemented.

Screenshots

screencapture-localhost-9323-2025-01-09-11_33_55

@meissadia meissadia force-pushed the 1099-e2e-mask-footer branch from e871363 to ecb8779 Compare January 9, 2025 19:09
@meissadia meissadia marked this pull request as ready for review January 9, 2025 19:10
@meissadia meissadia enabled auto-merge (squash) January 9, 2025 19:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Snapshot] Reduce flakiness of Footer snapshot tests
1 participant