Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert E2E tests from Puppeteer to Playwright #2041

Merged
merged 35 commits into from
Aug 9, 2023
Merged

Conversation

mikkamp
Copy link
Contributor

@mikkamp mikkamp commented Aug 3, 2023

Changes proposed in this Pull Request:

This PR converts the E2E tests from Puppeteer to Playwright and removes any dependencies on the @woocommerce/e2e-* packages. This is a followup from #2030

Note
Until we resolve issue #2002 we are still using Node 14, which means the latest compatible version of Playwright was 1.33. Ideally this will be updated once we upgrade Node.

Overview of the changes to convert tests:

  1. Conversion ID
    The test Conversion ID used to be set through the Connection Test page. This has now been replaced with a custom plugin tests/e2e/bin/test-data.php which adds some API endpoints to set this data.

  2. Site Settings
    The payment gateway COD used to be enabled through an API call at the start of the test, this is now done in the wp-env start up script.

  3. API requests
    All API requests use the basic auth plugin to send requests to the API. We use this for both WordPress and WooCommerce endpoints, see tests/e2e/utils/api.js

  4. Avoiding waitForSelector
    In our previous tests we often used waitForSelector, while this is still compatible with Playwright, I've replaced it with other Locators. This was done to simplify the tests and reduce the need for utility functions.

  5. Mocking API requests
    Instead of using a utility for mocking and spying on API requests, it's been replaced with the route.fulfill. We use this to mock connections to Jetpack or Google.

Closes #1599

Detailed test instructions:

  1. Checkout this PR and install packages: nvm use && npm ci
  2. Start the wp-env environment: npm run wp-env:up
  3. Install the Chromium browser for Playwright: npx playwright install chromium
  4. Start the E2E tests: npm run test:e2e
  5. Confirm that all the tests pass, and a test report is available in tests/e2e/test-result
  6. Check that we can run the tests in debug mode and step through them in the browser: npm run test:e2e-dev

Additional details:

Currently the E2E workflow is set to run only on release branches or manually. I've triggered a manual run, and all the tests passed: https://github.com/woocommerce/google-listings-and-ads/actions/runs/5753482975/job/15596778163

Changelog entry

  • Dev - Convert E2E tests from Puppeteer to Playwright.

@mikkamp mikkamp self-assigned this Aug 3, 2023
@codecov
Copy link

codecov bot commented Aug 3, 2023

Codecov Report

Merging #2041 (e7891a3) into develop (3ec92b8) will increase coverage by 0.1%.
The diff coverage is n/a.

❗ Current head e7891a3 differs from pull request most recent head 3bb245b. Consider uploading reports for the commit 3bb245b to get more accurate results

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##             develop   #2041     +/-   ##
===========================================
+ Coverage       58.2%   58.3%   +0.1%     
+ Complexity      4119    4114      -5     
===========================================
  Files            454     454             
  Lines          17701   17676     -25     
===========================================
  Hits           10297   10297             
+ Misses          7404    7379     -25     
Flag Coverage Δ
php-unit-tests 58.3% <ø> (+0.1%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed Coverage Δ
src/ConnectionTest.php 0.0% <ø> (ø)

@mikkamp mikkamp marked this pull request as ready for review August 3, 2023 16:57
@mikkamp mikkamp requested a review from a team August 3, 2023 16:57
@mikkamp mikkamp changed the title Convert E2E tests from Puppeteer to PlayWright Convert E2E tests from Puppeteer to Playwright Aug 4, 2023
@@ -0,0 +1,55 @@
<?php
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm aware this is a test file but would be nice to document a bit the functions.

module.exports = defineConfig( {
testDir: '../specs',

/* Maximum time one test can run for. */
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💅

/* Maximum time in milliseconds one test can run for. */


expect: {
/**
* Maximum time expect() should wait for the condition to be met.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maximum time, in milliseconds, expect() should wait for the condition to be met.

Comment on lines 18 to 19
/* Fail the build on CI if you accidentally left test.only in the source code. */
forbidOnly: !! process.env.CI,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💅 I don't understand well this comment.

I will suggest the one appearing below on the web:
https://playwright.dev/docs/test-configuration#:~:text=Whether%20to%20exit%20with%20an%20error%20if%20any%20tests%20are%20marked%20as%20test.only.%20Useful%20on%20CI.

Whether to exit with an error if any tests are marked as "test.only"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is mostly copying and pasting from other examples. I think we can just remove this setting all together.


/* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */
use: {
/* Maximum time each action such as `click()` can take. Defaults to 0 (no limit). */
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/* Maximum time in milliseconds each action such as click() can take. Defaults to 0 (no limit). */

Comment on lines 21 to 24
console.log( 'Admin state file does not exist.' );
} else {
console.log( 'Admin state file could not be deleted: ' + err );
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe console.error might be more accurate here.

Comment on lines 65 to 68
console.log(
`Admin log-in failed, Retrying... ${ i }/${ adminRetries }`
);
console.log( e );
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can use console.warn. Also, I don't understand well why we need to retry 5 times?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are the parts that we need to do the same as WC core, but it isn't reusable to the same extent it was when we had the e2e packages. So it's basically copied code from here with the comment further down:

This step was failing occasionally, and globalsetup doesn't retry, so make it retry

I've had it fail in one of the test runs I did in GitHub Actions, so it must be a bit flakey at times (which is resolved with the retry).

Maybe I should add some references where the code is copied from to help explain. I can change the to console.warn / error where appropriate though.

@puntope
Copy link
Contributor

puntope commented Aug 7, 2023

One test failed for me in local

✘ 7 [chromium] › gtag-events/gtag-events.test.js:74:2 › GTag events › Add to cart event is sent from a block shop page (2.0m)

Screenshot 2023-08-07 at 20 05 27

@puntope
Copy link
Contributor

puntope commented Aug 7, 2023

Same with the rest of Add to cart events...

I will try to check tomorrow why.

@mikkamp
Copy link
Contributor Author

mikkamp commented Aug 8, 2023

One test failed for me in local

As mentioned I'd suggest to run the tests from PR #2043 as I did improve a few tests that were still a bit flakey. If they still fail there then I'd suggest to run them with npm run test:e2e-dev and then step through them to see where they fail. It should show the selectors it's matched while doing so.

@mikkamp
Copy link
Contributor Author

mikkamp commented Aug 8, 2023

@puntope I went ahead and fixed those suggestions you made. Regarding running the tests maybe you can try with the following commands to start with a clean slate (I just ran that locally and all the tests pass):

npm run wp-env destroy
npm run wp-env:up
npm run test:e2e

@mikkamp mikkamp requested a review from puntope August 8, 2023 12:04
@puntope
Copy link
Contributor

puntope commented Aug 8, 2023

Thanks for the changes @mikkamp Unfortunately it is still stuck in the waitForRequest call. (Even if I merged #2043

Screen.Recording.2023-08-08.at.18.35.54.mov

@puntope
Copy link
Contributor

puntope commented Aug 9, 2023

Some conflicts to be resolved in tests/e2e/docker/initialize.sh

But ready to go. The issue was on my side likely.

@mikkamp mikkamp merged commit 92d15c7 into develop Aug 9, 2023
12 checks passed
@mikkamp mikkamp deleted the e2e/playwright-tests branch August 9, 2023 15:51
@eason9487 eason9487 added the changelog: dev Developer-facing only change. label Aug 15, 2023
@eason9487 eason9487 mentioned this pull request Aug 22, 2023
17 tasks
@puntope puntope mentioned this pull request Aug 29, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
changelog: dev Developer-facing only change.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement new E2E environment once its ready in Woo platform
3 participants