ci(editor): Run e2e tests in parallel and improve build caching (#5445)

* WIP: Cypress parallel CI run test

* Trigger action on branch push

* Change build artifacts path

* Make sure to checkout the repo for testing job

* Use Cypress action for installing

* Lock cypress action userd version

* Skip node install step since we're using cypress node16 container

* Let Cypress handle pnpm install

* Use setup-node action for caching pnpm

* Set CYPRESS_CACHE_FOLDER

* Set CYPRESS_CACHE_FOLDER

* Manually cache pnpm store

* Dont fix pnpm version

* Use caching action also in testing job

* Zip packages dist before uploading the artifacts and change caching key

* Use absolute build paths for zipping job

* Use zip command in action

* Use tar for zipping packages

* Debuggin directory ls

* Debugging caching of modules

* Attempt to fix permissions issue

* Porivde Cypress executable via `CYPRESS_RUN_BINARY`

* Cache /github/home

* Adjust caching keys

* Debug: search for cypress exec

* Debugging: List dirs

* Use pnpm install action to install node_modules

* Do not log /home/runner

* Use node_modules/.bin Cypress binary

* Use absolute path to nodue modules

* Run Cypress via custom command

* Try with patched cypress action

* Revert logging

* Manually specify cypress config file

* Use absolute paths

* Fix cypress config name

* Debug print cypress config

* Remove debugging, increase to 4 containers

* Increase amount of containers

* Add env-version matrix

* Replace node14 with node18 in testing matrix

* Remove debugging and add node 14

* Use just node14

* Use cypress:base and remove browser req

* Give more general timeouts

* Try with node16

* Change cache directive position

* Replace zip artifact upload with cache

* Cache full packages not just dist

* Test with variable inputs

* Add commit info message

* Remove wrongly commited code

* Allow WF API dispatch

* Try Chrome browser again for comparison

* Include Monaco in the build

* Make e2e workflow re-usable

* Comment out invalid reusable workflow args

* Use electron and add node 14 run

* Fix env arg

* Provide custom ci-build-id

* Refactor remaining e2e workflow to use reusable action

* Remove single matrix directive

* Refactor ci-pull-req

* Make lint job dependant on test jobs

* Disable debugging job

* Make containers dynamic

* Cleanup & install git for linting action

* Use regular buntu image for PR linting

* Debugging failing tests

* Remove fixed spec name

* Debug e2e env var

* Do not use realkeypress which crashes electron runner

* Debugging

* chore: remove console

* chore: remove console

* test: remove node 14 tests

* test: replace test branch with master

* test: use tests in current branch

* test: use relative path

* chore: clean up

* test: only trigger on approval

* ci: update test PR

* ci: use curr branch

* ci: only run 14 on schedule, not for slack command

* ci: only run test on approval

* ci: clean up branch, rename step

* ci: rename steps

* ci: clean up cancel

* ci: clean up env var

* ci: set var

* ci: use chromef

* ci: use electron

* chore: add console log

* chore: add console log

* ci: update to string

* ci: set all env options

* test: build

* ci: fix step issue

* Fix failing tests & upgrade to Cypress 12

* Allow WF dispatch of e2e reusable

* Fix wrong naming in e2e-tests workflow

* Redeploy

* Fix tests

* Fix NDV tests and remove skipping of webhooks execution tests

* Fix clipboard read command

* Fix execution failing tests

* Reset before each 15 and 3

* Fix flaky tests

* Cleanup and log envs

* Test fixes

* Default owner spec fixes

* Get rid of CYPRESS_RUN_ENV

* Increase amount of containers, cleanup and add mock for credentials test call

* Cleanup & fix PR tests unit tests

* Wait for WF to loade in sharing spec

* Do linting and unit tests first

* Use frozen lockfile

* Revert back ci pull request jobs order

* Refine credential input selector and move cy.waitForLoad to correct position in 15-scheduler spec

* test: build

* Wait for WF execution instead of arbitraty timeout in WF execution spec, change order of jobs for ci pull request

* Fix flaky 3-default owner spec and wait for execution list to load in 20-workflow-executions

* Use setup node action

* Remove caching for lint/unit tests

* Experiment with parallel test & lint on ci

* Provide cache key dynamically

* Run e2e in parallel on pr

* Only run node14 e2e on daily schedule

* Make sure to generate generate new ci-build-id on re-runs

* Remove debugging prints

* Address PR comments

* Rename custom onBeforeUnload handler

* Make sure 19-execution spec waits for wf to load properly before import fixtures

---------

Co-authored-by: Mutasem <[email protected]>

.github/workflows/ci-pull-requests.yml

@@ -1,41 +1,99 @@
-name: Test Pull Requests
+name: Build, unit/smoke test and lint branch
 
 on: [pull_request]
 
 jobs:
-  build:
+  install:
+    name: Install & Build
     runs-on: ubuntu-latest
-
-    timeout-minutes: 30
-
-    strategy:
-      matrix:
-        node-version: [14.x, 16.x]
-
     steps:
       - uses: actions/checkout@v3
+        with:
+          repository: n8n-io/n8n
+          ref: ${{ inputs.branch }}
 
       - uses: pnpm/[email protected]
 
-      - name: Use Node.js ${{ matrix.node-version }}
+      - name: Use Node.js 16
         uses: actions/setup-node@v3
         with:
-          node-version: ${{ matrix.node-version }}
-          cache: 'pnpm'
+          node-version: 16.x
+          cache: pnpm
 
       - name: Install dependencies
         run: pnpm install --frozen-lockfile
 
       - name: Build
         run: pnpm build
 
+      - name: Cache build artifacts
+        uses: actions/cache@v3
+        with:
+          path: |
+            /github/home/.cache
+            /github/home/.pnpm-store
+            ./node_modules
+            ./packages
+          key: ${{ github.sha }}-base:16.18.1-test-lint
+
+  unit-test:
+    name: Unit tests
+    runs-on: ubuntu-latest
+    needs: install
+    steps:
+      - uses: actions/checkout@v3
+        with:
+          repository: n8n-io/n8n
+          ref: ${{ inputs.branch }}
+
+      - name: Restore cached build artifacts
+        uses: actions/cache@v3
+        with:
+          path: |
+            /github/home/.cache
+            /github/home/.pnpm-store
+            ./node_modules
+            ./packages
+          key: ${{ github.sha }}-base:16.18.1-test-lint
+
+      - uses: pnpm/[email protected]
+
+      - name: Use Node.js 16
+        uses: actions/setup-node@v3
+        with:
+          node-version: 16.x
+          cache: pnpm
+
       - name: Test
         run: pnpm test
 
-      - name: Test E2E
-        run: |
-          pnpm cypress:install
-          pnpm test:e2e:smoke
+  lint:
+    name: Lint changes
+    runs-on: ubuntu-latest
+    needs: install
+    steps:
+      - uses: actions/checkout@v3
+        with:
+          repository: n8n-io/n8n
+          ref: ${{ inputs.branch }}
+
+      - name: Restore cached build artifacts
+        uses: actions/cache@v3
+        with:
+          path: |
+            /github/home/.cache
+            /github/home/.pnpm-store
+            ./node_modules
+            ./packages
+          key: ${{ github.sha }}-base:16.18.1-test-lint
+
+      - uses: pnpm/[email protected]
+
+      - name: Use Node.js 16
+        uses: actions/setup-node@v3
+        with:
+          node-version: 16.x
+          cache: pnpm
 
       - name: Fetch base branch for `git diff`
         run: git fetch origin ${{ github.event.pull_request.base.ref }}:${{ github.event.pull_request.base.ref }}
@@ -44,3 +102,18 @@ jobs:
         env:
           ESLINT_PLUGIN_DIFF_COMMIT: ${{ github.event.pull_request.base.ref }}
         run: pnpm lint
+
+  smoke-test:
+    name: E2E [Electron/Node 16]
+    uses: ./.github/workflows/e2e-reusable.yml
+    if: ${{ !contains(github.event.pull_request.labels.*.name, 'skip-e2e') }}
+    with:
+      branch: ${{ github.event.pull_request.base.ref }}
+      user: ${{ github.event.inputs.user || 'PR User' }}
+      spec: ${{ github.event.inputs.spec || 'e2e/0-smoke.cy.ts' }}
+      run-env: base:16.18.1
+      record: false
+      parallel: false
+      containers: '[1]'
+    secrets:
+      CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}

.github/workflows/e2e-reusable.yml

@@ -0,0 +1,150 @@
+name: Reusable e2e workflow
+
+on:
+  workflow_call:
+    inputs:
+      branch:
+        description: 'GitHub branch to test.'
+        required: false
+        type: string
+        default: 'master'
+      user:
+        description: 'User who kicked this off.'
+        required: false
+        type: string
+        default: 'schedule'
+      spec:
+        description: 'Specify specs.'
+        required: false
+        default: 'e2e/*'
+        type: string
+      run-env:
+        description: 'Node env version to run tests with.'
+        required: false
+        default: 'browsers:node16.18.0-chrome90-ff88'
+        type: string
+      cache-key:
+        description: 'Cache key for modules and build artifacts.'
+        required: false
+        default: ${{ github.sha }}-${{ inputs.run-env }}-e2e-modules
+        type: string
+      record:
+        description: 'Record test run.'
+        required: false
+        default: true
+        type: boolean
+      parallel:
+        description: 'Run tests in parallel.'
+        required: false
+        default: true
+        type: boolean
+      containers:
+        description: 'Number of containers to run tests in.'
+        required: false
+        default: '[1, 2, 3, 4, 5, 6, 7, 8]'
+        type: string
+    secrets:
+      CYPRESS_RECORD_KEY:
+        description: 'Cypress record key.'
+        required: true
+
+jobs:
+  # single job that generates and outputs a common id
+  prepare:
+    runs-on: ubuntu-latest
+    outputs:
+      uuid: ${{ steps.uuid.outputs.value }}
+    steps:
+      - name: Generate unique ID 💎
+        id: uuid
+        # take the current commit + timestamp together
+        # the typical value would be something like
+        # "sha-5d3fe...35d3-time-1620841214"
+        run: echo "value=sha-$GITHUB_SHA-time-$(date +"%s")" >> $GITHUB_OUTPUT
+
+  install:
+    runs-on: ubuntu-latest
+    needs: ['prepare']
+    container:
+      image: cypress/${{ inputs.run-env }}
+      options: --user 1001
+    steps:
+      - uses: actions/checkout@v3
+        with:
+          repository: n8n-io/n8n
+          ref: ${{ inputs.branch }}
+
+      - name: Setup pnpm
+        uses: pnpm/[email protected]
+        with:
+          run_install: true
+
+      - name: Cache pnpm modules
+        uses: actions/cache@v3
+        with:
+          path: |
+            /github/home/.cache
+            /github/home/.pnpm-store
+            ./node_modules
+            ./packages
+          key: ${{ inputs.cache-key }}
+
+      - name: Cypress build
+        uses: cypress-io/github-action@v5
+        with:
+          # Disable running of tests within install job
+          runTests: false
+          install: false
+          build: pnpm build
+
+      - name: Cypress install
+        run: pnpm cypress:install
+
+  testing:
+    runs-on: ubuntu-latest
+    container:
+      image: cypress/${{ inputs.run-env }}
+      options: --user 1001
+    needs: ['prepare', 'install']
+    strategy:
+      fail-fast: false
+      matrix:
+        containers: ${{ fromJSON(inputs.containers) }}
+    steps:
+      - uses: actions/checkout@v3
+        with:
+          repository: n8n-io/n8n
+          ref: ${{ inputs.branch }}
+
+      - name: Setup pnpm
+        uses: pnpm/[email protected]
+
+      - name: Restore cached pnpm modules
+        uses: actions/cache@v3
+        with:
+          path: |
+            /github/home/.cache
+            /github/home/.pnpm-store
+            ./node_modules
+            ./packages
+          key: ${{ inputs.cache-key }}
+
+      - name: Cypress run
+        uses: cypress-io/github-action@v5
+        with:
+          install: false
+          start: pnpm start
+          wait-on: 'http://localhost:5678'
+          wait-on-timeout: 120 #
+          record: ${{ inputs.record }}
+          parallel: ${{ inputs.parallel }}
+          # We have to provide custom ci-build-id key to make sure that this workflow could be run multiple times
+          # in the same parent workflow
+          ci-build-id: ${{ needs.prepare.outputs.uuid }}
+          spec: "/__w/n8n/n8n/cypress/${{ inputs.spec }}"
+          config-file: /__w/n8n/n8n/cypress.config.js
+        env:
+          CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          E2E_TESTS: true
+          COMMIT_INFO_MESSAGE: 🌳 ${{ inputs.branch }} 🖥️ ${{ inputs.run-env }} 🤖 ${{ inputs.user }} 🗃️ ${{ inputs.spec }}

.github/workflows/e2e-tests-pr.yml

@@ -7,53 +7,28 @@ on:
       - 'master'
 
 jobs:
-  runTests:
-    name: 'Run E2E tests'
-    runs-on: ubuntu-latest
-    if: ${{ !contains(github.event.pull_request.labels.*.name, 'skip-e2e') }}
-
-    timeout-minutes: 30
-
-    strategy:
-      matrix:
-        node-version: [16.x]
-
-    steps:
-      - name: 'Cancel previous runs'
-        uses: styfle/[email protected]
-        with:
-          access_token: ${{ github.token }}
-
-      - name: 'Cancel if PR is not approved'
-        if: github.event.review.state != 'approved'
-        uses: andymckay/[email protected]
-
-      - uses: actions/checkout@v3
-
-      - uses: pnpm/[email protected]
-
-      - name: Use Node.js ${{ matrix.node-version }}
-        uses: actions/setup-node@v3
-        with:
-          node-version: ${{ matrix.node-version }}
-          cache: 'pnpm'
-
-      - name: Install dependencies
-        run: pnpm install --frozen-lockfile
-
-      - name: Build
-        run: pnpm build
-
-      - name: Test E2E
-        run: |
-          pnpm cypress:install
-          pnpm test:e2e:all
-
-      - name: Upload artifacts
-        uses: actions/upload-artifact@v3
-        if: failure()
-        with:
-          name: cypress-screenshots
-          path: |
-            cypress/screenshots
-          retention-days: 1
+  # We disable this for now because cancelling runs makes the Cypress Cloud tests to hang.
+  # cancel-previous-runs:
+  #   runs-on: ubuntu-latest
+  #   name: 'Cancel previous e2e test runs'
+  #   strategy:
+  #     matrix:
+  #       node-version: [16.x]
+
+  #   steps:
+  #     - name: 'Cancel previous runs'
+  #       uses: styfle/[email protected]
+  #       with:
+  #         access_token: ${{ github.token }}
+
+  run-e2e-tests:
+    name: E2E [Electron/Node 16]
+    uses: ./.github/workflows/e2e-reusable.yml
+    if: ${{ github.event.review.state == 'approved' && !contains(github.event.pull_request.labels.*.name, 'skip-e2e') }}
+    with:
+      branch: ${{ github.event.pull_request.head.ref }}
+      user: ${{ github.event.pull_request.user.login || 'PR User' }}
+      spec: 'e2e/*'
+      run-env: base:16.18.1
+    secrets:
+      CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}