Usage with gitlab.ci and multiple shard ?

[samixchoumi]

Hello, does anyone have a sample project that works well with parallelisation using gitlab-ci file (so multiple shard), a SLACK_WEBHOOK_URL and this plugin for slack reporting ?

I tried multiple solution by looking at the documentation but it's still doing some weird behavior like sending all gitlab runner test execution instead of waiting for the merge to be done.

Actually i've something like :

.merge_reports:
  stage: merge_reports
  image: mcr.microsoft.com/playwright:v1.43.0-jammy
  script:
    - echo "Merging test reports..."
    - mkdir -p all-blob-reports
    - cp -r blob-report/* all-blob-reports/
    - npx playwright merge-reports --reporter json ./all-blob-reports > merged_tests_results.json
  artifacts:
    paths:
      - all-blob-reports/
      - merged_tests_results.json
    expire_in: 1 week

.notify_slack:
  stage: notify
  dependencies:
    - merge_reports
  script:
    - echo "Sending Slack notification..."
    - npm install playwright-slack-report
    - npx playwright-slack-report --json-results="./merged_tests_results.json"

And my test steps just run the script : "test": "npx bddgen && npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL",

I've my playwright config looking like that :

reporter: [
    [
      "./node_modules/playwright-slack-report/dist/src/SlackReporter.js",
      {
        sendResults: "always",
        slackWebHookUrl: process.env.SLACK_WEBHOOK_URL
      },
    ],
    cucumberReporter('html', { outputFile: 'cucumber-report/report.html' }),
    cucumberReporter('json', { outputFile: 'cucumber-report/report.json' }),
    ['blob'],
  ],

Do you have any idea on what is wrong here ? Or at least a working sample project with Gitlab and parallelization so I could take a look ?

[samixchoumi]

More information that could be useful, I know that my slack will not work as i don't have any config file but I don't understand why I need it.
And i'm pretty sure that the problem with the slack reporting for each gitlab runner is linked to sendResults: "always", from the config.

[samixchoumi]

So, by updating sendResults: "always", to sendResults: "off",, I don't have all shard report send with Slack, it's a bit better.
I tried with a config looking like that :

{
  "sendResults": "always",
  "slackLogLevel": "error",
  "sendUsingWebhook": {
    "webhookUrl": "hookurl"
  },
  "maxNumberOfFailuresToShow": 0,
  "disableUnfurl": true
}

But it doesn't seem to work either... I got this error message :

    for (const m of originalMeta) {
                    ^
TypeError: originalMeta is not iterable
    at replaceEnvVars (/builds/[MASKED]/qat/poc-playwright/node_modules/playwright-slack-report/dist/cli.js:161:21)
    at Command.<anonymous> (/builds/[MASKED]/qat/poc-playwright/node_modules/playwright-slack-report/dist/cli.js:82:22)
Node.js v22.1.0

[samixchoumi]

Okey, I think I've found a solution, but with this parameter on : "maxNumberOfFailuresToShow": 0,, why do I still have failure shown in my Slack message ?
My solution right now :

{
  "sendResults": "always",
  "slackLogLevel": "error",
  "sendUsingWebhook": {
    "webhookUrl": "$SLACK_WEBHOOK_URL"
  },
  "meta": [
    { "key": "branch", "value" : "$CI_COMMIT_BRANCH"},
    { "key": "commit", "value" : "$CI_COMMIT_SHA"}
  ],
  "maxNumberOfFailuresToShow": 0,
  "disableUnfurl": true
}

[samixchoumi]

It's funny because it show that : :warning: There are too many failures to display - 5 out of 31 failures shown but I don't want any error showing on my Slack message.
Just showing :white_check_mark: 52 | :x: 31 | :large_yellow_circle: 0 | :fast_forward: 0 is good for me.