-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
'failures' in mochawesome-report is not accurate. #29627
Comments
The Node.js and Ubuntu versions you are using are no longer supported. The version of Cypress is also old. You need to be using a minimum Node.js version You can also ask for help from the Cypress technical community on Discord |
Closing. We can reopen if an example is provided in our most recent version of Cypress demonstrating the issue. |
@MikeMcC399 I switched to testing on the required system, and I'm sure the problem still exists. |
I followed the documentation in https://docs.cypress.io/guides/tooling/reporters#Cypress-configuration-5 using:
and I was able to reproduce your issue. It seems if the order of the tests is changed, then the reported results are correct. So it's a bug, but you may be able to work around it. spec.cy.js describe('test', () => {
it('test3', () => {
throw new Error('This test case should fail');
})
it('test1', () => { })
it.skip('test2', () => { })
}) produces {
"stats": {
"suites": 1,
"tests": 3,
"passes": 1,
"pending": 1,
"failures": 1,
"start": "2024-06-14T11:19:52.429Z",
"end": "2024-06-14T11:19:52.829Z",
"duration": 400,
"testsRegistered": 3,
"passPercent": 50,
"pendingPercent": 33.33333333333333,
"other": 0,
"hasOther": false,
"skipped": 0,
"hasSkipped": false
},
"results": [
{
"uuid": "dc102c24-ebed-4c0d-a3d4-cb86a7dfb7f4",
"title": "",
"fullFile": "cypress\\e2e\\spec.cy.js",
"file": "cypress\\e2e\\spec.cy.js",
"beforeHooks": [],
"afterHooks": [],
"tests": [],
"suites": [
{
"uuid": "55d412a9-6995-412d-8e4d-f8c977172bde",
"title": "test",
"fullFile": "",
"file": "",
"beforeHooks": [],
"afterHooks": [],
"tests": [
{
"title": "test3",
"fullTitle": "test test3",
"timedOut": null,
"duration": 302,
"state": "failed",
"speed": null,
"pass": false,
"fail": true,
"pending": false,
"context": null,
"code": "throw new Error('This test case should fail');",
"err": {
"message": "Error: This test case should fail",
"estack": "Error: This test case should fail\n at Context.eval (webpack://cy-mochawesome/./cypress/e2e/spec.cy.js:3:10)",
"diff": null
},
"uuid": "32b09751-35ad-442f-85a1-0e067af5d120",
"parentUUID": "55d412a9-6995-412d-8e4d-f8c977172bde",
"isHook": false,
"skipped": false
},
{
"title": "test1",
"fullTitle": "test test1",
"timedOut": null,
"duration": 11,
"state": "passed",
"speed": "fast",
"pass": true,
"fail": false,
"pending": false,
"context": null,
"code": "",
"err": {},
"uuid": "9e195bf3-a980-4ecd-ad46-2aebd63698b1",
"parentUUID": "55d412a9-6995-412d-8e4d-f8c977172bde",
"isHook": false,
"skipped": false
},
{
"title": "test2",
"fullTitle": "test test2",
"timedOut": null,
"duration": 0,
"state": "pending",
"speed": null,
"pass": false,
"fail": false,
"pending": true,
"context": null,
"code": "",
"err": {},
"uuid": "1a6eae37-463f-4d35-88a4-6146e6b938cb",
"parentUUID": "55d412a9-6995-412d-8e4d-f8c977172bde",
"isHook": false,
"skipped": false
}
],
"suites": [],
"passes": [
"9e195bf3-a980-4ecd-ad46-2aebd63698b1"
],
"failures": [
"32b09751-35ad-442f-85a1-0e067af5d120"
],
"pending": [
"1a6eae37-463f-4d35-88a4-6146e6b938cb"
],
"skipped": [],
"duration": 313,
"root": false,
"rootEmpty": false,
"_timeout": 2000
}
],
"passes": [],
"failures": [],
"pending": [],
"skipped": [],
"duration": 0,
"root": true,
"rootEmpty": true,
"_timeout": 2000
}
],
"meta": {
"mocha": {
"version": "7.0.1"
},
"mochawesome": {
"options": {
"quiet": false,
"reportFilename": "mochawesome",
"saveHtml": false,
"saveJson": true,
"consoleReporter": "spec",
"useInlineDiffs": false,
"code": true
},
"version": "7.1.3"
},
"marge": {
"options": {
"reportDir": "cypress/results",
"overwrite": false,
"html": false,
"json": true
},
"version": "6.2.0"
}
}
} |
@MikeMcC399 Yes, at present, I can avoid this bug by putting the test marked skip at the end. But you don't need to pay attention to the order, it is definitely better. |
Current behavior
I have three cypress test cases,
one executed successfully,
one marked as skip,
and the other constructed an exception.
The following is the result of cypress.
But in the mochawesome-report, failures is 0.
Desired behavior
No response
Test code to reproduce
test.cy.js
Cypress Version
13.11.0
Node version
v18.20.3
Operating System
Ubuntu 20.04
Debug Logs
No response
Other
No response
The text was updated successfully, but these errors were encountered: