Fix issue where a test incorrectly reported as "Passed on Retry" #252
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nprizal
changed the title
nprizal
changed the title
nprizal
force-pushed
the
tet-491-passed-on-retry-count-incorrect
branch
from
c566cc7 to
c8f1647
Compare
niceking
approved these changes
Jan 16, 2025
| func testIdentifier(testCase plan.TestCase) string { | ||
| return testCase.Scope + "/" + testCase.Name + "/" + testCase.Path |
There was a problem hiding this comment.
Is path information available for all supported frameworks?
There was a problem hiding this comment.
Yes, except for Cypress, as we don't currently parse/read Cypress result because Cypress can't output result into file out of the box. We already read test path for RSpec and Playwright, so no changes are required. As for Jest, I updated the runner to read the test path in this PR.
There was a problem hiding this comment.
Ok, mute and retry aren't even supported in Cypress so that is fine :)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
We found a bug where a test marked as "Passed on retry". This is happening when there are different tests with the same
scopeandname. Currently,bktecwill usescopeandnameto identify a test. If there are 2 tests with the samescopeandnameit will treat those tests as the same test with 2 execution counts, leading to incorrect reporting. The initial implementation was made to follow the server-side implementation so thatbkteccan match a local test with the list of muted tests received from the server.This PR resolves the issues by introducing 2 different ways of identifying a test. When reading the result from test runner,
bktecwill identify a test usingscope,nameandpathto make sure tests with the samenameandscopecounted as separate test. However, when checking if a test is muted, it identifies the test usingscopeandnameonly to follow the server-side implementation.This PR will resolve #236