chore: try disabling concurrency for browserstack tests #7258
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We're still getting sporadic test failures with BrowserStack. So per one discussion online, attempting to run without concurrency to see if it improves stability. I note that the video from tests where we "fail to capture" shows a browser with an error: "Safari Can't Connect to the Server" "Safari can't open the page "localhost:9879/?id=75185158" because Safari can't connect to the server "localhost". https://automate.browserstack.com/dashboard/v2/builds/419c59decb8e12303e45e1f4bad1b945252df532/sessions/19712158833ce57592b66a50433965e4146fc972
User Test ResultsTest specification and instructions User tests are not required Test Artifacts
|
|
Additional ref: Suggestions that multiple CI sessions overlapping may also cause the issue? |
|
I've just analyzed the 50 most recent builds in both LM and web test builds and there is weak evidence that build overlap may be contributing to the failures? Last column show build overlap. We might not see all overlaps fail, but there do seem to be more failures when there is overlap.
|
|
AFAICT, this has not made a difference to stability. |
Yeah, that's been my running hypothesis. Build overlap definitely wouldn't be doing us any favors for test stability. I hadn't found any concrete references for that running theory, though. |
I found two somewhat-related refs, neither conclusive, but both suggestive:
We're certainly not the only people experiencing instability with BrowserStack + Karma, going by that first ref above! |
|
Changes in this pull request will be available for download in Keyman version 16.0.61-alpha |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We're still getting sporadic test failures with BrowserStack. So per one discussion online, attempting to run without concurrency to see if it improves stability.
I note that the video from tests where we "fail to capture" shows a browser with an error: "Safari Can't Connect to the Server" "Safari can't open the page "localhost:9879/?id=75185158" because Safari can't connect to the server "localhost".
https://automate.browserstack.com/dashboard/v2/builds/419c59decb8e12303e45e1f4bad1b945252df532/sessions/19712158833ce57592b66a50433965e4146fc972
Browserstack log entries from the build in question:
@keymanapp-test-bot skip