Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky e2e test #623

Open
MrSerth opened this issue Jun 27, 2024 · 3 comments
Open

Fix flaky e2e test #623

MrSerth opened this issue Jun 27, 2024 · 3 comments
Labels
bug Something isn't working

Comments

@MrSerth
Copy link
Member

MrSerth commented Jun 27, 2024

websocket_test.go:363: 
    	Error Trace:	/home/runner/work/poseidon/poseidon/tests/e2e/websocket_test.go:363
    	            				/home/runner/work/poseidon/poseidon/tests/e2e/websocket_test.go:170
    	            				/home/runner/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:115
    	Error:      	Not equal: 
    	            	expected: 1
    	            	actual  : 0
    	Test:       	TestE2ETestSuite/TestCommandMakeEnvironmentVariables/0
Unexpected EOF

=== RUN   TestE2ETestSuite/TestGetFileContent_Nomad/Permission_Denied
{"environment_id":"0","error":"unexpected EOF","level":"warning","msg":"Unexpected EOF for Execute","package":"nomad","runner_id":"0-8a48a4a7-6969-11ef-a609-000d3a600657","sentry-fingerprint":["nomad-unexpected-eof"],"time":"2024-09-02T20:25:50.041434Z"}
    runners_test.go:473: 
        	Error Trace:	/home/runner/work/poseidon/poseidon/tests/e2e/runners_test.go:473
        	            				/home/runner/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:115
        	Error:      	Not equal: 
        	            	expected: 424
        	            	actual  : 200
        	Test:       	TestE2ETestSuite/TestGetFileContent_Nomad/Permission_Denied
    runners_test.go:476: 
        	Error Trace:	/home/runner/work/poseidon/poseidon/tests/e2e/runners_test.go:476
        	            				/home/runner/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:115
        	Error:      	"" does not contain "file not found or insufficient permissions"
        	Test:       	TestE2ETestSuite/TestGetFileContent_Nomad/Permission_Denied

@MrSerth MrSerth added the bug Something isn't working label Jun 27, 2024
@mpass99 mpass99 self-assigned this Sep 27, 2024
@mpass99
Copy link
Contributor

mpass99 commented Sep 27, 2024

The second flaky test event can be reduced to the known upstream bug discussed in #589.
This occurs not only in production, but also in our e2e tests. Until the upstream issue is fixed, we can identify this cause by the line {"environment_id":"0","error":"unexpected EOF","level":"warning","msg":"Unexpected EOF for Execute","package":"nomad","runner_id":"0-8a48a4a7-6969-11ef-a609-000d3a600657","sentry-fingerprint":["nomad-unexpected-eof"],"time":"2024-09-02T20:25:50.041434Z"}.

We might even consider bumping nomad#12198, as the non-official Docker SDK produces this bug and has not fixed it since its disclosure two months ago.

@MrSerth
Copy link
Member Author

MrSerth commented Sep 28, 2024

Thanks for looking into the e2e test suite and pointing out the unexpected EOF bug we saw. Bumping nomad#12198 is a good idea and I've just added an upvote to the issue myself. Still, this probably only a long-term solution, since swapping the entire library is likely pretty complex for Nomad.

In the meantime, we could also try bumping go-dockerclient#1076, which would help for the e2e tests and the production use. Until then, I assume we need to live with these sporadic occurrences...

@MrSerth
Copy link
Member Author

MrSerth commented Oct 11, 2024

Soon, the second flaky e2e test should no longer occur: Nomad switched from from fsouza/go-dockerclient to the native docker client as I've detailed in #589 (comment).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants