Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Fix flaky espionage dispatch mission test #401

Closed
lanedirt opened this issue Oct 18, 2024 · 4 comments · Fixed by #415
Closed

[BUG] Fix flaky espionage dispatch mission test #401

lanedirt opened this issue Oct 18, 2024 · 4 comments · Fixed by #415
Labels
bug Something isn't working

Comments

@lanedirt
Copy link
Owner

lanedirt commented Oct 18, 2024

Describe the bug
This test sometimes fails when running jobs:

  FAILED  Tests\Feature\FleetDispatch\FleetDispatchEspionageTest > dispatch…   
  Target planet was not updated after espionage mission has arrived. Check target planet update logic on mission arrival.
Failed asserting that 1729147297 is less than 1729147297.

  at tests/Feature/FleetDispatch/FleetDispatchEspionageTest.php:191

For example, see this GitHub Action run:
https://github.com/lanedirt/OGameX/actions/runs/11373489520/job/31640153417

To Reproduce
Steps to reproduce the behavior:

  1. See GitHub action result: https://github.com/lanedirt/OGameX/actions/runs/11373489520/job/31640153417
  2. Most likely if you run the tests locally many times it will also sometimes fail. However I have not tested or confirmed this yet. Verify this by executing php artisan test in the ogame-app docker container.

Expected behavior
Test to always succeed, especially if nothing regarding espionage missions has been changed. The problem in this case can be caused by underlying game logic that is actually faulty, or it can be caused by something else that makes it flaky (i.e. fail sometimes while it shouldn't).

@lanedirt lanedirt added the bug Something isn't working label Oct 18, 2024
@rautamik
Copy link
Contributor

This issue can be reproduced locally (not tested in Docker), by running the fleet dispatch espionage tests in loop:
while php vendor/bin/phpunit --filter FleetDispatchEspionageTest --stop-on-failure; do echo; done

1) Tests\Feature\FleetDispatch\FleetDispatchEspionageTest::testDispatchFleetDebrisEspionageReport
Target planet was not updated after espionage mission has arrived. Check target planet update logic on mission arrival.
Failed asserting that 1729569638 is less than 1729569638.

tests/Feature/FleetDispatch/FleetDispatchEspionageTest.php:191

There is also another flaky case with the same testDispatchFleetDebrisEspionageReport that pops up when running only that test in the loop:
while php vendor/bin/phpunit --filter testDispatchFleetDebrisEspionageReport --stop-on-failure; do echo; done

1) Tests\Feature\FleetDispatch\FleetDispatchEspionageTest::testDispatchFleetDebrisEspionageReport
Debris field metal resources are not visible in espionage report.
Failed asserting that 2674 matches expected 1337.

tests/Feature/FleetDispatch/FleetDispatchEspionageTest.php:198

@lanedirt
Copy link
Owner Author

Hi @rautamik,

Thank you for testing this and for sharing the command to reproduce the error at will, that is very helpful! 👍

@lanedirt
Copy link
Owner Author

lanedirt commented Nov 1, 2024

Re-opening this issue: the flaky tests still occur in new PR's.

I'm working on refactoring how the tests interact with time as certain tests are setting time to static values (2024-01-01) and other tests do not and therefore use the current time which causes conflicts. If all tests use the same starting point this problem should go away once and for all.

@lanedirt
Copy link
Owner Author

lanedirt commented Dec 2, 2024

Fixed by #433

@lanedirt lanedirt closed this as completed Dec 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants