flaky tests

[xmunoz]

Thanks for creating an issue! Please fill out this form so we can be sure to have all the information we need, and to minimize back and forth.

• What are we trying to do?
  Eliminate test flakiness

• What feature or behavior is this required for?
  Quickly and easily reviewing PRs, and merging with confidence

• How could we solve this issue? (Not knowing is okay!)
  I don't know

• Anything else?

Here are the flaky tests:

• UpdatePatientInfoTest::changing fulfillment information#test_0001_should alter the information (OptionalRetry set up)
• LoggingCallsTest::logging Left voicemail#test_0002_should be visible on the call log after clicking Left voicemail (OptionalRetry set up)
• LoggingCallsTest::logging Left voicemail#test_0001_should close the modal when clicking Left voicemail (OptionalRetry set up)
• LoggingCallsTest::logging reached patient#test_0002_should be viewable on the call log (OptionalRetry set up)
• LoggingCallsTest::logging reached patient#test_0001_should redirect to the edit view when a patient has been reached: (OptionalRetry set up)
• LoggingCallsTest::verifying modal behavior and content#test_0001_should open a modal when clicking the call glyphicon OptionalRetry set up)

These are actually errors, not failures. Example: https://github.com/DARIAEngineering/dcaf_case_management/actions/runs/6327925858/job/17184902734

Sometimes we also get this error:

• UpdatePatientInfoTest::changing patient information#test_0001_should alter the patient information information (OptionalRetry set up)
  Example: https://github.com/DARIAEngineering/dcaf_case_management/actions/runs/6287048102/job/17071253257

There will either be 6 errors (above) or 1 error (below) for a given test run. I've never seen a case where there are 7 errors.

[colinxfleming]

I was hoping this was going to be low hanging fruit that we could slap extend Minitest::OptionalRetry (which will retry a failing test up to three times before declaring defeat) but this sounds like something's up with those, LoggingCallsTest in particular.

[xmunoz]

I was going to say that we haven't see this in a while, but looking at recent test runs, here is a recent example of a flaky failure (second failure mode): https://github.com/DARIAEngineering/dcaf_case_management/actions/runs/6785077889/job/18442620959