How to deal with long running processes?

[sckott]

(pulling info from a slack thread convo with @dtenenba on 2024-12-20 so it doesn't get lost)

We have some "long" running processes that we'll have to deal with for tests. Long here may be 30 seconds or multiple hours, both of which need specific handling - a request to get metadata about job X after 2 seconds may error or return a different set of data than the same request after job X has completed.

Some options:

• use test fixtures - record a real http response and then use that cached http response insetad of making a real request to the cromwell server - this will be very fast
• use mocks - similar to fixtures, but instead of the whole http response saved to a file that we read (the fixture) we just specify the returned json in a variable
• forget about tests that depend on longer running operations? seems not ideal
• on a cron schedule submit all the wdls/jsons that have longer running operations - say once a day - when we run tests we just use the results already in the test user DB

---

If things can take a long time and we're okay with that (we don't use the above approaches of fixtures, mocks, etc.), then we need a different testing setup than is typical (where you do X and Y and then run the unit test right away). How would that work?

Some background info:

• Self-hosted runners can have jobs that take up to 5 days
• You can trigger actions by calling the github api with a token. If you have a workflow that has on: workflow_dispatch: then you can do e.g.,

curl -X POST \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer YOUR_GITHUB_TOKEN" \
  https://api.github.com/repos/OWNER/REPO/actions/workflows/WORKFLOW_ID/dispatches \
  -d '{"ref":"main"}'

It's not clear yet what the solution is, but likely involves some system where one workflow (.yml file) submits workflows, then another runs tests that depend on those workflows (presumably all in final state on cromwell) - how the 2nd system knows when to run is up for debate; possibly some script polling for all workflows being done

[sckott]

Once we have a solution here, we'll want to add additional tests that deal with final states, e.g., right now the cromwell api tests are dealing with initial state after workflow submission, which of course only covers initial state conditions.

[sckott]

@dtenenba @tefirman @seankross Thoughts on this?

[seankross]

@sckott, based on our discussion, let me know if you agree with this (others are encouraged to weigh in too of course):

For testing the PROOF API, we'll have the following kinds of tests:

• Short jobs that submitted on PR, push, or any other circumstance that is appropriate.
• Fixtures that are updated frequently via cron.

Also, tests should only evaluate jobs when they have just been submitted, or they are completely finished running (at either the start or end of their lifecycle).

[sckott]

Yep, agree @seankross

[sckott]

Work started locally on my machine on branch vcr incorporating vcrpy and some other pkgs to cache http requests/responses. This will stay local only until we've worked out how to minimize any exposed sensitive strings in test fixtures (aka: vcr cassettes).

vcr notes and todos:

• vcr was branched off of api-tests-adding-more
• the change of dir cromwell-api to cromwellapi originally occurred on vcr branche has been cherry picked onto api-tests-adding-more
• make sure that anyone working on vcr cassettes locally is using the token for the test user in 1password
• maybe change vcr matchers to ignore the hostname that can change (e.g., gizmok99.fhcrc.org:44444) even for the same user