Hotfix: Handle UNAVAILABLE rocoto status in Bash CI

[DavidHuber-NOAA]

Description

From time to time, PBS pro cannot return a qstat response within a given time limit set by rocoto (default is 45 seconds). If that happens, then an UNAVAILABLE status will be returned for the given job. This PR adds checking for this status to allow CI processing to continue.

Refs #2755 christopherwharrop/rocoto#110

Type of change

• New feature (adds functionality)

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO
• Does this change require an update to any of the following submodules? NO (If YES, please add a link to any PRs that are pending.)

How has this been tested?

Visual inspection

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings
• New and existing tests pass with my changes

[TerrenceMcGuinness-NOAA]

This all needs to be married with the scripts that use it in bash and the pipeline and tested before this goes to a PR.
In practice the CI system flagged the PR in those UNKNOWN states for too long as Stalled specifically because they could not advance. I'm not sure yet if this update would change that.

[TerrenceMcGuinness-NOAA]

On closer inspection, I noticed the idea of having to deal with UNAVAIBLE (and Rocoto only does this for PBS BTW) is a good one but it is not specifically the cause for the "fix" as it would have continued anyway onto checking for STALL which is still a logically valid check. Why it seems to solve the issue is because it simply adds more time for the system to self-repair in this state. So this extra step to keep checking is indeed helpful and still valid.

[DavidHuber-NOAA]

After discussion with @TerrenceMcGuinness-NOAA, I will also add a check for UNKNOWN statuses.

[TerrenceMcGuinness-NOAA]

@DavidHuber-NOAA what David means is he adding the "extra" wait in these cases of UNKNOWN and UNAVAIBLE to give the the system extra time to self repair with subsequent runs of rocotorun. UNAVAIBLE is logically equivalent UNKNOWN but more specific to PBS. The extra logic may also cover for corner cases when all states are UNAVAIBLE adding robust-ness to the checker.

[WalterKolczynski-NOAA]

@DavidHuber-NOAA what David means is he adding the "extra" wait in these cases of UNKNOWN and UNAVAIBLE to give the the system extra time to self repair with subsequent runs of rocotorun. UNAVAIBLE is logically equivalent UNKNOWN but more specific to PBS. The extra logic may also cover for corner cases when all states are UNAVAIBLE adding robust-ness to the checker.

I think they designate different things. UNKNOWN means the scheduler no longer has information on a job. UNAVAILABLE means the scheduler did not respond before the time-out.

[TerrenceMcGuinness-NOAA]

Nice update, tested it and the side effects are all valid and work within the framework