Retry mechanism for API limit failures

[drpatelh]

Getting an error when running the Platform CLI via one of our NF pipelines (nf-aggregate). It would be great if there was a way to catch the API limit error and retry automatically.

Error executing process > 'NF_AGGREGATE:SEQERA_RUNS_DUMP (5O6m7WW5ZSeH9X)'

Caused by:
  Essential container in task exited

Command executed:

  tw \
       \
      --url=https://api.tower.nf/ \
      --access-token=$TOWER_ACCESS_TOKEN \
      runs \
      dump \
      -id=5O6m7WW5ZSeH9X \
      --workspace=scidev/testing \
      --output="5O6m7WW5ZSeH9X.tar.gz" \
      --add-task-logs --add-fusion-logs
  
  mkdir -p 5O6m7WW5ZSeH9X
  tar \
      -xvf \
      5O6m7WW5ZSeH9X.tar.gz \
      -C 5O6m7WW5ZSeH9X
  
  cat <<-END_VERSIONS > versions.yml
  "NF_AGGREGATE:SEQERA_RUNS_DUMP":
      seqera-cli: $(echo $(tw --version 2>&1) | sed 's/^.*Tower CLI version //; s/ *$//')
  END_VERSIONS

Command exit status:
  1

Command output:
  - Tower info
  - Workflow details

Command error:
   ERROR: Maximum request rate exceeded. Wait 0secs before issuing a new request
  - Tower info
  - Workflow details
  8:49AM INF shutdown filesystem start
  8:49AM INF shutdown filesystem done