google-batch quota error does not trigger job failure

[rivershah]

Using the google-batch provider, I notice that some batch errors are not propagating to dsub and it continues waiting to run jobs, when when should be aborting

$ dstat --provider google-batch --project <PROJECT_ID> --location <REGION> --jobs '<JOB_ID>' --users '<USER>' --status '*' --format json
[
  {
    "job-name": "<JOB_NAME>",
    "task-id": "<TASK_ID>",
    "last-update": "2025-01-01 13:07:03.664000",
    "status-message": "VM in Managed Instance Group meets error: Batch Error: code - CODE_GCE_QUOTA_EXCEEDED, description - error count is 4, latest message example: Instance '<INSTANCE_ID>' creation failed: Quota 'GPUS_PER_GPU_FAMILY' exceeded.  Limit: 0.0 in region <REGION>."
  }
]

The process that launched it has retries=0, yet it still shows no failure and is patiently

Waiting for job to complete...
Monitoring for failed tasks to retry...
*** This dsub process must continue running to retry failed tasks.