HTCondor tutorial: stalled launch of htcondor-pool-ap-mig

[jd-foster]

I hope this is the last question (and really appreciate your help so far).

The error is shown below, where it appears that htcondor-pool-ap-mig is expected to be available but isn't: when I inspect the Instance Groups console I can see it listed but the status is listed as Updating, with the hover tip reading "Instance group is resizing: 0 instances, scaling to 1" but the process has completely stalled (for 2.5 hours and counting). However, htcondor-pool-cm-mig is successfully created and running.

Are there any tips for diagnosing what has gone wrong and how to remedy this? Thanks again.

...
module.htcondor_access.module.htcondor_ap.google_compute_region_instance_group_manager.mig: Creation complete after 1m17s [id=projects/<PROJECT_ID>/regions/australia-southeast1/instanceGroupManagers/htcondor-pool-ap-mig]
module.htcondor_access.time_sleep.mig_warmup: Creating...
module.htcondor_access.time_sleep.mig_warmup: Still creating... [10s elapsed]
...
module.htcondor_access.time_sleep.mig_warmup: Creation complete after 2m0s [id=2023-08-29T01:39:53Z]
module.htcondor_access.data.google_compute_region_instance_group.ap: Reading...
module.htcondor_access.data.google_compute_region_instance_group.ap: Read complete after 1s [id=projects/<PROJECT_ID>/regions/australia-southeast1/instanceGroups/htcondor-pool-ap-mig]

Error: Resource postcondition failed

  on modules/embedded/community/modules/scheduler/htcondor-access-point/main.tf line 121, in data "google_compute_region_instance_group" "ap":
 121:       condition     = length(self.instances) == local.host_count
    ├────────────────
    │ local.host_count is 1
    │ self.instances is empty list of object

There should be 1 access points found
Error: exit status 1

[tpdownes]

@jd-foster the most likely answer is that you have consumed relatively limited quota for C2 CPUs in your region. A reasonable option would be to add the setting machine_type: n2-standard-4 to the central manager and access point modules. An advantage of C2 VM family is that it exposes the hardware in a more performant manner than the N2s, but that does not matter much for these bits of "infrastructure." They are, unfortunately, limited in availability.

Quota increases and be made through the IAM -> Quotas tab in the Cloud Console:

https://cloud.google.com/compute/resource-usage

You can confirm this theory by:

• browsing to https://console.cloud.google.com/logs
• streaming the logs (see option on right)
• searching for "QUOTA_EXCEEDED" in the text box or otherwise filtering for errors (they will appear as a red circle with exclamation points)

If you would like to use the C2 (or C2D, C3, H3) VM families, I suggest coordinating a quota increase requests with your FSR. They can guide you to the best region (of the 2) and help you assess your short-term/long-term needs.

[jd-foster]

Thanks for the response. I wasn't able find anything searching for quotas. However, the logs suggest this error might be the cause:

Constraint constraints/compute.vmExternalIpAccess violated for project <NNNN>. Add instance projects/<PROJECT_ID>/zones/australia-southeast1-c/instances/htcondor-pool-ap-hgx4 to the constraint to use external IP with it.

Does this seem likely or consistent?

[jd-foster]

Ok, I managed to resolve this by modifying community/examples/htc-htcondor.yaml as:

vars:
  project_id:  ## Set GCP Project ID Here ##
  deployment_name: htcondor-pool
  region: us-central1
  zone: us-central1-c
  disk_size_gb: 100
  new_image_family: htcondor-10x
  enable_shielded_vm: true
  enable_public_ips: false    ## <----- Added
...

settings:
      default_mig_id: $(htcondor_execute_point.mig_id)
      enable_public_ips: $(vars.enable_public_ips)    ## <----- Modified

[tpdownes]

Yes, this is it (I believe you're modifying the access-point module). The better way of doing that is to simply remove the line enable_public_ips: true entirely from the access-point module as the value defaults to false.

Additionally, anything that it listed in vars will automatically be applied to settings with a matching name, so the line as you've written it is explicit, but duplicative.

The access-point is the only HTCondor module where we have exposed the ability to have public IPs, on the theory that you may have users who need to SSH without access to gcloud. The best command to SSH without a public IP is:

gcloud compute ssh --zone "your-zone" "ap-hostname" --project "your-project --tunnel-through-iap