🤔 AgentPool not registering

[grabskimm]

Operator Version, Kind and Kubernetes Version

• Operator version: 2.0.0-beta3
• Kind: AgentPool
• Kubernetes version: v1.25.12 +rke2r1

YAML Manifest File

apiVersion: app.terraform.io/v1alpha2
kind: AgentPool
metadata:
  name: agent-pool-dev
spec:
  organization: TERRAFORM_CLOUD_ORG_NAME
  token:
    secretKeyRef:
      name: terraformrc
      key: teamtoken
  name: agent-pool-dev
  agentTokens:
    - name: agent-pool-1
    - name: agent-pool-2
    - name: agent-pool-3
  agentDeployment:
    replicas: 3
    spec:
      containers:
        - name: tfc-agent
          image: "hashicorp/tfc-agent:latest"

Output Log

2023-09-08T23:47:23Z    INFO    Agent Pool Controller   {"agentpool": "terraform/agent-pool-dev", "msg": "new reconciliation event"}
2023-09-08T23:47:23Z    INFO    Reconcile Agent Pool    {"agentpool": "terraform/agent-pool-dev", "msg": "reconciling agent pool"}
2023-09-08T23:47:23Z    INFO    Reconcile Agent Tokens  {"agentpool": "terraform/agent-pool-dev", "msg": "new reconciliation event"}
2023-09-08T23:47:23Z    INFO    Reconcile Agent Tokens  {"agentpool": "terraform/agent-pool-dev", "msg": "Kubernetes Secret \"agent-pool-dev-agent-pool\" exists"}
2023-09-08T23:47:23Z    INFO    Reconcile Agent Tokens  {"agentpool": "terraform/agent-pool-dev", "msg": "successfully reconcilied agent tokens"}
2023-09-08T23:47:23Z    INFO    Agent Pool Controller   {"agentpool": "terraform/agent-pool-dev", "msg": "successfully reconcilied agent pool"}

Kubectl Outputs

kubectl describe agentpool agent-pool-dev -n terraform
Name:         agent-pool-dev
Namespace:    terraform
Labels:       <none>
Annotations:  <none>
API Version:  app.terraform.io/v1alpha2
Kind:         AgentPool
Metadata:
  Creation Timestamp:  2023-09-08T23:17:25Z
  Finalizers:
    agentpool.app.terraform.io/finalizer
  Generation:  2
  Managed Fields:
    API Version:  app.terraform.io/v1alpha2
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"agentpool.app.terraform.io/finalizer":
    Manager:      manager
    Operation:    Update
    Time:         2023-09-08T23:17:25Z
    API Version:  app.terraform.io/v1alpha2
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:agentPoolID:
        f:agentTokens:
        f:observedGeneration:
    Manager:      manager
    Operation:    Update
    Subresource:  status
    Time:         2023-09-08T23:23:59Z
    API Version:  app.terraform.io/v1alpha2
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:agentTokens:
        f:name:
        f:organization:
        f:token:
          .:
          f:secretKeyRef:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2023-09-08T23:34:06Z
  Resource Version:  669232
  UID:               df98e249-d166-42c0-9ee6-345185a0b7d3
Spec:
  Agent Tokens:
    Name:        agent-pool-1
    Name:        agent-pool-2
    Name:        agent-pool-3
  Name:          agent-pool-dev
  Organization:  TERRAFORM_CLOUD_ORG_NAME
  Token:
    Secret Key Ref:
      Key:   teamtoken
      Name:  terraformrc
Status:
  Agent Pool ID:  apool-YWpBtoQYSRKjaWZS
  Agent Tokens:
    Created At:         1694215438
    Id:                 at-e5ch17BtjL1XMb54
    Last Used At:       -62135596800
    Name:               agent-pool-1
    Created At:         1694215438
    Id:                 at-ZuFfhzYiFwnrZx49
    Last Used At:       -62135596800
    Name:               agent-pool-2
    Created At:         1694215439
    Id:                 at-7SETHcSnMdZNmXEU
    Last Used At:       -62135596800
    Name:               agent-pool-3
  Observed Generation:  2
Events:
  Type    Reason                Age                   From                 Message
  ----    ------                ----                  ----                 -------
  Normal  AddFinalizer          28m                   AgentPoolController  Successfully added finalizer agentpool.app.terraform.io/finalizer to the object
  Normal  ReconcileAgentPool    28m                   AgentPoolController  Status.AgentPoolID is empty, creating a new agent pool
  Normal  ReconcileAgentPool    28m                   AgentPoolController  Successfully created a new agent pool with ID apool-YWpBtoQYSRKjaWZS
  Normal  ReconcileAgentPool    18m (x12 over 28m)    AgentPoolController  Successfully reconcilied agent pool ID apool-YWpBtoQYSRKjaWZS
  Normal  ReconcileAgentTokens  2m45s (x27 over 28m)  AgentPoolController  Reconcilied agent tokens in agent pool ID apool-YWpBtoQYSRKjaWZS

Question

Attempting to create a terraform agent with terraform operator, the agent pool creates successfully, however, the status remains on "Waiting for agents..." . Triggered a workflow and receiving a warning message as shows below.

Any advice?

References

Community Note

• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

[jrhouston]

Thanks for opening this @grabskimm – is there any output if you use kubectl logs on the tfc-agent pods themselves?

[grabskimm]

@jrhouston it keeps repeating the same as below

kubectl logs tf-operator-86c9db6fd8-gwmgk -n terraform manager | grep agent
2023-09-13T14:44:29Z    INFO    Reconcile Agent Tokens  {"agentpool": "terraform/agent-pool-dev", "msg": "new reconciliation event"}
2023-09-13T14:44:29Z    INFO    Reconcile Agent Tokens  {"agentpool": "terraform/agent-pool-dev", "msg": "Kubernetes Secret \"agent-pool-dev-agent-pool\" exists"}
2023-09-13T14:44:29Z    INFO    Reconcile Agent Tokens  {"agentpool": "terraform/agent-pool-dev", "msg": "successfully reconcilied agent tokens"}
2023-09-13T14:44:29Z    INFO    Agent Pool Controller   {"agentpool": "terraform/agent-pool-dev", "msg": "successfully reconcilied agent pool"}
2023-09-13T14:45:30Z    INFO    Agent Pool Controller   {"agentpool": "terraform/agent-pool-dev", "msg": "new reconciliation event"}
2023-09-13T14:45:31Z    INFO    Reconcile Agent Pool    {"agentpool": "terraform/agent-pool-dev", "msg": "reconciling agent pool"}
2023-09-13T14:45:31Z    INFO    Reconcile Agent Tokens  {"agentpool": "terraform/agent-pool-dev", "msg": "new reconciliation event"}
2023-09-13T14:45:31Z    INFO    Reconcile Agent Tokens  {"agentpool": "terraform/agent-pool-dev", "msg": "Kubernetes Secret \"agent-pool-dev-agent-pool\" exists"}
2023-09-13T14:45:31Z    INFO    Reconcile Agent Tokens  {"agentpool": "terraform/agent-pool-dev", "msg": "successfully reconcilied agent tokens"}
2023-09-13T14:45:31Z    INFO    Agent Pool Controller   {"agentpool": "terraform/agent-pool-dev", "msg": "successfully reconcilied agent pool"}
2023-09-13T14:46:32Z    INFO    Agent Pool Controller   {"agentpool": "terraform/agent-pool-dev", "msg": "new reconciliation event"}
2023-09-13T14:46:33Z    INFO    Reconcile Agent Pool    {"agentpool": "terraform/agent-pool-dev", "msg": "reconciling agent pool"}
2023-09-13T14:46:33Z    INFO    Reconcile Agent Tokens  {"agentpool": "terraform/agent-pool-dev", "msg": "new reconciliation event"}
2023-09-13T14:46:33Z    INFO    Reconcile Agent Tokens  {"agentpool": "terraform/agent-pool-dev", "msg": "Kubernetes Secret \"agent-pool-dev-agent-pool\" exists"}
2023-09-13T14:46:33Z    INFO    Reconcile Agent Tokens  {"agentpool": "terraform/agent-pool-dev", "msg": "successfully reconcilied agent tokens"}
2023-09-13T14:46:33Z    INFO    Agent Pool Controller   {"agentpool": "terraform/agent-pool-dev", "msg": "successfully reconcilied agent pool"}
2023-09-13T14:47:34Z    INFO    Agent Pool Controller   {"agentpool": "terraform/agent-pool-dev", "msg": "new reconciliation event"}
2023-09-13T14:47:35Z    INFO    Reconcile Agent Pool    {"agentpool": "terraform/agent-pool-dev", "msg": "reconciling agent pool"}
2023-09-13T14:47:35Z    INFO    Reconcile Agent Tokens  {"agentpool": "terraform/agent-pool-dev", "msg": "new reconciliation event"}
2023-09-13T14:47:35Z    INFO    Reconcile Agent Tokens  {"agentpool": "terraform/agent-pool-dev", "msg": "Kubernetes Secret \"agent-pool-dev-agent-pool\" exists"}
2023-09-13T14:47:35Z    INFO    Reconcile Agent Tokens  {"agentpool": "terraform/agent-pool-dev", "msg": "successfully reconcilied agent tokens"}
2023-09-13T14:47:35Z    INFO    Agent Pool Controller   {"agentpool": "terraform/agent-pool-dev", "msg": "successfully reconcilied agent pool"}
2023-09-13T14:48:36Z    INFO    Agent Pool Controller   {"agentpool": "terraform/agent-pool-dev", "msg": "new reconciliation event"}
2023-09-13T14:48:37Z    INFO    Reconcile Agent Pool    {"agentpool": "terraform/agent-pool-dev", "msg": "reconciling agent pool"}
2023-09-13T14:48:37Z    INFO    Reconcile Agent Tokens  {"agentpool": "terraform/agent-pool-dev", "msg": "new reconciliation event"}
2023-09-13T14:48:37Z    INFO    Reconcile Agent Tokens  {"agentpool": "terraform/agent-pool-dev", "msg": "Kubernetes Secret \"agent-pool-dev-agent-pool\" exists"}
2023-09-13T14:48:37Z    INFO    Reconcile Agent Tokens  {"agentpool": "terraform/agent-pool-dev", "msg": "successfully reconcilied agent tokens"}
2023-09-13T14:48:37Z    INFO    Agent Pool Controller   {"agentpool": "terraform/agent-pool-dev", "msg": "successfully reconcilied agent pool"}
2023-09-13T14:49:39Z    INFO    Agent Pool Controller   {"agentpool": "terraform/agent-pool-dev", "msg": "new reconciliation event"}
2023-09-13T14:49:39Z    INFO    Reconcile Agent Pool    {"agentpool": "terraform/agent-pool-dev", "msg": "reconciling agent pool"}
2023-09-13T14:49:39Z    INFO    Reconcile Agent Tokens  {"agentpool": "terraform/agent-pool-dev", "msg": "new reconciliation event"}
2023-09-13T14:49:39Z    INFO    Reconcile Agent Tokens  {"agentpool": "terraform/agent-pool-dev", "msg": "Kubernetes Secret \"agent-pool-dev-agent-pool\" exists"}
2023-09-13T14:49:39Z    INFO    Reconcile Agent Tokens  {"agentpool": "terraform/agent-pool-dev", "msg": "successfully reconcilied agent tokens"}
2023-09-13T14:49:39Z    INFO    Agent Pool Controller   {"agentpool": "terraform/agent-pool-dev", "msg": "successfully reconcilied agent pool"}

[grabskimm]

Any thoughts?

[marceloboeira]

What type of token did you use?

The operator (because of security concerns with the TFC APIs) swallows some errors and shows some misleading things from time to time (e.g.: 404 not found when the token is invalid instead of Unauthorised).

If you use a "org" level token it works partially since it can create workspaces but not agent pools, but it'd fail under the hood in some of these because that token can't manage agent pools tokens, plan/applies and such. You have to use a user or team-token AFAIK since those can request agent tokens...

There are three kinds of tokens available:
User tokens — each Terraform Cloud user can have any number of API tokens, which can make requests on their behalf.
Team tokens — each team can have one API token at a time. This is intended for performing plans and applies via a CI/CD pipeline.
Organization tokens — each organization can have one API token at a time. This is intended for automating the management of teams, team membership, and workspaces. The organization token cannot perform plans and applies.

https://developer.hashicorp.com/terraform/cloud-docs/api-docs

You should be able to test that by using CURL an trying to generate some agent pool tokens:
https://developer.hashicorp.com/terraform/cloud-docs/api-docs/agent-tokens

[jrhouston]

@grabskimm Did you figure this out?

There should be a Deployment created in your cluster called agents-of-agent-pool-dev can you share the output of kubectl describe from that, and do a kubectl logs on one of the actual agent pods to see if there's any error there?