Tainting Resources

[ekristen]

Have you considered how to support tainting of resources?

[isaaguilar]

Tainting of which kind of resources?

[ekristen]

Any resource controlled by terraform terraform taint <resource>

[isaaguilar]

Interesting. I have not personally used terraform taint before. Would you happen to have an example of how taint might fit into the terraform-operator/runner workflow?

[ekristen]

I'm fairly familiar with operators, written a couple myself, I've generally found using annotations as a decent way of handling ephemeral or single time tasks.

Let's say you have a terraform project that deploys 30 resources and you need to rebuild 1 resource, maybe it's a server, maybe it's AWS keys.

Once tainted on a new plan/apply it'll be rebuilt/replaced depending on the resource type.

How I've handled this in the past is by using annotations, but there might be better ways.

• tf-operator/taint=module.server.aws_instance.linux

The operator would see it, run a taint, remove the annotation, then run plan/apply or something to that affect.

Happy to discuss further and talk options.

Examples

• terraform taint aws_instance.linux
• terraform taint module.server.aws_instance.linux

[isaaguilar]

I'm going to make it a point to play around with taint this week so we can talk about fitting it into terraform-operator.

[isaaguilar]

Hi @ekristen I was thinking about this for a little bit and it dawned on me that tainting a resource after running is a perfect use case for spec.postrunScript.

For example, if this were my manifest file, let's call it taint-test.yaml:

# taint-test.yaml
apiVersion: tf.isaaguilar.com/v1alpha1
kind: Terraform
metadata:
  name: tainttest
spec:
  terraformVersion: 0.13.4
  terraformRunnerPullPolicy: IfNotPresent
  terraformModule:
    address: https://github.com/cloudposse/terraform-aws-test-module.git
  customBackend: |-
    terraform {
      backend "kubernetes" {
        secret_suffix    = "tainttest"
        in_cluster_config  = true
      }
    }
  applyOnCreate: true
  applyOnUpdate: true
  applyOnDelete: true
  ignoreDelete: false   # Make sure to clean up example and delete on `kubectl delete`
  postrunScript: |-
    #/bin/bash

    #
    # cd into the directory TerraformRunner executes terraform commands
    #
    cd /main_module  

    #
    # Taint the resource from this run. (No need to run init since 
    # the TerraformRunner has already initialized earlier
    #
    terraform taint "random_integer.example[0]"

Apply the manifest to create the resource.

$ kubectl apply -f taint-test.yaml
terraform.tf.isaaguilar.com/tainttest created

Check the logs to see that the postrunScript tainted the resource.

$ kubectl logs -l job-name=tainttest

...
Resource instance random_integer.example[0] has been marked as tainted.

Trigger terraform-operator to schedule a new run by deleting the Job resource.

$ kubectl delete job -l job-name=tainttest
job.batch "tainttest" deleted

The terraform-operator creates a new job. By tailing the logs again you'll see the taint in effect

$ kubectl logs -l job-name=tainttest

...
Terraform will perform the following actions:
  # random_integer.example[0] is tainted, so must be replaced
...

Cleanup as usual

$ kubectl delete -f taint-test.yaml
terraform.tf.isaaguilar.com "tainttest" deleted

[isaaguilar]

Ah, speaking out loud here, the use case we're seeking might be more of tainting "on demand". The annotation method would be a viable solution. Some small tweaks would need to be made to the TerraformRunner's run.sh script, which is a pain point of the operator because it requires new builds of the tfops images and might silently fail for those who are using a custom terraform runner.