Upgrading to tfe v0.45.0 wants to add value on all sensitive tfe_variable resources

[pndurette]

Hopefully this is only a question going from tfe v0.44.1 to v0.45.0, as this might be expected behaviour.

Terraform Cloud/Enterprise version

Terraform Cloud

Terraform version

1.3.9 (via Terraform Cloud, the version is pinned to ~> 1.3.0)

Terraform v1.3.9
on linux_amd64

Terraform Configuration Files

# Variables (terraform, sensitive, ignore value changes)
resource "tfe_variable" "tfvars_sensitive" {
  for_each = var.tfvars_sensitive

  key          = each.key
  description  = each.value.description
  category     = "terraform"
  sensitive    = true
  workspace_id = tfe_workspace.main.id

  lifecycle {
    ignore_changes = [value]
  }
}

Debug Output

I've followed the updates to tfe_variable in the v0.45.0 release notes with its related issues, bug fixes and breaking changes, but I can't see why my plan acts like this.

This tfe configuration manages a lot of workspaces, so full debug output would be very long (and would need to be scrubbed) but let me know if you need more and will happily provide!

Expected Behavior

No changes when updating from tfe v0.44.1 to v0.45.0

Actual Behavior

The speculative plan for going from tfe v0.44.1 to v0.45.0 wants to add (?) value to all tfe_variable that have sensitive = true (for all category), e.g.:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # <resource path> will be updated in-place
  ~ resource "tfe_variable" "envvars_sensitive" {
        id           = "var-some-id"
      + value        = (sensitive value)
        # (6 unchanged attributes hidden)
    }

# ....

Plan: 0 to add, 29 to change, 0 to destroy.

etc.

From what I suspect, this would replace all these variables with empty strings? It's hard to tell what's really going to happen, but it would be pretty harmful if it did.

I see this note was added in the tfe_variable docs, but I can't tell how/if this affects my case:

terraform-provider-tfe/website/docs/r/variable.html.markdown

Lines 85 to 89 in aad4a6e

	~> **NOTE:** When `sensitive` is set to true, Terraform cannot detect and repair
	drift if `value` is later changed out-of-band via the Terraform Cloud UI.
	Terraform will only change the value for a sensitive variable if you change
	`value` in the configuration, so that it no longer matches the last known value
	in the state.

Additional Context

I've read the entirety of #873 and #839 and just like the latter was happening, here we:

1. Create the tfe_variable in Terraform with lifecycle { ignore_changes = [value] }
2. In the UI, we go edit it to place in the value.

Which as been working well for us for a year

[brandonc]

@pndurette Hello, and thanks for your patience while we investigated this issue. The upgrade should not have triggered a change or addition to value. I did find a planned modification when the variable was created without a value, but I believe this was for adding readable_value with a null value (see my repro steps below)

In short, lifecycle { ignore_changes = [value] } should continue to do its job. Since the provider cannot read the value of any sensitive variable, it should not result in a plan to add a value without some other series of events I didn't capture.

Here were my steps:

1. I applied this config with provider 0.44.1:

terraform {
  required_providers {
    tfe = {
      version = "0.44.1"
    }
  }
}

resource "tfe_workspace" "variable_workspace" {
  name         = "variable-workspace"
  organization = "foo"
}

resource "tfe_variable" "sensitive_var" {
  key          = "sensitive_key"
  category     = "terraform"
  workspace_id = tfe_workspace.variable_workspace.id
  description  = "a useful description"
  sensitive = true

  lifecycle {
    ignore_changes = [value]
  }
}

2. I modified required_providers tfe to 0.45.0 and ran terraform init -upgrade
3. I modified the variable value in the UI
4. terraform apply then came up with this plan

Terraform will perform the following actions:

  # tfe_variable.sensitive_var will be updated in-place
  ~ resource "tfe_variable" "sensitive_var" {
        id           = "var-PSKgHAXpLr8R4t9z"
        # (6 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Applying it did not overwrite the value. However, in your case, I would expect the value to be overwritten because there is a value in your plan, possibly coming from some prior state? I couldn't reproduce that with terraform 1.3.9 or latest. Do you have any other ideas about where the value may have come from?

[TakayoshiNaito]

I am also encountering the same problem.

It also occurs with Terraform = v1.6.4 and tfe-provider=v0.50.0.

In my case, I originally set the value with null and then put the value on the TFC UI. In this case, it worked without setting ignore_changes.

However, after upgrading to v0.44, I found the following difference.

  # module.xxxx.tfe_variable.unmanaged_custom["TF_VAR_XXXXXX"] will be updated in-place
  ~ resource "tfe_variable" "unmanaged_custom" {
        id           = "var-XXXXXXXXXXXXXX"
      + value        = (sensitive value)
        # (5 unchanged attributes hidden)
    }

On the other hand, it may be working correctly despite the same implementation.

When I checked the "state file" to investigate the cause, I found that the former (which is updated) had a value of null while the latter (which is not updated) had a value of "" in the following section. This is despite the fact that both were set to null in the code.

          "attributes": {
            "category": "env",
            "description": "",
            "hcl": false,
            "id": "var-XXXXXXXXXX",
            "key": "TF_VAR_XXXXXXX",
            "readable_value": null,
            "sensitive": true,
            "value": "",                                   # "" or null 
            "variable_set_id": null,
            "workspace_id": "ws-XXXXXXXXXX"
          },

Can the above help to solve the problem?

[pndurette]

@brandonc Sorry for replying so late, haven't taken the time to come back to this (and still on tfe 0.44.1 because I'm not confident enough this won't wipe a lot of secrets)

Applying it did not overwrite the value. However, in your case, I would expect the value to be overwritten because there is a value in your plan, possibly coming from some prior state? I couldn't reproduce that with terraform 1.3.9 or latest. Do you have any other ideas about where the value may have come from?

I'm curious by what you mean by "possibly coming from some prior state"? Anything I can do? All I can do is a refresh and I'm still getting those.

The only difference in my code with your example is that I also have a precondition {} block in the lifecycle {} block.

I can't really think of anything and like you said I'm pretty sure this will wipe all those variables.

Any other idea of what I can try? Bit at a loss.

[pndurette]

update on that—
I finally bit the bullet and did the update (to 0.55.0) fully prepared to re-add all my sensitive variables, and despite the plan, and to my surprise, everything stayed as-is. So I thing this can be closed.