Support gossip encryption key as separate output to hcp_consul_cluster resurce

[lkysow]

Description

It would be helpful for the gossip encryption key to be a separate output from the hcp_consul_cluster resource. Currently it must be parsed from the consul_config_file output.

New or Affected Resource(s)

• hcp_consul_cluster

Potential Terraform Configuration

output `consul_gossip_encryption_key` {}

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[bcmdarroch]

Hi @lkysow! Thanks for adding this issue. I'll bring this to our cloud Consul team and see if we can get this on their board.

[lkysow]

Also having datacenter as an output would be useful as well.

[webdog]

Is there a mechanism during the apply to retrieve the gossip key from the config file output, to share with other terrraform resources during the same run?

From the feedback of @apparentlymart in this JSON-related HashiCorp discuss thread, I'd like to marshal the output with jsondecode to a terraform object to store the value as a secret on a Kubernetes cluster that is running Consul Enterprise (The gossip key is referenced in the values file as a secret key/value pair in kube).

consul_config_file (String) The cluster config encoded as a Base64 string.

locals {
  raw_config = base64decode(jsondecode(hcp_consul_cluster.server.consul_config_file))
  gossip_key = local.raw_config.encrypt
}

output "consul_gossip_key" {
  value = local.gossip_key
}

From testing and prior art, I understand that .encrypt is a key containing the gossip key stored within the output of the config file. When using either dot or index notation, I receive errors indicating that I cannot parse this object.

│ Error: Unsupported attribute
│
│   on modules/deploy-kubernetes/modules/hcp/outputs.tf line 20, in locals:
│   20:   gossip_key = local.raw_config.encrypt
│     ├────────────────
│     │ local.raw_config is a string, known only after apply
│
│ Can't access attributes on a primitive-typed value (string).

╷
│ Error: Invalid index
│
│   on modules/deploy-kubernetes/modules/hcp/outputs.tf line 20, in locals:
│   20:   gossip_key = local.raw_config["encrypt"]
│     ├────────────────
│     │ local.raw_config is a string, known only after apply
│
│ This value does not have any indices.

The resolution I can come up with is using -target on this module during terraform apply, and then re-running terraform apply on the rest of the infrastructure, passing this output to to the intended resource. As this code is intended for a tutorial on our Learn platform, I'd like to avoid this workflow. It is subject to error and can be an unnatural workflow for Consul learners who may have limited prior experience with terraform.

It appears we are capturing the config output here in a data source, generated in L186:

terraform-provider-hcp/internal/provider/data_source_consul_cluster.go

Line 186 in f6ef4b6

clientConfigFiles *consulmodels.HashicorpCloudConsul20210204GetClientConfigResponse) error {

...and passing along the consul_config_file attribute parsed from clientConfigFiles:

terraform-provider-hcp/internal/provider/data_source_consul_cluster.go

Lines 241 to 243 in f6ef4b6

	if err := d.Set("consul_config_file", clientConfigFiles.ConsulConfigFile.String()); err != nil {
	return err
	}

If the config is already stored as a JSON object from the lookup, I guess/assume the work needed to unpack the keys and values from the object would be to either add additional error conditions to setConsulDataSourceAttributes() or dynamically unpack all the values stored by the config file object.

I'm happy to raise my ✋ and contribute to a PR if this work is being blocked by other workstreams. I'm not an expert in golang by any means, but I'd be willing to learn 😺 - At minimum, I think offering both datacenter and gossip_key could be included in the changeset.

[webdog]

After digging around, here's an example of pulling the gossip key, looks like I may have swapped my json/base64 decoding. Oops!

https://github.com/hashicorp/terraform-aws-hcp-consul/blob/80e714415874e11b949869b00691eeaa6dbd0791/examples/hcp-eks-demo/main.tf#L93

gossip_encryption_key = jsondecode(base64decode(hcp_consul_cluster.main.consul_config_file))["encrypt"]

Examples like this would beneficial in the API docs for the provider, as there were none when initially investigating. cc @trujillo-adam for 👀

[apparentlymart]

Oh hmm yes this is an unfortunate case where Terraform had not quite enough information to return a useful error message. 😖

• jsondecode would normally fail if given a base64-encoded string that wasn't decoded yet, but because the Consul cluster isn't created yet we don't yet know the value of that string in order to syntax-check it.

  ╷
  │ Error: Error in function call
  │ 
  │   on example.tf line 4, in locals:
  │    4:   raw_config = base64decode(jsondecode(hcp_consul_cluster.main.consul_config_file))
  │     ├────────────────
  │     │ local.source is "eyJlbmNyeXB0IjoiaGVsbG8ifQ=="
  │ 
  │ Call to function "jsondecode" failed: invalid character 'e' looking for beginning of value.
  ╵
  

• base64decode would normally fail if given an object value instead of a string value, but because we don't yet know the content of the string given to jsonencode we therefore don't know the return type of that call, and so base64encode cannot return a type mismatch error yet, until the apply step.

• However, Terraform does know that base64encode can only ever return a string, and so it knows that it can't possibly be valid to access .encrypt on its result regardless of what it ultimately turns out to be, and so you get that reported as the error instead.

This is, I think, a good motivation for what this issue seems to have originally requested: if the needed information were exposed directly as an attribute of type string then Terraform's type checker could immediately know the type without first waiting to see what value this JSON string will take, and thus avoid the confusion caused by trying to decode a dynamically-generated JSON string here.