Make dataset + column creation idempotent to avoid punishing "deploy first" workflows

[xeger]

Problem Statement

• I typically run my services in an environment, and begin collecting traces, before I have defined any boards or triggers in my Terraform code.
• Honeycomb creates new datasets and columns on the fly
• If I deploy to some environments but not others, the Terraform code that I write to create boards and triggers may become valid to run in some environments, but not others, because it refers to columns or datasets that do not exist
• If, instead, I try to proactively create columns and datasets using Terraform, then that code be invalid in any environment where I have deployed

I'm forced to literally ship every change to production first, and then create a separate pull request later on that adds observability. This is a waste of my time and creates an observability gap.

Proposed Solution

For boards and columns, the state is trivial and I would argue that this provider should break ranks with Terraform and provide idempotent/upsert semantics. In other words: if the board or column already exists and if it is identical to the resource specification, the provider should simply pretend that create succeeded vs. erroring because the resource already exists.

Workarounds

One can switch to "observe first" i.e. decide on column and dataset names, ship a Terraform change to all environments, and then work on the service in question. That's alienating however.

[jharley]

Thanks for taking the time to open this issue, @xeger.

The idea of doing what are effectively "silent imports" is felt to be surprising behavior: especially when considering that the intention is to manage the lifecycle of a resource.

We have removed the spots where this use to happen in this code base for that reason, and view "error on conflict" as correct behavior in the Terraform ecosystem.

That all said, you are highlighting a rough edge that is top of mind (and felt by us as well) and currently being discussed internally to find a more holistic, product-wide solution.

I'll keep this issue open and report on progress periodically.

[xeger]

One thought: terraform can manage anything, but it's not always the best option for anything.

An official honeycomb CLI for calling management APIs, and featuring convergence behavior (potentially controlled via flags), might be a better fit for some.

The argument against, of course, is that that would be a ton of work. Making the terraform imports less silent (via an attribute or flag on the column and dataset resources, perhaps?) might be a quicker route to a solution.

Thanks for your consideration.

[jharley]

terraform can manage anything, but it's not always the best option for anything

Hard agree: I mentioned the other day in a discussion about this that it feels like Terraform has become the hammer (in the famous adage about everything looking like a nail) that many reach for because we don't really offer alternatives currently.

We are making progress with an OpenAPI specification and moving toward the shiny future of autogenerated client SDKs, which I'm hopeful will unlock (read: reduce maintenance burden in addition to delighting folks) new tools and support for additional ecosystems.