Support re-rolling unrolled recursive TF types

[iwahbe]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

The terraform type system is structural, meaning that subtypes are not named. This prevents terraform from describing recursive types. Terraform providers fake recursion by manually unrolling recursive types to a given depth, generating a huge number of types in the process.

Pulumi needs to generate an in-code representation of these types leading to PRs like this:

Pulumi's type system is nominal (mirroring most programming languages), and we do support recursive types. Recursive typing is worse in Pulumi then it is in TF (for bridged providers) because of the translation, but Pulumi providers can actually offer a better experience than TF providers here.

Solution

The bridge should have native support for re-rolling recursive types back into their recursive descriptions.

A MVP implementation enables designating a resource type on ProviderInfo as "unrolled recursive" on specific fields. The bridge would then handle re-rolling that type.

An ideal solution would automatically detect deep nesting, and then re-roll without manually marking as recursive. This would require inferring that a type is recursive, and on what fields it recurses on, then calling into the MVP solution.

Backwards compatibility

The MVP is fully backwards compatible, since it is off by default. Automatic detection is technically breaking, but we can configure it so that it only breaks providers with an unusable number of resources (wafv2 style).

Affected area/feature

Prior Art

• [v6] Refactor WafV2 types to recursive pulumi-aws#2589

[iwahbe]

This effectively blocks pulumi/pulumi-datadog#359, unless we want to ship a massive performance regression.

[iwahbe]

Looks like this is also blocking pulumi/pulumi-akamai#314.

[t0yv0]

Curious @iwahbe if you want to hand it over to me or @VenelinMartinov or still planning to finish out. Exciting feature.

[iwahbe]

Curious @iwahbe if you want to hand it over to me or @VenelinMartinov or still planning to finish out. Exciting feature.

I'm going to continue next iteration. I'd love some help finishing, it's a thornier technical issue then I expected.

[t0yv0]

There's some related work I've been trying to wrap up. There are coupled problems.

1. detect recursion emulation
2. detect type replicas
3. remove type replicas

This PR has reasonably good recursion detector, but it is written against PackageSpec instead of shim.Schema:

#2134

The next PR explores non-recursive type replica detection and removal at scale. It turns out scale makes things difficult-ish. The evolving approach is to have tooling that detects replicas and then has maintainer approve of them and pick names. Approval step is helpful to eliminate accidentally shared types that will evolve separately upstream. Naming is necessary to avoid super long tokens. Working on shim.Schema level instead of PackageSpec level seems advantageous for this use case.

pulumi/pulumi-aws#4449

This PR landed to give a mechanism for curtail type explosion at tfgen level:

#2409

[iwahbe]

pulumi/pulumi-akamai#366 is an instance of recursive types (and the subsequent compilation resource explosion).

[t0yv0]

The PR is not quite ready but I learned a lot from prototyping, I'd like to pitch this sometime soon-ish on this to move forward.