OmitType and TypeName to support sharing types

[t0yv0]

Introduce OmitType and TypeName flags to enforce type sharing.

Some of the upstream providers generate very large concrete schemata. TF is not being materially affected, just high RAM demands for in-memory processing. The example is inspired by QuickSight types in AWS. Pulumi is affected significantly. In Pulumi the default projection is going to generate named types for every instance of the shared schema. This leads to SDK bloat and issues with "filename too long."

With this change it is possible for the provider maintainer opt into explicit sharing of types, and ensure that the type names for the shared types have shorter meaningful prefixes.

At definition type the user can specify the type name to generate, which can be very short, and replace the automatically implied ReallyLongPrefixedTypeName like this:

"visuals": {
	Elem: &info.Schema{
		TypeName: tfbridge.Ref("Visual"),
	},
},

At reference time in another resource, the user can reuse an already generated type by token. This already worked before this change but had the downside of still generating unused helper types and causing SDK bloat.

"visuals": {
	Elem: &info.Schema{
		Type:     "testprov:index/Visual:Visual",
	},
},

With this change it is possible to instruct the bridge to stop generating the unused helper types:

"visuals": {
	Elem: &info.Schema{
		Type:     "testprov:index/Visual:Visual",
		OmitType: true
	},
},

[codecov]

Codecov Report

Attention: Patch coverage is 64.51613% with 11 lines in your changes missing coverage. Please review.

Project coverage is 57.66%. Comparing base (fbc2abd) to head (1370dde).
Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/tfgen/generate.go	50.00%	9 Missing and 1 partial ⚠️
pkg/tfgen/generate_schema.go	88.88%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2409   +/-   ##
=======================================
  Coverage   57.66%   57.66%           
=======================================
  Files         369      369           
  Lines       50121    50148   +27     
=======================================
+ Hits        28902    28920   +18     
- Misses      19641    19650    +9     
  Partials     1578     1578           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[t0yv0]

This does not check that the underlying type is structurally equal to the new replacement type. We could add this check.

[iwahbe]

This is a very surgical change, which is great. I would mark the new fields on info.Schema as experimental in their respective doc comments, just to hedge against future iteration on the design.

[iwahbe]

Nit:

I think this is clearer as an if statement. It looks odd to me to have 3 branches that all lead to the same body.

	if (t.typePrefixOverride != nil && other.typePrefixOverride == nil) ||
		(t.typePrefixOverride == nil && other.typePrefixOverride != nil) ||
		(t.typePrefixOverride != nil && other.typePrefixOverride != nil &&
			*t.typePrefixOverride != *other.typePrefixOverride) {
		return false
	}

That said, this is just an inequality check made painful by nil values. Nitpicking, I would express this as the inverse of an equality check:

Suggested change

	switch {
	case t.typePrefixOverride != nil && other.typePrefixOverride == nil:
	return false
	case t.typePrefixOverride == nil && other.typePrefixOverride != nil:
	return false
	case t.typePrefixOverride != nil && other.typePrefixOverride != nil &&
	*t.typePrefixOverride != *other.typePrefixOverride:
	return false
	}
	if eq := t.typePrefixOverride == other.typePrefixOverride ||
	(t.typePrefixOverride != nil && other.typePrefixOverride != nil &&
	*t.typePrefixOverride == *other.typePrefixOverride); !eq {
	return false
	}

[t0yv0]

This is an F# habit that may be out of place but I find individual cases more tractable than a large bool expr.

[VenelinMartinov]

Very nice! You mentioned tools to help provider maintainers spot these repeated types in the schemas. Is the plan to add these to upstream upgrades or tfgen or something else?

[t0yv0]

I've spent a lot of time last night getting Quicksight sharing rolled out to AWS against this functionality and found it can be made to work but it is quite awkward.

pulumi/pulumi-aws#4449

This PR reduces Quicksight from 30,000 to 7,000 types - there is still some work to specify sharing.

Couple of learnings there:

1. the user might want to recent the name of the auto-generated type and not just the prefix, since it is otherwise hard to predict the sharing token.

2. perhaps the better interface would be:

{ ShareTypeAs: "<token>" }

In AWS I built a stateful facade that emulates this by emitting {Prefix:""} on the first invocation and {Type: "<token>", OmitType: true} subsequently. It has some issues guessing the right token.

3. I've been relying heavily on a shared type detector from a different PR that operates on TypeSpec equality. It sounds like we could really use shim.Schema equality instead.

4. Once the provider builder confirms that certain types are shared, it can be extremely daunting to specify chasing down all the copies in complicated cases like Quicksight.

Perhaps we can automate the following flow.

1. go generate to compute a type sharing specification and check in as a yaml file. This specification specifies where the shared types are found and what are their suggested tokens.

2. Provider maintainer audits this to rename tokens as needed and "accept" that the discovered sharing is desirable, e.g. confirm that the identical types were generated from a shared func() invocation upstrema.

3. At resources.go time the sharing specification is loaded from disk and executed against ProviderInfo to find all copies of shared root types and specified in the corresponding info.Schema ; then tfgen proceeds to generate a schema with sharing

A question is what to do with dynamically bridged providers. It might be desirable there, since they are subject to type explosion as well. I think once we are certain of this functionality, we should default to detecting sharing in dynamically bridged providers, picking arbitrary names for the shared types.

[t0yv0]

Tweaked the API to make a lot more usable for QuickSight use case. PTAL.

[iwahbe]

LGTM

[pulumi-bot]

This PR has been shipped in release v3.91.0.