RFC: record aliases in provider

[mjeffryes]

This needs some cleaning still, but outlines a possible approach for recording the aliases in the provider shim so we can fully pack and unpack the provider info.

I've tested this locally using the aws provider (pulumi/pulumi-aws#2902). It appears to be working, and to offer a fairly significant speed up (>150ms in my local testing).

Feedback that would be most helpful:

1. ideas for how to test
2. Any suggestions for how to inject the aliasing with a lighter touch

[mjeffryes]

This was the most expedient way to make the aliases reachable for aws.

[mjeffryes]

The idea here was to introduce a wrapper for muxer.ProviderShim, just as I did for the base shim.Provider, but I took a short-cut and just modified ProviderShim directly for the POC

[mjeffryes]

This experiment has given me pretty good confidence that we're only using Set to add aliases today (Although since it's a public interface, it's possible for random providers to be calling it somewhere. It would be good to remove it in a future major release of the bridge.)

[mjeffryes]

This is ugly, but lets us discover aliases without adding a RangeAliases method to the base shim.Provider

[mjeffryes]

as discussed on pulumi/pulumi-aws#2902, a further improvement would be to just codegen the serialized form so the runtime can just load a bunch of maps directly into memory.

[mjeffryes]

This and the change below seem to eliminate the runtime use of Set on the bridge

[t0yv0]

Can we save this observation in a ticket? That'd be a good change to deprecate Set - except for this use case, which can be better named.

[mjeffryes]

Ian pointed out that this is probably a fruitless direction to pursue because there are too many non-serializable hooks in the ResourceInfo that are needed at runtime. We will have to take a more piecemeal approach.

[t0yv0]

You're so close @mjeffryes

I think a lot of this PR would still be needed.

It's an excellent observation that ResourceInfos have non-serializable things like Transformers, I apologize for missing that crucial bit in earlier design discussions. So we cannot turn them around through a serde boundary.

However as you're showing here it's entirely viable to capture side-effects at tfgen time, serialize them, and replay these side-effects at init-time to modify ResourceMap and ResourceInfos.

In case of ResourceMap, you've done most of the work here - it's AddAlias. Now it has to spy on these calls and do a type AddAlias struct { alias string, target string } trick to save aliasing at tfgen and replay at init-time.

Now, we need to also capture side-effects performed against ResourceInfo & friends.

It seems like we could try refactoring aliasing code to trace the effects.

Note the comment:

// Specifically, [ApplyAutoAliases] may perform the following actions:
//
// - Call [ProviderInfo.RenameResourceWithAlias] or [ProviderInfo.RenameDataSource]
// - Edit [ResourceInfo.Aliases]
// - Edit [SchemaInfo.MaxItemsOne]

If we could turn these into a data structure.. Basically ApplyAutoAliases could be doing two things. At tfgen time it does what it currently does, plus spies on the above effects and records them; at init-time it simply replays pre-recorded effects.

[mjeffryes]

I think a lot of this PR would still be needed.
I do think there is some value in this code just by virtue of working toward shrinking the mutable surface of shim.ResourceMap, I would be happy to refine this for that purpose if there's interest.

However, I think that it's not likely to be useful for the init latency problem.

If we could turn these into a data structure.. Basically ApplyAutoAliases could be doing two things. At tfgen time it does what it currently does, plus spies on the above effects and records them; at init-time it simply replays pre-recorded effects.

Yes, Ian and I came to the same conclusion. In fact, the first two steps (the actual aliases are already captured as a list of effects and then run in a second pass in ApplyAutoAliases, so we just need to make these data rather than lambdas and convert the MaxItemsOne transformations to use a similar approach. (In fact Ian pointed out that just switching MaxItemsOne to a two pass approach will save a lot of GC pressure from creating empty maps and discarding them.)