Test potential MustApplyAutoAliases speedup #1414

mjeffryes · 2023-10-03T21:22:00Z

Uses the build path for the binary to identify when we are in plugin runtime and disable sections of ApplyAutoAliases that are only needed for generating the bridge metadata file.

❯ (cd provider; go test -run="^$" -bench="BenchmarkProvider.*Gen$" -benchtime=10s)
goos: darwin
goarch: arm64
pkg: github.com/pulumi/pulumi-aws/provider/v6
BenchmarkProviderGen-12      	      58	 204080193 ns/op
BenchmarkProviderNoGen-12    	      68	 174488096 ns/op
PASS

I've extracted the bit of trying to guess whether we're in a tfgen binary or a plugin binary to it's own module so we could change out the implementation more easily in the future.

mjeffryes · 2023-10-03T21:23:19Z

pkg/tfbridge/auto_aliasing.go

+	/*
+		if err := md.Set(artifact, aliasMetadataKey, hist); err != nil {
+			// Set fails only when `hist` is not serializable. Because `hist` is
+			// composed of marshallable, non-cyclic types, this is impossible.
+			contract.AssertNoErrorf(err, "History failed to serialize")
+		}
+	*/


This change by itself yields:

❯ go test -run="^$" -bench="BenchmarkProvider$" -benchtime=10s goos: darwin goarch: arm64 pkg: github.com/pulumi/pulumi-aws/provider/v6 BenchmarkProvider-12 94 126865278 ns/op PASS

github-actions · 2023-10-03T21:27:22Z

Diff for pulumi-azuread with merge commit 37c59ac

github-actions · 2023-10-03T21:28:40Z

Diff for pulumi-random with merge commit 37c59ac

…e in tfgen

github-actions · 2023-10-03T22:39:03Z

Diff for pulumi-azuread with merge commit 8510cd5

github-actions · 2023-10-03T22:40:42Z

Diff for pulumi-random with merge commit 8510cd5

pkg/tfbridge/auto_aliasing.go

github-actions · 2023-10-04T17:25:12Z

Diff for pulumi-azuread with merge commit 874e28f

github-actions · 2023-10-04T17:27:00Z

Diff for pulumi-random with merge commit 874e28f

github-actions · 2023-10-04T17:31:11Z

Diff for pulumi-azuread with merge commit b218473

github-actions · 2023-10-04T17:34:13Z

Diff for pulumi-random with merge commit b218473

github-actions · 2023-10-04T22:24:53Z

Diff for pulumi-random with merge commit 2b2be54

github-actions · 2023-10-04T22:25:00Z

Diff for pulumi-azuread with merge commit 2b2be54

codecov-commenter · 2023-10-04T22:27:21Z

Codecov Report

Merging #1414 (d74f91f) into master (772baf6) will increase coverage by 0.83%.
Report is 4 commits behind head on master.
The diff coverage is 77.77%.

@@            Coverage Diff             @@
##           master    #1414      +/-   ##
==========================================
+ Coverage   57.01%   57.84%   +0.83%     
==========================================
  Files         147      284     +137     
  Lines       24400    39500   +15100     
==========================================
+ Hits        13911    22849    +8938     
- Misses       9505    15314    +5809     
- Partials      984     1337     +353

Files	Coverage Δ
pkg/tfbridge/auto_aliasing.go	`84.58% <92.30%> (+0.47%)`	⬆️
pkg/tfbridge/runtime.go	`64.28% <64.28%> (ø)`

... and 186 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

mjeffryes · 2023-10-04T23:11:33Z

Boo. Can't reproduce these test failures locally for some reason 😢

t0yv0 · 2023-10-05T12:59:48Z

pkg/tfbridge/runtime.go

+	}
+}
+
+func ReadRuntimeInfo() RuntimeInfo {


I'd be a bit careful exposing public API in pkg/tfbridge that's top visibility.

Do we want providers to use this? Until there is a need for providers to use this functionality, I would keep it private (or internal/).

I was using it from aws to benchmark each way, but I agree we should not make it visible

t0yv0 · 2023-10-05T13:09:32Z

Test failures : something around token tests is not recognizing the context it's running in I bet.

15% may be interesting. Let me see how far are we above the artificial alerting threshold. We need to go down 30% or so from the total. I'd be for taking some complexity here if it buys us 15% even if it doesn't get to where we need to be.

mjeffryes · 2023-10-06T21:12:25Z

Learned on a different thread that we may be initializing the plugin even more than 4 times in some programs; given that context, I agree with @t0yv0 that it's probably worth shipping this at least as a temporary measure until we have a chance to more carefully separate runtime and gen-time concerns.

github-actions · 2023-10-06T21:15:23Z

Diff for pulumi-random with merge commit 0920614

github-actions · 2023-10-06T21:15:27Z

Diff for pulumi-azuread with merge commit 0920614

github-actions · 2023-10-09T21:25:19Z

Diff for pulumi-azuread with merge commit e0c3359

github-actions · 2023-10-09T21:26:24Z

Diff for pulumi-random with merge commit e0c3359

iwahbe

LGTM

iwahbe · 2023-10-10T19:45:27Z

pkg/tfbridge/runtime.go

+func getRuntimeStage() runtimeStage {
+	return runtime.stage
+}


Nit: I would inline this helper function.

iwahbe · 2023-10-10T19:46:45Z

pkg/tfbridge/auto_aliasing.go

+func isTfgen() bool {
+	return getRuntimeStage() != resourceStage
+}


Nit: this function should be in the runtime.go file.

t0yv0 · 2023-10-10T20:21:41Z

pkg/tfbridge/auto_aliasing.go

-		eHasH, eHasI := applyResourceMaxItemsOneAliasing(e, &h.Elem.Fields, &info.Elem.Fields)
+		var eHasH, eHasI bool
+		if h.Elem != nil {
+			eHasH, eHasI = applyResourceMaxItemsOneAliasing(e, &h.Elem.Fields, &info.Elem.Fields)


Are we sure this here is safe? These functions edit SchemaInfo.MaxItemsOne to ensure backwards compatibility. If these edits are dropped at runtime, but only performed at compile-time, we'll get unexpected runtime serializer errors possibly.

applyResourceMaxItemsOneAliasing and applyMaxItemsOneAliasing panic if h.Elem is nil

The change in line 368 above means that sometimes h.elem will be nil here (ie. in the case that there was nothing in the metadata for this field), so we have to add the test. At runtime, it's safe to just skip because there's nothing in the metadata.

Although, it writing this I can see a cleaner way to handle this with a lot fewer conditionals, so I'll try that.

I'm worried that this changes semantics. I don't know what conditions precisely get us to "sometimes h.elem will be nil" but it sounds like it shouldn't be/do that because it didn't trigger before the flags here were introduced?

Are we quite certain that under both versions of the flag isTfgen() = {true,false} we're getting the same providerInfo back?

Something perhaps checkable with:

reflect.DeepEqual(a.Resources, b.Resouces)`

If this holds on a big provider I'd be less worried.

Hm the last version looks a lot cleaner! It seems to me only history edits are skipped, and there's short-cicuiting of edits when we know there's no history so there definitionally shouldn't be edits. This looks safer. Think about the invariant above but if you're confident we don't need further testing.

I'll run that test on aws before I merge, just to be sure.

So the Resources are not quite identical, but I think semantically the same. When running with isTfGen() == false we sometimes produce empty field maps, where the version with isTfgen() == true produces nil

Diff: --- Expected +++ Actual @@ -79007,4 +79007,3 @@ Elem: (*tfbridge.SchemaInfo)(<nil>), - Fields: (map[string]*tfbridge.SchemaInfo) { - }, + Fields: (map[string]*tfbridge.SchemaInfo) <nil>, Asset: (*tfbridge.AssetTranslation)(<nil>), @@ -79124,4 +79123,3 @@ Elem: (*tfbridge.SchemaInfo)(<nil>), - Fields: (map[string]*tfbridge.SchemaInfo) { - }, + Fields: (map[string]*tfbridge.SchemaInfo) <nil>, Asset: (*tfbridge.AssetTranslation)(<nil>), @@ -79168,4 +79166,3 @@ Elem: (*tfbridge.SchemaInfo)(<nil>), - Fields: (map[string]*tfbridge.SchemaInfo) { - }, + Fields: (map[string]*tfbridge.SchemaInfo) <nil>, Asset: (*tfbridge.AssetTranslation)(<nil>), @@ -79212,4 +79209,3 @@ Elem: (*tfbridge.SchemaInfo)(<nil>), - Fields: (map[string]*tfbridge.SchemaInfo) { - }, + Fields: (map[string]*tfbridge.SchemaInfo) <nil>, Asset: (*tfbridge.AssetTranslation)(<nil>), Test: TestProviderSame

Yes, nil vs empty should be fine here.

t0yv0 · 2023-10-10T20:22:13Z

pkg/tfbridge/runtime.go

+
+var runtime = initRuntimeInfo()
+
+func initRuntimeInfo() runtimeInfo {


github-actions · 2023-10-10T21:27:37Z

Diff for pulumi-azuread with merge commit 52bd873

github-actions · 2023-10-10T21:28:40Z

Diff for pulumi-random with merge commit 52bd873

mjeffryes · 2023-10-10T21:50:57Z

Cleaned up the logic in applyMaxItemsOneAliasing, and consolidated helper functions into runtime.go (reverified the benchmarked speedup and that gen is unaffected)

WIP: test potential MustApplyAutoAliases speedup

e89af80

mjeffryes commented Oct 3, 2023

View reviewed changes

Disable generating autoaliases if the BuildInfo.path doesn't say we'r…

62ced78

…e in tfgen

mjeffryes commented Oct 3, 2023

View reviewed changes

pkg/tfbridge/auto_aliasing.go Show resolved Hide resolved

mjeffryes requested review from iwahbe, a team and t0yv0 October 3, 2023 22:46

iwahbe reviewed Oct 3, 2023

View reviewed changes

pkg/tfbridge/auto_aliasing.go Outdated Show resolved Hide resolved

pkg/tfbridge/auto_aliasing.go Outdated Show resolved Hide resolved

mjeffryes mentioned this pull request Oct 4, 2023

Draft: Must apply auto alias speedup with global #1415

Closed

mjeffryes force-pushed the test_must_apply_auto_alias_speedup branch from b4e8dd1 to 62ced78 Compare October 4, 2023 17:27

Try using a global

08c845f

t0yv0 reviewed Oct 5, 2023

View reviewed changes

Make the runtime info package local

2b391da

mjeffryes changed the title ~~DO NOT MERGE: test potential MustApplyAutoAliases speedup~~ Test potential MustApplyAutoAliases speedup Oct 7, 2023

appease linter

5424cad

mjeffryes requested review from iwahbe and t0yv0 October 10, 2023 16:35

iwahbe approved these changes Oct 10, 2023

View reviewed changes

iwahbe reviewed Oct 10, 2023

View reviewed changes

t0yv0 reviewed Oct 10, 2023

View reviewed changes

pkg/tfbridge/runtime.go

var runtime = initRuntimeInfo()

func initRuntimeInfo() runtimeInfo {

Copy link

Member

t0yv0 Oct 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beautiful.

simplify

d74f91f

t0yv0 self-requested a review October 10, 2023 22:20

t0yv0 approved these changes Oct 10, 2023

View reviewed changes

t0yv0 mentioned this pull request Oct 12, 2023

v3.62.0 bridge release #1439

Closed

4 tasks

mjeffryes merged commit cad926a into master Oct 12, 2023
9 checks passed

mjeffryes deleted the test_must_apply_auto_alias_speedup branch October 12, 2023 20:00


		var runtime = initRuntimeInfo()

		func initRuntimeInfo() runtimeInfo {

Test potential MustApplyAutoAliases speedup #1414

Test potential MustApplyAutoAliases speedup #1414

Conversation

mjeffryes commented Oct 3, 2023 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Oct 3, 2023

github-actions bot commented Oct 3, 2023

github-actions bot commented Oct 3, 2023

github-actions bot commented Oct 3, 2023

github-actions bot commented Oct 4, 2023

github-actions bot commented Oct 4, 2023

github-actions bot commented Oct 4, 2023

github-actions bot commented Oct 4, 2023

github-actions bot commented Oct 4, 2023

github-actions bot commented Oct 4, 2023

codecov-commenter commented Oct 4, 2023 • edited Loading

Codecov Report

mjeffryes commented Oct 4, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

t0yv0 commented Oct 5, 2023

mjeffryes commented Oct 6, 2023

github-actions bot commented Oct 6, 2023

github-actions bot commented Oct 6, 2023

github-actions bot commented Oct 9, 2023

github-actions bot commented Oct 9, 2023

iwahbe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

t0yv0 Oct 10, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Oct 10, 2023

github-actions bot commented Oct 10, 2023

mjeffryes commented Oct 10, 2023 • edited Loading

mjeffryes commented Oct 3, 2023 •

edited

Loading

codecov-commenter commented Oct 4, 2023 •

edited

Loading

t0yv0 Oct 10, 2023 •

edited

Loading

mjeffryes commented Oct 10, 2023 •

edited

Loading