Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

initial pulumi preview (Go) on datadog SDK consumes more than 10GB of memory on V4 and 4GB on V3 #561

Closed
yishait-tailor opened this issue Jun 11, 2024 · 3 comments · Fixed by #570
Closed

initial pulumi preview (Go) on datadog SDK consumes more than 10GB of memory on V4 and 4GB on V3 #561

yishait-tailor opened this issue Jun 11, 2024 · 3 comments · Fixed by #570
Assignees
@iwahbe
Labels
kind/bug Some behavior is incorrect or out of spec resolution/fixed This issue was fixed

Comments

@yishait-tailor
Copy link

yishait-tailor commented Jun 11, 2024 •
edited
Loading

Describe what happened

Running pulumi preview on any resource in datadog SDK
with V4 consumes more than 10GB of memory on first run
V3 consumes more than 4GB of memory

very 2.0s: free -h                                                02d2802fe929: Tue Jun 11 07:20:45 2024

               total        used        free      shared  buff/cache   available
Mem:            15Gi        11Gi       1.6Gi       4.0Mi       2.8Gi       4.2Gi
Swap:             0B          0B          0B

using:

  • github.com/pulumi/pulumi-datadog/sdk/v4/go/datadog
  • github.com/pulumi/pulumi-datadog/sdk/v3/go/datadog

using the examples only with no changes
tested on several platforms all same behavior
using s3 as backend without pulumi account

Is there any way to avoid this?

Sample program

package main

import (
	"github.com/pulumi/pulumi-datadog/sdk/v4/go/datadog"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		_, err := datadog.NewMonitor(ctx, "foo", &datadog.MonitorArgs{
			Name:              pulumi.String("Name for monitor foo"),
			Type:              pulumi.String("metric alert"),
			Message:           pulumi.String("Monitor triggered. Notify: @hipchat-channel"),
			EscalationMessage: pulumi.String("Escalation message @pagerduty"),
			Query:             pulumi.String("avg(last_1h):avg:aws.ec2.cpu{environment:foo,host:foo} by {host} > 4"),
			MonitorThresholds: &datadog.MonitorMonitorThresholdsArgs{
				Warning:  pulumi.String("2"),
				Critical: pulumi.String("4"),
			},
			IncludeTags: pulumi.Bool(true),
			Tags: pulumi.StringArray{
				pulumi.String("foo:bar"),
				pulumi.String("team:fooBar"),
			},
		})
		if err != nil {
			return err
		}
		return nil
	})
}

Log output

root@02d2802fe929:/app/app# pulumi preview --logtostderr -v=9 > mac_log.txt
I0611 07:02:56.762808   23609 update.go:206] *** Starting Update(preview=true) ***
I0611 07:02:56.764645   23609 plugins.go:124] gatherPluginsFromProgram(): gathering plugins from language host
I0611 07:02:56.765220   23609 plugins.go:1800] GetPluginPath(language, go, <nil>): found on $PATH /pulumi/bin/pulumi-language-go
I0611 07:02:56.765347   23609 plugins.go:1823] GetPluginPath(language, go, <nil>): found next to current executable /pulumi/bin/pulumi-language-go
I0611 07:02:56.765391   23609 plugin.go:186] newPlugin(): Launching plugin 'go' from '/pulumi/bin/pulumi-language-go' with args: -root=/app/app,127.0.0.1:43685
I0611 07:02:56.811506   23609 langruntime_plugin.go:249] langhost[go].GetPluginInfo() executing
I0611 07:02:56.811942   23609 langruntime_plugin.go:136] langhost[go].GetRequiredPlugins(root=/app/app, program=/app/app, entryPoint=.) executing
I0611 07:02:56.871742   23609 langruntime_plugin.go:186] langhost[go].GetRequiredPlugins(root=/app/app, program=/app/app, entryPoint=.) success: #versions=1
I0611 07:02:56.871795   23609 plugins.go:137] gatherPluginsFromProgram(): plugin go <nil> () is required by language host
I0611 07:02:56.871802   23609 plugins.go:137] gatherPluginsFromProgram(): plugin datadog 4.28.0 () is required by language host
I0611 07:02:56.871806   23609 plugins.go:149] gatherPluginsFromSnapshot(): gathering plugins from snapshot
I0611 07:02:56.871810   23609 plugins.go:193] ensurePluginsAreInstalled(): beginning
I0611 07:02:56.871829   23609 plugins.go:1800] GetPluginPath(language, go, <nil>): found on $PATH /pulumi/bin/pulumi-language-go
I0611 07:02:56.871848   23609 plugins.go:1823] GetPluginPath(language, go, <nil>): found next to current executable /pulumi/bin/pulumi-language-go
I0611 07:02:56.871858   23609 plugins.go:203] ensurePluginsAreInstalled(): plugin go <nil> already installed
I0611 07:02:56.871911   23609 plugins.go:1874] GetPluginPath(resource, datadog, 4.28.0): enabling new plugin behavior
I0611 07:02:56.871929   23609 plugins.go:2008] SelectCompatiblePlugin(..., datadog): beginning
I0611 07:02:56.871934   23609 plugins.go:2041] SelectCompatiblePlugin(..., datadog): skipping plugin datadog-3.3.0: semver mismatch
I0611 07:02:56.871974   23609 plugins.go:229] ensurePluginsAreInstalled(): plugin datadog 4.28.0 not installed, doing install
I0611 07:02:56.871980   23609 plugins.go:248] installPlugin(datadog, 4.28.0): beginning install
I0611 07:02:56.871983   23609 plugins.go:262] installPlugin(datadog, 4.28.0): initiating download
I0611 07:02:56.872041   23609 plugins.go:473] plugin GitHub releases url: https://api.github.com/repos/pulumi/pulumi-datadog/releases/tags/v4.28.0
I0611 07:02:56.872063   23609 plugins.go:969] full plugin download url: https://api.github.com/repos/pulumi/pulumi-datadog/releases/tags/v4.28.0
I0611 07:02:57.687398   23609 plugins.go:508] datadog downloading from https://api.github.com/repos/pulumi/pulumi-datadog/releases/assets/163853504
I0611 07:02:57.688135   23609 plugins.go:969] full plugin download url: https://api.github.com/repos/pulumi/pulumi-datadog/releases/assets/163853504
[resource plugin datadog-4.28.0] installing
I0611 07:03:00.972644   23609 plugins.go:281] installPlugin(datadog, 4.28.0): extracting tarball to installation directory
I0611 07:03:01.277551   23609 ignore.go:44] Explicitly ignoring and discarding error: close /tmp/pulumi-plugin-tar1002728799: file already closed
I0611 07:03:01.277603   23609 plugins.go:288] installPlugin(datadog, 4.28.0): installation complete
I0611 07:03:01.280129   23609 plugins.go:236] ensurePluginsAreInstalled(): completed
I0611 07:03:01.280175   23609 plugins.go:340] computeDefaultProviderPlugins(): considering go
I0611 07:03:01.280180   23609 plugins.go:343] computeDefaultProviderPlugins(): skipping go, not a resource provider
I0611 07:03:01.280185   23609 plugins.go:340] computeDefaultProviderPlugins(): considering datadog-4.28.0
I0611 07:03:01.280189   23609 plugins.go:371] computeDefaultProviderPlugins(): plugin datadog-4.28.0 selected for package datadog (first seen)
I0611 07:03:01.280194   23609 log.go:73] computeDefaultProviderPlugins(): summary of default plugins:
I0611 07:03:01.280200   23609 plugins.go:379]   datadog         = 4.28.0
I0611 07:03:01.280618   23609 deployment_executor.go:218] deploymentExecutor.Execute(...): waiting for incoming events
I0611 07:03:01.280662   23609 step_executor.go:513] StepExecutor worker(7): worker coming online
I0611 07:03:01.280688   23609 step_executor.go:513] StepExecutor worker(7): worker waiting for incoming chains
I0611 07:03:01.280677   23609 step_executor.go:513] StepExecutor worker(2): worker coming online
I0611 07:03:01.280708   23609 step_executor.go:513] StepExecutor worker(2): worker waiting for incoming chains
I0611 07:03:01.280706   23609 langruntime_plugin.go:196] langhost[go].Run(pwd=/app/app,root=/app/app, program=/app/app, entryPoint=.,#args=0,proj=production,stack=production_service_pulumi,#config=1,dryrun=true) executing
I0611 07:03:01.280714   23609 step_executor.go:513] StepExecutor worker(3): worker coming online
I0611 07:03:01.280716   23609 step_executor.go:513] StepExecutor worker(3): worker waiting for incoming chains
I0611 07:03:01.280721   23609 step_executor.go:513] StepExecutor worker(4): worker coming online
I0611 07:03:01.280723   23609 step_executor.go:513] StepExecutor worker(4): worker waiting for incoming chains
I0611 07:03:01.280723   23609 rpc.go:78] Marshaling property for RPC[]: pulumi:tags={map[pulumi:template:{go}]}
I0611 07:03:01.280752   23609 rpc.go:78] Marshaling property for RPC[]: pulumi:template={go}
I0611 07:03:01.280842   23609 step_executor.go:513] StepExecutor worker(12): worker coming online
I0611 07:03:01.280852   23609 step_executor.go:513] StepExecutor worker(12): worker waiting for incoming chains
I0611 07:03:01.280845   23609 step_executor.go:513] StepExecutor worker(1): worker coming online
I0611 07:03:01.280855   23609 step_executor.go:513] StepExecutor worker(6): worker coming online
I0611 07:03:01.280862   23609 step_executor.go:513] StepExecutor worker(6): worker waiting for incoming chains
I0611 07:03:01.280863   23609 step_executor.go:513] StepExecutor worker(9): worker coming online
I0611 07:03:01.280868   23609 step_executor.go:513] StepExecutor worker(9): worker waiting for incoming chains
I0611 07:03:01.280858   23609 step_executor.go:513] StepExecutor worker(1): worker waiting for incoming chains
I0611 07:03:01.280874   23609 step_executor.go:513] StepExecutor worker(10): worker coming online
I0611 07:03:01.280876   23609 step_executor.go:513] StepExecutor worker(10): worker waiting for incoming chains
I0611 07:03:01.280877   23609 step_executor.go:513] StepExecutor worker(5): worker coming online
I0611 07:03:01.280880   23609 step_executor.go:513] StepExecutor worker(11): worker coming online
I0611 07:03:01.280732   23609 step_executor.go:513] StepExecutor worker(0): worker coming online
I0611 07:03:01.280889   23609 step_executor.go:513] StepExecutor worker(0): worker waiting for incoming chains
I0611 07:03:01.280890   23609 step_executor.go:513] StepExecutor worker(14): worker coming online
I0611 07:03:01.280895   23609 step_executor.go:513] StepExecutor worker(14): worker waiting for incoming chains
I0611 07:03:01.280896   23609 step_executor.go:513] StepExecutor worker(15): worker coming online
I0611 07:03:01.280948   23609 step_executor.go:513] StepExecutor worker(15): worker waiting for incoming chains
I0611 07:03:01.280881   23609 step_executor.go:513] StepExecutor worker(5): worker waiting for incoming chains
I0611 07:03:01.280884   23609 step_executor.go:513] StepExecutor worker(11): worker waiting for incoming chains
I0611 07:03:01.280901   23609 step_executor.go:513] StepExecutor worker(13): worker coming online
I0611 07:03:01.280965   23609 step_executor.go:513] StepExecutor worker(13): worker waiting for incoming chains
I0611 07:03:01.280667   23609 step_executor.go:513] StepExecutor worker(8): worker coming online
I0611 07:03:01.281157   23609 step_executor.go:513] StepExecutor worker(8): worker waiting for incoming chains

This is where it hangs while consuming more than 10GB of memory


I0611 07:03:01.281157   23609 step_executor.go:513] StepExecutor worker(8): worker waiting for incoming chains
I0611 07:04:13.787545   23609 source_eval.go:866] ResourceMonitor.SupportsFeature(id: resourceReferences) = true
I0611 07:04:13.788380   23609 source_eval.go:866] ResourceMonitor.SupportsFeature(id: outputValues) = true
I0611 07:04:13.789202   23609 source_eval.go:866] ResourceMonitor.SupportsFeature(id: deletedWith) = true
I0611 07:04:13.790264   23609 source_eval.go:866] ResourceMonitor.SupportsFeature(id: aliasSpecs) = true
I0611 07:04:13.791488   23609 source_eval.go:866] ResourceMonitor.SupportsFeature(id: transforms) = true
I0611 07:04:13.792677   23609 source_eval.go:1377] parsing source position &pulumirpc.SourcePosition{state:impl.MessageState{NoUnkeyedLiterals:pragma.NoUnkeyedLiterals{}, DoNotCompare:pragma.DoNotCompare{}, DoNotCopy:pragma.DoNotCopy{}, atomicMessageInfo:(*impl.MessageInfo)(nil)}, sizeCache:0, unknownFields:[]uint8(nil), Uri:"project://%2Fgo%2Fpkg%2Fmod%2Fgithub.com%2Fpulumi%2Fpulumi%2Fsdk%[email protected]%2Fgo%2Fpulumi%2Frun.go", Line:113, Column:0}: parse "project://%2Fgo%2Fpkg%2Fmod%2Fgithub.com%2Fpulumi%2Fpulumi%2Fsdk%[email protected]%2Fgo%2Fpulumi%2Frun.go": invalid URL escape "%2F"
I0611 07:04:13.792835   23609 source_eval.go:1830] ResourceMonitor.RegisterResource received: t=pulumi:pulumi:Stack, name=production-production_service_pulumi, custom=false, #props=0, parent=, protect=false, provider=, deps=[], deleteBeforeReplace=<nil>, ignoreChanges=[], aliases=[], customTimeouts=, providers=map[], replaceOnChanges=[], retainOnDelete=false, deletedWith=
I0611 07:04:13.792882   23609 source_eval.go:193] EvalSourceIterator produced a registration: t=pulumi:pulumi:Stack,name=production-production_service_pulumi,#props=0
I0611 07:04:13.792896   23609 deployment_executor.go:222] deploymentExecutor.Execute(...): incoming event (nil? false, <nil>)
I0611 07:04:13.792909   23609 deployment_executor.go:445] deploymentExecutor.handleSingleEvent(...): received RegisterResourceEvent
I0611 07:04:13.792976   23609 step_generator.go:1123] Planner decided to create 'urn:pulumi:production_service_pulumi::production::pulumi:pulumi:Stack::production-production_service_pulumi' (inputs=map[])
I0611 07:04:13.793163   23609 step_generator.go:286] Checking step create for urn:pulumi:production_service_pulumi::production::pulumi:pulumi:Stack::production-production_service_pulumi
I0611 07:04:13.793203   23609 step_executor.go:513] StepExecutor worker(7): worker received chain for execution
I0611 07:04:13.793382   23609 step_executor.go:513] StepExecutor worker(7): applying step create on urn:pulumi:production_service_pulumi::production::pulumi:pulumi:Stack::production-production_service_pulumi (preview true)
I0611 07:04:13.793431   23609 step_executor.go:513] StepExecutor worker(7): step create on urn:pulumi:production_service_pulumi::production::pulumi:pulumi:Stack::production-production_service_pulumi retired
I0611 07:04:13.793444   23609 step_executor.go:513] StepExecutor worker(7): worker waiting for incoming chains
I0611 07:04:13.793455   23609 source_eval.go:2015] stripping unknowns from RegisterResource response for urn urn:pulumi:production_service_pulumi::production::pulumi:pulumi:Stack::production-production_service_pulumi
I0611 07:04:13.793535   23609 source_eval.go:2062] ResourceMonitor.RegisterResource operation finished: t=pulumi:pulumi:Stack, urn=urn:pulumi:production_service_pulumi::production::pulumi:pulumi:Stack::production-production_service_pulumi, #outs=0
I0611 07:04:13.794665   23609 source_eval.go:2134] ResourceMonitor.RegisterResourceOutputs received: urn=urn:pulumi:production_service_pulumi::production::pulumi:pulumi:Stack::production-production_service_pulumi, #outs=0
I0611 07:04:13.794738   23609 source_eval.go:198] EvalSourceIterator produced a completion: urn=urn:pulumi:production_service_pulumi::production::pulumi:pulumi:Stack::production-production_service_pulumi,#outs=0
I0611 07:04:13.794748   23609 deployment_executor.go:222] deploymentExecutor.Execute(...): incoming event (nil? false, <nil>)
I0611 07:04:13.794761   23609 deployment_executor.go:451] deploymentExecutor.handleSingleEvent(...): received register resource outputs
I0611 07:04:13.794770   23609 source_eval.go:2158] ResourceMonitor.RegisterResourceOutputs operation finished: urn=urn:pulumi:production_service_pulumi::production::pulumi:pulumi:Stack::production-production_service_pulumi, #outs=0
I0611 07:04:13.794372   23609 source_eval.go:1377] parsing source position &pulumirpc.SourcePosition{state:impl.MessageState{NoUnkeyedLiterals:pragma.NoUnkeyedLiterals{}, DoNotCompare:pragma.DoNotCompare{}, DoNotCopy:pragma.DoNotCopy{}, atomicMessageInfo:(*impl.MessageInfo)(nil)}, sizeCache:0, unknownFields:[]uint8(nil), Uri:"project://%2Fgo%2Fpkg%2Fmod%2Fgithub.com%2Fpulumi%2Fpulumi-datadog%2Fsdk%[email protected]%2Fgo%2Fdatadog%2Fmonitor.go", Line:198, Column:0}: parse "project://%2Fgo%2Fpkg%2Fmod%2Fgithub.com%2Fpulumi%2Fpulumi-datadog%2Fsdk%[email protected]%2Fgo%2Fdatadog%2Fmonitor.go": invalid URL escape "%2F"
I0611 07:04:13.794933   23609 rpc.go:293] Unmarshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: escalationMessage={Escalation message @pagerduty}
I0611 07:04:13.794982   23609 rpc.go:293] Unmarshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: includeTags={true}
I0611 07:04:13.794994   23609 rpc.go:293] Unmarshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: message={Monitor triggered. Notify: @hipchat-channel}
I0611 07:04:13.795005   23609 rpc.go:293] Unmarshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: critical={4}
I0611 07:04:13.795016   23609 rpc.go:293] Unmarshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: warning={2}
I0611 07:04:13.795065   23609 rpc.go:293] Unmarshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: monitorThresholds={map[critical:{4} warning:{2}]}
I0611 07:04:13.795083   23609 rpc.go:293] Unmarshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: name={Name for monitor foo}
I0611 07:04:13.795096   23609 rpc.go:293] Unmarshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: query={avg(last_1h):avg:aws.ec2.cpu{environment:foo,host:foo} by {host} > 4}
I0611 07:04:13.795112   23609 rpc.go:293] Unmarshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: tags={[{foo:bar} {team:fooBar}]}
I0611 07:04:13.795122   23609 rpc.go:293] Unmarshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: type={metric alert}
I0611 07:04:13.795138   23609 source_eval.go:785] parseProviderRequest(datadog): semver version is the empty string
I0611 07:04:13.795165   23609 source_eval.go:400] handling default provider request for package datadog
I0611 07:04:13.795702   23609 source_eval.go:321] normalizeProviderRequest(datadog): default version hit on version 4.28.0
I0611 07:04:13.795776   23609 source_eval.go:338] normalizeProviderRequest(datadog-4.28.0): default pluginDownloadURL miss, sending empty string to engine
I0611 07:04:13.795786   23609 source_eval.go:352] normalizeProviderRequest(datadog-4.28.0): default pluginChecksums miss, sending empty map to engine
I0611 07:04:13.795800   23609 source_eval.go:458] checking if providers.ProviderRequest{version:(*semver.Version)(0x4001ddccd0), pkg:"datadog", pluginDownloadURL:"", pluginChecksums:map[string][]uint8(nil)} should be denied
I0611 07:04:13.796082   23609 source_eval.go:498] Did not find a config for 'pulumi'
I0611 07:04:13.796114   23609 source_eval.go:435] waiting for default provider for package datadog-4.28.0
I0611 07:04:13.796207   23609 source_eval.go:193] EvalSourceIterator produced a registration: t=pulumi:providers:datadog,name=default_4_28_0,#props=1
I0611 07:04:13.796214   23609 deployment_executor.go:222] deploymentExecutor.Execute(...): incoming event (nil? false, <nil>)
I0611 07:04:13.796217   23609 deployment_executor.go:445] deploymentExecutor.handleSingleEvent(...): received RegisterResourceEvent
I0611 07:04:13.796234   23609 registry.go:319] ProviderRegistry.Check(urn:pulumi:production_service_pulumi::production::pulumi:providers:datadog::default_4_28_0) executing (#olds=0,#news=1)
I0611 07:04:13.796690   23609 plugins.go:1874] GetPluginPath(resource, datadog, 4.28.0): enabling new plugin behavior
I0611 07:04:13.796786   23609 plugins.go:2008] SelectCompatiblePlugin(..., datadog): beginning
I0611 07:04:13.796806   23609 plugins.go:2041] SelectCompatiblePlugin(..., datadog): skipping plugin datadog-3.3.0: semver mismatch
I0611 07:04:13.796846   23609 plugins.go:2037] SelectCompatiblePlugin(..., datadog): best plugin datadog-4.28.0: semver match
I0611 07:04:13.796880   23609 plugins.go:1883] GetPluginPath(resource, datadog, 4.28.0): found in cache at /root/.pulumi/plugins/resource-datadog-v4.28.0/pulumi-resource-datadog
I0611 07:04:13.796922   23609 plugin.go:186] newPlugin(): Launching plugin 'datadog (resource)' from '/root/.pulumi/plugins/resource-datadog-v4.28.0/pulumi-resource-datadog' with args: 127.0.0.1:43685
I0611 07:04:13.835430   23609 provider_plugin.go:1799] Provider[datadog, 0x4002886700].GetPluginInfo() executing
I0611 07:04:13.836336   23609 provider_plugin.go:1824] Provider[datadog, 0x4002886700].GetPluginInfo() success (#version=4.28.0) success
I0611 07:04:13.836448   23609 provider_plugin.go:376] Provider[datadog, 0x4002886700].CheckConfig(urn:pulumi:production_service_pulumi::production::pulumi:providers:datadog::default_4_28_0) executing (#olds=0,#news=1)
I0611 07:04:13.836636   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].CheckConfig(urn:pulumi:production_service_pulumi::production::pulumi:providers:datadog::default_4_28_0).news]: version={4.28.0}
I0611 07:04:13.838583   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].CheckConfig(urn:pulumi:production_service_pulumi::production::pulumi:providers:datadog::default_4_28_0).inputs]: version={4.28.0}
I0611 07:04:13.838645   23609 provider_plugin.go:435] Provider[datadog, 0x4002886700].CheckConfig(urn:pulumi:production_service_pulumi::production::pulumi:providers:datadog::default_4_28_0) success: inputs=#1 failures=#0
I0611 07:04:13.838667   23609 registry.go:224] setProvider(urn:pulumi:production_service_pulumi::production::pulumi:providers:datadog::default_4_28_0::unconfigured)
I0611 07:04:13.838687   23609 step_generator.go:1123] Planner decided to create 'urn:pulumi:production_service_pulumi::production::pulumi:providers:datadog::default_4_28_0' (inputs=map[version:{4.28.0}])
I0611 07:04:13.838836   23609 step_generator.go:286] Checking step create for urn:pulumi:production_service_pulumi::production::pulumi:providers:datadog::default_4_28_0
I0611 07:04:13.838852   23609 step_executor.go:513] StepExecutor worker(2): worker received chain for execution
I0611 07:04:13.838889   23609 step_executor.go:513] StepExecutor worker(2): applying step create on urn:pulumi:production_service_pulumi::production::pulumi:providers:datadog::default_4_28_0 (preview true)
I0611 07:04:13.839454   23609 registry.go:481] ProviderRegistry.Create(urn:pulumi:production_service_pulumi::production::pulumi:providers:datadog::default_4_28_0) executing (#news=1)
I0611 07:04:13.839615   23609 provider_plugin.go:729] Provider[datadog, 0x4002886700].Configure() executing (#vars=1)
I0611 07:04:13.839625   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].Configure().inputs]: version={4.28.0}
I0611 07:04:13.839633   23609 registry.go:224] setProvider(urn:pulumi:production_service_pulumi::production::pulumi:providers:datadog::default_4_28_0::04da6b54-80e4-46f7-96ec-b56ff0331ba9)
I0611 07:04:13.839859   23609 step_executor.go:513] StepExecutor worker(2): step create on urn:pulumi:production_service_pulumi::production::pulumi:providers:datadog::default_4_28_0 retired
I0611 07:04:13.839978   23609 step_executor.go:513] StepExecutor worker(2): worker waiting for incoming chains
I0611 07:04:13.839986   23609 source_eval.go:444] registered default provider for package datadog-4.28.0: urn:pulumi:production_service_pulumi::production::pulumi:providers:datadog::default_4_28_0
I0611 07:04:13.840017   23609 source_eval.go:1830] ResourceMonitor.RegisterResource received: t=datadog:index/monitor:Monitor, name=foo, custom=true, #props=8, parent=urn:pulumi:production_service_pulumi::production::pulumi:pulumi:Stack::production-production_service_pulumi, protect=false, provider=urn:pulumi:production_service_pulumi::production::pulumi:providers:datadog::default_4_28_0::04da6b54-80e4-46f7-96ec-b56ff0331ba9, deps=[], deleteBeforeReplace=<nil>, ignoreChanges=[], aliases=[], customTimeouts=, providers=map[], replaceOnChanges=[], retainOnDelete=false, deletedWith=
I0611 07:04:13.840036   23609 source_eval.go:193] EvalSourceIterator produced a registration: t=datadog:index/monitor:Monitor,name=foo,#props=8
I0611 07:04:13.840062   23609 deployment_executor.go:222] deploymentExecutor.Execute(...): incoming event (nil? false, <nil>)
I0611 07:04:13.840068   23609 deployment_executor.go:445] deploymentExecutor.handleSingleEvent(...): received RegisterResourceEvent
I0611 07:04:13.840078   23609 registry.go:214] GetProvider(urn:pulumi:production_service_pulumi::production::pulumi:providers:datadog::default_4_28_0::04da6b54-80e4-46f7-96ec-b56ff0331ba9)
I0611 07:04:13.840873   23609 provider_plugin.go:813] Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo) executing (#olds=0,#news=8)
I0611 07:04:13.842757   23609 eventsink.go:59] [INFO] Datadog client successfully initialized, now validating...
I0611 07:04:13.842874   23609 eventsink.go:62] eventSink::Debug(<{%reset%}>[INFO] Datadog client successfully initialized, now validating...<{%reset%}>)
I0611 07:04:14.898913   23609 eventsink.go:59] [INFO] Datadog Client successfully validated.
I0611 07:04:14.899073   23609 eventsink.go:62] eventSink::Debug(<{%reset%}>[INFO] Datadog Client successfully validated.<{%reset%}>)
I0611 07:04:14.900407   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).news]: escalationMessage={Escalation message @pagerduty}
I0611 07:04:14.900561   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).news]: includeTags={true}
I0611 07:04:14.900596   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).news]: message={Monitor triggered. Notify: @hipchat-channel}
I0611 07:04:14.900637   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).news]: monitorThresholds={map[critical:{4} warning:{2}]}
I0611 07:04:14.900662   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).news]: critical={4}
I0611 07:04:14.900687   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).news]: warning={2}
I0611 07:04:14.900713   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).news]: name={Name for monitor foo}
I0611 07:04:14.900780   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).news]: query={avg(last_1h):avg:aws.ec2.cpu{environment:foo,host:foo} by {host} > 4}
I0611 07:04:14.900807   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).news]: tags={[{foo:bar} {team:fooBar}]}
I0611 07:04:14.900831   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).news]: type={metric alert}
I0611 07:04:14.906686   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: __defaults={[{noDataTimeframe} {notifyNoData} {requireFullWindow}]}
I0611 07:04:14.906803   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: escalationMessage={Escalation message @pagerduty}
I0611 07:04:14.906819   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: includeTags={true}
I0611 07:04:14.906825   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: message={Monitor triggered. Notify: @hipchat-channel}
I0611 07:04:14.906836   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: __defaults={[]}
I0611 07:04:14.906844   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: critical={4}
I0611 07:04:14.906850   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: warning={2}
I0611 07:04:14.906865   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: monitorThresholds={map[__defaults:{[]} critical:{4} warning:{2}]}
I0611 07:04:14.906871   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: name={Name for monitor foo}
I0611 07:04:14.906892   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: noDataTimeframe={10}
I0611 07:04:14.906901   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: notifyNoData={false}
I0611 07:04:14.906906   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: query={avg(last_1h):avg:aws.ec2.cpu{environment:foo,host:foo} by {host} > 4}
I0611 07:04:14.906914   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: requireFullWindow={true}
I0611 07:04:14.906923   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: tags={[{foo:bar} {team:fooBar}]}
I0611 07:04:14.907185   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: type={metric alert}
I0611 07:04:14.907210   23609 provider_plugin.go:887] Provider[datadog, 0x4002886700].Check(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo) success: inputs=#12 failures=#0
I0611 07:04:14.907250   23609 step_generator.go:1123] Planner decided to create 'urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo' (inputs=map[__defaults:{[{noDataTimeframe} {notifyNoData} {requireFullWindow}]} escalationMessage:{Escalation message @pagerduty} includeTags:{true} message:{Monitor triggered. Notify: @hipchat-channel} monitorThresholds:{map[__defaults:{[]} critical:{4} warning:{2}]} name:{Name for monitor foo} noDataTimeframe:{10} notifyNoData:{false} query:{avg(last_1h):avg:aws.ec2.cpu{environment:foo,host:foo} by {host} > 4} requireFullWindow:{true} tags:{[{foo:bar} {team:fooBar}]} type:{metric alert}])
I0611 07:04:14.907892   23609 step_generator.go:286] Checking step create for urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo
I0611 07:04:14.908003   23609 step_executor.go:513] StepExecutor worker(3): worker received chain for execution
I0611 07:04:14.908529   23609 step_executor.go:513] StepExecutor worker(3): applying step create on urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo (preview true)
I0611 07:04:14.909061   23609 registry.go:214] GetProvider(urn:pulumi:production_service_pulumi::production::pulumi:providers:datadog::default_4_28_0::04da6b54-80e4-46f7-96ec-b56ff0331ba9)
I0611 07:04:14.909290   23609 provider_plugin.go:1007] Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo) executing (#props=12)
I0611 07:04:14.909434   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: __defaults={[{noDataTimeframe} {notifyNoData} {requireFullWindow}]}
I0611 07:04:14.910025   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: escalationMessage={Escalation message @pagerduty}
I0611 07:04:14.910042   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: includeTags={true}
I0611 07:04:14.910052   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: message={Monitor triggered. Notify: @hipchat-channel}
I0611 07:04:14.910062   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: monitorThresholds={map[__defaults:{[]} critical:{4} warning:{2}]}
I0611 07:04:14.910070   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: __defaults={[]}
I0611 07:04:14.910080   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: critical={4}
I0611 07:04:14.910085   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: warning={2}
I0611 07:04:14.910091   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: name={Name for monitor foo}
I0611 07:04:14.910202   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: noDataTimeframe={10}
I0611 07:04:14.910223   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: notifyNoData={false}
I0611 07:04:14.910230   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: query={avg(last_1h):avg:aws.ec2.cpu{environment:foo,host:foo} by {host} > 4}
I0611 07:04:14.910237   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: requireFullWindow={true}
I0611 07:04:14.910250   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: tags={[{foo:bar} {team:fooBar}]}
I0611 07:04:14.910256   23609 rpc.go:78] Marshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).inputs]: type={metric alert}
I0611 07:04:14.912685   23609 eventsink.go:59] [DEBUG] Ignore the no_data_timeframe change of monitor 'Name for monitor foo' because notify_no_data is false.
I0611 07:04:14.912734   23609 eventsink.go:62] eventSink::Debug(<{%reset%}>[DEBUG] Ignore the no_data_timeframe change of monitor 'Name for monitor foo' because notify_no_data is false.<{%reset%}>)
I0611 07:04:14.913684   23609 eventsink.go:59] 2024/06/11 07:04:14 Error parsing float of old value (): strconv.ParseFloat: parsing "": invalid syntax
I0611 07:04:14.913731   23609 eventsink.go:62] eventSink::Debug(<{%reset%}>2024/06/11 07:04:14 Error parsing float of old value (): strconv.ParseFloat: parsing "": invalid syntax<{%reset%}>)
I0611 07:04:14.914705   23609 eventsink.go:59] 2024/06/11 07:04:14 Error parsing float of old value (): strconv.ParseFloat: parsing "": invalid syntax
I0611 07:04:14.914724   23609 eventsink.go:62] eventSink::Debug(<{%reset%}>2024/06/11 07:04:14 Error parsing float of old value (): strconv.ParseFloat: parsing "": invalid syntax<{%reset%}>)
I0611 07:04:14.918995   23609 eventsink.go:59] [DEBUG] Waiting for state to become: [success]
I0611 07:04:14.919020   23609 eventsink.go:62] eventSink::Debug(<{%reset%}>[DEBUG] Waiting for state to become: [success]<{%reset%}>)
I0611 07:04:15.123285   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).outputs]: escalationMessage={Escalation message @pagerduty}
I0611 07:04:15.123421   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).outputs]: id={}
I0611 07:04:15.123446   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).outputs]: includeTags={true}
I0611 07:04:15.123737   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).outputs]: message={Monitor triggered. Notify: @hipchat-channel}
I0611 07:04:15.123809   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).outputs]: critical={4}
I0611 07:04:15.123824   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).outputs]: criticalRecovery={}
I0611 07:04:15.123839   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).outputs]: ok={}
I0611 07:04:15.123896   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).outputs]: unknown={}
I0611 07:04:15.123904   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).outputs]: warning={2}
I0611 07:04:15.123911   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).outputs]: warningRecovery={}
I0611 07:04:15.123958   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).outputs]: monitorThresholds={map[critical:{4} criticalRecovery:{} ok:{} unknown:{} warning:{2} warningRecovery:{}]}
I0611 07:04:15.123975   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).outputs]: name={Name for monitor foo}
I0611 07:04:15.123994   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).outputs]: newHostDelay={300}
I0611 07:04:15.124020   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).outputs]: notifyNoData={false}
I0611 07:04:15.124035   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).outputs]: query={avg(last_1h):avg:aws.ec2.cpu{environment:foo,host:foo} by {host} > 4}
I0611 07:04:15.124048   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).outputs]: requireFullWindow={true}
I0611 07:04:15.124071   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).outputs]: tags={[{foo:bar} {team:fooBar}]}
I0611 07:04:15.124079   23609 rpc.go:293] Unmarshaling property for RPC[Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo).outputs]: type={metric alert}
I0611 07:04:15.124127   23609 provider_plugin.go:1098] Provider[datadog, 0x4002886700].Create(urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo) success: id=; #outs=12
I0611 07:04:15.125066   23609 step_executor.go:513] StepExecutor worker(3): step create on urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo retired
I0611 07:04:15.125809   23609 step_executor.go:513] StepExecutor worker(3): worker waiting for incoming chains
I0611 07:04:15.126580   23609 source_eval.go:2015] stripping unknowns from RegisterResource response for urn urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo
I0611 07:04:15.127677   23609 source_eval.go:2062] ResourceMonitor.RegisterResource operation finished: t=datadog:index/monitor:Monitor, urn=urn:pulumi:production_service_pulumi::production::datadog:index/monitor:Monitor::foo, #outs=12
I0611 07:04:15.127820   23609 rpc.go:78] Marshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: escalationMessage={Escalation message @pagerduty}
I0611 07:04:15.127842   23609 rpc.go:78] Marshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: id={}
I0611 07:04:15.127850   23609 rpc.go:78] Marshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: includeTags={true}
I0611 07:04:15.127858   23609 rpc.go:78] Marshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: message={Monitor triggered. Notify: @hipchat-channel}
I0611 07:04:15.127879   23609 rpc.go:78] Marshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: monitorThresholds={map[critical:{4} criticalRecovery:{} ok:{} unknown:{} warning:{2} warningRecovery:{}]}
I0611 07:04:15.127887   23609 rpc.go:78] Marshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: critical={4}
I0611 07:04:15.127893   23609 rpc.go:78] Marshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: criticalRecovery={}
I0611 07:04:15.127898   23609 rpc.go:78] Marshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: ok={}
I0611 07:04:15.127903   23609 rpc.go:78] Marshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: unknown={}
I0611 07:04:15.127909   23609 rpc.go:78] Marshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: warning={2}
I0611 07:04:15.128121   23609 rpc.go:78] Marshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: warningRecovery={}
I0611 07:04:15.128309   23609 rpc.go:78] Marshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: name={Name for monitor foo}
I0611 07:04:15.128336   23609 rpc.go:78] Marshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: newHostDelay={300}
I0611 07:04:15.128342   23609 rpc.go:78] Marshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: notifyNoData={false}
I0611 07:04:15.128351   23609 rpc.go:78] Marshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: query={avg(last_1h):avg:aws.ec2.cpu{environment:foo,host:foo} by {host} > 4}
I0611 07:04:15.128361   23609 rpc.go:78] Marshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: requireFullWindow={true}
I0611 07:04:15.128375   23609 rpc.go:78] Marshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: tags={[{foo:bar} {team:fooBar}]}
I0611 07:04:15.128398   23609 rpc.go:78] Marshaling property for RPC[ResourceMonitor.RegisterResource(datadog:index/monitor:Monitor,foo)]: type={metric alert}
I0611 07:04:15.154551   23609 langruntime_plugin.go:242] langhost[go].Run(pwd=/app/app,root=/app/app, program=/app/app, entryPoint=.,...,dryrun=true) success: progerr=, bail=false
I0611 07:04:15.154665   23609 deployment_executor.go:222] deploymentExecutor.Execute(...): incoming event (nil? true, <nil>)
I0611 07:04:15.154687   23609 step_executor.go:513] StepExecutor worker(-1): StepExecutor.waitForCompletion(): waiting for worker threads to exit
I0611 07:04:15.154779   23609 step_executor.go:513] StepExecutor worker(15): worker received nil chain, exiting
I0611 07:04:15.154818   23609 step_executor.go:513] StepExecutor worker(3): worker received nil chain, exiting
I0611 07:04:15.154823   23609 step_executor.go:513] StepExecutor worker(2): worker received nil chain, exiting
I0611 07:04:15.154829   23609 step_executor.go:513] StepExecutor worker(7): worker received nil chain, exiting
I0611 07:04:15.154833   23609 step_executor.go:513] StepExecutor worker(8): worker received nil chain, exiting
I0611 07:04:15.154837   23609 step_executor.go:513] StepExecutor worker(13): worker received nil chain, exiting
I0611 07:04:15.154841   23609 step_executor.go:513] StepExecutor worker(11): worker received nil chain, exiting
I0611 07:04:15.154845   23609 step_executor.go:513] StepExecutor worker(5): worker received nil chain, exiting
I0611 07:04:15.154850   23609 step_executor.go:513] StepExecutor worker(1): worker received nil chain, exiting
I0611 07:04:15.154854   23609 step_executor.go:513] StepExecutor worker(14): worker received nil chain, exiting
I0611 07:04:15.154857   23609 step_executor.go:513] StepExecutor worker(0): worker received nil chain, exiting
I0611 07:04:15.154860   23609 step_executor.go:513] StepExecutor worker(10): worker received nil chain, exiting
I0611 07:04:15.154865   23609 step_executor.go:513] StepExecutor worker(6): worker received nil chain, exiting
I0611 07:04:15.154868   23609 step_executor.go:513] StepExecutor worker(9): worker received nil chain, exiting
I0611 07:04:15.154872   23609 step_executor.go:513] StepExecutor worker(12): worker received nil chain, exiting
I0611 07:04:15.155182   23609 step_executor.go:513] StepExecutor worker(4): worker received nil chain, exiting
I0611 07:04:15.155211   23609 step_executor.go:513] StepExecutor worker(-1): StepExecutor.waitForCompletion(): worker threads all exited
I0611 07:04:15.155236   23609 step_executor.go:513] StepExecutor worker(-1): registered resource outputs urn:pulumi:production_service_pulumi::production::pulumi:pulumi:Stack::production-production_service_pulumi: old=#0, new=#0
I0611 07:04:15.155696   23609 deployment_executor.go:286] deploymentExecutor.Execute(...): step executor has completed
I0611 07:04:15.155814   23609 deployment_executor.go:143] deploymentExecutor.Execute(...): exiting provider canceller
I0611 07:04:15.161259   23609 update.go:211] *** Update(preview=true) complete ***
root@02d2802fe929:/app/app# root@02d2802fe929:/app/app# pulumi preview --logtostderr -v=9 > mac_log.txt
I0611 07:02:56.762808   23609 update.go:206] *** Starting Update(preview=true) ***
I0611 07:02:56.764645   23609 plugins.go:124] gatherPluginsFromProgram(): gathering plugins from language host
I0611 07:02:56.765220   23609 plugins.go:1800] GetPluginPath(language, go, <nil>): found on $PATH /pulumi/bin/pulumi-language-go
I0611 07:02:56.765347   23609 plugins.go:1823] GetPluginPath(language, go, <nil>): found next to current executable /pulumi/bin/pulumi-language-go
I0611 07:02:56.765391   23609 plugin.go:186] newPlugin(): Launching plugin 'go' from '/pulumi/bin/pulumi-language-go' with args: -root=/app/app,127.0.0.1:43685
I0611 07:02:56.811506   23609 langruntime_plugin.go:249] langhost[go].GetPluginInfo() executing
I0611 07:02:56.811942   23609 langruntime_plugin.go:136] langhost[go].GetRequiredPlugins(root=/app/app, program=/app/app, entryPoint=.) executing
I0611 07:02:56.871742   23609 langruntime_plugin.go:186] langhost[go].GetRequiredPlugins(root=/app/app, program=/app/app, entryPoint=.) success: #versions=1
I0611 07:02:56.871795   23609 plugins.go:137] gatherPluginsFromProgram(): plugin go <nil> () is required by language host
I0611 07:02:56.871802   23609 plugins.go:137] gatherPluginsFromProgram(): plugin datadog 4.28.0 () is required by language host
I0611 07:02:56.871806   23609 plugins.go:149] gatherPluginsFromSnapshot(): gathering plugins from snapshot
I0611 07:02:56.871810   23609 plugins.go:193] ensurePluginsAreInstalled(): beginning
I0611 07:02:56.871829   23609 plugins.go:1800] GetPluginPath(language, go, <nil>): found on $PATH /pulumi/bin/pulumi-language-go
I0611 07:02:56.871848   23609 plugins.go:1823] GetPluginPath(language, go, <nil>): found next to current executable /pulumi/bin/pulumi-language-go
I0611 07:02:56.871858   23609 plugins.go:203] ensurePluginsAreInstalled(): plugin go <nil> already installed
I0611 07:02:56.871911   23609 plugins.go:1874] GetPluginPath(resource, datadog, 4.28.0): enabling new plugin behavior
I0611 07:02:56.871929   23609 plugins.go:2008] SelectCompatiblePlugin(..., datadog): beginning
I0611 07:02:56.871934   23609 plugins.go:2041] SelectCompatiblePlugin(..., datadog): skipping plugin datadog-3.3.0: semver mismatch
I0611 07:02:56.871974   23609 plugins.go:229] ensurePluginsAreInstalled(): plugin datadog 4.28.0 not installed, doing install
I0611 07:02:56.871980   23609 plugins.go:248] installPlugin(datadog, 4.28.0): beginning install
I0611 07:02:56.871983   23609 plugins.go:262] installPlugin(datadog, 4.28.0): initiating download
I0611 07:02:56.872041   23609 plugins.go:473] plugin GitHub releases url: https://api.github.com/repos/pulumi/pulumi-datadog/releases/tags/v4.28.0

Affected Resource(s)

every initial pulumi preview run

Output of pulumi about

/app/app# pulumi about
CLI
Version      3.119.0
Go Version   go1.22.3
Go Compiler  gc

Plugins
KIND      NAME     VERSION
resource  datadog  4.28.0
language  go       unknown

Host
OS       debian
Version  11.9
Arch     aarch64

This project is written in go: executable='/usr/local/bin/go' version='go version go1.22.4 linux/arm64'

Current Stack: organization/production/production_service_pulumi

Found no resources associated with production_service_pulumi

Found no pending operations associated with production_service_pulumi

Backend
Name           02d2802fe929
URL            s3://company-pulumi
User           root
Organizations
Token type     personal

Dependencies:
NAME                                     VERSION
github.com/pulumi/pulumi-datadog/sdk/v4  v4.28.0
github.com/pulumi/pulumi/sdk/v3          v3.119.0

Pulumi locates its logs in /tmp by default

Additional context

Im trying to use this SDK in CI on ephemeral containers,
and i don't want to allocate such a high amount of memory just for the first run.

This happens with all Datadog SDK resources
And not on other pulumi SDKs

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).
@yishait-tailor yishait-tailor added kind/bug Some behavior is incorrect or out of spec needs-triage Needs attention from the triage team labels Jun 11, 2024
@yishait-tailor yishait-tailor changed the title initial pulumi preview on datadog.Monitor SDK consumes more than 10GB of memory on V4 and 4GB on V3 initial pulumi preview (Go) on datadog.Monitor SDK consumes more than 10GB of memory on V4 and 4GB on V3 Jun 11, 2024
@yishait-tailor yishait-tailor changed the title initial pulumi preview (Go) on datadog.Monitor SDK consumes more than 10GB of memory on V4 and 4GB on V3 initial pulumi preview (Go) on datadog SDK consumes more than 10GB of memory on V4 and 4GB on V3 Jun 11, 2024
@iwahbe
Copy link
Member

iwahbe commented Jun 11, 2024

Hey @yishait-tailor. Thanks for reporting this bug. The provider definitely shouldn't use that much memory. Just to clarify, what do you mean by the "first run"?

  • Are you saying that when you create a new stack with datadog, the first pulumi up uses 10gb
  • Are you saying that the initial run of pulumi up in each container takes 10gb?

Can you check how much memory go build uses to build your Pulumi program?

@iwahbe iwahbe added awaiting-feedback Blocked on input from the author and removed needs-triage Needs attention from the triage team labels Jun 11, 2024
@yishait-tailor
Copy link
Author

Hey @iwahbe!
thanks for the quick response,

by the looks of it, running go build is what causes this memory consumption
first run = if go build has not run yet on the container it will cause the issue,

Are you saying that when you create a new stack with datadog, the first pulumi up uses 10gb

  • pulumi up or pulumi preview - since they run go build in the bg, it causes the issue,

Are you saying that the initial run of pulumi up in each container takes 10gb?

  • can you clarify what you mean by "each" container?

the workaround is to create the docker image with the go binary file produced by running go build
and changing the Pulumi.yaml to use the binary,
we would prefer to avoid doing that...

runtime:
  name: go
  options:
    binary: go-binary-file

Thanks!

@pulumi-bot pulumi-bot added needs-triage Needs attention from the triage team and removed awaiting-feedback Blocked on input from the author labels Jun 13, 2024
@iwahbe
Copy link
Member

iwahbe commented Jun 14, 2024

Hey @yishait-tailor. That clarifies the problem nicely. The problem is that the go SDK for pulumi-datadog is simply too big.

I'll take a look at reducing the number of types we export.

This is related to pulumi/pulumi-terraform-bridge#1468.

@iwahbe iwahbe self-assigned this Jun 14, 2024
@iwahbe iwahbe removed the needs-triage Needs attention from the triage team label Jun 14, 2024
@iwahbe iwahbe mentioned this issue Jun 14, 2024
Merged
@iwahbe iwahbe closed this as completed in 43836a5 Jun 23, 2024
@pulumi-bot pulumi-bot added resolution/fixed This issue was fixed labels Jun 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@iwahbe iwahbe

Labels
kind/bug Some behavior is incorrect or out of spec resolution/fixed This issue was fixed
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Reduce recursive types pulumi/pulumi-datadog
3 participants
@iwahbe @pulumi-bot @yishait-tailor