AKS - splunk-otel-collector-agent failed to start on AKS cluster

[aqilbeig]

What happened?

Description

splunk-otel-collector-agent failed to start on AKS cluster

Steps to Reproduce

• Install the collector using helm install my-splunk-otel-collector --set="splunkPlatform.endpoint=https://127.0.0.1:8088/services/collector,splunkPlatform.token=xxxxxx,splunkPlatform.metricsIndex=k8s-metrics,splunkPlatform.index=main,clusterName=my-cluster" splunk-otel-collector-chart/splunk-otel-collector
• Collector agent pods went into CrashLoopBackoff with below error:
  Error: failed to start extensions: failed to bind to address 0.0.0.0:13133: listen tcp 0.0.0.0:13133: bind: address already in use; failed to shutdown pipelines: no existing monitoring routine is running
  2023/04/19 09:07:39 main.go:103: application run finished with error: failed to start extensions: failed to bind to address 0.0.0.0:13133: listen tcp 0.0.0.0:13133: bind: address already in use; failed to shutdown pipelines: no existing monitoring routine is running

Expected Result

Actual Result

Chart version

v0.4.0

Environment information

Environment

Cloud: (e.g., "EKS", "AKS", "GKE") : AKS
k8s version: (e.g., 1.21.0) : 1.24.9
OS: (e.g., "Ubuntu 20.04") : Ubuntu 18 LTS

Chart configuration

No response

Log output

2023/04/19 09:07:39 settings.go:331: Set config to [/conf/relay.yaml]
2023/04/19 09:07:39 settings.go:384: Set ballast to 165 MiB
2023/04/19 09:07:39 settings.go:400: Set memory limit to 450 MiB
2023-04-19T09:07:39.865Z	info	service/telemetry.go:90	Setting up own telemetry...
2023-04-19T09:07:39.866Z	info	service/telemetry.go:116	Serving Prometheus metrics	{"address": "0.0.0.0:8889", "level": "Basic"}
2023-04-19T09:07:39.867Z	info	kube/client.go:101	k8s filtering	{"kind": "processor", "name": "k8sattributes", "pipeline": "logs", "labelSelector": "", "fieldSelector": "spec.nodeName=aks-system3-16418380-vmss00003p"}
2023-04-19T09:07:39.867Z	info	[email protected]/memorylimiter.go:113	Memory limiter configured	{"kind": "processor", "name": "memory_limiter", "pipeline": "logs", "limit_mib": 450, "spike_limit_mib": 90, "check_interval": 2}
2023-04-19T09:07:39.868Z	info	service/service.go:129	Starting otelcol...	{"Version": "v0.75.0", "NumCPU": 16}
2023-04-19T09:07:39.868Z	info	extensions/extensions.go:41	Starting extensions...
2023-04-19T09:07:39.868Z	info	extensions/extensions.go:44	Extension is starting...	{"kind": "extension", "name": "health_check"}
2023-04-19T09:07:39.868Z	info	[email protected]/healthcheckextension.go:45	Starting health_check extension	{"kind": "extension", "name": "health_check", "config": {"Endpoint":"0.0.0.0:13133","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"Path":"/","ResponseBody":null,"CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
2023-04-19T09:07:39.868Z	info	service/service.go:155	Starting shutdown...
2023-04-19T09:07:39.868Z	info	healthcheck/handler.go:129	Health Check state change	{"kind": "extension", "name": "health_check", "status": "unavailable"}
2023-04-19T09:07:39.868Z	info	extensions/extensions.go:55	Stopping extensions...
2023-04-19T09:07:39.868Z	info	[email protected]/zpagesextension.go:109	Unregistered zPages span processor on tracer provider	{"kind": "extension", "name": "zpages"}
2023-04-19T09:07:39.868Z	info	service/service.go:169	Shutdown complete.
Error: failed to start extensions: failed to bind to address 0.0.0.0:13133: listen tcp 0.0.0.0:13133: bind: address already in use; failed to shutdown pipelines: no existing monitoring routine is running
2023/04/19 09:07:39 main.go:103: application run finished with error: failed to start extensions: failed to bind to address 0.0.0.0:13133: listen tcp 0.0.0.0:13133: bind: address already in use; failed to shutdown pipelines: no existing monitoring routine is running

No response

Additional context

No response

[jvoravong]

Hey @aqilbeig,
Are you by chance installing multiple instances of the chart in your cluster? If so, check out #572.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[atoulme]

Thanks @jvoravong @aqilbeig please let us know if you're still hitting this issue. I will close this issue soon as inactive.