googlecloudmonitoringreceiver gives no valid credentials found Error

[HazemAbdelmagid]

Component(s)

receiver/googlecloudmonitoring

What happened?

Description

I have been trying to integrate OTEL with googlecloudmonitoring receiver to collect metrics related to memorystore and then forward them to prometheus which exists in a different cluster but it is not working.
When I try to add the googlecloudmonitoring receiver configurations and give the service account of the collector access for collecting those metrics, it doesn't work and gives this error messges:

2024-12-01T16:28:37.861Z        error   graph/graph.go:426      Failed to start component       {"error": "no valid credentials found", "type": "Receiver", "id": "googlecloudmonitoring"}
2024-12-01T16:28:37.861Z        info    [email protected]/service.go:266 Starting shutdown...
2024-12-01T16:28:37.861Z        info    extensions/extensions.go:66     Stopping extensions...
2024-12-01T16:28:37.861Z        info    [email protected]/service.go:280 Shutdown complete.
Error: cannot start pipelines: no valid credentials found
2024/12/01 16:28:37 collector server run finished with error: cannot start pipelines: no valid credentials found

Steps to Reproduce

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: collector-with-ta
  namespace: opentelemetry-operator-system
spec:
  config:
    exporters:
      debug:
        verbosity: detailed
      googlecloudpubsub:
        project: <redacted>
        topic: projects/<redacted>/topics/metrics-pubsub
    processors:
      batch:
        timeout: 5s
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 10
    receivers:
      googlecloudmonitoring:
        collection_interval: 5m
        project_id: <redacted>
        metrics_list:
        - metric_name: "redis.googleapis.com/instance/stats/cpu_utilization"
    service:
      pipelines:
        metrics:
          exporters:
          - googlecloudpubsub
          - debug
          processors:
          - batch
          - memory_limiter
          receivers:
          - googlecloudmonitoring
      telemetry:
        metrics:
          address: 0.0.0.0:8888
  image: otel/opentelemetry-collector-contrib:0.112.0
  mode: statefulset
  replicas: 1
  serviceAccount: opentelemetry-collector-sa
---
apiVersion: v1
  kind: ServiceAccount
  metadata:
    annotations:
      iam.gke.io/gcp-service-account: opentelemetry-collector@<redacted>.iam.gserviceaccount.com
    name: opentelemetry-collector-sa
    namespace: opentelemetry-operator-system

gcloud projects get-iam-policy <redacted> --flatten="bindings[].members" --filter="bindings.members:serviceAccount:opentelemetry-collector@<redacted>.iam.gserviceaccount.com" --format="table(bindings.role)"
ROLE: roles/monitoring.viewer

Expected Result

OTEL collector to get the listed metrics.

Actual Result

error   graph/graph.go:426      Failed to start component       {"error": "no valid credentials found", "type": "Receiver", "id": "googlecloudmonitoring"}
2024-12-01T16:44:25.863Z        info    [email protected]/service.go:266 Starting shutdown...
2024-12-01T16:44:25.863Z        info    extensions/extensions.go:66     Stopping extensions...
2024-12-01T16:44:25.863Z        info    [email protected]/service.go:280 Shutdown complete.
Error: cannot start pipelines: no valid credentials found
2024/12/01 16:44:25 collector server run finished with error: cannot start pipelines: no valid credentials found

Collector version

v0.112.0

Environment information

Environment

GKE Cluster

OpenTelemetry Collector configuration

config:
    exporters:
      debug:
        verbosity: detailed
      googlecloudpubsub:
        project: <redacted>
        topic: projects/<redacted>/topics/metrics-pubsub
    processors:
      batch:
        timeout: 5s
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 10
    receivers:
      googlecloudmonitoring:
        collection_interval: 5m
        project_id: <redacted>
        metrics_list:
        - metric_name: "redis.googleapis.com/instance/stats/cpu_utilization"
    service:
      pipelines:
        metrics:
          exporters:
          - googlecloudpubsub
          - debug
          processors:
          - batch
          - memory_limiter
          receivers:
          - googlecloudmonitoring

Log output

error   graph/graph.go:426      Failed to start component       {"error": "no valid credentials found", "type": "Receiver", "id": "googlecloudmonitoring"}
2024-12-01T16:44:25.863Z        info    [email protected]/service.go:266 Starting shutdown...
2024-12-01T16:44:25.863Z        info    extensions/extensions.go:66     Stopping extensions...
2024-12-01T16:44:25.863Z        info    [email protected]/service.go:280 Shutdown complete.
Error: cannot start pipelines: no valid credentials found
2024/12/01 16:44:25 collector server run finished with error: cannot start pipelines: no valid credentials found

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/googlecloudmonitoring: @dashpole @TylerHelmuth @abhishek-at-cloudwerx

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[VihasMakwana]

@HazemAbdelmagid do you have GOOGLE_APPLICATION_CREDENTIALS environment variable set pointing to the credentials file?

[VihasMakwana]

For more details, see README.md and https://cloud.google.com/docs/authentication/application-default-credentials.

[HazemAbdelmagid]

@VihasMakwana why do I need to use GOOGLE_APPLICATION_CREDENTIALS since I am using a service account on K8s which has annotation for gcp service account that has been assigned monitoring.viewer role, and it's mentioned in the README.md file for the receiver the following:

Note that if your workload is running on Google Cloud Platform (GCP), the service account credentials will be used automatically without needing to set the environment variable manually.

[VihasMakwana]

Right. In that case, you don't need an env.
Here's the offending code:

opentelemetry-collector-contrib/receiver/googlecloudmonitoringreceiver/receiver.go

Lines 158 to 164 in bc7d967

	creds, err := google.FindDefaultCredentials(ctx, "https://www.googleapis.com/auth/monitoring.read")
	if err != nil {
	return fmt.Errorf("failed to find default credentials: %w", err)
	}
	if creds == nil || creds.JSON == nil {
	return errors.New("no valid credentials found")
	}

We're exiting on creds.JSON == nil but from the documentation,

// JSON contains the raw bytes from a JSON credentials file.
// This field may be nil if authentication is provided by the
// environment and not with a credentials file, e.g. when code is
// running on Google Cloud Platform.

So, maybe we don't need to check for creds.JSON?
cc: @dashpole

[dashpole]

Yes, we definitely should not be checking creds.JSON: https://pkg.go.dev/golang.org/x/oauth2/google#Credentials

 	// JSON contains the raw bytes from a JSON credentials file.
	// This field may be nil if authentication is provided by the
	// environment and not with a credentials file, e.g. when code is
	// running on Google Cloud Platform.
        JSON [][byte](https://pkg.go.dev/builtin#byte)

Looking at the source, it also isn't necessary to check if creds is nil, since it will always be non-nil if no error was returned: https://cs.opensource.google/go/x/oauth2/+/master:google/default.go;l=194;drc=d0e617c58cf747cf27df9762003502f814dd524c

[VihasMakwana]

I can file a fix for it, unless you're already working on it. Let me know!

[dashpole]

Go for it @VihasMakwana