Does k8sattributesprocessor works with prometheus receiver?

[gabrielgiussi]

The k8sattributesprocessor docs says

When running as an agent, the processor detects IP addresses of pods sending spans, metrics or logs to the agent and uses this information to extract metadata from pods.

But the prometheus metrics receiver is pull based and since I'm running the collector as an agent I configured ip 0.0.0.0

      prometheus:
        config:
          scrape_configs:
            - job_name: 'otel-collector'
              scrape_interval: 5s
              metrics_path: /stats/prometheus
              static_configs:
                - targets: ['0.0.0.0:9901']

So I'm thinking that even in the case the processor is also using the IP for a pull based receiver in this case it won't find anything querying k8s api with IP 0.0.0.0 (although I can't say if that's why I'm not seeing any label being added to the metrics or something else is happening).
I could configure the collector with an environment variable and fill that using the Downward API but I've already tried with status.podIP but didn't work, it failed saying it could scrap from that ip address.

My configuration is the following

apiVersion: v1
kind: ServiceAccount
metadata:
  name: otel-collector-service-account
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-agent
rules:
  - apiGroups: [""]
    resources: ["pods", "namespaces"]
    verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: otel-agent
subjects:
  - kind: ServiceAccount
    name: otel-collector-service-account
    namespace: default
roleRef:
  kind: ClusterRole
  name: otel-agent
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: collector-config
data:
  collector.yaml: |
    receivers:
      otlp:
        protocols: 
          grpc:
    processors:
    exporters:
      logging:
      prometheus:
        endpoint: "0.0.0.0:4318"
        namespace: test-space
        send_timestamps: true
        metric_expiration: 180m
        enable_open_metrics: true
        resource_to_telemetry_conversion:
          enabled: true
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [logging]
        metrics:
          receivers: [otlp]
          processors: []
          exporters: [prometheus]
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: agent-config
data:
  agent.yaml: |
    receivers:
      otlp:
        protocols: 
          grpc:
      prometheus:
        config:
          scrape_configs:
            - job_name: 'otel-collector'
              scrape_interval: 5s
              metrics_path: /stats/prometheus
              static_configs:
                - targets: ['${KUBE_POD_IP}:9901'] # it works with 0.0.0.0:9901 but no labels are being attached to metrics
    processors:
      k8sattributes:
        auth_type: "serviceAccount"
        passthrough: false
        filter:
          node_from_env_var: KUBE_NODE_NAME
        extract:
          metadata:
            - k8s.pod.name
            - k8s.pod.uid
            - k8s.deployment.name
            - k8s.cluster.name
            - k8s.namespace.name
            - k8s.node.name
            - k8s.pod.start_time
        pod_association:
          - from: resource_attribute
            name: k8s.pod.uid     
    exporters:
      otlp:
        endpoint: "opentelemetrycollector.default.svc.cluster.local:4317"
        tls:
          insecure: true
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [otlp]
        metrics:
          receivers: [prometheus]
          processors: [k8sattributes]
          exporters: [otlp]
---
apiVersion: v1
kind: Service
metadata:
  name: opentelemetrycollector
spec:
  ports:
    - name: grpc-otlp
      port: 4317
      protocol: TCP
      targetPort: 4317
    - name: prometheus
      port: 4318
      protocol: TCP
      targetPort: 4318
  selector:
    app.kubernetes.io/name: opentelemetrycollector
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: opentelemetrycollector
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: opentelemetrycollector
  template:
    metadata:
      labels:
        app.kubernetes.io/name: opentelemetrycollector
    spec:
      containers:
        - name: otelcol
          args:
            - --config=/conf/collector.yaml
          image: otel/opentelemetry-collector:0.61.0
          volumeMounts:
            - mountPath: /conf
              name: collector-config
      volumes:
        - configMap:
            items:
              - key: collector.yaml
                path: collector.yaml
            name: collector-config
          name: collector-config
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gabi
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      serviceAccountName: otel-collector-service-account
      containers:
        - name: nginx
          image: nginx:1.14.2
          ports:
            - containerPort: 80
        - name: envoy
          image: envoyproxy/envoy-dev:d00e4536195ec03b305a7411409910c7101aea25
          command: ["/bin/sh", "-c", "apt-get update && apt-get install -y wget && wget -O /tmp/envoy.yaml https://gist.githubusercontent.com/ggiussi/406229725fbd929a353d047fa4a6b513/raw/92462ab816145f5926c3765f6f7923915c53694d/envoy.yaml && envoy -c /tmp/envoy.yaml"]
          securityContext:
            allowPrivilegeEscalation: false
            runAsUser: 0
        - name: agent
          image: otel/opentelemetry-collector-contrib:0.61.0
          args:
          - --config=/conf/agent.yaml
          volumeMounts:
          - mountPath: /conf
            name: agent-config
          env:
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            - name: KUBE_POD_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.podIP
      volumes:
        - name: agent-config
          configMap:
            items:
            - key: agent.yaml
              path: agent.yaml
            name: agent-config

[jtama]

As your agent acts as a side car, he will share your ngxin pod IP. He can then export you metrics to your opentelemetrycollector (using otlp). Your collector will then be able to detect IP and k8sattributesprocessor will work as expected. Just move your k8sattributesprocessor configuration from agent to opentelemetrycollector

[gabrielgiussi]

Thanks @jtama .
I still don't know if we are going to deploy both as an agent and as a gateway, having only the agent seems enough for now so I wonder what can be done in that case.
In the meanwhile I'm using the more general metricstransform processor in combination with the k8s downward api to inject the pod name and other pod metadata, which also seems enough and simpler than the k8sattributesprocessor which also requires a ClusterRole to access the k8s api.