agent: support discovery mode on linux (#1108)

* agent: support discovery mode on linux

Co-authored-by: jvoravong <[email protected]>

.chloggen/discoverymode.yaml

@@ -0,0 +1,12 @@
+# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
+change_type: enhancement
+# The name of the component, or a single word describing the area of concern, (e.g. agent, clusterReceiver, gateway, operator, chart, other)
+component: agent
+# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
+note: 'linux: Adopt discovery mode in agent and provide agent.discovery.properties value mapping'
+# One or more tracking issues related to the change
+issues: [1108]
+# (Optional) One or more lines of additional information to render under the primary note.
+# These lines will be padded with 2 spaces and then inserted directly into the document.
+# Use pipe (|) for multiline entries.
+subtext:

docs/advanced-configuration.md

@@ -133,6 +133,57 @@ clusterReceiver:
   enabled: false
 ```
 
+## Discovery Mode
+
+*At this time only Linux installations are supported*
+
+The agent daemonset can be configured to run in [`--discovery` mode](https://docs.splunk.com/observability/en/gdi/opentelemetry/discovery-mode.html), initiating a "preflight" discovery phase to test bundled metric receiver configurations against `k8s_observer` discovered endpoints. Successfully discovered instances are then incorporated in the existing service config.
+
+You can additionally configure any necessary discovery properties for providing required auth or service-specific information:
+
+```yaml
+agent:
+  discovery:
+    enabled: true # disabled by default
+    properties:
+      extensions:
+        k8s_observer:
+          config:
+            auth_type: serviceAccount  # default auth_type value
+      receivers:
+        postgres:
+          config:
+            # auth fields reference environment variables populated by secret data below
+            username: '${env:POSTGRES_USER}'
+            password: '${env:POSTGRES_PASSWORD}'
+            tls:
+              insecure: true
+
+  extraEnvs:
+    # environment variables using a manually created "postgres-monitoring" secret
+    - name: POSTGRES_USER
+      valueFrom:
+        secretKeyRef:
+          name: postgres-monitoring
+          key: username
+    - name: POSTGRES_PASSWORD
+      valueFrom:
+        secretKeyRef:
+          name: postgres-monitoring
+          key: password
+```
+
+For discovery progress and statement evaluations, see the agent startup logs in product or via kubectl.
+
+```bash
+$ kubectl -n monitoring logs splunk-otel-collector-agent | grep -i disco
+Discovering for next 10s...
+Successfully discovered "postgresql" using "k8s_observer" endpoint "k8s_observer/e8a10f52-4f2a-468c-be7b-7f3c673b1c8e/(5432)".
+Discovery complete.
+```
+
+By default, the `docker_observer` and `host_observer` extensions are disabled for discovery in the helm chart.
+
 ## GKE Autopilot support
 
 If you want to run Splunk OTel Collector in [Google Kubernetes Engine

examples/discovery-mode/README.md

@@ -0,0 +1,3 @@
+# Discovery mode example
+
+This example configures the agent to run in --discovery mode using redis receiver discovery properties provided as values.

examples/discovery-mode/discovery-mode-values.yaml

@@ -0,0 +1,29 @@
+clusterName: CHANGEME
+splunkObservability:
+  realm: CHANGEME
+  accessToken: CHANGEME
+agent:
+  discovery:
+    enabled: true
+    properties:
+      receivers:
+        redis:
+          config:
+            password: '${env:REDIS_PASSWORD}'
+            username: '${env:REDIS_USERNAME}'
+  # these env vars reference a secret that must be created manually
+  extraEnvs:
+    - name: REDIS_USERNAME
+      valueFrom:
+        secretKeyRef:
+          name: redis-secret
+          key: username
+    - name: REDIS_PASSWORD
+      valueFrom:
+        secretKeyRef:
+          name: redis-secret
+          key: password
+
+clusterReceiver:
+  # disabled for example only. unrelated to discovery mode
+  enabled: false

examples/discovery-mode/rendered_manifests/clusterRole.yaml

@@ -0,0 +1,83 @@
+---
+# Source: splunk-otel-collector/templates/clusterRole.yaml
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: default-splunk-otel-collector
+  labels:
+    app.kubernetes.io/name: splunk-otel-collector
+    helm.sh/chart: splunk-otel-collector-0.91.0
+    app.kubernetes.io/managed-by: Helm
+    app.kubernetes.io/instance: default
+    app.kubernetes.io/version: "0.91.1"
+    app: splunk-otel-collector
+    chart: splunk-otel-collector-0.91.0
+    release: default
+    heritage: Helm
+rules:
+- apiGroups:
+  - ""
+  resources:
+  - events
+  - namespaces
+  - namespaces/status
+  - nodes
+  - nodes/spec
+  - nodes/stats
+  - nodes/proxy
+  - pods
+  - pods/status
+  - persistentvolumeclaims
+  - persistentvolumes
+  - replicationcontrollers
+  - replicationcontrollers/status
+  - resourcequotas
+  - services
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - apps
+  resources:
+  - daemonsets
+  - deployments
+  - replicasets
+  - statefulsets
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - extensions
+  resources:
+  - daemonsets
+  - deployments
+  - replicasets
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - batch
+  resources:
+  - jobs
+  - cronjobs
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+    - autoscaling
+  resources:
+    - horizontalpodautoscalers
+  verbs:
+    - get
+    - list
+    - watch
+- nonResourceURLs:
+  - /metrics
+  verbs:
+  - get
+  - list
+  - watch

examples/discovery-mode/rendered_manifests/clusterRoleBinding.yaml

@@ -0,0 +1,24 @@
+---
+# Source: splunk-otel-collector/templates/clusterRoleBinding.yaml
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: default-splunk-otel-collector
+  labels:
+    app.kubernetes.io/name: splunk-otel-collector
+    helm.sh/chart: splunk-otel-collector-0.91.0
+    app.kubernetes.io/managed-by: Helm
+    app.kubernetes.io/instance: default
+    app.kubernetes.io/version: "0.91.1"
+    app: splunk-otel-collector
+    chart: splunk-otel-collector-0.91.0
+    release: default
+    heritage: Helm
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: default-splunk-otel-collector
+subjects:
+- kind: ServiceAccount
+  name: default-splunk-otel-collector
+  namespace: default