[8.16](backport #5822) docs: add EDOT colletor kube-stack Helm values (…

…#6265)

* docs: add EDOT colletor kube-stack Helm values (#5822)

* docs: add EDOT colletor kube-stack Helm values

* chore: add changelog entry

* Update deploy/helm/edot-collector/kube-stack/README.md

Co-authored-by: Andrew Gizas <[email protected]>

* Update deploy/helm/edot-collector/kube-stack/README.md

Co-authored-by: Andrew Gizas <[email protected]>

* Update deploy/helm/edot-collector/kube-stack/README.md

Co-authored-by: Andrew Gizas <[email protected]>

* Update deploy/helm/edot-collector/kube-stack/README.md

Co-authored-by: Andrew Gizas <[email protected]>

* Update deploy/helm/edot-collector/kube-stack/README.md

Co-authored-by: Andrew Gizas <[email protected]>

* Update deploy/helm/edot-collector/kube-stack/README.md

Co-authored-by: Andrew Gizas <[email protected]>

* Update deploy/helm/edot-collector/kube-stack/README.md

Co-authored-by: Andrew Gizas <[email protected]>

* fix: values path relative path

* add kube-stack Helm integration test

* fix: relative values file

* ci: load values file with Helm Go package

* ci: remove agent's hardcoded image

* Update deploy/helm/edot-collector/kube-stack/README.md

Co-authored-by: Nathan L Smith <[email protected]>

* Update deploy/helm/edot-collector/kube-stack/README.md

Co-authored-by: Nathan L Smith <[email protected]>

* Update deploy/helm/edot-collector/kube-stack/README.md

Co-authored-by: Nathan L Smith <[email protected]>

* Update deploy/helm/edot-collector/kube-stack/README.md

Co-authored-by: Nathan L Smith <[email protected]>

* Update deploy/helm/edot-collector/kube-stack/README.md

Co-authored-by: Nathan L Smith <[email protected]>

* Update deploy/helm/edot-collector/kube-stack/README.md

Co-authored-by: Nathan L Smith <[email protected]>

* Update deploy/helm/edot-collector/kube-stack/README.md

Co-authored-by: Nathan L Smith <[email protected]>

* Update deploy/helm/edot-collector/kube-stack/README.md

Co-authored-by: Nathan L Smith <[email protected]>

* add kube-stack values file to Update mage target

* override elastic secrets

* fix: parametrize helm release name

* ci: eventually check created pods

* test 5min timeout

* Update deploy/helm/edot-collector/kube-stack/values.yaml

Co-authored-by: Edu González de la Herrán <[email protected]>

* Update deploy/helm/edot-collector/kube-stack/README.md

Co-authored-by: Edu González de la Herrán <[email protected]>

* Update deploy/helm/edot-collector/kube-stack/values.yaml

Co-authored-by: Edu González de la Herrán <[email protected]>

* fix: use constant for Helm values paths

* Update testing/integration/otel_helm_test.go

Co-authored-by: Mikołaj Świątek <[email protected]>

* fix: reference kube-stack Chart variables

* assert all pods are running

* chore: clarify pod description comment

---------

Co-authored-by: Andrew Gizas <[email protected]>
Co-authored-by: Nathan L Smith <[email protected]>
Co-authored-by: Edu González de la Herrán <[email protected]>
Co-authored-by: Mikołaj Świątek <[email protected]>
(cherry picked from commit 69c2f92)

# Conflicts:
#	magefile.go

* fix: pin elastic 8.16 version

---------

Co-authored-by: Roger Coll <[email protected]>

changelog/fragments/1729235281-kube-stack-helm-chart.yaml

@@ -0,0 +1,32 @@
+# Kind can be one of:
+# - breaking-change: a change to previously-documented behavior
+# - deprecation: functionality that is being removed in a later release
+# - bug-fix: fixes a problem in a previous version
+# - enhancement: extends functionality but does not break or fix existing behavior
+# - feature: new functionality
+# - known-issue: problems that we are aware of in a given version
+# - security: impacts on the security of a product or a user’s deployment.
+# - upgrade: important information for someone upgrading from a prior version
+# - other: does not fit into any of the other categories
+kind: other
+
+# Change summary; a 80ish characters long description of the change.
+summary: add EDOT colletor kube-stack Helm values
+
+# Long description; in case the summary is not enough to describe the change
+# this field accommodate a description without length limits.
+# NOTE: This field will be rendered only for breaking-change and known-issue kinds at the moment.
+#description:
+
+# Affected component; usually one of "elastic-agent", "fleet-server", "filebeat", "metricbeat", "auditbeat", "all", etc.
+component: "elastic-agent"
+
+# PR URL; optional; the PR number that added the changeset.
+# If not present is automatically filled by the tooling finding the PR where this changelog fragment has been added.
+# NOTE: the tooling supports backports, so it's able to fill the original PR number instead of the backport PR number.
+# Please provide it if you are adding a fragment for a different PR.
+#pr: https://github.com/owner/repo/1234
+
+# Issue URL; optional; the GitHub issue related to this changeset (either closes or is part of).
+# If not present is automatically filled by the tooling with the issue linked to the PR number.
+#issue: https://github.com/owner/repo/1234

deploy/helm/edot-collector/kube-stack/README.md

@@ -0,0 +1,66 @@
+## Kube-stack Helm Chart
+
+**More detailed documentation can be found [here](https://github.com/elastic/opentelemetry/blob/main/docs/kubernetes/operator/README.md).**
+
+The [kube-stack Helm Chart](https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-kube-stack#readme) is used to manage the installation of the OpenTelemetry operator (including its CRDs) and to configure a suite of EDOT collectors, which instrument various Kubernetes components to enable comprehensive observability and monitoring.
+
+The chart is installed with a provided default [`values.yaml`](./values.yaml) file that can be customized when needed.
+
+### DaemonSet collectors
+
+The OpenTelemetry components deployed within the DaemonSet EDOT collectors are responsible for observing specific signals from each node. To ensure complete data collection, these components must be deployed on every node in the cluster. Failing to do so will result in partial and potentially incomplete data.
+
+The DaemonSet collectors handle the following data:
+
+- Host Metrics: Collects host metrics specific to each node, utilizing the [hostmetrics receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/hostmetricsreceiver/README.md)
+- Kubernetes Metrics: Captures metrics related to the Kubernetes infrastructure on each node, utlilizing [kubeletstats](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/kubeletstatsreceiver/README.md) receiver
+- Logs: Utilizes [File Log Receiver receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver#readme) to gather logs from all Pods running on the respective node.
+- OTLP Traces: Utilizes [OTLP Receiver]( https://github.com/open-telemetry/opentelemetry-collector/blob/main/receiver/otlpreceiver#readme) which configures both HTTP and GRPC endpoints on the node to receive OTLP trace data.
+
+### Deployment collector
+
+The OpenTelemetry components deployed within a Deployment collector focus on gathering data at the cluster level rather than at individual nodes.  A Deployment instance of the collector operates as a standalone (unlike DaemonSet collector instances, which are deployed on every node)
+
+The Deployment collector handles the following data:
+
+- Kubernetes Events: Monitors and collects events occurring across the entire Kubernetes cluster, utilizing [Kubernetes Objects Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sobjectsreceiver#readme).
+- Cluster Metrics: Captures metrics that provide insights into the overall health and performance of the Kubernetes cluster, utilizing [Kubernetes Cluster Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sclusterreceiver#readme).
+
+### Auto-instrumentation
+
+The Helm Chart is configured to enable zero-code instrumentation using the [Operator's Instrumentation resource](https://github.com/open-telemetry/opentelemetry-operator/?tab=readme-ov-file#opentelemetry-auto-instrumentation-injection) for the following programming languages:
+
+- Go
+- Java
+- Node.js
+- Python
+- .NET
+
+
+### Installation
+
+1. Create the `opentelemetry-operator-system` Kubernetes namespace:
+```
+$ kubectl create namespace opentelemetry-operator-system
+```
+
+2. Create a secret in Kubernetes with the following command.
+   ```
+   kubectl create -n opentelemetry-operator-system secret generic elastic-secret-otel \
+     --from-literal=elastic_endpoint='YOUR_ELASTICSEARCH_ENDPOINT' \
+     --from-literal=elastic_api_key='YOUR_ELASTICSEARCH_API_KEY'
+   ```
+   Don't forget to replace
+   - `YOUR_ELASTICSEARCH_ENDPOINT`: your Elasticsearch endpoint (*with* `https://` prefix example: `https://1234567.us-west2.gcp.elastic-cloud.com:443`).
+   - `YOUR_ELASTICSEARCH_API_KEY`: your Elasticsearch API Key
+
+3. Execute the following commands to deploy the Helm Chart.
+
+```
+$ helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
+$ helm repo update
+$ helm upgrade --install --namespace opentelemetry-operator-system opentelemetry-kube-stack open-telemetry/opentelemetry-kube-stack --values ./values.yaml --version 0.3.3
+
+> [!NOTE]
+> Refer to the [compatibility matrix](https://github.com/elastic/opentelemetry/blob/main/docs/kubernetes/operator/README.md#compatibility-matrix) for a complete list of available manifests and associated helm chart versions.
+```