Create Elastic OpenTelemetry Collector docs

README.md

@@ -1,2 +1,35 @@
-# opentelemetry
-Get started with our Elastic Distros of OpenTelemetry
+# Elastic Distributions for OpenTelemetry
+
+[OpenTelemetry](https://opentelemetry.io/docs/) is a vendor- and tool-neutral observability framework for collecting, processing, and exporting telemetry data.
+The Elastic Distribution for OpenTelemetry (Elastic OTel Collector) is a supported distribution of the [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector) made up of mostly upstream components.
+You can send your telemetry data to Elastic Observability using OpenTelemetry the following ways:
+
+- Collect and send logs and host metrics to [Elastic Cloud](https://cloud.elastic.co/) using the Elastic OTel Collector.
+- Instrument your applications and send logs, traces, and metrics to [Elastic Cloud](https://cloud.elastic.co/) using the **Elastic Distributions for OpenTelemetry** extensions of select [OpenTelemetry language SDKs](https://opentelemetry.io/docs/languages/). The following languages are currently supported: Java, .NET, Node.js, and Python.
+- Configure an upstream [custom collector](https://opentelemetry.io/docs/collector/custom-collector/) or [contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) distribution for the collector to collect logs and metrics and send them to Elastic Observability.
+
+This diagram provides a quick overview on how the different components work together. Refer to the [components](docs/collector-components.md) for a more in-depth look.
+
+![Diagram of the OpenTelemetry flow](docs/images/elastic-otel-overview.png)
+
+## Collect infrastructure data with the Elastic Distribution for OpenTelemetry Collector
+
+These pages detail the components and how to configure the Elastic OTel Collector.
+
+- [Components](docs/collector-components.md): Get details on the components used to receive, process, and export telemetry data.
+- [Guided onboarding](docs/guided-onboarding.md): Use the guided onboarding in Kibana or in a serverless Observability project to send data using the Elastic OTel Collector.
+- [Manual configurations](docs/manual-configuration.md): Manually configure the Elastic OTel Collector to send data to Elastic Observability.
+- [Limitations](docs/collector-limitations.md): Understand the current limitations of the Elastic OTel Collector.
+
+## Collect application data with Elastic Distributions for OpenTelemetry language SDKs
+
+Elastic offers several distributions that extend [OpenTelemetry language SDKs](https://opentelemetry.io/docs/languages/). The following languages are currently supported:
+
+* [Java](https://github.com/elastic/elastic-otel-java)
+* [.NET](https://github.com/elastic/elastic-otel-dotnet)
+* [Node.js](https://github.com/elastic/elastic-otel-node)
+* [Python](https://github.com/elastic/elastic-otel-python)
+
+## Configure an upstream collector for Elastic
+
+[Configure a custom collector or a contrib distribution of the OpenTelemetry collector](docs/configure-upstream-collector.md): Configure a [custom collector](https://opentelemetry.io/docs/collector/custom-collector/) or [contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) distribution for the OpenTelemetry collector to collect logs and metrics and send them to Elastic Observability.

docs/collector-components.md

@@ -0,0 +1,130 @@
+# Elastic Distribution for OpenTelemetry Collector components
+
+The OpenTelemetry Collector uses the following components to receive, process, and export telemetry data:
+
+- [Receivers](collector-components.md#receivers): Collect telemetry from your host.
+- [Processors](collector-components.md#processors): Modify or transform telemetry data before sending it to the exporters.
+- [Exporters](collector-components.md#exporters): Send data to the backends or destinations.
+- [Extensions](collector-components.md#extensions): Provide additional functionalities and capabilities.
+
+The default configurations of the Elastic OTel Collector follows these flows.
+
+**MacOS and Linux Host metrics:**
+
+```mermaid
+flowchart LR
+    one["`Host metrics receiver`"]
+    two["`Elastic infra metrics processor`"]
+    three["`Attributes processor (dataset)`"]
+    four["`Resource processor (process)`"]
+    five["`Elasticsearch exporter`"]
+    one --> two --> three --> four --> five
+
+    style one fill: #e6f9f7,stroke:#333,stroke-width:1px
+    style two fill: ##f8e9e9,stroke:#333,stroke-width:1px
+    style three fill: ##f8e9e9,stroke:#333,stroke-width:1px
+    style four fill: ##f8e9e9,stroke:#333,stroke-width:1px
+    style five fill:#e6f9f7,stroke:#333,stroke-width:1px
+```
+
+**MacOS and Linux Logs**
+
+```mermaid
+flowchart LR
+    one["`File log receiver`"]
+    two["`Resource detection processor`"]
+    three["`Elasticsearch exporter`"]
+    one --> two --> three
+
+    style one fill: #e6f9f7,stroke:#333,stroke-width:1px
+    style two fill: ##f8e9e9,stroke:#333,stroke-width:1px
+    style three fill:#e6f9f7,stroke:#333,stroke-width:1px
+```
+
+**Kubernetes metrics**
+
+```mermaid
+flowchart LR
+    one["`Kubelet stats and host metrics receivers`"]
+    two["`Kubernetes attributes processor`"]
+    three["`Elastic infra metrics processor`"]
+    four["`Resource detection processors (EKS, GCP, K8s)`"]
+    five["`Resource processors (K8s, cloud)`"]
+    six["`Attributes processor (dataset)`"]
+    seven["`Resource processor`"]
+    eight["`Elasticsearch exporter`"]
+    one --> two --> three --> four --> five --> six --> seven --> eight
+
+    style one fill: #e6f9f7,stroke:#333,stroke-width:1px
+    style two fill: ##f8e9e9,stroke:#333,stroke-width:1px
+    style three fill: ##f8e9e9,stroke:#333,stroke-width:1px
+    style four fill: ##f8e9e9,stroke:#333,stroke-width:1px
+    style five fill: ##f8e9e9,stroke:#333,stroke-width:1px
+    style six fill: ##f8e9e9,stroke:#333,stroke-width:1px
+    style seven fill: ##f8e9e9,stroke:#333,stroke-width:1px
+    style eight fill:#e6f9f7,stroke:#333,stroke-width:1px
+```
+
+**Kubernetes, MacOS, and Linux logs**
+```mermaid
+flowchart LR
+    one["`File log receiver`"]
+    two["`Kubernetes attributes processor`"]
+    three["`Resource detection processors (system, EKS, GCP)`"]
+    four["`Resource processors (K8s, cloud)`"]
+    five["`Attributes processor (k8s_logs_dataset)`"]
+    six["`Elasticsearch and debug exporters`"]
+    one --> two --> three --> four --> five --> six
+
+    style one fill: #e6f9f7,stroke:#333,stroke-width:1px
+    style two fill: ##f8e9e9,stroke:#333,stroke-width:1px
+    style three fill: ##f8e9e9,stroke:#333,stroke-width:1px
+    style four fill: ##f8e9e9,stroke:#333,stroke-width:1px
+    style five fill: ##f8e9e9,stroke:#333,stroke-width:1px
+    style six fill: #e6f9f7,stroke:#333,stroke-width:1px
+```
+
+Refer to the following tables for more information on the components included in the Elastic Distribution for the OpenTelemetry Collector.
+Follow the links for OpenTelemetry documentation with more configuration details for each component.
+To set up the collector, go to the [guided onboarding](guided-onboarding.md) or [manual configuration](manual-configuration.md) to get started.
+
+## Receivers
+
+| Component  | Description |
+|---|---|
+| [`filelogreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/filelogreceiver/v0.105.0/receiver/filelogreceiver/README.md) | Collects logs from files on the local filesystem, supporting various formats and log rotation strategies. |
+| [`hostmetricsreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/hostmetricsreceiver/v0.105.0/receiver/hostmetricsreceiver/README.md) | Collects metrics from the host machine, such as CPU, memory, disk, and network usage. |
+| [`httpcheckreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/httpcheckreceiver/v0.105.0/receiver/httpcheckreceiver/README.md) | Performs HTTP checks to monitor the availability and response time of web services. |
+| [`k8sclusterreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/k8sclusterreceiver/v0.105.0/receiver/k8sclusterreceiver/README.md) | Gathers metrics and metadata from a Kubernetes cluster. |
+| [`k8sobjectsreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/k8sobjectsreceiver/v0.105.0/receiver/k8sobjectsreceiver/README.md) | Monitors changes to Kubernetes objects, and collects related metrics. |
+| [`kubeletstatsreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/kubeletstatsreceiver/v0.105.0/receiver/kubeletstatsreceiver/README.md) | Collects metrics from the Kubelet, including node and pod-level resource usage. |
+| [`otlpreceiver`](https://github.com/open-telemetry/opentelemetry-collector/blob/receiver/otlpreceiver/v0.105.0/receiver/otlpreceiver/README.md) | Receives metrics, traces, and logs in OpenTelemetry Protocol (OTLP) format. |
+
+## Processors
+
+| Component  | Description |
+|---|---|
+| [`attributesprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/attributesprocessor/v0.105.0/processor/attributesprocessor/README.md) | Modifies telemetry data attributes. |
+| [`filterprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/filterprocessor/v0.105.0/processor/filterprocessor/README.md) | Filters telemetry data to include or exclude specific data points. |
+| [`k8sattributesprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/k8sattributesprocessor/v0.105.0/processor/k8sattributesprocessor/README.md) | Enhances telemetry data with Kubernetes-specific metadata. |
+| [`resourcedetectionprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/resourcedetectionprocessor/v0.105.0/processor/resourcedetectionprocessor/README.md) | Detects resource attributes and adds them to telemetry data. |
+| [`resourceprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/resourceprocessor/v0.105.0/processor/resourceprocessor/README.md) | Allows resource attributes to be modified in telemetry data. |
+| [`transformprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/transformprocessor/v0.105.0/processor/transformprocessor/README.md) | Transforms telemetry data modifying based on specified rules. |
+| [`batchprocessor`](https://github.com/open-telemetry/opentelemetry-collector/blob/processor/batchprocessor/v0.105.0/processor/batchprocessor/README.md) | Batches telemetry data to improve export performance and manage load on back-end systems. |
+
+## Exporters
+
+| Component  | Description |
+|---|---|
+| [`elasticsearchexporter`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/exporter/elasticsearchexporter/v0.105.0/exporter/elasticsearchexporter/README.md) | Sends collected telemetry data to Elasticsearch for storage and analysis. |
+| [`fileexporter`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/exporter/fileexporter/v0.105.0/exporter/fileexporter/README.md) | Writes telemetry data to a file, useful for debugging or offline analysis. |
+| [`debugexporter`](https://github.com/open-telemetry/opentelemetry-collector/blob/exporter/debugexporter/v0.105.0/exporter/debugexporter/README.md) | Outputs telemetry data in a human-readable format for debugging purposes. |
+| [`otlpexporter`](https://github.com/open-telemetry/opentelemetry-collector/blob/exporter/otlpexporter/v0.105.0/exporter/otlpexporter/README.md) | Sends telemetry data in OTLP format to a specified endpoint. |
+| [`otlphttpexporter`](https://github.com/open-telemetry/opentelemetry-collector/blob/exporter/otlphttpexporter/v0.105.0/exporter/otlphttpexporter/README.md) | Sends telemetry data using HTTP with OTLP. |
+
+## Extensions
+
+| Component  | Description |
+|---|---|
+| [`filestorage`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/extension/storage/filestorage/v0.105.0/extension/storage/filestorage/README.md)| Provides file-based storage for temporary data, such as checkpoints and intermediate state. |
+| [`memorylimiterextension`](https://github.com/open-telemetry/opentelemetry-collector/blob/extension/memorylimiterextension/v0.105.0/extension/memorylimiterextension/README.md) | Limits the memory usage of the collector to prevent out-of-memory errors. |

docs/collector-limitations.md

@@ -0,0 +1,8 @@
+# Elastic Distribution for OpenTelemetry Collector limitations
+
+The Elastic OTel Collector has the following limitations:
+
+- Because of an upstream limitation, `host.network.*` metrics aren't present from the OpenTelemetry side.
+- `process.state` isn't present in the OpenTelemetry host metric. It's set to a dummy value of **Unknown** in the **State** column of the host processes table.
+- The Elasticsearch exporter handles the metadata fields, but **Host OS version** and **Operating system** may show as "N/A" and **Host IP** may show different values.
+- The CPU scraper needs to be enabled to collect the `systm.load.cores` metric, which affects the **Normalized Load** column in the **Hosts** table and the **Normalized Load** visualization on the host detailed view.

docs/configure-upstream-collector.md

@@ -0,0 +1,10 @@
+# Configure an upstream collector
+
+You can configure an upstream collector, like a [custom collector](https://opentelemetry.io/docs/collector/custom-collector/) or [contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) distribution for the collector, to collect logs and metrics and send them to Elastic Observability.
+
+For a more seamless experience, use the Elastic distribution for the OpenTelemetry collector.
+Refer to the [guided onboarding](guided-onboarding.md) docs or the [manual configuration](manual-configuration.md) docs for more on configuring the Elastic distribution.
+
+## Upstream collector configuration examples
+Refer to the OpenTelemetry documentation on [building a custom collector](https://opentelemetry.io/docs/collector/custom-collector/) for more on creating an upstream collector.
+Use the Elastic [example configurations](https://github.com/elastic/elastic-agent/tree/main/internal/pkg/otel/samples) as a reference when configuring your upstream collector.

docs/guided-onboarding.md

@@ -0,0 +1,32 @@
+# Collect logs and metrics using the guided onboarding
+The guided onboarding in Kibana or in a serverless Observability project walks you through collecting logs and metrics using the Elastic OTel Collector.
+For manual configuration, refer to [Manually configure the Elastic OTel Collector](manual-configuration.md).
+
+## Before you begin
+The onboarding has the following requirements and limitations:
+
+- The **Admin** role or higher is required to onboard system logs and metrics. To learn more, refer to [Assign user roles and privileges](https://www.elastic.co/docs/current/serverless/general/assign-user-roles).
+- Root privileges on the host are required to run the OpenTelemetry collector used in this quickstart.
+- The Elastic OTel Collector only works on Kubernetes, Linux, and MacOS systems.
+- Refer to [Elastic OpenTelemetry Collector limitations](collector-limitations.md) for known limitations when using the Elastic OTel Collector.
+
+## Collect your logs and metrics
+
+Follow these steps to collect logs and metrics using the Elastic OTel collector
+
+1. Open an [Elastic Cloud](cloud.elastic.co) deployment or a serverless Observability project.
+1. To open the guided onboarding, either:
+   1. In an Elastic Cloud deployment, open Kibana, and go to **Observability** → **Add Data**.
+   1. In a serverless Observability project, go to **Add Data**.
+1. Select **Collect and analyze logs**, and then select **OpenTelemetry**.
+1. Select the appropriate platform:
+   1. For **MacOS and Linux**, copy the command, open a terminal on your host, and run the command to download and configure the OpenTelemetry collector.
+   1. For **Kubernetes**, download the manifest.
+1. Copy the command under Step 2:
+   1. For **MacOS and Linux**, run the command in your terminal to start the Elastic OTel collector.
+   1. For **Kubernetes**, run the command from the directory where you downloaded the manifest to install the Elastic OTel Collector on every node of your cluster.
+
+Logs are collected from setup onward, so you won't see logs that occurred before starting the Elastic OTel Collector.
+The default log path is `/var/log/*`. To update the path, modify `otel.yml`.
+
+Under **Visualize your data**, you'll see links to **Logs Explorer** to view your logs and **Hosts** to view your host metrics.

docs/manual-configuration.md

@@ -0,0 +1,37 @@
+# Manually configure the Elastic Distribution for OpenTelemetry Collector
+Collecting logs and host metrics with the Elastic OTel Collector without using the [guided onboarding](guided-onboarding.md) requires some manual configuration.
+
+## Before you begin
+The Elastic OTel collector has the following requirements and limitations:
+
+- The **Admin** role or higher is required to onboard system logs and metrics. To learn more, refer to [Assign user roles and privileges](https://www.elastic.co/docs/current/serverless/general/assign-user-roles).
+- Root privileges on the host are required to run the OpenTelemetry collector used in this quickstart.
+- The Elastic OTel Collector only works on Kubernetes, Linux, and MacOS systems.
+- Refer to [Elastic OpenTelemetry Collector limitations](collector-limitations.md) for known limitations when using the Elastic Distribution for the OpenTelemetry collector.
+
+## Collect your logs and metrics
+
+To manually configure the Elastic OTel Collector, gather the following information:
+
+- **Your Elasticsearch endpoint**: From the help menu in Kibana or your serverless Observability project, select **Connection details** and copy the **Elasticsearch endpoint**.
+- **API key**:
+   - **Kibana:** From the help menu, select **Connection details** and select **Create and manage API keys**. From the **API keys** page, select **Create API key**. Give your API key a name, select **Create API key**, and copy the new API key.
+   - **Serverless:** From the help menu, select **Connection details** and select the **API key** tab. Give your API key a name, select **Create API key**, and copy the new API key.
+
+Then manually configure the Elastic OTel Collector to collect logs and metrics on a MacOS or Linux system:
+
+1. Download and extract the standalone Elastic Agent for your platform. For more on downloading and extracting a standalone Elastic Agent, refer to the first step in [Install standalone Elastic Agents](https://www.elastic.co/guide/en/fleet/current/install-standalone-elastic-agent.html).
+1. From the Elastic Agent base directory, go to the `otel_samples` directory. The `platformlogs_hostmetrics.yml` file has the configurations for the receivers, processors, and exporters needed to collect logs and host metrics.
+1. Copy the content of the `platformlogs_hostmetrics.yml` file.
+1. From the Elastic Agent base directory, open the `otel.yml` file, and replace the content with the copied content from `platformlogs_hostmetrics.yml`.
+1. Find and update the following settings in the configuration:
+    - `file_storage.directory`: Set to the directory where you want to store you OpenTelemetry data. <!-- do we want to recommend a specific folder for this? -->
+    - `elasticsearch.endpoint`: Set to your Elasticsearch endpoint you copied earlier.
+    - `elasticsearch.api_key`: Set to the API key you created earlier.
+1. Run the OpenTelemetry collector with the following command:
+   ```console
+   ./elastic-agent otel --config otel.yml
+   ```
+
+Logs are collected from setup onward, so you won't see logs that occurred before starting the collector.
+The default log path is `/var/log/*`. Update the path in the `otel.yml` file.