Create Elastic OpenTelemetry Collector docs

README.md

@@ -1,2 +1,28 @@
-# opentelemetry
-Get started with our Elastic Distros of OpenTelemetry
+# Elastic OpenTelemetry Distributions
+
+[OpenTelemetry](https://opentelemetry.io/docs/) is a vendor- and tool-neutral observability framework for collecting, processing, and exporting telemetry data.
+You can send your telemetry data to Elastic Observability using OpenTelemetry the following ways:
+
+- Use the Elastic Distribution for the [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector) to collect and send logs and host metrics to [Elastic Cloud](https://cloud.elastic.co/).
+- Use Elastic Distributions for OpenTelemetry Java, .NET, Node.js, and Python to instrument your applications and send logs, traces, and metrics to [Elastic Cloud](https://cloud.elastic.co/).
+- Configure a custom collector or contrib Distribution for the OpenTelemetry Collector to collect logs and metrics and send them to Elastic Observability.
+
+This diagram provides a quick overview on how the the different components work together. Refer to the [components](docs/collector-components.md) for a more in-depth look.
+
+![Diagram of the OpenTelemetry flow](docs/images/elastic-otel-overview.png)
+
+## Get started with the Elastic distribution of the OpenTelemetry Collector
+
+These pages detail the components and configuration of the Elastic distribution of the OpenTelemetry collector.
+
+- [Components](docs/collector-components.md): Details on the components used to receive, process, and export telemetry data.
+- [Guided onboarding](docs/guided-onboarding.md): Use the guided onboarding in Kibana or a serverless Observability project to send data using the Elastic OpenTelemetry Collector.
+- [Manual configurations](docs/manual-configuration.md): Manually configure the Elastic OpenTelemetry Collector to send data to Elastic Observability.
+- [Limitations](docs/collector-limitations.md): Current limitations of the Elastic OpenTelemetry Collector.
+
+## Configure Elastic distributions of OpenTelemetry programming languages
+<!-- Colleen can fill in these links? -->
+
+## Configure a custom or contrib collector for Elastic
+
+[Configure a custom collector or a contrib distribution of the collector](docs/configure-custom-collector.md): Configure a [custom](https://opentelemetry.io/docs/collector/custom-collector/) collector or [Contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) distribution of the collector to collect logs and metrics and send them to Elastic Observability.

docs/collector-components.md

@@ -0,0 +1,64 @@
+# Elastic OpenTelemetry Collector components
+
+The OpenTelemetry Collector uses the following components to receive, process, and export telemetry data:
+
+- [Receivers](collector-components.md#receivers): collect telemetry from your host.
+- [Processors](collector-components.md#processors): modify or transform telemetry data before sending it to the exporters.
+- [Exporters](collector-components.md#exporters): send data to the backends or destinations.
+- [Extensions](collector-components.md#extensions): provide additional functionalities and capabilities.
+
+The default configurations of the Elastic Distribution for the OpenTelemetry Collector follows these flows.
+
+**MacOS and Linux logs and host metrics:**
+
+![Diagram of the logs and metrics OpenTelemetry flow](images/infrastructure-elastic-otel-flow.png)
+
+**Kubernetes logs and host metrics:**
+
+![Diagram of Kubernetes logs and metrics OpenTelemetry flow](images/kubernetes-elastic-otel-flow.png)
+
+Refer to the following tables for more information on the components supported by the Elastic Distribution for the OpenTelemetry Collector.
+Follow the links for OpenTelemetry documentation with more configuration details for each component.
+To set up the collector, go to the [guided onboarding](docs/guided-onboarding.md) or [manual configuration](docs/manual-configuration.md) to get started.
+
+## Receivers
+
+| Component  | Description |
+|---|---|
+| [`filelogreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/filelogreceiver/v0.105.0/receiver/filelogreceiver/README.md) | Collects logs from files on the local filesystem, supporting various formats and log rotation strategies. |
+| [`hostmetricsreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/hostmetricsreceiver/v0.105.0/receiver/hostmetricsreceiver/README.md) | Collects metrics from the host machine, such as CPU, memory, disk, and network usage. |
+| [`httpcheckreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/httpcheckreceiver/v0.105.0/receiver/httpcheckreceiver/README.md) | Performs HTTP checks to monitor the availability and response time of web services. |
+| [`k8sclusterreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/k8sclusterreceiver/v0.105.0/receiver/k8sclusterreceiver/README.md) | Gathers metrics and metadata from a Kubernetes cluster. |
+| [`k8sobjectsreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/k8sobjectsreceiver/v0.105.0/receiver/k8sobjectsreceiver/README.md) | Monitors changes to Kubernetes objects, and collects related metrics. |
+| [`kubeletstatsreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/kubeletstatsreceiver/v0.105.0/receiver/kubeletstatsreceiver/README.md) | Collects metrics from the Kubelet, including node and pod-level resource usage. |
+| [`otlpreceiver`](https://github.com/open-telemetry/opentelemetry-collector/blob/receiver/otlpreceiver/v0.105.0/receiver/otlpreceiver/README.md) | Receives metrics, traces, and logs in OpenTelemetry Protocol (OTLP) format. |
+
+## Processors
+
+| Component  | Description |
+|---|---|
+| [`elasticinframetricsprocessor`](https://github.com/elastic/opentelemetry-collector-components/blob/processor/elasticinframetricsprocessor/v0.7.1/processor/elasticinframetricsprocessor/README.md)  | Processes infrastructure metrics to enhance and convert them for Elasticsearch. |
+| [`attributesprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/attributesprocessor/v0.105.0/processor/attributesprocessor/README.md) | Modifies telemetry data attributes. |
+| [`filterprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/filterprocessor/v0.105.0/processor/filterprocessor/README.md) | Filters telemetry data to include or exclude specific data points. |
+| [`k8sattributesprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/k8sattributesprocessor/v0.105.0/processor/k8sattributesprocessor/README.md) | Enhances telemetry data with Kubernetes-specific metadata. |
+| [`resourcedetectionprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/resourcedetectionprocessor/v0.105.0/processor/resourcedetectionprocessor/README.md) | Detects resource attributes and adds them to telemetry data. |
+| [`resourceprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/resourceprocessor/v0.105.0/processor/resourceprocessor/README.md) | Allows resource attributes to be modified in telemetry data. |
+| [`transformprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/transformprocessor/v0.105.0/processor/transformprocessor/README.md) | Transforms telemetry data modifying based on specified rules. |
+| [`batchprocessor`](https://github.com/open-telemetry/opentelemetry-collector/blob/processor/batchprocessor/v0.105.0/processor/batchprocessor/README.md) | Batches telemetry data to improve export performance and manage load on back-end systems. |
+
+## Exporters
+
+| Component  | Description |
+|---|---|
+| [`elasticsearchexporter`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/exporter/elasticsearchexporter/v0.105.0/exporter/elasticsearchexporter/README.md) | Sends collected telemetry data to Elasticsearch for storage and analysis. |
+| [`fileexporter`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/exporter/fileexporter/v0.105.0/exporter/fileexporter/README.md) | Writes telemetry data to a file, useful for debugging or offline analysis. |
+| [`debugexporter`](https://github.com/open-telemetry/opentelemetry-collector/blob/exporter/debugexporter/v0.105.0/exporter/debugexporter/README.md) | Outputs telemetry data in a human-readable format for debugging purposes. |
+| [`otlpexporter`](https://github.com/open-telemetry/opentelemetry-collector/blob/exporter/otlpexporter/v0.105.0/exporter/otlpexporter/README.md) | Sends telemetry data in OTLP format to a specified endpoint. |
+| [`otlphttpexporter`](https://github.com/open-telemetry/opentelemetry-collector/blob/exporter/otlphttpexporter/v0.105.0/exporter/otlphttpexporter/README.md) | Sends telemetry data using HTTP with OTLP. |
+
+## Extensions
+
+| Component  | Description |
+|---|---|
+| [`filestorage`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/extension/storage/filestorage/v0.105.0/extension/storage/filestorage/README.md)| Provides file-based storage for temporary data, such as checkpoints and intermediate state. |
+| [`memorylimiterextension`](https://github.com/open-telemetry/opentelemetry-collector/blob/extension/memorylimiterextension/v0.105.0/extension/memorylimiterextension/README.md) | Limits the memory usage of the collector to prevent out-of-memory errors. |

docs/collector-limitations.md

@@ -0,0 +1,8 @@
+# Elastic OpenTelemetry Collector limitations
+
+The Elastic Distribution for the OpenTelemetry Collector has the following limitations:
+
+- `host.network.*` metrics aren't present from OpenTelemetry side.
+- `process.state` isn't present in the OpenTelemetry host metric. It's set to a dummy value of **Unknown** in the **State** column of the host processes table.
+- The Elasticsearch exporter handles the metadata fields, but **Host OS version** and**Operating system** may show as "N/A" and **Host IP** may show different values.
+- The CPU scraper needs to be enabled to collect the `systm.load.cores` metric, which affects the **Normalized Load** column in the **Hosts** table and the **Normalized Load** visualization on the host detailed view.

docs/guided-onboarding.md

@@ -0,0 +1,32 @@
+# Collect logs and metrics using the guided onboarding
+The guided onboarding in Kibana or a serverless Observability project walks you through collecting logs and metrics using the Elastic OpenTelemetry Collector.
+If you prefer manually configuring the Elastic OpenTelemetry collector, refer to [Manually configure the collector](docs/manual-configuration.md).
+
+## Before you begin
+The onboarding has the following requirements and limitations:
+
+- The **Admin** role or higher is required to onboard system logs and metrics. To learn more, refer to <DocLink slug="/serverless/general/assign-user-roles" />.
+- Root privileges on the host are required to run the OpenTelemetry collector used in this quickstart.
+- The collector only work on Kubernetes, Linux, and MacOS systems.
+- Refer to [Elastic OpenTelemetry Collector limitations](collector-limitations.md) for known limitations when using Elastic Distribution for the OpenTelemetry collector.
+
+## Collect your logs and metrics
+
+Follow these steps to collect logs and metrics using the Elastic OpenTelemetry collector
+
+1. Open an [Elastic Cloud](cloud.elastic.co) deployment or a serverless Observability project.
+1. To open the guided onboarding, either:
+   1. In a Elastic Cloud deployment, open Kibana, and go to **Observability** → **Add Data**.
+   1. In a serverless Observability project, go to **Add Data**.
+1. Select **Collect and analyze logs**, and then select **OpenTelemetry**.
+1. Select the appropriate platform:
+   1. For **MacOS and Linux**, copy the command, open a terminal on your host, and run the command to download and configure the OpenTelemetry collector.
+   1. For **Kubernetes**, download the manifest.
+1. Copy the command under Step 2:
+   1. For **MacOS and Linux**, run the command in your terminal to start the OpenTelemetry collector.
+   1. For **Kubernetes**, run the command from the directory where you downloaded the manifest to install the collector on every node of your cluster.
+
+Logs are collected from setup onward, so you won't see logs that occurred before starting the collector.
+The default log path is `/var/log/*`. To update the path, modify the `otel.yml`.
+
+Under **Visualize your data**, you'll see links to **Logs Explorer** to view your logs and **Hosts** to view your host metrics.

docs/manual-configuration.md

@@ -0,0 +1,37 @@
+# Manually configure the collector
+Collecting logs and host metrics with the Elastic Distribution for the OpenTelemetry Collector without using the [guided onboarding](docs/guided-onboarding.md) requires some manual configuration.
+
+## Before you begin
+The onboarding has the following requirements and limitations:
+
+- The **Admin** role or higher is required to onboard system logs and metrics. To learn more, refer to <DocLink slug="/serverless/general/assign-user-roles" />.
+- Root privileges on the host are required to run the OpenTelemetry collector used in this quickstart.
+- The collector only work on Kubernetes, Linux, and MacOS systems.
+- Refer to [Elastic OpenTelemetry Collector limitations](collector-limitations.md) for known limitations when using the Elastic Distribution for the OpenTelemetry collector.
+
+## Collect your logs and metrics
+
+To manually configure the collector, gather the following information:
+
+- **Your Elasticsearch endpoint**: From the help menu in Kibana or your serverless Observability project, select **Connection details** and copy the **Elasticsearch endpoint**.
+- **API key**:
+   - **Kibana:** From the help menu, select **Connection details** and select **Create and manage API keys**. From the **API keys** page, select **Create API key**. Give your API key a name, select **Create API key**, and copy the new API key.
+   - **Serverless:** From the help menu, select **Connection details** and select the **API key** tab. Give your API key a name, select **Create API key**, and copy the new API key.
+
+The following steps guide you through manually configuring the Elastic OpenTelemetry Collector to collect logs and metrics on a MacOS or Linux system:
+
+1. Download and extract the standalone Elastic Agent for your platform. For more on downloading and extracting a standalone Elastic Agent, refer to the first step on [Install standalone Elastic Agents](https://www.elastic.co/guide/en/fleet/current/install-standalone-elastic-agent.html).
+1. From the Elastic Agent base directory, go to the `otel_samples` directory. The `platformlogs_hostmetrics.yml` file has the configurations for the receivers, processors, and exporters needed to collect logs and host metrics.
+1. Copy the content of the `platformlogs_hostmetrics.yml`.
+1. From the Elastic Agent base directory, open the `otel.yml` file, and replace the content with the copied content from `platformlogs_hostmetrics.yml`.
+1. Find and update the following settings in the configuration:
+    - `file_storage.directory`: Set to the directory where you want to store you OpenTelemetry data. <!-- do we want to recommend a specific folder for this? -->
+    - `elasticsearch.endpoint`: Set to your Elasticsearch endpoint you copied earlier.
+    - `elasticsearch.api_key`: Set to the API key you created earlier.
+1. Run the OpenTelemetry collector with the following command:
+   ```console
+   ./elastic-agent otel --config otel.yml
+   ```
+
+Logs are collected from setup onward, so you won't see logs that occurred before starting the collector.
+The default log path is `/var/log/*`. Update the path in the `otel.yml` file.