Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create Elastic OpenTelemetry Collector docs #1

Merged
merged 17 commits into from
Aug 6, 2024
37 changes: 35 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,35 @@
# opentelemetry
Get started with our Elastic Distros of OpenTelemetry
# Elastic Distributions for OpenTelemetry

[OpenTelemetry](https://opentelemetry.io/docs/) is a vendor-neutral observability framework for collecting, processing, and exporting telemetry data.
The Elastic Distribution for OpenTelemetry Collector (Elastic OTel Collector) is a supported, drop-in replacement distribution of the [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector) made up of mostly upstream components.
You can send your telemetry data to Elastic Observability using OpenTelemetry the following ways:
mdbirnstiehl marked this conversation as resolved.
Show resolved Hide resolved

- Collect and send logs and host metrics to [Elastic Cloud](https://cloud.elastic.co/) using the Elastic OTel Collector.
- Instrument your applications and send logs, traces, and metrics to [Elastic Cloud](https://cloud.elastic.co/) using the Elastic Distributions for select [OpenTelemetry language SDKs](https://opentelemetry.io/docs/languages/). Currently, Elastic provides distributions for the following language SDKs: Java, .NET, Node.js, and Python.
- Build and configure a [custom collector](https://opentelemetry.io/docs/collector/custom-collector/) or extend the [OpenTelemetry Collector Contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) distribution to collect logs and metrics and send them to Elastic Observability.

This diagram provides a quick overview on how the different components work together. Refer to the [components](docs/collector-components.md) for a more in-depth look.

![Diagram of the OpenTelemetry flow](docs/images/elastic-otel-overview.png)
mdbirnstiehl marked this conversation as resolved.
Show resolved Hide resolved

## Collect infrastructure data with the Elastic Distribution for OpenTelemetry Collector

These pages detail the components and how to configure the Elastic OTel Collector.

- [Components](docs/collector-components.md): Get details on the components used to receive, process, and export telemetry data.
- [Guided onboarding](docs/guided-onboarding.md): Use the guided onboarding in Kibana or in a serverless Observability project to send data using the Elastic OTel Collector.
- [Manual configurations](docs/manual-configuration.md): Manually configure the Elastic OTel Collector to send data to Elastic Observability.
- [Limitations](docs/collector-limitations.md): Understand the current limitations of the Elastic OTel Collector.

## Collect application data with Elastic Distributions for OpenTelemetry language SDKs

Elastic offers several Distributions that extend [OpenTelemetry language SDKs](https://opentelemetry.io/docs/languages/). The following languages are currently supported:

* [Java](https://github.com/elastic/elastic-otel-java)
* [.NET](https://github.com/elastic/elastic-otel-dotnet)
* [Node.js](https://github.com/elastic/elastic-otel-node)
* [Python](https://github.com/elastic/elastic-otel-python)

## Configure a custom collector or the OpenTelemetry Collector Contrib distribution

[Configure a custom collector or the OpenTelemetry Collector Contrib distribution](docs/configure-upstream-collector.md): Build and configure a [custom collector](https://opentelemetry.io/docs/collector/custom-collector/) or extend the [OpenTelemetry Collector Contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) distribution to collect logs and metrics and send them to Elastic Observability.
131 changes: 131 additions & 0 deletions docs/collector-components.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
# Elastic Distribution for OpenTelemetry Collector components

The OpenTelemetry Collector uses the following components to receive, process, and export telemetry data:

- [Receivers](collector-components.md#receivers): Collect telemetry from your host.
- [Processors](collector-components.md#processors): Modify or transform telemetry data before sending it to the exporters.
- [Exporters](collector-components.md#exporters): Send data to the backends or destinations.
- [Extensions](collector-components.md#extensions): Provide additional functionalities and capabilities.

The default configurations of the Elastic Distribution for the OpenTelemetry Collector follows these flows.

**MacOS and Linux Host metrics:**

```mermaid
flowchart LR
one["`Host metrics receiver`"]
two["`Elastic infra metrics processor`"]
three["`Attributes processor (dataset)`"]
four["`Resource processor (process)`"]
five["`Elasticsearch exporter`"]
one --> two --> three --> four --> five

style one fill: #e6f9f7,stroke:#333,stroke-width:1px
style two fill: ##f8e9e9,stroke:#333,stroke-width:1px
style three fill: ##f8e9e9,stroke:#333,stroke-width:1px
style four fill: ##f8e9e9,stroke:#333,stroke-width:1px
style five fill:#e6f9f7,stroke:#333,stroke-width:1px
```

**MacOS and Linux Logs**

```mermaid
flowchart LR
one["`File log receiver`"]
two["`Resource detection processor`"]
three["`Elasticsearch exporter`"]
one --> two --> three

style one fill: #e6f9f7,stroke:#333,stroke-width:1px
style two fill: ##f8e9e9,stroke:#333,stroke-width:1px
style three fill:#e6f9f7,stroke:#333,stroke-width:1px
```

**Kubernetes metrics**

```mermaid
flowchart LR
one["`Kubelet stats and host metrics receivers`"]
two["`Kubernetes attributes processor`"]
three["`Elastic infra metrics processor`"]
four["`Resource detection processors (EKS, GCP, K8s)`"]
five["`Resource processors (K8s, cloud)`"]
six["`Attributes processor (dataset)`"]
seven["`Resource processor`"]
eight["`Elasticsearch exporter`"]
one --> two --> three --> four --> five --> six --> seven --> eight

style one fill: #e6f9f7,stroke:#333,stroke-width:1px
style two fill: ##f8e9e9,stroke:#333,stroke-width:1px
style three fill: ##f8e9e9,stroke:#333,stroke-width:1px
style four fill: ##f8e9e9,stroke:#333,stroke-width:1px
style five fill: ##f8e9e9,stroke:#333,stroke-width:1px
style six fill: ##f8e9e9,stroke:#333,stroke-width:1px
style seven fill: ##f8e9e9,stroke:#333,stroke-width:1px
style eight fill:#e6f9f7,stroke:#333,stroke-width:1px
```

**Kubernetes, MacOS, and Linux logs**
```mermaid
flowchart LR
one["`File log receiver`"]
two["`Kubernetes attributes processor`"]
three["`Resource detection processors (system, EKS, GCP)`"]
four["`Resource processors (K8s, cloud)`"]
five["`Attributes processor (k8s_logs_dataset)`"]
six["`Elasticsearch and debug exporters`"]
one --> two --> three --> four --> five --> six

style one fill: #e6f9f7,stroke:#333,stroke-width:1px
style two fill: ##f8e9e9,stroke:#333,stroke-width:1px
style three fill: ##f8e9e9,stroke:#333,stroke-width:1px
style four fill: ##f8e9e9,stroke:#333,stroke-width:1px
style five fill: ##f8e9e9,stroke:#333,stroke-width:1px
style six fill: #e6f9f7,stroke:#333,stroke-width:1px
```

Refer to the following tables for more information on the components included in the Elastic OTel Collector.
Follow the links for OpenTelemetry documentation with more configuration details for each component.
To set up the Elastic OTel Collector, get started using the [guided onboarding](guided-onboarding.md) docs or the [manual configuration](manual-configuration.md) docs.

## Receivers

| Component | Description |
|---|---|
| [`filelogreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/filelogreceiver/v0.105.0/receiver/filelogreceiver/README.md) | Collects logs from files on the local filesystem, supporting various formats and log rotation strategies. |
| [`hostmetricsreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/hostmetricsreceiver/v0.105.0/receiver/hostmetricsreceiver/README.md) | Collects metrics from the host machine, such as CPU, memory, disk, and network usage. |
| [`httpcheckreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/httpcheckreceiver/v0.105.0/receiver/httpcheckreceiver/README.md) | Performs HTTP checks to monitor the availability and response time of web services. |
| [`k8sclusterreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/k8sclusterreceiver/v0.105.0/receiver/k8sclusterreceiver/README.md) | Gathers metrics and metadata from a Kubernetes cluster. |
| [`k8sobjectsreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/k8sobjectsreceiver/v0.105.0/receiver/k8sobjectsreceiver/README.md) | Monitors changes to Kubernetes objects, and collects related metrics. |
| [`kubeletstatsreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/kubeletstatsreceiver/v0.105.0/receiver/kubeletstatsreceiver/README.md) | Collects metrics from the Kubelet, including node and pod-level resource usage. |
| [`otlpreceiver`](https://github.com/open-telemetry/opentelemetry-collector/blob/receiver/otlpreceiver/v0.105.0/receiver/otlpreceiver/README.md) | Receives metrics, traces, and logs in OpenTelemetry Protocol (OTLP) format. |

## Processors

| Component | Description |
|---|---|
| [`elasticinframetricsprocessor`](https://github.com/elastic/opentelemetry-collector-components/blob/processor/elasticinframetricsprocessor/v0.7.1/processor/elasticinframetricsprocessor/README.md) | Processes infrastructure metrics to enhance and convert them for Elasticsearch. |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As of today this is not an upstream component. @AlexanderWert lets chat about this.

| [`attributesprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/attributesprocessor/v0.105.0/processor/attributesprocessor/README.md) | Modifies telemetry data attributes. |
mdbirnstiehl marked this conversation as resolved.
Show resolved Hide resolved
| [`filterprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/filterprocessor/v0.105.0/processor/filterprocessor/README.md) | Filters telemetry data to include or exclude specific data points. |
| [`k8sattributesprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/k8sattributesprocessor/v0.105.0/processor/k8sattributesprocessor/README.md) | Enhances telemetry data with Kubernetes-specific metadata. |
| [`resourcedetectionprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/resourcedetectionprocessor/v0.105.0/processor/resourcedetectionprocessor/README.md) | Detects resource attributes and adds them to telemetry data. |
| [`resourceprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/resourceprocessor/v0.105.0/processor/resourceprocessor/README.md) | Allows resource attributes to be modified in telemetry data. |
| [`transformprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/transformprocessor/v0.105.0/processor/transformprocessor/README.md) | Transforms telemetry data modifying based on specified rules. |
| [`batchprocessor`](https://github.com/open-telemetry/opentelemetry-collector/blob/processor/batchprocessor/v0.105.0/processor/batchprocessor/README.md) | Batches telemetry data to improve export performance and manage load on back-end systems. |

## Exporters

| Component | Description |
|---|---|
| [`elasticsearchexporter`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/exporter/elasticsearchexporter/v0.105.0/exporter/elasticsearchexporter/README.md) | Sends collected telemetry data to Elasticsearch for storage and analysis. |
| [`fileexporter`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/exporter/fileexporter/v0.105.0/exporter/fileexporter/README.md) | Writes telemetry data to a file, useful for debugging or offline analysis. |
| [`debugexporter`](https://github.com/open-telemetry/opentelemetry-collector/blob/exporter/debugexporter/v0.105.0/exporter/debugexporter/README.md) | Outputs telemetry data in a human-readable format for debugging purposes. |
| [`otlpexporter`](https://github.com/open-telemetry/opentelemetry-collector/blob/exporter/otlpexporter/v0.105.0/exporter/otlpexporter/README.md) | Sends telemetry data in OTLP format to a specified endpoint. |
| [`otlphttpexporter`](https://github.com/open-telemetry/opentelemetry-collector/blob/exporter/otlphttpexporter/v0.105.0/exporter/otlphttpexporter/README.md) | Sends telemetry data using HTTP with OTLP. |

## Extensions

| Component | Description |
|---|---|
| [`filestorage`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/extension/storage/filestorage/v0.105.0/extension/storage/filestorage/README.md)| Provides file-based storage for temporary data, such as checkpoints and intermediate state. |
| [`memorylimiterextension`](https://github.com/open-telemetry/opentelemetry-collector/blob/extension/memorylimiterextension/v0.105.0/extension/memorylimiterextension/README.md) | Limits the memory usage of the collector to prevent out-of-memory errors. |
11 changes: 11 additions & 0 deletions docs/collector-limitations.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Elastic Distribution for OpenTelemetry Collector limitations

mdbirnstiehl marked this conversation as resolved.
Show resolved Hide resolved
The Elastic Distribution for the OpenTelemetry Collector has the following limitations:

- Because of an upstream limitation, `host.network.*` metrics aren't present from the OpenTelemetry side.
- `process.state` isn't present in the OpenTelemetry host metric. It's set to a dummy value of **Unknown** in the **State** column of the host processes table.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ishleenk17 is this an upstream limitation?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The process.state is not present in the Hostmetrics receiver. So, yes its an upstream limitation.
At Elastic, we show the state as Unknown for all processes because of this limitation.

- The Elasticsearch exporter handles the resource attributes, but **Host OS version** and **Operating system** may show as "N/A".
- The CPU scraper needs to be enabled to collect the `systm.load.cores` metric, which affects the **Normalized Load** column in the **Hosts** table and the **Normalized Load** visualization on the host detailed view.
- The [`hostmetrics receiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver) doesn't support CPU and disk metrics on MacOS. These values will stay empty for collectors running on MacOS.
- The console shows error Log messages when the [`hostmetrics receiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver) can't access some of the process information due to permission issues.
- The console shows mapping errors initially until mapping occurs.
10 changes: 10 additions & 0 deletions docs/configure-upstream-collector.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Configure a custom collector or the OpenTelemetry Collector Contrib distribution

You can build and configure a [custom collector](https://opentelemetry.io/docs/collector/custom-collector/) or extend the [OpenTelemetry Collector Contrib ](https://github.com/open-telemetry/opentelemetry-collector-contrib) distribution to collect logs and metrics and send them to Elastic Observability.

For a more seamless experience, use the Elastic Distribution for the OpenTelemetry collector.
Refer to the [guided onboarding](guided-onboarding.md) docs or the [manual configuration](manual-configuration.md) docs for more on configuring the Elastic OTel collector.

## Upstream collector configuration examples
Refer to the OpenTelemetry documentation on [building a custom collector](https://opentelemetry.io/docs/collector/custom-collector/) for more on creating an upstream collector.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need more concrete steps on how exactly to build a custom collector that contains all the components required. Could we maybe even provide the config file for ocb? @ishleenk17 could you help with the concrete steps here?

Use the Elastic [example configurations](https://github.com/elastic/elastic-agent/tree/main/internal/pkg/otel/samples) as a reference when configuring your upstream collector.
32 changes: 32 additions & 0 deletions docs/guided-onboarding.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Collect logs and metrics using the guided onboarding
The guided onboarding in Kibana or in a serverless Observability project walks you through collecting logs and metrics using the Elastic Distribution for OpenTelemetry Collector.
To configure the Elastic OTel Collector manually, refer to the [manual configuration](manual-configuration.md) docs.

## Before you begin
The onboarding has the following requirements and limitations:

- The **Admin** role or higher is required to onboard system logs and metrics. To learn more, refer to [Assign user roles and privileges](https://www.elastic.co/docs/current/serverless/general/assign-user-roles).
- Root privileges on the host are required to run the OpenTelemetry collector used in this quickstart.
- The Elastic OTel Collector only works on Kubernetes, Linux, and MacOS systems.
- Refer to [Elastic OpenTelemetry Collector limitations](collector-limitations.md) for known limitations when using the Elastic OTel Collector.

## Collect your logs and metrics

Follow these steps to collect logs and metrics using the Elastic OTel collector:

1. Open an [Elastic Cloud](cloud.elastic.co) deployment or a serverless Observability project.
1. To open the guided onboarding, either:
1. In an Elastic Cloud deployment, open Kibana, and go to **Observability** → **Add Data**.
1. In a serverless Observability project, go to **Add Data**.
1. Select **Collect and analyze logs**, and then select **OpenTelemetry**.
1. Select the appropriate platform, and complete the following:
1. For **MacOS and Linux**, copy the command, open a terminal on your host, and run the command to download and configure the OpenTelemetry collector.
1. For **Kubernetes**, download the manifest.
1. Copy the command under Step 2:
1. For **MacOS and Linux**, run the command in your terminal to start the Elastic OTel collector.
1. For **Kubernetes**, run the command from the directory where you downloaded the manifest to install the Elastic OTel Collector on every node of your cluster.

Logs are collected from setup onward, so you won't see logs that occurred before starting the Elastic OTel Collector.
The default log path is `/var/log/*`. To update the path, modify `otel.yml`.

Under **Visualize your data**, you'll see links to **Logs Explorer** to view your logs and **Hosts** to view your host metrics.
Binary file added docs/images/elastic-otel-overview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/infrastructure-elastic-otel-flow.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/kubernetes-elastic-otel-flow.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading