opensearch-project · YANG-DB · May 30, 2023 · May 30, 2023 · May 30, 2023 · May 30, 2023
@@ -0,0 +1,55 @@
+![data-prepper](dataPrepper.svg)
+
+# What is Data Prepper
+
+[Data Prepper](https://github.com/opensearch-project/data-prepper/blob/main/docs/overview.md) is an open source utility service. Data Prepper is a server side data collector with abilities to filter, enrich, transform, normalize and aggregate data for downstream analytics and visualization. The broader vision for Data Prepper is to enable an end-to-end data analysis life cycle from gathering raw logs to facilitating sophisticated and actionable interactive ad-hoc analyses on the data.
+
+# What is Data Prepper Integration
+
+Data Prepper integration is concerned with the following aspects
+
+- Allow simple and automatic generation of all schematic structured
+    - traces ( including specific fields mapping to map to SS4O schema)
+    - services ( adding support for specific service mapping category)
+    - metrics (using the standard SS4O schema)
+
+- Add Dashboard Assets for correlation between traces-services-metrics
+
+- Add correlation queries to investigate traces based metrics
+
+# Data - Prepper Trace Fields
+Data Prepper uses the following [Traces](https://github.com/opensearch-project/data-prepper/blob/main/docs/schemas/trace-analytics/otel-v1-apm-span-index-template.md) mapping file
+The next fields are used:
+```text
+
+- traceId - A unique identifier for a trace. All spans from the same trace share the same traceId.
+- spanId - A unique identifier for a span within a trace, assigned when the span is created.
+- traceState - Conveys information about request position in multiple distributed tracing graphs.
+- parentSpanId - The spanId of this span's parent span. If this is a root span, then this field must be empty.
+- name - A description of the span's operation.
+- kind - The type of span. See OpenTelemetry - SpanKind.
+- startTime - The start time of the span.
+- endTime - The end time of the span.
+- durationInNanos - Difference in nanoseconds between startTime and endTime.
+- serviceName - Currently derived from the opentelemetry.proto.resource.v1.Resource associated with the span, the resource from the span originates.
+- events - A list of events. See OpenTelemetry - Events.
+- links - A list of linked spans. See OpenTelemetry - Links.
+- droppedAttributesCount - The number of attributes that were discarded.
+- droppedEventsCount - The number of events that were discarded.
+- droppedLinksCount - The number of links that were dropped.
+- span.attributes.* - All span attributes are split into a list of keywords.
+- resource.attributes.* - All resource attributes are split into a list of keywords.
+- status.code - The status of the span. See OpenTelemetry - Status.
+
+```
+There are some additional `trace.group` related fields which are not part of the [OTEL spec](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md) for traces
+```text
+- traceGroup - A derived field, the name of the trace's root span.
+- traceGroupFields.endTime - A derived field, the endTime of the trace's root span.
+- traceGroupFields.statusCode - A derived field, the status.code of the trace's root span.
+- traceGroupFields.durationInNanos - A derived field, the durationInNanos of the trace's root span.
+
+```
+### Metrics from Traces Processors
+
+Adding new processors for creating metrics for logs and traces that pass through [Data Prepper](https://opensearch.org/blog/Announcing-Data-Prepper-2.1.0/)
@@ -0,0 +1,78 @@
+# OTEL Astronomy Demo Application
+
+The following diagram presents the OTEL Astronomy shop services architecture:
+
+![](https://github.com/opensearch-project/opentelemetry-demo/raw/main/.github/img/docker-services-topology.png)
+
+
+### Trace Collectors
+Gaining a macro-level perspective on incoming data, such as sample counts and cardinality, is essential for comprehending the collector’s internal dynamics. However, when delving into the details, the interconnections can become complex. The Collector Data Flow Dashboard aims to demonstrate the capabilities of the OpenTelemetry demo application, offering a solid foundation for users to build upon.
+
+Monitoring data flow through the OpenTelemetry Collector is crucial for several reasons.
+- All services are traces in all the development languages
+- Auto instrumented
+- Manual spans and attributes
+- Span events
+- Span links
+
+Trace Headers are propagated across all services (**Context propagation**)
+
+
+### Metric Collectors
+Collecting all the KPI information into Prometheus time series storage including:
+- runtime metrics
+- HTTP / gRPC latency distribution
+
+### Data Flow Overview
+Collector Data Flow Dashboard provides valuable guidance on which metrics to monitor. Users can tailor their own dashboard variations by adding necessary metrics specific to their use cases, such as memory_delimiter processor or other data flow indicators. This demo dashboard serves as a starting point, enabling users to explore diverse usage scenarios and adapt the tool to their unique monitoring needs.
+
+![](https://github.com/opensearch-project/opentelemetry-demo/blob/main/.github/img/DemoFlow.png?raw=true)
+
+The diagram below provides an overview of the system components, showcasing the configuration derived from the OpenTelemetry Collector (otelcol) configuration file utilized by the OpenTelemetry demo application. Additionally, it highlights the observability data (traces and metrics) flow within the system.
+
+#### Simple purchase use case
+This flow diagram shows the trace evolution from the user selecting a purchased item going through different backend services until reaching the storage database.
+![](img/DemoFlow.png)
+
+#### Services
+- [All Services](https://otel.playground.opensearch.org/app/observability-traces#/services)
+  ![](https://raw.githubusercontent.com/YANG-DB/opensearch-catalog/otel-demo-integration/integrations/observability/otel-demo/static/services.png)
+
+- [accountingservice](https://otel.playground.opensearch.org/app/observability-traces#/services/accountingservice)
+- [adservice](https://otel.playground.opensearch.org/app/observability-traces#/services/adservice)
+- [cartservice](https://otel.playground.opensearch.org/app/observability-traces#/services/cartservice)
+- [checkoutservice](https://otel.playground.opensearch.org/app/observability-traces#/services/checkoutservice)
+- [currencyservice](https://otel.playground.opensearch.org/app/observability-traces#/services/currencyservice)
+- [emailservice](https://otel.playground.opensearch.org/app/observability-traces#/services/emailservice)
+- [featureflagservice](https://otel.playground.opensearch.org/app/observability-traces#/services/featureflagservice)
+- [frauddetectionservice](https://otel.playground.opensearch.org/app/observability-traces#/services/frauddetectionservice)
+- [frontend](https://otel.playground.opensearch.org/app/observability-traces#/services/frontend)
+- [frauddetectionservice](https://otel.playground.opensearch.org/app/observability-traces#/services/frauddetectionservice)
+- [loadgenerator](https://otel.playground.opensearch.org/app/observability-traces#/services/loadgenerator)
+- [paymentservice](https://otel.playground.opensearch.org/app/observability-traces#/services/paymentservice)
+- [productcatalogservice](https://otel.playground.opensearch.org/app/observability-traces#/services/productcatalogservice)
+- [quoteservice](https://otel.playground.opensearch.org/app/observability-traces#/services/quoteservice)
+- [recommendationservice](https://otel.playground.opensearch.org/app/observability-traces#/services/recommendationservice)
+- [shippingservice](https://otel.playground.opensearch.org/app/observability-traces#/services/shippingservice)
+---
+## OTEL collector pipelines
+
+### Logs
+![](https://github.com/opensearch-project/opentelemetry-demo/blob/main/src/otelcollector/img/logs-pipe.png?raw=true)
+
+### Metrics
+![](https://github.com/opensearch-project/opentelemetry-demo/blob/main/src/otelcollector/img/metrics-pipe.png?raw=true)
+
+### Traces
+![](https://github.com/opensearch-project/opentelemetry-demo/blob/main/src/otelcollector/img/traces-pipe.png?raw=true)
+
+
+---
+## Reference
+**_OTEL Demo info_**
+- [architecture](https://opentelemetry.io/docs/demo/architecture/)
+- [collector-data-flow-dashboard](https://opentelemetry.io/docs/demo/collector-data-flow-dashboard/)
+- [services](https://opentelemetry.io/docs/demo/services/)
+
+**_OTEL Demo repository_**
+- [OpenSearch: OpenTelemetry community demo](https://github.com/opensearch-project/opentelemetry-demo)
@@ -0,0 +1,79 @@
+
+![](https://raw.githubusercontent.com/opensearch-project/.github/main/profile/banner.jpg)
+# OpenSearch Observability OTEL Demo
+
+Welcome to the [OpenSearch](https://opensearch.org/docs/latest) OpenTelemetry [Demo](https://opentelemetry.io/docs/demo/) documentation, which covers how to install and run the demo, and some scenarios you can use to view OpenTelemetry in action.
+
+## Purpose
+The purpose of this demo is to demonstrate the different capabilities of OpenSearch Observability to investigate and reflect your system.
+
+![](../../../.github/img/DemoFlow.png)
+
+### Services
+[OTEL DEMO](https://opentelemetry.io/docs/demo/services/) Describes the list of services that are composing the Astronomy Shop.
+
+The main services that are open to user interactions:
+
+- [Dashboards](https://otel.playground.opensearch.org/)
+
+- [Demo Shop](https://shop.otel.playground.opensearch.org/)
+
+---
+
+### Screenshots
+
+_**The Shop**_
+
+
+![Shop](https://opentelemetry.io/docs/demo/screenshots/frontend-1.png)
+
+
+
+_**The load generator**_
+
+
+![](https://opentelemetry.io/docs/demo/screenshots/load-generator-ui.png)
+
+---
+
+### [Integrations](https://otel.playground.opensearch.org/app/integrations#/available)
+![](https://github.com/opensearch/opensearch-catalog/blob/otel-demo-integration/integrations/observability/otel-demo/static/dashboard1.png?raw=true)
+
+The integration service is a list of pre-canned assets that are loaded in a combined manner to allow users the ability for simple and automatic way to discover and review their services topology.
+
+These [OTEL demo](https://github.com/opensearch-project/opensearch-catalog/pull/91) integrations contain the following assets:
+- components & index template mapping
+    - [traces mapping](https://otel.playground.opensearch.org/app/opensearch_index_management_dashboards#/create-template/ss4o_traces_template)
+    - [logs mapping](https://otel.playground.opensearch.org/app/opensearch_index_management_dashboards#/create-template/ss4o_logs_template)
+    - [metrics mapping](https://otel.playground.opensearch.org/app/opensearch_index_management_dashboards#/create-template/ss4o_metrics_template)
+
+- Index Patterns
+    - [traces](https://otel.playground.opensearch.org/app/management/opensearch-dashboards/indexPatterns/)
+    - [logs](https://otel.playground.opensearch.org/app/management/opensearch-dashboards/indexPatterns/)
+    - [metrics](https://otel.playground.opensearch.org/app/management/opensearch-dashboards/indexPatterns/)
+
+- Index Aliases
+    - [traces](https://otel.playground.opensearch.org/app/opensearch_index_management_dashboards#/aliases)
+    - [logs](https://otel.playground.opensearch.org/app/opensearch_index_management_dashboards#/aliases)
+    - [metrics](https://otel.playground.opensearch.org/app/opensearch_index_management_dashboards#/aliases)
+
+- datasources
+    - [Prometheus](https://otel.playground.opensearch.org/app/datasources#/manage/prometheus)
+- data-stream & indices
+    - [Traces / Data-prepper](https://otel.playground.opensearch.org/app/opensearch_index_management_dashboards#/index-detail/otel-v1-apm-span-000001)
+    - [Traces / Jaeger](hhttps://otel.playground.opensearch.org/app/opensearch_index_management_dashboards#/index-detail/jaeger-span-2023-12-13)
+    - [Services / Data-prepper](https://otel.playground.opensearch.org/app/opensearch_index_management_dashboards#/index-detail/otel-v1-apm-service-map)
+    - [Services / Jaeger](https://otel.playground.opensearch.org/app/opensearch_index_management_dashboards#/index-detail/jaeger-service-2023-12-13)
+    - [Metrics / Data-prepper ](https://otel.playground.opensearch.org/app/opensearch_index_management_dashboards#/index-detail/otel-metrics-2023.12.13)
+    - [Log-Events / Data-prepper ](https://otel.playground.opensearch.org/app/opensearch_index_management_dashboards#/index-detail/otel-events-2023.12.13)
+
+
+- dashboards
+    - [OTEL Architecture](https://otel.playground.opensearch.org/app/dashboards#/view/67e37e40-f750-11ed-b6d0-850581e4a72d)
+    - [System-Wide AMP](https://otel.playground.opensearch.org/app/dashboards#/view/b0d09c20-9893-11ee-8bb8-69dd3b5541dd)
+---
+### Ingestion
+The ingestion capabilities for OpenSearch is to be able to support multiple pipelines:
+- [Data-Prepper](https://github.com/opensearch-project/data-prepper/) is an OpenSearch ingestion project that allows ingestion of OTEL standard signals using Otel-Collector
+- [Jaeger](https://opensearch.org/docs/latest/observing-your-data/trace/trace-analytics-jaeger/) is an ingestion framework which has a build in capability for pushing OTEL signals into OpenSearch
+- [Fluent-Bit](https://docs.fluentbit.io/manual/pipeline/outputs/opensearch) is an ingestion framework which has a build in capability for pushing OTEL signals into OpenSearch
@@ -0,0 +1,71 @@
+# OTEL Collector Pipeline
+
+
+### Traces
+The traces  pipeline consists of a receiver, multiple processors, and multiple exporters.
+
+![](img/traces-pipe.png)
+
+**Receiver (otlp):**
+This is where the data comes in from. In your configuration, the traces pipeline is using the otlp receiver. OTLP stands for OpenTelemetry Protocol. This receiver is configured to accept data over both gRPC and HTTP protocols. The HTTP protocol is also configured to allow CORS from any origin.
+
+**Processors (memory_limiter, batch, servicegraph):**
+Once the data is received, it is processed before being exported. The processors in the traces pipeline are:
+
+1. **memory_limiter:** This processor checks memory usage every second (check_interval: 1s) and ensures it does not exceed 4000 MiB (limit_mib: 4000). It also allows for a spike limit of 800 MiB (spike_limit_mib: 800).
+
+2. **batch:** This processor batches together traces before sending them on to the exporters, improving efficiency.
+
+3. **servicegraph:** This processor is specifically designed for creating a service graph from the traces. It is configured with certain parameters for handling latency histogram buckets, dimensions, store configurations, and so on.
+
+**Exporters (otlp, logging, spanmetrics, otlp/2):**
+After processing, the data is sent to the configured exporters:
+
+1. **otlp:** This exporter sends data to an endpoint configured as jaeger:4317 over OTLP with TLS encryption in insecure mode.
+
+2. **logging:** This exporter logs the traces.
+
+3. **spanmetrics:** This is likely a custom exporter defined as a connector in your configuration. It seems to be designed to create metrics from spans, but this is mostly conjecture based on the name.
+
+4. **otlp/logs:** This exporter sends data to an endpoint configured as dataprepper:21892 over OTLP with TLS encryption in insecure mode.
+
+### Metrics
+**Metrics Pipeline**
+
+This pipeline handles metric data.
+![](img/metrics-pipe.png)
+
+- **Receivers (otlp, spanmetrics):**
+
+Metric data comes in from the `otlp` receiver and the `spanmetrics` receiver.
+- **Processors (filter, memory_limiter, batch):**
+The data is then processed:
+1. **filter:** This processor excludes specific metrics. In this configuration, it is set to strictly exclude the queueSize metric.
+2. **memory_limiter:** Similar to the traces pipeline, this processor ensures memory usage doesn't exceed a certain limit.
+3. **batch:** This processor batches together metrics before sending them to the exporters, enhancing efficiency.
+
+- **Exporters (prometheus, logging):**
+The processed data is then exported:
+1. **prometheus:** This exporter sends metrics to an endpoint configured as
+2. **otelcol:9464**. It also converts resource information to Prometheus labels and enables OpenMetrics.
+3. **logging:** This exporter logs the metrics.
+
+### Logs
+
+**Logs Pipeline**
+
+This pipeline handles log data.
+![](img/logs-pipe.png)
+
+- **Receiver (otlp):**
+
+    Log data comes in from the otlp receiver. 
+- **Processors (memory_limiter, batch):**
+The data is then processed:
+1. **memory_limiter:** Similar to the traces and metrics pipelines, this processor ensures memory usage doesn't exceed a certain limit.
+2. **batch:** This processor batches together logs before sending them to the exporter, enhancing efficiency.
+
+- **Exporter (logging):**
+
+The processed data is then exported:
+ -  Logs Pipeline