diff --git a/_data-prepper/common-use-cases/log-enrichment.md b/_data-prepper/common-use-cases/log-enrichment.md index 0d8ce4ab7d..0c878dd76e 100644 --- a/_data-prepper/common-use-cases/log-enrichment.md +++ b/_data-prepper/common-use-cases/log-enrichment.md @@ -370,7 +370,7 @@ The `date` processor can generate timestamps for incoming events if you specify ### Deriving punctuation patterns -The [`substitute_string`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/substitute-string/) processor (which is one of the mutate string processors) lets you derive a punctuation pattern from incoming events. In the following example pipeline, the processor will scan incoming Apache log events and derive punctuation patterns from them: +The [`substitute_string`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/substitute_string/) processor (which is one of the mutate string processors) lets you derive a punctuation pattern from incoming events. In the following example pipeline, the processor will scan incoming Apache log events and derive punctuation patterns from them: ```yaml processor: diff --git a/_data-prepper/common-use-cases/trace-analytics.md b/_data-prepper/common-use-cases/trace-analytics.md index 033830351a..3deca7b632 100644 --- a/_data-prepper/common-use-cases/trace-analytics.md +++ b/_data-prepper/common-use-cases/trace-analytics.md @@ -32,7 +32,7 @@ To monitor trace analytics in Data Prepper, we provide three pipelines: `entry-p ### OpenTelemetry trace source -The [OpenTelemetry source]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/otel-trace-raw/) accepts trace data from the OpenTelemetry Collector. The source follows the [OpenTelemetry Protocol](https://github.com/open-telemetry/opentelemetry-specification/tree/master/specification/protocol) and officially supports transport over gRPC and the use of industry-standard encryption (TLS/HTTPS). +The [OpenTelemetry source]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/otel_traces/) accepts trace data from the OpenTelemetry Collector. The source follows the [OpenTelemetry Protocol](https://github.com/open-telemetry/opentelemetry-specification/tree/master/specification/protocol) and officially supports transport over gRPC and the use of industry-standard encryption (TLS/HTTPS). ### Processor @@ -49,8 +49,8 @@ OpenSearch provides a generic sink that writes data to OpenSearch as the destina The sink provides specific configurations for the trace analytics feature. These configurations allow the sink to use indexes and index templates specific to trace analytics. The following OpenSearch indexes are specific to trace analytics: -* otel-v1-apm-span –- The *otel-v1-apm-span* index stores the output from the [otel_traces_raw]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/otel-trace-raw/) processor. -* otel-v1-apm-service-map –- The *otel-v1-apm-service-map* index stores the output from the [service_map_stateful]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/service-map-stateful/) processor. +* otel-v1-apm-span –- The *otel-v1-apm-span* index stores the output from the [otel_traces_raw]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/otel_traces/) processor. +* otel-v1-apm-service-map –- The *otel-v1-apm-service-map* index stores the output from the [service_map_stateful]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/service_map/) processor. ## Trace tuning diff --git a/_data-prepper/pipelines/configuration/processors/date.md b/_data-prepper/pipelines/configuration/processors/date.md index c44a10ba16..4f65f7b593 100644 --- a/_data-prepper/pipelines/configuration/processors/date.md +++ b/_data-prepper/pipelines/configuration/processors/date.md @@ -15,6 +15,13 @@ The `date` processor adds a default timestamp to an event, parses timestamp fiel The following table describes the options you can use to configure the `date` processor. + + Option | Required | Type | Description :--- | :--- | :--- | :--- diff --git a/_data-prepper/pipelines/configuration/processors/delete_entries.md b/_data-prepper/pipelines/configuration/processors/delete_entries.md index c9a93a1f3e..e7c022c6a7 100644 --- a/_data-prepper/pipelines/configuration/processors/delete_entries.md +++ b/_data-prepper/pipelines/configuration/processors/delete_entries.md @@ -14,6 +14,13 @@ The `delete_entries` processor deletes entries, such as key-value pairs, from an You can configure the `delete_entries` processor with the following options. + + | Option | Required | Description | :--- | :--- | :--- | `with_keys` | Yes | An array of keys for the entries to be deleted. | diff --git a/_data-prepper/pipelines/configuration/processors/grok.md b/_data-prepper/pipelines/configuration/processors/grok.md index 16f72c4968..3724278adf 100644 --- a/_data-prepper/pipelines/configuration/processors/grok.md +++ b/_data-prepper/pipelines/configuration/processors/grok.md @@ -1,6 +1,6 @@ --- layout: default -title: Grok +title: grok parent: Processors grand_parent: Pipelines nav_order: 50 @@ -14,6 +14,13 @@ The Grok processor uses pattern matching to structure and extract important keys The following table describes options you can use with the Grok processor to structure your data and make your data easier to query. + + Option | Required | Type | Description :--- | :--- |:--- | :--- `break_on_match` | No | Boolean | Specifies whether to match all patterns (`true`) or stop once the first successful match is found (`false`). Default is `true`. diff --git a/_data-prepper/pipelines/configuration/processors/otel-metrics.md b/_data-prepper/pipelines/configuration/processors/otel_metrics.md similarity index 93% rename from _data-prepper/pipelines/configuration/processors/otel-metrics.md rename to _data-prepper/pipelines/configuration/processors/otel_metrics.md index 08fb72810e..6fc82f5deb 100644 --- a/_data-prepper/pipelines/configuration/processors/otel-metrics.md +++ b/_data-prepper/pipelines/configuration/processors/otel_metrics.md @@ -16,7 +16,7 @@ To get started, add the following processor to your `pipeline.yaml` configuratio ``` yaml processor: - - otel_metrics_raw_processor: + - otel_metrics: ``` {% include copy.html %} @@ -24,6 +24,13 @@ processor: You can use the following optional parameters to configure histogram buckets and their default values. A histogram displays numerical data by grouping data into buckets. You can use histogram buckets to view sets of events that are organized by the total event count and aggregate sum for all events. For more detailed information, see [OpenTelemetry Histograms](https://opentelemetry.io/docs/reference/specification/metrics/data-model/#histogram). + + | Parameter | Default value | Description | | :--- | :--- | :--- | | `calculate_histogram_buckets` | `True` | Whether or not to calculate histogram buckets. | diff --git a/_data-prepper/pipelines/configuration/processors/otel-trace-raw.md b/_data-prepper/pipelines/configuration/processors/otel_traces.md similarity index 82% rename from _data-prepper/pipelines/configuration/processors/otel-trace-raw.md rename to _data-prepper/pipelines/configuration/processors/otel_traces.md index 395956a668..6d26a5aca8 100644 --- a/_data-prepper/pipelines/configuration/processors/otel-trace-raw.md +++ b/_data-prepper/pipelines/configuration/processors/otel_traces.md @@ -1,6 +1,6 @@ --- layout: default -title: otel_trace +title: otel_traces parent: Processors grand_parent: Pipelines nav_order: 75 @@ -23,6 +23,13 @@ This processor includes the following parameters. The following table describes the options you can use to configure the `otel_trace` processor. + + Option | Required | Type | Description :--- | :--- | :--- | :--- trace_flush_interval | No | Integer | Represents the time interval in seconds to flush all the descendant spans without any root span. Default is 180. diff --git a/_data-prepper/pipelines/configuration/processors/service-map-stateful.md b/_data-prepper/pipelines/configuration/processors/service_map.md similarity index 77% rename from _data-prepper/pipelines/configuration/processors/service-map-stateful.md rename to _data-prepper/pipelines/configuration/processors/service_map.md index a05f44863a..b62e222fd5 100644 --- a/_data-prepper/pipelines/configuration/processors/service-map-stateful.md +++ b/_data-prepper/pipelines/configuration/processors/service_map.md @@ -14,6 +14,13 @@ The `service_map` processor uses OpenTelemetry data to create a distributed serv The following table describes the option you can use to configure the `service_map` processor. + + Option | Required | Type | Description :--- | :--- | :--- | :--- window_duration | No | Integer | Represents the fixed time window, in seconds, during which service map relationships are evaluated. Default value is 180. @@ -32,7 +39,7 @@ The following table describes common [Abstract processor](https://github.com/ope | `recordsOut` | Counter | Metric representing the egress of records from a pipeline component. | | `timeElapsed` | Timer | Metric representing the time elapsed during execution of a pipeline component. | -The `service-map-stateful` processor includes following custom metrics: +The `service_map` processor includes following custom metrics: * `traceGroupCacheCount`: The number of trace groups in the trace group cache. * `spanSetCount`: The number of span sets in the span set collection. \ No newline at end of file diff --git a/_data-prepper/pipelines/configuration/processors/split-string.md b/_data-prepper/pipelines/configuration/processors/split_string.md similarity index 76% rename from _data-prepper/pipelines/configuration/processors/split-string.md rename to _data-prepper/pipelines/configuration/processors/split_string.md index 3959ae5acd..a8058dd530 100644 --- a/_data-prepper/pipelines/configuration/processors/split-string.md +++ b/_data-prepper/pipelines/configuration/processors/split_string.md @@ -11,6 +11,13 @@ nav_order: 100 The `split_string` processor splits a field into an array using a delimiting character and is a [mutate string](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/mutate-string-processors#mutate-string-processors) processor. The following table describes the options you can use to configure the `split_string` processor. + + Option | Required | Type | Description :--- | :--- | :--- | :--- entries | Yes | List | List of entries. Valid values are `source`, `delimiter`, and `delimiter_regex`. diff --git a/_data-prepper/pipelines/configuration/processors/string-converter.md b/_data-prepper/pipelines/configuration/processors/string_converter.md similarity index 66% rename from _data-prepper/pipelines/configuration/processors/string-converter.md rename to _data-prepper/pipelines/configuration/processors/string_converter.md index 32055791b8..8d3df165fb 100644 --- a/_data-prepper/pipelines/configuration/processors/string-converter.md +++ b/_data-prepper/pipelines/configuration/processors/string_converter.md @@ -11,6 +11,13 @@ nav_order: 105 The `string_converter` processor converts a string to uppercase or lowercase. You can use it as an example for developing your own processor. The following table describes the option you can use to configure the `string_converter` processor. + + Option | Required | Type | Description :--- | :--- | :--- | :--- upper_case | No | Boolean | Whether to convert to uppercase (`true`) or lowercase (`false`). diff --git a/_data-prepper/pipelines/configuration/processors/substitute-string.md b/_data-prepper/pipelines/configuration/processors/substitute_string.md similarity index 75% rename from _data-prepper/pipelines/configuration/processors/substitute-string.md rename to _data-prepper/pipelines/configuration/processors/substitute_string.md index 5d18bf6a4f..6958ff8e42 100644 --- a/_data-prepper/pipelines/configuration/processors/substitute-string.md +++ b/_data-prepper/pipelines/configuration/processors/substitute_string.md @@ -14,6 +14,13 @@ The `substitute_string` processor matches a key's value against a regular expres The following table describes the options you can use to configure the `substitute_string` processor. + + Option | Required | Type | Description :--- | :--- | :--- | :--- entries | Yes | List | List of entries. Valid values are `source`, `from`, and `to`. diff --git a/_data-prepper/pipelines/configuration/processors/trim-string.md b/_data-prepper/pipelines/configuration/processors/trim_string.md similarity index 67% rename from _data-prepper/pipelines/configuration/processors/trim-string.md rename to _data-prepper/pipelines/configuration/processors/trim_string.md index 46b6ad4af1..97927949a2 100644 --- a/_data-prepper/pipelines/configuration/processors/trim-string.md +++ b/_data-prepper/pipelines/configuration/processors/trim_string.md @@ -10,6 +10,13 @@ nav_order: 120 The `trim_string` processor removes white space from the beginning and end of a key and is a [mutate string](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/mutate-string-processors#mutate-string-processors) processor. The following table describes the option you can use to configure the `trim_string` processor. + + Option | Required | Type | Description :--- | :--- | :--- | :--- with_keys | Yes | List | A list of keys to trim the white space from. diff --git a/_data-prepper/pipelines/configuration/processors/truncate.md b/_data-prepper/pipelines/configuration/processors/truncate.md index 3714d80847..8b4c3d19e9 100644 --- a/_data-prepper/pipelines/configuration/processors/truncate.md +++ b/_data-prepper/pipelines/configuration/processors/truncate.md @@ -14,6 +14,13 @@ The `truncate` processor truncates a key's value at the beginning, the end, or o You can configure the `truncate` processor using the following options. + + Option | Required | Type | Description :--- | :--- | :--- | :--- `entries` | Yes | String list | A list of entries to add to an event. diff --git a/_observing-your-data/trace/ta-dashboards.md b/_observing-your-data/trace/ta-dashboards.md index 595dce6ca2..c7cf0a5091 100644 --- a/_observing-your-data/trace/ta-dashboards.md +++ b/_observing-your-data/trace/ta-dashboards.md @@ -48,7 +48,7 @@ The **Trace Analytics** application includes two options: **Services** and **Tra The plugin requires you to use [Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/) to process and visualize OTel data and relies on the following Data Prepper pipelines for OTel correlations and service map calculations: - [Trace analytics pipeline]({{site.url}}{{site.baseurl}}/data-prepper/common-use-cases/trace-analytics/) -- [Service map pipeline]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/service-map-stateful/) +- [Service map pipeline]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/service_map/) ### Standardized telemetry data