Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MAINT: documentation update for certain processors #7713

Merged
merged 3 commits into from
Jul 31, 2024
Merged

MAINT: documentation update for certain processors #7713

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
29fc181
MAINT: documentation update for processors
chenqi0805 Jul 15, 2024
3fb5625
Merge branch 'main' into maint/processor-auto-generation-campaign
chenqi0805 Jul 30, 2024
87a088f
MAINT: fix hyper links
chenqi0805 Jul 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion _data-prepper/common-use-cases/log-enrichment.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -370,7 +370,7 @@ The `date` processor can generate timestamps for incoming events if you specify

### Deriving punctuation patterns

The [`substitute_string`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/substitute-string/) processor (which is one of the mutate string processors) lets you derive a punctuation pattern from incoming events. In the following example pipeline, the processor will scan incoming Apache log events and derive punctuation patterns from them:
The [`substitute_string`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/substitute_string/) processor (which is one of the mutate string processors) lets you derive a punctuation pattern from incoming events. In the following example pipeline, the processor will scan incoming Apache log events and derive punctuation patterns from them:

```yaml
processor:
Expand Down
6 changes: 3 additions & 3 deletions _data-prepper/common-use-cases/trace-analytics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@

### OpenTelemetry trace source

The [OpenTelemetry source]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/otel-trace-raw/) accepts trace data from the OpenTelemetry Collector. The source follows the [OpenTelemetry Protocol](https://github.com/open-telemetry/opentelemetry-specification/tree/master/specification/protocol) and officially supports transport over gRPC and the use of industry-standard encryption (TLS/HTTPS).
The [OpenTelemetry source]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/otel_traces/) accepts trace data from the OpenTelemetry Collector. The source follows the [OpenTelemetry Protocol](https://github.com/open-telemetry/opentelemetry-specification/tree/master/specification/protocol) and officially supports transport over gRPC and the use of industry-standard encryption (TLS/HTTPS).

Check failure on line 35 in _data-prepper/common-use-cases/trace-analytics.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[Vale.Terms] Use 'OTel' instead of 'otel'.

Raw Output:
{"message": "[Vale.Terms] Use 'OTel' instead of 'otel'.", "location": {"path": "_data-prepper/common-use-cases/trace-analytics.md", "range": {"start": {"line": 35, "column": 105}}}, "severity": "ERROR"}

### Processor

Expand All @@ -49,8 +49,8 @@

The sink provides specific configurations for the trace analytics feature. These configurations allow the sink to use indexes and index templates specific to trace analytics. The following OpenSearch indexes are specific to trace analytics:

* otel-v1-apm-span –- The *otel-v1-apm-span* index stores the output from the [otel_traces_raw]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/otel-trace-raw/) processor.
* otel-v1-apm-service-map –- The *otel-v1-apm-service-map* index stores the output from the [service_map_stateful]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/service-map-stateful/) processor.
* otel-v1-apm-span –- The *otel-v1-apm-span* index stores the output from the [otel_traces_raw]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/otel_traces/) processor.
* otel-v1-apm-service-map –- The *otel-v1-apm-service-map* index stores the output from the [service_map_stateful]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/service_map/) processor.

## Trace tuning

Expand Down
7 changes: 7 additions & 0 deletions _data-prepper/pipelines/configuration/processors/date.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,13 @@ The `date` processor adds a default timestamp to an event, parses timestamp fiel

The following table describes the options you can use to configure the `date` processor.

<!--
This table is autogenerated. Do not edit it.
- name: date
- pluginType: processor
- source: https://github.com/opensearch-project/data-prepper/blob/c4455a7785bc2da4358067c217be7085e0bc8d0f/data-prepper-plugins/date-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/date/DateProcessorConfig.java
-->


Option | Required | Type | Description
:--- | :--- | :--- | :---
Expand Down
7 changes: 7 additions & 0 deletions _data-prepper/pipelines/configuration/processors/delete_entries.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,13 @@ The `delete_entries` processor deletes entries, such as key-value pairs, from an

You can configure the `delete_entries` processor with the following options.

<!--
This table is autogenerated. Do not edit it.
- name: delete_entries
- pluginType: processor
- source: https://github.com/opensearch-project/data-prepper/blob/c4455a7785bc2da4358067c217be7085e0bc8d0f/data-prepper-plugins/mutate-event-processors/src/main/java/org/opensearch/dataprepper/plugins/processor/mutateevent/DeleteEntryProcessorConfig.java
-->

| Option | Required | Description |
:--- | :--- | :---
| `with_keys` | Yes | An array of keys for the entries to be deleted. |
Expand Down
9 changes: 8 additions & 1 deletion _data-prepper/pipelines/configuration/processors/grok.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
layout: default
title: Grok
title: grok
parent: Processors
grand_parent: Pipelines
nav_order: 50
Expand All @@ -14,6 +14,13 @@ The Grok processor uses pattern matching to structure and extract important keys

The following table describes options you can use with the Grok processor to structure your data and make your data easier to query.

<!--
This table is autogenerated. Do not edit it.
- name: grok
- pluginType: processor
- source: https://github.com/opensearch-project/data-prepper/blob/c4455a7785bc2da4358067c217be7085e0bc8d0f/data-prepper-plugins/grok-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/grok/GrokProcessorConfig.java
-->

Option | Required | Type | Description
:--- | :--- |:--- | :---
`break_on_match` | No | Boolean | Specifies whether to match all patterns (`true`) or stop once the first successful match is found (`false`). Default is `true`.
Expand Down
9 changes: 8 additions & 1 deletion .../configuration/processors/otel-metrics.md → .../configuration/processors/otel_metrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,21 @@ To get started, add the following processor to your `pipeline.yaml` configuratio

``` yaml
processor:
- otel_metrics_raw_processor:
- otel_metrics:
```
{% include copy.html %}

## Configuration

You can use the following optional parameters to configure histogram buckets and their default values. A histogram displays numerical data by grouping data into buckets. You can use histogram buckets to view sets of events that are organized by the total event count and aggregate sum for all events. For more detailed information, see [OpenTelemetry Histograms](https://opentelemetry.io/docs/reference/specification/metrics/data-model/#histogram).

<!--
This table is autogenerated. Do not edit it.
- name: otel_metrics
- pluginType: processor
- source: https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/otel-metrics-raw-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/otelmetrics/OtelMetricsRawProcessorConfig.java
-->

| Parameter | Default value | Description |
| :--- | :--- | :--- |
| `calculate_histogram_buckets` | `True` | Whether or not to calculate histogram buckets. |
Expand Down
9 changes: 8 additions & 1 deletion ...onfiguration/processors/otel-trace-raw.md → ...s/configuration/processors/otel_traces.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
layout: default
title: otel_trace
title: otel_traces
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably clarify somewhere that otel_trace was the old name.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our deprecated name was otel_trace_raw. I never saw otel_trace anywhere in our repo

parent: Processors
grand_parent: Pipelines
nav_order: 75
Expand All @@ -23,6 +23,13 @@ This processor includes the following parameters.

The following table describes the options you can use to configure the `otel_trace` processor.

<!--
This table is autogenerated. Do not edit it.
- name: otel_traces
- pluginType: processor
- source: https://github.com/opensearch-project/data-prepper/blob/c4455a7785bc2da4358067c217be7085e0bc8d0f/data-prepper-plugins/otel-trace-raw-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/oteltrace/OtelTraceRawProcessorConfig.java
-->

Option | Required | Type | Description
:--- | :--- | :--- | :---
trace_flush_interval | No | Integer | Represents the time interval in seconds to flush all the descendant spans without any root span. Default is 180.
Expand Down
9 changes: 8 additions & 1 deletion ...ration/processors/service-map-stateful.md → ...s/configuration/processors/service_map.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,13 @@ The `service_map` processor uses OpenTelemetry data to create a distributed serv

The following table describes the option you can use to configure the `service_map` processor.

<!--
This table is autogenerated. Do not edit it.
- name: service_map
- pluginType: processor
- source: https://github.com/opensearch-project/data-prepper/blob/c4455a7785bc2da4358067c217be7085e0bc8d0f/data-prepper-plugins/service-map-stateful/src/main/java/org/opensearch/dataprepper/plugins/processor/ServiceMapProcessorConfig.java
-->

Option | Required | Type | Description
:--- | :--- | :--- | :---
window_duration | No | Integer | Represents the fixed time window, in seconds, during which service map relationships are evaluated. Default value is 180.
Expand All @@ -32,7 +39,7 @@ The following table describes common [Abstract processor](https://github.com/ope
| `recordsOut` | Counter | Metric representing the egress of records from a pipeline component. |
| `timeElapsed` | Timer | Metric representing the time elapsed during execution of a pipeline component. |

The `service-map-stateful` processor includes following custom metrics:
The `service_map` processor includes following custom metrics:

* `traceGroupCacheCount`: The number of trace groups in the trace group cache.
* `spanSetCount`: The number of span sets in the span set collection.
7 changes: 7 additions & 0 deletions .../configuration/processors/split-string.md → .../configuration/processors/split_string.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,13 @@ nav_order: 100

The `split_string` processor splits a field into an array using a delimiting character and is a [mutate string](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/mutate-string-processors#mutate-string-processors) processor. The following table describes the options you can use to configure the `split_string` processor.

<!--
This table is autogenerated. Do not edit it.
- name: split_string
- pluginType: processor
- source: https://github.com/opensearch-project/data-prepper/blob/c4455a7785bc2da4358067c217be7085e0bc8d0f/data-prepper-plugins/mutate-string-processors/src/main/java/org/opensearch/dataprepper/plugins/processor/mutatestring/SplitStringProcessorConfig.java
-->

Option | Required | Type | Description
:--- | :--- | :--- | :---
entries | Yes | List | List of entries. Valid values are `source`, `delimiter`, and `delimiter_regex`.
Expand Down
7 changes: 7 additions & 0 deletions ...figuration/processors/string-converter.md → ...figuration/processors/string_converter.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,13 @@ nav_order: 105

The `string_converter` processor converts a string to uppercase or lowercase. You can use it as an example for developing your own processor. The following table describes the option you can use to configure the `string_converter` processor.

<!--
This table is autogenerated. Do not edit it.
- name: string_converter
- pluginType: processor
- source: https://github.com/opensearch-project/data-prepper/blob/c4455a7785bc2da4358067c217be7085e0bc8d0f/data-prepper-plugins/common/src/main/java/org/opensearch/dataprepper/plugins/processor/StringProcessor.java
-->

Option | Required | Type | Description
:--- | :--- | :--- | :---
upper_case | No | Boolean | Whether to convert to uppercase (`true`) or lowercase (`false`).
Expand Down
7 changes: 7 additions & 0 deletions ...iguration/processors/substitute-string.md → ...iguration/processors/substitute_string.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,13 @@ The `substitute_string` processor matches a key's value against a regular expres

The following table describes the options you can use to configure the `substitute_string` processor.

<!--
This table is autogenerated. Do not edit it.
- name: substitute_string
- pluginType: processor
- source: https://github.com/opensearch-project/data-prepper/blob/c4455a7785bc2da4358067c217be7085e0bc8d0f/data-prepper-plugins/mutate-string-processors/src/main/java/org/opensearch/dataprepper/plugins/processor/mutatestring/SubstituteStringProcessorConfig.java
-->

Option | Required | Type | Description
:--- | :--- | :--- | :---
entries | Yes | List | List of entries. Valid values are `source`, `from`, and `to`.
Expand Down
7 changes: 7 additions & 0 deletions ...s/configuration/processors/trim-string.md → ...s/configuration/processors/trim_string.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,13 @@ nav_order: 120

The `trim_string` processor removes white space from the beginning and end of a key and is a [mutate string](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/mutate-string-processors#mutate-string-processors) processor. The following table describes the option you can use to configure the `trim_string` processor.

<!--
This table is autogenerated. Do not edit it.
- name: trim_string
- pluginType: processor
- source: https://github.com/opensearch-project/data-prepper/blob/c4455a7785bc2da4358067c217be7085e0bc8d0f/data-prepper-plugins/mutate-string-processors/src/main/java/org/opensearch/dataprepper/plugins/processor/mutatestring/WithKeysConfig.java
-->

Option | Required | Type | Description
:--- | :--- | :--- | :---
with_keys | Yes | List | A list of keys to trim the white space from.
Expand Down
7 changes: 7 additions & 0 deletions _data-prepper/pipelines/configuration/processors/truncate.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,13 @@ The `truncate` processor truncates a key's value at the beginning, the end, or o

You can configure the `truncate` processor using the following options.

<!--
This table is autogenerated. Do not edit it.
- name: truncate
- pluginType: processor
- source: https://github.com/opensearch-project/data-prepper/blob/c4455a7785bc2da4358067c217be7085e0bc8d0f/data-prepper-plugins/truncate-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/truncate/TruncateProcessorConfig.java
-->

Option | Required | Type | Description
:--- | :--- | :--- | :---
`entries` | Yes | String list | A list of entries to add to an event.
Expand Down
2 changes: 1 addition & 1 deletion _observing-your-data/trace/ta-dashboards.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ The **Trace Analytics** application includes two options: **Services** and **Tra
The plugin requires you to use [Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/) to process and visualize OTel data and relies on the following Data Prepper pipelines for OTel correlations and service map calculations:

- [Trace analytics pipeline]({{site.url}}{{site.baseurl}}/data-prepper/common-use-cases/trace-analytics/)
- [Service map pipeline]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/service-map-stateful/)
- [Service map pipeline]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/service_map/)

### Standardized telemetry data

Expand Down
Loading