Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add single writer principle note to deployment documentation #5166

Open
wants to merge 22 commits into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
3386452
add single-writer-principle note to collector docs
michael2893 Sep 8, 2024
8f9a85e
Merge branch 'main' into michael2893-update-collector-docs-gateway
michael2893 Sep 8, 2024
9da5e86
fix linting issue on github links, correct spelling
michael2893 Sep 9, 2024
1695255
response to some review comments
michael2893 Sep 27, 2024
f9f7ff3
Merge branch 'main' into michael2893-update-collector-docs-gateway
michael2893 Sep 27, 2024
9cf717a
Merge branch 'main' into michael2893-update-collector-docs-gateway
michael2893 Sep 30, 2024
0822a25
Update content/en/docs/collector/deployment/gateway.md
michael2893 Jan 2, 2025
be42e53
Update content/en/docs/collector/deployment/gateway.md
michael2893 Jan 2, 2025
6eaa78e
Update content/en/docs/collector/deployment/gateway.md
michael2893 Jan 2, 2025
6a22294
Update content/en/docs/collector/deployment/gateway.md
michael2893 Jan 2, 2025
9409f01
add single-writer-principle note to collector docs
michael2893 Sep 8, 2024
87e12a3
fix linting issue on github links, correct spelling
michael2893 Sep 9, 2024
d054969
response to some review comments
michael2893 Sep 27, 2024
aff3fe7
Update content/en/docs/collector/deployment/gateway.md
michael2893 Jan 2, 2025
94bb694
Update content/en/docs/collector/deployment/gateway.md
michael2893 Jan 2, 2025
01b9a83
Update content/en/docs/collector/deployment/gateway.md
michael2893 Jan 2, 2025
69bec1b
Merge remote-tracking branch 'origin/michael2893-update-collector-doc…
michael2893 Jan 6, 2025
abf5a93
resolve conflicts
michael2893 Jan 6, 2025
d47901e
Merge branch 'main' into michael2893-update-collector-docs-gateway
michael2893 Jan 6, 2025
72bd07a
resolve conflicting files
michael2893 Jan 6, 2025
9011b56
Resolve package json update conflict
michael2893 Jan 6, 2025
f7162ba
Results from /fix:all
opentelemetrybot Jan 7, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions content/en/docs/collector/deployment/gateway/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -251,3 +251,45 @@ Cons:
https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor
[spanmetrics-connector]:
https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/spanmetricsconnector

## Multiple collectors and the single-writer principle

All metric data streams within OTLP must have a
[single writer](/docs/specs/otel/metrics/data-model/#single-writer). When
deploying multiple collectors in a gateway configuration, it's important to
ensure that all metric data streams have a single writer and a globally unique
identity.

### Potential problems

Concurrent access from multiple applications that modify or report on the same
data can lead to data loss or degraded data quality. For example, you might see
inconsistent data from multiple sources on the same resource, where the
different sources can overwrite each other because the resource is not uniquely
identified.

There are patterns in the data that may provide some insight into whether this
is happening or not. For example, upon visual inspection, a series with
unexplained gaps or jumps in the same series may be a clue that multiple
collectors are sending the same samples. You might also see errors in your
backend. For example, with a Prometheus backend:

`Error on ingesting out-of-order samples`

This error could indicate that identical targets exist in two jobs, and the
order of the timestamps is incorrect. For example:

- Metric `M1` received at `T1` with a timestamp 13:56:04 with value `100`
- Metric `M1` received at `T2` with a timestamp 13:56:24 with value `120`
- Metric `M1` received at `T3` with a timestamp 13:56:04 with value `110`
- Metric `M1` received at time 13:56:24 with value `120`
- Metric `M1` received at time 13:56:04 with value `110`

### Best practices

- Use the
[Kubernetes attributes processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/k8sattributesprocessor)
to add labels to different Kubernetes resources.
- Use the
[resource detector processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/resourcedetectionprocessor/README.md)
to detect resource information from the host and collect resource metadata.