[Thanos Query] Query latency rises every 2 hours #7190
Unanswered
Kot-o-pes
asked this question in
Questions & Answers
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hello, ive encountered a strange case where query time increases every 2 hours and leads to lots of nodata alerts and degraded performance
i have a setup of following components
thanos-query-frontend - > thanos query -> fanout to prometheuses inside k8s clusters added via sidecar as a store in thanos query
thanos-query flags
ExecStart=/opt/thanos/current/thanos query --http-address 0.0.0.0:19192 --grpc-address 0.0.0.0:19182 --query.replica-label replica --query.replica-label prometheus_replica --query.replica-label receive_replica --log.format=json --store.sd-files="/etc/thanos/stores.yaml" --query.auto-downsampling --query.max-concurrent=1000 --query.max-concurrent-select=1000 --query.timeout=60s --query.default-step=60s --store.response-timeout=120s --store.limits.request-series=10000 --log.level=error --query.promql-engine=thanos
query-frontend
ExecStart=/opt/thanos/current/thanos query-frontend --http-address 0.0.0.0:19195 --query-frontend.downstream-url="http://0.0.0.0:19192" --labels.response-cache-config-file=/etc/thanos/cache.yaml --query-range.response-cache-config-file=/etc/thanos/cache.yaml --log.format=json --query-range.max-query-parallelism=1000 --labels.max-query-parallelism=1000 --query-range.split-interval=12h --query-range.request-downsampled --query-frontend.log-queries-longer-than=30s
thanos version 32.5
it looks like its related to prometheus creating block every 2 hours but i cant seem to find how it can cause such delays
Beta Was this translation helpful? Give feedback.
All reactions