[store] - Index Header unexpected number of S3 requests

[toledompm]

Hi people :)
Wanted to clear up some doubts about thanos store index-header. Ever since we enabled time partitioning I noticed a big spike in AWS S3 HeadObject Operations (about 100M per day 😱), I expected there would be an increase since each store builds it’s own index, but AWS S3 Api requests increased > 10x.
Is this expected and is there anything we can do to make index-header more efficient, or is something wrong with or thanos cluster?

More info:

• Currently running 6 shards, each looking at a period of 5 days
• Store and Compactor configs:

store
--log.level=debug
--data-dir=/data
--objstore.config-file=/etc/thanos/store/config.yaml
--store.enable-index-header-lazy-reader
--min-time=-5d
---
compact
--log.level=info
--data-dir=/data/thanos-compact
--objstore.config-file=/etc/thanos/store/config.yaml
--deduplication.replica-label=__replica__
--deduplication.replica-label=receive_replica
--deduplication.replica-label=receive_cluster
--wait
--wait-interval=5m
--retention.resolution-1h=30d
--retention.resolution-5m=10d
--retention.resolution-raw=3d
---
# /etc/thanos/store/config.yaml
type: S3
config:
  bucket: "####"
  endpoint: "s3.#####.amazonaws.com"
  region: "#####"
  trace:
    enable: true

[toledompm]

Got some good results by decreasing concurrency and increasing sync duration:

--block-meta-fetch-concurrency=8
--block-sync-concurrency=8
--sync-block-duration=2h

To minimize query impact we had to:

• Configure thanos store overlapping (set at 1 day, ie. store-1.minTime = 5d and store-2.maxTime = 5d)
• Configured thanos receiver retention to 3h, so live metric queries wont be affected by sync block delay

[toledompm]

S3 daily costs where cut in half instantly, still gonna mess arround a bit with these values to try to optmize it even more.