Thanos cant query from object store

[TheCodingSheikh]

Thanos, Prometheus and Golang version used:
Kubernetes, sidecar in kube prom stack chart
And thanos bitnami chart

Object Storage Provider:
Ceph
What happened:

we are facing an issue with Thanos that it can't query after a certain date even though the side car are pushing metrics until current date but in the Thanos front end I can see the object store has a max time of 2024-02-18
I tried thanos tools bucket verify it didnt show any issue
Logs for sidecar, compactor, storegateway didnt include anything that seems an issue
What you expected to happen:
I cant query up to date
How to reproduce it (as minimally and precisely as possible):

Full logs to relevant components:

Anything else we need to know:

[dosubot]

Hey @TheCodingSheikh! 👋 I'm here to help you with your Thanos issue. I can assist with bug fixes, answer questions, and even guide you on becoming a contributor. Let's squash those bugs together!

I found a similar unsolved discussion titled Store gateway returns no metrics even if blocks loaded, which might be related to your issue. Additionally, there are a few closed issues that might provide some insights:

1. Thanos store does not show data in downsampled block - This issue is closed but does not have comments or a solution.
2. Thanos query get No data if the time is older than 2 weeks - The problem was solved by running the thanos store command with different parameters, allowing Thanos to connect to the object store.
3. thanos-store can not query long-time ago data - This issue is also closed.

For your specific issue, ensure that the --max-time flag is set correctly in your Thanos Store Gateway configuration, as it determines the max time for data available in the object store. This flag can be set to a constant time in RFC3339 format or a time duration relative to the current time, such as -1d or 2h45m ^[1]. Additionally, verify that your Store Gateway configurations and flags are set correctly, as they can affect its ability to query data from the object store ^[1][2].

To continue talking to Dosu, mention @dosu.

[TheCodingSheikh]

One thing to mention, the bucket size is now 2T
When i do thanos tools bucket inspect i can only see from 2023-09-12 to 2024-02-08

[yeya24]

Is your issue cannot query object store or cannot query object store with latest timestamp?

I can see the object store has a max time of 2024-02-18

So your expectation is to see max time of now?

It doesn't sound like an issue of store gateway itself. There is a metric in sidecar for lastest timestamp of uploaded block.