Why not dedupe in Thanos query proxy?

[pentlander]

Thanos query does not deduplicate series when using the store API, it only does it using the http api. Specifically, it says:

Deduplication should be done on the highest level (just before PromQL).

I'm wondering what the reasoning behind that is. We're currently running a federated thanos query setup and just recently realized that deduplication is only occurring at the highest layer. It would be nice if we could more evenly distribute the work between layers, otherwise we'll just remove the second query layer as it just adds overhead.

Edit: Seems like this is relevant: #2700 though I don't understand the explanation. proxy.go reads in the Series which unmarshals the AggrChunks, so the deduping might as well be performed at the same time. Maybe I'm missing something, are the inner Chunks unmarshalled lazily?

[pentlander]

I took an allocs pprof of one of the lower level Thanos queries that's being used as a proxy. It appears that most of the allocations are spent unmarshalling the series, which includes the chunks:

[pentlander]

Seems like if the goal is to actually function as a proxy, this technique should be used to avoid decoding chunks: https://blog.najaryan.net/posts/partial-protobuf-encoding/

[GiedriusS]

This looks interesting, it would be worth exploring this further 👍 Note that our data field in Chunk is already bytes so we don't do any decoding there:

			m.Data = append(m.Data[:0], dAtA[iNdEx:postIndex]...)
			if m.Data == nil {
				m.Data = []byte{}
			}

We can probably save on RAM by #4557 which is a work in progress.