Three node monolith cluster's distributor's ownership are unbalanced - advice needed.

[jzakhar]

We have a three node monolith cluster running on metal servers with an upstream f5 doing load balancing between the three.

We've noticed that one distributor gets 95% of ownership and all three distributor's only have one token each.
If we restart the mimir process on the distributor with 95% ownership, it just moves to one of the other two nodes..

Is that expected or intentional?
I tried defining num_tokens under ring: in the distributor stanza but it just spits out mimir[1576515]: line 59: field num_tokens not found in type distributor.RingConfig

I went to see if we could enable spread-minimizing tokens but that requires zone-awareness enabled.
We just have three nodes with a replication factor set to three I don't see how we enable zone-awareness in those circumstances.

Is this normal? will the other nodes start taking more traffic in as more arrives?
Do we have a config issue?
Any advice or help understanding what's happening here would be appreciated.

[Nachtfalkeaw]

Distributor is stateless and it just distributes the load. So I think it is not a bottleneck.

Make sure that at your f5 loadbalancer you disabled the source_persistent options. nour source_address or cookie or other persistence. just set it to "None". You do not need any stickyness and you should avoid that the F5 is doing any whch then sends all to one distributor.

Better check the distribution of the ingesters which should be more or less equal.