Metrics from Loki Ruler do not appear in Mimir

[JDuskey]

Describe the bug

Metrics sent to Mimir from the Loki ruler's remote-write feature do not appear in Mimir, even though Mimir receives them.
Nginx receives a request with the user agent LokiReplayWAL UserAgent type. It appears to process it fine, with a 200 status code and no issues reported in the debug logs, however the metric does not appear when searched in Mimir. The recording rules were created using the alerting UI in grafana, and show as "ok".

Expected behavior

Metrics sent to Mimir from Loki's remote-write implementation are usable

Environment

• Infrastructure: kubernetes
• Deployment tool: helm

[JDuskey]

After further investigation, if I send directly to prometheus instead of mimir, the metric appears. If anyone is reading this and having the same issue, my current workaround is to have the loki ruler remote write to the prometheus instance that is scraping the cluster metrics, which sends them into Mimir.

[dimitarvdimitrov]

This was also shared in the community slack. My understanding was that the problem was caused by suing different no-auth tenants in loki and mimir. In Loki it is fake, whereas in Mimir it is anonymous (Nginx adds this header when it's not present). The way that @JDuskey reported solving this was to disable multitenancy on Mimir

mimir:
  structuredConfig:
    multitenancy_enabled: false

[MaxDiOrio]

Same issue here. Disabling multitenancy doesn't seem like the best idea, especially if it's needed on the Mimir side but not on the Loki side. Makes more sense to let Loki specify the tenant used in the ruler section when sending to Mimir, or the fake/anonymous no tenant defined should be the same. If fake, it should be fake in both application, or if anonymous, should be anonymous in both applications.

[hugree]

What this ever solved without disabling multitenancy? How to change fake to a different tenant value in Loki when sending recording rules to Mimir?

[pstibrany]

Internally we have a component called "alertmanager instance mapper", which maps "hosted logs" (Loki) tenant ID and "hosted metrics" (Mimir) into "hosted alerts" tenant ID (completely separate IDs, only used by alertmanager). Unfortunately this mapping depends on our internal tenants database (exposed via private API).

I'm not aware of publicly available component doing the same.

Technically all that needs to happen is to remap X-Scope-OrgID header values from Loki to alertmanager (Mimir). Perhaps simple nginx rule would do?