Can Mimir AlertManager be used for Loki Ruler?

[hobbsh]

Hi,

I have been unable to get alerts to actually notify when using Alert Rules in the Loki helm chart, pointing alertmanager_url at the Mimir Alertmanager instance. I've searched through the docs extensively but I can't seem to uncover what I'm missing. Is there a catch or is this just not designed to work?

I can see rule evaluating logs in the ruler and the alert is even Firing in the Grafana Alerting dashboard, but nothing is being notified and I have not found evidence of errors/misconfiguration anywhere.

[thmshmm]

hey, it should work, using it in this setup as well.

loki:
  structuredConfig:
    auth_enabled: true
    ruler:
      alertmanager_url: http://lgtm-distributed-mimir-alertmanager:8080/alertmanager

Do you have multi-tenancy enabled everywhere? Loki in that case forwards the X-Scope-OrgID header.

[hobbsh]

So I think remote_write was part of the issue, but now we have a tenant mismatch because the default tenant is anonymous on Mimir and fake on Loki. How is this typically reconciled? Should we set the tenant name to anonymous on the ruler remote_write clients?

[pstibrany]

If your Mimir is configured with multitenancy disabled, I'd expect that Mimr ignores the tenant coming from Loki.

Looking at the remote-write config options in Loki ruler, I'd perhaps try setting remote_write.add_org_id_header option to false, so that Loki doesn't report the tenant to alertmanager. (But I have no direct experience trying to setup Loki ruler with Mimir alertmanager, just guessing here.)

[hobbsh]

It looks like that option is only available in Loki 3.x - we are still on 2.9 because the loki-distributed chart does not support the new version yet.

[zktaiga]

Thought I'd chime in since I've made it work, was in a similar situation: different default tenants between Mimir/Loki. Ended up aligning the tenant names to match, and then just setting alertmanager_url.

By the way, I don't think remote_write has anything to do here. Rules are evaluated by the Loki Ruler and then sent to Mimir's Alertmanager (they don't go through Mimir). As far as I can see, remote_write just generates metrics and pushes them to Mimir for collection.

My 2 cents with the loki chart in simple scalable mode:

loki:
  auth_enabled: true
  rulerConfig:
    wal:
      dir: /var/loki/ruler-wal
    alertmanager_url: http://mimir-alertmanager.mimir.svc:8080/alertmanager
    external_url: https://grafana.com
    enable_alertmanager_v2: true
    enable_api: true
    external_labels:
      origin: loki
    remote_write:
      enabled: true
      client:
        url: http://mimir-nginx.mimir.svc:80/api/v1/push

This way, Grafana can connect to Loki's ruler API, you can create datasource-managed alerts (Loki ruler), they'll be created in object storage and then loaded by the ruler. Hope this helps!

[hobbsh]

Ok so what I had to do since I don't have access to disabling the org id header for the Loki ruler is to just add an AlertManager config for the fake tenant with mimirtool. Maybe I needed this either way, but with this remote_write config and a duplicate config for the fake tenant (just passing --id=fake to mimirtool alertmanager load like we do for anonymous tenant), it works!

        ruler:
          enable_api: true
          alertmanager_url: http://mimir-distributed-alertmanager.mimir:8080/alertmanager
          enable_alertmanager_v2: true
          enable_sharding: true
          remote_write:
            enabled: true
            clients:
              fake:
                url: http://mimir-distributed-gateway.mimir:8080/api/v1/push