Mimir and S3 configuration to OCI Object Storage

[lee-spo]

Hi All,

Im very new to this but I'm trying to deploy mimir into a Kubernetes cluster and have the backend storage be on OCI Object storage using the S3-compatible storage options.

I cant get the deployment to work, I'm using the small.yaml as my base and then adding what I think is the correct common sections to add the storage into it, though the small.yaml already has some persistent storage so I've removed those lines for the items the docs seem to suggest you can use S3, could anyone help with were I might be going wrong in the config file for the values.yaml?

mimir:
  structuredConfig:
    common:
      storage:
        backend: s3
        s3:
          endpoint: https://namespace.compat.objectstorage.region.oraclecloud.com
          region: region
          secret_access_key: password
          access_key_id: password

    blocks_storage:
      s3:
        bucket_name: mimir-blocks
    alertmanager_storage:
      s3:
        bucket_name: mimir-alertmanager
    ruler_storage:
      s3:
        bucket_name: mimir-ruler

alertmanager:
  replicas: 2
  resources:
    limits:
      memory: 1.4Gi
    requests:
      cpu: 1
      memory: 1Gi
  statefulSet:
    enabled: true

compactor:
  persistentVolume:
    size: 20Gi
  resources:
    limits:
      memory: 2.1Gi
    requests:
      cpu: 1
      memory: 1.5Gi

distributor:
  replicas: 2
  resources:
    limits:
      memory: 5.7Gi
    requests:
      cpu: 2
      memory: 4Gi

ingester:
  persistentVolume:
    size: 50Gi
  replicas: 3
  resources:
    limits:
      memory: 12Gi
    requests:
      cpu: 3.5
      memory: 8Gi
  topologySpreadConstraints: {}
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: target # support for enterprise.legacyLabels
                operator: In
                values:
                  - ingester
          topologyKey: 'kubernetes.io/hostname'

        - labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/component
                operator: In
                values:
                  - ingester
          topologyKey: 'kubernetes.io/hostname'

  zoneAwareReplication:
    topologyKey: 'kubernetes.io/hostname'

admin-cache:
  enabled: true
  replicas: 2

chunks-cache:
  enabled: true
  replicas: 2

index-cache:
  enabled: true
  replicas: 3

metadata-cache:
  enabled: true

results-cache:
  enabled: true
  replicas: 2

minio:
  enabled: false

overrides_exporter:
  replicas: 1
  resources:
    limits:
      memory: 128Mi
    requests:
      cpu: 100m
      memory: 128Mi

querier:
  replicas: 1
  resources:
    limits:
      memory: 5.6Gi
    requests:
      cpu: 2
      memory: 4Gi

query_frontend:
  replicas: 1
  resources:
    limits:
      memory: 2.8Gi
    requests:
      cpu: 2
      memory: 2Gi

ruler:
  replicas: 1
  resources:
    limits:
      memory: 2.8Gi
    requests:
      cpu: 1
      memory: 2Gi

store_gateway:
  persistentVolume:
    size: 10Gi
  replicas: 3
  resources:
    limits:
      memory: 2.1Gi
    requests:
      cpu: 1
      memory: 1.5Gi
  topologySpreadConstraints: {}
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: target # support for enterprise.legacyLabels
                operator: In
                values:
                  - store-gateway
          topologyKey: 'kubernetes.io/hostname'

        - labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/component
                operator: In
                values:
                  - store-gateway
          topologyKey: 'kubernetes.io/hostname'
  zoneAwareReplication:
    topologyKey: 'kubernetes.io/hostname'

nginx:
  replicas: 1
  resources:
    limits:
      memory: 731Mi
    requests:
      cpu: 1
      memory: 512Mi

# Grafana Enterprise Metrics feature related
admin_api:
  replicas: 1
  resources:
    limits:
      memory: 128Mi
    requests:
      cpu: 100m
      memory: 64Mi

gateway:
  replicas: 1
  resources:
    limits:
      memory: 731Mi
    requests:
      cpu: 1
      memory: 512Mi

[56quarters]

Hello,

In what way is the deployment not working? Can you share some logs from components that connect to object storage (ingesters, compactor, store-gateways)?

[lee-spo]

Hi Nick,

Thanks for the quick reply.

Basically, the ruler pod is in a CrashLoopBackOff state and all other pods are pending

Pods shown below

NAME READY STATUS RESTARTS AGE
mimir-alertmanager-0 0/1 Pending 0 10h
mimir-chunks-cache-0 0/2 Pending 0 10h
mimir-chunks-cache-1 0/2 Pending 0 10h
mimir-compactor-0 0/1 Pending 0 10h
mimir-distributor-d7588c756-j84v5 0/1 Pending 0 10h
mimir-distributor-d7588c756-tw576 0/1 Pending 0 10h
mimir-index-cache-0 2/2 Running 0 10h
mimir-index-cache-1 0/2 Pending 0 10h
mimir-index-cache-2 2/2 Running 0 10h
mimir-ingester-zone-a-0 0/1 Pending 0 10h
mimir-ingester-zone-b-0 0/1 Pending 0 10h
mimir-ingester-zone-c-0 0/1 Pending 0 10h
mimir-metadata-cache-0 0/2 Pending 0 10h
mimir-nginx-645dccf44f-p8fwq 0/1 Pending 0 10h
mimir-overrides-exporter-6dc7599c7f-bt457 1/1 Running 0 10h
mimir-querier-7dcbcd6b49-8ql5c 0/1 Pending 0 10h
mimir-query-frontend-5765d4c5f8-58rfl 0/1 Pending 0 10h
mimir-query-scheduler-c64f746c8-p9rj7 1/1 Running 0 10h
mimir-query-scheduler-c64f746c8-tl6xl 1/1 Running 0 10h
mimir-results-cache-0 2/2 Running 0 10h
mimir-results-cache-1 2/2 Running 0 10h
mimir-rollout-operator-56995dd678-b9qw8 1/1 Running 0 10h
mimir-ruler-d67bc77cb-hqtsz 0/1 CrashLoopBackOff 127 (2m7s ago) 10h
mimir-store-gateway-zone-a-0 0/1 Pending 0 10h
mimir-store-gateway-zone-b-0 0/1 Pending 0 10h
mimir-store-gateway-zone-c-0 0/1 Pending 0 10h

Log from the ruler Pod, Interestingly doesn't mention object storage

ts=2024-08-20T07:03:33.942899507Z caller=main.go:225 level=info msg="Starting application" version="(version=2.13.0, branch=HEAD, revision=4775ec1)"
ts=2024-08-20T07:03:33.946764525Z caller=server.go:352 level=info msg="server listening on addresses" http=[::]:8080 grpc=[::]:9095
ts=2024-08-20T07:03:33.953042942Z caller=memberlist_client.go:435 level=info msg="Using memberlist cluster label and node name" cluster_label= node=mimir-ruler-d67bc77cb-hqtsz-64b59f4a
ts=2024-08-20T07:03:33.95562438Z caller=memberlist_client.go:541 level=info msg="memberlist fast-join starting" nodes_found=1 to_join=4
ts=2024-08-20T07:03:33.955951649Z caller=memberlist_client.go:561 level=info msg="memberlist fast-join finished" joined_nodes=1 elapsed_time=1.399379ms
ts=2024-08-20T07:03:33.955967168Z caller=memberlist_client.go:573 level=info phase=startup msg="joining memberlist cluster" join_members=dns+mimir-gossip-ring.mimir.svc.cluster.local.:7946
ts=2024-08-20T07:03:33.957031583Z caller=memberlist_client.go:580 level=info phase=startup msg="joining memberlist cluster succeeded" reached_nodes=1 elapsed_time=1.059295ms
ts=2024-08-20T07:03:33.964076391Z caller=resolver.go:87 level=error msg="failed to lookup IP addresses" host=mimir-metadata-cache.mimir.svc err="no such host"
ts=2024-08-20T07:03:33.964099555Z caller=resolver.go:134 level=warn msg="IP address lookup yielded no results. No host found or no addresses found" host=mimir-metadata-cache.mimir.svc
ts=2024-08-20T07:03:33.964457843Z caller=log.go:103 level=error msg="error running application" err="no server address resolved for \nruler-storage-cache\ngithub.com/grafana/mimir/pkg/ruler.wrapBucketWithCache\n\t/__w/mimir/mimir/pkg/ruler/storage.go:67\ngithub.com/grafana/mimir/pkg/ruler.NewRuleStore\n\t/__w/mimir/mimir/pkg/ruler/storage.go:47\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).initRulerStorage\n\t/__w/mimir/mimir/pkg/mimir/modules.go:807\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:136\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:852\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:227\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:271\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nerror initialising module: ruler-storage\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:852\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:227\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:271\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"

[lee-spo]

Hi,

Just an update to say ive now fixed this and things seem to be running, basically it was two problems

1. I had used the small.yaml and left in the cache options i.e admin-cache and chunks-cache which is redis setup but I wont't / didn't setup redis
2. The compartment I placed the buckets into wasn't the tenancy-compatible S3 compartment so had to move the buckets as in OCI you can only have compatible S3 buckets in a single compartment across the tenancy

mimir:
  structuredConfig:
    common:
      storage:
        backend: s3
        s3:
          endpoint: namespace.compat.objectstorage.region.oraclecloud.com
          region: region
          secret_access_key: "key"
          access_key_id: "id"
          insecure: false

    blocks_storage:
      s3:
        bucket_name: mimir-blocks
    alertmanager_storage:
      s3:
        bucket_name: mimir-alert
    ruler_storage:
      s3:
        bucket_name: mimir-ruler

alertmanager:
  replicas: 2
  statefulSet:
    enabled: true

compactor:
  persistentVolume:
    size: 20Gi

distributor:
  replicas: 2

ingester:
  persistentVolume:
    size: 50Gi
  replicas: 3
  topologySpreadConstraints: {}
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: target # support for enterprise.legacyLabels
                operator: In
                values:
                  - ingester
          topologyKey: 'kubernetes.io/hostname'

        - labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/component
                operator: In
                values:
                  - ingester
          topologyKey: 'kubernetes.io/hostname'

  zoneAwareReplication:
    topologyKey: 'kubernetes.io/hostname'

minio:
  enabled: false

overrides_exporter:
  replicas: 1

querier:
  replicas: 1

query_frontend:
  replicas: 1

ruler:
  replicas: 1

store_gateway:
  persistentVolume:
    size: 10Gi
  replicas: 3
  topologySpreadConstraints: {}
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: target # support for enterprise.legacyLabels
                operator: In
                values:
                  - store-gateway
          topologyKey: 'kubernetes.io/hostname'

        - labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/component
                operator: In
                values:
                  - store-gateway
          topologyKey: 'kubernetes.io/hostname'
  zoneAwareReplication:
    topologyKey: 'kubernetes.io/hostname'

nginx:
  replicas: 1

# Grafana Enterprise Metrics feature related
admin_api:
  replicas: 1

gateway:
  replicas: 1

[56quarters]

Thanks for the follow up!

[lee-spo]

no problem at all, always annoys me when I see people saying its fixed without any details of how so...