bug:"restricted database access" after upgrade from 0.24.3 -> 0.24.5

[knuurr]

Chart name and version
chart: victoria-metrics-k8s-stack
version: between 0.24.3 and 0.24.5

Describe the bug
I am investigating an issue where our dashboards fail to load after upgrade from 0.24.3 to 0.24.5 version of victoria-metrics-k8s-stack chart.

We deploy our dashboards mostly through Git and Argo CD, using appropriate labels.

Here is how example dashboard ConfigMap file may look when provisioned:

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-dashboard
  namespace: monitoring
  labels:
    grafana_dashboard: "1"
  annotations:
    grafana_folder: "infra"
data:
  dashboard-data.json: |-
    # here goes json for dashboard
    # ....

When being on 0.24.5 (and 0.25.x, I didn't test any higher version) many (majority) of our dashboards isn't visible in dashboard UI at all. In Pod log this is what I can find:

logger=provisioning.dashboard type=file name=default t=2024-11-22T14:16:09.540294443Z level=warn msg="Not saving new dashboard due to restricted database access" provisioner=default file=/var/lib/grafana/dashboards/default/infra/elasticache.json folderId=0  

logger=provisioning.dashboard type=file name=default t=2024-11-22T14:16:09.545660724Z level=warn msg="Not saving new dashboard due to restricted database access" provisioner=default file=/var/lib/grafana/dashboards/default/infra/clickhouse.json folderId=0  

logger=provisioning.dashboard type=file name=default t=2024-11-22T14:16:19.469646818Z level=warn msg="Not saving new dashboard due to restricted database access" provisioner=default file=/var/lib/grafana/dashboards/default/infra/bees.json folderId=0  

logger=provisioning.dashboard type=file name=default t=2024-11-22T14:16:19.471034211Z level=warn msg="Not saving new dashboard due to restricted database access" provisioner=default file=/var/lib/grafana/dashboards/default/kubernetes-views-namespaces.json folderId=0  

logger=provisioning.dashboard type=file name=default t=2024-11-22T14:16:19.472675922Z level=warn msg="Not saving new dashboard due to restricted database access" provisioner=default file=/var/lib/grafana/dashboards/default/infra/chromium-server.json folderId=0  

logger=provisioning.dashboard type=file name=default t=2024-11-22T14:16:19.473087614Z level=warn msg="Not saving new dashboard due to restricted database access" provisioner=default file=/var/lib/grafana/dashboards/default/infra/k8s-staging-2-chromium-metrics-exporter.json folderId=0  

logger=provisioning.dashboard type=file name=default t=2024-11-22T14:16:19.473420462Z level=warn msg="Not saving new dashboard due to restricted database access" provisioner=default file=/var/lib/grafana/dashboards/default/alertmanager-overview.json folderId=0  

logger=provisioning.dashboard type=file name=default t=2024-11-22T14:16:19.4806884Z level=warn msg="Not saving new dashboard due to restricted database access" provisioner=default file=/var/lib/grafana/dashboards/default/infra/clickhouse.json folderId=0  

logger=provisioning.dashboard type=file name=default t=2024-11-22T14:16:19.485881429Z level=warn msg="Not saving new dashboard due to restricted database access" provisioner=default file=/var/lib/grafana/dashboards/default/victoriametrics-cluster.json folderId=0

Those are the only values that are related to dashboard provisioning:

            grafana:
              enabled: true
              # ...
              sidecar:
                dashboards:
                  provider:
                    foldersFromFilesStructure: true
                  folderAnnotation: "grafana_folder"
                  searchNamespace: monitoring
                datasources:
                  enabled: true
                notifiers:
                  enabled: true
                alerts:
                  enabled: true

Could you help me investigate potential issue here? Could it be change in chart template/deployment? Could it be change in Grafana itself (or any components used)?

[AndrewChubatiuk]

could you try to reinstall dashboards by disabling and then reenabling them again. it looks like grafana issues which happens during an attempt to import duplicated dashboards

[AndrewChubatiuk]

hi @knuurr
have you resolved an issue?