Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Velero Service Monitor is not discovered by Prometheus Operator (kube-prometheus -stack) #609

Open
darnone opened this issue Jul 24, 2024 · 4 comments

Comments

@darnone
Copy link

darnone commented Jul 24, 2024

What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.)
I have deployed velero into a namespace called velero using helmfile and I have prometheus-operator running in a namespace called monitoring. I have activated metrics with the following values:

metrics:
  enabled: true
  scrapeInterval: 30s
  scrapeTimeout: 10s

  # Pod annotations for Prometheus
  podAnnotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8085"
    prometheus.io/path: "/metrics"

  serviceMonitor:
    autodetect: false
    enabled: true
    annotations: 
      "helm.sh/hook": post-install,post-upgrade
    namespace: monitoring

The service monitor is created in the monitoring namespace but prometheus does not see it as a target. The service monitor produced is:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: velero
  namespace: monitoring
  annotations:
    helm.sh/hook: post-install,post-upgrade
  labels:
    app.kubernetes.io/name: velero
    app.kubernetes.io/instance: velero
    app.kubernetes.io/managed-by: Helm
    helm.sh/chart: velero-7.1.1
spec:
  namespaceSelector:
    matchNames:
      - velero
  selector:
    matchLabels:
      app.kubernetes.io/name: velero
      app.kubernetes.io/instance: velero
  endpoints:
  - port: http-monitoring
    interval: 30s
    scrapeTimeout: 10s

What did you expect to happen:

I expected prometheus to find the servicemonitor under service discovery and targets in prometheus UI and scrape metrics

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

nothing appears abnormal in the prometheus or velero logs

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

I am using helmfile v0.166.0
The kube-prometheus-stack is working for default k8s services

Environment:

  • helm version (use helm version): v3.15.3
  • helm chart version and app version: velero-7.1.1
  • Kubernetes version: v1.28.11-eks-db838b0
  • Kubernetes installer & version: asdf
  • Cloud provider or hardware configuration: AWS EKS
  • OS (e.g. from /etc/os-release): Amazon EC2 Linux
@darnone
Copy link
Author

darnone commented Jul 24, 2024

I have written to the Velero-users channel on clock but I have not received a reply.

@darnone darnone changed the title Velero Service Monitor I snot discovered on Velero Service Monitor is not discovered by Prometheus Operator (kube-prometheus -stack) Jul 24, 2024
@darnone
Copy link
Author

darnone commented Jul 25, 2024

I am attaching the velero pod and deployment
velero-manifests.zip

@phac008
Copy link

phac008 commented Jul 25, 2024

try adding additionalLabels to serviceMonitor values

 serviceMonitor:
    additionalLabels:
      release: kube-prometheus-stack

@darnone
Copy link
Author

darnone commented Jul 25, 2024

That did it. What led you find that fix?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants