Prometheus related several problems/questions

[tirelibirefe]

Hello,
I have an installed Strimzi 0.27.0 on my aws-eks k8s. The operator and kafka is in different namespaecs and the operator watches all namespaces. "*"

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:41:01Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.2-eks-0389ca3", GitCommit:"8a4e27b9d88142bbdd21b997b532eb6d493df6d2", GitTreeState:"clean", BuildDate:"2021-07-31T01:34:46Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.23) and server (1.21) exceeds the supported minor version skew of +/-1

I've installed Prometheus stuff as explained in documentation but there is no Prometheus service in namespace. This is the first problem.

$ k get svc -n kafka
NAME                             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                               AGE
alertmanager                     ClusterIP   10.100.66.200   <none>        9093/TCP                              4h17m
alertmanager-operated            ClusterIP   None            <none>        9093/TCP,9094/TCP,9094/UDP            4h17m
kuap-kafka-bootstrap             ClusterIP   10.100.43.123   <none>        9091/TCP,9092/TCP,9093/TCP            19h
kuap-kafka-brokers               ClusterIP   None            <none>        9090/TCP,9091/TCP,9092/TCP,9093/TCP   19h
kuap-kafka-connect-connect-api   ClusterIP   10.100.185.80   <none>        8083/TCP                              18h
kuap-zookeeper-client            ClusterIP   10.100.51.209   <none>        2181/TCP                              19h
kuap-zookeeper-nodes             ClusterIP   None            <none>        2181/TCP,2888/TCP,3888/TCP            19h
prometheus-operator              ClusterIP   None            <none>        8080/TCP                              17h

alertmanager pod keeps crashing: This is second problem.

k get pods -n kafka
NAME                                          READY   STATUS        RESTARTS   AGE
alertmanager-alertmanager-0                   0/2     Terminating   0          1s
kafka-client                                  1/1     Running       0          8h
kafkacat-67ff5975cd-wwgbg                     1/1     Running       0          8h
kuap-entity-operator-746bfc5675-2lm5r         3/3     Running       0          19h
kuap-kafka-0                                  1/1     Running       0          18h
kuap-kafka-1                                  1/1     Running       0          18h
kuap-kafka-2                                  1/1     Running       0          18h
kuap-kafka-connect-connect-76bf5bbc5d-9tw5s   1/1     Running       0          19h
kuap-kafka-exporter-56d449685d-qt265          1/1     Running       0          19h
kuap-zookeeper-0                              1/1     Running       0          19h
kuap-zookeeper-1                              1/1     Running       0          19h
kuap-zookeeper-2                              1/1     Running       0          19h
prometheus-operator-85d84bb848-7kld5          1/1     Running       0          17h

I have another Prometheus/Grafana/AlertManager in another namespace. When I install Strimzi/metrics, the other Promethus goes crazy. This is the third problem.

if there is a workaround or known issue(s), knowing them before dig into details would be great.

Thanks & Regards

[scholzj]

I've installed Prometheus stuff as explained in documentation but there is no Prometheus service in namespace. This is the first problem.

You do not seem to have Prometheus deployed. Just the Prometheus Opertor. So I guess that is why are you missing the service and why the alert manager is possibly crashing?

I have another Prometheus/Grafana/AlertManager in another namespace. When I install Strimzi/metrics, the other Promethus goes crazy. This is the third problem.

Not sure we can help with that since we do not really know how that is configured or used. In general, you do not need a separate Prometheus to monitor Strimzi. You can use the one you already have if you have it. Maybe they conflict with each other?

[scholzj]

CC @ppatierno @sknot-rh

[tirelibirefe]

Hello @scholzj it's nice to see you again; you helped me in 2019 and 2020 a lot. Now I am here again. Thanks for your fast feedback.

Actually prometheus is installed.

k get prometheus -n kafka
NAME         VERSION   REPLICAS   AGE
prometheus             1          48s
$ k get svc -n kafka
NAME                             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                               AGE
alertmanager                     ClusterIP   10.100.242.11   <none>        9093/TCP                              51s
alertmanager-operated            ClusterIP   None            <none>        9093/TCP,9094/TCP,9094/UDP            51s
my-kafka-bootstrap             ClusterIP   10.100.43.123   <none>        9091/TCP,9092/TCP,9093/TCP            27h
my-kafka-brokers               ClusterIP   None            <none>        9090/TCP,9091/TCP,9092/TCP,9093/TCP   27h
my-kafka-connect-connect-api   ClusterIP   10.100.185.80   <none>        8083/TCP                              27h
my-zookeeper-client            ClusterIP   10.100.51.209   <none>        2181/TCP                              27h
my-zookeeper-nodes             ClusterIP   None            <none>        2181/TCP,2888/TCP,3888/TCP            27h

describe prometheus

k describe prometheus -n kafka
Name:         prometheus
Namespace:    kafka
Labels:       app=strimzi
Annotations:  <none>
API Version:  monitoring.coreos.com/v1
Kind:         Prometheus
Metadata:
  Creation Timestamp:  2022-01-11T00:05:56Z
  Generation:          1
  Managed Fields:
    API Version:  monitoring.coreos.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
        f:labels:
          .:
          f:app:
      f:spec:
        .:
        f:additionalScrapeConfigs:
          .:
          f:key:
          f:name:
        f:alerting:
          .:
          f:alertmanagers:
        f:enableAdminAPI:
        f:podMonitorSelector:
          .:
          f:matchLabels:
            .:
            f:app:
        f:replicas:
        f:resources:
          .:
          f:requests:
            .:
            f:memory:
        f:ruleSelector:
          .:
          f:matchLabels:
            .:
            f:app:
            f:role:
        f:serviceAccountName:
        f:serviceMonitorSelector:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2022-01-11T00:05:56Z
  Resource Version:  56227286
  UID:               a97e0259-b112-4e70-bf1d-bfc57e8af667
Spec:
  Additional Scrape Configs:
    Key:   prometheus-additional.yaml
    Name:  additional-scrape-configs
  Alerting:
    Alertmanagers:
      Name:          alertmanager
      Namespace:     kafka
      Port:          alertmanager
  Enable Admin API:  false
  Pod Monitor Selector:
    Match Labels:
      App:   strimzi
  Replicas:  1
  Resources:
    Requests:
      Memory:  400Mi
  Rule Selector:
    Match Labels:
      App:               strimzi
      Role:              alert-rules
  Service Account Name:  prometheus-server
  Service Monitor Selector:
Events:  <none>

I just upgraded my other existing Prometheus to the latest release and when I install bundle operator under "kafka" namespace, things go crazy in my other Prometheus :

k get pods -n monitoring
NAME                                                     READY   STATUS        RESTARTS   AGE
alertmanager-prometheus-kube-prometheus-alertmanager-0   0/2     Terminating   0          5s
prometheus-grafana-77949b89cc-f8wrj                      3/3     Running       0          41m
prometheus-kube-prometheus-operator-67458985cb-6z6wt     1/1     Running       0          41m
prometheus-kube-state-metrics-79f9cf87df-6b2wh           1/1     Running       0          45m
prometheus-prometheus-kube-prometheus-prometheus-0       0/2     Terminating   0          7s
prometheus-prometheus-node-exporter-2grjf                1/1     Running       0          45m
prometheus-prometheus-node-exporter-bmvr7                1/1     Running       0          45m
prometheus-prometheus-node-exporter-ksk4q                1/1     Running       0          45m

I tried to install Prometheus staff without install operator (because I've Prometheus Operator in other namespace); the installation destructed the other prometheus .

k apply -f metrics/prometheus-install/strimzi-pod-monitor.yaml -n kafka
k apply -f metrics/prometheus-install/prometheus-rules.yaml -n kafka
k apply -f metrics/prometheus-install/prometheus.yaml -n kafka
k apply -f metrics/prometheus-install/alert-manager-config.yaml -n kafka
k apply -f metrics/prometheus-install/alert-manager.yaml -n kafka

..investigating; to be continued...

[scholzj]

Well, k describe prometheus -n kafka would really just mean you created the Prometheus custom resource. But if the Pod is not there (such as prometheus-prometheus-kube-prometheus-prometheus-0 in the other namespace), the operator didn't deployed it for some reason.

I guess your problem is that you have different versions of Prometheus Operator installed in your cluster. That does not work well because Kubernetes is not really multitenant and they have to share the CRDs etc. So I guess that might be breaking the other Prometheus cluster. I guess you would need to use the same version in both namespaces or use just one Prometheus and scrape the metrics from many namespaces.

I'm afraid I'm not really Prometheus expect, so I'm not sure I would be able to help much more since I do not really know how to troubleshoot it :-(.

[tirelibirefe]

I am totally agree with you. Actually having 2 Prometheus installations in the same k8s is not something I would prefer because of the troubles you mentioned. It would be hard to maintain.

I've modified strimzi-pod-monitor.yaml like below:

...
  namespaceSelector:
    matchNames:
      - kafka
...

Could you please advise; if I install following configs to the other/existing Prometheus namespace, can I monitor Strimzi-Kafka which is under "Kafka" namespace?

...or should I configure anything else?

k apply -f metrics/prometheus-install/strimzi-pod-monitor.yaml -n monitoring
k apply -f metrics/prometheus-install/prometheus-rules.yaml -n monitoring
k apply -f metrics/prometheus-install/alert-manager-config.yaml -n monitoring
k apply -f metrics/prometheus-install/alert-manager.yaml -n monitoring

[tirelibirefe]

@ppatierno @sknot-rh pls advise,

I modified strimzi-pod-monitor.yaml like below:
...
namespaceSelector:
matchNames:
- kafka
...

..and applied...

k create secret generic additional-scrape-configs --from-file=metrics/prometheus-install/prometheus-additional.yaml -n monitoring
k apply -f metrics/prometheus-install/strimzi-pod-monitor.yaml -n monitoring
k apply -f metrics/prometheus-install/prometheus-rules.yaml -n monitoring

The relevant part of my Prometheus is below:

...
    ## Namespaces to be selected for PrometheusRules discovery.
    ## If nil, select own namespace. Namespaces to be selected for ServiceMonitor discovery.
    ## See https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#namespaceselector for usage
    ##
    ruleNamespaceSelector: {}
...

...```
serviceMonitorSelector: {}
## Example which selects ServiceMonitors with label "prometheus" set to "somelabel"
# serviceMonitorSelector:
# matchLabels:
# prometheus: somelabel

## Namespaces to be selected for ServiceMonitor discovery.
##
serviceMonitorNamespaceSelector: {}
## Example which selects ServiceMonitors in namespaces with label "prometheus" set to "somelabel"
# serviceMonitorNamespaceSelector:
#   matchLabels:
#     prometheus: somelabel

...

$ k get prometheus prometheus-kube-prometheus-prometheus -n monitoring -o yaml

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
annotations:
meta.helm.sh/release-name: prometheus
meta.helm.sh/release-namespace: monitoring
creationTimestamp: "2022-01-11T00:03:55Z"
generation: 1
labels:
app: kube-prometheus-stack-prometheus
app.kubernetes.io/instance: prometheus
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/part-of: kube-prometheus-stack
app.kubernetes.io/version: 30.0.1
chart: kube-prometheus-stack-30.0.1
heritage: Helm
release: prometheus
name: prometheus-kube-prometheus-prometheus
namespace: monitoring
resourceVersion: "56226148"
uid: bd6da3f9-8aa4-45b9-8a50-de931db484b7
spec:
alerting:
alertmanagers:
- apiVersion: v2
name: prometheus-kube-prometheus-alertmanager
namespace: monitoring
pathPrefix: /
port: http-web
enableAdminAPI: false
externalUrl: http://prometheus.s2.ku.edu.tr/
image: quay.io/prometheus/prometheus:v2.32.1
listenLocal: false
logFormat: logfmt
logLevel: info
paused: false
podMonitorNamespaceSelector: {}
podMonitorSelector:
matchLabels:
release: prometheus
portName: http-web
probeNamespaceSelector: {}
probeSelector:
matchLabels:
release: prometheus
replicas: 1
retention: 10d
routePrefix: /
ruleNamespaceSelector: {}
ruleSelector:
matchLabels:
release: prometheus
securityContext:
fsGroup: 2000
runAsGroup: 2000
runAsNonRoot: true
runAsUser: 1000
serviceAccountName: prometheus-kube-prometheus-prometheus
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector:
matchLabels:
release: prometheus
shards: 1
version: v2.32.1

My existing Prometheus doesn't discover anything.

What am I missing? Could you please advise?

Thanks&Regards

[tirelibirefe]

I added following servicemonitor; nothing changed:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kafka-service-monitor
  labels:
    app: strimzi
spec:
  selector:
    matchLabels:
      strimzi.io/kind: Kafka
  namespaceSelector:
    matchNames:
      - kafka
  endpoints:
  #### job_name: kubernetes-cadvisor
  # configured in additional secret

  #### job_name: kube-state-metrics
  - port: metrics
    honorLabels: true
    interval: 10s
    scrapeTimeout: 10s
    path: /metrics
    scheme: http
    relabelings:
    - sourceLabels: [__meta_kubernetes_endpoints_name]
      separator: ;
      regex: prometheus-kube-state-metrics
      replacement: $1
      action: keep
    - separator: ;
      regex: __meta_kubernetes_service_label_(.+)
      replacement: $1
      action: labelmap
    - sourceLabels: [__meta_kubernetes_namespace]
      separator: ;
      regex: (.*)
      targetLabel: namespace
      replacement: $1
      action: replace
    - sourceLabels: [__meta_kubernetes_namespace]
      separator: ;
      regex: (.*)
      targetLabel: kubernetes_namespace
      replacement: $1
      action: replace
    - sourceLabels: [__meta_kubernetes_service_name]
      separator: ;
      regex: (.*)
      targetLabel: kubernetes_name
      replacement: $1
      action: replace
    - sourceLabels: [__meta_kubernetes_pod_node_name]
      separator: ;
      regex: (.*)
      targetLabel: node_name
      replacement: $1
      action: replace
    - sourceLabels: [__meta_kubernetes_pod_host_ip]
      separator: ;
      regex: (.*)
      targetLabel: node_ip
      replacement: $1
      action: replace

  #### job_name: node-exporter
  - port: metrics
    honorLabels: true
    interval: 10s
    scrapeTimeout: 10s
    path: /metrics
    scheme: http
    relabelings:
    - sourceLabels: [__meta_kubernetes_endpoints_name]
      separator: ;
      regex: prometheus-node-exporter
      replacement: $1
      action: keep
    - separator: ;
      regex: __meta_kubernetes_service_label_(.+)
      replacement: $1
      action: labelmap
    - sourceLabels: [__meta_kubernetes_namespace]
      separator: ;
      regex: (.*)
      targetLabel: namespace
      replacement: $1
      action: replace
    - sourceLabels: [__meta_kubernetes_namespace]
      separator: ;
      regex: (.*)
      targetLabel: kubernetes_namespace
      replacement: $1
      action: replace
    - sourceLabels: [__meta_kubernetes_service_name]
      separator: ;
      regex: (.*)
      targetLabel: kubernetes_name
      replacement: $1
      action: replace
    - sourceLabels: [__meta_kubernetes_pod_node_name]
      separator: ;
      regex: (.*)
      targetLabel: node_name
      replacement: $1
      action: replace
    - sourceLabels: [__meta_kubernetes_pod_host_ip]
      separator: ;
      regex: (.*)
      targetLabel: node_ip
      replacement: $1
      action: replace

  #### job_name: kubernetes-services
  # configured in additional secret

  #### job_name: kubernetes-pods
  # configured in additional secret

  #### job_name: kubernetes-nodes-kubelet
  # configured in additional secret

[tirelibirefe]

@Gentlemen; pls advise... Need to your adises...