Adjust Custom Resource configuration to support open metric types StateSet and Info

[chrischdi]

What would you like to be added:

Kube-state-metrics v2.5.0 added the experimental Custom-Resource State Metrics #1710 (🎉 awesome 👍 ).

The current implementation allows the definition of metrics which directly refer to:

• numbers (all kinds of int or float)
• strings (only if they can be parsed to a float)
• or booleans (which result in values 1 or 0)

because the getNum function expects that 0.

This kind of metric maps to the definition of a Gauge in the OpenMetrics specification.

For the [Cluster API] project we also want to implement Info and StateSet metrics.

Because the current implementation always expects to get a numbered value due to calling getNum / the implementation of getNum it is currently not possible to define metrics which are of type Info or StateSet.

Why is this needed:

To support the additional metric types Info and StateSet.

Info and StateSet already get exposed by kube-state-metrics for core kubernetes resources. E.g.:

• Info: kube_pod_info
• StateSet: kube_pod_status_phase

Describe the solution you'd like

By additional configuration options at the MetricsPre struct in

kube-state-metrics/pkg/customresourcestate/config.go

Line 36 in 138e1e6

type MetricPer struct {

.

It may be worth to consider the following user stories:

• As a user I would like to define a static value for a custom resource metrics for creating metrics of type Info.
• As a user I would like to define a list of values for a custom resource metric for creating metrics of type StateSet.
  • This list of values gets compared to the actual value determined via path which then results in the value of 1 or 0 (like a boolean).
  • For a Custom Resource, each value of the list results in a metric entry but each metric entry differs on a label which has the value as its value.

For Info metrics: it may be enough to be able to set a fixed value of 1 instead of parsing a value from a path.
The other functionality would be available via the LabelsFromPath variable.

Additional context

• Slack discussion: https://kubernetes.slack.com/archives/CJJ529RUY/p1654593034854759
• Cluster API metrics issue: Cluster API State Metrics implementation kubernetes-sigs/cluster-api#6458
• Cluster API State metrics proposal (contains metrics we want to implement): https://github.com/kubernetes-sigs/cluster-api/blob/main/docs/proposals/20220411-cluster-api-state-metrics.md

[chrischdi]

I did a small POC implementation which would fullfill both needs:

https://github.com/kubernetes/kube-state-metrics/compare/master...chrischdi:poc-additional-metric-types?expand=1

Of course this would need more considerations and note that this is only a "dirty hack implementation":

• Should the config be structured in some other way to have a cleaner distinction between the metric types?!
• This implementation should implement an additional validation to ensure only one of Value, ValueFrom or ValueFromList is set.

Edit: example config:

kind: CustomResourceStateMetrics
spec: 
  resources: 
    - subsystem: machinedeployment
      namespace: capi
      groupVersionKind: 
        group: cluster.x-k8s.io
        kind: MachineDeployment
        version: v1beta1
      metrics: 
        - name: spec_replicas
          each: 
            path: 
              - spec
              - replicas
        - name: info
          each: 
            value: 1
            labelsFromPath:
              version: [spec, template, spec, version]
        - name: phase
          each: 
            path:
            - status
            - phase
            valueFromList:
            - Running
            - Foo

[chrischdi]

cc @mrueg follow up for the short slack discussion we had 😊

[chrischdi]

@mrueg or others, what would be best to continue on this issue?

Seen from a configuration perspective I have two kind of versions in my head:

1. The chaotic one (as linked above), which I think is not that good from a UX
2. Create a kind of typed configuration, similar to what is done in kubernetes' persistentVolumeClaim so each type could have its own struct for specific configuration options (xref POC implementation):

metrics:
  # a gauge metric
  - name: spec_replicas
    each: 
      path: 
        - spec
        - replicas
      gauge: {}
  # a info metric
  - name: info
    each: 
      info: {}
      labelsFromPath:
        version: [spec, template, spec, version]
  # a stateSet metric
  - name: phase
    each: 
      path:
      - status
      - phase
      stateSet:
        list:
        - Running
        - ScalingUp
        labelName: phase

I'd be happy to continue working on this 😊

[chrischdi]

I updated the POC to work with feedback from @bavarianbidi (thx) and changed the config layout a little bit. Note: code is not really cleaned up but works for the poc :-)

kind: CustomResourceStateMetrics
spec: 
  resources: 
    - subsystem: machinedeployment
      namespace: capi
      groupVersionKind: 
        group: cluster.x-k8s.io
        kind: MachineDeployment
        version: v1beta1
      metrics:
        # a gauge per metric
        - name: status_conditions
          each: 
            gauge:
              path: 
                - status
                - conditions
              valueFrom:
                - status
              labelFromKey: reason
              labelsFromPath:
                type: [type]
        # a gauge metric
        - name: spec_replicas
          each: 
            gauge:
              path: 
                - spec
                - replicas
        # a info metric
        - name: info
          each: 
            info:
              labelsFromPath:
                version: [spec, template, spec, version]
        # a stateSet metric
        - name: phase
          each: 
            stateSet:
              path:
              - status
              - phase
              list:
              - Running
              - ScalingUp
              labelName: phase

example metrics output:

capi_machinedeployment_status_conditions{type="Available"} 0
capi_machinedeployment_status_conditions{type="Ready"} 0
capi_machinedeployment_spec_replicas 3
capi_machinedeployment_info{version="v1.23.3"} 1
capi_machinedeployment_phase{phase="Running"} 0
capi_machinedeployment_phase{phase="ScalingUp"} 1

Object used:

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  annotations:
    machinedeployment.clusters.x-k8s.io/revision: "1"
  creationTimestamp: "2022-06-26T18:08:09Z"
  generation: 1
  labels:
    cluster.x-k8s.io/cluster-name: capi-quickstart
    topology.cluster.x-k8s.io/deployment-name: md-0
    topology.cluster.x-k8s.io/owned: ""
  name: capi-quickstart-md-0-bsl6p
  namespace: default
  ownerReferences:
  - apiVersion: cluster.x-k8s.io/v1beta1
    kind: Cluster
    name: capi-quickstart
    uid: 5a84fae0-3623-4ff4-a54f-8f5a2b82762a
  resourceVersion: "4443"
  uid: b7ccc503-f2c5-4b86-8504-e731eba2d59f
spec:
  clusterName: capi-quickstart
  minReadySeconds: 0
  progressDeadlineSeconds: 600
  replicas: 3
  revisionHistoryLimit: 1
  selector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: capi-quickstart
      topology.cluster.x-k8s.io/deployment-name: md-0
      topology.cluster.x-k8s.io/owned: ""
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        cluster.x-k8s.io/cluster-name: capi-quickstart
        topology.cluster.x-k8s.io/deployment-name: md-0
        topology.cluster.x-k8s.io/owned: ""
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
          kind: KubeadmConfigTemplate
          name: capi-quickstart-md-0-bootstrap-slfbb
          namespace: default
      clusterName: capi-quickstart
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: DockerMachineTemplate
        name: capi-quickstart-md-0-infra-rgk5m
        namespace: default
      version: v1.23.3
status:
  conditions:
  - lastTransitionTime: "2022-06-26T18:08:09Z"
    message: Minimum availability requires 3 replicas, current 0 available
    reason: WaitingForAvailableMachines
    severity: Warning
    status: "False"
    type: Ready
  - lastTransitionTime: "2022-06-26T18:08:09Z"
    message: Minimum availability requires 3 replicas, current 0 available
    reason: WaitingForAvailableMachines
    severity: Warning
    status: "False"
    type: Available
  observedGeneration: 1
  phase: ScalingUp
  replicas: 3
  selector: cluster.x-k8s.io/cluster-name=capi-quickstart,topology.cluster.x-k8s.io/deployment-name=md-0,topology.cluster.x-k8s.io/owned=
  unavailableReplicas: 3
  updatedReplicas: 3