Custom workflow metrics.

[Lejboelle]

Hi,

In KFP v1, it was possible to add custom workflow metrics, e.g., in the spec/templates field for a given component.
Ex.

- name: my-component
  metrics:
    prometheus:
      - name: component_duration
        help: run duration seconds
        labels:
        - key: name
          value: my-component-duration
        - key: duration
          value: run-time

Or even on a pipeline (spec) level.

However, since the template change in v2, this does not seem possible. At least, it doesn't work for me.
I tried adding metrics in the deploymentSpec/executors fields for a given component.

Anyone knows if it is possible to add custom workflow metrics in v2 pipelines?
One of the things I'm also interested in is getting the argo_workflows_count metric with at namespace (profile) label, which is not just the kubeflow label.

Alternatively, something like a prometheus push gateway should be utilized.