Draft: Prometheus from control plane

[voximity]

Targets:

• Control plane should spin up and manage a Prometheus container
• Prometheus container should actively scrape agents running nodes
• Guarantee outcomes using PromQL queries

[Meshiest]

I think we may want the option to connect to an existing Prometheus container so the control plane can be run in a container without needing docker in docker mounts

[voximity]

I think we may want the option to connect to an existing Prometheus container so the control plane can be run in a container without needing docker in docker mounts

That's fine, but the control plane needs to be able to write to the Prometheus config file wherever the Prometheus is, so currently having it in Docker only is the most feasible

[Meshiest]

https://prometheus.io/docs/guides/file-sd/

the Prometheus instance will listen for changes to the file and automatically update the scrape target list

scrape_configs:
- job_name: 'node'
  file_sd_configs:
  - files:
    - 'targets.json'

allows for targets.json to have

[
  {
    "labels": {
      "job": "node"
    },
    "targets": [
      "localhost:9100"
    ]
  }
]

[Meshiest]

this looks more like what we want https://github.com/prometheus/pushgateway

[voximity]

this looks more like what we want https://github.com/prometheus/pushgateway

I think we would have to modify snarkOS to make this work since they use a scrape exporter by default. The metrics exporter they use does support push gateways though

[Meshiest]

agent can scrape the node locally (with env + own node id context), then push it to a desired service

[voximity]

Closing as we've decided to not make the control plane spin up its own Prometheus instance and instead let agents optionally use push gateway to a Prometheus server specified by the control plane.