OpenCost Multi Cluster Setup and documentation

[wamak9]

Is your feature request related to a problem? Please describe.
Documentations on multi cluster setup with opencost.

Describe the solution you'd like
I am trying to figure out how opencost can be expanded for a multi cluster setup ? do I install it individually on every cluster I have ? Is there any client server process I can follow ?
Is it similar to kubecost setup?
At the moment, there is no clear way(atleast for me ) to understand how open cost be setup for a multi cluster use case.

Describe alternatives you've considered
Add examples and documentation on this can be setup.

Additional context
Add examples and check in under the opencost.

[mattray]

There's no public documentation available yet but a few folks have got this working. Perhaps @Pokom can weigh-in?

I'm going to move this over to https://github.com/opencost/opencost-website since it's documentation-related

[wamak9]

Thanks for the update @mattray , It would be nice if we can get few examples and documentation setup who wants to try multi cluster setup.
@Pokom -- looking forward for the examples.

[jainsourabh2]

Is there a support for multi cluster available? @Pokom Can you help with the steps for the same?

[pandvan]

We are looking for the same. Struggling to setup OpenCost or Kubecost in a single cluster that makes use of vCluster to provision virtual clusters inside it. We were able to consolidate Prometheus metrics in a single, central instance. We would like to achieve the same also for cost monitoring.
If someone has any tip on how to setup this, it would be very appreciated.

[AjayTripathy]

Hi @pandvan we haven't set up Opencost in such a configuration, but would love documentation if you're able to figure it out.

[pandvan]

Basically our architecture consists of a main (hosting) EKS cluster that contains (hosts) several "virtual" k3s clusters using vCluster project by Loft. Nodes are tainted in order to allow schedule of worklows only for a specific cluster (main or one of vclusters).
In the main cluster we've a full setup of the Kube Prometheus Stack that scrapes metrics only for nodes belonging to main cluster (leveraging a feature of Prometheus Operator that can create a service with only a specific subset of node endpoints).
Each vCluster, on the other hand, has an agent-only Kube Prometheus Stack deployment that grabs metrics for such vCluster and its dedicated nodes. Then all metrics scraped by kubelet and kube-state-metrics jobs are sent to the main stack using remote write and adding few labels to mark with vCluster they are related to.

The issue with this setup is how to tell OpenCost / Kubecost to make use of these metrics that are collected from a "different" Kubernetes cluster as it directly talks with API server and emit kube-state-metrics likes metrics to have a "view" of all workloads.