Add docs on building your own service graphs (#3804)

Co-authored-by: Kim Nylander <[email protected]>
Co-authored-by: Ilia Mochalov <[email protected]>
Co-authored-by: Kim Nylander <[email protected]>

docs/sources/tempo/metrics-generator/service_graphs/custom-service-graphs.md

@@ -0,0 +1,67 @@
+---
+title: Create custom service graphs
+menuTitle: Create custom service graphs
+description: Build your own service graphs using the Node Graph visualization from metrics generated by Tempo.
+weight: 300
+---
+
+# Create custom service graphs
+
+You can craft custom service graphs based on the metrics generated from metrics-generator.
+
+## Before you begin
+
+For custom service graphs, you'll use:
+- Tempo metrics-generator
+- Grafana deployment of version v10 or higher
+- Grafana [Node graph panel](https://grafana.com/docs/grafana/latest/panels-visualizations/visualizations/node-graph/)
+
+The metrics-generator creates multiple metrics, including `traces_service_graph_request_total`. This metric holds:
+- relationships between services
+- total number of requests performed between services
+
+### Create one or more Grafana dashboards
+1. In Grafana, [create a new dashboard](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/create-dashboard/).
+2.  Add two variables:
+    - Data source of type `Prometheus`.
+  ![Custom service graph Grafana data source variable](../custom-service-graph-var-datasource.png)
+    - Service of type `Label values`, enable multi-value option
+  ![Custom service graph service variable](../custom-service-graph-var-service.png)
+
+### Add a panel
+
+1. Create a panel with a single query called `edges`.
+2. Select your Prometheus data source with metrics from the metrics-generator.
+3. Query using the following example:
+   ```
+   label_join(
+     label_join(
+       label_join(
+         sum(increase(traces_service_graph_request_total{server=~"$service"}[5m])) by (server, client) > 0
+         or
+         sum(increase(traces_service_graph_request_total{client=~"$service"}[5m])) by (server, client) > 0,
+       "source", "", "client"),
+     "target", "", "server"),
+   "id", "-", "server", "client")
+   ```
+4. Use Instant query type.
+5. If you encounter issues, switch to the Table data visualization. Refer to the [Node graph panel](/docs/grafana/latest/panels-visualizations/visualizations/node-graph/) documentation for more options and expected data shape.
+
+![Custom service graph panel view](../custom-service-graph-with-query.png)
+
+All data transformations are done with the Prometheus `label_join` operators.
+
+Explanation of the query:
+
+- The first `label_join` creates a new field `id` which is required by the Node graph panel.
+- The second and third `label_join` copy the client and server labels to source and target respectively, as these names are also required by Node graph panel.
+- `traces_service_graph_request_total` is queried twice with the `OR` operator to get a combination of requests from and to all of the selected services.
+
+This query gets most of the job done. However, it's limitations can't be compensated for even using the Grafana transform data feature.
+These limitations include:
+
+- Unable to add request stats to nodes and edges, such as req/sec and error rates
+- Unable to add custom icons for nodes
+
+These limitations can be overcome by wrapping the Prometheus query into a REST API, which provides more flexibility with query results data transformations.
+You can use [Grafana Infinity data source](/grafana/plugins/yesoreyeram-infinity-datasource/) to visualize data from REST API.

docs/sources/tempo/metrics-generator/service_graphs/enable-service-graphs.md

@@ -4,7 +4,7 @@ aliases:
 - /docs/tempo/latest/metrics-generator/service_graphs/
 title: Enable service graphs
 description: Learn how to enable service graphs
-weight: 300
+weight: 200
 ---
 
 ## Enable service graphs

docs/sources/tempo/metrics-generator/service_graphs/estimate-cardinality.md

@@ -2,7 +2,7 @@
 title: Estimate cardinality from traces
 menuTitle: Estimate cardinality
 description: Service graphs help you understand the structure of a distributed system and the connections and dependencies between its components.
-weight: 300
+weight: 400
 ---
 
 ## Estimate cardinality from traces

docs/sources/tempo/metrics-generator/service_graphs/metrics-queries.md

@@ -2,7 +2,7 @@
 title: Service graph metrics queries
 menuTitle: Metrics queries
 description: Use PromQL queries to access metrics from service graphs
-weight: 300
+weight: 500
 ---
 
 # Service graph metrics queries