traces_service_graph_request_messaging_system_seconds metrics is not produced, while enabled.

[cbos]

I am running Tempo 2.6.1.
My full Tempo config can be found here: https://github.com/cbos/observability-toolkit/blob/main/config/tempo/tempo-config.yaml

For the service graph processor I have I this config:

  processor:
    service_graphs:
      wait: 30s
      max_items: 10000
      workers: 10
      dimensions: []
      #enable_client_server_prefix: false
      peer_attributes:
        - peer.service
        - db.name
        - db.system
        - messaging.destination.name
      # Enables additional labels for services and virtual nodes. Default false
      enable_virtual_node_label: true
      # If enabled another histogram will be produced for interactions over messaging systems middlewares
      # If this feature is relevant over long time ranges (high latencies) - consider increasing
      # `wait` value for this processor. Default false
      # enable_messaging_system_latency_histogram: false
      enable_messaging_system_latency_histogram: true

All metrics are generated:

But traces_service_graph_request_messaging_system_seconds is missing. That is introduced with #3453

My test setup is a modified setup of OpenTelemetry demo:
https://github.com/cbos/opentelemetry-demo/tree/jms-support

Producer
https://github.com/cbos/opentelemetry-demo/tree/jms-support/src/adservice
JMS system is ActiveMQ (messagequeue in the setup)
Consumer
https://github.com/cbos/opentelemetry-demo/tree/jms-support/src/analytics

So the client-server matching is done correctly and the connection_type is 'messaging_system'.

tempo/modules/generator/processor/servicegraphs/servicegraphs.go

Line 380 in deece3c

if p.Cfg.EnableMessagingSystemLatencyHistogram && e.ConnectionType == store.MessagingSystem {

I checked the logging of Tempo, but I don't see the warning "producerSpanEndTime must be smaller than consumerSpanStartTime. maybe the peers clocks are not synced"

Based on what I can see all conditions are met to create the metrics, but these are not produced.
What can I do to find the problem to get it fixed?

[joe-elliott]

When we create a processor we overlay per tenant overrides on the default config and pass it in.

tempo/modules/generator/config.go

Line 67 in b6bd537

func (cfg *ProcessorConfig) copyWithOverrides(o metricsGeneratorOverrides, userID string) (ProcessorConfig, error) {

I'm wondering if we're overwriting the "true" in your config with false here. Most config options seem to have some kind of guard code to not override if its an empty value, but we just take whatever is in the overrides for this particular value.

tempo/modules/generator/config.go

Line 132 in b6bd537

copyCfg.ServiceGraphs.EnableMessagingSystemLatencyHistogram = o.MetricsGeneratorProcessorServiceGraphsEnableMessagingSystemLatencyHistogram(userID)

Perhaps try something like:

overrides:
  defaults:
    metrics_generator:
      processors: [service-graphs, span-metrics, local-blocks]
      processor:
        service_graphs:
          enable_messaging_system_latency_histogram: true

This is confusing. Even if this is not your issue, we need to optionally overwrite this field and not blindly do it.

[cbos]

@joe-elliott
Thanks for you reply!

I tested it with the additional override settings and it works now:

The metrics are available now.

So it looks like the way the settings are handled is causing some problems at the moment.

[joe-elliott]

I do think this is still an issue. There doesn't seem to be anyway for a handful of settings to use the actual config since they are always overwritten by the overrides config here:

tempo/modules/generator/config.go

Lines 124 to 134 in b6bd537

	copyCfg.SpanMetrics.DimensionMappings = o.MetricsGeneratorProcessorSpanMetricsDimensionMappings(userID)
	copyCfg.SpanMetrics.EnableTargetInfo = o.MetricsGeneratorProcessorSpanMetricsEnableTargetInfo(userID)
	copyCfg.SpanMetrics.TargetInfoExcludedDimensions = o.MetricsGeneratorProcessorSpanMetricsTargetInfoExcludedDimensions(userID)
	copyCfg.ServiceGraphs.EnableClientServerPrefix = o.MetricsGeneratorProcessorServiceGraphsEnableClientServerPrefix(userID)
	copyCfg.ServiceGraphs.EnableMessagingSystemLatencyHistogram = o.MetricsGeneratorProcessorServiceGraphsEnableMessagingSystemLatencyHistogram(userID)
	copyCfg.ServiceGraphs.EnableVirtualNodeLabel = o.MetricsGeneratorProcessorServiceGraphsEnableVirtualNodeLabel(userID)

The settings above it have a way to detect "unset" and only overwrite the config value if they are set. Let's add similar logic to the settings highlighted in the link above.