New component: Signal to metrics connector

[lahsivjar]

The purpose and use-cases of the new component

Signal to metrics connector can produce metrics from all signal types (logs, metrics, and traces).

The currently available connectors like countconnector, spanmetricsconnector, sumconnector can perform specific tasks and become a bit hard to use with evolving use-cases. Signal to metrics connector will utilize OTTL expressions to make data manipulation from the incoming signals configurable, thus allowing the component to serve a wider range of use-cases.

The OTTL expressions can be configured depending on the metric types. For example: for histograms and exponential histograms, count and value could be retrieved based on separate OTTL expressions whereas, for sum metric the value could be retrieved from a single value OTTL expression.

Example configuration for the component

Generic configuration

signaltometrics:
  spans:
    - name: span.duration
      description: Span duration as exponential histogram
      unit: us
      attributes: # categorize by attributes
        - key: attribute.foo
      exponential_histogram:
        count: Int(AdjustedCount()) # Adjusted count here is calculated as a custom OTTL converter
        value: Microseconds(end_time - start_time)
  datapoints:
    - name: datapoint.count
      description: Sum all the attribute value on the datapoint with name datapoint.foo
      conditions:
        - IsDouble(attributes["datapoint.foo"]) or IsInt(attributes["datapoint.foo"]) # only evaluate if datapoint.foo is a number
      sum:
        value: Double(attributes["datapoint.foo"])
  logs:
    - name: logrecord.count
      description: Count of log records
      sum:
        value: "1" # increment by 1 for each log record

Count connector use-cases

signaltometrics:
  spans:
    - name: trace.span.count
      description: Counts all spans
      sum:
        value: "1" # count each span as 1
  datapoints:
    - name: metric.datapoint.count
      description: Count of datapoints
      sum:
        value: "1" # count each datapoint as 1
  logs:
    - name: log.record.count
      description: Count of log records
      sum:
        value: "1" # count each log record as 1

Sum connector use-cases

signaltometrics:
  logs:
    - name: my.example.metric.name
      description: Sum of all attribute values with key attribute.with.numerical.value
      sum:
        value: Double(attributes["attribute.with.numerical.value"])

Span metrics connector use-cases

signaltometrics:
  spans:
    - name: calls
      description: Counts all spans
      attributes:
        - key: span.name
        - key: span.kind
        - key: status.code
      sum:
        value: "1" # count each span as 1
    - name: duration
      description: Counts all spans
      attributes:
        - key: span.name
        - key: span.kind
        - key: status.code
      unit: us
      exponential_histogram:
        value: Microseconds(end_time - start_time)

Convert a gauge metric to a histogram

signaltometrics:
  datapoints:
    - name: gauge.to.exphistogram
      description: An exponential histogram created from gague values
      include_resource_attributes:
        - key: resource.foo
      attributes:
        - key: datapoint.foo
      conditions:
        - metric.type == 1 # select all gauges
      exponential_histogram:
        count: "1" # 1 count for each datapoint
        value: Double(value_int) + value_double # handle both int and double

Telemetry data types supported

traces, metrics, and logs

Is this a vendor-specific component?

NO

Code Owner(s)

@lahsivjar

Sponsor (optional)

@ChrsMark

Additional context

We, at Elastic, have a working version of this component with a lot of the features for spans, datapoints, and, logrecords already developed and we would like to contribute the component upstream as we believe it could be useful to the community.

[djaglowski]

I think this is a great idea. Am I correct in understanding that it depends on #35621?

[lahsivjar]

Am I correct in understanding that it depends on #35621?

Partially. In our current implementation, we have circumvented this by an internal get editor function which we use to create OTTL statement from data expressions - so not exactly a blocker.

[ChrsMark]

I'm willing to sponsor this.

[jluncc]

hi, can it use log convert to gauge? if not, is there any way to do that ?

[lahsivjar]

hi, can it use log convert to gauge? if not, is there any way to do that ?

👋 @jluncc , The component will be able to produce counters or histograms using log records, some examples are available here. Can you elaborate on your exact usage scenario?

[jluncc]

hi, can it use log convert to gauge? if not, is there any way to do that ?

👋 @jluncc , The component will be able to produce counters or histograms using log records, some examples are available here. Can you elaborate on your exact usage scenario?

@lahsivjar thanks your reply

i have another issue here show more detail: #36710