Cannot disable/fix OpenTelemetry errors in AWS Lambda logs

[imanshafiei540]

I am using AWS Lambda with Python 3.9 and OpenTelemetry SDK 1.17.0 to send metrics to an ADOT collector. The collector then exports the metrics to an endpoint. While everything is working as expected, I am seeing a few error logs in the Lambda log group, making it hard to maintain the logs.

I need help with either disabling these logs or fixing the errors. Here are some details of the environment variables and configuration files I'm using:

Tools:

1. AWS Lambda
2. Python==3.9
3. opentelemetry-sdk==1.17.0
4. arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-python-amd64-ver-1-17-0:1 (AWS Distro for OpenTelemetry Lambda Support)

Environment variables:

AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument
OPENTELEMETRY_COLLECTOR_CONFIG_FILE=/var/task/src/otel-config.yaml
OTEL_METRICS_EXPORTER=otlp
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OPENTELEMETRY_EXTENSION_LOG_LEVEL=error

Configuration file (otel-config.yaml):

receivers:
  otlp:
    protocols:
      http:

exporters:
  otlp:
    endpoint: https://<ANOTHER_SERVICE>:4317

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [otlp]

Here are the error logs that I'm receiving in the Lambda log group:

[WARNING] 2023-05-01T23:56:03.374Z An instrument with name http.server.duration, type Histogram, unit ms and description measures the duration of the inbound HTTP request has been created already.

[ERROR] 2023-05-01T23:56:04.958Z 42a36f0d-6bde-428d-833a-fc34f295a4a9 Failed to export batch code: 404, reason: 404 page not found

[ERROR] 2023-05-01T23:56:04.959Z 42a36f0d-6bde-428d-833a-fc34f295a4a9 Invalid type <class 'NoneType'> of value None
Traceback (most recent call last):
File "/opt/python/opentelemetry/exporter/otlp/proto/http/metric_exporter/init.py", line 345, in _translate_attributes
output.append(_translate_key_values(key, value))
File "/opt/python/opentelemetry/exporter/otlp/proto/http/metric_exporter/init.py", line 429, in _translate_key_values
return KeyValue(key=key, value=_translate_value(value))
File "/opt/python/opentelemetry/exporter/otlp/proto/http/metric_exporter/init.py", line 423, in _translate_value
raise Exception(f"Invalid type {type(value)} of value {value}")
Exception: Invalid type <class 'NoneType'> of value None

I would appreciate any help or suggestions to resolve these errors or disable the logs. Thank you.

[dipit-malhotra]

@imanshafiei540 Were you able to fix this ?

[imanshafiei540]

Hello @dipit-malhotra
Yes and No, I was able to solve some of them by creating new PRs, so they should be okay in newer versions. But, I'm still getting some errors like Failed to export batch code: 404. Let me know what version you're using, maybe we can use the additional information to debug more. The environment variables you're using could be also useful.

[mattbrandman]

Were you able to resolved the failed to export error? Still seeing this in my logs and not sure where it's coming from.

[tylerbenson]

I wonder if the new layer with the decouple processor will help here.