2.17.0rc1

New Features

• ASM

  • Adds support for session fingerprints.

• LLM Observability

  • When not using a provider integration (OpenAI, Anthropic, or Bedrock) with the LangChain integration, token metrics will be appended to the LLM Observability llm span.
  • When langchain's chat_model.with_structured_output(..., method="json_mode") is used, or response_format={"type": "json_object"} is passed into a langchain chat model invocation, the LLM Observability span will be an llm span instead of a workflow span.

• Single Step Instrumentation

  • Adds requirements.json to SSI artifact for bailing out on unsupported systems.

• Tracing

  • Adds support for expanding AWS request/response payloads into flattened span tags.
  • Updates the service naming algorithm to infer the base service name when DD_SERVICE is not set, replacing instances of unnamed-python-service. Ensures that a more meaningful service name is used whenever possible, enhancing clarity in service identification.

Bug Fixes

• ASM/Threats

  • The new user events policy is preventing users PII to be added by default as span tags. To allow customers using the Django auto instrumentation to still have those information, new environment variables have been added. In particular DD_DJANGO_INCLUDE_EMAIL (false by default), will tag user events with user email as before.

• Code Security/IAST

  • Adds googlecloudsdk and google auth to the Code Security deny list.
  • Resolves an issue where importing the google.cloud.storage.batch module would fail raising an ImportError.

• Crashtracking

  • Fixes an issue where the use of the crashtracking component could result in zombie processes.

• Lib-Injection

  • Adds more commands to the auto-injection denylist.
  • Ensures we do not import the user installed ddtrace if it is present.
  • Fixes injection guardrail check when sys.argv is not available.

• LLM Observability

  • Resolves an issue where annotating spans with non-ASCII language input/output values resulted in encoded unicode being submitted.

• Profiling

  • Fixes a data race where span information associated with a thread was read and updated concurrently, leading to segfaults
  • Fixes an issue where cpu-time was not profiled for services using gunicorn, when DD_PROFILING_STACK_V2_ENABLED was set.
  • Fixes an issue where enabling native exporter via DD_PROFILING_EXPORT_LIBDD_ENABLED, DD_PROFILING_TIMELINE_ENABLED or DD_PROFILING_STACK_V2_ENABLED turned off live heap profiling.
  • The lock profiler would log a warning if it couldn't determine a name for a lock, and it would try determining a name multiple times for the same lock. This lead to excessive log spam. Downgrade this to a debug log and only try to determine the name once.
  • Fixes an issue where the profiler was allocating too much memory from ensure_binary_or_empty() function, on Python versions before 3.12, with DD_PROFILING_EXPORT_LIBDD_ENABLED or DD_PROFILING_TIMELINE_ENABLED.
  • Fixes an issue where the sample pool could deadlock after fork() by clearing it in the child process.
  • When a Python thread finishes, this change frees memory used for mapping its thread id to Span. The mapping is populated and used when DD_PROFILING_ENDPOINT_COLLECTION_ENABLED and DD_PROFILING_STACK_V2_ENABLED were set to enable grouping of profiles for endpoints.

• Tracing

  • pymongo: Adds type checking to solve an issue where NoneType instead of expected Pin object would throw an error in TracedTopology method.

2.14.7

Bug Fixes

• Code Security:

  • Add googlecloudsdk and google auth to the Code Security deny list.

• Profiling:

  • Fixes an issue where cpu-time was not profiled for services using gunicorn, when DD_PROFILING_STACK_V2_ENABLED was set.

  • Fixes an issue where the sample pool could deadlock after fork() by clearing it in the child process.

2.16.1

Bug Fixes

• Threats

  • The new user events policy is preventing users PII to be added by default as span tags. To allow customers using the Django auto instrumentation to still have those information, new environment variables have been added. In particular DD_DJANGO_INCLUDE_EMAIL (false by default), will tag user events with user email as before.

• Code Security

  • Add googlecloudsdk and google auth to the Code Security deny list.

• Crashtracking

  • Fixes an issue where the use of the crashtracking component could result in zombie processes.

• Lib-Injection

  • This fix adds more commands to the auto-injection denylist.
  • This fix ensures we do not import the user installed ddtrace if it is present.

• LLM Observability

  • Resolves an issue where annotating spans with non-ASCII language input/output values resulted in encoded unicode being submitted.

• Profiling

  • Fixes an issue where cpu-time was not profiled for services using gunicorn, when DD_PROFILING_STACK_V2_ENABLED was set.
  • Fixes an issue where the profiler was allocating too much memory from ensure_binary_or_empty() function, on Python versions before 3.12, with DD_PROFILING_EXPORT_LIBDD_ENABLED or DD_PROFILING_TIMELINE_ENABLED.
  • Fixes an issue where the sample pool could deadlock after fork() by clearing it in the child process.

2.15.2

Bug Fixes

• Profiling:
  • Fixes an issue where enabling native exporter via DD_PROFILING_EXPORT_LIBDD_ENABLED, DD_PROFILING_TIMELINE_ENABLED or DD_PROFILING_STACK_V2_ENABLED turned off live heap profiling.
  • Fixes an issue where the profiler was allocating too much memory from ensure_binary_or_empty() function, on Python versions before 3.12, with DD_PROFILING_EXPORT_LIBDD_ENABLED or DD_PROFILING_TIMELINE_ENABLED.
  • When a Python thread finishes, this change frees memory used for mapping its thread id to Span. The mapping is populated and used when DD_PROFILING_ENDPOINT_COLLECTION_ENABLED and DD_PROFILING_STACK_V2_ENABLED were set to enable grouping of profiles for endpoints.
  • Resolves an issue where asyncio task names are not captured by stack v2, when DD_PROFILING_STACK_V2_ENABLED is set.
• Tracing:
  • pymongo: Adds type checking to solve an issue where NoneType instead of expected Pin object would throw an error in TracedTopology method.

2.14.6

Bug Fixes

• Profiling

  • Fixes an issue where enabling native exporter via DD_PROFILING_EXPORT_LIBDD_ENABLED, DD_PROFILING_TIMELINE_ENABLED or DD_PROFILING_STACK_V2_ENABLED turned off live heap profiling.
  • Fixes an issue where the profiler was allocating too much memory from ensure_binary_or_empty() function, on Python versions before 3.12, with DD_PROFILING_EXPORT_LIBDD_ENABLED or DD_PROFILING_TIMELINE_ENABLED.
  • When a Python thread finishes, this change frees memory used for mapping its thread id to Span. The mapping is populated and used when DD_PROFILING_ENDPOINT_COLLECTION_ENABLED and DD_PROFILING_STACK_V2_ENABLED were set to enable grouping of profiles for endpoints.
  • Resolves an issue where asyncio task names are not captured by stack v2, when DD_PROFILING_STACK_V2_ENABLED is set.

• Tracing

  • pymongo: Adds type checking to solve an issue where NoneType instead of expected Pin object would throw an error in TracedTopology method.

2.16.0

New Features

• LLM Observability

  • When starting LLM and embedding spans, the model_name argument is now optional and will default to custom. This applies to both inline methods (e.g. LLMObs.llm()) and function decorators (e.g. @llm).
  • Introduces the ability to add metadata for evaluation metrics via the submit_evaluation method. For more information, see submitting evaluations with the SDK.

• Tracing

  • Introduces support for Baggage as defined by the OpenTelemetry specification.
  • botocore: Adds span pointers for successful DynamoDB BatchWriteItem spans. Table Primary Keys will need to be provided with the ddtrace.config.botocore.dynamodb_primary_key_names_for_tables option or the DD_BOTOCORE_DYNAMODB_TABLE_PRIMARY_KEYS environment variable to correctly handle the PutRequest items.
  • botocore: Adds span pointers for successful DynamoDB TransactWriteItems spans. Table Primary Keys will need to be provided with the ddtrace.config.botocore.dynamodb_primary_key_names_for_tables option or the DD_BOTOCORE_DYNAMODB_TABLE_PRIMARY_KEYS environment variable to correctly handle the Put items.
  • botocore: Adds ddtrace.config.botocore.add_span_pointers option or the DD_BOTOCORE_ADD_SPAN_POINTERS environment variable to control adding span pointers to some successful AWS API requests. This option is enabled by default.

Bug Fixes

• CI Visibility

  • Fixes a bug where CODEOWNERS would incorrectly fail to discard line-level trailing comments (eg: @code/owner # my comment would result in codeowners being parsed as @code/owner, #, my, and comment)
  • Fixes unnecessary logging of an exception that would appear when trying to upload git metadata in an environment without functioning git (eg: missing git binary or .git directory)

• Code Security

  • Resolves an issue where importing the google.cloud.storage.batch module would fail raising an ImportError

• Dynamic Instrumentation

  • Fixes an issue that prevented dynamic span tags probes from adding the requested tags to the requested span.

• LLM Observability

  • Resolves two issues with annotation contexts:
    • annotations registered via annotation contexts were being applied globally. Annotations are now only applied to the current trace context and do not pollute to other threads & processes.
    • annotations from nested annotation contexts were applied in a non-deterministic order. Annotations are now applied in the order they were registered.
  • Resolves an issue where input and output values equal to zero were not being annotated on workflow, task, agent and tool spans when using LLMObs.annotate.
  • Resolves errors where the disabled setting was being ignored when forking.

• Profiling

  • Fixes a data race where span information associated with a thread was read and updated concurrently, leading to segfaults.
  • Fixes an issue where enabling native exporter via DD_PROFILING_EXPORT_LIBDD_ENABLED, DD_PROFILING_TIMELINE_ENABLED or DD_PROFILING_STACK_V2_ENABLED turned off live heap profiling.
  • When a Python thread finishes, this change frees memory used for mapping its thread id to Span. The mapping is populated and used when DD_PROFILING_ENDPOINT_COLLECTION_ENABLED and DD_PROFILING_STACK_V2_ENABLED were set to enable grouping of profiles for endpoints.
  • Resolves an issue where asyncio task names are not captured by stack v2, when DD_PROFILING_STACK_V2_ENABLED is set.
  • Resolves an issue where endpoint profiling for stack v2 throws TypeError exception when it is given a Span with None span_type.

• Tracing

  • Resolves the issue where tracer flares would not be generated if unexpected types were received in the AGENT_CONFIG remote configuration product.
  • elasticsearch: Resolves an issue where span tags were not fully populated on "sampled" spans, causing metric dimensions to be incorrect when spans were prematurely marked as sampled, including resource_name.

Other Changes

• LLM Observability
  • Updates the merging behavior for tags when LLMObs.annotate is called multiple times on the same span so that the latest value for a tag key overrides the previous value.

2.16.0rc2

Bug Fixes

• Code Security
  • Resolves an issue where importing the google.cloud.storage.batch module would fail raising an ImportError
• Profiling
  • Fixes a data race where span information associated with a thread was read and updated concurrently, leading to segfaults
  • Fixes an issue where enabling native exporter via DD_PROFILING_EXPORT_LIBDD_ENABLED, DD_PROFILING_TIMELINE_ENABLED or DD_PROFILING_STACK_V2_ENABLED turned off live heap profiling.
  • When a Python thread finishes, this change frees memory used for mapping its thread id to Span. The mapping is populated and used when DD_PROFILING_ENDPOINT_COLLECTION_ENABLED and DD_PROFILING_STACK_V2_ENABLED were set to enable grouping of profiles for endpoints.

2.15.1

Bug Fixes

• LLM Observability: This fix resolves two issues with annotation contexts:
  - annotations registered via annotation contexts were being applied globally. Annotations are now only applied to the current trace context and do not pollute to other threads & processes.
  - annotations from nested annotation contexts were applied in a non-deterministic order. Annotations are now applied in the order they were registered.

• CI Visibility: fixes a bug where CODEOWNERS would incorrectly fail to discard line-level trailing comments (eg: @code/owner # my comment would result in codeowners being parsed as @code/owner, #, my, and comment)

• CI Visibility: fixes unnecessary logging of an exception that would appear when trying to upload git metadata in an environment without functioning git (eg: missing git binary or .git directory)

• dynamic instrumentation: Fixes an issue that prevented dynamic span tags probes from adding the requested tags to the requested span.

• Code Security: This fix resolves an issue where importing the google.cloud.storage.batch module would fail raising an ImportError

• profiling: fix a data race where span information associated with a thread was read and updated concurrently, leading to segfaults

• profiling: resolves an issue where endpoint profiling for stack v2 throws TypeError exception when it is given a Span with None span_type.

Other Changes

• LLM Observability: Updates the merging behavior for tags when LLMObs.annotate is called multiple times on the same span so that the latest value for a tag key overrides the previous value.

2.14.5

Bug Fixes

• LLM Observability: This fix resolves an issue where LLMObs.enable() did not patch google_generativeai library.
• CI Visibility: fixes a bug where CODEOWNERS would incorrectly fail to discard line-level trailing comments (eg: @code/owner # my comment would result in codeowners being parsed as @code/owner, #, my, and comment)
• CI Visibility: fixes unnecessary logging of an exception that would appear when trying to upload git metadata in an environment without functioning git (eg: missing git binary or .git directory)
• elasticsearch: this fix resolves an issue where span tags were not fully populated on "sampled" spans, causing metric dimensions to be incorrect when spans were prematurely marked as sampled, including resource_name.
• Code security: This fix resolves an issue where partial matches on function names we aimed to patch were being patched instead of full matches on them.
• Code Security: This fix resolves an issue where importing the google.cloud.storage.batch module would fail raising an ImportError
• profiling: Improves the error message when the native exporter fails to load and stops profiling from starting if ddtrace is also being injected.
• profiling: fix a data race where span information associated with a thread was read and updated concurrently, leading to segfaults
• profiling: resolves an issue where endpoint profiling for stack v2 throws TypeError exception when it is given a Span with None span_type.

2.16.0rc1

New Features

• LLM Observability

  • When starting LLM and embedding spans, the model_name argument is now optional and will default to custom. This applies to both inline methods (e.g. LLMObs.llm()) and function decorators (e.g. @llm).
  • Introduces the ability to add metadata for evaluation metrics via the submit_evaluation method. For more information, see submitting evaluations with the SDK.

• Tracing

  • Introduces support for Baggage as defined by the OpenTelemetry specification.
  • botocore: Adds span pointers for successful DynamoDB BatchWriteItem spans. Table Primary Keys will need to be provided with the ddtrace.config.botocore.dynamodb_primary_key_names_for_tables option or the DD_BOTOCORE_DYNAMODB_TABLE_PRIMARY_KEYS environment variable to correctly handle the PutRequest items.
  • botocore: Adds span pointers for successful DynamoDB TransactWriteItems spans. Table Primary Keys will need to be provided with the ddtrace.config.botocore.dynamodb_primary_key_names_for_tables option or the DD_BOTOCORE_DYNAMODB_TABLE_PRIMARY_KEYS environment variable to correctly handle the Put items.
  • botocore: Adds ddtrace.config.botocore.add_span_pointers option or the DD_BOTOCORE_ADD_SPAN_POINTERS environment variable to control adding span pointers to some successful AWS API requests. This option is enabled by default.
  • elasticsearch: Resolves an issue where span tags were not fully populated on "sampled" spans, causing metric dimensions to be incorrect when spans were prematurely marked as sampled, including resource_name.

Bug Fixes

• CI Visibility

  • Fixes a bug where CODEOWNERS would incorrectly fail to discard line-level trailing comments (eg: @code/owner # my comment would result in codeowners being parsed as @code/owner, #, my, and comment)
  • Fixes unnecessary logging of an exception that would appear when trying to upload git metadata in an environment without functioning git (eg: missing git binary or .git directory)

• Dynamic Instrumentation

  • Fixes an issue that prevented dynamic span tags probes from adding the requested tags to the requested span.

• LLM Observability

  • Resolves two issues with annotation contexts:
    • annotations registered via annotation contexts were being applied globally. Annotations are now only applied to the current trace context and do not pollute to other threads & processes.
    • annotations from nested annotation contexts were applied in a non-deterministic order. Annotations are now applied in the order they were registered.
  • Resolves an issue where input and output values equal to zero were not being annotated on workflow, task, agent and tool spans when using LLMObs.annotate.
  • Resolves errors where the disabled setting was being ignored when forking.

• Profiling

  • Resolves an issue where asyncio task names are not captured by stack v2, when DD_PROFILING_STACK_V2_ENABLED is set.
  • Resolves an issue where endpoint profiling for stack v2 throws TypeError exception when it is given a Span with None span_type.

• Tracing

  • Resolves the issue where tracer flares would not be generated if unexpected types were received in the AGENT_CONFIG remote configuration product.

Other Changes

• LLM Observability
  • Updates the merging behavior for tags when LLMObs.annotate is called multiple times on the same span so that the latest value for a tag key overrides the previous value.