chore: TEST mq #10432
Merged
chore: TEST mq #10432
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#9515) This change: - Removes the usage of `TelemetryModuleWatchdog` - A strategy of diffing `sys.modules.keys()` is now used instead of watching for imports. - Removes tracking of module origin - Since we are only interested in distribution name + version we don't need to keep track of all imported module origins, and then look up distributions from that. We can use import names to directly look up the distribution they are apart of. - Add support for namespace packages - Added the use of `importlib_metdata.packages_distributions()` to ensure we are able to resolve namespace packages to their actual distribution name. - We only add `importlib_metadata` for Python <3.8, but `packages_distributions` was added in 3.10, the footprint is fairly small, so it is just vendored for the cases when we don't have it available (keeps `importlib_metadata` requirement fairly broad as well) The main change here is to have the TelemetryWriter keep track of which `sys.modules.keys()` we have already resolved and sent information about. Then on each periodic run, compare current `sys.modules.keys()` against the cache, and send any new modules. We also are no longer using origins to try and resolve the distributions for a given module, instead we use `importlib_metadata.distribution(name)`, if that fails to resolve, then we will check to see if that import name is mapped to any namespace packages, which we then try grab the distributions for that. (this means any given module name could resolve to multiple distributions to report about). I have really struggled to get a solid benchmark/performance measurement going, the only details I have are the internal app I was testing against went from 100s to about 2s of overhead added for dependency resolution, and the test suite itself still validates that the previous behavior is occurring. ## Checklist - [x] Change(s) are motivated and described in the PR description - [x] Testing strategy is described if automated tests are not included in the PR - [x] Risks are described (performance impact, potential for breakage, maintainability) - [x] Change is maintainable (easy to change, telemetry, documentation) - [x] [Library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) are followed or label `changelog/no-changelog` is set - [x] Documentation is included (in-code, generated user docs, [public corp docs](https://github.com/DataDog/documentation/)) - [x] Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) - [x] If this PR changes the public interface, I've notified `@DataDog/apm-tees`. ## Reviewer Checklist - [ ] Title is accurate - [ ] All changes are related to the pull request's stated goal - [ ] Description motivates each change - [ ] Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - [ ] Testing strategy adequately addresses listed risks - [ ] Change is maintainable (easy to change, telemetry, documentation) - [ ] Release note makes sense to a user of the library - [ ] Author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - [ ] Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)
Updates the aiobotocore integration latest version and regenerates lockfiles. - [x] checklist Co-authored-by: emmettbutler <[email protected]> Co-authored-by: Quinna Halim <[email protected]>
This adds (in more of a PoC state than anything) a small utility to
visualize snapshots, that can be useful when looking at PRs that change
snapshots.
This is somewhere in the area of "done for fun", but it has more
potential for good use down the road, hopefully.
Example:
```
% hatch run snapshot_viewer:view tests/snapshots/tests.opentelemetry.test_trace.test_distributed_trace_with_flask_app\[with_ddtrace_run\].json
========================= Showing snapshot: tests/snapshots/tests.opentelemetry.test_trace.test_distributed_trace_with_flask_app[with_ddtrace_run].json =========================
flask
┌ - flask.try_trigger_before_first_request_functions
│
│ flask
├ - flask.preprocess_request
│
│ flask flask None
flask ├ - flask.dispatch_request ─ - /otel ─ - otel-flask-manual-span
┌ - GET /otel ┤
│ │ flask
│ ├ - flask.process_response
│ │
None flask │ │ flask
Snapshots ─ - test-otel-distributed-trace ─ - GET /otel ┤ ├ - flask.do_teardown_request
│ │
│ │ flask
│ └ - flask.do_teardown_appcontext
│
│ flask
└ - flask.response
```
## Checklist
- [x] PR author has checked that all the criteria below are met
- The PR description includes an overview of the change
- The PR description articulates the motivation for the change
- The change includes tests OR the PR description describes a testing
strategy
- The PR description notes risks associated with the change, if any
- Newly-added code is easy to change
- The change follows the [library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
- The change includes or references documentation updates if necessary
- Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))
## Reviewer Checklist
- [x] Reviewer has checked that all the criteria below are met
- Title is accurate
- All changes are related to the pull request's stated goal
- Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes
- Testing strategy adequately addresses listed risks
- Newly-added code is easy to change
- Release note makes sense to a user of the library
- If necessary, author has acknowledged and discussed the performance
implications of this PR as reported in the benchmarks PR comment
- Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)
Taint the query string source for IAST integration with FastAPI ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)
Updates the algoliasearch integration latest version and regenerates lockfiles. - [x] checklist Co-authored-by: emmettbutler <[email protected]> Co-authored-by: Quinna Halim <[email protected]>
## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) Co-authored-by: quinna-h <[email protected]> Co-authored-by: Quinna Halim <[email protected]>
This job is useful to run, to be able to manually check the results, but given that it fails every time and we ignore the failure in the pipeline, there is no reason to block the entire pipeline waiting for it to finish. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [ ] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)
Updates the anthropic integration latest version and regenerates lockfiles. - [x] checklist Co-authored-by: brettlangdon <[email protected]> Co-authored-by: Quinna Halim <[email protected]>
Updates the pyodbc integration latest version and regenerates lockfiles. - [x] checklist Co-authored-by: quinna-h <[email protected]> Co-authored-by: Quinna Halim <[email protected]>
## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) Co-authored-by: quinna-h <[email protected]> Co-authored-by: Quinna Halim <[email protected]>
Updates the aiohttp integration latest version and regenerates lockfiles. - [x] checklist Co-authored-by: brettlangdon <[email protected]> Co-authored-by: Quinna Halim <[email protected]>
Updates the pynamodb integration latest version and regenerates lockfiles. - [x] checklist Co-authored-by: emmettbutler <[email protected]> Co-authored-by: erikayasuda <[email protected]> Co-authored-by: Quinna Halim <[email protected]>
## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) Co-authored-by: quinna-h <[email protected]> Co-authored-by: Quinna Halim <[email protected]>
Updates the mongoengine integration latest version and regenerates lockfiles. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) Co-authored-by: quinna-h <[email protected]> Co-authored-by: Quinna Halim <[email protected]>
This PR adds instrumentation to submit langchain vector similarity search spans to LLM Observability. The similarity search spans sent to LLM Observability will contain the following I/O data: input.value: a text field annotating the search query output.value: The output document objects for the vector similarity search Ticket : https://datadoghq.atlassian.net/browse/MLOB-938 ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --------- Co-authored-by: Yun Kim <[email protected]>
…ls (#10352) ## Note This branches from #10334 so it will be easier to review once merged. ## Description - Move the `%` operator aspect to 100% native. This makes it almost twice as fast. 🚀 However, even with this there is a lot of overhead on this aspect so more work will have to be done on it. - Replace instance like calls to static methods if `Initializer` (`initializer-foo()`) with static method calls (`Initializer::foo()`) which further improves times in some aspects up to 28% (and it's more correct). - Use the new `TRY_CATCH_MACRO` on `aspect_operator_add` and `aspect_operator_addinplace`. Signed-off-by: Juanjo Alvarez <[email protected]> ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --------- Signed-off-by: Juanjo Alvarez <[email protected]>
## Code Security FastAPI support Cookies & Headers sources (APPSEC-54623 & APPSEC-54619 tasks) Previous sources tasks: #10331 ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)
…10393) Currently, the build pipeline isn't triggered automatically upon an open PR unless we merge or push from main ([example](#10392)). This update adds the branch prefix `upgrade-latest-` to the 'allowlist' of branches that trigger the build and deploy pipeline, in order to have the auto-generated PRs trigger the appropriate jobs. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)
We fix the debug log call pertaining to the new RC service auto-restart feature by adding the expected missing arguments. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)
erikayasuda
requested review from
rachelyangdog
and removed request for
a team
erikayasuda
requested review from
emmettbutler and
juanjux
and removed request for
a team,
rachelyangdog,
emmettbutler and
juanjux
![datadog-datadog-prod-us1[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
Comment on lines
+62
to
+63
| if origin == OriginType.HEADER and name.lower() in ["cookie", "cookies"]: | ||
| origin = OriginType.COOKIE |
There was a problem hiding this comment.
🔴 Code Quality Violation
too many nesting levels (...read more)
Avoid to nest too many loops together. Having too many loops make your code harder to understand.
Prefer to organize your code in functions and unit of code you can clearly understand.
Learn More
Comment on lines
+151
to
+152
| if output_value is not None: | ||
| meta_dict["output"].update({"value": output_value}) |
There was a problem hiding this comment.
🔴 Code Quality Violation
too many nesting levels (...read more)
Avoid to nest too many loops together. Having too many loops make your code harder to understand.
Prefer to organize your code in functions and unit of code you can clearly understand.
Learn More
Comment on lines
+87
to
+88
| if name in already_imported: | ||
| continue |
There was a problem hiding this comment.
🔴 Code Quality Violation
too many nesting levels (...read more)
Avoid to nest too many loops together. Having too many loops make your code harder to understand.
Prefer to organize your code in functions and unit of code you can clearly understand.
Learn More
Comment on lines
+62
to
+63
| if origin == OriginType.HEADER and name.lower() in ["cookie", "cookies"]: | ||
| origin = OriginType.COOKIE |
There was a problem hiding this comment.
🔴 Code Quality Violation
too many nesting levels (...read more)
Avoid to nest too many loops together. Having too many loops make your code harder to understand.
Prefer to organize your code in functions and unit of code you can clearly understand.
Learn More
Comment on lines
+151
to
+152
| if output_value is not None: | ||
| meta_dict["output"].update({"value": output_value}) |
There was a problem hiding this comment.
🔴 Code Quality Violation
too many nesting levels (...read more)
Avoid to nest too many loops together. Having too many loops make your code harder to understand.
Prefer to organize your code in functions and unit of code you can clearly understand.
Learn More
Comment on lines
+87
to
+88
| if name in already_imported: | ||
| continue |
There was a problem hiding this comment.
🔴 Code Quality Violation
too many nesting levels (...read more)
Avoid to nest too many loops together. Having too many loops make your code harder to understand.
Prefer to organize your code in functions and unit of code you can clearly understand.
Learn More
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Checklist
Reviewer Checklist