feat(llmobs): submit langchain embedding spans

[yahya-mouman]

Changes

This PR adds instrumentation to submit langchain embedding spans to LLM Observability. The embedding spans sent to LLM Observability will contain the following I/O data:

input.documents: when part of a workflow captures the embeddings as a json list. when it is not a workflow but a single embedding operation captures each embedding input as a Document dictionary containing a text field
output.value: instead of unnecessarily storing very long vector values, adds a placeholder text [X embeddings returned with size Y]

Ticket : https://datadoghq.atlassian.net/browse/MLOB-937

Checklist

• PR author has checked that all the criteria below are met
• The PR description includes an overview of the change
• The PR description articulates the motivation for the change
• The change includes tests OR the PR description describes a testing strategy
• The PR description notes risks associated with the change, if any
• Newly-added code is easy to change
• The change follows the library release note guidelines
• The change includes or references documentation updates if necessary
• Backport labels are set (if applicable)

Reviewer Checklist

• Reviewer has checked that all the criteria below are met
• Title is accurate
• All changes are related to the pull request's stated goal
• Avoids breaking API changes
• Testing strategy adequately addresses listed risks
• Newly-added code is easy to change
• Release note makes sense to a user of the library
• If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[github-actions]

CODEOWNERS have been resolved as:

releasenotes/notes/feat-llmobs-submit-langchain-embedding-spans-89c8704ef41cfee3.yaml  @DataDog/apm-python
tests/contrib/langchain/cassettes/langchain_community/openai_embedding_query_integration.yaml  @DataDog/ml-observability
ddtrace/contrib/langchain/patch.py                                      @DataDog/ml-observability
ddtrace/llmobs/_integrations/langchain.py                               @DataDog/ml-observability
tests/contrib/langchain/test_langchain_llmobs.py                        @DataDog/ml-observability

[Yun-Kim]

Implementation looks great, just a couple questions 😄 can we add tests in test_langchain_llmobs.py to class TestLLMObsLangchainCommunity and class TestLangchainTraceStructureWithLlmIntegrations?

[datadog-datadog-prod-us1]

Library Vulnerabilities

    

[yahya-mouman]

Implementation looks great, just a couple questions 😄 can we add tests in test_langchain_llmobs.py to class TestLLMObsLangchainCommunity and class TestLangchainTraceStructureWithLlmIntegrations?

I updated and addressed all the previous comments. The only thing left now is writing the test case but before doing that, I wanted to confirm the implementation details especially the parsing difference between the workflow vs non-workflow cases 🙏

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-07-31 16:49:43

Comparing candidate commit 1601844 in PR branch yahya/add-langchain-embedding-support with baseline commit 1710fe3 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 214 metrics, 2 unstable metrics.

[Yun-Kim]

Looks mostly great, I added a comment/suggestion on the input documents formatting.
Just needs tests, and release note similar to this PR.

[Yun-Kim]

@yahya-mouman there are significant flakiness/skipping issues in the Langchain test suite. Let's merge this PR once #9768 is merged.

[Yun-Kim]

Should be good to go after the testing comments are addressed now that the flakiness should be addressed!

[yahya-mouman]

Should be good to go after the testing comments are addressed now that the flakiness should be addressed!

Great job on fixing the flakiness of the tests 🥇 I believe everything is done and good to go with this PR