From ce6e4bb645af42360fc385ba29db44767a8674a1 Mon Sep 17 00:00:00 2001 From: Harrison Chase Date: Fri, 29 Nov 2024 12:40:35 -0500 Subject: [PATCH 1/6] improve integration docs --- .../how_to/integrations/index.mdx | 71 +++++++++--- .../integrations/standard-tests-chat-model.md | 104 ++++++++++++++++++ 2 files changed, 157 insertions(+), 18 deletions(-) create mode 100644 docs/docs/contributing/how_to/integrations/standard-tests-chat-model.md diff --git a/docs/docs/contributing/how_to/integrations/index.mdx b/docs/docs/contributing/how_to/integrations/index.mdx index eeb0f73b3fdbb..b4f16f854029c 100644 --- a/docs/docs/contributing/how_to/integrations/index.mdx +++ b/docs/docs/contributing/how_to/integrations/index.mdx @@ -4,14 +4,16 @@ pagination_next: contributing/how_to/integrations/package # Contribute Integrations -LangChain integrations are packages that provide access to language models, vector stores, and other components that can be used in LangChain. +Integrations are a core component of LangChain. +LangChain provides standard interfaces for several different components (language models, vector stores, etc) that are crucial when building LLM applications. -This guide will walk you through how to contribute new integrations to LangChain, by -publishing an integration package to PyPi, and adding documentation for it -to the LangChain Monorepo. -These instructions will evolve over the next few months as we improve our integration -processes. +## Why contribute an integration to LangChain? + +- **Discoverability:** LangChain is the most used framework for building LLM applications, with over 20 million monthly downloads. LangChain integrations are discoverable by a large community of GenAI builders. +- **Interoptability:** LangChain components expose a standard interface, allowing developers to easily swap them for each other. If you implement a LangChain integration, any developer using a different component will easily be able to swap yours in. +- **Best Practices:** Through their standard interface, LangChain components encourage and facilitate best practices (streaming, async, etc) + ## Components to Integrate @@ -22,8 +24,7 @@ supported in LangChain ::: -While any component can be integrated into LangChain, at this time we are only accepting -new integrations in the docs of the following kinds: +While any component can be integrated into LangChain, there are specific types of integrations we encourage more: @@ -60,18 +61,52 @@ new integrations in the docs of the following kinds: ## How to contribute an integration -The only step necessary to "be" a LangChain integration is to add documentation -that will render on this site (https://python.langchain.com/). +In order to contribute an integration, you should follow these steps: + +1. Confirm that your integration is in the [list of components](#components-to-integrate) we are currently encouraging. +2. Implement your integration +3. Implement the standard tests ([see below](#standard-tests)) for your integration and successfully run them. +4. Publish your integration in a Python package to PyPi. +5. [Optional] Open and merge a PR to add documentation for your integration to the official LangChain docs. +6. [Optional] Engage with the LangChain team for joint co-marketing ([see below](#co-marketing)). + + +## Standard Tests + +Testing is a critical part of the development process that ensures your code works as expected and meets the desired quality standards. +In the LangChain ecosystem, we have 2 main types of tests: **unit tests** and **integration tests**. +These standard tests help maintain compatibility between different components and ensure reliability. + +**Unit Tests**: Unit tests are designed to validate the smallest parts of your code—individual functions or methods—ensuring they work as expected in isolation. They do not rely on external systems or integrations. + +**Integration Tests**: Integration tests validate that multiple components or systems work together as expected. For tools or integrations relying on external services, these tests often ensure end-to-end functionality. + +Each type of integration has its own set of standard tests. Please see the correct guide for setting up standard tests for your integration: + +- [Chat Models] +- [Tools] +- [Toolkits] +- [Retrievers] +- [Document Loaders] +- [Retrievers] +- [Document Loaders] +- [Vector Stores] +- [Embedding Models] + +## Co-Marketing -As a prerequisite to adding your integration to our documentation, you must: +With over 20 million monthly downloads, LangChain has a large audience of developers building LLM applications. +Besides just adding integrations, we also like to show them examples of cool tools or APIs they can use. -1. Confirm that your integration is in the [list of components](#components-to-integrate) we are currently accepting. -2. [Implement your package](./package.mdx) and publish it to a public github repository. -3. [Implement the standard tests](./standard_tests) for your integration and successfully run them. -4. [Publish your integration](./publish.mdx) by publishing the package to PyPi and add docs in the `docs/docs/integrations` directory of the LangChain monorepo. +While traditionally called "co-marketing", we like to thing of this more as "co-education". +For that reason, while we are happy to highlight your integration through our social media channels, we prefer to highlight examples that also serve some educational purpose. +Our main social media channels are Twitter and LinkedIn. -Once you have completed these steps, you can submit a PR to the LangChain monorepo to add your integration to the documentation. +Here are some heuristics for types of content we are excited to promote: -## Further Reading +- **Integration announcement:** If you announce the integration with a link to the LangChain documentation page, we are happy to re-tweet/re-share on Twitter/LinkedIn. +- **Educational content:** We highlight good educational content on the weekends - if you write a good blog or make a good YouTube video, we are happy to share there! Note that we prefer content that is NOT framed as "here's how to use integration XYZ", but rather "here's how to do ABC", as we find that is more educational and helpful for developers. +- **End-to-end applications:** End-to-end applications are great resources for developers looking to build. We prefer to highlight applications that are more complex/agentic in nature, and that use [LangGraph](https://github.com/langchain-ai/langgraph) as the orchestration framework. We get particularly excited about anything involving long-term memory, human-in-the-loop interaction patterns, or multi-agent architectures. +- **Research:** We love highlighting novel research! Whether it is research built on top of LangChain or that integrates with it. -To get started, let's learn [how to bootstrap a new integration package](./package.mdx) for LangChain. +// TODO: set up some form to gather these requests. \ No newline at end of file diff --git a/docs/docs/contributing/how_to/integrations/standard-tests-chat-model.md b/docs/docs/contributing/how_to/integrations/standard-tests-chat-model.md new file mode 100644 index 0000000000000..dee6a7f0133a5 --- /dev/null +++ b/docs/docs/contributing/how_to/integrations/standard-tests-chat-model.md @@ -0,0 +1,104 @@ +# Standard Tests (Chat Model) + +This guide walks through how to create standard tests for a custom Chat Model that you have developed. + +## Setup + +First we need to install certain dependencies. These include: + +- `pytest`: For running tests +- `pytest-socket`: For running unit tests +- `pytest-asyncio`: For testing async functionality +- `langchain-tests`: For importing standard tests +- `langchain-core`: This should already be installed, but is needed to define our integration. + +```shell +pip install -U langchain-core pytest pytest-socket pytest-asyncio langchain-tests +``` + +## Add and configure standard tests +There are 2 namespaces in the langchain-tests package: + +[unit tests](../../../concepts/testing.mdx#unit-tests) (langchain_tests.unit_tests): designed to be used to test the component in isolation and without access to external services +[integration tests](../../../concepts/testing.mdx#integration-tests) (langchain_tests.integration_tests): designed to be used to test the component with access to external services (in particular, the external service that the component is designed to interact with). + +Both types of tests are implemented as [pytest class-based test suites](https://docs.pytest.org/en/7.1.x/getting-started.html#group-multiple-tests-in-a-class). + +By subclassing the base classes for each type of standard test (see below), you get all of the standard tests for that type, and you can override the properties that the test suite uses to configure the tests. + +Here's how you would configure the standard unit tests for the custom chat model: + +```python +# title="tests/unit_tests/test_chat_models.py" +from typing import Type + +from my_package.chat_models import MyChatModel +from langchain_tests.unit_tests import ChatModelUnitTests + + +class TestChatParrotLinkUnit(ChatModelUnitTests): + @property + def chat_model_class(self) -> Type[MyChatModel]: + return MyChatModel + + @property + def chat_model_params(self) -> dict: + # These should be parameters used to initialize your integration for testing + return { + "model": "bird-brain-001", + "temperature": 0, + "parrot_buffer_length": 50, + } +``` + +```python +# title="tests/integration_tests/test_chat_models.py" +from typing import Type + +from my_package.chat_models import MyChatModel +from langchain_tests.integration_tests import ChatModelIntegrationTests + + +class TestChatParrotLinkIntegration(ChatModelIntegrationTests): + @property + def chat_model_class(self) -> Type[MyChatModel]: + return MyChatModel + + @property + def chat_model_params(self) -> dict: + # These should be parameters used to initialize your integration for testing + return { + "model": "bird-brain-001", + "temperature": 0, + "parrot_buffer_length": 50, + } +``` + +## Run standard tests + +After setting tests up, you would run these with the following commands from your project root: + +```shell +# run unit tests without network access +pytest --disable-socket --allow-unix-socket --asyncio-mode=auto tests/unit_tests + +# run integration tests +pytest --asyncio-mode=auto tests/integration_tests +``` + +## Test suite information and troubleshooting + +What tests are run to test this integration? + +If a test fails, what does that mean? + +You can find information on the tests run for this integration in the [Standard Tests API Reference](https://python.langchain.com/api_reference/standard_tests/index.html). + +// TODO: link to exact page for this integration test suite information + +## Skipping tests + +Sometimes you do not need your integration to pass all tests, as you only rely on specific subsets of functionality. +If this is the case, you can turn off specific tests by... + +// TODO: add instructions for how to turn off tests \ No newline at end of file From 027f49c661a48c17e433190a1d95af6f60de6c0d Mon Sep 17 00:00:00 2001 From: ccurme Date: Mon, 2 Dec 2024 17:55:57 -0500 Subject: [PATCH 2/6] docs[patch]: reorganize integration guides (#28457) Proposal is: - Each type of component (chat models, embeddings, etc) has a dedicated guide. - This guide contains detail on both implementation and testing via langchain-tests. - We delete the monolithic standard-tests guide. --- docs/docs/concepts/testing.mdx | 2 +- docs/docs/contributing/how_to/index.mdx | 1 - .../how_to/integrations/chat_models.md | 323 ++++++++++++ .../how_to/integrations/index.mdx | 29 +- .../how_to/integrations/package.mdx | 206 +------- .../how_to/integrations/publish.mdx | 4 +- .../integrations/standard-tests-chat-model.md | 104 ---- .../how_to/integrations/standard_tests.ipynb | 497 ------------------ docs/docs/contributing/index.mdx | 1 - docs/vercel.json | 4 + 10 files changed, 350 insertions(+), 821 deletions(-) create mode 100644 docs/docs/contributing/how_to/integrations/chat_models.md delete mode 100644 docs/docs/contributing/how_to/integrations/standard-tests-chat-model.md delete mode 100644 docs/docs/contributing/how_to/integrations/standard_tests.ipynb diff --git a/docs/docs/concepts/testing.mdx b/docs/docs/concepts/testing.mdx index cd0114f31e1b3..e125025e23510 100644 --- a/docs/docs/concepts/testing.mdx +++ b/docs/docs/concepts/testing.mdx @@ -78,4 +78,4 @@ class TestParrotMultiplyToolUnit(ToolsUnitTests): return {"a": 2, "b": 3} ``` -To learn more, check out our guide on [how to add standard tests to an integration](../../contributing/how_to/integrations/standard_tests). +To learn more, check out our guide guides on [contributing an integration](/docs/contributing/how_to/integrations/#standard-tests). diff --git a/docs/docs/contributing/how_to/index.mdx b/docs/docs/contributing/how_to/index.mdx index 675586340f427..340efb9599931 100644 --- a/docs/docs/contributing/how_to/index.mdx +++ b/docs/docs/contributing/how_to/index.mdx @@ -7,4 +7,3 @@ - [**Start Here**](integrations/index.mdx): Help us integrate with your favorite vendors and tools. - [**Package**](integrations/package): Publish an integration package to PyPi -- [**Standard Tests**](integrations/standard_tests): Ensure your integration passes an expected set of tests. diff --git a/docs/docs/contributing/how_to/integrations/chat_models.md b/docs/docs/contributing/how_to/integrations/chat_models.md new file mode 100644 index 0000000000000..e5e9bcc866c57 --- /dev/null +++ b/docs/docs/contributing/how_to/integrations/chat_models.md @@ -0,0 +1,323 @@ +--- +pagination_next: contributing/how_to/integrations/publish +pagination_prev: contributing/how_to/integrations/index +--- +# How to implement and test a chat model integration + +This guide walks through how to implement and test a custom [chat model](/docs/concepts/chat_models) that you have developed. + +For testing, we will rely on the `langchain-tests` dependency we added in the previous [bootstrapping guide](/docs/contributing/how_to/integrations/package). + +## Implementation + +Let's say you're building a simple integration package that provides a `ChatParrotLink` +chat model integration for LangChain. Here's a simple example of what your project +structure might look like: + +```plaintext +langchain-parrot-link/ +├── langchain_parrot_link/ +│ ├── __init__.py +│ └── chat_models.py +├── tests/ +│ ├── __init__.py +│ └── test_chat_models.py +├── pyproject.toml +└── README.md +``` + +Following the [bootstrapping guide](/docs/contributing/how_to/integrations/package), +all of these files should already exist, except for +`chat_models.py` and `test_chat_models.py`. We will implement these files in this guide. + +To implement `chat_models.py`, we copy the [implementation](/docs/how_to/custom_chat_model/#implementation) from our +[Custom Chat Model Guide](/docs/how_to/custom_chat_model). Refer to that guide for more detail. + +
+ chat_models.py +```python title="langchain_parrot_link/chat_models.py" +from typing import Any, Dict, Iterator, List, Optional + +from langchain_core.callbacks import ( + CallbackManagerForLLMRun, +) +from langchain_core.language_models import BaseChatModel +from langchain_core.messages import ( + AIMessage, + AIMessageChunk, + BaseMessage, +) +from langchain_core.messages.ai import UsageMetadata +from langchain_core.outputs import ChatGeneration, ChatGenerationChunk, ChatResult +from pydantic import Field + + +class ChatParrotLink(BaseChatModel): + """A custom chat model that echoes the first `parrot_buffer_length` characters + of the input. + + When contributing an implementation to LangChain, carefully document + the model including the initialization parameters, include + an example of how to initialize the model and include any relevant + links to the underlying models documentation or API. + + Example: + + .. code-block:: python + + model = ChatParrotLink(parrot_buffer_length=2, model="bird-brain-001") + result = model.invoke([HumanMessage(content="hello")]) + result = model.batch([[HumanMessage(content="hello")], + [HumanMessage(content="world")]]) + """ + + model_name: str = Field(alias="model") + """The name of the model""" + parrot_buffer_length: int + """The number of characters from the last message of the prompt to be echoed.""" + temperature: Optional[float] = None + max_tokens: Optional[int] = None + timeout: Optional[int] = None + stop: Optional[List[str]] = None + max_retries: int = 2 + + def _generate( + self, + messages: List[BaseMessage], + stop: Optional[List[str]] = None, + run_manager: Optional[CallbackManagerForLLMRun] = None, + **kwargs: Any, + ) -> ChatResult: + """Override the _generate method to implement the chat model logic. + + This can be a call to an API, a call to a local model, or any other + implementation that generates a response to the input prompt. + + Args: + messages: the prompt composed of a list of messages. + stop: a list of strings on which the model should stop generating. + If generation stops due to a stop token, the stop token itself + SHOULD BE INCLUDED as part of the output. This is not enforced + across models right now, but it's a good practice to follow since + it makes it much easier to parse the output of the model + downstream and understand why generation stopped. + run_manager: A run manager with callbacks for the LLM. + """ + # Replace this with actual logic to generate a response from a list + # of messages. + last_message = messages[-1] + tokens = last_message.content[: self.parrot_buffer_length] + ct_input_tokens = sum(len(message.content) for message in messages) + ct_output_tokens = len(tokens) + message = AIMessage( + content=tokens, + additional_kwargs={}, # Used to add additional payload to the message + response_metadata={ # Use for response metadata + "time_in_seconds": 3, + }, + usage_metadata={ + "input_tokens": ct_input_tokens, + "output_tokens": ct_output_tokens, + "total_tokens": ct_input_tokens + ct_output_tokens, + }, + ) + ## + + generation = ChatGeneration(message=message) + return ChatResult(generations=[generation]) + + def _stream( + self, + messages: List[BaseMessage], + stop: Optional[List[str]] = None, + run_manager: Optional[CallbackManagerForLLMRun] = None, + **kwargs: Any, + ) -> Iterator[ChatGenerationChunk]: + """Stream the output of the model. + + This method should be implemented if the model can generate output + in a streaming fashion. If the model does not support streaming, + do not implement it. In that case streaming requests will be automatically + handled by the _generate method. + + Args: + messages: the prompt composed of a list of messages. + stop: a list of strings on which the model should stop generating. + If generation stops due to a stop token, the stop token itself + SHOULD BE INCLUDED as part of the output. This is not enforced + across models right now, but it's a good practice to follow since + it makes it much easier to parse the output of the model + downstream and understand why generation stopped. + run_manager: A run manager with callbacks for the LLM. + """ + last_message = messages[-1] + tokens = str(last_message.content[: self.parrot_buffer_length]) + ct_input_tokens = sum(len(message.content) for message in messages) + + for token in tokens: + usage_metadata = UsageMetadata( + { + "input_tokens": ct_input_tokens, + "output_tokens": 1, + "total_tokens": ct_input_tokens + 1, + } + ) + ct_input_tokens = 0 + chunk = ChatGenerationChunk( + message=AIMessageChunk(content=token, usage_metadata=usage_metadata) + ) + + if run_manager: + # This is optional in newer versions of LangChain + # The on_llm_new_token will be called automatically + run_manager.on_llm_new_token(token, chunk=chunk) + + yield chunk + + # Let's add some other information (e.g., response metadata) + chunk = ChatGenerationChunk( + message=AIMessageChunk(content="", response_metadata={"time_in_sec": 3}) + ) + if run_manager: + # This is optional in newer versions of LangChain + # The on_llm_new_token will be called automatically + run_manager.on_llm_new_token(token, chunk=chunk) + yield chunk + + @property + def _llm_type(self) -> str: + """Get the type of language model used by this chat model.""" + return "echoing-chat-model-advanced" + + @property + def _identifying_params(self) -> Dict[str, Any]: + """Return a dictionary of identifying parameters. + + This information is used by the LangChain callback system, which + is used for tracing purposes make it possible to monitor LLMs. + """ + return { + # The model name allows users to specify custom token counting + # rules in LLM monitoring applications (e.g., in LangSmith users + # can provide per token pricing for their model and monitor + # costs for the given LLM.) + "model_name": self.model_name, + } +``` +
+ +## Testing + +To implement our test files, we will subclass test classes from the `langchain_tests` package. These test classes contain the tests that will be run. We will just need to configure what model is tested, what parameters it is tested with, and specify any tests that should be skipped. + +### Setup + +First we need to install certain dependencies. These include: + +- `pytest`: For running tests +- `pytest-socket`: For running unit tests +- `pytest-asyncio`: For testing async functionality +- `langchain-tests`: For importing standard tests +- `langchain-core`: This should already be installed, but is needed to define our integration. + +If you followed the previous [bootstrapping guide](/docs/contributing/how_to/integrations/package/), these should already be installed. + +### Add and configure standard tests +There are two namespaces in the langchain-tests package: + +[unit tests](../../../concepts/testing.mdx#unit-tests) (`langchain_tests.unit_tests`): designed to be used to test the component in isolation and without access to external services +[integration tests](../../../concepts/testing.mdx#integration-tests) (`langchain_tests.integration_tests`): designed to be used to test the component with access to external services (in particular, the external service that the component is designed to interact with). + +Both types of tests are implemented as [pytest class-based test suites](https://docs.pytest.org/en/7.1.x/getting-started.html#group-multiple-tests-in-a-class). + +By subclassing the base classes for each type of standard test (see below), you get all of the standard tests for that type, and you can override the properties that the test suite uses to configure the tests. + +Here's how you would configure the standard unit tests for the custom chat model: + +```python +# title="tests/unit_tests/test_chat_models.py" +from typing import Type + +from my_package.chat_models import MyChatModel +from langchain_tests.unit_tests import ChatModelUnitTests + + +class TestChatParrotLinkUnit(ChatModelUnitTests): + @property + def chat_model_class(self) -> Type[MyChatModel]: + return MyChatModel + + @property + def chat_model_params(self) -> dict: + # These should be parameters used to initialize your integration for testing + return { + "model": "bird-brain-001", + "temperature": 0, + "parrot_buffer_length": 50, + } +``` + +And here is the corresponding snippet for integration tests: + +```python +# title="tests/integration_tests/test_chat_models.py" +from typing import Type + +from my_package.chat_models import MyChatModel +from langchain_tests.integration_tests import ChatModelIntegrationTests + + +class TestChatParrotLinkIntegration(ChatModelIntegrationTests): + @property + def chat_model_class(self) -> Type[MyChatModel]: + return MyChatModel + + @property + def chat_model_params(self) -> dict: + # These should be parameters used to initialize your integration for testing + return { + "model": "bird-brain-001", + "temperature": 0, + "parrot_buffer_length": 50, + } +``` + +These two snippets should be written into `tests/unit_tests/test_chat_models.py` and `tests/integration_tests/test_chat_models.py`, respectively. + +:::note + +LangChain standard tests test a range of behaviors, from the most basic requirements to optional capabilities like multi-modal support. The above implementation will likely need to be updated to specify any tests that should be ignored. See [below](#skipping-tests) for detail. + +::: + +### Run standard tests + +After setting tests up, you would run these with the following commands from your project root: + +```shell +# run unit tests without network access +pytest --disable-socket --allow-unix-socket --asyncio-mode=auto tests/unit_tests + +# run integration tests +pytest --asyncio-mode=auto tests/integration_tests +``` + +Our objective is for the pytest run to be successful. That is, + +1. If a feature is intended to be supported by the model, it passes; +2. If a feature is not intended to be supported by the model, it is skipped. + +### Skipping tests + +LangChain standard tests test a range of behaviors, from the most basic requirements (generating a response to a query) to optional capabilities like multi-modal support, tool-calling, or support for messages generated from other providers. Tests for "optional" capabilities are controlled via a [set of properties](https://python.langchain.com/api_reference/standard_tests/unit_tests/langchain_tests.unit_tests.chat_models.ChatModelTests.html) that can be overridden on the test model subclass. + + +### Test suite information and troubleshooting + +What tests are run to test this integration? + +If a test fails, what does that mean? + +You can find information on the tests run for this integration in the [Standard Tests API Reference](https://python.langchain.com/api_reference/standard_tests/index.html). + +// TODO: link to exact page for this integration test suite information diff --git a/docs/docs/contributing/how_to/integrations/index.mdx b/docs/docs/contributing/how_to/integrations/index.mdx index b4f16f854029c..913057a477528 100644 --- a/docs/docs/contributing/how_to/integrations/index.mdx +++ b/docs/docs/contributing/how_to/integrations/index.mdx @@ -64,12 +64,23 @@ While any component can be integrated into LangChain, there are specific types o In order to contribute an integration, you should follow these steps: 1. Confirm that your integration is in the [list of components](#components-to-integrate) we are currently encouraging. -2. Implement your integration -3. Implement the standard tests ([see below](#standard-tests)) for your integration and successfully run them. -4. Publish your integration in a Python package to PyPi. +2. [Bootstrap your integration](/docs/contributing/how_to/integrations/package/). +3. Implement and test your integration following the [component-specific guides](#component-specific-guides). +4. [Publish your integration](/docs/contributing/how_to/integrations/publish/) in a Python package to PyPi. 5. [Optional] Open and merge a PR to add documentation for your integration to the official LangChain docs. 6. [Optional] Engage with the LangChain team for joint co-marketing ([see below](#co-marketing)). +## Component-specific guides + +The guides below walk you through both implementing and testing specific components: + +- [Chat Models](/docs/contributing/how_to/integrations/chat_models) +- [Tools] +- [Toolkits] +- [Retrievers] +- [Document Loaders] +- [Vector Stores] +- [Embedding Models] ## Standard Tests @@ -81,17 +92,7 @@ These standard tests help maintain compatibility between different components an **Integration Tests**: Integration tests validate that multiple components or systems work together as expected. For tools or integrations relying on external services, these tests often ensure end-to-end functionality. -Each type of integration has its own set of standard tests. Please see the correct guide for setting up standard tests for your integration: - -- [Chat Models] -- [Tools] -- [Toolkits] -- [Retrievers] -- [Document Loaders] -- [Retrievers] -- [Document Loaders] -- [Vector Stores] -- [Embedding Models] +Each type of integration has its own set of standard tests. Please see the relevant [component-specific guide](#component-specific-guides) for details on testing your integration. ## Co-Marketing diff --git a/docs/docs/contributing/how_to/integrations/package.mdx b/docs/docs/contributing/how_to/integrations/package.mdx index 8189ce97f8ba3..e2ba8b78ff9a5 100644 --- a/docs/docs/contributing/how_to/integrations/package.mdx +++ b/docs/docs/contributing/how_to/integrations/package.mdx @@ -1,6 +1,6 @@ --- -pagination_next: contributing/how_to/integrations/standard_tests -pagination_prev: contributing/how_to/integrations/index +pagination_next: contributing/how_to/integrations/index +pagination_prev: null --- # How to bootstrap a new integration package @@ -46,7 +46,7 @@ We will also add some `test` dependencies in a separate poetry dependency group. you are not using Poetry, we recommend adding these in a way that won't package them with your published package, or just installing them separately when you run tests. -`langchain-tests` will provide the [standard tests](../standard_tests) we will use later. +`langchain-tests` will provide the [standard tests](/docs/contributing/how_to/integrations/#standard-tests) we will use later. We recommended pinning these to the latest version: Note: Replace `` with the latest version of `langchain-tests` below. @@ -64,203 +64,7 @@ poetry install --with test You're now ready to start writing your integration package! -## Writing your integration - -Let's say you're building a simple integration package that provides a `ChatParrotLink` -chat model integration for LangChain. Here's a simple example of what your project -structure might look like: - -```plaintext -langchain-parrot-link/ -├── langchain_parrot_link/ -│ ├── __init__.py -│ └── chat_models.py -├── tests/ -│ ├── __init__.py -│ └── test_chat_models.py -├── pyproject.toml -└── README.md -``` - -All of these files should already exist from step 1, except for -`chat_models.py` and `test_chat_models.py`! We will implement `test_chat_models.py` -later, following the [standard tests](../standard_tests) guide. - -To implement `chat_models.py`, let's copy the implementation from our -[Custom Chat Model Guide](../../../../how_to/custom_chat_model). - -
- chat_models.py -```python title="langchain_parrot_link/chat_models.py" -from typing import Any, Dict, Iterator, List, Optional - -from langchain_core.callbacks import ( - CallbackManagerForLLMRun, -) -from langchain_core.language_models import BaseChatModel -from langchain_core.messages import ( - AIMessage, - AIMessageChunk, - BaseMessage, -) -from langchain_core.messages.ai import UsageMetadata -from langchain_core.outputs import ChatGeneration, ChatGenerationChunk, ChatResult -from pydantic import Field - - -class ChatParrotLink(BaseChatModel): - """A custom chat model that echoes the first `parrot_buffer_length` characters - of the input. - - When contributing an implementation to LangChain, carefully document - the model including the initialization parameters, include - an example of how to initialize the model and include any relevant - links to the underlying models documentation or API. - - Example: - - .. code-block:: python - - model = ChatParrotLink(parrot_buffer_length=2, model="bird-brain-001") - result = model.invoke([HumanMessage(content="hello")]) - result = model.batch([[HumanMessage(content="hello")], - [HumanMessage(content="world")]]) - """ - - model_name: str = Field(alias="model") - """The name of the model""" - parrot_buffer_length: int - """The number of characters from the last message of the prompt to be echoed.""" - temperature: Optional[float] = None - max_tokens: Optional[int] = None - timeout: Optional[int] = None - stop: Optional[List[str]] = None - max_retries: int = 2 - - def _generate( - self, - messages: List[BaseMessage], - stop: Optional[List[str]] = None, - run_manager: Optional[CallbackManagerForLLMRun] = None, - **kwargs: Any, - ) -> ChatResult: - """Override the _generate method to implement the chat model logic. - - This can be a call to an API, a call to a local model, or any other - implementation that generates a response to the input prompt. - - Args: - messages: the prompt composed of a list of messages. - stop: a list of strings on which the model should stop generating. - If generation stops due to a stop token, the stop token itself - SHOULD BE INCLUDED as part of the output. This is not enforced - across models right now, but it's a good practice to follow since - it makes it much easier to parse the output of the model - downstream and understand why generation stopped. - run_manager: A run manager with callbacks for the LLM. - """ - # Replace this with actual logic to generate a response from a list - # of messages. - last_message = messages[-1] - tokens = last_message.content[: self.parrot_buffer_length] - ct_input_tokens = sum(len(message.content) for message in messages) - ct_output_tokens = len(tokens) - message = AIMessage( - content=tokens, - additional_kwargs={}, # Used to add additional payload to the message - response_metadata={ # Use for response metadata - "time_in_seconds": 3, - }, - usage_metadata={ - "input_tokens": ct_input_tokens, - "output_tokens": ct_output_tokens, - "total_tokens": ct_input_tokens + ct_output_tokens, - }, - ) - ## - - generation = ChatGeneration(message=message) - return ChatResult(generations=[generation]) - - def _stream( - self, - messages: List[BaseMessage], - stop: Optional[List[str]] = None, - run_manager: Optional[CallbackManagerForLLMRun] = None, - **kwargs: Any, - ) -> Iterator[ChatGenerationChunk]: - """Stream the output of the model. - - This method should be implemented if the model can generate output - in a streaming fashion. If the model does not support streaming, - do not implement it. In that case streaming requests will be automatically - handled by the _generate method. - - Args: - messages: the prompt composed of a list of messages. - stop: a list of strings on which the model should stop generating. - If generation stops due to a stop token, the stop token itself - SHOULD BE INCLUDED as part of the output. This is not enforced - across models right now, but it's a good practice to follow since - it makes it much easier to parse the output of the model - downstream and understand why generation stopped. - run_manager: A run manager with callbacks for the LLM. - """ - last_message = messages[-1] - tokens = str(last_message.content[: self.parrot_buffer_length]) - ct_input_tokens = sum(len(message.content) for message in messages) - - for token in tokens: - usage_metadata = UsageMetadata( - { - "input_tokens": ct_input_tokens, - "output_tokens": 1, - "total_tokens": ct_input_tokens + 1, - } - ) - ct_input_tokens = 0 - chunk = ChatGenerationChunk( - message=AIMessageChunk(content=token, usage_metadata=usage_metadata) - ) - - if run_manager: - # This is optional in newer versions of LangChain - # The on_llm_new_token will be called automatically - run_manager.on_llm_new_token(token, chunk=chunk) - - yield chunk - - # Let's add some other information (e.g., response metadata) - chunk = ChatGenerationChunk( - message=AIMessageChunk(content="", response_metadata={"time_in_sec": 3}) - ) - if run_manager: - # This is optional in newer versions of LangChain - # The on_llm_new_token will be called automatically - run_manager.on_llm_new_token(token, chunk=chunk) - yield chunk - - @property - def _llm_type(self) -> str: - """Get the type of language model used by this chat model.""" - return "echoing-chat-model-advanced" - - @property - def _identifying_params(self) -> Dict[str, Any]: - """Return a dictionary of identifying parameters. - - This information is used by the LangChain callback system, which - is used for tracing purposes make it possible to monitor LLMs. - """ - return { - # The model name allows users to specify custom token counting - # rules in LLM monitoring applications (e.g., in LangSmith users - # can provide per token pricing for their model and monitor - # costs for the given LLM.) - "model_name": self.model_name, - } -``` -
+See our [component-specific guides](/docs/contributing/how_to/integrations/#component-specific-guides) for detail on implementing and testing each component. ## Push your package to a public Github repository @@ -272,4 +76,4 @@ This is only required if you want to publish your integration in the LangChain d ## Next Steps -Now that you've implemented your package, you can move on to [testing your integration](../standard_tests) for your integration and successfully run them. +Now that you've bootstrapped your package, you can move on to [implementing and testing](/docs/contributing/how_to/integrations/#component-specific-guides) your integration. diff --git a/docs/docs/contributing/how_to/integrations/publish.mdx b/docs/docs/contributing/how_to/integrations/publish.mdx index 72b5e2e3634a5..3e4f86364d4c8 100644 --- a/docs/docs/contributing/how_to/integrations/publish.mdx +++ b/docs/docs/contributing/how_to/integrations/publish.mdx @@ -1,5 +1,5 @@ --- -pagination_prev: contributing/how_to/integrations/standard_tests +pagination_prev: contributing/how_to/integrations/index pagination_next: null --- @@ -12,7 +12,7 @@ Now that your package is implemented and tested, you can: ## Publishing your package to PyPi -This guide assumes you have already implemented your package and written tests for it. If you haven't done that yet, please refer to the [implementation guide](../package) and the [testing guide](../standard_tests). +This guide assumes you have already implemented your package and written tests for it. If you haven't done that yet, please refer to the [component-specific guides](/docs/contributing/how_to/integrations/#component-specific-guides). Note that Poetry is not required to publish a package to PyPi, and we're using it in this guide end-to-end for convenience. You are welcome to publish your package using any other method you prefer. diff --git a/docs/docs/contributing/how_to/integrations/standard-tests-chat-model.md b/docs/docs/contributing/how_to/integrations/standard-tests-chat-model.md deleted file mode 100644 index dee6a7f0133a5..0000000000000 --- a/docs/docs/contributing/how_to/integrations/standard-tests-chat-model.md +++ /dev/null @@ -1,104 +0,0 @@ -# Standard Tests (Chat Model) - -This guide walks through how to create standard tests for a custom Chat Model that you have developed. - -## Setup - -First we need to install certain dependencies. These include: - -- `pytest`: For running tests -- `pytest-socket`: For running unit tests -- `pytest-asyncio`: For testing async functionality -- `langchain-tests`: For importing standard tests -- `langchain-core`: This should already be installed, but is needed to define our integration. - -```shell -pip install -U langchain-core pytest pytest-socket pytest-asyncio langchain-tests -``` - -## Add and configure standard tests -There are 2 namespaces in the langchain-tests package: - -[unit tests](../../../concepts/testing.mdx#unit-tests) (langchain_tests.unit_tests): designed to be used to test the component in isolation and without access to external services -[integration tests](../../../concepts/testing.mdx#integration-tests) (langchain_tests.integration_tests): designed to be used to test the component with access to external services (in particular, the external service that the component is designed to interact with). - -Both types of tests are implemented as [pytest class-based test suites](https://docs.pytest.org/en/7.1.x/getting-started.html#group-multiple-tests-in-a-class). - -By subclassing the base classes for each type of standard test (see below), you get all of the standard tests for that type, and you can override the properties that the test suite uses to configure the tests. - -Here's how you would configure the standard unit tests for the custom chat model: - -```python -# title="tests/unit_tests/test_chat_models.py" -from typing import Type - -from my_package.chat_models import MyChatModel -from langchain_tests.unit_tests import ChatModelUnitTests - - -class TestChatParrotLinkUnit(ChatModelUnitTests): - @property - def chat_model_class(self) -> Type[MyChatModel]: - return MyChatModel - - @property - def chat_model_params(self) -> dict: - # These should be parameters used to initialize your integration for testing - return { - "model": "bird-brain-001", - "temperature": 0, - "parrot_buffer_length": 50, - } -``` - -```python -# title="tests/integration_tests/test_chat_models.py" -from typing import Type - -from my_package.chat_models import MyChatModel -from langchain_tests.integration_tests import ChatModelIntegrationTests - - -class TestChatParrotLinkIntegration(ChatModelIntegrationTests): - @property - def chat_model_class(self) -> Type[MyChatModel]: - return MyChatModel - - @property - def chat_model_params(self) -> dict: - # These should be parameters used to initialize your integration for testing - return { - "model": "bird-brain-001", - "temperature": 0, - "parrot_buffer_length": 50, - } -``` - -## Run standard tests - -After setting tests up, you would run these with the following commands from your project root: - -```shell -# run unit tests without network access -pytest --disable-socket --allow-unix-socket --asyncio-mode=auto tests/unit_tests - -# run integration tests -pytest --asyncio-mode=auto tests/integration_tests -``` - -## Test suite information and troubleshooting - -What tests are run to test this integration? - -If a test fails, what does that mean? - -You can find information on the tests run for this integration in the [Standard Tests API Reference](https://python.langchain.com/api_reference/standard_tests/index.html). - -// TODO: link to exact page for this integration test suite information - -## Skipping tests - -Sometimes you do not need your integration to pass all tests, as you only rely on specific subsets of functionality. -If this is the case, you can turn off specific tests by... - -// TODO: add instructions for how to turn off tests \ No newline at end of file diff --git a/docs/docs/contributing/how_to/integrations/standard_tests.ipynb b/docs/docs/contributing/how_to/integrations/standard_tests.ipynb deleted file mode 100644 index 8bee8a45dd200..0000000000000 --- a/docs/docs/contributing/how_to/integrations/standard_tests.ipynb +++ /dev/null @@ -1,497 +0,0 @@ -{ - "cells": [ - { - "cell_type": "raw", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, - "source": [ - "---\n", - "pagination_next: contributing/how_to/integrations/publish\n", - "pagination_prev: contributing/how_to/integrations/package\n", - "---" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# How to add standard tests to an integration\n", - "\n", - "When creating either a custom class for yourself or to publish in a LangChain integration, it is important to add standard tests to ensure it works as expected. This guide will show you how to add standard tests to a custom chat model, and you can **[Skip to the test templates](#standard-test-templates-per-component)** for implementing tests for each integration type.\n", - "\n", - "## Setup\n", - "\n", - "If you're coming from the [previous guide](../package), you have already installed these dependencies, and you can skip this section.\n", - "\n", - "First, let's install 2 dependencies:\n", - "\n", - "- `langchain-core` will define the interfaces we want to import to define our custom tool.\n", - "- `langchain-tests` will provide the standard tests we want to use. Recommended to pin to the latest version: \n", - "\n", - ":::note\n", - "\n", - "Because added tests in new versions of `langchain-tests` can break your CI/CD pipelines, we recommend pinning the \n", - "version of `langchain-tests` to avoid unexpected changes.\n", - "\n", - ":::\n", - "\n", - "import Tabs from '@theme/Tabs';\n", - "import TabItem from '@theme/TabItem';\n", - "\n", - "\n", - " \n", - "If you followed the [previous guide](../package), you should already have these dependencies installed!\n", - "\n", - "```bash\n", - "poetry add langchain-core\n", - "poetry add --group test pytest pytest-socket pytest-asyncio langchain-tests==\n", - "poetry install --with test\n", - "```\n", - " \n", - " \n", - "```bash\n", - "pip install -U langchain-core pytest pytest-socket pytest-asyncio langchain-tests\n", - "\n", - "# install current package in editable mode\n", - "pip install --editable .\n", - "```\n", - " \n", - "" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Let's say we're publishing a package, `langchain_parrot_link`, that exposes the chat model from the [guide on implementing the package](../package). We can add the standard tests to the package by following the steps below." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "And we'll assume you've structured your package the same way as the main LangChain\n", - "packages:\n", - "\n", - "```plaintext\n", - "langchain-parrot-link/\n", - "├── langchain_parrot_link/\n", - "│ ├── __init__.py\n", - "│ └── chat_models.py\n", - "├── tests/\n", - "│ ├── __init__.py\n", - "│ └── test_chat_models.py\n", - "├── pyproject.toml\n", - "└── README.md\n", - "```\n", - "\n", - "## Add and configure standard tests\n", - "\n", - "There are 2 namespaces in the `langchain-tests` package: \n", - "\n", - "- [unit tests](../../../concepts/testing.mdx#unit-tests) (`langchain_tests.unit_tests`): designed to be used to test the component in isolation and without access to external services\n", - "- [integration tests](../../../concepts/testing.mdx#unit-tests) (`langchain_tests.integration_tests`): designed to be used to test the component with access to external services (in particular, the external service that the component is designed to interact with).\n", - "\n", - "Both types of tests are implemented as [`pytest` class-based test suites](https://docs.pytest.org/en/7.1.x/getting-started.html#group-multiple-tests-in-a-class).\n", - "\n", - "By subclassing the base classes for each type of standard test (see below), you get all of the standard tests for that type, and you\n", - "can override the properties that the test suite uses to configure the tests.\n", - "\n", - "### Standard chat model tests\n", - "\n", - "Here's how you would configure the standard unit tests for the custom chat model:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# title=\"tests/unit_tests/test_chat_models.py\"\n", - "from typing import Tuple, Type\n", - "\n", - "from langchain_parrot_link.chat_models import ChatParrotLink\n", - "from langchain_tests.unit_tests import ChatModelUnitTests\n", - "\n", - "\n", - "class TestChatParrotLinkUnit(ChatModelUnitTests):\n", - " @property\n", - " def chat_model_class(self) -> Type[ChatParrotLink]:\n", - " return ChatParrotLink\n", - "\n", - " @property\n", - " def chat_model_params(self) -> dict:\n", - " return {\n", - " \"model\": \"bird-brain-001\",\n", - " \"temperature\": 0,\n", - " \"parrot_buffer_length\": 50,\n", - " }" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# title=\"tests/integration_tests/test_chat_models.py\"\n", - "from typing import Type\n", - "\n", - "from langchain_parrot_link.chat_models import ChatParrotLink\n", - "from langchain_tests.integration_tests import ChatModelIntegrationTests\n", - "\n", - "\n", - "class TestChatParrotLinkIntegration(ChatModelIntegrationTests):\n", - " @property\n", - " def chat_model_class(self) -> Type[ChatParrotLink]:\n", - " return ChatParrotLink\n", - "\n", - " @property\n", - " def chat_model_params(self) -> dict:\n", - " return {\n", - " \"model\": \"bird-brain-001\",\n", - " \"temperature\": 0,\n", - " \"parrot_buffer_length\": 50,\n", - " }" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "and you would run these with the following commands from your project root\n", - "\n", - "\n", - " \n", - "\n", - "```bash\n", - "# run unit tests without network access\n", - "poetry run pytest --disable-socket --allow-unix-socket --asyncio-mode=auto tests/unit_tests\n", - "\n", - "# run integration tests\n", - "poetry run pytest --asyncio-mode=auto tests/integration_tests\n", - "```\n", - "\n", - " \n", - " \n", - "\n", - "```bash\n", - "# run unit tests without network access\n", - "pytest --disable-socket --allow-unix-socket --asyncio-mode=auto tests/unit_tests\n", - "\n", - "# run integration tests\n", - "pytest --asyncio-mode=auto tests/integration_tests\n", - "```\n", - "\n", - " \n", - "" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Test suite information and troubleshooting\n", - "\n", - "For a full list of the standard test suites that are available, as well as\n", - "information on which tests are included and how to troubleshoot common issues,\n", - "see the [Standard Tests API Reference](https://python.langchain.com/api_reference/standard_tests/index.html).\n", - "\n", - "An increasing number of troubleshooting guides are being added to this documentation,\n", - "and if you're interested in contributing, feel free to add docstrings to tests in \n", - "[Github](https://github.com/langchain-ai/langchain/tree/master/libs/standard-tests/langchain_tests)!" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Standard test templates per component:\n", - "\n", - "Above, we implement the **unit** and **integration** standard tests for a tool. Below are the templates for implementing the standard tests for each component:\n", - "\n", - "
\n", - " Chat Models\n", - "

Note: The standard tests for chat models are implemented in the example in the main body of this guide too.

" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# title=\"tests/unit_tests/test_chat_models.py\"\n", - "from typing import Type\n", - "\n", - "from langchain_parrot_link.chat_models import ChatParrotLink\n", - "from langchain_tests.unit_tests import ChatModelUnitTests\n", - "\n", - "\n", - "class TestChatParrotLinkUnit(ChatModelUnitTests):\n", - " @property\n", - " def chat_model_class(self) -> Type[ChatParrotLink]:\n", - " return ChatParrotLink\n", - "\n", - " @property\n", - " def chat_model_params(self) -> dict:\n", - " return {\n", - " \"model\": \"bird-brain-001\",\n", - " \"temperature\": 0,\n", - " \"parrot_buffer_length\": 50,\n", - " }" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# title=\"tests/integration_tests/test_chat_models.py\"\n", - "from typing import Type\n", - "\n", - "from langchain_parrot_link.chat_models import ChatParrotLink\n", - "from langchain_tests.integration_tests import ChatModelIntegrationTests\n", - "\n", - "\n", - "class TestChatParrotLinkIntegration(ChatModelIntegrationTests):\n", - " @property\n", - " def chat_model_class(self) -> Type[ChatParrotLink]:\n", - " return ChatParrotLink\n", - "\n", - " @property\n", - " def chat_model_params(self) -> dict:\n", - " return {\n", - " \"model\": \"bird-brain-001\",\n", - " \"temperature\": 0,\n", - " \"parrot_buffer_length\": 50,\n", - " }" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "
\n", - "
\n", - " Embedding Models" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# title=\"tests/unit_tests/test_embeddings.py\"\n", - "from typing import Tuple, Type\n", - "\n", - "from langchain_parrot_link.embeddings import ParrotLinkEmbeddings\n", - "from langchain_tests.unit_tests import EmbeddingsUnitTests\n", - "\n", - "\n", - "class TestParrotLinkEmbeddingsUnit(EmbeddingsUnitTests):\n", - " @property\n", - " def embeddings_class(self) -> Type[ParrotLinkEmbeddings]:\n", - " return ParrotLinkEmbeddings\n", - "\n", - " @property\n", - " def embedding_model_params(self) -> dict:\n", - " return {\"model\": \"nest-embed-001\", \"temperature\": 0}" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# title=\"tests/integration_tests/test_embeddings.py\"\n", - "from typing import Type\n", - "\n", - "from langchain_parrot_link.embeddings import ParrotLinkEmbeddings\n", - "from langchain_tests.integration_tests import EmbeddingsIntegrationTests\n", - "\n", - "\n", - "class TestParrotLinkEmbeddingsIntegration(EmbeddingsIntegrationTests):\n", - " @property\n", - " def embeddings_class(self) -> Type[ParrotLinkEmbeddings]:\n", - " return ParrotLinkEmbeddings\n", - "\n", - " @property\n", - " def embedding_model_params(self) -> dict:\n", - " return {\"model\": \"nest-embed-001\"}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "
\n", - "
\n", - " Tools/Toolkits" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# title=\"tests/unit_tests/test_tools.py\"\n", - "from typing import Type\n", - "\n", - "from langchain_parrot_link.tools import ParrotMultiplyTool\n", - "from langchain_tests.unit_tests import ToolsUnitTests\n", - "\n", - "\n", - "class TestParrotMultiplyToolUnit(ToolsUnitTests):\n", - " @property\n", - " def tool_constructor(self) -> Type[ParrotMultiplyTool]:\n", - " return ParrotMultiplyTool\n", - "\n", - " @property\n", - " def tool_constructor_params(self) -> dict:\n", - " # if your tool constructor instead required initialization arguments like\n", - " # `def __init__(self, some_arg: int):`, you would return those here\n", - " # as a dictionary, e.g.: `return {'some_arg': 42}`\n", - " return {}\n", - "\n", - " @property\n", - " def tool_invoke_params_example(self) -> dict:\n", - " \"\"\"\n", - " Returns a dictionary representing the \"args\" of an example tool call.\n", - "\n", - " This should NOT be a ToolCall dict - i.e. it should not\n", - " have {\"name\", \"id\", \"args\"} keys.\n", - " \"\"\"\n", - " return {\"a\": 2, \"b\": 3}" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# title=\"tests/integration_tests/test_tools.py\"\n", - "from typing import Type\n", - "\n", - "from langchain_parrot_link.tools import ParrotMultiplyTool\n", - "from langchain_tests.integration_tests import ToolsIntegrationTests\n", - "\n", - "\n", - "class TestParrotMultiplyToolIntegration(ToolsIntegrationTests):\n", - " @property\n", - " def tool_constructor(self) -> Type[ParrotMultiplyTool]:\n", - " return ParrotMultiplyTool\n", - "\n", - " @property\n", - " def tool_constructor_params(self) -> dict:\n", - " # if your tool constructor instead required initialization arguments like\n", - " # `def __init__(self, some_arg: int):`, you would return those here\n", - " # as a dictionary, e.g.: `return {'some_arg': 42}`\n", - " return {}\n", - "\n", - " @property\n", - " def tool_invoke_params_example(self) -> dict:\n", - " \"\"\"\n", - " Returns a dictionary representing the \"args\" of an example tool call.\n", - "\n", - " This should NOT be a ToolCall dict - i.e. it should not\n", - " have {\"name\", \"id\", \"args\"} keys.\n", - " \"\"\"\n", - " return {\"a\": 2, \"b\": 3}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "
\n", - "
\n", - " Vector Stores" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# title=\"tests/integration_tests/test_vectorstores_sync.py\"\n", - "\n", - "from typing import AsyncGenerator, Generator\n", - "\n", - "import pytest\n", - "from langchain_core.vectorstores import VectorStore\n", - "from langchain_parrot_link.vectorstores import ParrotVectorStore\n", - "from langchain_standard_tests.integration_tests.vectorstores import (\n", - " AsyncReadWriteTestSuite,\n", - " ReadWriteTestSuite,\n", - ")\n", - "\n", - "\n", - "class TestSync(ReadWriteTestSuite):\n", - " @pytest.fixture()\n", - " def vectorstore(self) -> Generator[VectorStore, None, None]: # type: ignore\n", - " \"\"\"Get an empty vectorstore for unit tests.\"\"\"\n", - " store = ParrotVectorStore()\n", - " # note: store should be EMPTY at this point\n", - " # if you need to delete data, you may do so here\n", - " try:\n", - " yield store\n", - " finally:\n", - " # cleanup operations, or deleting data\n", - " pass\n", - "\n", - "\n", - "class TestAsync(AsyncReadWriteTestSuite):\n", - " @pytest.fixture()\n", - " async def vectorstore(self) -> AsyncGenerator[VectorStore, None]: # type: ignore\n", - " \"\"\"Get an empty vectorstore for unit tests.\"\"\"\n", - " store = ParrotVectorStore()\n", - " # note: store should be EMPTY at this point\n", - " # if you need to delete data, you may do so here\n", - " try:\n", - " yield store\n", - " finally:\n", - " # cleanup operations, or deleting data\n", - " pass" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "
" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": ".venv", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.11.4" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/docs/docs/contributing/index.mdx b/docs/docs/contributing/index.mdx index d2c1231789d8d..67930710585dc 100644 --- a/docs/docs/contributing/index.mdx +++ b/docs/docs/contributing/index.mdx @@ -17,7 +17,6 @@ More coming soon! We are working on tutorials to help you make your first contri - [**Documentation**](how_to/documentation/index.mdx): Help improve our docs, including this one! - [**Code**](how_to/code/index.mdx): Help us write code, fix bugs, or improve our infrastructure. - [**Integrations**](how_to/integrations/index.mdx): Help us integrate with your favorite vendors and tools. -- [**Standard Tests**](how_to/integrations/standard_tests): Ensure your integration passes an expected set of tests. ## Reference diff --git a/docs/vercel.json b/docs/vercel.json index 869e2e6b506ee..46d5aff7c8df7 100644 --- a/docs/vercel.json +++ b/docs/vercel.json @@ -113,6 +113,10 @@ { "source": "/docs/contributing/:path((?:faq|repo_structure|review_process)/?)", "destination": "/docs/contributing/reference/:path" + }, + { + "source": "/docs/contributing/how_to/integrations/standard_tests(/?)", + "destination": "/docs/contributing/how_to/integrations/#standard-tests" } ] } From ff214eb503902ed2928f73a13d666fdd3733ee38 Mon Sep 17 00:00:00 2001 From: ccurme Date: Tue, 3 Dec 2024 15:31:20 -0500 Subject: [PATCH 3/6] Update docs/docs/contributing/how_to/integrations/index.mdx Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com> --- docs/docs/contributing/how_to/integrations/index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs/contributing/how_to/integrations/index.mdx b/docs/docs/contributing/how_to/integrations/index.mdx index 913057a477528..5077904734020 100644 --- a/docs/docs/contributing/how_to/integrations/index.mdx +++ b/docs/docs/contributing/how_to/integrations/index.mdx @@ -99,7 +99,7 @@ Each type of integration has its own set of standard tests. Please see the relev With over 20 million monthly downloads, LangChain has a large audience of developers building LLM applications. Besides just adding integrations, we also like to show them examples of cool tools or APIs they can use. -While traditionally called "co-marketing", we like to thing of this more as "co-education". +While traditionally called "co-marketing", we like to think of this more as "co-education". For that reason, while we are happy to highlight your integration through our social media channels, we prefer to highlight examples that also serve some educational purpose. Our main social media channels are Twitter and LinkedIn. From 01045580f9d1a8c26fdf95163af489825714d282 Mon Sep 17 00:00:00 2001 From: ccurme Date: Tue, 3 Dec 2024 16:32:50 -0500 Subject: [PATCH 4/6] docs[patch]: add to chat model contributing guide (#28490) --- .../how_to/integrations/chat_models.md | 46 ++++++++++++++++--- .../how_to/integrations/index.mdx | 2 +- 2 files changed, 40 insertions(+), 8 deletions(-) diff --git a/docs/docs/contributing/how_to/integrations/chat_models.md b/docs/docs/contributing/how_to/integrations/chat_models.md index e5e9bcc866c57..3df0a478f7fb3 100644 --- a/docs/docs/contributing/how_to/integrations/chat_models.md +++ b/docs/docs/contributing/how_to/integrations/chat_models.md @@ -206,6 +206,17 @@ class ChatParrotLink(BaseChatModel): ``` +:::tip + +The model from the [Custom Chat Model Guide](/docs/how_to/custom_chat_model) is tested +against the standard unit and integration tests in the LangChain Github repository. +You can always use this as a starting point. + +- [Model implementation](https://github.com/langchain-ai/langchain/blob/master/libs/standard-tests/tests/unit_tests/custom_chat_model.py) +- [Tests](https://github.com/langchain-ai/langchain/blob/master/libs/standard-tests/tests/unit_tests/test_custom_chat_model.py) + +::: + ## Testing To implement our test files, we will subclass test classes from the `langchain_tests` package. These test classes contain the tests that will be run. We will just need to configure what model is tested, what parameters it is tested with, and specify any tests that should be skipped. @@ -225,8 +236,8 @@ If you followed the previous [bootstrapping guide](/docs/contributing/how_to/int ### Add and configure standard tests There are two namespaces in the langchain-tests package: -[unit tests](../../../concepts/testing.mdx#unit-tests) (`langchain_tests.unit_tests`): designed to be used to test the component in isolation and without access to external services -[integration tests](../../../concepts/testing.mdx#integration-tests) (`langchain_tests.integration_tests`): designed to be used to test the component with access to external services (in particular, the external service that the component is designed to interact with). +- [Unit tests](../../../concepts/testing.mdx#unit-tests) (`langchain_tests.unit_tests`): designed to be used to test the component in isolation and without access to external services +- [Integration tests](../../../concepts/testing.mdx#integration-tests) (`langchain_tests.integration_tests`): designed to be used to test the component with access to external services (in particular, the external service that the component is designed to interact with). Both types of tests are implemented as [pytest class-based test suites](https://docs.pytest.org/en/7.1.x/getting-started.html#group-multiple-tests-in-a-class). @@ -309,15 +320,36 @@ Our objective is for the pytest run to be successful. That is, ### Skipping tests -LangChain standard tests test a range of behaviors, from the most basic requirements (generating a response to a query) to optional capabilities like multi-modal support, tool-calling, or support for messages generated from other providers. Tests for "optional" capabilities are controlled via a [set of properties](https://python.langchain.com/api_reference/standard_tests/unit_tests/langchain_tests.unit_tests.chat_models.ChatModelTests.html) that can be overridden on the test model subclass. +LangChain standard tests test a range of behaviors, from the most basic requirements (generating a response to a query) to optional capabilities like multi-modal support and tool-calling. Tests for "optional" capabilities are controlled via a set of properties that can be overridden on the test model subclass. + +You can see the entire list of properties in the API reference [here](https://python.langchain.com/api_reference/standard_tests/unit_tests/langchain_tests.unit_tests.chat_models.ChatModelTests.html). These properties are shared by both unit and integration tests. + +For example, to enable integration tests for image inputs, we can implement + +```python +@property +def supports_image_inputs(self) -> bool: + return True +``` + +on the integration test class. + +The API references for individual test methods include instructions on whether and how +they can be skipped. See details: + +- [Unit tests API reference](https://python.langchain.com/api_reference/standard_tests/unit_tests/langchain_tests.unit_tests.chat_models.ChatModelUnitTests.html) +- [Integration tests API reference](https://python.langchain.com/api_reference/standard_tests/integration_tests/langchain_tests.integration_tests.chat_models.ChatModelIntegrationTests.html) ### Test suite information and troubleshooting -What tests are run to test this integration? +Each test method documents: -If a test fails, what does that mean? +1. Troubleshooting tips; +2. (If applicable) how test can be skipped. -You can find information on the tests run for this integration in the [Standard Tests API Reference](https://python.langchain.com/api_reference/standard_tests/index.html). +This information along with the full set of tests that run can be found in the API +reference. See details: -// TODO: link to exact page for this integration test suite information +- [Unit tests API reference](https://python.langchain.com/api_reference/standard_tests/unit_tests/langchain_tests.unit_tests.chat_models.ChatModelUnitTests.html) +- [Integration tests API reference](https://python.langchain.com/api_reference/standard_tests/integration_tests/langchain_tests.integration_tests.chat_models.ChatModelIntegrationTests.html) diff --git a/docs/docs/contributing/how_to/integrations/index.mdx b/docs/docs/contributing/how_to/integrations/index.mdx index 5077904734020..3f63247525889 100644 --- a/docs/docs/contributing/how_to/integrations/index.mdx +++ b/docs/docs/contributing/how_to/integrations/index.mdx @@ -64,7 +64,7 @@ While any component can be integrated into LangChain, there are specific types o In order to contribute an integration, you should follow these steps: 1. Confirm that your integration is in the [list of components](#components-to-integrate) we are currently encouraging. -2. [Bootstrap your integration](/docs/contributing/how_to/integrations/package/). +2. Create a package with the required dependencies (see example [here](/docs/contributing/how_to/integrations/package/)). 3. Implement and test your integration following the [component-specific guides](#component-specific-guides). 4. [Publish your integration](/docs/contributing/how_to/integrations/publish/) in a Python package to PyPi. 5. [Optional] Open and merge a PR to add documentation for your integration to the official LangChain docs. From 3fe135f24257934372047d01d9ca990aadfaa1a1 Mon Sep 17 00:00:00 2001 From: ccurme Date: Wed, 4 Dec 2024 16:26:23 -0500 Subject: [PATCH 5/6] docs[patch]: add vector store contribution guide (#28518) --- .../how_to/integrations/index.mdx | 2 +- .../how_to/integrations/vector_stores.mdx | 593 ++++++++++++++++++ 2 files changed, 594 insertions(+), 1 deletion(-) create mode 100644 docs/docs/contributing/how_to/integrations/vector_stores.mdx diff --git a/docs/docs/contributing/how_to/integrations/index.mdx b/docs/docs/contributing/how_to/integrations/index.mdx index 3f63247525889..e86f3ed69dbdc 100644 --- a/docs/docs/contributing/how_to/integrations/index.mdx +++ b/docs/docs/contributing/how_to/integrations/index.mdx @@ -79,7 +79,7 @@ The guides below walk you through both implementing and testing specific compone - [Toolkits] - [Retrievers] - [Document Loaders] -- [Vector Stores] +- [Vector Stores](/docs/contributing/how_to/integrations/vector_stores) - [Embedding Models] ## Standard Tests diff --git a/docs/docs/contributing/how_to/integrations/vector_stores.mdx b/docs/docs/contributing/how_to/integrations/vector_stores.mdx new file mode 100644 index 0000000000000..d298c6b6bd5e3 --- /dev/null +++ b/docs/docs/contributing/how_to/integrations/vector_stores.mdx @@ -0,0 +1,593 @@ +--- +pagination_next: contributing/how_to/integrations/publish +pagination_prev: contributing/how_to/integrations/index +--- +# How to implement and test a vector store integration + +This guide walks through how to implement and test a custom [vector store](/docs/concepts/vectorstores) that you have developed. + +For testing, we will rely on the `langchain-tests` dependency we added in the previous [bootstrapping guide](/docs/contributing/how_to/integrations/package). + +## Implementation + +Let's say you're building a simple integration package that provides a `ParrotVectorStore` +vector store integration for LangChain. Here's a simple example of what your project +structure might look like: + +```plaintext +langchain-parrot-link/ +├── langchain_parrot_link/ +│ ├── __init__.py +│ └── vectorstores.py +├── tests/ +│ ├── __init__.py +│ └── test_vectorstores.py +├── pyproject.toml +└── README.md +``` + +Following the [bootstrapping guide](/docs/contributing/how_to/integrations/package), +all of these files should already exist, except for +`vectorstores.py` and `test_vectorstores.py`. We will implement these files in this guide. + +First we need an implementation for our vector store. This implementation will depend +on your chosen database technology. `langchain-core` includes a minimal +[in-memory vector store](https://python.langchain.com/api_reference/core/vectorstores/langchain_core.vectorstores.in_memory.InMemoryVectorStore.html) +that we can use as a guide. You can access the code [here](https://github.com/langchain-ai/langchain/blob/master/libs/core/langchain_core/vectorstores/in_memory.py). +For convenience, we also provide it below. + +
+ vectorstores.py +```python title="langchain_parrot_link/vectorstores.py" +from __future__ import annotations + +import json +import uuid +from collections.abc import Iterator, Sequence +from pathlib import Path +from typing import Any, Callable, Optional + +from langchain_core.documents import Document +from langchain_core.embeddings import Embeddings +from langchain_core.load import dumpd, load +from langchain_core.vectorstores import VectorStore +from langchain_core.vectorstores.utils import _cosine_similarity as cosine_similarity +from langchain_core.vectorstores.utils import maximal_marginal_relevance + + +class InMemoryVectorStore(VectorStore): + """In-memory vector store implementation. + + Uses a dictionary, and computes cosine similarity for search using numpy. + """ + + def __init__(self, embedding: Embeddings) -> None: + """Initialize with the given embedding function. + + Args: + embedding: embedding function to use. + """ + self.store: dict[str, dict[str, Any]] = {} + self.embedding = embedding + + @property + def embeddings(self) -> Embeddings: + return self.embedding + + def delete(self, ids: Optional[Sequence[str]] = None, **kwargs: Any) -> None: + if ids: + for _id in ids: + self.store.pop(_id, None) + + async def adelete(self, ids: Optional[Sequence[str]] = None, **kwargs: Any) -> None: + self.delete(ids) + + def add_documents( + self, + documents: list[Document], + ids: Optional[list[str]] = None, + **kwargs: Any, + ) -> list[str]: + """Add documents to the store.""" + texts = [doc.page_content for doc in documents] + vectors = self.embedding.embed_documents(texts) + + if ids and len(ids) != len(texts): + msg = ( + f"ids must be the same length as texts. " + f"Got {len(ids)} ids and {len(texts)} texts." + ) + raise ValueError(msg) + + id_iterator: Iterator[Optional[str]] = ( + iter(ids) if ids else iter(doc.id for doc in documents) + ) + + ids_ = [] + + for doc, vector in zip(documents, vectors): + doc_id = next(id_iterator) + doc_id_ = doc_id if doc_id else str(uuid.uuid4()) + ids_.append(doc_id_) + self.store[doc_id_] = { + "id": doc_id_, + "vector": vector, + "text": doc.page_content, + "metadata": doc.metadata, + } + + return ids_ + + async def aadd_documents( + self, documents: list[Document], ids: Optional[list[str]] = None, **kwargs: Any + ) -> list[str]: + """Add documents to the store.""" + texts = [doc.page_content for doc in documents] + vectors = await self.embedding.aembed_documents(texts) + + if ids and len(ids) != len(texts): + msg = ( + f"ids must be the same length as texts. " + f"Got {len(ids)} ids and {len(texts)} texts." + ) + raise ValueError(msg) + + id_iterator: Iterator[Optional[str]] = ( + iter(ids) if ids else iter(doc.id for doc in documents) + ) + ids_: list[str] = [] + + for doc, vector in zip(documents, vectors): + doc_id = next(id_iterator) + doc_id_ = doc_id if doc_id else str(uuid.uuid4()) + ids_.append(doc_id_) + self.store[doc_id_] = { + "id": doc_id_, + "vector": vector, + "text": doc.page_content, + "metadata": doc.metadata, + } + + return ids_ + + def get_by_ids(self, ids: Sequence[str], /) -> list[Document]: + """Get documents by their ids. + + Args: + ids: The ids of the documents to get. + + Returns: + A list of Document objects. + """ + documents = [] + + for doc_id in ids: + doc = self.store.get(doc_id) + if doc: + documents.append( + Document( + id=doc["id"], + page_content=doc["text"], + metadata=doc["metadata"], + ) + ) + return documents + + async def aget_by_ids(self, ids: Sequence[str], /) -> list[Document]: + """Async get documents by their ids. + + Args: + ids: The ids of the documents to get. + + Returns: + A list of Document objects. + """ + return self.get_by_ids(ids) + + def _similarity_search_with_score_by_vector( + self, + embedding: list[float], + k: int = 4, + filter: Optional[Callable[[Document], bool]] = None, + **kwargs: Any, + ) -> list[tuple[Document, float, list[float]]]: + # get all docs with fixed order in list + docs = list(self.store.values()) + + if filter is not None: + docs = [ + doc + for doc in docs + if filter(Document(page_content=doc["text"], metadata=doc["metadata"])) + ] + + if not docs: + return [] + + similarity = cosine_similarity([embedding], [doc["vector"] for doc in docs])[0] + + # get the indices ordered by similarity score + top_k_idx = similarity.argsort()[::-1][:k] + + return [ + ( + Document( + id=doc_dict["id"], + page_content=doc_dict["text"], + metadata=doc_dict["metadata"], + ), + float(similarity[idx].item()), + doc_dict["vector"], + ) + for idx in top_k_idx + # Assign using walrus operator to avoid multiple lookups + if (doc_dict := docs[idx]) + ] + + def similarity_search_with_score_by_vector( + self, + embedding: list[float], + k: int = 4, + filter: Optional[Callable[[Document], bool]] = None, + **kwargs: Any, + ) -> list[tuple[Document, float]]: + return [ + (doc, similarity) + for doc, similarity, _ in self._similarity_search_with_score_by_vector( + embedding=embedding, k=k, filter=filter, **kwargs + ) + ] + + def similarity_search_with_score( + self, + query: str, + k: int = 4, + **kwargs: Any, + ) -> list[tuple[Document, float]]: + embedding = self.embedding.embed_query(query) + docs = self.similarity_search_with_score_by_vector( + embedding, + k, + **kwargs, + ) + return docs + + async def asimilarity_search_with_score( + self, query: str, k: int = 4, **kwargs: Any + ) -> list[tuple[Document, float]]: + embedding = await self.embedding.aembed_query(query) + docs = self.similarity_search_with_score_by_vector( + embedding, + k, + **kwargs, + ) + return docs + + def similarity_search_by_vector( + self, + embedding: list[float], + k: int = 4, + **kwargs: Any, + ) -> list[Document]: + docs_and_scores = self.similarity_search_with_score_by_vector( + embedding, + k, + **kwargs, + ) + return [doc for doc, _ in docs_and_scores] + + async def asimilarity_search_by_vector( + self, embedding: list[float], k: int = 4, **kwargs: Any + ) -> list[Document]: + return self.similarity_search_by_vector(embedding, k, **kwargs) + + def similarity_search( + self, query: str, k: int = 4, **kwargs: Any + ) -> list[Document]: + return [doc for doc, _ in self.similarity_search_with_score(query, k, **kwargs)] + + async def asimilarity_search( + self, query: str, k: int = 4, **kwargs: Any + ) -> list[Document]: + return [ + doc + for doc, _ in await self.asimilarity_search_with_score(query, k, **kwargs) + ] + + def max_marginal_relevance_search_by_vector( + self, + embedding: list[float], + k: int = 4, + fetch_k: int = 20, + lambda_mult: float = 0.5, + **kwargs: Any, + ) -> list[Document]: + prefetch_hits = self._similarity_search_with_score_by_vector( + embedding=embedding, + k=fetch_k, + **kwargs, + ) + + try: + import numpy as np + except ImportError as e: + msg = ( + "numpy must be installed to use max_marginal_relevance_search " + "pip install numpy" + ) + raise ImportError(msg) from e + + mmr_chosen_indices = maximal_marginal_relevance( + np.array(embedding, dtype=np.float32), + [vector for _, _, vector in prefetch_hits], + k=k, + lambda_mult=lambda_mult, + ) + return [prefetch_hits[idx][0] for idx in mmr_chosen_indices] + + def max_marginal_relevance_search( + self, + query: str, + k: int = 4, + fetch_k: int = 20, + lambda_mult: float = 0.5, + **kwargs: Any, + ) -> list[Document]: + embedding_vector = self.embedding.embed_query(query) + return self.max_marginal_relevance_search_by_vector( + embedding_vector, + k, + fetch_k, + lambda_mult=lambda_mult, + **kwargs, + ) + + async def amax_marginal_relevance_search( + self, + query: str, + k: int = 4, + fetch_k: int = 20, + lambda_mult: float = 0.5, + **kwargs: Any, + ) -> list[Document]: + embedding_vector = await self.embedding.aembed_query(query) + return self.max_marginal_relevance_search_by_vector( + embedding_vector, + k, + fetch_k, + lambda_mult=lambda_mult, + **kwargs, + ) + + @classmethod + def from_texts( + cls, + texts: list[str], + embedding: Embeddings, + metadatas: Optional[list[dict]] = None, + **kwargs: Any, + ) -> InMemoryVectorStore: + store = cls( + embedding=embedding, + ) + store.add_texts(texts=texts, metadatas=metadatas, **kwargs) + return store + + @classmethod + async def afrom_texts( + cls, + texts: list[str], + embedding: Embeddings, + metadatas: Optional[list[dict]] = None, + **kwargs: Any, + ) -> InMemoryVectorStore: + store = cls( + embedding=embedding, + ) + await store.aadd_texts(texts=texts, metadatas=metadatas, **kwargs) + return store + + @classmethod + def load( + cls, path: str, embedding: Embeddings, **kwargs: Any + ) -> InMemoryVectorStore: + """Load a vector store from a file. + + Args: + path: The path to load the vector store from. + embedding: The embedding to use. + kwargs: Additional arguments to pass to the constructor. + + Returns: + A VectorStore object. + """ + _path: Path = Path(path) + with _path.open("r") as f: + store = load(json.load(f)) + vectorstore = cls(embedding=embedding, **kwargs) + vectorstore.store = store + return vectorstore + + def dump(self, path: str) -> None: + """Dump the vector store to a file. + + Args: + path: The path to dump the vector store to. + """ + _path: Path = Path(path) + _path.parent.mkdir(exist_ok=True, parents=True) + with _path.open("w") as f: + json.dump(dumpd(self.store), f, indent=2) + +``` +
+ +All vector stores must inherit from the [VectorStore](https://python.langchain.com/api_reference/core/vectorstores/langchain_core.vectorstores.base.VectorStore.html) +base class. This interface consists of methods for writing, deleting and searching +for documents in the vector store. + +`VectorStore` supports a variety of synchronous and asynchronous search types (e.g., +nearest-neighbor or maximum marginal relevance), as well as interfaces for adding +documents to the store. See the [API Reference](https://python.langchain.com/api_reference/core/vectorstores/langchain_core.vectorstores.base.VectorStore.html) +for all supported methods. The required methods are tabulated below: + +| Method/Property | Description | +|------------------------ |------------------------------------------------------| +| `add_documents` | Add documents to the vector store. | +| `delete` | Delete selected documents from vector store (by IDs) | +| `get_by_ids` | Get selected documents from vector store (by IDs) | +| `similarity_search` | Get documents most similar to a query. | +| `embeddings` (property) | Embeddings object for vector store. | +| `from_texts` | Instantiate vector store via adding texts. | + +Note that `InMemoryVectorStore` implements some optional search types, as well as +convenience methods for loading and dumping the object to a file, but this is not +necessary for all implementations. + +:::tip + +The in-memory vector store is tested against the standard tests in the LangChain +Github repository. You can always use this as a starting point. + +- [Model implementation](https://github.com/langchain-ai/langchain/blob/master/libs/core/langchain_core/vectorstores/in_memory.py) +- [Tests](https://github.com/langchain-ai/langchain/blob/master/libs/standard-tests/tests/unit_tests/test_in_memory_vectorstore.py) + +::: + +## Testing + +To implement our test files, we will subclass test classes from the `langchain_tests` +package. These test classes contain the tests that will be run. We will just need to +configure what vector store implementation is tested. + +### Setup + +First we need to install certain dependencies. These include: + +- `pytest`: For running tests +- `pytest-asyncio`: For testing async functionality +- `langchain-tests`: For importing standard tests +- `langchain-core`: This should already be installed, but is needed to define our integration. + +If you followed the previous [bootstrapping guide](/docs/contributing/how_to/integrations/package/), +these should already be installed. + +### Add and configure standard tests + +The `langchain-test` package implements suites of tests for testing vector store +integrations. By subclassing the base classes for each standard test, you +get all of the standard tests for that type. + +:::note + +The full set of tests that run can be found in the API reference. See details: + +- [Sync tests API reference](https://python.langchain.com/api_reference/standard_tests/integration_tests/langchain_tests.integration_tests.vectorstores.ReadWriteTestSuite.html) +- [Async tests API reference](https://python.langchain.com/api_reference/standard_tests/integration_tests/langchain_tests.integration_tests.vectorstores.AsyncReadWriteTestSuite.html) + +::: + +Here's how you would configure the standard tests for a typical vector store (using +`ParrotVectorStore` as a placeholder): + +```python +# title="tests/integration_tests/test_vectorstores.py" + +from typing import AsyncGenerator, Generator + +import pytest +from langchain_core.vectorstores import VectorStore +from langchain_parrot_link.vectorstores import ParrotVectorStore +from langchain_tests.integration_tests.vectorstores import ( + AsyncReadWriteTestSuite, + ReadWriteTestSuite, +) + + +class TestSync(ReadWriteTestSuite): + @pytest.fixture() + def vectorstore(self) -> Generator[VectorStore, None, None]: # type: ignore + """Get an empty vectorstore.""" + store = ParrotVectorStore(self.get_embeddings()) + # note: store should be EMPTY at this point + # if you need to delete data, you may do so here + try: + yield store + finally: + # cleanup operations, or deleting data + pass + + +class TestAsync(AsyncReadWriteTestSuite): + @pytest.fixture() + async def vectorstore(self) -> AsyncGenerator[VectorStore, None]: # type: ignore + """Get an empty vectorstore.""" + store = ParrotVectorStore(self.get_embeddings()) + # note: store should be EMPTY at this point + # if you need to delete data, you may do so here + try: + yield store + finally: + # cleanup operations, or deleting data + pass +``` + +There are separate suites for testing synchronous and asynchronous methods. +Configuring the tests consists of implementing pytest fixtures for setting up an +empty vector store and tearing down the vector store after the test run ends. + +For example, below is the `ReadWriteTestSuite` for the [Chroma](https://python.langchain.com/docs/integrations/vectorstores/chroma/) +integration: + +```python +from typing import Generator + +import pytest +from langchain_core.vectorstores import VectorStore +from langchain_tests.integration_tests.vectorstores import ReadWriteTestSuite + +from langchain_chroma import Chroma + + +class TestSync(ReadWriteTestSuite): + @pytest.fixture() + def vectorstore(self) -> Generator[VectorStore, None, None]: # type: ignore + """Get an empty vectorstore.""" + store = Chroma(embedding_function=self.get_embeddings()) + try: + yield store + finally: + store.delete_collection() + pass +``` + +Note that before the initial `yield`, we instantiate the vector store with an +[embeddings](/docs/concepts/embedding_models/) object. This is a pre-defined +["fake" embeddings model](https://python.langchain.com/api_reference/standard_tests/integration_tests/langchain_tests.integration_tests.vectorstores.ReadWriteTestSuite.html#langchain_tests.integration_tests.vectorstores.ReadWriteTestSuite.get_embeddings) +that will generate short, arbitrary vectors for documents. You can use a different +embeddings object if desired. + +In the `finally` block, we call whatever integration-specific logic is needed to +bring the vector store to a clean state. This logic is executed when the test run +ends (e.g., even if tests fail). + +### Run standard tests + +After setting tests up, you would run them with the following command from your project root: + +```shell +pytest --asyncio-mode=auto tests/integration_tests +``` + +### Test suite information and troubleshooting + +Each test method documents: + +1. Troubleshooting tips; +2. (If applicable) how test can be skipped. + +This information along with the full set of tests that run can be found in the API +reference. See details: + +- [Sync tests API reference](https://python.langchain.com/api_reference/standard_tests/integration_tests/langchain_tests.integration_tests.vectorstores.ReadWriteTestSuite.html) +- [Async tests API reference](https://python.langchain.com/api_reference/standard_tests/integration_tests/langchain_tests.integration_tests.vectorstores.AsyncReadWriteTestSuite.html) From 0ab8e5cfe05ea7817ffaf27c45f61ed3d2654e38 Mon Sep 17 00:00:00 2001 From: Chester Curme Date: Wed, 4 Dec 2024 16:56:12 -0500 Subject: [PATCH 6/6] nit --- docs/docs/contributing/how_to/integrations/vector_stores.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/docs/contributing/how_to/integrations/vector_stores.mdx b/docs/docs/contributing/how_to/integrations/vector_stores.mdx index d298c6b6bd5e3..a4bf74dc58229 100644 --- a/docs/docs/contributing/how_to/integrations/vector_stores.mdx +++ b/docs/docs/contributing/how_to/integrations/vector_stores.mdx @@ -568,8 +568,8 @@ that will generate short, arbitrary vectors for documents. You can use a differe embeddings object if desired. In the `finally` block, we call whatever integration-specific logic is needed to -bring the vector store to a clean state. This logic is executed when the test run -ends (e.g., even if tests fail). +bring the vector store to a clean state. This logic is executed in between each test +(e.g., even if tests fail). ### Run standard tests