Release/v0.28.0

.github/pull_request_template.md

@@ -0,0 +1,5 @@
+- [ ] I have read and agree to the contributing guidelines for [submitting new pull requests](https://github.com/griptape-ai/griptape?tab=readme-ov-file#submitting-pull-requests).
+
+## Describe your changes
+
+## Issue ticket number and link

.github/workflows/code-checks.yml

@@ -70,3 +70,16 @@ jobs:
             verbose: true
           env:
             CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
+  spell-check:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: [ "3.12" ]
+    steps:
+      - name: Checkout actions
+        uses: actions/checkout@v3
+      - name: Init environment
+        uses: ./.github/actions/init-environment
+      - name: Run linter
+        run: make check/spell

.github/workflows/docs-integration-tests.yml

@@ -119,6 +119,9 @@ jobs:
       PUSHER_KEY: ${{ secrets.INTEG_PUSHER_KEY }}
       PUSHER_SECRET: ${{ secrets.INTEG_PUSHER_SECRET }}
       PUSHER_CLUSTER: ${{ secrets.INTEG_PUSHER_CLUSTER }}
+      ZENROWS_API_KEY: ${{ secrets.INTEG_ZENROWS_API_KEY }}
+      QDRANT_CLUSTER_ENDPOINT: ${{ secrets.INTEG_QDRANT_CLUSTER_ENDPOINT }}
+      QDRANT_CLUSTER_API_KEY: ${{ secrets.INTEG_QDRANT_CLUSTER_API_KEY }}
     services:
       postgres:
         image: ankane/pgvector:v0.5.0

.pre-commit-config.yaml

@@ -25,3 +25,10 @@ repos:
     entry: make check/types
     language: system
     types: [python]
+- repo: local
+  hooks:
+  - id: typos
+    name: Typos
+    entry: make check/spell
+    language: system
+    types: [python]

CHANGELOG.md

@@ -6,6 +6,85 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## Unreleased
 
+## [0.28.0] - 2024-07-09
+### Added
+- `RagEngine` is an abstraction for implementing modular RAG pipelines.
+  - `RagContext` is a container object for passing around RAG context. 
+  - RAG stages:
+    - `QueryRagStage` for parsing and expanding queries.
+    - `RetrievalRagStage` for retrieving content.
+    - `ResponseRagStage` for augmenting and generating outputs.
+  - RAG modules:
+    - Retrieval:
+      - `VectorStoreRetrievalRagModule` for retrieving text chunks from vector stores.
+      - `TextLoaderRetrievalRagModule` for retrieving data with text loaders in real time.
+      - `TextChunksRerankRagModule` for re-ranking retrieved results.
+    - Response:
+      - `MetadataBeforeResponseRagModule` for appending metadata.
+      - `RulesetsBeforeResponseRagModule` for appending rulesets.
+      - `PromptResponseRagModule` for generating responses based on retrieved text chunks.
+      - `TextChunksResponseRagModule` for responding with retrieved text chunks.
+      - `FootnotePromptResponseRagModule` for responding with automatic footnotes from text chunk references.
+- `RagClient` tool for exposing `RagEngines` to LLM agents.
+- `RagTask` task for including `RagEngines` in any structure.
+- Rerank drivers:
+  - `CohereRerankDriver` for using the Cohere rerank API.
+- `utils.execute_futures_list()` for executing a list of futures.
+- `LocalVectorStoreDriver.persist_file` for persisting vectors and chunks in a text file.
+- `Entry.to_artifact()` for easy vector store entry conversions into Griptape artifacts.
+- `BaseVectorStoreDriver.does_entry_exist()` to check if an entry exists in the vector store.
+- `GoogleWebSearchDriver` to web search with the Google Customsearch API.
+- `DuckDuckGoWebSearchDriver` to web search with the DuckDuckGo search SDK.
+- `ProxyWebScraperDriver` to web scrape using proxies.
+- Parameter `session` on `AmazonBedrockStructureConfig`.
+- Parameter `meta` on `TextArtifact`.
+- `VectorStoreClient` improvements:
+  - `VectorStoreClient.query_params` dict for custom query params.
+  - `VectorStoreClient.process_query_output_fn` for custom query output processing logic.
+- Parameter `fail_fast` to `Structure`.
+- `BooleanArtifact` for handling boolean values.
+- `typos` to dev dependencies to catch typos in code/docs.
+- `Message` for storing messages in a `PromptStack`. Messages consist of a role, content, and usage.
+- `DeltaMessage` for storing partial messages in a `PromptStack`. Multiple `DeltaMessage` can be combined to form a `Message`.
+- `TextMessageContent` for storing textual content in a `Message`.
+- `ImageMessageContent` for storing image content in a `Message`.
+- Support for adding `TextArtifact`s, `ImageArtifact`s, and `ListArtifact`s to `PromptStack`.
+- Support for image inputs to `OpenAiChatPromptDriver`, `AzureOpenAiChatPromptDriver`, `AmazonBedrockPromptDriver`, `AnthropicPromptDriver`, and `GooglePromptDriver`.
+- Input/output token usage metrics to all Prompt Drivers. 
+- `FinishPromptEvent.input_token_count` and `FinishPromptEvent.output_token_count`.
+- Support for storing Artifacts as inputs/outputs in Conversation Memory Runs.
+- `Agent.input` for passing Artifacts as input.
+- Support for `PromptTask`s to take `TextArtifact`s, `ImageArtifact`s, and `ListArtifact`s as input.
+- Parameters `sort_key` and `sort_key_value` on `AmazonDynamoDbConversationMemoryDriver` for tables with sort keys.
+- `Reference` for supporting artifact citations in loaders and RAG engine modules.
+
+### Changed
+- **BREAKING**: Moved/renamed `griptape.utils.PromptStack` to `griptape.common.PromptStack`.
+- **BREAKING**: Renamed `PromptStack.inputs` to `PromptStack.messages`.
+- **BREAKING**: Moved `PromptStack.USER_ROLE`, `PromptStack.ASSISTANT_ROLE`, and `PromptStack.SYSTEM_ROLE` to `Message`.
+- **BREAKING**: Updated return type of `PromptDriver.try_run` from `TextArtifact` to `Message`.
+- **BREAKING**: Updated return type of `PromptDriver.try_stream` from `Iterator[TextArtifact]` to `Iterator[DeltaMessage]`.
+- **BREAKING**: Removed `BasePromptEvent.token_count` in favor of `FinishPromptEvent.input_token_count` and `FinishPromptEvent.output_token_count`.
+- **BREAKING**: Removed `StartPromptEvent.prompt`. Use `StartPromptEvent.prompt_stack` instead.
+- **BREAKING**: Removed `Agent.input_template` in favor of `Agent.input`.
+- **BREAKING**: `BasePromptDriver.run` now returns a `Message` instead of a `TextArtifact`. For compatibility, `Message.value` contains the Message's Artifact value
+- **BREAKING**: `BaseVectorStoreDriver.upsert_text_artifact()` and `BaseVectorStoreDriver.upsert_text()` use artifact/string values to generate `vector_id` if it wasn't implicitly passed. This change ensures that we don't generate embeddings for the same content every time.
+- **BREAKING**: Removed `VectorQueryEngine` in favor of `RagEngine`.
+- **BREAKING**: Removed `TextQueryTask` in favor of `RagTask`.
+- **BREAKING**: `TextArtifactStorage` now requires `vector_store_driver` and `rag_engine` in place of `vector_query_engine`.
+- **BREAKING**: Moved `load_artifacts()` from `BaseQueryEngine` to `BaseVectorStoreDriver`.
+- **BREAKING**: Merged `BaseVectorStoreDriver.QueryResult` into `BaseVectorStoreDriver.Entry`.
+- **BREAKING**: Replaced `query_engine` with `vector_store_driver` in `VectorStoreClient`.
+- **BREAKING**: removed parameters `google_api_lang`, `google_api_key`, `google_api_search_id`, `google_api_country` on `WebSearch` in favor of `web_search_driver`.
+- **BREAKING**: removed `VectorStoreClient.top_n` and `VectorStoreClient.namespace` in favor of `VectorStoreClient.query_params`.
+- **BREAKING**: All `futures_executor` fields renamed to `futures_executor_fn` and now accept callables instead of futures; wrapped all future `submit` calls with the `with` block to address future executor shutdown issues.
+- `GriptapeCloudKnowledgeBaseClient` migrated to `/search` api.
+- Default Prompt Driver model in `GoogleStructureConfig` to `gemini-1.5-pro`.
+
+### Fixed
+- `CoherePromptDriver` to properly handle empty history.
+- `StructureVisualizer.to_url()` by wrapping task IDs in single quotes. 
+
 ## [0.27.2] - 2024-06-27
 
 ### Fixed
@@ -86,6 +165,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Streaming not working when using deprecated `Structure.stream` field.
 - Raw Tool output being lost when being executed by ActionsSubtask.
 - Re-order Workflow tasks on every task execution wave.
+- `Workflow.insert_task()` enumerates by parent id equality, opposed to object equality.
 - Web Loader to catch Exceptions and properly return an ErrorArtifact.
 - Conversation Memory entry only added if `output_task.output` is not `None` on all `Structures`
 - `TextArtifacts` contained in `ListArtifact` returned by `WebSearch.search` to properly formatted stringified JSON.
@@ -105,6 +185,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `PusherEventListenerDriver` to enable sending of framework events over a Pusher WebSocket.
 
 ### Changed
+- **BREAKING**: Updated OpenAI-based image query drivers to remove Vision from the name.
+- **BREAKING**: `off_prompt` now defaults to `False` on all Tools, making Task Memory something that must be explicitly opted into.
 - **BREAKING**: Removed `StructureConfig.global_drivers`. Pass Drivers directly to the Structure Config instead. 
 - **BREAKING**: Removed `StructureConfig.task_memory` in favor of configuring directly on the Structure.  
 - **BREAKING**: Updated OpenAI-based image query drivers to remove Vision from the name.
@@ -120,6 +202,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 - Extra fields being excluded when using `SerializableMixin.from_dict`.
+- Validation of `max_tokens` < 0 on `BaseChunker`
 
 ## [0.25.1] - 2024-05-15
 
@@ -211,6 +294,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `MarkdownifyWebScraperDriver` for scraping text from web pages using playwright and converting to markdown using markdownify.
 - `VoyageAiEmbeddingDriver` for use with VoyageAi's embedding models. 
 - `AnthropicStructureConfig` for providing Structures with Anthropic Prompt and VoyageAi Embedding Driver configuration.
+- `QdrantVectorStoreDriver` to integrate with Qdrant vector databases.
 
 ### Fixed
 - Improved system prompt in `ToolTask` to support more use cases.
@@ -316,7 +400,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `ImageLoader` for loading images files into `ImageArtifact`s.
 
 - Support for all Tokenizers in `OpenAiChatPromptDriver`, enabling OpenAI drop-in clients such as Together AI.
-- `AmazonSageMakerEmbeddingDriver` for using Amazon SageMaker to generate embeddings. Thanks @KaushikIyer16!
+- `AmazonSageMakerJumpstartEmbeddingDriver` for using Amazon SageMaker to generate embeddings. Thanks @KaushikIyer16!
 - Claude 2.1 support in `AnthropicPromptDriver` and `AmazonBedrockPromptDriver` via `BedrockClaudePromptModelDriver`.
 - `CodeExecutionTask` for executing code as a Task without the need for an LLM.
 - `BedrockLlamaPromptModelDriver` for using Llama models on Amazon Bedrock.

Makefile

@@ -16,8 +16,8 @@ publish: ## Push git tag and publish version to PyPI.
 install: ## Install all dependencies.
 	@poetry install --with dev --with test --with docs --all-extras
 
-.PHONY: test
-test: test/unit test/integration ## Run all tests.
+.PHONY: test  ## Run all tests.
+test: test/unit test/integration
 
 .PHONY: test/unit
 test/unit: ## Run unit tests.
@@ -40,7 +40,7 @@ format: ## Format project.
 	@poetry run ruff format .
 
 .PHONY: check
-check: check/format check/lint check/types ## Run all checks.
+check: check/format check/lint check/types check/spell ## Run all checks.
 
 .PHONY: check/format
 check/format:
@@ -53,6 +53,14 @@ check/lint:
 .PHONY: check/types
 check/types:
 	@poetry run pyright griptape/
+
+.PHONY: check/spell
+check/spell:
+	@poetry run typos 
+
+.PHONY: docs
+docs: ## Build documentation.
+	@poetry run mkdocs build
 
 .DEFAULT_GOAL := help
 .PHONY: help

README.md

@@ -6,7 +6,7 @@
 [![Checked with pyright](https://microsoft.github.io/pyright/img/pyright_badge.svg)](https://microsoft.github.io/pyright/)
 [![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
 [![codecov](https://codecov.io/github/griptape-ai/griptape/graph/badge.svg?token=HUBqUpl3NB)](https://codecov.io/github/griptape-ai/griptape)
-[![Griptape Discord](https://dcbadge.vercel.app/api/server/gnWRz88eym?compact=true&style=flat)](https://discord.gg/gnWRz88eym)
+[![Griptape Discord](https://dcbadge.vercel.app/api/server/gnWRz88eym?compact=true&style=flat)](https://discord.gg/griptape)
 
 Griptape is a modular Python framework for building AI-powered applications that securely connect to your enterprise data and APIs. It offers developers the ability to maintain control and flexibility at every step.
 
@@ -50,7 +50,7 @@ Drivers facilitate interactions with external resources and services:
 
 Engines wrap Drivers and provide use-case-specific functionality:
 
-- 📊 **Query Engines** execute Retrieval Augmented Generation (RAG) queries.
+- 📊 **RAG Engine** is an abstraction for implementing modular Retrieval Augmented Generation (RAG) pipelines.
 - 🛠️ **Extraction Engines** extract JSON or CSV data from unstructured text.
 - 📝 **Summary Engines** generate summaries from textual content.
 - 🖼️ **Image Generation Engines** generate images from textual descriptions.
@@ -92,7 +92,7 @@ from griptape.structures import Agent
 from griptape.tools import WebScraper, FileManager, TaskMemoryClient
 
 agent = Agent(
-    input_template="Load {{ args[0] }}, summarize it, and store it in a file called {{ args[1] }}.",
+    input="Load {{ args[0] }}, summarize it, and store it in a file called {{ args[1] }}.",
     tools=[
         WebScraper(off_prompt=True),
         TaskMemoryClient(off_prompt=True),
@@ -162,12 +162,6 @@ Thank you for considering contributing to Griptape! Before you start, please rea
 
 If you have identified a bug, want to propose a new feature, or have a question, please submit an issue through our public [issue tracker](https://github.com/griptape-ai/griptape/issues). Before submitting a new issue, please check the existing issues to ensure it hasn't been reported or discussed before.
 
-### New Griptape Tools
-
-Griptape's extensibility allows anyone to develop and distribute tools independently. With rare exceptions for Tools providing broadly applicable functionality, new Griptape Tools should be managed as their own projects and not submitted to the core framework. Pull requests for new tools (unless addressing an [existing issue](https://github.com/griptape-ai/griptape/issues)) will be closed.
-
-The [Griptape Tool Template](https://github.com/griptape-ai/tool-template) provides the recommended structure, step-by-step instructions, basic automation, and usage examples for new Tools. In the Template, select **Use this template** then **Create a new repository** to begin a new Tool project.
-
 ### Submitting Pull Requests
 
 We welcome and encourage pull requests. To streamline the process, please follow these guidelines:
@@ -178,13 +172,27 @@ We welcome and encourage pull requests. To streamline the process, please follow
 
 3. **Unit Tests:** Ensure that your pull request passes all existing unit tests. Additionally, if you are introducing new code, please include new unit tests to validate its functionality.
 
+Run `make test/unit` to execute the test suite locally.
+
 4. **Documentation:** Every pull request must include updates to documentation or explicitly explain why a documentation update is not required. Documentation is crucial for maintaining a comprehensive and user-friendly project.
 
-5. **Code Style:** Griptape uses [Ruff](https://github.com/astral-sh/ruff) to enforce style guidelines. You can ensure that your code is formatted accordingly and will pass formatting checks using `pre-commit`. See [Tools](#tools) for information on how to configure this and other dev tools.
+Run `make docs` to build the documentation locally.
+
+5. **Code Checks:** Griptape a variety of tools to enforce code quality and style. Your code must pass all checks before it can be merged.
+
+Run `make check` to run all code checks locally.
+
+6. **Changelog:** If your pull request introduces a notable change, please update the [changelog](https://github.com/griptape-ai/griptape/blob/dev/CHANGELOG.md).
+
+### New Griptape Tools
+
+Griptape's extensibility allows anyone to develop and distribute tools independently. With rare exceptions for Tools providing broadly applicable functionality, new Griptape Tools should be managed as their own projects and not submitted to the core framework. Pull requests for new tools (unless addressing an [existing issue](https://github.com/griptape-ai/griptape/issues)) will be closed.
+
+The [Griptape Tool Template](https://github.com/griptape-ai/tool-template) provides the recommended structure, step-by-step instructions, basic automation, and usage examples for new Tools. In the Template, select **Use this template** then **Create a new repository** to begin a new Tool project.
 
-### Tools
+### Dev and Test Dependencies
 
-Install dev dependencies via Make:
+Install all dependencies via Make:
 
 ```shell
 make install

_typos.toml

@@ -0,0 +1,8 @@
+[default.extend-words]
+# Don't correct the state ND
+ND = "ND"
+# Don't correct pirates using acronyms
+mornin = "mornin"
+
+[files]
+extend-exclude = ["docs/assets", "tests/resources/"]

docs/examples/load-query-and-chat-marqo.md

@@ -2,11 +2,10 @@
 import os
 from griptape import utils
 from griptape.drivers import MarqoVectorStoreDriver
-from griptape.engines import VectorQueryEngine
 from griptape.loaders import WebLoader
 from griptape.structures import Agent
 from griptape.tools import VectorStoreClient
-from griptape.drivers import LocalVectorStoreDriver, OpenAiEmbeddingDriver, OpenAiChatPromptDriver
+from griptape.drivers import OpenAiEmbeddingDriver
 
 # Define the namespace
 namespace = "griptape-ai"
@@ -18,14 +17,12 @@ vector_store = MarqoVectorStoreDriver(
     index=os.environ["MARQO_INDEX_NAME"],
     embedding_driver=OpenAiEmbeddingDriver()
 )
-# Initialize the query engine
-query_engine = VectorQueryEngine(vector_store_driver=vector_store, prompt_driver=OpenAiChatPromptDriver(model="gpt-3.5-turbo"))
 
 # Initialize the knowledge base tool
 vector_store_tool = VectorStoreClient(
     description="Contains information about the Griptape Framework from www.griptape.ai",
-    query_engine=query_engine,
     namespace=namespace,
+    vector_store_driver=vector_store
 )
 
 # Load artifacts from the web

docs/examples/multi-agent-workflow.md

@@ -7,7 +7,7 @@ Additionally, this architecture opens us up to using services such as [Griptape
 ```python
 import os
 
-from griptape.drivers import WebhookEventListenerDriver, LocalStructureRunDriver
+from griptape.drivers import WebhookEventListenerDriver, LocalStructureRunDriver, GoogleWebSearchDriver
 from griptape.events import EventListener, FinishStructureRunEvent
 from griptape.rules import Rule, Ruleset
 from griptape.structures import Agent, Workflow
@@ -38,8 +38,10 @@ def build_researcher():
         id="researcher",
         tools=[
             WebSearch(
-                google_api_key=os.environ["GOOGLE_API_KEY"],
-                google_api_search_id=os.environ["GOOGLE_API_SEARCH_ID"],
+                web_search_driver=GoogleWebSearchDriver(
+                    api_key=os.environ["GOOGLE_API_KEY"],
+                    search_id=os.environ["GOOGLE_API_SEARCH_ID"],
+                ),
             ),
             WebScraper(
                 off_prompt=True,

docs/examples/multiple-agent-shared-memory.md

@@ -8,13 +8,9 @@ The `MongoDbAtlasVectorStoreDriver` assumes that you have a vector index configu
 
 ```python
 import os
-from griptape.tools import WebScraper, VectorStoreClient, TaskMemoryClient
+from griptape.tools import WebScraper, TaskMemoryClient
 from griptape.structures import Agent
 from griptape.drivers import AzureOpenAiEmbeddingDriver, AzureMongoDbVectorStoreDriver
-from griptape.engines import VectorQueryEngine, PromptSummaryEngine, CsvExtractionEngine, JsonExtractionEngine
-from griptape.memory import TaskMemory 
-from griptape.artifacts import TextArtifact
-from griptape.memory.task.storage import TextArtifactStorage
 from griptape.config import AzureOpenAiStructureConfig
 
 

docs/examples/query-webpage.md

@@ -22,7 +22,7 @@ results = vector_store.query(
     namespace="griptape"
 )
 
-values = [BaseArtifact.from_json(r.meta["artifact"]).value for r in results]
+values = [r.to_artifact().value for r in results]
 
 print("\n\n".join(values))
 ```