Release/v0.30.0 (#1088)

Co-authored-by: Andrew French <[email protected]> Co-authored-by: Vasily Vasinov <[email protected]> Co-authored-by: Matt Vallillo <[email protected]> Co-authored-by: dylanholmes <[email protected]> Co-authored-by: Michal <[email protected]> Co-authored-by: Zach Giordano <[email protected]> Co-authored-by: Ikko Eltociear Ashimine <[email protected]> Co-authored-by: torabshaikh <[email protected]> Co-authored-by: Aodhan Roche <[email protected]> Co-authored-by: Kyle Roche <[email protected]> Co-authored-by: Emily Danielson <[email protected]> Co-authored-by: CJ Kindel <[email protected]> Co-authored-by: hkhajgiwale <[email protected]> Co-authored-by: Harsh Khajgiwale <[email protected]> Co-authored-by: Anush <[email protected]> Co-authored-by: datashaman <[email protected]> Co-authored-by: Stefano Lottini <[email protected]> Co-authored-by: James Clarendon <[email protected]>
griptape-ai · Aug 20, 2024 · a2520d3 · a2520d3
1 parent 694232c
commit a2520d3
Show file tree

Hide file tree

Showing 734 changed files with 11,453 additions and 9,647 deletions.
diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md
@@ -13,11 +13,7 @@ assignees: ''
 A clear and concise description of what the bug is.
 
 **To Reproduce**
-Steps to reproduce the behavior:
-1. Go to '...'
-2. Click on '....'
-3. Scroll down to '....'
-4. See error
+A minimal, reproducible code example.
 
 **Expected behavior**
 A clear and concise description of what you expected to happen.

diff --git a/.github/actions/init-bare-environment/action.yml b/.github/actions/init-bare-environment/action.yml
@@ -27,7 +27,7 @@ runs:
 
       - name: Install dependencies
         if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
-        run: poetry install --no-interaction --no-root --with test --with dev
+        run: poetry install --no-interaction --with test --with dev
         shell: bash
 
       - name: Activate venv 

diff --git a/.github/actions/init-environment/action.yml b/.github/actions/init-environment/action.yml
@@ -27,7 +27,7 @@ runs:
 
       - name: Install dependencies
         if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
-        run: poetry install --no-interaction --no-root --with test --with dev --all-extras
+        run: poetry install --no-interaction --with test --with dev --with docs --all-extras
         shell: bash
 
       - name: Activate venv 

diff --git a/.github/workflows/docs-integration-tests.yml b/.github/workflows/docs-integration-tests.yml
@@ -1,20 +1,15 @@
 name: Docs Integration Tests
 
 on:
-  pull_request_review:
-    types: [submitted]
   push:
-    branches:
-      - main
-      - dev
+    branches: [ "main", "dev" ]
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
   cancel-in-progress: true
 
 jobs:
   test:
-    if: github.event.review.state == 'APPROVED' || github.event_name == 'push'
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
@@ -124,6 +119,8 @@ jobs:
       ZENROWS_API_KEY: ${{ secrets.INTEG_ZENROWS_API_KEY }}
       QDRANT_CLUSTER_ENDPOINT: ${{ secrets.INTEG_QDRANT_CLUSTER_ENDPOINT }}
       QDRANT_CLUSTER_API_KEY: ${{ secrets.INTEG_QDRANT_CLUSTER_API_KEY }}
+      ASTRA_DB_API_ENDPOINT: ${{ secrets.INTEG_ASTRA_DB_API_ENDPOINT }}
+      ASTRA_DB_APPLICATION_TOKEN: ${{ secrets.INTEG_ASTRA_DB_APPLICATION_TOKEN }}
     services:
       postgres:
         image: ankane/pgvector:v0.5.0
@@ -143,22 +140,7 @@ jobs:
         uses: actions/checkout@v3
       - name: Init environment
         uses: ./.github/actions/init-environment
-      - name: Get changed files
-        id: changed-files
-        uses: tj-actions/changed-files@v44
-        if: github.event_name == 'pull_request_review'
-        with:
-          files: |
-             **.md
-      - name: List all changed files
-        env:
-          ALL_CHANGED_FILES: ${{ steps.changed-files.outputs.all_changed_files }}
-        run: |
-          for file in ${ALL_CHANGED_FILES}; do
-            echo "$file was changed"
-          done
       - name: Run integration tests
-        if: steps.changed-files.outputs.any_changed == 'true' || github.event_name == 'push'
         run: make test/integration
         env:
           DOCS_ALL_CHANGED_FILES: ${{ steps.changed-files.outputs.all_changed_files || '' }}
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -6,6 +6,66 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## Unreleased
 
+### Added
+- `AstraDbVectorStoreDriver` to support DataStax Astra DB as a vector store.
+- Ability to set custom schema properties on Tool Activities via `extra_schema_properties`.
+- Parameter `structure` to `BaseTask`.
+- Method `try_find_task` to `Structure`.
+- `TranslateQueryRagModule` `RagEngine` module for translating input queries.
+- Global event bus, `griptape.events.EventBus`, for publishing and subscribing to events.
+- Global object, `griptape.configs.Defaults`, for setting default values throughout the framework.
+- Unique name generation for all `RagEngine` modules.
+- `ExtractionTool` for having the LLM extract structured data from text.
+- `PromptSummaryTool` for having the LLM summarize text.
+- `QueryTool` for having the LLM query text.
+- Support for bitshift composition in `BaseTask` for adding parent/child tasks.
+- `JsonArtifact` for handling de/seralization of values.
+- `Chat.logger_level` for setting what the `Chat` utility sets the logger level to. 
+- `FuturesExecutorMixin` to DRY up and optimize concurrent code across multiple classes.
+- `utils.execute_futures_list_dict` for executing a dict of lists of futures.
+- `GriptapeCloudConversationMemoryDriver` to store conversation history in Griptape Cloud.
+- `griptape.utils.decorators.lazy_property` for creating lazy properties.
+
+### Changed
+- **BREAKING**: Removed all uses of `EventPublisherMixin` in favor of `EventBus`.
+- **BREAKING**: Removed `EventPublisherMixin`.
+- **BREAKING**: Removed `Pipeline.prompt_driver` and `Workflow.prompt_driver`. Set this via `griptape.configs.Defaults.drivers.prompt_driver` instead. `Agent.prompt_driver` has not been removed.
+- **BREAKING**: Removed `Pipeline.stream` and `Workflow.stream`. Set this via `griptape.configs.Defaults.drivers.prompt_driver.stream` instead. `Agent.stream` has not been removed.
+- **BREAKING**: Removed `Structure.embedding_driver`, set this via `griptape.configs.Defaults.drivers.embedding_driver` instead.
+- **BREAKING**: Removed `Structure.custom_logger` and `Structure.logger_level`, set these via `logging.getLogger(griptape.configs.Defaults.logger_name)` instead.
+- **BREAKING**: Removed `BaseStructureConfig.merge_config`.
+- **BREAKING**: Renamed `StructureConfig` to `DriversConfig`, moved to `griptape.configs.drivers` and renamed fields accordingly.
+- **BREAKING**: `RagContext.output` was changed to `RagContext.outputs` to support multiple outputs. All relevant RAG modules were adjusted accordingly.
+- **BREAKING**: Removed before and after response modules from `ResponseRagStage`.
+- **BREAKING**: Moved ruleset and metadata ingestion from standalone modules to `PromptResponseRagModule`.
+- **BREAKING**: Dropped `Client` from all Tool names for better naming consistency. 
+- **BREAKING**: Dropped `_client` suffix from all Tool packages. 
+- **BREAKING**: Added `Tool` suffix to all Tool names for better naming consistency. 
+- **BREAKING**: Removed `TextArtifactStorage.query` and `TextArtifactStorage.summarize`. 
+- **BREAKING**: Removed `TextArtifactStorage.rag_engine`, and `TextArtifactStorage.retrieval_rag_module_name`.
+- **BREAKING**: Removed `TextArtifactStorage.summary_engine`, `TextArtifactStorage.csv_extraction_engine`, and `TextArtifactStorage.json_extraction_engine`.
+- **BREAKING**: Removed `TaskMemory.summarize_namespace` and `TaskMemory.query_namespace`.
+- **BREAKING**: Removed `Structure.rag_engine`.
+- **BREAKING**: Split `JsonExtractionEngine.template_generator` into `JsonExtractionEngine.system_template_generator` and `JsonExtractionEngine.user_template_generator`.
+- **BREAKING**: Split `CsvExtractionEngine.template_generator` into `CsvExtractionEngine.system_template_generator` and `CsvExtractionEngine.user_template_generator`.
+- **BREAKING**: Changed `JsonExtractionEngine.template_schema` from a `run` argument to a class attribute. 
+- **BREAKING**: Changed `CsvExtractionEngine.column_names` from a `run` argument to a class attribute. 
+- **BREAKING**: Removed `JsonExtractionTask`, and `CsvExtractionTask` use `ExtractionTask` instead.
+- **BREAKING**: Removed `TaskMemoryClient`, use `QueryClient`, `ExtractionTool`, or `PromptSummaryTool` instead.
+- **BREAKING**: `BaseTask.add_parent/child` now take a `BaseTask` instead of `str | BaseTask`.
+- Engines that previously required Drivers now pull from `griptape.configs.Defaults.drivers_config` by default.
+- `BaseTask.add_parent/child` will now call `self.structure.add_task` if possible.
+- `BaseTask.add_parent/child` now returns `self`, allowing for chaining.
+- `Chat` now sets the `griptape` logger level to `logging.ERROR`, suppressing all logs except for errors.
+
+### Fixed
+- `JsonExtractionEngine` failing to parse json when the LLM outputs more than just the json.
+- Exception when adding `ErrorArtifact`'s to the Prompt Stack.
+- Concurrency bug in `BaseVectorStoreDriver.upsert_text_artifacts`.
+- Schema issues with Tools that use lists.
+- Issue with native Tool calling and streaming with `GooglePromptDriver`.
+- Description not being used properly in `StructureRunTool`.
+
 ## [0.29.2] - 2024-08-16
 
 ### Fixed
@@ -36,6 +96,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `@observable` decorator for selecting which functions/methods to provide observability for.
 - `GenericArtifact` for storing any data.
 - `BaseTextArtifact` for text-based Artifacts to subclass.
+- `HuggingFacePipelineImageGenerationDriver` for generating images locally with HuggingFace pipelines.
+- `BaseImageGenerationPipelineDriver` as the base class for drivers interfacing with HuggingFace image generation pipelines.
+- `StableDiffusion3ImageGenerationPipelineDriver` for local text-to-image generation using a Stable Diffusion 3 pipeline.
+- `StableDiffusion3Img2ImgImageGenerationPipelineDriver` for local image-to-image generation using a Stable Diffusion 3 pipeline.
+- `StableDiffusion3ControlNetImageGenerationPipelineDriver` for local ControlNet image generation using a Stable Diffusion 3 pipeline.
+- Optional `params` field to `WebSearch`'s `search` schema that the LLM can be steered into using.
 
 ### Changed
 - **BREAKING**: `BaseVectorStoreDriver.upsert_text_artifacts` optional arguments are now keyword-only arguments.
@@ -55,6 +121,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **BREAKING**: Renamed `drivers-vector-postgresql` extra to `drivers-vector-pgvector`.
 - **BREAKING**: Update `marqo` dependency to `^3.7.0`.
 - **BREAKING**: Removed `drivers-sql-postgresql` extra. Use `drivers-sql` extra and install necessary drivers (i.e. `psycopg2`) separately.
+- **BREAKING**: `api_key` and `search_id` are now required fields in `GoogleWebSearchDriver`.
+- **BREAKING**: `web_search_driver` is now required fields in the `WebSearch` Tool.
+- `GoogleWebSearchDriver` and `DuckDuckGoWebSearchDriver` now use `kwargs` passed to the `run` method.
 - Removed unnecessary `sqlalchemy-redshift` dependency in `drivers-sql-amazon-redshift` extra.
 - Removed unnecessary `transformers` dependency in `drivers-prompt-huggingface` extra.
 - Removed unnecessary `huggingface-hub` dependency in `drivers-prompt-huggingface-pipeline` extra.
@@ -378,6 +447,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Default model of `AmazonBedrockStructureConfig` to `anthropic.claude-3-sonnet-20240229-v1:0`.
 - `AnthropicPromptDriver` and `BedrockClaudePromptModelDriver` to use Anthropic's Messages API.
 - `OpenAiVisionImageQueryDriver` now has a required field `max_tokens` that defaults to 256
+- `GriptapeCloudStructureRunDriver` now outputs a `BaseArtifact` instead of a `TextArtifact`
 
 ## [0.23.2] - 2024-03-15
 
@@ -407,7 +477,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `JsonExtractionTask` for convenience over using `ExtractionTask` with a `JsonExtractionEngine`.
 - `CsvExtractionTask` for convenience over using `ExtractionTask` with a `CsvExtractionEngine`.
 - `OpenAiVisionImageQueryDriver` to support queries on images using OpenAI's Vision model.
-- `ImageQueryClient` allowing an Agent to make queries on images on disk or in Task Memory.
+- `ImageQueryTool` allowing an Agent to make queries on images on disk or in Task Memory.
 - `ImageQueryTask` and `ImageQueryEngine`.
 
 ### Fixed 

diff --git a/Makefile b/Makefile
@@ -23,6 +23,7 @@ install/core: ## Install core dependencies.
 .PHONY: install/all
 install/all: ## Install all dependencies.
 	@poetry install --with dev --with test --with docs --all-extras
+	@poetry run pre-commit install
 
 .PHONY: install/dev
 install/dev: ## Install dev dependencies.
@@ -68,7 +69,7 @@ check/lint:
 
 .PHONY: check/types
 check/types:
-	@poetry run pyright griptape/
+	@poetry run pyright griptape $(shell find docs -type f -path "*/src/*")
 
 .PHONY: check/spell
 check/spell:

diff --git a/README.md b/README.md
@@ -89,57 +89,78 @@ With Griptape, you can create Structures, such as Agents, Pipelines, and Workflo
 
 ```python
 from griptape.structures import Agent
-from griptape.tools import WebScraper, FileManager, TaskMemoryClient
+from griptape.tools import WebScraperTool, FileManagerTool, PromptSummaryTool
 
 agent = Agent(
     input="Load {{ args[0] }}, summarize it, and store it in a file called {{ args[1] }}.",
     tools=[
-        WebScraper(off_prompt=True),
-        TaskMemoryClient(off_prompt=True),
-        FileManager()
+        WebScraperTool(off_prompt=True),
+        PromptSummaryTool(off_prompt=True),
+        FileManagerTool()
     ]
 )
 agent.run("https://griptape.ai", "griptape.txt")
 ```
 
 And here is the output:
 ```
-[04/02/24 13:51:09] INFO     ToolkitTask 85700ec1b0594e1a9502c0efe7da6ef4
+[08/12/24 14:48:15] INFO     ToolkitTask c90d263ec69046e8b30323c131ae4ba0
                              Input: Load https://griptape.ai, summarize it, and store it in a file called griptape.txt.
-[04/02/24 13:51:15] INFO     Subtask db6a3e7cb2f549128c358149d340f91c
-                             Thought: First, I need to load the content of the website using the WebScraper action. Then, I will use the TaskMemoryClient action to
-                             summarize the content. Finally, I will save the summarized content to a file using the FileManager action.
+[08/12/24 14:48:16] INFO     Subtask ebe23832cbe2464fb9ecde9fcee7c30f
                              Actions: [
                                {
-                                 "name": "WebScraper",
+                                 "tag": "call_62kBnkswnk9Y6GH6kn1GIKk6",
+                                 "name": "WebScraperTool",
                                  "path": "get_content",
                                  "input": {
                                    "values": {
                                      "url": "https://griptape.ai"
                                    }
-                                 },
-                                 "tag": "load_website_content"
+                                 }
                                }
                              ]
-[04/02/24 13:51:16] INFO     Subtask db6a3e7cb2f549128c358149d340f91c
-                             Response: Output of "WebScraper.get_content" was stored in memory with memory_name "TaskMemory" and artifact_namespace
-                             "752b38bb86da4baabdbd9f444eb4a0d1"
-[04/02/24 13:51:19] INFO     Subtask c3edba87ebf845d4b85e3a791f8fde8d
-                             Thought: Now that the website content is loaded into memory, I need to summarize it using the TaskMemoryClient action.
-                             Actions: [{"tag": "summarize_content", "name": "TaskMemoryClient", "path": "summarize", "input": {"values": {"memory_name": "TaskMemory",
-                             "artifact_namespace": "752b38bb86da4baabdbd9f444eb4a0d1"}}}]
-[04/02/24 13:51:25] INFO     Subtask c3edba87ebf845d4b85e3a791f8fde8d
-                             Response: Output of "TaskMemoryClient.summarize" was stored in memory with memory_name "TaskMemory" and artifact_namespace
-                             "c4f131c201f147dcab07be3925b46294"
-[04/02/24 13:51:33] INFO     Subtask 06fe01ca64a744b38a8c08eb152aaacb
-                             Thought: Now that the content has been summarized and stored in memory, I need to save this summarized content to a file named 'griptape.txt'
-                             using the FileManager action.
-                             Actions: [{"tag": "save_summarized_content", "name": "FileManager", "path": "save_memory_artifacts_to_disk", "input": {"values": {"dir_name":
-                             ".", "file_name": "griptape.txt", "memory_name": "TaskMemory", "artifact_namespace": "c4f131c201f147dcab07be3925b46294"}}}]
-                    INFO     Subtask 06fe01ca64a744b38a8c08eb152aaacb
-                             Response: saved successfully
-[04/02/24 13:51:35] INFO     ToolkitTask 85700ec1b0594e1a9502c0efe7da6ef4
-                             Output: The summarized content of the website https://griptape.ai has been successfully saved to a file named 'griptape.txt'.
+[08/12/24 14:48:17] INFO     Subtask ebe23832cbe2464fb9ecde9fcee7c30f
+                             Response: Output of "WebScraperTool.get_content" was stored in memory with memory_name "TaskMemory" and artifact_namespace
+                             "cecca28eb0c74bcd8c7119ed7f790c95"
+[08/12/24 14:48:18] INFO     Subtask dca04901436d49d2ade86cd6b4e1038a
+                             Actions: [
+                               {
+                                 "tag": "call_o9F1taIxHty0mDlWLcAjTAAu",
+                                 "name": "PromptSummaryTool",
+                                 "path": "summarize",
+                                 "input": {
+                                   "values": {
+                                     "summary": {
+                                       "memory_name": "TaskMemory",
+                                       "artifact_namespace": "cecca28eb0c74bcd8c7119ed7f790c95"
+                                     }
+                                   }
+                                 }
+                               }
+                             ]
+[08/12/24 14:48:21] INFO     Subtask dca04901436d49d2ade86cd6b4e1038a
+                             Response: Output of "PromptSummaryTool.summarize" was stored in memory with memory_name "TaskMemory" and artifact_namespace
+                             "73765e32b8404e32927822250dc2ae8b"
+[08/12/24 14:48:22] INFO     Subtask c233853450fb4fd6a3e9c04c52b33bf6
+                             Actions: [
+                               {
+                                 "tag": "call_eKvIUIw45aRYKDBpT1gGKc9b",
+                                 "name": "FileManagerTool",
+                                 "path": "save_memory_artifacts_to_disk",
+                                 "input": {
+                                   "values": {
+                                     "dir_name": ".",
+                                     "file_name": "griptape.txt",
+                                     "memory_name": "TaskMemory",
+                                     "artifact_namespace": "73765e32b8404e32927822250dc2ae8b"
+                                   }
+                                 }
+                               }
+                             ]
+                    INFO     Subtask c233853450fb4fd6a3e9c04c52b33bf6
+                             Response: Successfully saved memory artifacts to disk
+[08/12/24 14:48:23] INFO     ToolkitTask c90d263ec69046e8b30323c131ae4ba0
+                             Output: The content from https://griptape.ai has been summarized and stored in a file called `griptape.txt`.
 ```
 
 During the run, the Griptape Agent loaded a webpage with a [Tool](https://docs.griptape.ai/stable/griptape-tools/), stored its full content in [Task Memory](https://docs.griptape.ai/stable/griptape-framework/structures/task-memory.md), queried it to answer the original question, and finally saved the answer to a file.

diff --git a/_typos.toml b/_typos.toml
@@ -1,3 +1,6 @@
+[default]
+extend-ignore-re = ["call_[[:alnum:]]+"]
+
 [default.extend-words]
 # Don't correct the state ND
 ND = "ND"

diff --git a/docs/assets/css/extra.css b/docs/assets/css/extra.css
@@ -5,8 +5,3 @@
 .md-typeset table:not([class]) {
     display: table;
 }
-
-/* Hide the code block title since we're using it for other purposes.*/
-.filename {
-    display: none !important
-}