Replace Image Query Drivers with Prompt Drivers #1340
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅ 📢 Thoughts on this report? Let us know! |
collindutter
force-pushed
the
refactor/image-query
branch
2 times, most recently
from
39eba58 to
ab48bb6
Compare
collindutter
force-pushed
the
refactor/image-query
branch
from
ab48bb6 to
030aef0
Compare
| from griptape.structures import Agent | ||
| from griptape.tools import ImageQueryTool | ||
|
|
||
| # Create an Image Query Driver. | ||
| driver = OpenAiImageQueryDriver(model="gpt-4o") | ||
| driver = OpenAiChatPromptDriver(model="gpt-4o") | ||
|
|
||
| # Create an Image Query Tool configured to use the engine. | ||
| tool = ImageQueryTool( |
There was a problem hiding this comment.
I'm curious about the use case when an ImageQueryTool is still useful? if the prompt is "describe this image", the model should be able to do that now without using a tool.
There was a problem hiding this comment.
There is still value in having a Tool that can query images from the file system/task memory. Long term this functionality should maybe be baked into the File Manager Tool but that would require refactors outside the scope of this PR.
There was a problem hiding this comment.
got it. in that case, may be good to add ImageQueryTool to this list.
| @@ -40,7 +40,6 @@ Drivers facilitate interactions with external resources and services: | |||
| - 🔢 **Embedding Drivers** generate vector embeddings from textual inputs. | |||
| - 💾 **Vector Store Drivers** manage the storage and retrieval of embeddings. | |||
| - 🎨 **Image Generation Drivers** create images from text descriptions. | |||
| - 🔎 **Image Query Drivers** query images from text queries. | |||
There was a problem hiding this comment.
Up to you on this, but may be good to add a line in the prompt drivers that they can now handle multi-modal queries or something so it doesn't seem like we don't support it at all.
There was a problem hiding this comment.
Maybe this list could focus on functionality (like verbs), then mention which driver gives it to you 🤷 .
collindutter
force-pushed
the
refactor/image-query
branch
2 times, most recently
from
c8cc248 to
2767e54
Compare
collindutter
force-pushed
the
refactor/image-query
branch
from
2767e54 to
d1741e4
Compare
| @@ -40,7 +40,6 @@ Drivers facilitate interactions with external resources and services: | |||
| - 🔢 **Embedding Drivers** generate vector embeddings from textual inputs. | |||
| - 💾 **Vector Store Drivers** manage the storage and retrieval of embeddings. | |||
| - 🎨 **Image Generation Drivers** create images from text descriptions. | |||
| - 🔎 **Image Query Drivers** query images from text queries. | |||
There was a problem hiding this comment.
Maybe this list could focus on functionality (like verbs), then mention which driver gives it to you 🤷 .
Describe your changes
Image Query Drivers were added early on before Prompt Drivers supported image inputs. Now they provide no value other than some syntactic niceties. This PR removes them and improves some syntax with Prompt Drivers.
Note that Image Query Tool has been kept since we don't have a Prompt Tool (though maybe we should...separate discussion).
Added
PromptStack.from_artifactfactory method for creating a Prompt Stack with a user message from an Artifact.Changed
ImageQueryDrivers, usePromptDrivers instead.ImageQueryTask, usePromptTaskinstead.ImageQueryTool.image_query_drivertoImageQueryTool.prompt_driver.BasePromptDriver.runcan now accept an Artifact in addition to a Prompt Stack.Issue ticket number and link
NA
📚 Documentation preview 📚: https://griptape--1340.org.readthedocs.build//1340/