Skip to content

Commit

Permalink
Refactor Artifacts
Browse files Browse the repository at this point in the history
  • Loading branch information
collindutter committed Sep 5, 2024
1 parent a1ad5b7 commit 346c8ec
Show file tree
Hide file tree
Showing 65 changed files with 500 additions and 486 deletions.
Empty file added .ignore
Empty file.
19 changes: 19 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## Unreleased

### Added
- `BaseArtifact.to_bytes()` method to convert an Artifact to bytes.

### Changed
- **BREAKING**: Removed `MediaArtifact`, use `ImageArtifact` or `AudioArtifact` instead.
- **BREAKING**: Removed `BooleanArtifact`, use `JsonArtifact` instead.
- **BREAKING**: Removed `CsvRowArtifact`.
- **BREAKING**: `CsvLoader`, `DataframeLoader`, and `SqlLoader` now return `list[TextArtifact]`.
- **BREAKING**: Removed `ImageArtifact.media_type`.
- **BREAKING**: Removed `AudioArtifact.media_type`.
- **BREAKING**: Removed `BlobArtifact.dir_name`.
- **BREAKING**: Moved `ImageArtifact.prompt` and `ImageArtifact.model` into `ImageArtifact.meta`.
- **BREAKING**: `ImageArtifact.to_text()` now returns the base64 encoded image.
- Updated `JsonArtifact` value converter to properly handle more types.
- `AudioArtifact` now subclasses `BaseArtifact` instead of `MediaArtifact`.
- `ImageArtifact` now subclasses `BaseArtifact` instead of `MediaArtifact`.
- Passing a dictionary as the value to `TextArtifact` will convert to a key-value formatted string.
- Removed `__add__` method from `BaseArtifact`, implemented it where necessary.

## [0.31.0] - 2024-09-03

**Note**: This release includes breaking changes. Please refer to the [Migration Guide](./MIGRATION.md#030x-to-031x) for details.
Expand Down
117 changes: 117 additions & 0 deletions MIGRATION.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,123 @@
# Migration Guide

This document provides instructions for migrating your codebase to accommodate breaking changes introduced in new versions of Griptape.
## 0.31.X to 0.32.X

### Removed `MediaArtifact`

`MediaArtifact` has been removed. Use `ImageArtifact` or `AudioArtifact` instead.

#### Before

```python
image_media = MediaArtifact(
b"image_data",
media_type="image",
format="jpeg"
)

audio_media = MediaArtifact(
b"audio_data",
media_type="audio",
format="wav"
)
```

#### After
```python
image_artifact = ImageArtifact(
b"image_data",
format="jpeg"
)

audio_artifact = AudioArtifact(
b"audio_data",
format="wav"
)
```

### Removed `BooleanArtifact`

`BooleanArtifact` has been removed. Use `JsonArtifact` instead.

#### Before

```python
boolean_artifact = BooleanArtifact("true")

print(boolean_artifact.value) # Value is True
```

#### After
```python
json_artifact = JsonArtifact("true")

print(json_artifact.value) # Value is True
```

### Removed `CsvRowArtifact`

`CsvRowArtifact` has been removed. Use `TextArtifact` instead.

#### Before

```python
CsvRowArtifact({"name": "John", "age": 30})
```

#### After
```python
TextArtifact("name: John\nAge: 30")
```

### `CsvLoader`, `DataframeLoader`, and `SqlLoader` return types

`CsvLoader`, `DataframeLoader`, and `SqlLoader` now return a tuple of `list[TextArtifact]` instead of `list[CsvRowArtifact]`.

#### Before

```python
results = CsvLoader().load(Path("people.csv").read_text())

print(results[0].value) # {"name": "John", "age": 30}
```

#### After
```python
results = CsvLoader().load(Path("people.csv").read_text())

print(results[0].value) # name: John\nAge: 30
print(results[0].meta["row"]) # 0
```

### Moved `ImageArtifact.prompt` and `ImageArtifact.model` to `ImageArtifact.meta`

`ImageArtifact.prompt` and `ImageArtifact.model` have been moved to `ImageArtifact.meta`.

#### Before

```python
image_artifact = ImageArtifact(
b"image_data",
format="jpeg",
prompt="Generate an image of a cat",
model="DALL-E"
)

print(image_artifact.prompt, image_artifact.model) # Generate an image of a cat, DALL-E
```

#### After
```python
image_artifact = ImageArtifact(
b"image_data",
format="jpeg",
meta={"prompt": "Generate an image of a cat", "model": "DALL-E"}
)

print(image_artifact.meta["prompt"], image_artifact.meta["model"]) # Generate an image of a cat, DALL-E
```


## 0.30.X to 0.31.X

Expand Down
60 changes: 27 additions & 33 deletions docs/griptape-framework/data/artifacts.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,60 +5,54 @@ search:

## Overview

**[Artifacts](../../reference/griptape/artifacts/base_artifact.md)** are used for passing different types of data between Griptape components. All tools return artifacts that are later consumed by tasks and task memory.
Artifacts make sure framework components enforce contracts when passing and consuming data.

**[Artifacts](../../reference/griptape/artifacts/base_artifact.md)** are used to store data that can be provided as input to or received as output from a Language Learning Model (LLM).

## Text

A [TextArtifact](../../reference/griptape/artifacts/text_artifact.md) for passing text data of arbitrary size around the framework. It can be used to count tokens with [token_count()](../../reference/griptape/artifacts/text_artifact.md#griptape.artifacts.text_artifact.TextArtifact.token_count) with a tokenizer.
It can also be used to generate a text embedding with [generate_embedding()](../../reference/griptape/artifacts/text_artifact.md#griptape.artifacts.text_artifact.TextArtifact.generate_embedding)
and access it with [embedding](../../reference/griptape/artifacts/text_artifact.md#griptape.artifacts.text_artifact.TextArtifact.embedding).
[TextArtifact](../../reference/griptape/artifacts/text_artifact.md)s store textual data. They can be used to count tokens using the [token_count()](../../reference/griptape/artifacts/text_artifact.md#griptape.artifacts.text_artifact.TextArtifact.token_count) method with a tokenizer, generate a text embedding through the [generate_embedding()](../../reference/griptape/artifacts/text_artifact.md#griptape.artifacts.text_artifact.TextArtifact.generate_embedding) method, and access the embedding with the [embedding](../../reference/griptape/artifacts/text_artifact.md#griptape.artifacts.text_artifact.TextArtifact.embedding) property.

[TaskMemory](../../reference/griptape/memory/task/task_memory.md) automatically stores [TextArtifact](../../reference/griptape/artifacts/text_artifact.md)s returned by tool activities and returns artifact IDs back to the LLM.
[TaskMemory](../../reference/griptape/memory/task/task_memory.md) automatically stores `TextArtifacts` returned by tool activities and provides their IDs back to the LLM.

## Csv Row
## Image

A [CsvRowArtifact](../../reference/griptape/artifacts/csv_row_artifact.md) for passing structured row data around the framework. It inherits from [TextArtifact](../../reference/griptape/artifacts/text_artifact.md) and overrides the
[to_text()](../../reference/griptape/artifacts/csv_row_artifact.md#griptape.artifacts.csv_row_artifact.CsvRowArtifact.to_text) method, which always returns a valid CSV row.
[ImageArtifact](../../reference/griptape/artifacts/image_artifact.md)s store image data. They include binary image data and metadata such as MIME type, dimensions, and prompt and model information for images returned by [image generation drivers](../drivers/image-generation-drivers.md). They inherit functionality from [BlobArtifacts](#blob).

## Info
## Audio

An [InfoArtifact](../../reference/griptape/artifacts/info_artifact.md) for passing short notifications back to the LLM without task memory storing them.
[AudioArtifact](../../reference/griptape/artifacts/audio_artifact.md)s store audio content, including binary audio data and metadata such as format, duration, and prompt and model information for audio returned by generative models. They inherit from [BlobArtifacts](#blob).

## Error
## Action

An [ErrorArtifact](../../reference/griptape/artifacts/error_artifact.md) is used for passing errors back to the LLM without task memory storing them.
[ActionArtifact](../../reference/griptape/artifacts/action_artifact.md)s represent actions taken by the LLM. Currently, the only supported action is [ToolAction](../../reference/griptape/common/actions/tool_action.md), which is used to execute a [Tool](../../griptape-framework/tools/index.md).

## Blob
## JSON

A [BlobArtifact](../../reference/griptape/artifacts/blob_artifact.md) for passing binary large objects (blobs) back to the LLM.
Treat it as a way to return unstructured data, such as images, videos, audio, and other files back from tools.
Each blob has a [name](../../reference/griptape/artifacts/base_artifact.md#griptape.artifacts.base_artifact.BaseArtifact.name) and
[dir](../../reference/griptape/artifacts/blob_artifact.md#griptape.artifacts.blob_artifact.BlobArtifact.dir_name) to uniquely identify stored objects.
[JsonArtifact](../../reference/griptape/artifacts/json_artifact.md)s store JSON-serializable data. Any data assigned to the `value` property is converted using `json.dumps(json.loads(value))`.

[TaskMemory](../../reference/griptape/memory/task/task_memory.md) automatically stores [BlobArtifact](../../reference/griptape/artifacts/blob_artifact.md)s returned by tool activities that can be reused by other tools.
## Generic

## Image
[GenericArtifact](../../reference/griptape/artifacts/generic_artifact.md)s act as an escape hatch for passing any type of data that does not fit into any other artifact type. While generally not recommended, they are suitable for specific scenarios. For example, see [talking to a video](../../examples/talk-to-a-video.md), which demonstrates using a `GenericArtifact` to pass a Gemini-specific video file.

An [ImageArtifact](../../reference/griptape/artifacts/image_artifact.md) is used for passing images back to the LLM. In addition to binary image data, an Image Artifact includes image metadata like MIME type, dimensions, and prompt and model information for images returned by [image generation Drivers](../drivers/image-generation-drivers.md). It inherits from [BlobArtifact](#blob).
## System Artifacts

## Audio
These Artifacts don't map to an LLM modality. They must be transformed in some way before they can be used as LLM input.

An [AudioArtifact](../../reference/griptape/artifacts/audio_artifact.md) allows the Framework to interact with audio content. An Audio Artifact includes binary audio content as well as metadata like format, duration, and prompt and model information for audio returned generative models. It inherits from [BlobArtifact](#blob).
### Blob

## Boolean
[BlobArtifact](../../reference/griptape/artifacts/blob_artifact.md)s store binary large objects (blobs) and are used to pass unstructured data back to the LLM via [InfoArtifact](#info).

A [BooleanArtifact](../../reference/griptape/artifacts/boolean_artifact.md) is used for passing boolean values around the framework.
`TaskMemory` automatically stores `BlobArtifacts` returned by tool activities, allowing them to be reused by other tools.

!!! info
Any object passed on init to `BooleanArtifact` will be coerced into a `bool` type. This might lead to unintended behavior: `BooleanArtifact("False").value is True`. Use [BooleanArtifact.parse_bool](../../reference/griptape/artifacts/boolean_artifact.md#griptape.artifacts.boolean_artifact.BooleanArtifact.parse_bool) to convert case-insensitive string literal values `"True"` and `"False"` into a `BooleanArtifact`: `BooleanArtifact.parse_bool("False").value is False`.
### Info

## Generic
[InfoArtifact](../../reference/griptape/artifacts/info_artifact.md)s store short notifications that are passed back to the LLM without being stored in Task Memory.

### Error

[ErrorArtifact](../../reference/griptape/artifacts/error_artifact.md)s store errors that are passed back to the LLM without being stored in Task Memory.

A [GenericArtifact](../../reference/griptape/artifacts/generic_artifact.md) can be used as an escape hatch for passing any type of data around the framework.
It is generally not recommended to use this Artifact type, but it can be used in a handful of situations where no other Artifact type fits the data being passed.
See [talking to a video](../../examples/talk-to-a-video.md) for an example of using a `GenericArtifact` to pass a Gemini-specific video file.
### List

## Json
[ListArtifact](../../reference/griptape/artifacts/list_artifact.md)s store lists of Artifacts that can be passed to the LLM.

A [JsonArtifact](../../reference/griptape/artifacts/json_artifact.md) is used for passing JSON-serliazable data around the framework. Anything passed to `value` will be converted using `json.dumps(json.loads(value))`.
6 changes: 3 additions & 3 deletions docs/griptape-framework/data/loaders.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,15 +22,15 @@ Inherits from the [TextLoader](../../reference/griptape/loaders/text_loader.md)

## SQL

Can be used to load data from a SQL database into [CsvRowArtifact](../../reference/griptape/artifacts/csv_row_artifact.md)s:
Can be used to load data from a SQL database into [TextArtifact](../../reference/griptape/artifacts/text_artifact.md)s:

```python
--8<-- "docs/griptape-framework/data/src/loaders_2.py"
```

## CSV

Can be used to load CSV files into [CsvRowArtifact](../../reference/griptape/artifacts/csv_row_artifact.md)s:
Can be used to load CSV files into [TextArtifact](../../reference/griptape/artifacts/text_artifact.md)s:

```python
--8<-- "docs/griptape-framework/data/src/loaders_3.py"
Expand All @@ -42,7 +42,7 @@ Can be used to load CSV files into [CsvRowArtifact](../../reference/griptape/art
!!! info
This driver requires the `loaders-dataframe` [extra](../index.md#extras).

Can be used to load [pandas](https://pandas.pydata.org/) [DataFrame](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html)s into [CsvRowArtifact](../../reference/griptape/artifacts/csv_row_artifact.md)s:
Can be used to load [pandas](https://pandas.pydata.org/) [DataFrame](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html)s into [TextArtifact](../../reference/griptape/artifacts/text_artifact.md)s:

```python
--8<-- "docs/griptape-framework/data/src/loaders_4.py"
Expand Down
18 changes: 8 additions & 10 deletions griptape/artifacts/__init__.py
Original file line number Diff line number Diff line change
@@ -1,30 +1,28 @@
from .base_artifact import BaseArtifact
from .error_artifact import ErrorArtifact
from .info_artifact import InfoArtifact
from .base_system_artifact import BaseSystemArtifact

from .text_artifact import TextArtifact
from .json_artifact import JsonArtifact
from .blob_artifact import BlobArtifact
from .boolean_artifact import BooleanArtifact
from .csv_row_artifact import CsvRowArtifact
from .list_artifact import ListArtifact
from .media_artifact import MediaArtifact
from .image_artifact import ImageArtifact
from .audio_artifact import AudioArtifact
from .json_artifact import JsonArtifact
from .action_artifact import ActionArtifact
from .generic_artifact import GenericArtifact

from .error_artifact import ErrorArtifact
from .info_artifact import InfoArtifact
from .list_artifact import ListArtifact


__all__ = [
"BaseArtifact",
"BaseSystemArtifact",
"ErrorArtifact",
"InfoArtifact",
"TextArtifact",
"JsonArtifact",
"BlobArtifact",
"BooleanArtifact",
"CsvRowArtifact",
"ListArtifact",
"MediaArtifact",
"ImageArtifact",
"AudioArtifact",
"ActionArtifact",
Expand Down
13 changes: 9 additions & 4 deletions griptape/artifacts/action_artifact.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,20 @@
from attrs import define, field

from griptape.artifacts import BaseArtifact
from griptape.mixins import SerializableMixin

if TYPE_CHECKING:
from griptape.common import ToolAction


@define()
class ActionArtifact(BaseArtifact, SerializableMixin):
class ActionArtifact(BaseArtifact):
"""Represents the LLM taking an action to use a Tool.
Attributes:
value: The Action to take. Currently only supports ToolAction.
"""

value: ToolAction = field(metadata={"serializable": True})

def __add__(self, other: BaseArtifact) -> ActionArtifact:
raise NotImplementedError
def to_text(self) -> str:
return str(self.value)
26 changes: 21 additions & 5 deletions griptape/artifacts/audio_artifact.py
Original file line number Diff line number Diff line change
@@ -1,12 +1,28 @@
from __future__ import annotations

from attrs import define
from attrs import define, field

from griptape.artifacts import MediaArtifact
from griptape.artifacts import BaseArtifact


@define
class AudioArtifact(MediaArtifact):
"""AudioArtifact is a type of MediaArtifact representing audio."""
class AudioArtifact(BaseArtifact):
"""Stores audio data.
media_type: str = "audio"
Attributes:
value: The audio data.
format: The audio format, e.g. "wav" or "mp3".
"""

value: bytes = field(metadata={"serializable": True})
format: str = field(kw_only=True, metadata={"serializable": True})

@property
def mime_type(self) -> str:
return f"audio/{self.format}"

def to_bytes(self) -> bytes:
return self.value

def to_text(self) -> str:
return f"Audio, format: {self.format}, size: {len(self.value)} bytes"
Loading

0 comments on commit 346c8ec

Please sign in to comment.