Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Target for Bedrock Flows #96

Merged
merged 1 commit into from
Dec 11, 2024
Merged

Conversation

athewsey
Copy link
Contributor

Issue #, if available: Closes #80

Description of changes:

This PR adds a basic agenteval Target for testing Bedrock Flows. Key caveats include:

  1. As far as I'm aware, Flows are stateless and this PR doesn't attempt to do anything about that on agenteval side: So it's mainly expecting single-turn tests
  2. I was surprised to see the existing BedrockAgentTarget doesn't seem to actually do anything when it receives error chunks in the response stream... I wrote the flow target to treat these as an error and raise them for now, but maybe different behaviour is expected?
  3. While the InvokeFlow API seems to imply all sorts of interesting things like multiple inputs and multiple outputs... As far as I can tell at the moment (from building flows in the console) there's always exactly one input called FlowInputNode. This sits pretty nicely anyway with agenteval's "string prompt in, string response out" assumptions.

Testing & docs checklist:

  • Added a (basic) unit test for the new target
  • Checked the overall agenteval run works with a simple Flow & couple of single-turn scripts in my account
  • Updated the "targets" section of the docs site

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Copy link
Contributor

@sharonxiaohanli sharonxiaohanli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@sharonxiaohanli sharonxiaohanli merged commit 198a1a1 into awslabs:main Dec 11, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Native Bedrock Prompt Flow target?
2 participants