Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #, if available: Closes #80
Description of changes:
This PR adds a basic agenteval Target for testing Bedrock Flows. Key caveats include:
BedrockAgentTarget
doesn't seem to actually do anything when it receives error chunks in the response stream... I wrote the flow target to treat these as an error andraise
them for now, but maybe different behaviour is expected?InvokeFlow
API seems to imply all sorts of interesting things like multiple inputs and multiple outputs... As far as I can tell at the moment (from building flows in the console) there's always exactly one input calledFlowInputNode
. This sits pretty nicely anyway with agenteval's "string prompt in, string response out" assumptions.Testing & docs checklist:
agenteval run
works with a simple Flow & couple of single-turn scripts in my accountBy submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.