Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Major project refactor #44

Merged
merged 4 commits into from
May 1, 2024
Merged

Major project refactor #44

merged 4 commits into from
May 1, 2024

Conversation

tonykchen
Copy link
Contributor

@tonykchen tonykchen commented May 1, 2024

Issue #, if available:
#14

Description of changes:

  • Reorganized and consolidated a majority of the modules to simplify the codebase.
  • Added a --filter CLI argument to run specific tests.
  • Refactored the tests configuration from a list to a mapping, where the test name serves as the key. This allows us to fetch the tests more efficiently when --filter is specified.
  • Implemented logging for evaluator token I/O.
  • Added provisioned throughput support for evaluator models (will implement PT support for targets in a later PR).
  • Remove Claude 3 Haiku as an available evaluator model for the initial release.
  • Implemented target and evaluator factories which encapsulates logic needed to create these objects.
  • Updated user documentation with changes made.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Copy link
Contributor

@sharonxiaohanli sharonxiaohanli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tonykchen tonykchen merged commit 0a0569d into awslabs:main May 1, 2024
4 checks passed
@tonykchen tonykchen deleted the refactor-plan branch May 1, 2024 18:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants