Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Amazon Nova Models Support #613

Merged
merged 9 commits into from
Dec 3, 2024
Merged

Amazon Nova Models Support #613

merged 9 commits into from
Dec 3, 2024

Conversation

bigadsoleiman
Copy link
Collaborator

Issue #, if available:

Description of changes:
Adding support of new Amazon Nova models

  • Amazon Nova Micro
  • Amazon Nova Lite
  • Amazon Nova Pro
  • Amazon Nova Canvas
  • Amazon Nova Reels

Backend

  • Prevented the layer builder to remove boto3* botocore* since we need to install them from .whl this is until new boto3 apis are made public
  • Added to langchain adapter a new regex to register bedrock.amazon.nova to use existing BedrockChatAdapter this will only work for Nova Micro since the other has to go through multimodal adapter.
  • Changed SAFE_STR_REGEX to support multi-path s3 paths (i.e. private/user_id/videos/..../*.mp4) since the video generations forcefully puts videos under "randomid/output.mp4" under any S3 URI passed as output path.
  • Changed MultiModalBase to support file upload/retrieval to/from S3, converse API and generation of non-text content (video/image)
  • Adapted Claude/ Idefics adapters to new MultiModalBase class by returning the expected new structure.
  • Created Nova adapter from MultiModalBase (using Converse API where currently possible)
    • Video generation relies on a new API bedrock.start_async_invoke which generates 6s video in background asynchronously in ~4m.

UI

  • chat-message.tsx has been broken down in smaller components.
  • added a chat-message-media-display.tsx to handle the display of media (image/video)
    • For video generation it assumes that if the S3 signed url is not avaible the video is still generating and does a long polling (every 30seconds) until the video is availble. This is a temporary solution to support async generation. A more robust backend workflow (stepfunctions?) should be implemented for long term.
  • added video-dialog for video uploads (max 10MB)
  • added seed options defaulting to 0 in llm-config-dialog, used only in image/video generation.
  • added support for single selection of multiple OUTPUT modalities in the chat-input-panel.tsx based on model's OUTPUT_MODALITIES, default to first or only available one.

Ran vet-all for linting and testing.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Copy link
Collaborator

@charles-marion charles-marion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the change!.

@bigadsoleiman bigadsoleiman merged commit 4cd3cda into main Dec 3, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

2 participants