Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Support NVLM-D #1601

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

[WIP] Support NVLM-D #1601

wants to merge 1 commit into from

Conversation

amosyou
Copy link
Contributor

@amosyou amosyou commented Oct 7, 2024

Motivation

NVLM-D is a decoder-only VLM released by NVIDIA recently that has great performance across different benchmarks. I've been trying to get a better understanding of VLM/LLM architectures, figured supporting a model would be a good start.

Modifications

NVLM-D is based off InternViT-6B as the vision encoder, and Qwen as the LLM. This should work out nicely since we can reuse the Qwen implementation. There's also some interesting modifications I'll need to make (I think) for their tile tags, and modify the image_processors.py file to accomodate a new VLM.

Checklist

  • Format your code according to the Contributor Guide.
  • Add unit tests as outlined in the Contributor Guide.
  • Update documentation as needed, including docstrings or example tutorials.

@merrymercy
Copy link
Contributor

merrymercy commented Oct 11, 2024

Thanks for the contribution. Please fix the lint, and let us know when the PR is ready.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants