Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unified pipeline for models & support phi3 model #45

Merged
merged 24 commits into from
Jul 3, 2024

Conversation

guoqingbao
Copy link
Collaborator

This PR provides a unified pipeline for models and adds the phi3 model under the unified pipeline. The major changes are:

  1. The LLaMaPipeline is removed and replaced with DefaultPipeline (and default loader), which can serve different models such as LLaMa and phi3.
  2. The phi3 model is added and tested, achieving around 100 tokens/second generation speed on A100 (BF16).
  3. Configuration for models is simplified.
  4. The padding strategy is optimized.
  5. ReadMe is revised to reflect the recent changes.

More models are expected to be added using the DefaultPipeline.

Mixed precision is used for the phi3 model because I found rope and rmsnorm require at least FP32 for long sequence generation (e.g., over 2000 tokens prompt).

Command line to run Phi3 3.8B chat service

cargo run --release -- --port 2000 --weight-path /home/phi3-3.8b/ phi3 --repeat-last-n 64

Please ignore the previous commit messages :)

Mention #44

@guoqingbao guoqingbao changed the title Unfied pipeline for models & support phi3 model Unified pipeline for models & support phi3 model Jul 2, 2024
Copy link
Owner

@EricLBuehler EricLBuehler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@EricLBuehler EricLBuehler merged commit 743a8b2 into EricLBuehler:master Jul 3, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants