Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add PagedAttention #13

Merged
merged 54 commits into from
Dec 3, 2023
Merged

Add PagedAttention #13

merged 54 commits into from
Dec 3, 2023

Conversation

EricLBuehler
Copy link
Owner

Adds the PagedAttention implementation.

@EricLBuehler
Add new paged attention basic layout
ec98e4f
@EricLBuehler
Update readme
f6dc172
@EricLBuehler
Add commend
859338c
@EricLBuehler
Add calls to vllm paged attention impl via FFI
680df93
@EricLBuehler
Debug state
15a8194
@EricLBuehler
Use cc .std, .opt_level, .include, still debug
da1cd99
@EricLBuehler
Use setup.py
c0c9e0e
@EricLBuehler
Better install
268a963
@EricLBuehler
Add build.rs
fec3e9e
@EricLBuehler
Add some installation instructions
4c21697
@EricLBuehler
Update installation instructions
2e1575f
@EricLBuehler
Add pytorch installation method
e382d8e
@EricLBuehler
Update pytorch installation method
11cfa2d
@EricLBuehler
Update pytorch installation method
7e61a17
@EricLBuehler
Update pytorch installation method
be6bf5f
@EricLBuehler
Update pytorch installation method
56a14c2
@EricLBuehler
Update readme
1d4700a
@EricLBuehler
Add the Compiling PagedAttention CUDA kernels section
75cd454
@EricLBuehler
Fix typo
8eeb250
@EricLBuehler
Make directions explicit
e453c05
@EricLBuehler
Only compile specific kernel
1df0f81
@EricLBuehler
Small update
32a72c0
@EricLBuehler
Add basic framework of PagedAttention.forward
0746cc5
@EricLBuehler
Add initial BlockDiagonalCausalMask
3dfb563
@EricLBuehler
temporary save
c7cfca9
@EricLBuehler
Implement attention bias
97e91c5
@EricLBuehler
Refactor, mostly implement PagedAttention
eba2ce8
@EricLBuehler
Clippy fixes
ad0dfd6
@EricLBuehler
Implement .copy_ manually
75105db
@EricLBuehler
Hide memory-efficient-attention behind a feature flag
d76f42e
@EricLBuehler EricLBuehler added the enhancement New feature or request label Dec 3, 2023
@EricLBuehler EricLBuehler self-assigned this Dec 3, 2023
@EricLBuehler EricLBuehler mentioned this pull request Dec 3, 2023
Closed
32 tasks
@EricLBuehler EricLBuehler marked this pull request as draft December 3, 2023 12:18
@EricLBuehler EricLBuehler added the tracking Tracking issue or PR for a new feature. label Dec 3, 2023
@EricLBuehler EricLBuehler changed the title Add Paged attention Add PagedAttention Dec 3, 2023
@EricLBuehler EricLBuehler marked this pull request as ready for review December 3, 2023 20:27
EricLBuehler
EricLBuehler commented Dec 3, 2023
View reviewed changes
Cargo.toml Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
src/paged_attention/attn_bias.rs Outdated Show resolved Hide resolved
@EricLBuehler EricLBuehler merged commit fae2610 into master Dec 3, 2023
5 checks passed
@EricLBuehler EricLBuehler deleted the paged_attention branch December 3, 2023 21:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@EricLBuehler EricLBuehler

Labels
enhancement New feature or request tracking Tracking issue or PR for a new feature.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@EricLBuehler