Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

increase cuda_cpy block size #996

Merged
merged 1 commit into from
Oct 23, 2024
Merged

increase cuda_cpy block size #996

merged 1 commit into from
Oct 23, 2024

Conversation

bssrdf
Copy link
Contributor

@bssrdf bssrdf commented Oct 22, 2024

This PR gives a small performance boost to cuda backend cpy op. All test cases in test-backend-ops passed.

@bssrdf bssrdf mentioned this pull request Oct 22, 2024
Closed
JohannesGaessler
JohannesGaessler approved these changes Oct 22, 2024
View reviewed changes
Copy link
Collaborator

@JohannesGaessler JohannesGaessler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should be able to get even better performance by replacing the scalar copies with copies of half2/float2.

@bssrdf
Copy link
Contributor Author

bssrdf commented Oct 23, 2024

Thanks for approving. I read somewhere that vectorized global memory read does not improve throughput much. It helps with loading from shared memory.

@JohannesGaessler JohannesGaessler merged commit d51c6c0 into ggerganov:master Oct 23, 2024
4 checks passed
@bssrdf bssrdf deleted the bump-cuda-cpy-block-size branch October 28, 2024 02:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@JohannesGaessler JohannesGaessler JohannesGaessler approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@bssrdf @JohannesGaessler