Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add AMD gfx110* support #2641

Open
cazlo opened this issue Oct 13, 2024 · 0 comments · May be fixed by #2642
Open

Add AMD gfx110* support #2641

cazlo opened this issue Oct 13, 2024 · 0 comments · May be fixed by #2642

Comments

@cazlo
Copy link

cazlo commented Oct 13, 2024

Feature request

Add support for the gfx1101 and gfx1100 GPUs. Currently the official docs indicate lack of support for this hardware.

Motivation

Allow developers who have a 7900xt or 7900xtx GPU to utilize it to run local LLMs using TGI.

I have successfully run the image rocm/pytorch:rocm6.2.3_ubuntu22.04_py3.10_pytorch_release_2.3.0 on such a machine and demonstrated basic GPU accelerated compute smokechecks, so I think it is possible, if we change some configuration items in the build.

Your contribution

Looking at the upstream pytorch AMD docker build definition pytorch is built with gfx1100 support.

Looking at TGI AMD docker build definition, pytorch is not currently compiled with gfx1100 support.

Also some of these env vars related to ROCm tuning may need to change to support the consumer GPUs:
https://github.com/huggingface/text-generation-inference/blob/main/Dockerfile_amd#L323-L326

I plan to experiment with a rebuild configured with gfx1100 to see if it is achievable without further modifications.

cazlo added a commit to cazlo/text-generation-inference that referenced this issue Oct 13, 2024
@cazlo cazlo linked a pull request Oct 13, 2024 that will close this issue
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant