build: RHEL8 PyTorch Backend #7524
Conversation
* Update README and versions for 2.48.0 / 24.07
[24.07] Revert vllm to 0.5.0.post1
#7440) Co-authored-by: Misha Chornyi <[email protected]> Co-authored-by: Francesco Petrini <[email protected]>
Downgrade ompi version to v4.1.5rc2
More general capture of Linux Distros tritonserver compiling except for --filesystem=gcs and --filesystem=s3 adding sbsa installation for dcgm updating RUNPATH directory for tritonserver executable cleanup removed build_ubuntu and updated build.py
* Merge ORT RHEL changes Into Kyle's RHEL dev
build.py
Outdated
| df += """ | ||
| # Point to python 3.10 which is what triton expects the default to be | ||
| ENV PYVER 3.10 | ||
| ENV PYTHONPATH /opt/python/v | ||
| RUN ln -sf /opt/python/cp${PYVER/./}* ${PYTHONPATH} | ||
| ENV PYBIN ${PYTHONPATH}/bin | ||
| ENV PYTHON_BIN_PATH ${PYBIN}/python${PYVER} | ||
| ENV PATH ${PYBIN}:${PATH} |
There was a problem hiding this comment.
Why do we need custom location for python installation?
There was a problem hiding this comment.
The manylinux container we are using as a base has several versions of python installed. By default, it will use Python3.7 which is incompatible with some of the packages we need to install. So here we default the version to 3.10.
There was a problem hiding this comment.
Why don't we use proper version of Python aligned with all DLFW products instead and expanding existing image?
There was a problem hiding this comment.
Python 3.10 is the properly aligned version for Triton and PyTorch as far as I can tell: https://docs.nvidia.com/deeplearning/frameworks/support-matrix/index.html. Realistically, the best place to fix this would be the origin--the base container. I'm not sure what the rationale is for having it default to 3.7.
There was a problem hiding this comment.
A fix at that level would prevent us and the Pytorch team from having to do this in the first place.
Goal: Support the PyTorch Backend on RHEL8 systems. Importantly, these changes will disable optional PyTorch optional features such as
TRITON_PYTORCH_ENABLE_TORCHTRTandTRITON_ENABLE_NVTX, which are not currently supported for the RHEL8 build.PyTorch Backend: triton-inference-server/pytorch_backend#137