feat: Enable inference serving capabilities on sagemaker endpoint #536
Conversation
gwang111
changed the title
gwang111
changed the title
gwang111
force-pushed
the
inference-serving
branch
3 times, most recently
from
9cee77b to
bb132ea
Compare
build_artifacts/v2/v2.2/v2.2.0/dirs/etc/inference-server/server.py
Outdated
Show resolved
Hide resolved
build_artifacts/v2/v2.2/v2.2.0/dirs/etc/inference-server/server.py
Outdated
Show resolved
Hide resolved
build_artifacts/v2/v2.2/v2.2.0/dirs/etc/inference-server/server.py
Outdated
Show resolved
Hide resolved
build_artifacts/v2/v2.2/v2.2.0/dirs/etc/inference-server/server.py
Outdated
Show resolved
Hide resolved
| if requirements_txt.is_file(): | ||
| try: | ||
| subprocess.check_call( | ||
| [sys.executable, "-m", "pip", "install", "-r", str(requirements_txt)] |
There was a problem hiding this comment.
SMD recommends that we use micromamba instead of pip for installing dependencies.
There was a problem hiding this comment.
I couldn't find any micromamba resources for installed runtime dependencies without pip. Will reach out to Tian on this
build_artifacts/v2/v2.2/v2.2.0/dirs/etc/inference-server/server.py
Outdated
Show resolved
Hide resolved
build_artifacts/v2/v2.2/v2.2.0/dirs/etc/inference-server/tornado_server/sync_server.py
Outdated
Show resolved
Hide resolved
| CODE_DIRECTORY = "SAGEMAKER_INFERENCE_CODE_DIRECTORY" | ||
| CODE = "SAGEMAKER_INFERENCE_CODE" |
There was a problem hiding this comment.
we will review these with Saurabh/PM since these will be customer facing.
There was a problem hiding this comment.
will schedule a meeting next week
|
Can you move your changes under "template" folder?
For any changes/additions in cpu/gpu.env.in file, please create an issue (example) so that we'll add them to the next image major/minor releases |
Moved the code to the v3 template folder since this code will be launching in the next major version release |
gwang111
force-pushed
the
inference-serving
branch
10 times, most recently
from
271755a to
9865a75
Compare
gwang111
force-pushed
the
inference-serving
branch
from
9865a75 to
4153ebb
Compare
| astream() runs a provided async generator fn in an async manner. | ||
| """ | ||
|
|
||
| async def stream(self, generator: Generator): |
There was a problem hiding this comment.
Can we accept Iterator instead? It will work with Generators too since they're a subtype of Iterator and LangChain stream implementations return [Async]Iterator.
Description of changes:
Added source code to enable serving capabilities on SageMaker Endpoint.
Testing
TODO:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.