Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: Making tritonclient.http and tritonclient.grpc health checks uniform #814

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

KrishnanPrash
Copy link
Contributor

This PR is meant to address a silent error not being caught in tritonclient.http.is_server_live() and tritonclient.http.is_server_ready().

Currently if we start a tritonserver instance with a health restricted feature specification, with the following command:

tritonserver --model-repo ... --grpc-restricted-protocol=health:health-key=health-value  --http-restricted-api=health:health-key=health-value

Then if we use tritonclient to query for health information, we receive the following:

>>> import tritonclient.http as httpclient
>>> import tritonclient.grpc as grpcclient
>>> http_client = httpclient.InferenceServerClient("localhost:8000")
>>> grpc_client = grpcclient.InferenceServerClient("localhost:8001")
>>> grpc_client.is_server_ready(headers={"triton-grpc-protocol-health-key":"health-value"})
True
>>> http_client.is_server_ready(headers={"health-key":"health-value"})
True
>>> grpc_client.is_server_ready(headers={"invalid-key":"invalid-value"})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/tritonclient/grpc/_client.py", line 344, in is_server_ready
    raise_error_grpc(rpc_error)
  File "/usr/local/lib/python3.10/dist-packages/tritonclient/grpc/_utils.py", line 77, in raise_error_grpc
    raise get_error_grpc(rpc_error) from None
tritonclient.utils.InferenceServerException: [StatusCode.UNAVAILABLE] This protocol is restricted, expecting header 'triton-grpc-protocol-health-key'
>>> http_client.is_server_ready(headers={"invalid-key":"invalid-value"}) # Should be erroring out with InferenceServerException: This API is restricted, expecting header 'health-key`
False 

From the above example, we can see that tritonclient provides different behavior for grpc and http in terms of erroring out when invalid credentials are presented.

@rmccorm4 rmccorm4 changed the title Making tritonclient.http and tritonclient.grpc health checks uniform refactor: Making tritonclient.http and tritonclient.grpc health checks uniform Dec 20, 2024
@rmccorm4 rmccorm4 self-assigned this Dec 20, 2024
@rmccorm4 rmccorm4 requested review from rmccorm4 and GuanLuo December 20, 2024 23:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

3 participants