Fix /vertex
payload parsing when MESSAGES_API_ENABLED
#2552
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR fixes the input payload parsing on Vertex AI i.e. when compiled with the
google
feature and deployed in a Vertex AI environment (replicable with the following environment variablesAIP_MODE=PREDICTION
,AIP_HTTP_PORT=80
,AIP_PREDICT_ROUTE=/predict
, andAIP_HEALTH_ROUTE=/health
); when the environment variableMESSAGES_API_ENABLED
is set to true, meaning that the/vertex
endpoint is a proxy for/v1/chat/completions
endpoint instead of the default/generate
.Note
As already discussed with @Narsil this is most likely not the best approach, here are some things that I tried:
ChatRequest
, but that still requires us to have a "duplicated" struct holding the generation kwargs, which is not idealChatRequest
whilst just leavingmessages
there, and using#[serde(flatten)]
over the generation kwargs; but that's not ideal eitherAny feedback is appreciated!
Who can review?
@Narsil as of our previous conversation today, feel free to take over the PR, or even close it in favour of a potentially better PR! Thanks in advance Nicolas! 🤗