Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for multimodal in Vertex #1338

Merged
merged 6 commits into from
Sep 9, 2024
Merged

feat: add support for multimodal in Vertex #1338

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
f0b5f93
feat: add support for multimodal in Vertex
ArthurGoupil Jul 10, 2024
2179040
Nit changes and remove tools if multimodal
ArthurGoupil Sep 5, 2024
d2ef445
revert model name change
ArthurGoupil Sep 5, 2024
eab152c
Fix tools/multimodal condition
ArthurGoupil Sep 5, 2024
5a06d79
Merge branch 'main' into feat/multimodal-vertex
nsarrazin Sep 9, 2024
1601442
chores(lint): fix formatting
nsarrazin Sep 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 14 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -762,21 +762,31 @@ MODELS=`[
{
"name": "gemini-1.5-pro",
"displayName": "Vertex Gemini Pro 1.5",
"multimodal": true,
"endpoints" : [{
"type": "vertex",
"project": "abc-xyz",
"location": "europe-west3",
"model": "gemini-1.5-pro-preview-0409", // model-name

"model": {
ArthurGoupil marked this conversation as resolved.
Show resolved Hide resolved
"id": "some-id",
},
// Optional
"safetyThreshold": "BLOCK_MEDIUM_AND_ABOVE",
"apiEndpoint": "", // alternative api endpoint url,
// Optional
"tools": [{
ArthurGoupil marked this conversation as resolved.
Show resolved Hide resolved
"googleSearchRetrieval": {
"disableAttribution": true
}
}]
}],
"multimodal": {
"image": {
"supportedMimeTypes": ["image/png", "image/jpeg", "image/webp"],
"preferredMimeType": "image/png",
"maxSizeInMB": 5,
"maxWidth": 2000,
"maxHeight": 1000;
}
}
}]
},
]`
Expand Down
58 changes: 46 additions & 12 deletions src/lib/server/endpoints/google/endpointVertex.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import type { Endpoint } from "../endpoints";
import { z } from "zod";
import type { Message } from "$lib/types/Message";
import type { TextGenerationStreamOutput } from "@huggingface/inference";
import { createImageProcessorOptionsValidator, makeImageProcessor } from "../images";

export const endpointVertexParametersSchema = z.object({
weight: z.number().int().positive().default(1),
Expand All @@ -27,10 +28,28 @@ export const endpointVertexParametersSchema = z.object({
])
.optional(),
tools: z.array(z.any()).optional(),
multimodal: z
.object({
image: createImageProcessorOptionsValidator({
supportedMimeTypes: [
"image/png",
"image/jpeg",
"image/webp",
"image/avif",
"image/tiff",
"image/gif",
],
preferredMimeType: "image/webp",
maxSizeInMB: Infinity,
maxWidth: 4096,
maxHeight: 4096,
}),
})
.default({}),
});

export function endpointVertex(input: z.input<typeof endpointVertexParametersSchema>): Endpoint {
const { project, location, model, apiEndpoint, safetyThreshold, tools } =
const { project, location, model, apiEndpoint, safetyThreshold, tools, multimodal } =
endpointVertexParametersSchema.parse(input);

const vertex_ai = new VertexAI({
Expand Down Expand Up @@ -73,7 +92,8 @@ export function endpointVertex(input: z.input<typeof endpointVertexParametersSch
stopSequences: parameters?.stop,
temperature: parameters?.temperature ?? 1,
},
tools,
// tools and multimodal are mutually exclusive
tools: !multimodal ? tools : undefined,
Copy link
Contributor

@pocman pocman Sep 5, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

condition should not be on !multimodal (otherwise, the model without picture provided will never use the other tools) but on files.length == 0.

If not files is provided for this specific query to vertex, then tools can be provided

});

// Preprompt is the same as the first system message.
Expand All @@ -83,16 +103,30 @@ export function endpointVertex(input: z.input<typeof endpointVertexParametersSch
messages.shift();
}

const vertexMessages = messages.map(({ from, content }: Omit<Message, "id">): Content => {
return {
role: from === "user" ? "user" : "model",
parts: [
{
text: content,
},
],
};
});
const vertexMessages = await Promise.all(
messages.map(async ({ from, content, files }: Omit<Message, "id">): Promise<Content> => {
const imageProcessor = makeImageProcessor(multimodal.image);
const processedFiles =
files && files.length > 0
? await Promise.all(files.map(async (file) => imageProcessor(file)))
: [];

return {
role: from === "user" ? "user" : "model",
parts: [
...processedFiles.map((processedFile) => ({
inlineData: {
data: processedFile.image.toString("base64"),
mimeType: processedFile.mime,
},
})),
{
text: content,
},
],
};
})
);

const result = await generativeModel.generateContentStream({
contents: vertexMessages,
Expand Down