Is the gRPC in BentoML stable and actively maintained?

[ryan-3m]

As far as I know, gRPC is a new feature added after version 1.x of BentoML. Is it still considered experimental?

I have some high-traffic tasks, such as processing videos, for which REST would incur too much overhead. Therefore, I have to choose gRPC.

[parano]

Hi @ryan-3m, yes the gRPC support is actively being developed and maintained, however still considered a beta feature. It would be great to hear your feedback on this.

[optiluca]

OK, so the "server side" of gRPC is still supported? I can't find any documentation at all. Apologies if I'm missing something, I've not actually tried BentoML yet, but am trying to understand if it could be a fit for my projects (which involve models with relatively low compute requirements, but large input/output sizes, meaning that the communication overhead can become dominant).
Thanks!

[aarnphm]

For the new 1.2 API we haven't update gRPC API yet.

gRPC API are supported in 1.1 API style. What is your specific requirements that utilise gRPC better than just normal HTTP?

I think the incentives behind gRPC are suited when you have multiple services in different languages, then gRPC makes sense.

[aarnphm]

Additionally, I don't think communication overhead is a big concern even for HTTP, given we are just sending bytes over the wire (I might be wrong).

we might want to investigate with Zmq and probably do another internal benchmark if migrating to a language other than Python makes sense. This has been floating around the office for a while now 😄

[optiluca]

The desire to use gRPC is driven by the fact that sending (binary) protobufs is much more efficient than sending (text) JSONs. Plus using proto means I can automatically generate clients for multiple programming languages, as you say.

But you're right, I guess the fact I associate HTTP REST with JSONs is just because that's what's customary, and I could probably send binary data as a message and get performance that should be somewhere in between.

[oogou11]

In my project based on the Python language, I adopted gRPC communication, primarily to construct multiple LLM microservices. Previously, I used HTTP communication and directly transmitted tensors. When testing communication latency (communication only) on my server, the results were: gRPC - 1.308 seconds, HTTP - 4.53 seconds, with tensor.shape = 1*4.19 million. However, when using gRPC communication, I first applied the tolist operation to the tensor.

environment:

• bentoml: 1.3.3
• GPU: NVIDIA GeForce RTX 4090
• CUDA: 12.4
• Operating System: Ubuntu 20.04.6

[MrBeeMovie]

Hi @ryan-3m, yes the gRPC support is actively being developed and maintained, however still considered a beta feature. It would be great to hear your feedback on this.

I have been trying to run the sentence-transformers example repo you guys have but serve-grpc always fails with:

bentoml serve-grpc
2024-03-19T12:23:19-0400 [WARNING] [cli] Deprecated build option: 'docker.env' is used, please use 'envs' instead.
Error: [serve] `serve-grpc` failed: <class '_bentoml_sdk.service.factory.Service'> type doesn't support gRPC serving

I have made sure to install bentoml[grpc]. Any ideas?

[EraYaN]

According to a conversation in slack, the new 1.2 API is not yet supported by gRPC. So you would need to use the one 1.1 style bento service for gRPC to work (not a class but functions).