Documentation for RPC Protocol

[ecyht2]

Is there any documentation for the RPC protocol? I think it would be nice to have it documented. I have an idea on how it works by reading the source code but I think it would be helpful to have a formal documentation about it. I am thinking of implementing my own server just for fun.

[rgerganov]

The RPC protocol is quite simple. There are several RPC commands which are used to implement the backend interfaces defined in ggml-backend-impl.h.

The format of the RPC request and RPC response is:

RPC request:
 | rpc_cmd (1 byte) | request_size (8 bytes) | request_data (request_size bytes) |

RPC response:
 | response_size (8 bytes) | response_data (response_size bytes) |

The RPC server is handling one client at a time, in a single thread. There is some work-in-progress for server-to-server communication but it is not ready yet.

That's it. If you have any questions, I'd be happy to answer.