diff --git a/README.md b/README.md index e57c3390..b6bec7d0 100644 --- a/README.md +++ b/README.md @@ -193,14 +193,188 @@ Online documentation [here](https://mransan.github.io/ocaml-protoc/dev/pbrt/Pbrt | ------------- | ------------- | ----------| | | Type definition along with a `default` constructor function to conveniently create values of that type | | | --make | `make` constructor functions | | -| --binary | Binary encodings | pbrt | -| --yojson | JSON encoding using the widely popular [yojson](https://github.com/mjambon/yojson) library | pbrt_yojson | +| --binary | Binary encodings | `pbrt` | +| --yojson | JSON encoding using the widely popular [yojson](https://github.com/mjambon/yojson) library | `pbrt_yojson` | | --bs | BuckleScript encoding using the BuckleScript core binding to JS json library | [bs-ocaml-protoc-json][3] | -| --pp | pretty printing functions based on the Format module. | pbrt | -| --services | RPC definitions. | pbrt_services | +| --pp | pretty printing functions based on the Format module. | `pbrt` | +| --services | RPC definitions. | `pbrt_services` | [3]:https://www.npmjs.com/package/bs-ocaml-protoc-json +### Services + +With the `--services` option, ocaml-protoc now generates stubs for service +declarations. + +For example with the given `calculator.proto` file: + +```proto +syntax = "proto3"; + +message I32 { + int32 value = 0; +} + +message AddReq { + int32 a = 1; + int32 b = 2; +} + +service Calculator { + rpc add(AddReq) returns (I32); + + rpc add_stream(stream I32) returns (I32); +} +``` + +Using `ocaml-protoc --binary --services --ml_out=. calculator.proto`, we get the normal +type definitions, but also this service definition: + +```ocaml +(** Calculator service *) +module Calculator : sig + open Pbrt_services + open Pbrt_services.Value_mode + + module Client : sig + val add : (add_req, unary, i32, unary) Client.rpc + val add_stream : (i32, stream, i32, unary) Client.rpc + end + + module Server : sig + (** Produce a server implementation from handlers *) + val make : + add:((add_req, unary, i32, unary) Server.rpc -> 'handler) -> + add_stream:((add_req, stream, i32, unary) Server.rpc -> 'handler) -> + unit -> 'handler Pbrt_services.Server.t + end +end +``` + +This can then potentially be used with libraries that implement specific protobuf-based +network protocols, such as [ocaml-grpc](https://github.com/dialohq/ocaml-grpc) +or [ocaml-twirp](https://github.com/c-cube/ocaml-twirp), or other custom protocols. + +Protobuf service endpoints take a single type and return a single type, but they have the ability +to stream either side. We represent this ability with the `Pbrt_services.Value_mode` types: + +```ocaml +(** Whether there's a single value or a stream of them *) +module Value_mode = struct + type unary + type stream +end +``` + +#### Client-side + +A `(req, req_kind, res, res_kind) Client.rpc` is a bundle describing a single RPC endpoint, +from the client perspective. It contains the RPC name, service, etc. alongside encoders for +the request type `req`, and decoders for the response type `res`. + +The phantom types `req_kind` and `res_kind` represent the value mode for request, +respectively response. Here we see that `Calculator.Client.add` is unary for both +(it takes a single argument and returns a single value) +but `Calculator.Client.add_stream` takes a string of `i32` as parameters before +returning a single result. + +With transports such as grpc, all [4 combinations](https://grpc.io/docs/what-is-grpc/core-concepts/#rpc-life-cycle) +are possible. With twirp over HTTP 1.1, only unary mode is supported. + +#### Server-side + +On the server side, ocaml-protoc generates individual stubs, +like on the client side; but it also generates _services_ as bundles +of endpoints. One service corresponds to a `service` declaration +in the `.proto` file. + +
+ +Detailed explanation of how server-side services work + + +In practice, in something like twirp, a service could be added to a web server +by adding each endpoint to a single HTTP route; or a twirp-aware router could +directly map incoming HTTP queries to services. + +The trickiest part here is that the type `'handler Pbrt_services.Server.t` is +parametric. Indeed it'd be hard for the generated code to cater to every possible +combination of network transport and concurrency library (eio, lwt, async, etc.). + +Instead, the code is generic over `'handler` (the type of a query handler for a _single_ +endpoint; e.g. a HTTP endpoint for a single route). The function +```ocaml + module Server : sig + val make : + add:((add_req, unary, i32, unary) Server.rpc -> 'handler) -> + add_stream:((add_req, stream, i32, unary) Server.rpc -> 'handler) -> + unit -> 'handler Pbrt_services.Server.t + end +``` +seen previously is used to build the `'handler service` by asking the user +to provide a handler for each method. The builder for `add` is given a +description of the `add` endpoint (with decoders for requests; and encoders +for responses), and must return a handler that knows how to decode the request, +add numbers, and turn that back into a response. + +Libraries will provide facilities to build such handlers, so that the user +only has to provide the actual logic (here, adding numbers). For example +in `twirp_tiny_httpd` (part of `ocaml-twirp`), implementing a +server looks like this[^1]: + +[^1]: we use a different `.proto` because twirp doesn't handle streams. + +```proto +syntax = "proto3"; + +message I32 { + int32 value = 0; +} + +message AddReq { + int32 a = 1; + int32 b = 2; +} + +message AddAllReq { + repeated int32 ints = 1; +} + +service Calculator { + rpc add(AddReq) returns (I32); + + rpc add_all(AddAllReq) returns (I32); +} +``` + +```ocaml +let add (a : add_req) : i32 = default_i32 ~value:Int32.(add a.a a.b) () + +let add_all (a : add_all_req) : i32 = + let l = ref 0l in + List.iter (fun x -> l := Int32.add !l x) a.ints; + default_i32 ~value:!l () + +let calc_service : Twirp_tiny_httpd.handler Pbrt_services.Server.t = + Calculator.Server.make + ~add:(fun rpc -> Twirp_tiny_httpd.mk_handler rpc add) + ~add_all:(fun rpc -> Twirp_tiny_httpd.mk_handler rpc add_all) + () + +let() = + let server = Tiny_httpd.create ~port:1234 () in + Twirp_tiny_httpd.add_service ~prefix:(Some "twirp") server calc_service; + Tiny_httpd.run_exn server +``` + +Here we see that all the logic is in `add` and `add_all`, which know nothing +about protobuf or serialization. A `calc_service` bundle, using the `Twirp_tiny_httpd.handler` +type for each handler, is built from them. Finally, a HTTP server is created, +the service is added to it (binding some routes), and we enter the +server's main loop. + +
+ ### Protobuf <-> OCaml mapping see [here](doc/protobuf_ocaml_mapping.md).