dune config for SSR with native code on the server

[andreypopp]

So I'm working on a way to SSR React components on server but compiling them to native code instead of running inside Node.js (yes, this means need to reimplement some parts of React in native OCaml, partly done).

The code looks like this:

let%component hello ~name () =
  let greeting = use (Remote.run (Api.get_greeting ())) in
  let%browser_only () =
    use_effect' (fun () -> Js.log "executes only in browser") [||]
  in
  jsx.div [| textf "%s, %s!" greeting name |]

So this code compiles separately for browser (using melange) and separately for server (using ocamlopt). There are few interesting things here to note:

1. %browser_only form allows to mark parts of the code (only functions, evals and lazy values) to be executed in browser only (no support from the type system here, at least yet)

2. Such code might depend on libs which have different implementations for browser and for native. In this case I have Remote and Api modules doing different things.

Now to the "problems" I have with the build system config (quoting b/c I can workaround them but the workarounds are very verbose).

For the code above I have roughly (omiiting some details first) this scheme:

(library
 (name example)
 (libraries react_browser)
 (modes melange)
 (preprocess (pps react_ppx)))

(subdir
 native
 (library
  (name example_native)
  (modes native)
  (libraries react_server)
  (preprocess (pps react_ppx -native)))
 (copy_files#
  (files ../**[!.pp].ml)))

As you can see I'm using (subdir) stanza to:

1. define a lib for native with a different name, modes, preprocess and libraries.
2. copy .ml sources from the parent dir

Now this doesn't look so bad... but given that the lib names are different for native vs melange libs, when the dependency graph grows it becomes harder to specify dependency and make the code compile.

The reason is not only that I need to specify different libs as dependencies for browser vs native libs but also need to expose different libs under the same name so the same code compiles for both cases. So the final build config looks like this.

First original lib for browser but notice I've added example_deps lib as dependency and then opened its root module in flags:

(library
 (name example)
 (libraries react_browser example_deps)
 (flags :standard -open Example_deps)
 (modes melange))

Now example_deps bundles all example deps and exposes them under "normalized" names in its root module:

(subdir
 deps
 (library
  (name example_deps)
  (modes melange)
  (libraries dep1_browser dep2_browser))
 (rule
  (target example_deps.ml)
  (action
   (with-stdout-to
    %{target}
    (echo
     "module Dep1 = Dep1_browser module Dep2 = Dep2_browser")))))

Now the same is done for native:

(subdir
 native
 (library
  (name example_native)
  (modes native)
  (libraries react_server example_native_deps)
  (flags :standard -open Example_native_deps))
 (copy_files#
  (files ../**[!.pp].ml))
 (subdir
  deps
  (library
   (name example_native_deps)
   (modes native)
   (libraries dep1_native dep2_native))
  (rule
   (target example_native_deps.ml)
   (action
    (with-stdout-to
     %{target}
     (echo
      "module Dep1 = Dep1_native module Dep2 = Dep2_native"))))))

So to recap, the problems I see here:

1. Need to define separate libs for native vs melange builds
2. Need to supply separate preprocess configs to those libs (I think it's ok to be able to do that as native vs melange libs might require completely different ppx but in my case the same ppx can work in two modes and I'd prefer if dune's melange mode had a separate ppx rules so it can communicate to ppx that it runs for melange mode, thus I don't have to pass flags to ppx)
3. Need to create _deps libs to expose dependencies under proper names.

I wonder if there's a better way to solve this?

Why not dune's virtual libs? I've tried them but they have a limitation that the implementations of virtual libs cannot expose more than the virtual libs’ .mli define.

That means I cannot set different type equalities in a proper way for — for example if I have a vlib promise, then promise-server impl to have type 'a promise = 'a Lwt.t and promise-browser to have type 'a promise = 'a Js.Promise.t. I cannot do that. Even more I cannot have converters from one promise type to another as I cannot expose more than vlib defines (and vlib cannot define more support by both native and melange).

Link to the current WIP -
https://github.com/andreypopp/react_of_ocaml/blob/data/example/browser/dune

[davesnx]

I have done exact same steps under server-reason-react, exact same ppx for removing server code (browser_only) as well! but with a few details different:

• I have different ppxes for melange/ocamlopt: reason-react-ppx and server-reason-react-ppx
• For promises (or other libraries) I used exact same interfaces (aantron/promises, this is time consuming and would love virtual libs to solve this

Some things that we might need to consider

• I have a port of Belt that works on native (this could eventually be upstreamed)
• A ppx that does pipe_first
• A ppx that implements a minimum version of regex (using pcre)
• A patched Js module that works and is half implemented (there are some parts that are hard/impossible to do) which is alright at this point.
• I avoided using melange.ppx on the native side, since it will try to link to C code (when it's mean to be JS bindings).
• Unicode hasn't been tackled yet under server-reason-react, would be nice to fix.

The decent result for now, has been copy_files into all levels of dependencies, so any library that needs to be universal will eventually have 2 separate versions. This is done since even when using browser_only the code needs to be compiled anyway, so stubs with the same interface are helpful. That can be a problem. For example, most of the code I'm migrating over use Js stuff which is currently implemented but some parts might be missing (currently crashing at runtime), for the opposite Belt works great.

[jchavarri]

That means I cannot set different type equalities in a proper way for — for example if I have a vlib promise, then promise-server impl to have type 'a promise = 'a Lwt.t and promise-browser to have type 'a promise = 'a Js.Promise.t. I cannot do that.

I ran into this issue while trying vlibs when migrating Ahrefs monorepo to Melange. One solution is to make the type abstract, but this comes with its own set of downsides, and makes some things just impossible.

For promises (or other libraries) I used exact same interfaces (aantron/promises, this is time consuming and would love virtual libs to solve this

If the interfaces are exactly the same, it should be possible to create a virtual library out of it, no? Although worth trying, I believe that virtual libs would not be much cleaner than the current approach using two separate dependency trees: with virtual libraries, one would need to define three libraries: the virtual one, plus the two implementations, then choose the implementations from melange.emit or executable.

Turning the discussion around: what would be the "ideal" solution for this kind of setup? I imagine having melange.emit and executable share the same set of libraries. But then, how would one implement conditional compilation for cases when we need to differ between both platforms?

[andreypopp]

Turning the discussion around: what would be the "ideal" solution for this kind of setup? I imagine having melange.emit and executable share the same set of libraries. But then, how would one implement conditional compilation for cases when we need to differ between both platforms?

So the solution, I think, which is going to work is to introduce a (melange.library) stanza which mimics the (library) configuration but:

• (melange.library) will form a separate library namespace so it's going to be possible to have same named (melange.library) and (library) in the same workspace
• (melange.library) (and (melange.emit)) will resolve its dependencies (specified by (libraries ...)) within that new melange namespace
• (melange.library) will have its own ppx pipeline (so we can have ppx which work both with native and melange)

Such (melange.library) is going to be used for Melange-only libs which depend on Melange FFI / Js specific APIs.

Then we can re-introduce (library (modes :standard melange)) but this time it will work differently — it'll produce (melange.library) automatically for melange mode, e.g.:

(library (name app) (modes :standard melange) (libraries dep))

will be the same as writing:

(library (name app) (modes :standard) (libraries dep))
(melange.library (name app) (modes melange) (libraries dep))

(note that for native dep will be resolved among libraries and for melange — among melange libraries)

Such (library) build configuration will be used for code which doesn't use melange FFI/JS specific APIs (so can be compiled both with ocamlopt and melc). In this case the expectations is that all FFIs/low level details are sorted out in dependencies (which will be resolved to different libs for native vs melange).

Example:

(melange.library
  (name react) ; produces a wrapped lib with top level module React
  (description "bindings to React.js"))

(library
  (name react) ; produces a wrapped lib with top level module React
  (description "native implementation of React SSR"))

(library
  (name my_app)
  (modes native melange)
  (libraries react) ; for native will resolve to library "react", for melange - to melange.library "react"
  (preprocess (pps react.ppx))) ; react.ppx knows when it runs for native vs for melange

---

The minimum viable alternative I'm trying now is to just allow to specify a name for the top level module for wrapped libs (dune patch here).

In this case I still have to duplicate libraries definitions and use copy_files# but at least I don't need to add a library just to wrap and rename dependencies accordingly:

(melange.library
  (name react_browser)
  (wrapped (as react)) ; produces a wrapped lib with top level module React
  (description "bindings to React.js"))

(library
  (name react_server)
  (wrapped (as react)) ; produces a wrapped lib with top level module React
  (description "native implementation of React SSR"))

(library
  (name my_app_server)
  (modes native)
  (libraries react_server)
  (preprocess (pps react.ppx -server)))
(subdir browser
  (library
    (name my_app_browser)
    (modes melange)
    (libraries react_browser)
    (preprocess (pps react.ppx -browser)))
  (copy_files# "../*.ml"))
)

[jchavarri]

Thanks for the explanation and examples!

will be the same as writing:

(library (name app) (modes :standard) (libraries dep))
(melange.library (name app) (modes :standard) (libraries dep))

I think you meant (modes melange) in the last line.

In terms of implementation, would we need a new melange.library stanza? If it is equivalent to using modes maybe we can leave it like that.

Leaving aside how this is expressed in terms of stanzas, trying to summarize the proposal, which would consist in:

1. Get Melange its own library namespace
2. Get Melange its own ppx pipeline

I think one more thing should be added, which is lift the limitation that one module can belong to a single library, at least for modules that belong to a melange library and a native library. That would allow to get rid of the copy_files stanzas. (Maybe this is automatically achieved with 1. above).

(melange.library) will have its own ppx pipeline (so we can have ppx which work both with native and melange)

Can you elaborate on how this would make things better than the current approach? I think one would still need two different configs, like (preprocess (pps react_ppx)) and (preprocess (pps react_ppx -native)) above.

The minimum viable alternative I'm trying now is to just allow to specify a name for the top level module for wrapped libs (dune patch here).

This is pretty useful on its own. Maybe can be submitted / suggested upstream?

[andreypopp]

I think you meant (modes melange) in the last line.

Yes, fixed.

In terms of implementation, would we need a new melange.library stanza? If it is equivalent to using modes maybe we can leave it like that.

I've proposed a new stanza because we need a way to produce the lib with the same name, same top level module from same sources.

Then I've proposed to make (modes melange) to implicitly define (melange.library) with the same config — this actually maybe confusing and we shouldn't do that. Instead adding (melange.library (from library-name)) form or something like that to produce melange.library out of library definition.

(melange.library) will have its own ppx pipeline (so we can have ppx which work both with native and melange)

Can you elaborate on how this would make things better than the current approach? I think one would still need two different configs, like (preprocess (pps react_ppx)) and (preprocess (pps react_ppx -native)) above.

If melange.library will have its own ppx pipeline then we can communicate to ppx that it is being run for melange (env var or some other mechanism), so no special CLI argument is needed to set explicitly.

[actionshrimp]

I was just about to ask a similar question, but this thread describes my current challenge with migrating to dune integrated melange more or less exactly 😄

Previously we had a shared 'Shared_util' module containing things like:

(* src-shared-util/shared_util.ml, as part of a lib called Shared_util_lib *)
module String_map = struct
  include CCMap.Make(CCString)
end

(* src-shared-util-bs/shared_util.ml *)
module String_map = struct
  include Belt.Set.String
  
  (* e.g. matching the `add` sig of CCMap where 
      the args are flipped so the two interfaces are the same *)
  let add k v t = Belt.Map.String.set t k v
  ...
end

This allowed us to define shared code modules using the Util.String_map interface, but code which ran only on the front or back end could still use Belt.Set.String.* or CCMap.* fns as felt natural. With virtual libs it sounds like we would run into the restrictions mentioned above.

We'd hooked this up with this in the dune file for shared code libs for the native build using:

(flags (:standard -open Shared_util_lib))

and adding only src-shared-util-bs to the paths in bsconfig.json, which made the Shared_util module available on that side.

So it feels like we'd essentially created the two 'namespaces' using the two build systems as a bit of a hack. It sounds like the proposed melange.library solution would work perfectly here.

One issue we had with this previous setup was editor tooling - each file had to be assigned to either the frontend realm or the backend realm for things like jumping to definition of these dual-implemention libs in a monorepo setup - I guess we would default shared code to jump to definition of functions in the 'native' impl? Not really a huge issue in practice but thought I would mention it.

[jchavarri]

I've tried them but they have a limitation that the implementations of virtual libs cannot expose more than the virtual libs’ .mli define.

@andreypopp have you tried this workaround? ocaml/dune#5997 (comment)

[andreypopp]

Unwrapped libraries is what I've ended up using, yes. But without virtual libraries, as I need to compile same code twice (as I'm applying different preprocessors to server vs browser builds).

[jchavarri]

I have been thinking lately about ways to fix the issues @andreypopp brought up in this discussion, and @davesnx has been also experiencing. My goal was to fix them without the need to add new stanzas to Dune, which imo has some downsides as it makes Dune implementation more complex, and has some costs in documentation and maintenance that we'd have to assume.

After some experiments, I realized that maybe leveraging Dune contexts through the library enabled_if field could help solve the problems. The idea would be to have OCaml libraries be defined in the default Dune context, and Melange libraries be defined in a newly created melange context.

This would allow using same library names across both platforms (wrapped), as well as avoid having to use copy_files to have shared modules, while keeping specific ppx or flags configurations on each platform for max flexibility. It should also allow to define shared libraries in the same folder, without having to resort to js, native and shared folders, and also define extra modules that can be platform specific, but still part of the same library.

The main downside is that users would have to add enabled_if field to each library. There might be some performance impact as well, for projects where the majority of libraries use (modes :standard melange), because with 2 contexts Dune has to build all artifacts twice and there's less work reuse.

While testing the dual context idea, I ran into some limitations with Dune, where it currently forbids having two libs with the same name defined in different contexts. But with the help of @rgrinberg (thx!) we managed to figure things out in ocaml/dune#9839 (still unmerged).

I have created a small demo using the version of Dune from that PR in https://github.com/jchavarri/melange-universal-demo. The src/world/dune file shows how it's possible to define the same library world in both native and melange contexts. From this point, it should be easy to use different deps on each side, apply ppxs, etc.

If we want to move forward with this approach, there are some remaining issues to be fixed, besides making sure that that Dune PR is approved and merged:

• Editor integration: currently dune ocaml merlin doesn't seem to find artifacts that are not in the default context, I created a repro in merlin: show behavior with non-default context ocaml/dune#10035, and I have some plans to make a suggestion in the vscode-ocaml tracker so that it allows users to choose from which context the editor / merlin should get the information from
• Errors: at the moment dune shows information about the context where executed rules are running when using display=short, but it doesn't show context information when there are build errors. I think this would be a nice addition as one of the main sources of confusions with "universal" modules is to figure out if some build error is coming from the Melange side or from the native side.

Another point that I haven't thought much about is how publication of packages that use this kind of approach would work.

Please feel free to comment, ask, suggest or explore as you will. As this is all exploratory work, any kind of feedback is super appreciated.

[andreypopp]

This all sounds very good!

The main downside is that users would have to add enabled_if field to each library.

Maybe in the future, when the approach described her has proven to be sound, there could still be a separate melange.library stanza. A full copy of library but which will configured automatically to belong to the "default" melange context?

Another point that I haven't thought much about is how publication of packages that use this kind of approach would work.

For "universal" code, I'm thinking it should be possible to publish several packages from the same sources, e.g. pkg and melange-pkg. The only difference between them would be different dependencies and a build command which would differ only if --context.