Build stdlibs with rewatch or bsb and integrate Core

[cknitt]

Before integrating Core into the compiler repo, we should sort out the standard library build procedure. We should get rid of the ninja scripts and build everything with rewatch (or maybe even still bsb at first).

It turns out to be difficult to set this up because of (among other things) cyclic dependencies that need to be resolved. A lot things need to be shuffled around. It would be easier to iterate on this outside of the compiler repo IMHO.

My proposal is therefore to create a separate repo rescript-stdlib to work on a monorepo structure where all standard libs can be built with a single build command and without relying on any pre-existing artifacts in lib/ocaml. For this, we do not even need the v12 compiler, it should work fine with v11, too. This rescript-stdlib repo would be a temporary one, allowing us to iterate quickly on the standard library build infrastructure. Once we everything is working, it can be taken over into the compiler repo.

A rough plan could look like this:

• 1. Re-add the -nostdlib option, see Re-add -nostdlib option and make it actually do something #6824.
• Creeate separate repo for the work on the standard libraries
• Add a "stdlib mini" (the absolutely essential stuff to get the rest to build), runtime (caml_*.res) and belt and get everything to compile with "-nostdlib -nopervasives" in curried mode
• Switch to uncurried mode and get runtime to compile
• Add Core
• Add (part of?) the old Js namespace as a compatibility package
• Add (part of?) the old OCaml 4.06 stdlib as a compatibility package
• Clean up

[cknitt]

I think the resulting standard library packages could then even be published as separate npm packages (see #6183) independent of a v12 compiler release. They could be used/tested in any ReScript project that sets the -nostdlib compiler option and adds them as dependencies.

[cknitt]

@zth @cristianoc @jfrolich @rolandpeelen what do you think?

[jfrolich]

I think it's probably easier to get everything compiled the same way using rewatch in the compiler repo, and refactor from there. This allows us to have the two options side-by-side. Feels like the above approach takes too many steps at once. Sorry I didn't get to finalizing the PR where we compile the stdlib with rewatch but I don't have the impression that it's hard. We just have to determine if we keep the cyclic deps, or refactor them out, we went for the last and that seemed to work. Just need to spend some time to make sure everything is properly compiling.

[cristianoc]

I think the stdlib mini would be pretty nice to have, as it isolates all the special cases. If I understand correctly, everything else can be built user-level on top of it without tricks/surprises. So it seems doable and independent from the details of the other steps.

[cknitt]

Feels like the above approach takes too many steps at once.

Thanks for the feedback. My feeling was the opposite, that doing it in the compiler would mean taking too many steps at once. IMHO it is hard to refactor all this until it has the proper structure while still keeping the old build process working. (Unless we copy everything to a new dir in the compiler repo, but then we may as well put it into a separate repo temporarily to be able to iterate quickly, like we did for Core itself.)

I think the stdlib mini would be pretty nice to have, as it isolates all the special cases. If I understand correctly, everything else can be built user-level on top of it without tricks/surprises. So it seems doable and independent from the details of the other steps.

Definitely! Everything can be built from scratch in user level, starting with the stdlib mini. (Even with bsb, and even with the v11 compiler with -nostdlib). I'll put up a repo tomorrow with what I have now (basically step 1-4).

[cknitt]

Ok, so here is my repo: https://github.com/cknitt/rescript-stdlib

It uses compiler v11.1.3-rc.1 with -nostdlib -nopervasives. This ensures that everything is indeed built from scratch.

You can clone it and run npx rewatch to build with rewatch, or npx rescript to build with bsb.

It will then build in one go:

• a mini stdlib (not quite mini enough yet, I had to add some part of Js, too)
• belt, depending only on the mini stdlib
• runtime (caml_*), depending only on the mini stdlib

This is based on #6772, but further modified to get it to build in this setup.

Next up would be adding Core.

[cknitt]

Moved my repo to https://github.com/rescript-lang/experimental-rescript-stdlib-build

I have also added Core in the meantime and adapted it a bit to fit into the structure.

[cknitt]

Some questions:

1. I want to be able to build runtime, belt, Js, and the OCaml stdlib without relying on Core being present (and am already doing so now for the first two). A consequence of this is that common type definitions cannot have Core as their source. I have collected these type definitions in https://github.com/rescript-lang/experimental-rescript-stdlib-build/blob/main/stdlib-mini/src/js_types.res for now. Maybe at least some of these should actually be builtin types?
2. What we need to be able to rely on though when compiling ReScript code is runtime (caml_*) to be present, see Compiler should not rely on Js or Belt module being present #6836. Therefore the question is, shouldn't the basic types and the "stdlib mini" actually be part of the runtime as caml_types and caml_pervasives or something like that?

What do you think @cristianoc @zth?

[cknitt]

Done in #7108 and #7116 (with a simpler approach, no monorepo like in https://github.com/rescript-lang/experimental-rescript-stdlib-build).