Skip to content

The State of the PPX Transition

Jump to bottom
Sonja Heinze edited this page Jul 26, 2021 · 4 revisions

Issues ppxlib was Facing

The issues we were facing and are tackling with our current plan are the following:

  1. To keep the PPX ecosystem cross-compiler compatible a) ppxlib was handling and abstracting away parts of the unstable compiler-libs API; b) OMP/ppxlib was keeping the AST migration information up-to-date. That situation required us and the compiler developers to coordinate for each compiler release in order to guarantee that the new compiler could be used in projects with PPXs.
  2. To guarantee new feature support, ppxlib needs to bump the ppxlib AST to the newest version, which implies a breaking change. That was an issue for a homogeneous and up-to-date PPX ecosystem.
  3. Not all PPXs had migrated from OMP to ppxlib. That was also an issue for a homogeneous and up-to-date PPX ecosystem.

Our Three-Part, Multi-Step Plan and its Current State

Part One: Implementing Astlib in Six Steps

The first part works towards continuous cross-compiler compatibility (Issue 1. above) while making the situation of still having PPXs based on OMP (Issue 3. above) even more of a problem. It consists of implementing an interface module called Astlib between ppxlib and the compiler, then upstreaming it to the compiler. Here are its six steps:

  1. Moving the OMP driver and other OMP features from OMP to ppxlib. That was done in August 2020, and it introduced OMP2. This was the start of having the PPX ecosystem split into the two incompatible worlds of OMP1 PPXs on one hand and ppxlib PPXs on the other hand.
  2. Reducing the compiler-libs usage in ppxlib as much as possible to keep Astlib minimal. That has been completed over time.
  3. Implementing a ppxlib internal version of Astlib as a layer between ppxlib and the compiler. This is currently (July 2021) the state on our main branch, but it hasn't been released yet.
  4. Working on an upstream version of Astlib for the compiler and adapting the tooling to update the AST migration logic to the compiler. That's next on our ToDo list.
  5. Discussing the upstream version and the workflow to update the AST migrations with the compiler developers, with the aim to upstream it.
  6. Simultaneously to upstreaming Astlib, implementing the logic to use our internal Astlib on old compilers and the upstream Astlib on new compilers into ppxlib.

Part Two: Porting PPXs to Re-unite the Ecosystem

Moving the driver from OMP to ppxlib (Step 1. above) made the fact that some PPXs were still based on OMP (Issue 3. above) even more of a problem since it split the PPX world in two. For some PPX developers, the reason to avoid porting their PPXs to ppxlib was that ppxlib depended on base and stdio, so we decided to tackle this situation by three means:

  • Dropping the base and the stdio dependencies, which was done in August 2020.
  • Porting and reviewing some of the most important PPXs ourselves. So far we've ported js_of_ocaml, bisect_ppx, and tyxml with the help of the respective maintainers, and we've also reviewed several ports.
  • Spreading the word about the need to port PPXs and asking for help. A lot of people have joined the effort!

In summer 2020, we made a non-exhaustive list of PPXs that needed to be ported. By now (in July 2021), all PPXs on that list have been ported with the only exception of ppx_import, which has a port, but it's not quite ready to be merged.

You can find the full list of opam packages that are still stuck in the OMP1 world by filtering for them in opam's health check pipeline. However, notice that is a generated list, so it also contains libraries that intrinsically form part of the OMP1 ecosystem (such as ppx_tools_versioned), PPXs that have already been ported but haven't relesed their port on opam yet, deprecated PPXs that aren't marked as deprecated, and PPXs that only transitively depend on OMP1.

Part Three: Sending Patch PRs when Bumping the AST

Our current plan doesn't provide a solution for the problem that a few PPXs break whenever we bump the ppxlib AST (Issue 2. above), but it does make handling the problem more efficient and takes away the burden from the PPX developers. Since the AST bump to 4.10, whenever we bump the AST, we create a workspace with all PPXs fulfilling a certain standard, fix the PPXs that break all at once, and open patch PRs. We call that workspace the ppx-universe. You can find a ppx-universe snapshot on our repo.

Our plan is to make the tooling that creates and interacts with the workspace more sophisticated. We use opam-monorepo as main tool. For the rest, we are currently using a ppxlib-specific bash script.

Clone this wiki locally