Skip to content

roadmap v1

Jump to bottom
Jeremie Dimino edited this page May 15, 2019 · 1 revision

ppx 1.0.0 roadmap

This document is a work in progress.

This document describes the roadmap for ppx 1.0.0. The goal is to build a new stable foundation for the ppx ecosystem, as currently the ppx stack is complicated and not very resilient to compiler upgrades.

The expectation is that Ppx 1.0.0 will effectively supersede everything else. In particular, all other projects such as ocaml-migrate-parsetree, ppx_deriving, ppx_tools, ppx_tools_versioned and ppxlib. However, the code of ppx is initially forked from ppxlib.

As such, we must prepare ppx to be a solid foundation for the whole ppx ecosystem, and soothe the rough edges that might get in the way of its adoption. The main difficulty is backward compatiblity, and this document describes how we are going to solve this problem.

While all of this happen, we will also communicate the plan and advancement on discuss.ocaml.org so that people who care about ppx can follow what's happening, understand how this work will affect them and what benefit they can expect from it.

If you have any questions or concerns about this plan, please contact Jeremie Dimino.

Backward compatiblity

Currently, ppxlib uses one selected version of the AST, for instance 4.07. When OCaml 4.08 comes out, the following events happen:

  • ocaml-migrate-parsetree learns how to convert between the 4.07 and 4.08 AST. This allows ppxlib to work with OCaml 4.08 and code using ppxlib to build against 4.08, however users of ppx rewriters cannot use the new 4.08 features as the 4.07 AST cannot represent them
  • ppxlib is updated to use the 4.08 AST. This is a breaking change
  • all authors of ppx rewriters have to upgrade to the new version of ppxlib, based on the 4.08 AST. If they are lucky, they don't need to upgrade

There are several issues with this approach:

  • it is a lot of churn for a lot of people
  • we can't test the new versions of OCaml until all of the above has happened

The plan is to arrive in a situation where:

  • ppx builds at all time, even against OCaml trunk
  • existing OCaml files are always parsed and preprocessed the same way, allowing to test OCaml trunk at any time

We plan to achieve this by the introduction of a new Astlib library that will sit between the compiler libraries and ppx.

Astlib

Astlib will be a small connector library between the compiler and ppx. It will provide a minimal stable API for ppx to rely on. The plan is to make it live in the compiler directly. However, in order to make the transition to the new ppx world easier for users, we will backport this library to older compilers.

Astlib will be released as an astlib package but will be developed directly in the github.com/ocaml-ppx/ppx repository, in order to make the development of astlib and ppx easier.

When building the astlib library, we will inspect the current compiler version. If it is a compiler that provides Astlib, then the astlib library will just redirect to compiler library. Otherwise, we will use the implementation embed in the reposiotry.

API of Astlib

Astlib will expose the following functionality:

  • parsers for a few public entry point of the grammar: structure, signature, expression, ...
  • printers for the same entry points
  • a stable AST

The stable AST

The stable AST exposed by Astlib will be a stable AST that is different from the one used inside the compiler. The goal is that a given file parses to exactly the same AST value, not matter the version of the compiler.

This AST will be a more general AST that can represent not only the current AST, but also any future ones. In pratice, we will use a single type t and declare the various types structure, expression, ... as private instance of this type.

Astlib and Ppx

Ppx will use the AST provided by Astlib directly. In addition, it will provide viewers (as in ppx_view viewers) for convenience so that users do not have to deal with the encoding into the stable AST.

When traversing the AST, Astlib will fail when it encounters an encoding it doesn't know about. This will ensure that it doesn't silently ignore parts of a file because it cannot interpret the new features.

Effectively, when the language evolves, the following will happen:

  • a new version of OCaml will be released
  • we will update ppx to handle the new constructors
  • we will make a new release of ppx
  • users will be able to enjoy both the new language features and ppx rewriters

During all this time, existing OCaml files will still be interpreted and preprocessed in exactly the same way they used to.

Ppx 1.0.0

Dependencies

The current dependencies of ppxlib are:

  • dune
  • base
  • stdio
  • ocaml-compiler-libs
  • ocaml-migrate-parsetree
  • ppx_derivers

The dependencies of ppx will be:

  • dune
  • astlib
  • ppx_derivers

Inverting the dependency on ocaml-migrate-parsetree

Instead of ppx depending on ocaml-migrate-parsetree, once ppx is ready we plan to make ocaml-migrate-parsetree depend on ppx. This will allow to mix older ppx rewriters based on ocaml-migrate-parsetree and newer ones based on ppx.

ppx_derivers

This library allows users to mix derivers based on ppx_deriving and derivers based on ppxlib. Users of ppx_deriving will be able to switch to Ppx.Deriving, which offer a similar functionality.

However, to allow mixing derivers based on ppxlib, ppx_deriving and ppx during the transition period, ppx will continue to depend on ppx_derivers for two years.

Dropping the dependency on Base

We will drop the dependency on Base as I believe it would slow down the adoption of ppx. Indeed, so far Base hasn't convinced everybody, and I have seen people who didn't want to depend on ppxlib because of Base.

This will also make things easier for Jane Street developers as currently the situation between Base and ppxlib is a bit complicated. Indeed, Base is developed inside Jane Street, ppxlib is developed on github and ppxlib depend on Base. This causes a very weird workflow for people working on Base and/or ppxlib.

Instead, we will use the OCaml standard library and have a small overlay inside ppxlib to add the few things that are missing, a bit like what is done in dune.

Roadmap

Items under the same bullet can be done in parallel

  • preparation:
    • write github.com/ocaml-ppx/ppx/astlib
    • drop dependency on ocaml-migrate-parsetree and ppx_derivers
    • drop dependency on base and stdio
    • rename ppxlib to ppx in the github.com/ocaml-ppx/ppx repository
  • delete ppx.ast (ast folder in ppxlib) and use astlib instead
  • make ocaml-migrate-parsetree depend ppx and plug the ocaml-migrate-parsetree driver into the ppx one
  • release ppx 1.0.0 and the ocaml-migrate-parsetree 2.0.0

Jane Street will then port all of its ppx rewriters to ppx, and we hope that open source developers will find the new stability guarantee offered by ppx attractive enough to switch to it.

Third party tools such a reason

In addition to ppx rewriters, there are other tools that currently rely on the concrete types of the AST, such as Reason. The plan to accomodate these tools is to provide conversion functions between the version of the AST they are currently using and the stable AST. Then, if possible they should switch and use the stable AST directly.

Clone this wiki locally