Source map improvements #1716

vouillon · 2024-10-18T10:18:26Z

I started working on improving the source map output of Wasm_of_ocaml, and got a bit side-tracked...

Many changes to preserve debug information from the bytecode as well as possible and put this information in the source map at appropriate places for the JavaScript debuggers.

compiler/lib/js_output.ml

compiler/lib/js_traverse.ml

compiler/lib/ocaml_compiler.ml

compiler/lib/specialize_js.ml

compiler/tests-compiler/effects_continuations.ml

toplevel/examples/lwt_toplevel/dune

compiler/lib/generate.ml

compiler/lib/parse_bytecode.ml

compiler/lib/js_output.ml

Explain a bit how they are used by the debuggers.

The debuggers do not stop on some statements, like function declarations. So there is no point in outputting some debug information there.

The best position is at the beginning of the expression after the `=`.

There are some locations which are marked as ghost locations, but which actually make sense.

compiler/lib/js_output.ml

- Do not repeat the output if the location has not changed - Start a new source map mapping before outputing the debug info as a comment (we need to start a mapping right after a return statement, moving it after a comment would not work) - Outputting an identifier name will not change the location (there is no point in doing that)

This is simpler than to associate a location to all instructions. Also, this avoid the issue of locations getting lost because an instruction gets optimized away.

In particular, no longer track the location of variables. This may make sense during code generation, though: the location of a statement should be the earliest location of the effectful expressions it contains (if there is any).

If the first block of the function starts with an event, we use the event's location. This is useful in case of tail-calls, where some code is generated before this first block. If we don't have an event, we set the initial location to unkown to prevent previous locations to bleed into the function body.

There is some code at the beginnning of exception handlers for which we have no debugging information.

Use a monad to abstract the operations on the expression queue and the effect information.

Statements will get the earliest location of the expressions performing a call it contains, if there is any. Then, for an expression `if (e) {...}`, we don't move first to the location of the conditional before going back to the location within `e` which might be strictly before. When queueing expressions, we ignore their locations if they don't perform a call; when flushed, we use the current location. This again prevent locations to be reordered.

The generated code is also hidden by using a dummy file.

Firefox assumes that a mapping stops at the end of a line, which is inconvenient. When this happens, we repeat the mapping on the next line.

Outputting this source map would fail on 32-bit architectures since its size, once Base64-encoded, would be over the string length limit of 16 MiB.

…tion

…ore robust

We ignore any event at the end of a block. This happens when the block ends with a return or a branch. In case of a return, it's a tail call, so we don't have an event anyway. For a branch, the target block will start with an event which is would take precedence anyway.

When pretty-printing, there is no ambiguity since we always have a space after the return statement. In compact mode, use a newline instead.

CHANGES: ## Features/Changes * Misc: update testsuite to OCaml 5.2 * Misc: CI uses opam.2.2 and no longer use sunset repo * Misc: yojson is no longer optional * Misc: reduce the diff with the wasm_of_ocaml fork * Misc: finalize support for OCaml 5.3 * Compiler: speedup global_flow/global_deadcode pass on large bytecode * Compiler: improved global dead code elimination (ocsigen/js_of_ocaml#2206) * Compiler: speedup json parsing, relying on Yojson.Raw (ocsigen/js_of_ocaml#1640) * Compiler: Decode sourcemap mappings only when necessary (ocsigen/js_of_ocaml#1664) * Compiler: mark [TextEncoder] as reserved * Compiler: add support for the Wasm backend in parts of the pipeline, in prevision for the merge of wasm_of_ocaml * Compiler: introduce a Targetint module that follows the semantic of the backend (js or wasm) * Compiler: warn on joo_global_object * Compiler: revisit static env handling (ocsigen/js_of_ocaml#1708) * Compiler: Emit index source_map to avoid changing mappings (ocsigen/js_of_ocaml#1714, ocsigen/js_of_ocaml#1715) * Compiler: improved source map generation (ocsigen/js_of_ocaml#1716) * Runtime: change Sys.os_type on windows (Cygwin -> Win32) * Runtime: backtraces are really expensive, they need to be explicitly requested at compile time (--enable with-js-error) or at startup (OCAMLRUNPARAM=b=1) * Runtime: allow dynlink of precompiled js with separate compilation (ocsigen/js_of_ocaml#1676) * Runtime: reimplement the runtime of weak and ephemeron (ocsigen/js_of_ocaml#1707) * Lib: Modify Typed_array API for compatibility with WebAssembly * Lib: add details element and toggle event (ocsigen/js_of_ocaml#1728) * Toplevel: no longer set globals for toplevel initialization * Runtime: precompute constants used in `caml_lxm_next` (ocsigen/js_of_ocaml#1730) * Runtime: cleanup runtime ## Bug fixes * Runtime: fix parsing of unsigned integers (0u2147483648) (ocsigen/js_of_ocaml#1633, ocsigen/js_of_ocaml#1666) * Runtime: fix incorrect pos_in after unmarshalling * Runtime: make float_of_string stricter (ocsigen/js_of_ocaml#1609) * Toplevel: fix missing primitives with separate compilation * Compiler: fix link of packed modules with separate compilation * Compiler: Fixed the static evaluation of some equalities (ocsigen/js_of_ocaml#1659) * Compiler: fix global analysis bug (subsumes ocsigen/js_of_ocaml#1556)

vouillon requested a review from hhugo October 18, 2024 10:18

vouillon force-pushed the source-maps branch 4 times, most recently from 2913f20 to 4e10e29 Compare October 18, 2024 14:30