Feature to track multicore compatibility of libraries

[gadmm]

In order to be used in a multicore setting, libraries typically need to be written with multicore in mind. OCaml 4 libraries must be adapted to multicore in various ways (e.g. eliminating reliance on a global lock, adjusting to possible new FFI guidelines, etc.). OCaml 4 libraries should be assumed non-muticore-compatible, unless the authors indicate explicitly that thought and work has been put into ensuring it.

The tooling is currently missing in the ecosystem for keeping track of multicore-readiness of libraries. I propose to discuss this problem here. For instance, there could be some kind of tagging in opam to track library capabilities, with a "multicore" tag that is absent by default.

cc @kayceesrk

[hannesm]

Thanks for your constructive proposal, I had a similar question at the opam-repository issue tracker some time ago (and was surprised by the answer) ocaml/opam-repository#25524

[avsm]

Why was my answer surprising, @hannesm? What I described there still seems broadly correct to me. I think the way to realistically document parallellism-safety isn't at the opam level, but at the odoc level. Given that we now have documentation generation centrally, it's up to library authors to document what locks their OCaml interfaces require -- and nothing is stopping that as far as I can see from a technical perspective.

[dbuenzli]

OCaml 4 libraries should be assumed non-muticore-compatible, unless the authors indicate explicitly that thought and work has been put into ensuring it.

That position sounds extreme to me. In OCaml 4 you already had to mention if your libraries were thread-safe or not (e.g. grepping through my code bases here, here or there).

I would take the opposite stance, which is the one that was adopted upstream. Assume thread-safety and document unsafeties. You can give as much "thought and work", stuff is going to slip through anyways.

[kayceesrk]

After discussing with @avsm elsewhere, I've come to a similar conclusion @dbuenzli. We haven't done anything special for thread safety in OCaml 4. I do not see a strong reason to handle things differently with parallelism.

[dbuenzli]

But that's not to say I would not be interested by:

The tooling is currently missing in the ecosystem for keeping track of multicore-readiness of libraries

:–)

[avsm]

Yeah, it would be really useful to have some Merlin or LSP support for querying parallel-safety docstrings/alerts from compiled dependencies in the environment in a more structured way. I haven't had a chance to think through what that might look like concretely, though.

[dbuenzli]

Why not.

From a library sanitization perspective I think I would rather like a simple lint tool that looks up for non-atomic top-level refs, parallel unsafe use of Format (likely to be my largest issue), etc.

Speaking of parallel unsafe use of Format I noticed yesterday that 10 years ago I removed to_string functions from Gg which were using the pp functions with Format.str_formatter on the ground that they were not thread-safe…

[avsm]

Yeah, Format's been by far the biggest culprit for me when using (the amazing) TSAN variant to do development.

[dra27]

opam's tags field could be using for this, but I'm not sure that there's anything which needs changing in opam itself to be able to do that - is there anything you have in mind that would be specifically wanted in the opam client itself?

[dra27]

Unsurprisingly, I come quite strongly down on what we did with 5.0 - that is, not blocking single-domain-only-but-not-necessarily-checked-for-multicore packages from working on 5.0 by default. Fracturing the existing single-core ecosystem as part of 5.0 would have been a big mistake, and a vast amount of effort was poured into trying not to do that.

It's easy in opam-repository to mark packages which have not been tested for parallelism and then set-up an opam switch which forbids these packages. However, that brings a lot of bureaucracy for new releases of still-untested packages, so I'm, not sure it would be that useful. Sadly, there isn't as easy a way to do the converse - have a switch which can be set to only install packages which have been tested.