Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature to track multicore compatibility of libraries #6194

Open
gadmm opened this issue Sep 7, 2024 · 10 comments
Open

Feature to track multicore compatibility of libraries #6194

gadmm opened this issue Sep 7, 2024 · 10 comments

Comments

@gadmm
Copy link

gadmm commented Sep 7, 2024

In order to be used in a multicore setting, libraries typically need to be written with multicore in mind. OCaml 4 libraries must be adapted to multicore in various ways (e.g. eliminating reliance on a global lock, adjusting to possible new FFI guidelines, etc.). OCaml 4 libraries should be assumed non-muticore-compatible, unless the authors indicate explicitly that thought and work has been put into ensuring it.

The tooling is currently missing in the ecosystem for keeping track of multicore-readiness of libraries. I propose to discuss this problem here. For instance, there could be some kind of tagging in opam to track library capabilities, with a "multicore" tag that is absent by default.

cc @kayceesrk

@hannesm
Copy link
Member

hannesm commented Sep 7, 2024

Thanks for your constructive proposal, I had a similar question at the opam-repository issue tracker some time ago (and was surprised by the answer) ocaml/opam-repository#25524

@avsm
Copy link
Member

avsm commented Sep 7, 2024

Why was my answer surprising, @hannesm? What I described there still seems broadly correct to me. I think the way to realistically document parallellism-safety isn't at the opam level, but at the odoc level. Given that we now have documentation generation centrally, it's up to library authors to document what locks their OCaml interfaces require -- and nothing is stopping that as far as I can see from a technical perspective.

@dbuenzli
Copy link
Contributor

dbuenzli commented Sep 7, 2024

OCaml 4 libraries should be assumed non-muticore-compatible, unless the authors indicate explicitly that thought and work has been put into ensuring it.

That position sounds extreme to me. In OCaml 4 you already had to mention if your libraries were thread-safe or not (e.g. grepping through my code bases here, here or there).

I would take the opposite stance, which is the one that was adopted upstream. Assume thread-safety and document unsafeties. You can give as much "thought and work", stuff is going to slip through anyways.

@kayceesrk
Copy link

kayceesrk commented Sep 9, 2024

After discussing with @avsm elsewhere, I've come to a similar conclusion @dbuenzli. We haven't done anything special for thread safety in OCaml 4. I do not see a strong reason to handle things differently with parallelism.

@dbuenzli
Copy link
Contributor

dbuenzli commented Sep 9, 2024

But that's not to say I would not be interested by:

The tooling is currently missing in the ecosystem for keeping track of multicore-readiness of libraries

:–)

@avsm
Copy link
Member

avsm commented Sep 9, 2024

Yeah, it would be really useful to have some Merlin or LSP support for querying parallel-safety docstrings/alerts from compiled dependencies in the environment in a more structured way. I haven't had a chance to think through what that might look like concretely, though.

@dbuenzli
Copy link
Contributor

dbuenzli commented Sep 9, 2024

Why not.

From a library sanitization perspective I think I would rather like a simple lint tool that looks up for non-atomic top-level refs, parallel unsafe use of Format (likely to be my largest issue), etc.

Speaking of parallel unsafe use of Format I noticed yesterday that 10 years ago I removed to_string functions from Gg which were using the pp functions with Format.str_formatter on the ground that they were not thread-safe…

@avsm
Copy link
Member

avsm commented Sep 9, 2024

Yeah, Format's been by far the biggest culprit for me when using (the amazing) TSAN variant to do development.

@dra27
Copy link
Member

dra27 commented Sep 10, 2024

opam's tags field could be using for this, but I'm not sure that there's anything which needs changing in opam itself to be able to do that - is there anything you have in mind that would be specifically wanted in the opam client itself?

@dra27
Copy link
Member

dra27 commented Sep 10, 2024

Unsurprisingly, I come quite strongly down on what we did with 5.0 - that is, not blocking single-domain-only-but-not-necessarily-checked-for-multicore packages from working on 5.0 by default. Fracturing the existing single-core ecosystem as part of 5.0 would have been a big mistake, and a vast amount of effort was poured into trying not to do that.

It's easy in opam-repository to mark packages which have not been tested for parallelism and then set-up an opam switch which forbids these packages. However, that brings a lot of bureaucracy for new releases of still-untested packages, so I'm, not sure it would be that useful. Sadly, there isn't as easy a way to do the converse - have a switch which can be set to only install packages which have been tested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants