opam proposes to install packages without explanation

[ElectreAAS]

Hello and a sincere thank you to all opam maintainers for everything you do

Description of the issue

I have seen weird behaviours from opam upgrade: it will sometimes (reproduction notes below) propose installing new packages without explaining why.

$ opam install a
The following actions will be performed:
=== install 5 packages
  ∗ b                   1.0.0     [required by a]
  ∗ c                   1.0.0     [required by a]
  ∗ d                   1.0.0     [required by b]
  ∗ e                   1.0.0
  ∗ a                   1.0.0

Proceed with ∗ 5 installations? [y/n] y

In this forced example, should the package e really be proposed for installation?
In case I say yes, and then opam remove e, it will get removed without proposing to recompile anything, the way it would if it was an optional dependency.
It is not listed as a root in opam list --roots, however it will not get removed with a opam remove -a. In the same vein, opam why e won't show anything...

Reproduction

After a lot of fiddling around with unstable switches, my example moved from mirage-no-solo5 to ounit and then to ocaml-index. I think it's one of the smallest examples possible (starting from an almost empty switch):

> opam install ocaml-lsp-server
The following actions will be performed:
=== install 18 packages
  ∗ chrome-trace        3.16.0    [required by ocaml-lsp-server]
  ∗ dune-rpc            3.16.0    [required by ocaml-lsp-server]
  ∗ dyn                 3.16.0    [required by ocaml-lsp-server]
  ∗ fiber               3.7.0     [required by ocaml-lsp-server]
  ∗ jsonrpc             1.19.0    [required by ocaml-lsp-server]
  ∗ lsp                 1.19.0    [required by ocaml-lsp-server]
  ∗ merlin-lib          5.2.1-502 [required by ocaml-lsp-server]
  ∗ ocaml-index         1.1
  ∗ ocaml-lsp-server    1.19.0
  ∗ ocamlc-loc          3.16.0    [required by ocaml-lsp-server]
  ∗ ocamlformat-rpc-lib 0.26.2    [required by ocaml-lsp-server]
  ∗ ordering            3.16.0    [required by ocaml-lsp-server]
  ∗ pp                  1.2.0     [required by ocaml-lsp-server]
  ∗ ppx_yojson_conv_lib v0.17.0   [required by ocaml-lsp-server]
  ∗ spawn               v0.15.1   [required by ocaml-lsp-server]
  ∗ stdune              3.16.0    [required by ocaml-lsp-server]
  ∗ xdg                 3.16.0    [required by ocaml-lsp-server]
  ∗ yojson              2.2.2     [required by ocaml-lsp-server]

Proceed with ∗ 18 installations? [y/n] n

Here ocaml-index isn't justified like the others, and removing it doesn't cause any recompilation.
At this point, opam upgrade will propose installing it, which looks weird. opam remove -a will also propose installing it, which looks even weirder.
opam why ocaml-index doesn't explain anything.

From fiddling with a few packages earlier, I understand it's got something to do with post dependencies, whatever that is. This makes me think that this is the root cause of the issue raised in #6109.

[kit-ty-kate]

Thanks for the comprehensive report!

The reason post dependencies are not ignored when looking at the reason calculation is that we use the action graph to compute the reason since we have it on hand. However post dependencies, per their nature, have to detach themselves from their parent node and avoid cycles. One way to fix this could be to add "ghost edge" (i'll have a look at the different options in ocamlgraph to see if there is something there already) which are not used in normal use but only used during the reason calculation.

For the issue in opam why, the cause is different and the fix should be a bit simpler (although it would involve some sort of cycle detection so it might not be that simple)

Here ocaml-index isn't justified like the others, and removing it doesn't cause any recompilation. At this point, opam upgrade will propose installing it, which looks weird. opam remove -a will also propose installing it, which looks even weirder. opam why ocaml-index doesn't explain anything.

mmh, i was able to reproduce this. I think the issue here is that you're not supposed to be able to remove it without removing the parent package but the information that there is a package depending on it is lost somewhere somehow.