feat!: decide on explicit requirements first

[baszalmstra]

Fixes #60

An implementation of the ideas discussed in #60. This PR opts to decide on requirements from the root over any requirement that the root does not directly depend on. This has the effect that the highest versions of requirements from the root are most likely picked instead of them being constrained by transitive dependencies.

The test_explicit_root_requirements test shows this in action. With the code before this PR the test would result in

a=1 # root requirement
b=2	# transitive requirement
c=1 # root requirement

whereas now the result is:

a=1 # root requirement
b=1 # transitive requirement
c=5 # root requirement

Note that the root requirements are optimized at the cost of the transitive requirement.

• I still have to test what the impact on performance is for this PR.

[baszalmstra]

I used the attached notebook to generate the following images to get a better idea what the performance impact of this method:

This shows that when running on a large set of examples with single requirements there is no real difference with the code from main.

This shows the differences in solve times for individual packages:

I conclude that at least for solving for specific solvables there is no noticable performance difference. 👍

But it would be good to profile this when solving for multiple requirements at the same time which will likely have a larger impact. Ill try that later.

[baszalmstra]

I modified the benchmark program to select 1000 (deterministic) random set of requirements from the snapshot. I think this provides a better overview of what is going on.

These results show that on average solving becomes slower and although there are some cases that become (significantly) faster overall the solver becomes slightly slower (and in some cases significantly so).

I am a little unsure what to do. These changes also result in "better" outcomes where the root requirements are optimized. So I'm inclined to merge this.

Thoughts? @jjerphan @wolfv

[jjerphan]

Hi @baszalmstra,

Thank you for posting those results. Let me block some time this week so that I can review it.

[jjerphan]

I have not dived into the implementation, I am just looking at the second results given in #61 (comment).

I am a little unsure what to do. These changes also result in "better" outcomes where the root requirements are optimized. So I'm inclined to merge this.

I am also unsure about what to do here. Functionally, this is interesting, but do we want to have this new behavior be the default if some users know regression? Could this be introduced as a new alternative strategy?

[baszalmstra]

Yeah, perhaps we can abstract the decision making logic and allow users to inject certain strategies.

[baszalmstra]

We ran into this again today with pixi.

If you solve for python ~=3.10.0 and pytorch >2,<3 you get the CPU version of pytorch regardless of whether CUDA is available. When you change the pytorch requirement to 2.3.1 you get the CUDA enabled version.

The reason this happens is that there are lots of pytorch variants available so the solver will decide on it very late. Before it decides on pytorch it will decide on an openmp implementation and it will pick one that makes any CUDA enabled pytorch version incompatible. This is not what we want and it's also very hard for a user to make it do the right thing without messing with build strings.

I am therefore inclined to fix-up this PR and merge it, even though the performance might degrade a little. I am quite confident that once we merge #37 the performance regression introduced here will no longer be an issue anyway.

[jjerphan]

Up to you, @baszalmstra. I will come back to resolvo when I can.