Poetry version solving fails

[jmwilkinson]

Description

Trying to version solve with very general requirements:

package-a deps:

[tool.poetry.dependencies]                                                                             |
python = "~3"  
PyYAML="^6.0.1"
package-b="^1.1.4"

package-b was built using setup tools, and does not have a pyproject yaml file.
deps:

PyYAML>="6.0.0"
python=">=3.7"

Nets the error:

The current project's supported Python range (>=3,<4) is not compatible with some of the required packages Python requirement:
  - pyyaml requires Python >=3.6, so it will not be satisfied for Python >=3,<3.6

Because pyyaml (6.0.1) requires Python >=3.6
 and no versions of pyyaml match >6.0.1,<7.0.0, pyyaml is forbidden.
So, because package-a depends on PyYAML (^6.0.1), version solving failed.

The error message very obviously states the supported version range is >=3,<4, but then fails claiming >=3,<3.6 is outside this range.

This happens for quite a few packages.

Workarounds

None I've discovered so far.

Poetry Installation Method

pipx

Operating System

MacOS

Poetry Version

Poetry (version 1.8.3)

Poetry Configuration

cache-dir = # redacted
certificates.artifactory.cert = true
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
keyring.enabled = true
# redacted, private repo
solver.lazy-wheel = true
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # redacted
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true

Python Sysconfig

No response

Example pyproject.toml

No response

Poetry Runtime Logs

I can attach the relevant solve logic failure:


Found 0 compatible packages for package-b
   1: fact: no versions of pyyaml match >6.0.1,<7.0.0
   1: conflict: no versions of pyyaml match >6.0.1,<7.0.0
   1: derived: not PyYAML (>6.0.1,<7.0.0)
   1: conflict: pyyaml (6.0.1) requires Python >=3.6
   1: ! pyyaml (==6.0.1) is partially satisfied by not PyYAML (>6.0.1,<7.0.0)
   1: ! which is caused by "no versions of pyyaml match >6.0.1,<7.0.0"
   1: ! thus: pyyaml is forbidden
   1: ! PyYAML (>=6.0.1,<7.0.0) is satisfied by PyYAML (>=6.0.1,<7.0.0)
   1: ! which is caused by "package-a depends on PyYAML (^6.0.1)"
   1: ! thus: version solving failed
   1: Version solving took 1.019 seconds.
   1: Tried 1 solutions.

[clintonroy]

You've requested that your project support all versions 3.x of python. You've also requested pyyaml, which only works for python version 3.6 and greater. As a particular example, you've requested that your package work for python 3.5, but that conflicts with pyaml needing at least 3.6.

https://python-poetry.org/docs/faq/#why-is-poetry-telling-me-that-the-current-projects-supported-python-range-is-not-compatible-with-one-or-more-packages-python-requirements

[jmwilkinson]

I appreciate the link, it helps explain the situation.

I think it's important to point out that this is a counter-intuitive use of dependencies. Specifying dependency ranges indicates which ranges are operable with the package, not what ranges are required. The point of version solving is to determine the usable set of packages (or none, if it cannot be solved).

The approach poetry takes here seems to run counter to the rules set forth by python versioning: https://packaging.python.org/en/latest/specifications/version-specifiers/#compatible-release

When multiple candidate versions match a version specifier, the preferred version SHOULD be the latest version as determined by the consistent ordering defined by the standard Version scheme.

This is also in line with intuitive expectations, and the way all other package managers I'm aware of specify dependencies.

If the python dependency is removed, poetry has the same error, but for a different, more complex version range (presumeably some default set by internally by poetry).

[clintonroy]

Specifying dependency ranges indicates which ranges are operable with the package, not what ranges are required are necessary.

The difference between these two ideas is lost on me.

If a package is published with the metadata as is, someone running python3.5 would get a pip error installing the package, because pyyaml could not be installed.

Make the python requirement a subset of the python required of the dependencies.

If there's any suggested improvements to the documentation or error message, then please give them.

[jmwilkinson]

That may not have been very well worded, sorry. I'll try to clarify.

A version range specification is a shorthand for a set of versions that a package can remain operable with. So, say some Package A requires Dependency B version range X resolves to the set {1, 2, 3}, this indicates that the code within that package may select any of those versions, but not, for example, 4.

The complexity arises when Package A also requires Dependency C, which then in turn requires Dependency B itself, but with versions {2, 3}.

At this point, solving has two options:

1. Error, claiming the problem is not solvable, as the sets of allowed version from Package A and Dependency C for Dependency B do not match exactly. This is what Poetry is doing here.
2. Mark B@2 and B@3 as candidate versions when trying to resolve, and then attempt to resolve to the latest one, working backwards. If there are no remaining candidate versions, then error appropriately. This is what version solvers generally do.

In the first case, the range is an indicator of compatibility for code and all of its dependencies.

This is counter-intuitive because there exists a set of valid packages that could be used, but the version solver errors without attempting to discover that set.

One of the primary benefits of general version ranges is not needing to constantly update versions as various dependencies get updated. This benefit is lost in the case (like here) where Package A allows for a superset of Dependency C, but the fact that the superset is allows is exactly the problem. There is an inversion here- for poetry's version solver to work, it necessitates that higher-level packages have subsets of their dependencies rather than supersets, ensuring frequent version specification updates.

In fact, this either leads to the undesirable behavior where Package A installs an older version of Package C that had matching version ranges, despite a newer one working perfectly fine, or inconsistent behavior discovering an older version of Package C with a matching version range then bypasses this error and allows the latest compatible version to be selected. I have not tested which of these scenarios poetry uses.

Obviously, my primary recommendation would be to alter the version solving algorithm poetry uses.

Baring that, perhaps clarifying the requirement in the error message would help? But I think there will always be confusion around this, as it is unexpected behavior.

The current project's supported Python range (>=3,<4) does not precisely match the range of some of the required packages' Python requirement:

• pyyaml requires Python >=3.6, so it will not be satisfied for Python >=3,<3.6

[dimbleby]

your proposed change to the error message is definitely worse: it is not true that the current project supported python range must "precisely match" the range from requirements. "compatible" per the current wording is correct.

possibly what you have not appreciated is that python is not like other requirements. If a user running python 3.3 types poetry install - or pip install for that matter - then the installer does not get to "select" python 3.6, and the existence of a compatible solution using python 3.6 does not help them.

anyway this all is working as expected and has been discussed often enough to make it into the FAQ (and if you search you will find a handful of issues too).

Feel free to submit a merge request improving the docs or the error message - being careful that what you are proposing is indeed an improvement! - and please close this.

[jmwilkinson]

I'm really trying to understand the reasoning here.

possibly what you have not appreciated is that python is not like other requirements. If a user running python 3.3 types poetry install - or pip install for that matter - then the installer does not get to "select" python 3.6, and the existence of a compatible solution using python 3.6 does not help them.

While its true that the installer does not select the compatible python version, if the user is running python 3.6, why error? This is fundamentally what I am missing. It seems like the expected behavior would be to only error if the user is not running a compatible version. But Poetry will error if the python version is not a subset, even if the user is running a version within the range intersection, and is thus compatible.

I think this is exacerbated by there apparently being a default value, forcing all projects to override it:

The current project's supported Python range (>=2.7,<2.8 || >=3.4) is not compatible with some of the required packages Python requirement:
  - pyyaml requires Python >=3.6, so it will not be satisfied for Python >=2.7,<2.8 || >=3.4,<3.6

Because pyyaml (6.0.1) requires Python >=3.6
 and no versions of pyyaml match >6.0.1,<7.0.0, pyyaml is forbidden.
So, because my-project depends on PyYAML (^6.0.1), version solving failed.

This is the error if no python version is set.

So is the official recommendation to manually determine the intersection of all dependencies, and then keep the project's python version in line with that intersection on every update? And if so, why? What benefit is there in doing that?

[jmwilkinson]

I see. The primary previous issue is here: #1413, closed a few months ago for inactivity, with the recommendation to open a new issue for related bugs.

The discussion seems to be almost exactly the same as the one I have raised here. I do not see an answer to the questions here on that thread, though the closest I believe is the following comment:

poetry is a tool for developing python packages and a such a tool it makes sure, that you don't define invalid dependency definitions. poetry is not a tool to install packages into a given environment like pip. That's why all dependencies must be resolvable for all python version that the project aims to be compatible with.

This seems to misunderstand both the meaning of "invalid dependencies" and the general usage of dependency managers and version solvers. Of course, it is the prerogative of the Poetry maintainers to dictate the direction of the project, and I respect that. It may, however, be valuable to the community to have a broader discussion around this topic. The very fact that an FAQ had to be created since this behavior is so commonly misunderstood should probably serve as an indicator that perhaps it should be rethought.

If I wanted to open this as a community discussion topic, what would be the appropriate way to do that?

[dimbleby]

If I wanted to open this as a community discussion topic, what would be the appropriate way to do that?

here and discord are where conversations happen.

but this particular conversation has happened enough times that I am confident in recommending that there is little value in trying to have it again. If your project is not installable on (say) python 3.1, poetry takes the view that you should provide a requires-python that excludes (say) python 3.1.

[Secrus]

While its true that the installer does not select the compatible python version, if the user is running python 3.6, why error?

This is because Poetry solves the dependency graph for a whole range of Python versions, not just the one currently being used. Just the fact that you are at a given point running a version that is compatible with a given dependency A version, doesn't mean that the same dependency version will work on a different Python version. If Poetry were to work like you describe it, your project would break in runtime for Python versions that dependency A doesn't support.

[jmwilkinson]

Could you describe this in more detail? I am struggling to see how it would break in runtime any more than it would break today.

An example of the way I'd expect Poetry to work: let's say I have python 3.10, and my project says it will work for python 3.6 - 4.0. I have a dependency that requires python 3.9 - 4.0. I install in my environment, and it works fine. I then go to install in production, running python 3.7, and I receive an error, because the dependency version is incompatible. This let's me know I need to update the version in production before I can update the project.

With how poetry works today, the only difference is that I explicitly make my project python version 3.9 - 4.0, to match my dependency. Then it still works in my environment, and when I install in production, it still breaks, and I still need to update.

What am I missing here?