How poetry run should handle non-installed scripts?

[wagnerluis1982]

TL;DR

Roughly there are 3 options for poetry run handling non-installed scripts, in the order of my preference:

1. Install the script in the virtualenv when poetry detects a script is not installed.
2. Disallow to run uninstalled scripts at all.
3. Keep as is ☹️

I think option (1) is the best because would only enhance Poetry without introducing any incompatibility. To be honest, I think no one is relying on the broken sys.argv[0] poetry emits when script is not installed.

Background

The command poetry run is able to execute any command in the virtualenv, but has a special behavior with entry points (the ones defined at [tool.poetry.scripts]).

Currently poetry run can execute entry points even though they are uninstalled, i.e. they don't exist physically in the file system. But this, inadvertently, is confusing the users due to how poetry historically fills sys.argv[0], which led to several issues created in the past (#480, #965, #2435, #7528).

For instance, using latest poetry version, with the following entry point, poetry run dev may result in a different sys.argv[0] depending if the script is installed or no.

[tool.poetry.scripts]
dev = 'my_package.main:run'

• sys.argv = ['dev'] if script is not installed.
• sys.argv = ['/home/wagner/.cache/pypoetry/virtualenvs/my_package-Iqs9p7vq-py3.7/bin/dev'] if script is installed.

Important quotes from issue #7528 which shows how confusing this is for users

...did you install the entry point via poetry install?

oh, do I need to run poetry install again after adding the script line?

Originally posted by @delucca in #7528 (comment)

Since I didn't add any new dependencies, I didn't know I had to run poetry install before testing.

Originally posted by @delucca in #7528 (comment)

I feel the wrongness feeling [of poetry run] is being fueled by the fact that poetry run may have different behaviors regarding entry point (installed / non-installed).

So, I feel that poetry should forbid at all to run non installed entry points. What do maintainers think about it?

Originally posted by @wagnerluis1982 in #7528 (comment)

At the moment, I don't see a use case why it should be necessary to run non installed entry points. However, before we forbid it we should deprecate it and see if people do complain.

Originally posted by @radoering in #7528 (comment)

[wagnerluis1982]

I was checking the feasibility to accomplish "Install the script in the virtualenv when poetry detects a script is not installed".

It is much trickier than I thought. Because the only guarantee we have in the poetry run is a python environment (virtualenv, system env, mock env, ...)

To install the script to the file system, first I would need to know which kind of env, this is completely out of scope of poetry run.

[finswimmer]

Digging up, I traced to 06a2bd1, in which the ancient poetry script (now removed) was merged with poetry run.

Wow, that's quite old. I tend to just remove the special handling, if no one find a reason why it should be kept ...

[delucca]

@wagnerluis1982 I don't have a clear picture of Poetry's internals, but IMHO (as a user) if I execute poetry run ... I don't think it needs to work "automatically"

If having to run poetry install before running a recently added script is a requirement, I think Poetry could simply say that in the result of the command. Like this (using your example, but removing the prompt):

$ poetry run dev
WARNING: 'dev' is an entry point defined in pyproject.toml, but it is not installed.

You need to install the project in order to make it available. To do so simply run poetry install.

The problem (in my opinion) is that having to run install is not usual for package managers in general (like npm and others). So, people are used to execute the run command directly. There is nothing wrong with this (since you only need to run once), the only problem is not being able to handle that exception properly

If Poetry simply explain it in the prompt it should be good enough IMO

[wagnerluis1982]

@delucca to block poetry run for uninstalled scripts was my first idea.

I tend to agree completely that an error message should be enough. If I understood correctly, @finswimmer also agrees to remove special handling.

@radoering, what are your thoughts? Still think that we should deprecate first, then remove?

[radoering]

I still think that we should deprecate things first, if we can avoid a hard failure. Of course, some users overlook warnings, but many users will notice and fix it without experiencing a hard failure.

[wagnerluis1982]

I still think that we should deprecate things first, if we can avoid a hard failure.

Understandable, in that case, my PR #7606 is aligned to the deprecation perspective.

[alexanderilyin]

Hi everyone,

I'm on Poetry (version 1.6.1) and in Dockerfile I basically do this:

poetry lock \
  -vvv \
  --no-ansi \
  --no-interaction

poetry install -vvv \
    --extras mystuff \
    --no-ansi \
    --no-interaction

In pyproject.toml I have the following:

[tool.poetry.scripts]
myscript = 'myscript.cli:command'

The rest of the Dockefile has this:

ENTRYPOINT [ "poetry", "run", "myscript" ]
CMD ["--help"]

And I'm getting the warning when running the script:

$ docker run --rm myimage

Warning: 'myscript' is an entry point defined in pyproject.toml, but it's not installed as a script. You may get improper `sys.argv[0]`.

The support to run uninstalled scripts will be removed in a future release.

Run `poetry install` to resolve and get rid of this message.

Usage: myscript [OPTIONS] COMMAND [ARGS]...
…

It's not clear what's the problem so far, given that I'm doing poetry install … when making an image. Just from the warning it seems that I have to run install twice and it does not look right. Is there anything else user should do except running poetry install …?

[alexanderilyin]

Well, main symptom seems to be that this part is missing when poetry install … runs as part of docker build …

…
Installing the current project: myscript (0.3.1)
  - Building package myscript in editable mode
  - Adding myscript.pth to /workspaces/myscript/.venv/lib/python3.11/site-packages for /workspaces/myscript
  - Adding the myscript script to /workspaces/myscript/.venv/bin
  - Adding the myscript-0.3.1.dist-info directory to /workspaces/myscript/.venv/lib/python3.11/site-packages

It feels like it could have something to do with tty not available during docker build …. What I'm trying to do is to distribute myscript as installed in editable mode and make it container's entrypoint so I can use docker image as executable.

[alexanderilyin]

It seems that I maybe somehow run it with --no-root option?

• https://github.com/python-poetry/poetry/blob/47a3a19c442c21aa5b2836ea092dda8ae4d13151/src/poetry/console/commands/install.py#L155C25-L156

[alexanderilyin]

• https://github.com/python-poetry/poetry/blob/1.6.1/src/poetry/console/commands/install.py#L158-L164

That comment should have been a log message.

#10 23.41   ModuleOrPackageNotFound
#10 23.41 
#10 23.41   No file/folder found for package myscript

[alexanderilyin]

Shame on me, I had basically this in my Dockerfile. What did I expect?

RUN \
  --mount=type=secret,id=foo \
  --mount=type=secret,id=bar \
  --mount=type=ssh \
  --mount=type=bind,source=install.sh,target=/tmp/install.sh \
  --mount=type=bind,source=poetry.lock,target=/workspaces/myscript/poetry.lock \
  --mount=type=bind,source=poetry.toml,target=/workspaces/myscript/poetry.toml \
  --mount=type=bind,source=pyproject.toml,target=/workspaces/myscript/pyproject.toml,readwrite \
  /tmp/install.sh
ADD . /workspaces/myscript

[radoering]

That comment should have been a log message.

It already is in the master branch. (I had the same thought a few weeks ago. 😃)

[earonesty]

i think the support should stay. it's ok for it to log a warning, but it's terribly useful to run in "dev mode" without requiring an installation every time that you might forget you did, and now wind up testing the wrong thing. honestly, i loathe installation of dev packages while working. it just leads to errors. for ci/cd it makes sense (its important), but for coding, it's bad.

[staffanf]

But you will need to execute poetry install anyway at least once, no?

Not really if you only run one script that just has other project files as imports?

My case is that we run this as part of a cmake build during cmake configure.
Keeping it fast is a priority and poetry install && poetry run is 2-3 x slower than just poetry run in this case.
If there was a fast scriptable way to know if I need to do install then I would be fine with the deprecation, but currently the fastest way I found was to just run use poetry run directly.

[wagnerluis1982]

But you will need to execute poetry install anyway at least once, no?

Not really if you only run one script that just has other project files as imports?

In that case, I think you can simply create a custom script that appends the source folder to the sys.path before using it.

[staffanf]

In that case, I think you can simply create a custom script that appends the source folder to the sys.path before using it.

Hmm, I just might use that. What I'd really what is a really fast way to check if the venv is installed or not.
Also, if I had some custom things in https://python-poetry.org/docs/pyproject/#packages, I guess I would need to install anyhow?

[wagnerluis1982]

What I'd really what is a really fast way to check if the venv is installed or not.

But why would you need a venv if you said that you don't need to install any dependency for the script works?

[staffanf]

Well, preferrably, we want to use poetry in the same way everywhere. Having special cases here and there in the code is not great for maintainability.
In general we tell everyone to just use poetry install and the poetry run.

We use a "mono-repo" and the cmake part is one subproject and the python utils are other sub-projects.
Some parts of the python utils needs dependencies (like requests for handling rest apis calls) but this specific part was about calling a self-written python module to get a git based version.
This needs to be done as fast as possible since it's done during every cmake build.
So, in this case we don't need the venv because currently we have no external imports in the scripts involved, but doing it some other way introduces a maintainability issue in our code. But then again, we do not want to rely on deprecated behavior either.

[earonesty]

You don't need poetry install every time if you only changed the content of the script.

oh, ok. i did add a script recently. not sure why that matters (it's easy to override arg0), but that's not too bad.