Unable to export to single virtualenv & if I even should?? #18704

p-baum · 2023-04-08T09:59:07Z

p-baum
Apr 8, 2023

I'm clearly missing something. My build file and dependencies are recognised. I'm trying to export for my IDE (vscode) but the export produces multiple envs (python-default, tools/black, tools/pylint, etc). What I want is everything in one env where I can point my IDE.

I've read the documentation but even that states 'You can specify the --resolve flag multiple times to export multiple virtualenvs at once.'

What am I missing? Why would one want to generate multiple different envs. Is there a "trick" to link them somwhow?

I think I'm missing some basic understanding which the docs are not providing.

Answered by p-baum

Apr 10, 2023

So I switched to version 2.16.0rc0 and simply added the tools I want (pylint, black, isort) to my requirements.txt and

pants generate-lockfiles --resolve=python-default
pants export --py-resolve-format=symlinked_immutable_virtualenv --resolve=python-default

set dist/export/python/virtualenvs/python-default/3.10.10/bin/python as Interpreter

and vscode linting and black formatting is working. "Sort Imports" however is not. Running the fmt goal does however sort the imports.

for my sanity's sake I will sort them with the fmt goal for now. I would of course be happy to hear suggestions on how I might get this working in vscode.

View full answer

p-baum · 2023-04-08T11:02:49Z

p-baum
Apr 8, 2023
Author

#12449 states "This is addressed in a different way by #18481, which allows a tool to be installed from a user resolve.". pull 18481 does not explain what a "user resolve" is. The documentation mentions resolves in various place but I cant find any section that specifcally describes what is meant by "user resolves".

I am currently trying to piece together the clues I can find about this concept.

1 reply

benjyw Apr 9, 2023
Maintainer Sponsor

A "user resolve" is just a resolve as described in https://www.pantsbuild.org/v2.16/docs/python-lockfiles.

Prior to 2.16 "tool resolves" and "user resolves" were two separate concept. In 2.16 they are unified, and so that terminology is going away.

p-baum · 2023-04-08T12:35:26Z

p-baum
Apr 8, 2023
Author

These don't work:

python_sources(name="lib")

python_sources(
    name="dev",
    dependencies=[
        ":lib",
    ],
    resolve=parametrize("pylint","black")
)

python_requirements()


UnrecognizedResolveNamesError: Unrecognized resolve name from the field `resolve` in the target src/browser/browser/__init__.py:dev@resolve=pylint: pylint

All valid resolve names: ['python-default']

python_sources(name="lib")

python_sources(
    name="dev",
    dependencies=[
        ":lib",
        "3rdparty/python:pylint",
        "3rdparty/python:black",
    ],
)

python_requirements()

ResolveError: The address 3rdparty/python:black from the `dependencies` field of the target src/browser/browser/__init__.py:dev does not exist.

The target name ':black' is not defined in the directory 3rdparty/python. Did you mean one of these target names?

  * :python-default

1 reply

benjyw Apr 9, 2023
Maintainer Sponsor

You have not defined 3rdparty/python:black I assume, so this error is correct?

p-baum · 2023-04-08T12:49:47Z

p-baum
Apr 8, 2023
Author

https://www.pantsbuild.org/docs/setting-up-an-ide#tool-virtualenvs states:

This allows you to configure your editor to use the same version of the tool as Pants does for workflows like formatting on save.

Please how for more that 1 tool at a time?

0 replies

p-baum · 2023-04-08T13:24:09Z

p-baum
Apr 8, 2023
Author

https://www.pantsbuild.org/docs/python-third-party-dependencies#tool-lockfiles states:

Pants distributes a lockfile with each tool by default.

I'm thinking i will try to add the tool locks files as dev dependencies somehow. Where are they? I've run pants generate-lockfiles but I cant find them.

0 replies

p-baum · 2023-04-09T08:17:42Z

p-baum
Apr 9, 2023
Author

I'm try to be too clever for my own good. A solution is to use the PYTHONPATH env variable (vscode settings "python.envFile") to include multiple paths:

PYTHONPATH=src/base1:src/base2:dist/export/python/virtualenvs/tools

after running pants export --symlink-python-virtualenv ::

[end monologue]

3 replies

benjyw Apr 9, 2023
Maintainer Sponsor

Glad you found a solution! To clarify though, pants export :: is deprecated (the :: never made sense). So the new way is pants export --resolve=resolve1 --resolve=resolve2....

In this, resolve1, resolve2 etc. are resolves that you've defined as described here.

You can have multiple resolves for different parts of your codebase (although the fewer the better, one is ideal, but you can't always get away with that). And you also use resolves for Python-based tools, e.g., pytest, black, flake8 etc. "tool lockfiles" used to be a separate concept, but as of the upcoming Pants 2.16, if you need custom versions of tools, you use a regular resolve. So you have flexibility - you can have one resolve for your code and one for each tool, or one resolve for your code and one extra resolve for all tools, or your tools can be in the same resolve as your code (this is useful for example for pytest, which is both a runtime library and a tool, so needs to be listed in your code's resolve and in the pytest tool resolve, if they share a resolve you only need to list it in one place).

I hope that made some kind of sense?

benjyw Apr 9, 2023
Maintainer Sponsor

And of course all of this is just for third-party code. Your PYTHONPATH trick also brings in the first-party source roots.

Another way of doing that, if your first-party code can be packaged as a distribution, is via editable installs into the exported venv, support for which is being added here: #18639

p-baum Apr 9, 2023
Author

Thankyou. After multiple reads I just found the sentence I was missing:

"resolves" - logical names given to lockfiles so that they are easy to reference.

I dont yet understand the need to call "lockfiles names" "resolves". It would have saved me a lot of suffering if they were just called "lockfile names. I guess there's reasons for that I am yet to discover.

Thankyou again.

benjyw · 2023-04-09T20:42:38Z

benjyw
Apr 9, 2023
Maintainer Sponsor

Yeah, potentially the word "resolve" should go away in favor of "lockfile".

…

On Sun, Apr 9, 2023, 1:28 PM Paul ***@***.***> wrote: Thankyou. After multiple reads I just found the sentence I was missing: "resolves" - logical names given to lockfiles so that they are easy to reference. I dont yet understand the need to call "lockfiles names" "resolves". It would have saved me a lot of suffering if they were just called "lockfile names. I guess there's reasons for that I am yet to discover. Thankyou again. — Reply to this email directly, view it on GitHub <#18704 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAD5F7D7JB3N3S7WDQUU7ULXAMLVHANCNFSM6AAAAAAWXLS5UQ> . You are receiving this because you commented.Message ID: ***@***.***>

2 replies

p-baum Apr 9, 2023
Author

If there really is no need for it that would help the noobs massively! Then we could stick to using "resolves" as a verb - further simplifying understanding and communications.

benjyw Apr 10, 2023
Maintainer Sponsor

There is a distinction - a resolve is a logical thing and a lockfile is part of its implementation (you could imagine a resolve with no lockfile), but in practice I'm not sure this distinction is important enough to warrant the confusion.

p-baum · 2023-04-10T08:22:03Z

p-baum
Apr 10, 2023
Author

Ive had to remove my answer. It appeared to work at first but since then I've been unable to repeat it. In order to check if linting even works I manually set the pylint path in vscode (python.linting.pylintPath) to the tool path (${workspaceFolder}/dist/export/python/virtualenvs/pylint/3.10.10/bin/pylint). It fails to find my third party imports :(

I suspect it has something to do with my python version 3.10.10 or python3.10 being inserted into my venv paths:

dist/export/python/virtualenvs/python-default/3.10.10/lib/python3.10/site-packages

I can manually set my tool paths like this but then I have to manually change them all when my python version upgrades. yuck.

But I dont know what I can do about vscode/pylint not finding my site-packages? It accepts the environment. Is anybody using this with vscode? How are you doing this? Why is this so damn hard? Surely setting up a dev env is the first thing any newcomer wants to do.

0 replies

p-baum · 2023-04-10T09:02:31Z

p-baum
Apr 10, 2023
Author

So I switched to version 2.16.0rc0 and simply added the tools I want (pylint, black, isort) to my requirements.txt and

pants generate-lockfiles --resolve=python-default
pants export --py-resolve-format=symlinked_immutable_virtualenv --resolve=python-default

set dist/export/python/virtualenvs/python-default/3.10.10/bin/python as Interpreter

and vscode linting and black formatting is working. "Sort Imports" however is not. Running the fmt goal does however sort the imports.

for my sanity's sake I will sort them with the fmt goal for now. I would of course be happy to hear suggestions on how I might get this working in vscode.

6 replies

p-baum May 5, 2023
Author

fmt isort is only working because I added isort to my requirements.txt.

As far as I can tell, exporting isort to its own virtualenv (and any other tool) is for vscode users worthless.

There seems to be no possibility to combine these individual environments for vscode consumption. This is why I was forced to use v2.16.

I'm still clueless as to how I can get vscode sort imports to play nicely (or vice versa)

p-baum May 5, 2023
Author

Furthermore I now run the risk of linking incompatible tool versions. I read somewhere (and now can't find again) that there's a way to find these versions in order to pin my requirements to match. But then how do I know when to upgrade them? I guess I have to manually check them every time I start using a new pants version.

I suspect I am missing something. I find it hard to believe that vscode would be so hard to setup considering its wide usage:
https://survey.stackoverflow.co/2022/#section-most-popular-technologies-integrated-development-environment

dijonkitchen May 5, 2023

Yeah, this is all kind of confusing since pants can obviously run a bunch of commands and their dependencies, but it's difficult to inspect where those deps are and utilize the same ones. At least in the source code, they should be here: https://github.com/pantsbuild/pants/tree/main/3rdparty/python

p-baum May 6, 2023
Author

The tool versions can be seen with pants help tools.

For 2.16 this means:

pylint>=2.13.0,<3
black>=22.6.0,<24
isort>=5.9.3,<6.0
pytest==7.0.1

p-baum May 6, 2023
Author

I've added

"python.sortImports.path": "${workspaceFolder}/dist/export/python/virtualenvs/python-default/3.10.10/bin/isort"

to settings.json and now "sort imports" is working :)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to export to single virtualenv & if I even should?? #18704

{{title}}

Replies: 8 comments 13 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Unable to export to single virtualenv & if I even should?? #18704

p-baum Apr 8, 2023

Replies: 8 comments · 13 replies

p-baum Apr 8, 2023 Author

benjyw Apr 9, 2023 Maintainer Sponsor

p-baum Apr 8, 2023 Author

benjyw Apr 9, 2023 Maintainer Sponsor

p-baum Apr 8, 2023 Author

p-baum Apr 8, 2023 Author

p-baum Apr 9, 2023 Author

benjyw Apr 9, 2023 Maintainer Sponsor

benjyw Apr 9, 2023 Maintainer Sponsor

p-baum Apr 9, 2023 Author

benjyw Apr 9, 2023 Maintainer Sponsor

p-baum Apr 9, 2023 Author

benjyw Apr 10, 2023 Maintainer Sponsor

p-baum Apr 10, 2023 Author

p-baum Apr 10, 2023 Author

p-baum May 5, 2023 Author

p-baum May 5, 2023 Author

dijonkitchen May 5, 2023

p-baum May 6, 2023 Author

p-baum May 6, 2023 Author

p-baum
Apr 8, 2023

Replies: 8 comments 13 replies

p-baum
Apr 8, 2023
Author

benjyw Apr 9, 2023
Maintainer Sponsor

p-baum
Apr 8, 2023
Author

benjyw Apr 9, 2023
Maintainer Sponsor

p-baum
Apr 8, 2023
Author

p-baum
Apr 8, 2023
Author

p-baum
Apr 9, 2023
Author

benjyw Apr 9, 2023
Maintainer Sponsor

benjyw Apr 9, 2023
Maintainer Sponsor

p-baum Apr 9, 2023
Author

benjyw
Apr 9, 2023
Maintainer Sponsor

p-baum Apr 9, 2023
Author

benjyw Apr 10, 2023
Maintainer Sponsor

p-baum
Apr 10, 2023
Author

p-baum
Apr 10, 2023
Author

p-baum May 5, 2023
Author

p-baum May 5, 2023
Author

p-baum May 6, 2023
Author

p-baum May 6, 2023
Author