Supporting custom entry points other than console_scripts in pex envs

[achimnol]

I'd like to be able to use importlib.metadata.entry_points() with arbitrary entry point names in pex environments, i.e., when running my code via ./pants run, ./pants test, or as a packaged pex binary. In other words, I'd like to use the same power of "editable installation" in pex environments. So I'm finding a way for this (related: #11386), as this is one of the remaining pieces to completely migrate our workflows.

As some of you know, I've been workarounding this for a while using a custom entrypoint loader that scans BUILD files with AST parsing for development setups. This is also mentioned in my PyCon KR/JP 2022 talks ([1], [2]).

Currently we can only specify the full entrypoint attribute in the python_distribution() target only, not in the pex_binary() target. The pex_binary() target only supports a single console-script-like entrypoint. It would be nice to add entry_points to pex_binary() and probably share its contents with the python_distribution() target's entry_points.

If this is supported, I could exploit the full convenience of pex packages. When installing our client SDK in some customer sites, there are cases that we cannot easily modify the target virtual environment. For example, to inject a self-contained SDK inside arbitrary immutable containers, it will be sufficient to mount just a single file and a single directory: the pex package and a standalone python build.

• Will there be any way to support custom entry points by writing a Pants plugin? (e.g., populating auto-generated package metadata inside the pex environments)
• If it is not feasible to do with plugins, what parts should I/we look into to implement this feature?

[benjyw]

So it sounds like what you want is a way to set some metadata on a pex, so that importlib.metadata.entry_points() can be used to introspect that metadata?

[benjyw]

(ps I deleted some simply incorrect commentary I wrote earlier)

[benjyw]

Note that a pex_binary's "entry_point" field is not an entry point in the importlib.metadata sense, but just "what code to run when the pex is invoked", so best not to get those mixed up.

[benjyw]

I did a bit of experimenting, and you can achieve this today with no Pex or Pants changes with something like:

src_root/anyname.dist_info/entry_points.txt:

[mygroup]
myentrypoint = entry.point:location

src_root/foo/bar.py:

from importlib.metadata import entry_points

print("Entry points: ")
for grp, in_group in entry_points().items():
    print(f"  Group: {grp}")
    for ep in in_group:
        print(f"    {ep}")

src_root/foo/BUILD:

python_sources()

src_root/BUILD:

resources(
  name="dist_info", 
  sources=["anyname.dist-info/**/*"]
)

pex_binary(
  name = "demo",
  dependencies = ["src_root/foo", ":dist_info"],
  entry_point = "foo.bar",
)

Now:

$ ./pants package src_root:demo
19:26:23.41 [INFO] Wrote dist/src_root/demo.pex
$ ./dist/src_root/demo.pex 
Entry points: 
  Group: mygroup
    EntryPoint(name='myentrypoint', value='entry.point:location', group='mygroup')
$ ./pants run src_root:demo
Entry points: 
  Group: mygroup
    EntryPoint(name='myentrypoint', value='entry.point:location', group='mygroup')
$ 

Note that it's important that the anyname.dist-info directory is in a source root, so that the entry_points.txt has the right relative path, but apparently it's not important that anyname be any particular string.

[benjyw]

Anyway, a PR to implement this would be great!

[dbari]

With pex_binary I get the same behavior, so I think it's equivalent? There was some warning about changed behavior in pants 2.13.0, but I'm still on 2.12.0 in my main project, so I'm not familiar with it.

As for the PR it would indeed be great, but there are still some subtle and important details to work out on the concept level. For example, specifying which entry points to depend on. If we depend on all entry points in the repository, in many cases we would inject too many dependencies, including some false ones. We could say we want the entry points of a specific python_distribution by adding the distribution itself as dependency, but then we hit #16985 and related issues (#15801, #15688 and others), while also depending on the whole distribution. Another way would be to specify a regex for groups and entrypoints, injecting only their exact files/targets as dependencies. This probably would optimal, but it's currently way over my knowledge level of pants plugins and internal structures.

I'll first try some external workarounds (generated entry_points.txt and maybe some entries in BUILD files?) to see what works out well from a user and functionality perspective and help flesh out the concept.

[benjyw]

Maybe I'm misunderstanding what is being asked for, but I assume the idea would be to infer a dependency on all the listed entry points for a pex_binary target (same as we do today for python_distribution) - once we have support for listing those on the pex_binary?

[dbari]

I believe I drifted a bit from the initial topic described by @achimnol, sorry for that. Although related, what I meant in the last couple of posts was not the entry points provided by the pex_binary, but the ones used internally and the first-party dependencies that they pull. My topic is covered quite well by the aforementioned 3 issues and your workaround with the custom entry_points.txt file works as a temporary solution, so I won't continue the discussion in the wrong direction. If I have any new insights on this problem, I'll share them in the respective issue or open a new discussion thread like this.

[benjyw]

Oh... yeah, that's a bit different. Dep inference for runtime deps is tricky. It could be made to work (similar to how "string imports" work today, but with an extra indirection via the entry point name) but would not be easy. Probably just adding explicit deps is the workaround for now...

[tgolsson]

I am having issues with this as well, though mostly for local workflows. I'm porting a bunch of loose packages into Pants. Our main application is centered around using "plugins" for various computation workflows, with a centralized entrypoint containing workload and configuration management. So app.console_script.foo will lookup all entrypoints in the app.engines namespace and use those to execute various workloads.

Our old workflow was centered around "editable" installs for development, which meant that for the most part entry_points worked. The sharpest edge was that you needed to reinstall if you added an entry_point, but that is quite rare. And anything beyond that was automated with CI or scripts, so the nitty-gritty of installing the right plugins for the right cloud image etc was never a concern.

In Pants; this workflow seemingly gets much worse. Using the above approach works; but due to our heavy set of dependencies (torch, tensorflow, scipy, etc) it ends up being ridiculously slow - I'm measuring > 1minute to just build the pex with the same depset. I've instead opted to make the "host" python_sources() depend on the plugin and the mock entrypoints. This is OK from an iteration-speed standpoint, but ends up being circular and breaks during packaging.

[tgolsson]

Sort of. I want different ones when deploying - I have plugin1, plugin2, etc. When running locally I want all to be available, but when building docker images I normally only have a subset.

[benjyw]

Ah ok, so you have to compose the ones you want in via different explicit deps, meaning you need different python_sources()/python_tests() to depend on the relevant entry_points.txt targets

[tgolsson]

Ok ; that seems slightly inconvenient and fragile. I think what I'd want for local development is something that mirrors what ./pants run python_source does in terms of multiple pex files and copying the source but that isn't tied to a specific distribution. Maybe a python_binary? Something like this:

python_binary(
    name="exec",
    dependencies=["//src/py/backend/src", "//src/py/plugin1/src", ..., "//src/py:mock-dist-info"],
    entrypoint="backend.entry_point.foo"
)

Then I can easily decouple and define this "joint" environment externally, and nothing inside my python_distributions have to be aware.

[tgolsson]

OK; nevermind the above - I can just define another source root or package; and small shims like this:

python_source(
   name="foo",
   source="foo.py",
   dependencies["//src/py/plugin1:package", ..., "//src/py:mock-dist-info"],
)

from backend.entry_point import foo
foo.main()

A bit of extra work but keeps the main distribution clean!

[benjyw]

Glad you've found a solution! If you can think of a neater way to do this, perhaps with a plugin or a macro, let us know!