Pass (additional) index url to pip install command

[moooeeeep]

The PipManager currently installs all pip osdeps in one big install command. It can only install from the default package index, and allows besides the version range spec that can be passed with the package name no further options.

Pip provides a lot of options. I am especially interested in --index-url and --extra-index-url. These would allow us to install packages from non-standard package indexes, e.g., CPU-only builds of torch from this index. It would also allow us to host our own internal package index, and install pip packages as osdeps from there.

To facilitate this, the osdeps syntax of pip packages would need to be extended, and the pip install command might need to be split up appropriately, to not leak the additional arguments to the installation of other packages.

An alternative to some of these use cases might be to define autobuild packages for these packages. But autobuild packages are currently not installed with pip, but with python setup.py install (related issue: rock-core/autobuild#108), which might not work with some workspaces or packages. Also we would need to duplicate the dependencies of those packages in our package set then.

Any comments / suggestions?

[doudou]

The osdep system nowadays supports passing per-package arguments. The bundler manager for instance accepts 'version' to specify a package version.

I think you could handle the index URL this way. You would indeed have to have more than one pip call, which would have to be handled by the python package manager.

I am lucky and never had to dig into python package managers. Isn't there something standard closer to bundler, where you could list all the dependency requirements (including index URLs) and have them installed at once by the tool ?

[2maz]

@moooeeeep

Pip provides a lot of options. I am especially interested in --index-url and --extra-index-url. These would allow us to install packages from non-standard package indexes, e.g., CPU-only builds of torch from this index. It would also allow us to host our own internal package index, and install pip packages as osdeps from there.

Did you look into setting up a workspace specific pip configuration, e.g. via pip config or via PIP_CONFIG_FILE env variable. There you can set the extra-index-url without changing autoproj, although this will not be package specific.

Regarding the installation of python_package:
You can have a look at a proof-of-concept that I started - check autobuild and autoproj. Dependencies are loaded from setup.cfg or pyproject.toml files into autoproj.

@doudou

I am lucky and never had to dig into python package managers. Isn't there something standard closer to bundler, where you could list all the dependency requirements (including index URLs) and have them installed at once by the tool ?

You can already use "pip install -r requirements.txt", where requirements.txt is a collection of the commandline arguments including the list of packages - if that is what you are looking for. Maybe in combination with exporting a fixed dependency list via 'pip freeze'.
Otherwise there is conda and there is poetry as main players. But I hesitate to suggest integrating those with regard to maintainability inside autoproj.

[moooeeeep]

Did you look into setting up a workspace specific pip configuration, e.g. via pip config or via PIP_CONFIG_FILE env variable. There you can set the extra-index-url without changing autoproj, although this will not be package specific.

Thanks for the hint - I didn't know that option existed! For a global --extra-index-url this seems workable.

I've seen a related discussion here, regarding passing certain options on a per-package basis (here: --pre): pypa/pip#8426
This seems to be a hard problem in general. (Simply) installing the osdeps one after each other might not be as straight-forward as I thought.