How to specify a python interpreter in a virtualenv on the run using PEX_PYTHON？

[zw615]

Hi, there!

I am trying to run a packed python .pex file with the command ./xxx.pex main.py. However, I just discover that there are a few necessary packages missing in the pex file. So I try to specify the custom python interpreter in a virtualenv with all the necessary packages with the PEX_PYTHON environment variable. Strangely, PEX_PYTHON=/path/to/venv/python ./xxx.pex main.py does not work, and so does /path/to/venv/python ./xxx.pex main.py. I wonder how I can specify the python environment or add the necessary packages without needing to regenerate or change the .pex file?

Thanks a lot!

[jsirois]

@zeyuwang615 PEX files are intended to be the equivalent of a hermetic, fully self-contained virtualenv. By default a PEX purposely ignores all packages on your system and just uses the Python interpreters found there. To ask PEX to not be hermetic, you must use PEX_INHERIT_PATH: https://pex.readthedocs.io/en/v2.1.131/api/vars.html#PEX_INHERIT_PATH

For example, given this example.py:

$ cat example.py
import colors
import cowsay


print(colors.green(cowsay.get_output_string("tux", "Hello Venv!")))

And this PEX with missing deps (no colors, which is provided by ansicolors):

$ pex cowsay --exe example.py -o example.pex

You get:

$ ./example.pex
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/jsirois/.pex/unzipped_pexes/1344f920839cad8e829ba76ae4ad89bce5cbaf68/__main__.py", line 106, in <module>
    bootstrap_pex(__entry_point__, execute=__execute__, venv_dir=__venv_dir__)
  File "/home/jsirois/.pex/unzipped_pexes/1344f920839cad8e829ba76ae4ad89bce5cbaf68/.bootstrap/pex/pex_bootstrapper.py", line 615, in bootstrap_pex
    pex.PEX(entry_point).execute()
  File "/home/jsirois/.pex/unzipped_pexes/1344f920839cad8e829ba76ae4ad89bce5cbaf68/.bootstrap/pex/pex.py", line 560, in execute
    sys.exit(self._wrap_coverage(self._wrap_profiling, self._execute))
  File "/home/jsirois/.pex/unzipped_pexes/1344f920839cad8e829ba76ae4ad89bce5cbaf68/.bootstrap/pex/pex.py", line 467, in _wrap_coverage
    return runner(*args)
  File "/home/jsirois/.pex/unzipped_pexes/1344f920839cad8e829ba76ae4ad89bce5cbaf68/.bootstrap/pex/pex.py", line 498, in _wrap_profiling
    return runner(*args)
  File "/home/jsirois/.pex/unzipped_pexes/1344f920839cad8e829ba76ae4ad89bce5cbaf68/.bootstrap/pex/pex.py", line 603, in _execute
    return self.execute_entry(
  File "/home/jsirois/.pex/unzipped_pexes/1344f920839cad8e829ba76ae4ad89bce5cbaf68/.bootstrap/pex/pex.py", line 800, in execute_entry
    return self.execute_module(entry_point.module)
  File "/home/jsirois/.pex/unzipped_pexes/1344f920839cad8e829ba76ae4ad89bce5cbaf68/.bootstrap/pex/pex.py", line 808, in execute_module
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "/usr/lib/python3.10/runpy.py", line 224, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/jsirois/.pex/unzipped_pexes/1344f920839cad8e829ba76ae4ad89bce5cbaf68/__pex_executable__.py", line 1, in <module>
    import colors
ModuleNotFoundError: No module named 'colors'

Here I set up a venv with the missing dep, which it sounds like you did:

$ python -mvenv colors.venv
$ colors.venv/bin/pip install ansicolors
Collecting ansicolors
  Using cached https://files.pythonhosted.org/packages/53/18/a56e2fe47b259bb52201093a3a9d4a32014f9d85071ad07e9d60600890ca/ansicolors-1.1.8-py2.py3-none-any.whl
Installing collected packages: ansicolors
Successfully installed ansicolors-1.1.8
You are using pip version 18.1, however version 21.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

And here I tell PEX it's ok to let the outside environment leak in:

$ PEX_INHERIT_PATH=prefer colors.venv/bin/python ./example.pex
  ___________
| Hello Venv! |
  ===========
                \
                 \
                  \
                   .--.
                  |o_o |
                  |:_/ |
                 //   \ \
                (|     | )
               /'\_   _/`\
               \___)=(___/

And that penguin is, in fact, green!

All that said, this is generally not what you want to do! You should try to build the PEX file to contain the dependencies needed for each interpreter / platform it will be run on. See --python, --platform and --complete-platform.