How to install private dependency into source code root?

[jacobg]

I have a python project that gets deployed to Google Cloud Functions. This deployment consists of the source code folder and a requirements.txt file. This application uses some private packages coming from another directory in my monorepo that are locally installed into the application source code root directory using pip install -t. Here is Google's documentation on this use case:
https://cloud.google.com/functions/docs/writing/specifying-dependencies-python#using_private_dependencies

Is poetry able to install local npm packages into the application directory rather than environment site packages? Or is there a way to hook in an install script to do that?

[sinoroc]

I do not think you need a special feature of poetry to do this. Calling poetry run pip install --target ./external Library is probably good enough. But I am not familiar with this "Google Cloud Functions", so no guarantees.

[jacobg]

Thanks @sinoroc. There's an extra catch, which is that these private external packages are namespace packages using the same namespace. And there seems to be a pip -t bug in that use case as discussed here:
pypa/pip#1924

I created a custom script to install such packages, as pasted below. So I guess what might be best would be if poetry was able to hook in custom script calls when calling poetry install.

#!/usr/bin/env python

"""
The purpose of this script is to pip install the core namespace
packages into a specified target directory. The reason why the
caller can't simply call `pip install -t <target> <package> --upgrade`
is that it doesn't seem like pip is able to install multiple
subpackages of a single namespace to the same target directory.
See issue here: https://github.com/pypa/pip/issues/1924
So we instead install to a temporary directory, and move
it to the actual target directory.
"""

import argparse
import os
import shutil
import subprocess
import tempfile

CURRENT_DIR = os.path.dirname(os.path.abspath(__file__))


def run(package_name, target_directory):

    # create core directory in target if it does not exist
    target_core_directory = os.path.join(target_directory, 'core')
    if not os.path.exists(target_core_directory):
        os.mkdir(target_core_directory)

    # run pip install to temporary directory
    temp_directory = tempfile.mkdtemp()
    return_code = subprocess.call([
        'pip',
        'install',
        '-t',
        temp_directory,
        os.path.join(CURRENT_DIR, package_name)
    ])
    if return_code != 0:
        raise RuntimeError(f'pip install failed with return code {return_code}')

    # Move contents of core directory from install directory to target core directory,
    # overwriting previous installation.
    target_package_directory = os.path.join(target_core_directory, package_name)
    if os.path.exists(target_package_directory):
        shutil.rmtree(target_package_directory)
    shutil.move(os.path.join(temp_directory, 'core', package_name),
                target_core_directory)

    # delete temporary directory
    shutil.rmtree(temp_directory)


if __name__ == '__main__':
    parser = argparse.ArgumentParser(
        description=__doc__,
        formatter_class=argparse.RawDescriptionHelpFormatter
    )
    parser.add_argument('--package',
                        required=True,
                        help='Package name')
    parser.add_argument('--target',
                        required=True,
                        help='Path to target')

    args = parser.parse_args()

    run(args.package, args.target)

[sinoroc]

Aside: You might want to replace ['pip', 'install', ...] with [sys.executable, '-m', 'pip', 'install', ...], to make sure you are installing for the right interpreter.

[sinoroc]

Is the issue pypa/pip#1924 really still occuring with recent Python interpreters? In that discussion they mention "this issue largely goes away with Python 3.3".

[jacobg]

Thanks. When I tried it out a few months ago, that was the only solution that worked for me. I'll plan to try again soon with a fresh mind.

[jacobg]

As I think about it more, I might just need to think of these private packages as being part of my application source code, i.e., run the script outside of poetry. The script is basically like a source code generator of sorts.