third_party python imports and Visual Studio Code (pylance) #2207
Replies: 2 comments 2 replies
-
|
Any idea what's outputting that extra directory? The problem is presumably that one of the non-source python rules is creating that folder. The please out directory isn't very clean. We've thought about creating a python virtual env under This should be pretty straight forward. We just need to put a Something like this might help. You can stick this at the top of your def pip_library(name:str, version:str, hashes:list=None, package_name:str=None,
test_only:bool&testonly=False, deps:list=[], post_install_commands:list=None,
install_subdirectory:str=None, repo:str=None, use_pypi:bool=None, patch:str|list=None,
visibility:list=None, zip_safe:bool=True, licences:list=None, pip_flags:str=None,
strip:list=[]):
"""Provides a build rule for third-party dependencies to be installed by pip.
Args:
name (str): Name of the build rule.
version (str): Specific version of the package to install.
hashes (list): List of acceptable hashes for this target.
package_name (str): Name of the pip package to install. Defaults to the same as 'name'.
outs (list): List of output files / directories. Defaults to [name]. If subdirectory is used outs[0] is used as the prefix.
test_only (bool): If True, can only be used by test rules or other test_only libraries.
deps (list): List of rules this library depends on.
post_install_commands (list): Commands run after pip install has completed.
install_subdirectory (bool): Forces the package to install into a subdirectory with this name.
repo (str): Allows specifying a custom repo to fetch from.
use_pypi (bool): If True, will check PyPI as well for packages.
patch (str | list): A patch file or files to be applied after install.
visibility (list): Visibility declaration for this rule.
zip_safe (bool): Flag to indicate whether a pex including this rule will be zip-safe.
licences (list): Licences this rule is subject to. Default attempts to detect from package metadata.
pip_flags (str): List of additional flags to pass to pip.
strip (list): Files to strip after install. Note that these are done at any level.
"""
pip_target_dir = "pip_install"
package_name = package_name or name
package_name = f'{package_name}=={version}'
post_install_commands = post_install_commands or []
use_pypi = CONFIG.USE_PYPI if use_pypi is None else use_pypi
index_flag = '' if use_pypi else '--no-index'
pip_flags = pip_flags or CONFIG.PIP_FLAGS
repo_flag = ''
repo = repo or CONFIG.PYTHON_DEFAULT_PIP_REPO
if repo:
if repo.startswith('//') or repo.startswith(':'): # Looks like a build label, not a URL.
repo_flag = f'-f %(location {repo})'
deps += [repo]
else:
repo_flag = '-f ' + repo
target = f"{pip_target_dir}/{name}"
if install_subdirectory:
target += '/' + install_subdirectory
pip_tool = "$TOOLS_PIP" if CONFIG.PIP_TOOL else '$TOOLS_PYTHON -m pip'
# Unset these compiler and linker flags as macOS doesn't have all these options.
pip_compile_flags = 'CFLAGS=" -Wl,--build-id=none -g0 " LDFLAGS=" -Wl,--build-id=none"' if CONFIG.HOSTOS != 'darwin' else 'CFLAGS=" -g0"'
pip_cmd = (
f'mkdir {pip_target_dir} && '
f'PIP_CONFIG_FILE=/dev/null {pip_compile_flags} {pip_tool} install --isolated --no-deps --prefix= --no-compile '
f'--no-cache-dir --default-timeout=60 --target={target}'
)
# TODO(jpoole): This is a total mess on pip's side with --user flag. https://github.com/pypa/pip/issues/1668
# It seems like most things have been resolved (Oct 2019ish) however a lot of python installs are probably not
# updated yet. Essentially the "user" default will be turned off when --target is passed. Right now, it complains
# that --prefix (and therefore --target) are incompatible with --user. See https://github.com/pypa/pip/pull/7002
if CONFIG.HOSTOS == 'darwin' and not CONFIG.DISABLE_VENDOR_FLAGS:
# Fix for Homebrew which fails with a superficially similar issue.
# https://github.com/Homebrew/brew/blob/master/docs/Homebrew-and-Python.md suggests fixing with --install-option
# but that prevents downloading binary wheels. This is more fiddly but seems to work.
# Unfortunately it does *not* work similarly on the Debian problem :(
pip_cmd = 'echo "[install]\nprefix=" > setup.cfg; ' + pip_cmd
if CONFIG.OS == 'linux' and not CONFIG.DISABLE_VENDOR_FLAGS:
# Fix for https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=830892
# tl;dr: Debian has broken --target with a custom patch, the only way to fix is to pass --system
# which is itself Debian-specific, so we need to find if we're running on Debian. AAAAARGGGHHHH...
pip_cmd = f'[ -f /etc/debian_version ] && SYS_FLAG="--system" || SYS_FLAG=""; {pip_cmd} $SYS_FLAG'
pip_cmd += f' {repo_flag} {index_flag} {pip_flags} {package_name}'
if strip:
pip_cmd += ' && find . %s | xargs rm -rf' % ' -or '.join(['-name "%s"' % s for s in strip])
if install_subdirectory:
pip_cmd += f' && touch {target}/__init__.py && rm -rf {target}/*.egg-info {target}/*.dist-info'
if patch:
patches, c = _patch_cmd(patch)
pip_cmd += ' && ' + ' && '.join(c)
if post_install_commands:
pip_cmd = ' && '.join([pip_cmd] + post_install_commands)
tools = {'pip': [CONFIG.PIP_TOOL]} if CONFIG.PIP_TOOL else {'python': [CONFIG.DEFAULT_PYTHON_INTERPRETER]}
srcs = build_rule(
name = name,
tag = "srcs",
cmd = pip_cmd,
test_only = test_only,
output_dirs = [pip_target_dir],
srcs = patches if patch else [],
building_description = 'Fetching...',
tools = tools,
sandbox = False,
labels = ["dlink:plz-out/python/venv"]
)
# Don't include the dependency whl's into our whl. They are later picked up by py_binary anyway.
cmd = f'$TOOLS_JARCAT z --suffix="" --exclude_suffix=whl --include_other -i . -r $PKG/{name}:$PKG'
if not licences:
cmd += ' && find . -name METADATA -or -name PKG-INFO | grep -v "^./build/" | xargs grep -E "License ?:" | grep -v UNKNOWN | cat || true'
return build_rule(
name = name,
cmd = cmd,
outs = [name + '.whl'],
srcs = [srcs],
deps = [] + deps,
building_description = 'Collecting...',
hashes = hashes,
test_only = test_only,
licences = licences,
tools = {
'jarcat': [CONFIG.JARCAT_TOOL],
},
post_build = None if licences else _add_licences,
labels = ['py', 'pip:' + package_name] + (['py:zip-unsafe'] if not zip_safe else []),
visibility = visibility,
)I've got a PR here (#2210) |
Beta Was this translation helpful? Give feedback.
-
|
I think it's something in the existing pip_library definition that causes the duplicate directories. You can replicate it with the please-examples repo. (Note this doesn't finish cleanly for me because there's no precompiled protoc binary for M1 Macs, but it generates enough in the plz-out directory to see the effect.) I think it's this line that creates the outer directory: https://github.com/thought-machine/please/blob/master/rules/python_rules.build_defs#L388 Then pip installs the package in that directory, which has the effect of duplicating the directory. But I agree that creating a virtualenv directory is a better solution. That will also allow running the python program locally without having to use "plz run". I tried your new definition for pip_library but got this error: These are the definitions for those two packages: But I'm not sure why (or what) is trying to copy N.B. The protoc issue I mentioned in the previous comment should go away from 3.20: |
Beta Was this translation helpful? Give feedback.
-
|
@Tatskaari, I tested this and it doesn't quite work. Only 1 pip_library gets linked to python/venv and then the remaining ones are skipped because this line is checking if the destination exists and it does already after the first lib creates python/venv. |
Beta Was this translation helpful? Give feedback.
-
|
For reference, I opened this issue: #2309. I also have a possible fix to PR. Asking about it here: https://gitter.im/please-build/Lobby?at=62102bc88db2b95f0a2bfa13. |
Beta Was this translation helpful? Give feedback.
-
I'm really enjoying please and trying to make it my main build tool.
The problem I'm currently facing is getting Visual Studio Code to recognise dependencies from my third_party directory.
In my .plzconfig I've set it up so I can import things without the
third_party.pythonprefix:and in my VS Code settings.json I've set it to look in plz-out to find the dependencies:
This works for most dependencies, but it fails to find sub modules, I think because of the directory structure in plz-out. The dependency is facenet-python:
and the imports are:
Pylance can resolve the first line, but can't find the second line. The generated directory structure is:
It's the duplicate "facenet_pytorch/facenet_pytorch" directory that's causing the problem, i.e. if I change the second import to:
from facenet_pytorch.facenet_pytorch.models.mtcnn import fixed_image_standardizationbut I obviously don't want to do that.
Any ideas?
Beta Was this translation helpful? Give feedback.
All reactions