Skip to content

Commit

Permalink
[uv] port pyright script from pip -> uv (#20199)
Browse files Browse the repository at this point in the history
## Summary & Motivation

Companion PR: dagster-io/internal#8518

Make `scripts/run-pyright.py` use `uv` instead of `pip`. Experiment with
size n=2 on BK shows a ~5-10m speedup in the OSS pyright step from
14m40s to (9m39s, 6m13s).

`uv` also integrates some utilities from `pip-tools` that may be able to
improve `make rebuild_pyright_pins` (though the process is already sped
up a lot by this PR). I'll attempt to incorporate this functionality in
a future PR.

## Notes on editable installs

For some reason, even though no editable installs are altered in the
pins, two imports in our examples started being flagged as type errors--
in both cases it was a sibling module import being flagged as
unresolvable. Must be some subtle difference in how editable installs
are being done between `pip` and `uv`, even with
`--editable-mode=compat` set. In any case I was able to resolve the
errors by making these two instances relative imports, which is arguably
better in these cases anyway as it doesn't implicitly assume a working
directory.

## Notes on `uv venv` python interpreter resolution

The most tricky part of this was discovering and correcting for the
python resolution logic `uv` uses to create virtual environments. Unlike
the built-in `venv` package, `uv` (because it does not actually use
python at all) _does not use the python it is installed under by
default_. That is:

```
$ which pip  # /some/pip (which goes with /some/python)
$ pip install uv
$ uv venv .venv  # the created venv may not be using /some/python!
```

This had significant consequences on BK internal. The images used in
internal contain a python at both:

- `/bin/python`
- `/usr/local/bin/python`

The one at `/usr/local/bin/python` is the one used for all our tests and
is the default target for `pip install`. But when creating a virtual
env, `uv` actually chooses `/bin/python` by default. At time of this PR,
the pyright env cannot be built against `/bin/python` (3.11.2) for
obscure reasons having to do with PEP 517 build isolation and the
failure of `pendulum==2.1.2` to accurately require `setuptools` as a
buildtime dependency 💀. There's also some other nonsense with `xmlsec`.
To fix these issues I had to point `uv` to `/usr/local/bin/python` when
creating the venv on BK internal.

## How I Tested These Changes

BK
  • Loading branch information
smackesey authored Mar 4, 2024
1 parent 0ac6235 commit fc28696
Show file tree
Hide file tree
Showing 6 changed files with 149 additions and 124 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ def build_repo_wide_pyright_steps() -> List[CommandStep]:
CommandStepBuilder(":pyright: pyright")
.run(
"curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain nightly -y",
"pip install -e python_modules/dagster[pyright] -e python_modules/dagster-pipes",
"pip install uv -e python_modules/dagster[pyright] -e python_modules/dagster-pipes",
"make pyright",
)
.on_test_image(AvailablePythonVersion.get_default())
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
from create_table import iris_dataset
from dagster_duckdb.resource import DuckDBResource

from .create_table import iris_dataset

# ruff: isort: split
# start_example
from dagster import asset

Expand Down
3 changes: 2 additions & 1 deletion examples/with_wandb/with_wandb/assets/example/train.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,10 @@
import torch.nn as nn
import torchvision.transforms as transforms
import wandb
from fashion_data import fashion
from torch.autograd import Variable

from .fashion_data import fashion

hyperparameter_defaults = dict(
dropout=0.5,
channels_one=16,
Expand Down
69 changes: 37 additions & 32 deletions pyright/alt-1/requirements-pinned.txt
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,9 @@ asn1crypto==1.5.1
astroid==3.1.0
asttokens==2.4.1
async-lru==2.0.4
async-timeout==4.0.3
attrs==23.2.0
Babel==2.14.0
babel==2.14.0
backoff==2.2.1
beautifulsoup4==4.12.3
bleach==6.1.0
Expand Down Expand Up @@ -76,6 +77,7 @@ distlib==0.3.8
docker==7.0.0
docstring-parser==0.15
duckdb==0.10.0
exceptiongroup==1.2.0
execnet==2.0.2
executing==2.0.1
fastjsonschema==2.19.1
Expand All @@ -86,7 +88,7 @@ frozenlist==1.4.1
fsspec==2024.2.0
gcsfs==2024.2.0
google-api-core==2.17.1
google-api-python-client==2.119.0
google-api-python-client==2.120.0
google-auth==2.28.1
google-auth-httplib2==0.2.0
google-auth-oauthlib==1.2.0
Expand Down Expand Up @@ -118,32 +120,32 @@ ipython==8.22.1
isodate==0.6.1
isoduration==20.11.0
isort==5.13.2
jaraco.classes==3.3.1
jaraco-classes==3.3.1
jedi==0.19.1
Jinja2==3.1.3
jinja2==3.1.3
jmespath==1.0.1
joblib==1.3.2
json5==0.9.17
json5==0.9.18
jsonpointer==2.4
jsonschema==4.21.1
jsonschema-specifications==2023.12.1
jupyter-client==8.6.0
jupyter-core==5.7.1
jupyter-events==0.9.0
jupyter-lsp==2.2.3
jupyter_client==8.6.0
jupyter_core==5.7.1
jupyter_server==2.12.5
jupyter_server_terminals==0.5.2
jupyter-server==2.12.5
jupyter-server-terminals==0.5.2
jupyterlab==4.1.2
jupyterlab_pygments==0.3.0
jupyterlab_server==2.25.3
jupyterlab-pygments==0.3.0
jupyterlab-server==2.25.3
keyring==24.3.1
kiwisolver==1.4.5
leather==0.4.0
Logbook==1.5.3
logbook==1.5.3
makefun==1.15.2
Mako==1.3.2
mako==1.3.2
markdown-it-py==3.0.0
MarkupSafe==2.1.5
markupsafe==2.1.5
mashumaro==3.12
matplotlib==3.8.3
matplotlib-inline==0.1.6
Expand All @@ -154,7 +156,7 @@ mistune==3.0.2
mock==3.0.5
more-itertools==10.2.0
morefs==0.2.0
msgpack==1.0.7
msgpack==1.0.8
multidict==6.0.5
multimethod==1.11.2
mypy==1.8.0
Expand All @@ -166,7 +168,7 @@ nbformat==5.9.2
nest-asyncio==1.6.0
networkx==3.2.1
nodeenv==1.8.0
notebook_shim==0.2.4
notebook-shim==0.2.4
numpy==1.26.4
oauth2client==4.1.3
oauthlib==3.2.2
Expand All @@ -185,11 +187,12 @@ pathspec==0.11.2
pendulum==3.0.0
pexpect==4.9.0
pillow==10.2.0
pip==24.0
platformdirs==3.11.0
pluggy==1.4.0
polars==0.20.13
-e examples/project_fully_featured
prometheus_client==0.20.0
prometheus-client==0.20.0
prompt-toolkit==3.0.43
protobuf==4.25.3
psutil==5.9.8
Expand All @@ -204,11 +207,11 @@ pyasn1==0.5.1
pyasn1-modules==0.3.0
pycparser==2.21
pydantic==2.6.3
pydantic_core==2.16.3
Pygments==2.17.2
PyJWT==2.8.0
pydantic-core==2.16.3
pygments==2.17.2
pyjwt==2.8.0
pylint==3.1.0
pyOpenSSL==24.0.0
pyopenssl==24.0.0
pyparsing==3.1.1
pyright==1.1.349
pyspark==3.5.1
Expand All @@ -220,13 +223,13 @@ pytest-mock==3.3.1
pytest-rerunfailures==10.0
pytest-runner==5.2
pytest-xdist==3.3.1
python-dateutil==2.8.2
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
python-json-logger==2.0.7
python-slugify==8.0.4
pytimeparse==1.1.8
pytz==2024.1
PyYAML==6.0.1
pyyaml==6.0.1
pyzmq==25.1.2
referencing==0.33.0
requests==2.31.0
Expand All @@ -243,15 +246,16 @@ s3transfer==0.10.0
scikit-learn==1.4.1.post1
scipy==1.12.0
seaborn==0.13.2
Send2Trash==1.8.2
send2trash==1.8.2
setuptools==69.1.1
six==1.16.0
slack_sdk==3.27.1
slack-sdk==3.27.1
sniffio==1.3.1
snowflake-connector-python==3.7.1
snowflake-sqlalchemy==1.5.1
sortedcontainers==2.4.0
soupsieve==2.5
SQLAlchemy==1.4.51
sqlalchemy==1.4.51
sqlparse==0.4.4
stack-data==0.6.3
starlette==0.37.1
Expand Down Expand Up @@ -282,22 +286,22 @@ types-mock==5.1.0.20240106
types-paramiko==3.4.0.20240205
types-pkg-resources==0.1.3
types-protobuf==4.24.0.20240129
types-pyOpenSSL==24.0.0.20240228
types-pyopenssl==24.0.0.20240228
types-python-dateutil==2.8.19.20240106
types-pytz==2024.1.0.20240203
types-PyYAML==6.0.12.12
types-pyyaml==6.0.12.12
types-requests==2.31.0.6
types-simplejson==3.19.0.20240218
types-six==1.16.21.20240106
types-SQLAlchemy==1.4.53.34
types-six==1.16.21.20240301
types-sqlalchemy==1.4.53.34
types-tabulate==0.9.0.20240106
types-toml==0.10.8.7
types-tzlocal==5.1.0.1
types-urllib3==1.26.25.14
typing-extensions==4.10.0
typing-inspect==0.9.0
typing_extensions==4.10.0
tzdata==2024.1
universal_pathlib==0.2.1
universal-pathlib==0.2.1
uri-template==1.3.0
uritemplate==4.1.1
urllib3==1.26.18
Expand All @@ -311,6 +315,7 @@ webcolors==1.13
webencodings==0.5.1
websocket-client==1.7.0
websockets==12.0
wheel==0.42.0
wrapt==1.16.0
yarl==1.9.4
zipp==3.17.0
Loading

0 comments on commit fc28696

Please sign in to comment.