Add CI requirements update script

[texodus]

No description provided.

[timkpaine]

Can we use this opportunity to move the requirements to a folder and exclude the folder from distribution (in MANIFEST.in)

[texodus]

Why?

[edit] Even if they were disted - why is that a bad thing?

[timkpaine]

They're only used for testing, and it's confusing because many other libraries use similarly named files as their runtime deps

[texodus]

What is the difference between CI deps and runtime deps?

[timkpaine]

When an end user does pip install perspective-python and installs perspective from pypi, they will pull down a dynamic set of dependencies based on their currently installed dependencies and perspective's defined bounds. These are our runtime deps. In CI, we don't do this, we install a fixed set of known-good dependencies to run lints and tests against. These are our CI deps.

Our runtime deps are defined in setup.py, but some repos use requirements.txt. I think it would be good to clearly put these deps in their own folder so people don't look and think "Oh im on Python3.8, perspective only supports the fixed set of pinned dependencies in requirements-38.txt and that doesn't work for me". A good example of this is airflow, which creates the same type of "known-good" requirements files as we do, but they put them in their own space: https://github.com/apache/airflow/tree/constraints-2-7.

[texodus]

You'd have to dig 2 levels deeps into our source repo to even know these files exist. How is moving them to 3 levels deep going to correct developers' expectations?

[timkpaine]

They're at the root of our python project. Its a tiny change and it doesn't seem like its a big deal from your perspective, so I think its a good thing to do.

[texodus]

pandas has the same configuration we have - developer/ci requirements in requirements-dev.txt in the root of the project next to setup.py, and no one is confused. Moreover, the pip documentation says that the file name can be anything, and mentions absolutely nothing about conventional use.

I will change this because I'm tired of discussing it, but I'm doing so under protest.

[dhirschfeld]

As a user who is heavily involved in python packaging, I can confirm that there is a strong convention for storing runtime dependencies in a file named requirements.txt.

Using a file named requirements.txt for anything other than runtime dependencies would be quite surprising/confusing.

The convention is also extended to optional dependencies (or extras) where requirements-<name>.txt maps to the optional dependency group <name>. It is quite often that you will see requirments-dev.txt, requirements-test.txt and requirements-docs.txt files alongside the requirements.txt file.

This convention is strong enough that there is a hatch-requirements-txt plugin for the hatch packaging tool to dynamically map the contents of these files into the pyproject.toml.

requirements-38.txt isn't requirements.txt but perhaps it's close enough to still cause confusion. If it costs nothing to reduce the possibility of confusion, that's probably a good thing.

[tomjakubowski]

I'm not sure what pandas requirements-dev.txt at their project root is used for, but in CI they use sets of requirements from a ci/deps folder. Their per-python version dependencies specified with conda environment.yml files, which is similar to what we're doing with requirements.txt.

https://github.com/pandas-dev/pandas/tree/main/ci/deps

I think it's nice when CI-specific stuff is kept in a separate, clearly labeled place like that. Makes it easy to find them when you need them, and also easy to ignore when you're just doing local development.