Python client -- issues with version coupling and automatic releases

[melange396]

The version of the delphi_epidata python client is bound to the version of the repository. This means that a new version of the client is automatically published to PyPI whenever we do a release of delphi-epidata, even if the client code has not actually changed. This can confuse or annoy users, as it might encourage them to do frequent but unnecessary upgrades. There are also concerns regarding what to do if we make changes to the client interface that might break users' existing workflows -- in such a case, Semantic Versioning necessitates incrementing the "major" version number, but this may be undesirable to apply to the entire repository.

We can address this in a number of ways:

• Uncouple versions between the two, which lets us number the client version however we need or want. This would involve removing the client from the existing "bumpversion" configuration, and would require creating a different release process (either automatic or manual -- automatic being more work up front, while manual requires remembering to perform some number of extra steps when changing the client).
• Keep the versions coupled, but only do the PyPI release if/when the client code actually changes (again, either automatic or manual with the same caveats). Changes to the client (other than just version number) would be found in the file https://github.com/cmu-delphi/delphi-epidata/blob/dev/src/client/delphi_epidata.py and anything under the directory https://github.com/cmu-delphi/delphi-epidata/tree/dev/src/client/packaging/pypi . I suggest this path, at least initially.

This discussion came up in #1288. Additional points are made and details listed in that PR, starting with comment #1288 (comment) .

Similar concerns may exist for the other clients in this repo.

[melange396]

We are going to do the second bullet point above, as it integrates well with checking against the most current PyPI release as mentioned in #1281 (comment)

[rzats]

@melange396 one quick question before I create a PR for this!

Ideally I'd like to automatically release the Python and JS clients if the relevant files have been changed, and there are GitHub Actions that enable this. However:

Changes to the client (other than just version number) would be found in the file https://github.com/cmu-delphi/delphi-epidata/blob/dev/src/client/delphi_epidata.py and anything under the directory https://github.com/cmu-delphi/delphi-epidata/tree/dev/src/client/packaging/pypi

Currently, a couple files in these paths have bumpversion enabled:

[bumpversion:file:src/client/delphi_epidata.py] (recently changed)

[bumpversion:file:src/client/packaging/npm/package.json] (managed by bumpversion before)

[bumpversion:file:src/client/packaging/pypi/setup.py] (managed by bumpversion before)

So they would be changed with every release either way.

I have two potential solutions for this:

1. Do not automate the client release, instead simply decouple the Python & JS release steps as an extra manual workflow (and modify the release guide to mention this)

• This decoupled workflow would simply work like create-release.yml - put in a version number to get a Python and/or JS release.

2. Rethink how we single-source the version again. This would mean excluding delphi_epidata.py, package.json and setup.py from bumpversion, and figuring out another way to update the versions there.

• As Dmitry has mentioned, there are a couple suggested ways of doing this. The 4th suggestion (a separate VERSION file) might make sense, as it is supported by JavaScript as well.

Which do you think is preferable?

[melange396]

There are other possibilities too:

• (1) Make the python client release steps a conditional part of the regular release process, but only when there are changes to https://github.com/cmu-delphi/delphi-epidata/blob/dev/src/client/packaging/pypi/CHANGELOG.md . This will have a side-effect benefit of ensuring people properly document changes to the client before it gets released.
• (2) Allow the python client to have its own version progression, not linked to the version of the acquisition and server code. This is how it is done for the delphi_utils package in covidcast-indicators. I dont love this (and im not exactly sure why, besides the drawback i mention below), but there is some logic to having the separation.
  • There are two .bumpversion.cfg files, one for delphi_utils and one for the rest.
  • The delphi_utils version bumping and PyPI release are conditional on changes to those files (here and here).
  • AFAICT this solution has a drawback in that its not currently possible to do anything but a "patch" release for delphi_utils and not the rest, or vice-versa.
• (3) Semi-detatch the versioning, syncing the client version to the base version and doing the PyPI release only on client code changes. This can be accomplished like so:
  • Take the two python client file mentions out of .bumpversion.cfg
  • Put them into their own at src/client/packaging/pypi/.bumpversion.cfg (with lines for [bumpversion:file:setup.py] and [bumpversion:file:../../delphi_epidata.py], hopefully itll accept the "up" directory pathing).
  • Conditionally initiate a bump2version command in https://github.com/cmu-delphi/delphi-epidata/blob/dev/.github/workflows/create-release.yml using that .../pypi/ as the working directory, when changes are detected in any of the client files...
    • but dont run it with ${{ github.event.inputs.versionName }} as the version argument;
    • use ${{ steps.version.outputs.next_tag }} instead.
  • Finally, in https://github.com/cmu-delphi/delphi-epidata/blob/dev/.github/workflows/release-helper.yml , make the PyPI step again conditional on changes to the client files.

I like the last one best but i could be convinced to go along with the CHANGELOG-dependent one.

[rzats]

@melange396 I've implemented your last idea in #1465!