Move examples out of package directory

[wigging]

The current structure of the WaterTAP project is shown below. The directory that represents the WaterTAP package is at watertap/watertap/; the contents of this directory is what gets published to PyPI as a Python package.

watertap/
|-- bin/
|-- docs/
|-- tutorials/
|-- watertap/
|     |-- examples/
|          |-- chemistry/
|          |-- custom_model_demo/
|          |-- edb/
|          |-- flowsheets/
|-- README.md
|-- setup.py

When the released version of WaterTAP is installed from PyPI using pip install watertap, the examples also get installed with the package. This adds about 3.5 MB to the watertap package. As you can see in my Conda environment, an examples directory is seen inside the package at:

/opt/miniconda3/envs/watertap2/lib/python3.10/site-packages/watertap/examples

This occurs because the examples are located inside the WaterTAP package directory at watertap/watertap/examples. The examples directory should be moved out of the package directory and to the top level of the project. This prevents the examples from getting installed with the package. The watertap package itself does not provide any mechanisms for a user to directly run the examples so there's no need to include them within the package.

To prevent the examples from getting installed with the package, I propose the project structure shown below.

watertap/
|-- bin/
|-- docs/
|-- examples/
|     |-- chemistry/
|     |-- custom_model_demo/
|     |-- edb/
|     |-- flowsheets/
|     |-- tutorials/
|-- watertap/
|     |-- ...
|-- README.md
|-- setup.py

Unfortunately, the current watertap package seems to rely on code and tests that are located in the examples directory. All of the code that the package relies on should be moved into the package directory and removed from the examples directory.

Providing package examples in a separate repository or at the top level of a Python project is the expected approach of the Python community. This is demonstrated by packages such as NumPy, SciPy, Pandas, Matplotlib, Seaborn, Django, and Flask. They do not put the examples directory inside the package. These packages put examples in a separate GitHub repository or in a top level project directory. Examples are not part of the package's source code. Some domain specific packages that also do this are Astropy, PyRK, Pycroscopy, liionpack, PyBaMM. As such, WaterTAP should adhere to best practices and not put examples inside the watertap package.

[wigging]

So based on some offline discussion, it sounds like the examples are more like plugins or extensions or utilities for watertap. If you look at watertap/examples/chemistry/chemical_scaling_utils.py, it contains helper functions for running tests. Why is this considered an example? Seems like it should be part of a utilities module inside of the watertap package.

[dangunter]

I agree that "real" examples need to teach something, though they may have dependencies that are merely functional. These should move to a different repo and be refactored entirely to use the same toolchain that IDAES will be using in a month or two (hopefully I get to it by then) -- I think it would be more efficient to wait until the IDAES side of it settles out and just copy.

[wigging]

It looks like the IDAES examples are in the examples-pse repository. Is this what you are referring to for the IDAES side of it?

[wigging]

As a first step, I suggest moving the examples out of the watertap source code and into a top level directory. This keeps the examples in the same repository and the tests for the examples can still run as changes are made to the watertap package. Moving the examples to a separate repo will make it more difficult to test the examples as changes are made to watertap. It would also require a separate GitHub Actions workflow to maintain.

[ksbeattie]

@wigging, that's right the IDAES examples are in the examples-pse repo, which was the result of an identical effort to move those examples out of the idaes src tree and have them "installed" as a separate "package" (quoted here since it never really used PyPi for that).

It's also the case (as mentioned by @dangunter here) that there has been a long effort (mostly done on the side by @dangunter and @lbianchi-lbl) to move that examples-pse repo into a new examples repo and use PyPi for installing the ideas examples as a truly separate package. That effort is nearly finished now (possibly to be announced with the idaes-pse May release). We're planning on seeing how that works for idaes and then decide how it might be applied to the other PSE+ / dependent projects (like WaterTAP, PARETO, etc.).

There are a variety of considerations that need to be taken into account when making changes like these. I can't recall all the technical ones in detail at the movement but often those are moot when compared to identifying who will own the new structure (managing releases, dependencies, testing, documentation, etc.) because of our very limited (and fractured across institutions) funding. That said, I'm optimistic that the move in idaes will bear fruit that can be applied to WaterTAP.

[wigging]

What are the advantages of putting the examples into a completely separate repo compared to having the examples at the top-level of the package's repo?

[andrewlee94]

The main reason IDAES did this was to make it easier for users to find them. If a user does a pip install, the package gets buried in the Python packages which can be hard to find. By making the examples a separate repository and package, we were able to give the users some control over where the examples ended up (i.e by not getting the examples as part of the pip install but through another means, such as git clone or the IDAES Command line tools).

[wigging]

What do you mean by the package is getting buried by Python packages?

the package gets buried in the Python packages

[andrewlee94]

The file path to where the files are located - they end up somewhere in the anaconda site packages which most users will have no idea about how to find (I don't even know where to go looking for those half the time). Examples are generally something where the user needs to look at the actual code, as opposed to just importing and running the code. Thus, we wanted to make sure users could put the files in a place of their choosing.

Now, at the moment that is a moot point for WaterTAP as the examples already end up in the site packages anyway if the user does a pip install.

[wigging]

@andrewlee94 Yes, that is exactly why I'm suggesting the watertap examples need to be moved out of the watertap package source directory.

[TimBartholomew]

Hey all just jumping in here. I think there's a couple things being talked about at once.

First, I think Gavin is just proposing moving the examples directory out of the source code. Currently it sits in watertap/watertap/examples, where he wants to move it to watertap/examples. So same repository, but out of the package source directory.

Second, I think several folks have mentioned that IDAES examples have moved around and that they are not under the idaes-pse repository. I don't think this is what Gavin is suggesting, I'm not sure what the pros and cons of that are, but that could be something we decide to do after we move the examples from the package source directory.

[wigging]

Yes, this is what I'm doing. I'm just moving the examples from watertap/watertap/examples to watertap/examples. The examples remain in the same repository but they are moved out of the package.