Examples: To pin mesa version or not to pin?

[jackiekazil]

In the process of working on this ticket: #963, I realized that our examples are going to continue to grow and grow. The purpose of the requirements.txt file in the examples is to make sure that the version of Mesa will always work with the model. When we end up with 100+ examples, do we want to try to make sure that all Mesa examples are able to work with the latest release? It seems like a losing battle that will keep slowing us down.

I would like to propose the following:

• We make sure all contributed examples are pinned to a version.
• We select a limited set to always keep up to date with main - maybe these are calls "highlighted_models" or "demo_models". These should be the classic ones where we focus on the demonstration of features. These models are not pinned on purposed because these are also used for testing of changes in Mesa.

Thoughts?

[Corvince]

I agree that it doesn't make sense that we try to manually keep up with the Version.

Imho we should pin the the version for every example. It would still be possible to test the models with the latest version and also possible to cook up a script that bumps the version number if testing succeeds

[rht]

These models are not pinned on purposed because these are also used for testing of changes in Mesa.

What I had in mind is that these should have been covered by test cases instead, because we can then make assertions of the values. With the examples also doubling as test cases, we can only check whether the code runs.
Though, the examples could also be used for integration testing, if we ensure some properties of the simulations output have certain values.

[jackiekazil]

@rht so are you agreeing with @Corvince then in suggesting everything is pinned and we target and then certain subset for testing and updating?

[rht]

Yes, for pinning everything to the most recent release, if all examples are moved to the examples repo. I think this has less maintenance overhead, of not having to keep tracking of 2 versions of the library in 1 repo.

The concern I had with implementing #963 (comment) probably can be solved with duplicating the code, if needed, or do import from a separate mesa_examples namespace, import mesa_examples; mesa_examples.Schelling, taken from the examples repo. Anyway, there should be workarounds, I think.

[rht]

Do we pin, or not pin? One more argument for pinning is that the examples at each Git revision are more reproducible (this is actually a general argument in support of pinning). And it's possible to do Git bisect to pinpoint a problem, should it happen.