Refactoring mesa.space

[adamamer20]

Roadmap

• Abstract SpaceDF #39
• Abstract DiscreteSpaceDF #40
• Abstract GridDF #41
• Concrete GridPandas #34
• Concrete GridPolars #35
• Concrete ContinousSpaceDF #36
• Concrete MultiSpaceDF #37
• Concrete GeoGridDF #38
• feat: Adding Concrete NetworkDF #47

Class Structure and Design

SpaceDF

• Base class for all spaces
• Defines a common interface for all spaces
• Stores agent positions in the _agents attribute in a DataFrame/GeoDataFrame. This approach avoids many missing values in the AgentSetDF and reduces confusion when multiple spaces of the same type are present (e.g., an agent in both a grid and in a social network)
• TODO: Implement a pos property for AgentContainer that performs an instant join with the space _agents DataFrame

MultiSpaceDF

• Stores the collection of spaces in a model, similar to AgentsDF

DiscreteSpaceDF

• Defines an interface for GridDF and NetworkDF
• Supports setting cell properties at both class level (setting an attribute of the class) and cell level (using the set_cells method)
• In addition to _agents stores cell properties in the _cells attribute in a DataFrame. The _cells DataFrame explicitly stores only cells that have agents or have specific properties different from the class level

ContinousSpaceDF

• Stores agents in a GeoDataFrame (only geopandas in the future)
• Bridges mesa and mesa-geo
• All agents are geoagents and shapely objects
• Can be used as a continuous space in mesa without setting CRS, or as a geospace in mesa-geo by setting CRS

GridDF

• Implemented in both GridPolars and GridPandas versions
• Supports multiple dimensions
• The '_empty_grid' attribute stores the remaining capacity of the grid.

GeoGridDF

• Position in the class hierarchy still to be determined
• Is it possible to create a unique class or attempt to merge with GridDF? Probably not possible to merge as GridDF supports multiple dimensions, while shapely objects are limited to 2 dimensions.

[rht]

Can't wait for this feature. This is the only blocker to me implementing mesa-frames version of Sugarscape constant growback. It boils down to implementing the move within the space: https://github.com/projectmesa/mesa-examples/blob/668eb974ffdaaf2d20ca9a5fd7ba828f1b043426/examples/sugarscape_cg/sugarscape_cg/agents.py#L40-L61. Only eat and die part of the step can be DF-vectorized as of now.

[adamamer20]

Can't wait for this feature. This is the only blocker to me implementing mesa-frames version of Sugarscape constant growback. It boils down to implementing the move within the space: https://github.com/projectmesa/mesa-examples/blob/668eb974ffdaaf2d20ca9a5fd7ba828f1b043426/examples/sugarscape_cg/sugarscape_cg/agents.py#L40-L61. Only eat and die part of the step can be DF-vectorized as of now.

Let me ensure I understood correctly the issue. The available spaces are the neighborhood spaces which are not occupied. The problem is that in the traditional Sugarscape model, since the step is sequential for agents, there is a race condition. If parallelized, we could have two agents that select the same space at the same time.
What if we were to run an iteration across all agents, select randomly one of the agents which have a duplicate neighborhood, and then run the choice algorithm again on the agents with duplicates that are not selected? This approach is not as efficient as parallelizing everything at once, but it shouldn't require many iterations to resolve conflicts.

[rht]

Yeah, that (update the position in parallel, then do tie-breaking) is the method used in the FLAME 2 implementation of Sugarscape instantaneous growback (i.e. simpler version of constant growback, where the sugar grows to max value in the next step right after being consumed; constant growback has a growing rate), in figure 6. Here is their code.

[adamamer20]

I see, that's very interesting that it's so recent! Do you think tie-breaking should be left to the user or handled by the library? Depending on the model, it can be either very simple or very complex to handle tie-breaking, similar to the challenge of vectorizing user-defined functions. Perhaps it would be better to include a section in the documentation on how to deal with race conditions?

[rht]

I think the basic version move_to_empty and move_to_empty_neighborhood should be handled by the library. They can be a starting point for users who want to implement more complex move logic. The former is used by the Schelling model. But the Boltzmann wealth model and Wolf sheep simply use moving to anywhere within the neighborhood.

[adamamer20]

@rht Do you think it would be better to convert DataFrames to GeoDataFrames with the respective geo-library if space is added to the model or simply adding a position/geometry column?
I think the first way would be much more powerful for the end-user but geopolars is currently 0.1.0-alpha4 and breaking changes are expected. Nevertheless, I would still go for the geolibraries.

[rht]

Which one is faster to implement? I have no strong opinion on either, as long as there could be a common API -- unlikely? Because Mesa-Geo and Mesa's current space.py have different API. If designing a common API is too much at this stage, my only requirement is that it shouldn't be too much boilerplate code for user who starts learning by doing Schelling or Boltzmann wealth model.

Note that in core Mesa, we are moving away from the old space.py implementation to a new cell-based implementation, https://github.com/projectmesa/mesa/tree/main/mesa/experimental/cell_space, that is conceptually simpler (see e.g. the get neighborhood method, https://github.com/projectmesa/mesa/blob/2cee49efeac07ea44342b918b2db6fb170649793/mesa/experimental/cell_space/cell.py#L132) but also sufficiently performant (the PR itself).

But the structure in core Mesa that is closer to Mesa-Geo is the new experimental PropertyLayer (the PR for more context and examples).

[adamamer20]

Thank you for the references; they are very useful. So you're moving to a cell-based space to achieve better granularity of the environment (i.e., by setting properties of each single cell with the PropertyLayer) and to create an easier abstraction for interacting with and creating different types of spaces more easily.

To create a unique API that aligns with the future of mesa and works for mesa-geo objects, we can create the cell collection in a GeoPandas DataFrame (GeoPolars could also be implemented in the future). GeoPandas allows the geometry column to be a series of shapely objects, representing the geometry of the cell itself (e.g., a box for a rectangular grid, a complex polygon for continuous spaces). The coordinate reference system (CRS) can also be set, aligning it with mesa-geo. The other columns of the DataFrame would contain the properties for each cell (e.g., maximum number of agents or other custom properties in the PropertyLayer).

I am unsure why there are so many classes with similar functionality but different implementations and interfaces in mesa. This complexity can make it difficult for end-users to understand and use multiple classes with custom behaviors, and it can also be challenging for developers to comprehend the layers of abstraction.

I propose creating a single unique (abstract?) class that stores cells in a GeoDataFrame, with other columns representing the "properties" of the PropertyLayer. This class would combine the functionalities of PropertyLayer/_PropertyGrid, CellCollection, DiscreteSpace, SingleGrid, and MultiGrid. From this abstract class, we could create child classes for each type of space (e.g., network, rectangular grid, etc.).

Additionally, I would move the move_to method of CellAgent to the AgentContainer.

I think the end result would mirror the structure of AgentSet and AgentsDF with CellSet and CellsDF respectively to implement multiple type of cells and store all cells in the model with GeoDataFrames instead of DataFrames.

[rht]

I am unsure why there are so many classes with similar functionality but different implementations and interfaces in mesa. This complexity can make it difficult for end-users to understand and use multiple classes with custom behaviors, and it can also be challenging for developers to comprehend the layers of abstraction.

Yes I agree, sorry for the mess (old space, property layer, cell space, and the implementations). Mesa was in a flux of development last year, and there is not enough explicit communication to the public on which one is recommended for the current best practice, which one should be used for integration, and so on. Docs, tutorials, and how-to's are lacking.
For clarity, in the cell_space folder, the concrete classes of grid.py and network.py should be in their own folder.

There is also a problem that the cell_space version of Mesa-Geo doesn't exist yet.

I propose creating a single unique (abstract?) class that stores cells in a GeoDataFrame, with other columns representing the "properties" of the PropertyLayer. This class would combine the functionalities of PropertyLayer/_PropertyGrid, CellCollection, DiscreteSpace, SingleGrid, and MultiGrid. From this abstract class, we could create child classes for each type of space (e.g., network, rectangular grid, etc.).

Yes, that would make sense for mesa-frames. In Mesa, the PropertyLayer is separate from grid/space because it is implemented as a np.ndarray, and operation on it would be a vectorized operation, which regular old space/cell space doesn't support.

Optional question: for the GeoDataFrame, how does it work for a sparse agents situation (e.g. 10 agents, in a grid of 100x100)? Wouldn't it be 10k rows of mostly empty agent content, but yet each must have a property? This is optional because I think we can still design a common abstract interface while postponing a more specific implementation.

Additionally, I would move the move_to method of CellAgent to the AgentContainer.

SGTM. The move_to was inspired by NetLogo natural-language-like syntax, so you could say agent.move_to(x) like a sentence would. It currently lives in CellAgent not for a deliberate reason of moving in only the discrete spaces and nothing else; it could have been in mesa.Agent as well, and implemented for GeoAgent.

[adamamer20]

Yes I agree, sorry for the mess (old space, property layer, cell space, and the implementations). Mesa was in a flux of development last year, and there is not enough explicit communication to the public on which one is recommended for the current best practice, which one should be used for integration, and so on. Docs, tutorials, and how-to's are lacking.
For clarity, in the cell_space folder, the concrete classes of grid.py and network.py should be in their own folder.

I think it's normal; mesa is evolving a lot, and keeping up with documentation is hard. However, I think it should be a priority for mesa 3.0. As a user familiar with mesa, I struggle to understand which classes to use, what has been deprecated or how things work without looking at the source code. I can also help to improve docs at the end of the GSOC period if we have time.

Optional question: for the GeoDataFrame, how does it work for a sparse agents situation (e.g. 10 agents, in a grid of 100x100)? Wouldn't it be 10k rows of mostly empty agent content, but yet each must have a property? This is optional because I think we can still design a common abstract interface while postponing a more specific implementation.

That is a very good point. I think there's a tradeoff here. Either we have very granular cells (i.e., each cell can have its own property) with a large memory usage for sparse models, or we have less granularity (each cell in the set has the same property) with more efficient memory usage (we could store geometries at the agent level and store property attributes for cells at the cell class level). What do you think works better?

Maybe we can have users using both.

[rht]

I think it's normal; mesa is evolving a lot, and keeping up with documentation is hard.

Yeah, it's a good problem to have (revamping the API). The old API is getting clunky to use.

I can also help to improve docs at the end of the GSOC period if we have time.

Don't worry about this for now. Focus on making mesa-frames more awesome, as there are lots of things to do, mainly the classic models, for Sampy use case, and using GPU to scale to millions of agents and huge space. Also, I'm still reading on hypothesis and deal.

Either we have very granular cells (i.e., each cell can have its own property) with a large memory usage for sparse models, or we have less granularity (each cell in the set has the same property) with more efficient memory usage (we could store geometries at the agent level and store property attributes for cells at the cell class level). What do you think works better?

From the user/API standpoint, I think we should have a sparse flag that is very simple to toggle (along the line of in NumPy where you can switch to a sparse matrix seamlessly). And from the developer standpoint, we will have to worry about the fast move_to_empty, get_neighbors (e.g. which loops over all agents and check their distance instead of the neighborhood).

[EwoutH]

Thanks for the feedback. I agree work is needed on the examples and tutorials. I think we're doing a good job with docstring though, but if you disagrees or find areas for improvement, please point them out!

I have created open issues with some resources for anyone that wants to improve the examples:

• Use new AgentSet functionality in examples mesa-examples#111
• Use new Cell Space functionality in examples mesa-examples#123
• Use new Discrete Event Scheduling in examples mesa-examples#124

And we have projectmesa/mesa-examples#118 incoming which uses both the AgentSet and the Cell Space.

If I can find the time I will also work on them somewhat myself. Unfortunately, that keeps being a big if.

[adamamer20]

Thank you for the references! I agree that docstrings are generally sufficient for understanding what a component does, but as you mentioned, examples are lacking. I almost always have to look at the source code to understand how to actually use it in practice. It's great that there are open issues to work on examples though! If I may make a suggestion, in addition to adding examples to mesa-examples, I would recommend including short examples in the docstrings, similar to what you often see in the polars, pandas, or numpy API documentation. These are great for quickly understanding what a component does.