Growing an ABM Ecosystem

[tpike3]

I have been researching the idea of what a simulation ecosystem would look like to make it easy to use for rapid simulation development.

To that end I have built two prototype repositories:
Mesa Data
Mesa Behaviors

Working with some collaborators we also wrote an article that was picked up by Simulation: Growing the simulation ecosystem: introducing Mesa Data to provide transparent, accessible, and extensible data pipelines for simulation development also available from arXiv

[jackiekazil]

excited to see the work go forward -- this is awesome stuff -- my only concern with these is that is doesn't allow for anyone to feel like they have ownership -- it needs to be a site off of github where people can self register for that... like how Django Packages does it -- https://djangopackages.org/

[wang-boyu]

Would it make sense to integrate Mesa Data as built-in datasets?

For example GeoPandas has some built-in datasets so that you can do the following to easily load them:

geopandas.read_file(gpd.datasets.get_path('naturalearth_cities'))

Similarly in Seaborn:

seaborn.load_dataset("fmri")

This is a similar idea to #1228.

[jackiekazil]

@wang-boyu I like that idea, but it should still be an additional download at the very least.
Also -- there should be though on how does one add/grow additional datasets. If 20 people wanted to add a set -- how are they added? What is added? Now, what if that is 100 people?

[rht]

The GeoPandas example dataset can be found in https://github.com/geopandas/geopandas/tree/51864acf3dd0bcbc74b2a922c6e012d7e57e46b5/geopandas/datasets. There are only a few of them, and hence why it is manageable. The biggest file is about 600 KB.

We can do both vendoring small sample datasets (in the Git repo) and huge dataset (in a website, e.g. https://www.nltk.org/nltk_data/, see also https://hash.ai/data?sort=popularity).

[wang-boyu]

Seaborn's data is here as a separate repo: https://github.com/mwaskom/seaborn-data

We could probably start with small datasets and focus on the API or use cases first. Thereafter if it works well and there're more demands, we can think about alternative ways of storing the data.

[rht]

Yeah, sounds good.

[tpike3]

My impression is PyData (part of NumFocus) is a conference, but do they have affiliated data repos or do you know if they are looking at such a thing (@jackiekazil )? My impression has always been most data sources lead back to a few key datasets. One example being population data always seems to lead back to landscan data. So my hopeful thought is if some data repos already exists via NumFocus/PyData then we can link pipelines to them and get some economies of scale.

[jackiekazil]

@tpike3 my knowledge of PyData is just a conference. I don't know of general data stores that can be shared other than the data resource owners themselves -- eg Open street map for map data and Census for US populations, etc etc.