The future of data collection #1944
Replies: 9 comments 52 replies
-
|
Here is an illustration of the API overlap with AgentSet "gini":collect(model.agents, "wealth", function=calculate_gini)with AgentSet "gini": lambda model: calculate_gini(model.agents.get("wealth))"n_quiescent":collect(model.get_agents_of_type(Citizen), "condition", func=lambda x: len(entry for entry in x if entry=='Quiescent'))with AgentSet "n_quiescent": lambda model: len(model.agents.select(agent_type=Citizen, filter_func=lambda a: a.condition == "Quiescent")) |
Beta Was this translation helpful? Give feedback.
-
I think this may refer specifically to nested conditionals. While a chain of method calls should be considered to be something along the line of the paradigm of the UNIX pipes, that are found in Bash pipes, Golang interfaces, and DataFrame composition of operations. |
Beta Was this translation helpful? Give feedback.
-
Has a problem in that |
Beta Was this translation helpful? Give feedback.
-
Exactly, but that can be left to the internal logic of the DataCollector class to sort out. For example, you built up an internal |
Beta Was this translation helpful? Give feedback.
-
|
Even if it can work under the hood, even if the behavior can be documented, I still find it problematic to deviate from the known behavior of Python dict, with the dict elements becoming ordered, and the next elements that may depend on the previous element being defined first. The dict elements should have all been defined simultaneously. |
Beta Was this translation helpful? Give feedback.
-
|
I am not sure what you are trying to say. I was trying to describe the internal logic of the It seems that you are saying that the behavior of a class ( |
Beta Was this translation helpful? Give feedback.
-
Just for reference, this information is outdated. Python dictionaries used to be unordered. In Python 3.6 insertion order became in implementation detail of CPython (the reference implementation of Python). But since Python 3.7 insertion order is guaranteed, so it is perfectly fine to rely on it. That said the mental model for dictionaries is still set-orientaded (which I think is the right model). So I agree that it would be confusing if this works DataCollector(model, collectors={"wealth": collect(model.agents, "wealth"),
"gini": collect("wealth", func=calculate_gini)}
})but this doesn't DataCollector(model, collectors={"gini": collect("wealth", func=calculate_gini),
"wealth": collect(model.agents, "wealth")}
})So we still would have to work around this problem internally which complicates the code. But I don't think we need tiered data collectors at all. I think they are a bit hard to understand and provide little benefit. At least how I understand it, they are basically a performance optimization, so you don't need to loop over all agents more than once. For small to medium models I don't think its a problem at all. For larger models or if you really do lots of simulation runs, yes it can matter. But than a better solution would anyway be to calculate your derivate variables afterwards. That is you just collect the wealth attribute, turn your data collection into a pandas dataframe and calculate the gini coefficient from the dataframe. That probably is even faster, because pandas can parallelize the calculations across all rows. This way you also don't mix any logic into your data collector. I think it is actually bad practice to calculate things in the data collector. It should basically an observer. If you have the gini coefficient in your model definition feel free to collect it. Otherwise calculate it as part of the data analysis. So for me the callable should be only used to filter your objects (e.g. only a certain type, or based on a condition) |
Beta Was this translation helpful? Give feedback.
-
I haven't tried a quick implementation, but it can be done in just five lines of code, so it is not that complicated from a code base point of view. The understandability of the API to the user is a more significant concern.
I agree with this in principle. However, from practical experience, I know it can be convenient to support it (for example, to reduce the amount of data that needs to be stored, not needing to touch the source code itself if you want to quickly track something on the fly). It also would be a backward incompatible change. |
Beta Was this translation helpful? Give feedback.
-
|
I think users shouldn't be imposed on whether they should or should not process the raw data they have collected. With a small change in the API model.datacollector.collect({"wealth": lambda model: model.agents.get("wealth")})
.process({"gini": lambda data: calculate_gini(data["wealth"])})
.process({"wealth**gini": lambda data: data["wealth"] ** data["gini"]})
.store(["gini", "wealth**gini"]) # optional, if the user needs to store only a subset of the data collectedThis is no longer a tiered data collection, but rather a series of processing over the first collection, and so is easier to explain. |
Beta Was this translation helpful? Give feedback.
-
|
An API along these lines also occurred to me. What I like is that it is declarative and clear. The question for me is whether the user should specify what to collect and process when initializing the Or do we want to be even more radical and fully rerig datacollection on top of the observer design pattern? |
Beta Was this translation helpful? Give feedback.
-
|
This is my summary of the problems in the current data collector. I made a summary for the rest of @projectmesa/maintainers. Needs your opinion so that this can happen in time just before the 3.0 release. I think this should not be a GSoC 2024 project. Data collection problems:
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks for this useful summary. I broadly agree. Some additional thoughts below On 5. Yes, it is important that the datacollection mechanism is open for extension so users can easily replace the default behavior with their own custom (database) solution. The question is whether to do this at the level of each individual collect call, at the level of a complete run, or both. |
Beta Was this translation helpful? Give feedback.
-
Users sometimes may want to get all the data collected of a particular group. With the current data collector, it would be
I think @EwoutH wants an easy (minimal ceremony) way to apply statistics on the measures. The "object" in the current data collector is too indirect to be manipulated. I think your idea (and in @Corvince's wish list) of making it a concrete object solves this issue, because it allows one to do Optional comment: This seems to stem from the fact that Python's dict-key and Python object-attribute are not fully equivalent, unlike in JavaScript where I can specify an object as a dictionary const quiescents = {wealth: ..., x: ..., y: ...}And I'd be able to access wealth via |
Beta Was this translation helpful? Give feedback.
-
Only as a last resort. But the idea of measure as a concrete object/attribute of a model, seems to render this unnecessary. |
Beta Was this translation helpful? Give feedback.
-
You actually already wrote this down before I did. And I somehow missed it. So point 9 already has a proposed solution. |
Beta Was this translation helpful? Give feedback.
-
|
I suggest we try to contain the discussion on DataCollection here rather than having it spread over multiple locations. I am getting confused trying to find all the useful ideas and discussions. So rather than respond in #1933, I'll respond here. In 1933, @rht wrote
I am not entirely sure about this. Dataframes, for me, are associated with analyzing the results of a run. So, in my branch, Measures in my understanding are
So is State a single thing, or can it be multiple things? For example, an agent's position is clearly part of the agent (and by extension) model state. However, most of the time, position will be some tuple. So, somewhere, we have to translate the position into its elements. Do we want to do this in Measure, which would imply having multiple "fields" in a measure, or do we handle this downstream wherever Measure is being used? I personally am inclined to handle this further downstream. To continue the position example, in data collection, we might want to split position into x, y, (and z). For visualization, however, this splitting might not be required. So, I am unsure if we need multiple attributes/functions on a Measure. Instead, in my current thinking Measure always reflects a single state variable. |
Beta Was this translation helpful? Give feedback.
-
|
I should have provided a problem that I want to solve, which is problem 8 in #1944 (comment). I think measure-containing-multiple-attributes/functions may solve problem 8, because it could be considered a preemptive organization method, instead of downstream, for the user to group-by attributes/functions for 1 group in 1 measure. The problem with the user doing group-by at downstream, is that while they may be aware which measures are from the same group, the Mesa framework isn't, because it is not specified explicitly.
How does multi-indexing work? An example to see it in action would help. Between the 2 choices, I prefer whichever has an overall simpler structure, both user facing (easy to conceptualize) and the underlying code. |
Beta Was this translation helpful? Give feedback.
-
This is not a problem. I suppose the framework does have an awareness of each measure's group. If I implement a function combine_into_one_dataframe(measure1, measure2, ...)in #2024's implementation, I can always check each measure's |
Beta Was this translation helpful? Give feedback.
-
I meant pandas multi indexing
That is basically in line with how I have approached it in my branch. Each |
Beta Was this translation helpful? Give feedback.
-
|
A quick update from my side. I have been trying to figure out a way to make it possible to access the value of Measure as an attribute. So the basic idea is that the following code works. class Measure:
def __init__(self, group, function):
self.group = group
self.function = function
def get_value(self):
return self.function(self.group)
class MyModel(Model):
def __init__(self, *args, **kwargs):
# some initiliaziation code goes here
self.gini = Measure(self.agents, "wealth", calculate_gini)
if __name__ == '__main__':
model = MyModel()
print(model.gini) # should actually do model.gini.get_value()This turns out to be not trivial because in this example class Measure:
def __init__(self, model, identifier, *args, **kwargs):
self.model = model
self.identifier = identifier
def get_value(self):
return getattr(self.model, self.identifier)
class MeasureDescriptor:
def __set_name__(self, owner, name):
self.public_name = name
self.private_name = "_" + name
def __get__(self, obj, owner):
return getattr(obj, self.private_name).get_value()
def __set__(self, obj, value):
setattr(obj, self.private_name, value)
class Model:
def __setattr__(self, name, value):
if isinstance(value, Measure) and not name.startswith("_"):
klass = type(self)
descr = MeasureDescriptor()
descr.__set_name__(klass, name)
setattr(klass, name, descr)
descr.__set__(self, value)
else:
super().__setattr__(name, value)
def __init__(self, identifier, *args, **kwargs):
self.gini = Measure(self, "identifier")
self.identifier = identifier
if __name__ == '__main__':
model1 = Model(1)
model2 = Model(2)
print(model1.gini)
print(model2.gini)To make I hope this explanation is clear enough. I admit it is a bit convoluted. It is also one of the only ways I have been able to come up with so far that makes it possible for Measures to behave as if they are normal attributes. Please let me know what you think of this direction for implementing Measure or whether the complexity is not worth it, and we forego the idea of having Measure behave as if it is an attribute that returns a simple value (e.g., int, float, string). |
Beta Was this translation helpful? Give feedback.
-
|
I'd say the ReactJS-state API of the |
Beta Was this translation helpful? Give feedback.
-
Ok, this is slightly different from what I was trying to achieve and, indeed, easier. The consequence of going this route is that at the level of Collectors, you need to differentiate between collectors that collect attributes and collectors that collect measures. The former can do I am not familiar with either Solara's At the moment, I am trying to develop a conceptual design and working implementation for data collection that does not rely on pub/sub. Ideally, this design is such that the public API can stay in place even if the implementation is switched to pub/sub. Once we all agree on the conceptual design, I'll try to demonstrate how the implementation becomes much easier if you use pub/sub (see @EwoutS comment in #1947). |
Beta Was this translation helpful? Give feedback.
-
|
@Corvince what is your take on the ReactJS angle on the Measure class? I asked ChatGPT about the similarity/difference between ReactJS state management and pub/sub pattern.
Difference:
|
Beta Was this translation helpful? Give feedback.
-
|
I think the ChatGPT answer regarding the differences is pretty good. But first on useState and reactive. Reacts useState and solaras reactive are fundamentally different strategies to solve the same problem. The latter is not the former combined in a single object. Most importantly useState can only be used inside a component, or, in solara terms with a rendering context (that @solara.component provides). It returns a plain Python object with no magic attached. What triggers state updates are the side effects of the set_* function, which "inform" the rendering context of the state update. On the other hand solaras reactive can be used anywhere and is not bound to any component, providing global state (there is an equivalent use_reactive for local state). But again, something needs to track these changes. So if you access a reactive variable inside a @solara.component, the component actually subscribes to the reactive object, which publishes its changes. So in summary, yes, solaras reactive and pub/sub are strongly related. UseState, not so much. You can read more about reactive in the Vue docs Regarding the API, after seeing the implementation of @quaquel I think the solution provided by @rht in his PR probably suitable. Especially it indeed could be turned into being reactive (or pub/sub) with the same API. Then again, maybe Measure and MeasureDescriptor can indeed be merged and we get the best of both worlds? |
Beta Was this translation helpful? Give feedback.
-
To the best of my knowledge, it is technically impossible to combine Measure and MeasureDescriptor. Descriptors are defined on classes, not on instances. So, in my example with 2 model instances, you have two different Measure instances (one per Model instance) but only 1 ModelDescriptor instance (on the Model class). This is also one reason for my code example's rather complicated |
Beta Was this translation helpful? Give feedback.
-
|
Thanks a lot for this. I think we're on the right track, I would only change the abstraction level on which the There are basically the following problems:
Then there is the complication that you sometimes have an object with members with attributes (like an AgentSet) and sometimes just have an object with attributes directly (like a Model). So basically there three levels that need to be defined:
You can already see how complicated this can possibly get. I will try to think about some possible abstractions, but feel free to built on this in the mean time. |
Beta Was this translation helpful? Give feedback.
-
|
This is a fair summary of the state of the conversation. At present, I guess we have identified four classes: DataCollector/CollectorContainer, Collector, Measure, and Group. Each of these might have several subclasses (e.g., AgentSetCollector). Let my try to summarize: Group is a collection of objects, typically agents, and can be static or dynamic. Dynamic groups are groups were membership changes over the course of the simulation. This can be because agents are added or removed from the model, or because membership is based on the agent being in some state. Measure reflects a state variable of the model at a given time instant. Ideally, it would be possible to access the value of a measure through attribute lookup Collector collects measures and attributes and stores them. This allows one to capture the change of them over the course of the simulation. DataCollector/CollectorContainer, in my thinking, is more of a convenience class. It contains all collectors, and a single collect call on it invokes collect on all collectors. |
Beta Was this translation helpful? Give feedback.
-
|
Quick update: I discussed this conceptual design with the scientific programmer working on several large MESA-based ABMs. Based on our discussion, this design seems to cover virtually all of their use cases. She particularly liked the ideas of dynamic/conditional Groups/AgentSets, the separation between Measure (part of the state of the Model at a given time instant) and Collector (tracking state over time), and the idea of making Collectors easily extendible on the storage side of things. |
Beta Was this translation helpful? Give feedback.
-
|
An interesting thought we had: It might be useful to let an "aggerate" collector collect another "group" collector. So for example, a group collector collects wealth, and then the aggerate collector aggerates to a single value. |
Beta Was this translation helpful? Give feedback.
-
That's point no. 6 in the summary: #1944 (comment) |
Beta Was this translation helpful? Give feedback.
-
It's great to have user validation of the idea, in that it covers complex use cases. Just needs to make sure the DynamicAgentSet is performant. Does she have an additional wish list regarding with the data collection? |
Beta Was this translation helpful? Give feedback.
-
|
On Group: I can see how groups can be used outside of the measure and data collection use case. They may be reused to organize agents step execution as well, e.g. if I want only the quiescent citizens in the Epstein civil violence to take certain actions. def step(self): # of a model
# Instead of
self.agents.select(agent_type=Citizen, filter_func=lambda a: a.condition == "Quiescent").do("rest")
# we do
self.quiescents.do("rest")What about doing addition on the groups # The drawback being this is not cacheable
(self.quiescents + self.injured_cops).do("rest")
# Needs to be
self.needs_rest = Group(self.quiescents + self.injured_cops)
self.needs_rest.do("rest") |
Beta Was this translation helpful? Give feedback.
-
The class would need to have a base set of agents (or default to model._agents if no base set is provided). Next You could expand on this, just as we discussed with Measure, by having some kind of updated flag that is reset each time tick, while having a force_update keyword to override the flag if necessary. This can be a performance saver if if the DynamicAgentSet/ConditionalAgentSet is used several times without any intervening state changes to any of the agents within the set.
I don't know enough about hash.ai to comment, but in principle pub/sub or the Observer design pattern is a a software design idea, and I would not conflate it with what I called a flow library which is a discrete simulation concept. |
Beta Was this translation helpful? Give feedback.
-
I wrote this in a way that could be easily misunderstood. I did not mean message passing in the way that hash.ai to be the only viable way instead of pub/sub. I wanted to say that it is one major contender to pub/sub as the backend implemententation. And so, would need to compare contrast between the 2 implementations. I have not decided on either. |
Beta Was this translation helpful? Give feedback.
-
|
hash.ai's agent messages are specifically for communication among agents that are not neighbors. They are not used for communicating between agents and non-agent objects, as such their design shouldn't necessarily be taken into consideration as a backend for dynamic AgentSet. The only other alternative is that for a condition-based membership, that it is always computed from scratch. This has a simple naive implementation, but not performant. |
Beta Was this translation helpful? Give feedback.
-
|
I have done some more work on pub/sub and datacollection in my datacollection branch. I hope to find time over the weekend to do an actual performance comparison between 2 implementations for ConditionalAgentSet.
Note that I prefer ConditionalAgentSet over DynamicAgentSet as a name. |
Beta Was this translation helpful? Give feedback.
-
|
I committed an alternative implementation along the lines of point 2. Initial testing doesn't show a massive performance difference between pub/sub and the naive implementation, at least for the Epstein model. |
Beta Was this translation helpful? Give feedback.
-
The problem is an extension/detailing of 6. Let me try to explain in a bit more detail one of the details I am currently stuck on. The basic idea of a Collector is that it retrieves one or more attributes from an object or collection of objects, and optionally applies a callable to it. The issue now is that there is no way to specify the return of this optional callable in the current design. This return matters because it affects how data is stored in the collector and how it will be turned into a dataframe in So, for example, we are retrieving One idea I had after the conversation with @EwoutH is that the entire problem is analogous to e.g., pandas.DataFrame.apply. In case of collecting data from a collection of objects and next applying a callable to it, the user should specify the "axis" over which this function will operate. If you operate over the "columns", you are aggregating the information across all objects, while if you operate over "rows", the function is applied to the collected data for each object separately. I hope this helps to clarify the issue. |
Beta Was this translation helpful? Give feedback.
-
Yes, I like this. You could even do both, in some order. I feel we're close! What's needed to get this done? |
Beta Was this translation helpful? Give feedback.
-
|
mainly time from my side I guess, but with my talk last week done, I hope to get back to this and devs. |
Beta Was this translation helpful? Give feedback.
-
|
Played a bit around a few days ago. Now that we have our very powerful AgentSet, API seems to be able to get simpler: datacollector = DataCollector(
collectors = [
c(target=Model, attributes=["n_agents"], methods=calculate_energy),
c(target=Wolf, attributes=["sheep_eaten"]),
c(target=Sheep, attributes=["age"], methods=calculate_energy),
c(target=model.agents, attributes=["energy"], agg={"energy": np.mean}),
]Few notes:
c(target=Model, attributes=["n_agents"], methods=calculate_energy),gives {
f"{Model.__name__}_{n_agents}": {...},
f"{Model.__name__}_{calculate_energy}": {...},
}
Just one approach. Don't know if it's the best. |
Beta Was this translation helpful? Give feedback.
-
|
It is indeed becoming simpler, but we are not there yet.
# partial and incomplete sketch of collect func. signature
def collect(name:str, target:Any, attributes:strList[str], func:callable):
... # logic goes here
wealth = collect("wealth", model.agents, "wealth")
gini = collect("gini", wealth_c, method="calculate_gini")
avg_energy = collect("average energy", model.agents, "energy", np.mean) # no need to use agg in agentset here. 3 I don't fully follow your |
Beta Was this translation helpful? Give feedback.
-
|
@EwoutH I like this. I think I like this a lot! Let me think a bit more about this, but I think this is very close! Not sure I share the concerns of @quaquel . But let me think about this a bit more |
Beta Was this translation helpful? Give feedback.
-
|
These remarks are not so much concerns, but more puzzle pieces that are still missing. I think the main one is point 1. How can we make it easy to say that you want to collect data for a given Agent class or a subset of agents that is dynamically changing over time? |
Beta Was this translation helpful? Give feedback.
-
There has been quite some discussion in various places about changing data collection. This is my attempt to think this through in some more detail. It is heavily inspired by a suggestion by @Corvince at some point.
In the general case, data collection is taking an object and extracting from this one or more attributes, and optionally applying a callable to this. It might involve only an object and a callable applied to this object in specific cases. This object can be the model, an agent, an agentset, a space, or some user-defined class.
So, it seems sensible to create a separate Collector class that implements this basic logic. Because the behavior of AgentSet is a bit different from other objects (i.e., AgentSet.get instead of relying on getattr), I believe it makes sense to have 2 Collector classes: BaseCollector and AgentSetCollector (PEP 20, flat is better than nested). Rather than burden the user with this distinction, it is possible to use a factory function (e.g.,
collect(obj, attrs, func=None)) to create the appropriate Collector instance.Ideally, data should only be extracted once. So, in the case of the Boltzman wealth model, the data collector should be smart enough to extract the
wealthattribute only once from the agentset. This can relatively easily be realized by maintaining an internal mapping of all objects and the attributes to be retrieved from them. Moreover, extracting all relevant attributes from a given object in one go might be possible to avoid unnecessary iteration. This would, however, require a minor update toAgentSet.getso thatattr_nametakes a string or list of strings.I believe it is possible to design and implement this new-style DataCollector so that the current one can be implemented on top of it for backward compatibility.
Like with the current DataCollector, data collection should happen whenever
data_collector.collectis called. However, I believe it is paramount that the data collector also always extracts the current simulation time. Only by having the simulation time for each call tocollectcan you produce a clean and complete time series of the dynamics of the model over time. In fact, these time stamps could become part of the index/column labels of the DataFrames when turning the retrieved data into a DataFrame.Like with the current DataCollector, it should be easy to turn any retrieved data into a DatafFrame. This can easily be done through a
to_dataframemethod on the Collector class.So, what could the resulting API look like?
So, does the basis idea of object, retrieval of one or more attributes, and/or applying a callable make sense? Have a missed a key concern? Is there something obviously wrong or missing in the sketch of the API?
Beta Was this translation helpful? Give feedback.
All reactions