cattrs support

[HexDecimal]

I was asked to support cattrs and other generic serialization libraries. With the boilerplate required to do this externally it would be better to add cattrs support directly to this package instead.

This has to do the following:

• Save and load anonymous Entity objects which are always a (world, uid) pair.
• Save and load anonymous objects without knowing their type ahead of time. These may be components, tags, or uid's.

Code examples from @slavfox

How to handle object() as uid?

# EntityId: some opaque type with a zero-argument constructor EntityId()
# ecs.entdict: dict[EntityId, collection[Any]]

# note: only typechecks with --enable-recursive-aliases
JsonValue: TypeAlias = dict[str, "JsonValue"] | list["JsonValue"] | int | float | bool | str | None

def serialize_ecs(ecs):
    idmap: dict[EntityId, int] = {id_: i for i, id_ in enumerate(ecs.entdict)}
    return {
        idmap[id_]: [serialize_value(c, idmap=idmap) for c in components] 
        for id_, c in ecs.entdict.items()
    }

def serialize_value(value: Any, idmap: dict[EntityId, int]) -> JsonValue:
    match value:
        case int(_) | float(_) | bool(_) | str(_) | None:
            return value
        case [*xs]:
            return [serialize_value(x, idmap) for x in xs]
        case {**xs}:
            return {str(k): serialize_value(v, idmap) for (k, v) in xs}
        case EntityId():
            return idmap[value]
        case _:
            return {
                field.name: serialize_value(field.value) for field in get_ecs_fields(value)
            }

How to handle component types held by World?

The common pattern for using cattr is tagged enums, which you can think of as, roughly:

serializers = {}
deserializers = {}

# Decorator to register a class with cattr
def cattr(cls_):
    serializers[cls_] = ...
    cls_key = make_cls_key(cls_)
    deserializers[cls_key] = ...
    ...
    return cls_


def serialize(inst):
    return {
        "__type": make_cls_key(type(inst)), 
        **serializers[type(inst)](inst)
    }

def deserialize(serialized):
    key = serialized.pop("__type")
    return deserializers[key](serialized)

The cattrs docs are pretty good, but it will take me a while to internalize them.

[HexDecimal]

The more I look into it the less likely it seems I'll be able to register the ECS World with cattrs. At the very least I'll need to pass the converter into the serialization function otherwise the converter configuration is lost.

For components I could probably do something like this for their types and data:

import importlib
from typing import Any

import cattrs


def unstructure_generic(obj: object, converter: cattrs.Converter) -> dict[str, Any]:
    """Unstructure any type or instance to a qualified path and its data."""
    if isinstance(obj, type):
        return {"module": obj.__module__, "type": obj.__name__}
    return {
        "module": obj.__class__.__module__,
        "type": obj.__class__.__name__,
        "data": converter.unstructure(obj, obj.__class__),
    }


def structure_generic(blob: dict[str, Any], converter: cattrs.Converter) -> Any:
    """Structure a previously unstructured generic object."""
    out_type: type[Any] = getattr(importlib.import_module(blob["module"]), blob["type"])
    if "data" not in blob:
        return out_type
    return converter.structure(blob["data"], out_type)

This should work for everything except object()

[slavfox]

Dropping a reminder to myself and a notice that I plan to take a look at this tomorrow, will need to look deeper into tcod-ecs to figure out how to handle this

[HexDecimal]

So cattrs only works with simple callable objects, but both the converter and any extra state can be tracked by making a new class and registering that classes methods with cattrs.

class _ECSConverter:
    ... # Weak references to known Worlds and plain object are tracked by this class

    def unstructure_uid(self, obj: object) -> dict[Any, str]:
        """Unstructure an object used as an ID, tracking seen objects."""

    def structure_uid(self, blob: dict[Any, str]) -> Any:
        """Structure an object used as an ID, can return the same object if it is duplicate."""

    def unstructure_entity(self, entity: Entity) -> dict[Any, str]:
        """Unstructure a (world, uid) pair."""

    def structure_entity(self, blob: dict[Any, str]) -> Entity:
        """Structure a (world, uid) pair."""

    def unstructure_world(self, world: World) -> dict[Any, str]:
        """Unstructure a World instance."""

    def structure_world(self, blob: dict[Any, str]) -> World:
        """Structure an object used as an ID, can return the same object if it is duplicate."""


def handle_tcod_ecs(converter: cattrs.Converter) -> None:
    """Configure a cattrs converter to support ECS instances."""
    ecs_converter = _ECSConverter(converter)  # Bind to converter
    converter.register_unstructure_hook(World, ecs_converter.unstructure_world)
    converter.register_structure_hook(World, ecs_converter.structure_world)
    converter.register_unstructure_hook(Entity, ecs_converter.unstructure_entity)
    converter.register_structure_hook(Entity, ecs_converter.structure_entity)
    converter.register_unstructure_hook(Any, ecs_converter.unstructure_uid)
    converter.register_structure_hook(Any, ecs_converter.structure_uid)

I think this solves the issue of how to handle when a world or uid is encountered multiple times.

[HexDecimal]

My attempt to write a cattrs converter with #5 is officially a failure. It works, saves and loads without issues and handles cycles and whatever I throw at it, but the resulting format is huge mess that I don't want to support. It might be good for me to look at how other tools serialize this kind of system.

I vastly underestimated the complexity of unstructuring dictionaries with complex keys. Python does not like the keys being unstructured into unhashable dictionaries and even if I used frozendict some formats might not be able to handle serializing such keys.

cattrs doesn't seem to have good support for cycles in general and adding such support is possible but very awkward. It isn't that kind of tool. I'm not sure of the best way to resolve this. The main issue other than cycles is that the world instance becomes a black box for all of the types it contains. You mentioned registering types which seems like the only real way to resolve that, though I still don't like that idea and it doesn't handle cycles.

I might try this again some time after I've become a lot more familiar with cattrs.