Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for Descriptors in Model Fields #11148

Open
4 of 13 tasks
nickyoung-github opened this issue Dec 18, 2024 · 3 comments
Open
4 of 13 tasks

Support for Descriptors in Model Fields #11148

nickyoung-github opened this issue Dec 18, 2024 · 3 comments

Comments

@nickyoung-github
Copy link

nickyoung-github commented Dec 18, 2024

Initial Checks

  • I have searched Google & GitHub for similar requests and couldn't find anything
  • I have read and followed the docs and still think this feature is missing

Description

[Raising this issue as a prelude to submitting PRs for pydantic and pydantic_core, with some suggested solutions]

Pydantic has features which break the use of descriptors for model fields. Principally:

  1. __init__() calls self.__pydantic_validator__.validate_python, which ends up setting self.__dict__ directly
  2. BaseMode.__setattr__() sets model fields via self.__dict__[name] = ...
  3. model_dump() and model_dump_json() access __dict__ directly (via ModelSerializer.get_inner_value() in pydantic_core)
  4. __repr_args__, __iter__ and __eq__ for BaseModel access model fields via self.__dict__
  5. collect_model_fields() will actually delete descriptors in some cases

You can see this behaviour with the following code:

from pydantic import BaseModel
from pydantic._internal._model_construction import ModelMetaclass
from pydantic_core import PydanticUndefined
from typing import Any


class FieldDescriptor:
    """ Example descriptor, just to show storage somewhere other than __dict__ """

    def __init__(self, name: str, default: Any):
        self.__default = default
        self.__name = name
        self.__values = {}

    def __get__(self, instance, owner):
        if instance is not None:
            try:
                return self.__values[id(instance)][self.__name]
            except KeyError:
                return self.__default

        return self

    def __set__(self, instance, value):
        self.__values.setdefault(id(instance), {})[self.__name] = value


class DescriptorMeta:
    def __new__(cls, cls_name: str, bases: tuple[type[Any], ...], namespace: dict[str, Any], **kwargs):
        # Convert all the model fields into FieldDescriptors
        descriptors = {}
        for name, typ in namespace.get("__annotations__", {}).items():
            value = namespace.get(name, PydanticUndefined)
            if value is PydanticUndefined or not hasattr(value, "__get__"):
                descriptors[name] = FieldDescriptor(name, value)

        ret = super().__new__(cls, cls_name, bases, namespace, **kwargs)
        for name, descriptor in descriptors.items():
            # We can't just put them in namespace because collect_model_fields() will delete them
            setattr(ret, name, descriptor)

        return ret


class PydanticDescriptorMeta(DescriptorMeta, ModelMetaclass):
    pass


class Foo(BaseModel, metaclass=PydanticDescriptorMeta):
    s: str
    i: int


>>> f = Foo(s="mystr", i=123)
>>> f.model_dump()
{'s': 'nick', 'i': 123}
>>>
>>> f.i
PydanticUndefined

However, if we use a dataclass and TypeAdapter, it all works beautifully!

from dataclasses import dataclass
from pydantic import TypeAdapter


class DataclassDescriptorMeta(DescriptorMeta, type):
    pass


@dataclass
class Bar(metaclass=DataclassDescriptorMeta):
    s: str
    i: int


>>> b = Bar(s="mystr", i=123)
>>> TypeAdapter(Bar).dump_json(b)
b'{"s":"mystr","i":123}'
>>>
>>> b.i
123

I presume the existing implementation and its reliance on direct access to __dict__ was done for a reason, so I'm trying to make this work without wholesale changes. My approach is to add __pydantic_descriptor_fields__, to identify those which use a descriptor, and add functionality slightly in the mould of __pydantic_extra__.

PRs to follow ...

Affected Components

@nickyoung-github
Copy link
Author

nickyoung-github commented Dec 23, 2024

A proposed solution: introduce ModelFieldDescriptor as a base class for descriptors one wishes to use for model fields. This serves two purposes:

  1. It makes the intent obvious and negates the need for special-casing a la property, cached_property etc
  2. It allows use to associate FieldInfo with the field

With my proposed changes, the above example can be re-written as:

from pydantic import ModelFieldDescriptor
from pydantic import BaseModel, Field
from pydantic.fields import FieldInfo
from pydantic_core import PydanticUndefined
from typing import Any


class FieldDescriptor(ModelFieldDescriptor):
    def __init__(self, /, default: Any = PydanticUndefined, field=None):
        super().__init__(default=default, field=field)
        self.__values = {}

    def __get__(self, instance, owner):
        if instance is not None:
            try:
                return self.__values[id(instance)][self.name]
            except KeyError:
                return self.field.default

        return self

    def __set__(self, instance, value):
        self.__values.setdefault(id(instance), {})[self.name] = value


class Foo(BaseModel):
    s: str = FieldDescriptor(field=Field("Nick", alias="String"))
    i: int

    @property
    def ii(self) -> int:
        return self.i

>>> f = Foo(i=123)
>>> f
Foo(i=123, s='Nick')
>>> f.__dict__
{'i': 123}
>>> f.s
'Nick'
>>> f.i
123
>> f.model_dump()
{'i': 123, 's': 'Nick'}
>> Foo.__pydantic_descriptor_fields__
{'s'}

@nickyoung-github
Copy link
Author

@nickyoung-github
Copy link
Author

nickyoung-github commented Jan 6, 2025

After discussion with @Vilicos, it's clear that more debate on descriptors is needed. In case anyone wants such functionality in the meantime, you can find a stop-gap solution here and here - caveat emptor!

@nickyoung-github nickyoung-github changed the title Better Support for Descriptors in Model Fields Support for Descriptors in Model Fields Jan 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant