Merge pull request #124 from lsst-sqre/tickets/DM-37051a

DM-37051: Add some Pydantic helper functions from Gafaelfawr

.github/workflows/ci.yaml

@@ -41,6 +41,7 @@ jobs:
           - "3.8"
           - "3.9"
           - "3.10"
+          - "3.11"
 
     steps:
       - uses: actions/checkout@v3

.github/workflows/periodic.yaml

@@ -19,6 +19,7 @@ jobs:
           - "3.8"
           - "3.9"
           - "3.10"
+          - "3.11"
 
     steps:
       - uses: actions/checkout@v3

.pre-commit-config.yaml

@@ -21,8 +21,8 @@ repos:
     rev: v1.12.1
     hooks:
       - id: blacken-docs
-        additional_dependencies: [black==22.8.0]
-        args: [-l, '79', -t, py39]
+        additional_dependencies: [black==22.10.0]
+        args: [-l, '77', -t, py39]
 
   - repo: https://github.com/PyCQA/flake8
     rev: 6.0.0

CHANGELOG.md

@@ -5,11 +5,17 @@ Headline template:
 X.Y.Z (YYYY-MM-DD)
 -->
 
-## 3.4.0 (unreleased)
+## 3.4.0 (2022-11-29)
 
 - `safir.logging.configure_logging` and `safir.logging.configure_uvicorn_logging` now accept `Profile` and `LogLevel` enums in addition to strings for the `profile` and `log_level` parameters.
   The new enums are preferred.
   Support for enums was added in preparation for changing the FastAPI template to use `pydantic.BaseSettings` for its `Configuration` class and validate the values of the `profile` and `log_level` configuration parameters.
+- Add new function `safir.pydantic.normalize_datetime`, which can be used as a Pydantic validator for `datetime` fields.
+  It also allows seconds since epoch as input, and ensures that the resulting `datetime` field in the model is timezone-aware and in UTC.
+- Add new function `safir.pydantic.to_camel_case`, which can be used as a Pydantic alias generator when a model needs to accept attributes with camel-case names.
+- Add new function `safir.pydantic.validate_exactly_one_of`, which can be used as a model validator to ensure that exactly one of a list of fields was set to a value other than `None`.
+- In `safir.testing.kubernetes.patch_kubernetes`, correctly apply a spec to the mock of `kubernetes_asyncio.client.ApiClient`.
+  Due to an ordering bug in previous versions, the spec was previously a mock object that didn't apply any constraints.
 
 ## 3.3.0 (2022-09-15)
 

docs/api.rst

@@ -43,5 +43,8 @@ API reference
 .. automodapi:: safir.middleware.x_forwarded
    :include-all-objects:
 
+.. automodapi:: safir.pydantic
+   :include-all-objects:
+
 .. automodapi:: safir.testing.kubernetes
    :include-all-objects:

docs/user-guide/arq.rst

@@ -64,7 +64,9 @@ If your app uses a configuration system like ``pydantic.BaseSettings``, this exa
             redis_settings = RedisSettings(
                 host=url_parts.hostname or "localhost",
                 port=url_parts.port or 6379,
-                database=int(url_parts.path.lstrip("/")) if url_parts.path else 0,
+                database=int(url_parts.path.lstrip("/"))
+                if url_parts.path
+                else 0,
             )
             return redis_settings
 

docs/user-guide/database.rst

@@ -312,7 +312,9 @@ For example:
        engine = create_database_engine(
            config.database_url, config.database_password
        )
-       await initialize_database(engine, logger, schema=Base.metadata, reset=True)
+       await initialize_database(
+           engine, logger, schema=Base.metadata, reset=True
+       )
        await engine.dispose()
        async with LifespanManager(main.app):
            yield main.app
@@ -351,7 +353,9 @@ To get a new async database connection, use code like the following:
    from .config import config
 
 
-   engine = create_database_engine(config.database_url, config.database_password)
+   engine = create_database_engine(
+       config.database_url, config.database_password
+   )
    session = await create_async_session(engine)
 
    # ... use the session here ...
@@ -424,7 +428,10 @@ As with :ref:`async database sessions <probing-db-connection>`, you can pass a `
    logger = structlog.get_logger(config.logger_name)
    stmt = select(User)
    session = create_sync_session(
-       config.database_url, config.database_password, logger, statement=stmt
+       config.database_url,
+       config.database_password,
+       logger,
+       statement=stmt,
    )
 
 Applications that use `~safir.database.create_sync_session` must declare a dependency on `psycopg2 <https://pypi.org/project/psycopg2/>`__ in their pip dependencies.

docs/user-guide/gafaelfawr.rst

@@ -18,7 +18,9 @@ To get that username, use the `~safir.dependencies.gafaelfawr.auth_dependency` F
 
 
    @app.get("/route")
-   async def get_rounte(user: str = Depends(auth_dependency)) -> Dict[str, str]:
+   async def get_rounte(
+       user: str = Depends(auth_dependency),
+   ) -> Dict[str, str]:
        # Route implementation using user.
        return {"some": "data"}
 

docs/user-guide/index.rst

@@ -22,3 +22,4 @@ User guide
    x-forwarded
    ivoa
    kubernetes
+   pydantic

docs/user-guide/pydantic.rst

@@ -0,0 +1,94 @@
+#############################
+Utilities for Pydantic models
+#############################
+
+Several validation and configuration problems arise frequently with Pydantic models.
+Safir offers some utility functions to assist in solving them.
+
+Normalizing datetime fields
+===========================
+
+Pydantic supports several input formats for `~datetime.datetime` fields, but the resulting `~datetime.datetime` object may be timezone-naive.
+Best practice for Python code is to only use timezone-aware `~datetime.datetime` objects in the UTC time zone.
+
+Pydantic provides a utility function, `~safir.pydantic.normalize_datetime`, that can be used as a validator for a `~datetime.datetime` model field.
+It ensures that any input is converted to UTC and is always timezone-aware.
+
+Here's an example of how to use it:
+
+.. code-block:: python
+
+   class Info(BaseModel):
+       last_used: Optional[datetime] = Field(
+           None,
+           title="Last used",
+           description="When last used in seconds since epoch",
+           example=1614986130,
+       )
+
+       _normalize_last_used = validator(
+           "last_used", allow_reuse=True, pre=True
+       )(normalize_datetime)
+
+Multiple attributes can be listed as the initial arguments of `~pydantic.validator` if there are multiple fields that need to be checked.
+
+Accepting camel-case attributes
+===============================
+
+Python prefers ``snake_case`` for all object attributes, but some external sources of data (Kubernetes custom resources, YAML configuration files generated from Helm configuration) require or prefer ``camelCase``.
+
+Thankfully, Pydantic supports converting from camel-case to snake-case on input using what Pydantic calls an "alias generator."
+Safir provides `~safir.pydantic.to_camel_case`, which can be used as that alias generator.
+
+To use it, add a configuration block to any Pydantic model that has snake-case attributes but needs to accept them in camel-case form:
+
+.. code-block:: python
+
+   class Model(BaseModel):
+       some_field: str
+
+       class Config:
+           alias_generator = to_camel_case
+           allow_population_by_field_name = True
+
+By default, only the generated aliases (so, in this case, only the camel-case form of the attribute, ``someField``) are supported.
+The additional setting ``allow_population_by_field_name``, tells Pydantic to allow either ``some_field`` or ``soemField`` in the input.
+
+Adding this configuration can be tedious if you have a lot of models.
+In that case, consider making a subclass of `~pydantic.BaseModel` with this ``Config`` subclass, and having all of your models that need to support camel-case inherit from that class.
+
+Requiring exactly one of a list of attributes
+=============================================
+
+Occasionally, you will have reason to write a model with several attributes, where one and only one of those attributes may be set.
+For example:
+
+.. code-block:: python
+
+   class Model(BaseModel):
+       docker: Optional[DockerConfig] = None
+       ghcr: Optional[GHCRConfig] = None
+
+The intent here is that only one of those two configurations will be present: either Docker or GitHub Container Registry.
+However, Pydantic has no native way to express that, and the above model will accept input where neither or both of those attributes are set.
+
+Safir provides a function, `~safir.pydantic.validate_exactly_one_of`, designed for this case.
+It takes a list of fields, of which exactly one must be set, and builds a validation function that checks this property of the model.
+
+So, in the above example, the full class would be:
+
+.. code-block:: python
+
+   class Model(BaseModel):
+       docker: Optional[DockerConfig] = None
+       ghcr: Optional[GHCRConfig] = None
+
+       _validate_type = validator("ghcr", always=True, allow_reuse=True)(
+           validate_exactly_one_of("docker", "ghcr")
+       )
+
+Note the syntax, which is a little odd since it is calling a decorator on the results of a function builder.
+
+The argument to `~pydantic.validator` must always be the last of the possible attributes that may be set, ensuring that any other attributes have been seen when the validator runs.
+``always=True`` must be set to ensure the validator runs regardless of which attribute is set.
+``allow_reuse=True`` must be set due to limitations in Pydantic.

pyproject.toml

@@ -17,6 +17,7 @@ classifiers = [
     "Programming Language :: Python :: 3.8",
     "Programming Language :: Python :: 3.9",
     "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
     "Intended Audience :: Developers",
     "Natural Language :: English",
     "Operating System :: POSIX",

src/safir/pydantic.py

@@ -0,0 +1,165 @@
+"""Utility functions for Pydantic models."""
+
+from __future__ import annotations
+
+from datetime import datetime, timezone
+from typing import Any, Callable, Dict, Optional, Union
+
+__all__ = [
+    "normalize_datetime",
+    "to_camel_case",
+    "validate_exactly_one_of",
+]
+
+
+def normalize_datetime(
+    v: Optional[Union[int, datetime]]
+) -> Optional[datetime]:
+    """Pydantic validator for datetime fields.
+
+    Supports `~datetime.datetime` fields given in either any format supported
+    by Pydantic natively, or in seconds since epoch (which Pydantic doesn't
+    support).  This validator ensures that datetimes are always stored in the
+    model as timezone-aware UTC datetimes.
+
+    Parameters
+    ----------
+    v
+        The field representing a `~datetime.datetime`.
+
+    Returns
+    -------
+    datetime.datetime or None
+        The timezone-aware `~datetime.datetime` or `None` if the input was
+        `None`.
+
+    Examples
+    --------
+    Here is a partial model that uses this function as a validator.
+
+    .. code-block:: python
+
+       class Info(BaseModel):
+           last_used: Optional[datetime] = Field(
+               None,
+               title="Last used",
+               description="When last used in seconds since epoch",
+               example=1614986130,
+           )
+
+           _normalize_last_used = validator(
+               "last_used", allow_reuse=True, pre=True
+           )(normalize_datetime)
+    """
+    if v is None:
+        return v
+    elif isinstance(v, int):
+        return datetime.fromtimestamp(v, tz=timezone.utc)
+    elif v.tzinfo and v.tzinfo.utcoffset(v) is not None:
+        return v.astimezone(timezone.utc)
+    else:
+        return v.replace(tzinfo=timezone.utc)
+
+
+def to_camel_case(string: str) -> str:
+    """Convert a string to camel case.
+
+    Intended for use with Pydantic as an alias generator so that the model can
+    be initialized from camel-case input, such as Kubernetes objects or
+    settings from Helm charts.
+
+    Parameters
+    ----------
+    string
+        Input string.
+
+    Returns
+    -------
+    str
+        String converted to camel-case with the first character in lowercase.
+
+    Examples
+    --------
+    To support ``camelCase`` input to a model, use the following settings:
+
+    .. code-block:: python
+
+       class Model(BaseModel):
+           some_field: str
+
+           class Config:
+               alias_generator = to_camel_case
+               allow_population_by_field_name = True
+
+    This must be added to every class that uses ``snake_case`` for an
+    attribute and that needs to be initialized from ``camelCase``. If there
+    are a lot of those classes, consider making a derivative class of
+    `~pydantic.BaseModel` that sets these configuration options, and having
+    all of your model classes inherit from it.
+    """
+    components = string.split("_")
+    return components[0] + "".join(c.title() for c in components[1:])
+
+
+def validate_exactly_one_of(
+    *settings: str,
+) -> Callable[[Any, Dict[str, Any]], Any]:
+    """Generate a validator imposing a one and only one constraint.
+
+    Sometimes, models have a set of attributes of which one and only one may
+    be set. Ideally this is represented properly in the type system, but
+    occasionally it's more convenient to use a validator. This is a validator
+    generator that can produce a validator function that ensures one and only
+    one of an arbitrary set of attributes must be set.
+
+    Parameters
+    ----------
+    *settings
+        List of names of attributes, of which one and only one must be set.
+        At least two attribute names must be listed.
+
+    Returns
+    -------
+    Callable
+        The validator.
+
+    Examples
+    --------
+    Use this inside a Pydantic class as a validator as follows:
+
+    .. code-block:: python
+
+       class Foo(BaseModel):
+           foo: Optional[str] = None
+           bar: Optional[str] = None
+           baz: Optional[str] = None
+
+           _validate_options = validator("baz", always=True, allow_reuse=True)(
+               validate_exactly_one_of("foo", "bar", "baz")
+           )
+
+    The attribute listed as the first argument to the ``validator`` call must
+    be the last attribute in the model definition so that any other attributes
+    have already been seen.
+    """
+    if len(settings) < 2:
+        msg = "validate_exactly_one_of takes at least two field names"
+        raise ValueError(msg)
+
+    if len(settings) == 2:
+        options = f"{settings[0]} and {settings[1]}"
+    else:
+        options = ", ".join(settings[:-1]) + ", and " + settings[-1]
+
+    def validator(v: Any, values: Dict[str, Any]) -> Any:
+        seen = v is not None
+        for setting in settings:
+            if setting in values and values[setting] is not None:
+                if seen:
+                    raise ValueError(f"only one of {options} may be given")
+                seen = True
+        if not seen:
+            raise ValueError(f"one of {options} must be given")
+        return v
+
+    return validator

src/safir/testing/kubernetes.py

@@ -353,9 +353,10 @@ def mock_kubernetes() -> Iterator[MockKubernetesApi]:
             mock_class = patcher.start()
             mock_class.return_value = mock_api
             patchers.append(patcher)
+        mock_api_client = Mock(spec=client.ApiClient)
+        mock_api_client.close = AsyncMock()
         with patch.object(client, "ApiClient") as mock_client:
-            mock_client.return_value = Mock(spec=client.ApiClient)
-            mock_client.return_value.close = AsyncMock()
+            mock_client.return_value = mock_api_client
             os.environ["KUBERNETES_PORT"] = "tcp://10.0.0.1:443"
             yield mock_api
             del os.environ["KUBERNETES_PORT"]