diff --git a/.gitignore b/.gitignore index 2a6d0162..d8c5b177 100644 --- a/.gitignore +++ b/.gitignore @@ -2,3 +2,5 @@ db.sqlite3 __pycache__ .venv + +addon_service/static/gravyvalet_code_docs/ diff --git a/Dockerfile b/Dockerfile index 736f8f3b..42f67efa 100644 --- a/Dockerfile +++ b/Dockerfile @@ -8,18 +8,25 @@ COPY . /code/ WORKDIR /code # END gv-base -# BEGIN gv-local -FROM gv-base as gv-local +# BEGIN gv-dev +FROM gv-base as gv-dev # install dev and non-dev dependencies: RUN pip3 install --no-cache-dir -r requirements/dev-requirements.txt # Start the Django development server CMD ["python", "manage.py", "runserver", "0.0.0.0:8004"] -# END gv-local +# END gv-dev + +# BEGIN gv-docs +FROM gv-dev as gv-docs +RUN python -m gravyvalet_code_docs.build +# END gv-docs # BEGIN gv-deploy FROM gv-base as gv-deploy # install non-dev and release-only dependencies: RUN pip3 install --no-cache-dir -r requirements/release.txt +# copy auto-generated static docs (without the dev dependencies that built them) +COPY --from=gv-docs /code/addon_service/static/gravyvalet_code_docs/ /code/addon_service/static/gravyvalet_code_docs/ # collect static files into a single directory: RUN python manage.py collectstatic --noinput # note: no CMD in gv-deploy -- depends on deployment diff --git a/README.md b/README.md index fe6772fa..d6bcfe89 100644 --- a/README.md +++ b/README.md @@ -1,26 +1,70 @@ -![Center for Open Science Logo](https://mfr.osf.io/export?url=https://osf.io/download/24697/?direct=%26mode=render&format=2400x2400.jpeg) +# 🥣 gravyvalet -# OSF Addon Service (GravyValet) +`gravyvalet` fetches, serves, and holds small ladlefuls of precious bytes. -Welcome to the Open Science Framework's base server for addon integration with our RESTful API (osf.io). This server acts as a gateway between the OSF and external APIs. Authenticated users or machines can access various resources through common file storage and citation management APIs via the OSF. Institutional members can also add their own integrations, tailoring addon usage to their specific communities. +together with [waterbutler](https://waterbutler.readthedocs.io) +(which fetches and serves whole streams of bytes, but holds nothing), +gravyvalet provides an api to support "osf addons", +whereby you can share controlled access to online accounts +(e.g. cloud storage) with your collaborators on [osf](https://osf.io). -## Setting up GravyValet Locally +(NOTE: gravyvalet is still under active development and changes may happen suddenly, +tho current docs may or may not be available at https://addons.staging.osf.io/docs ) +# how to... + +## ...set up gravyvalet for local development with osf, using docker + +0. have [osf running](https://github.com/CenterForOpenScience/osf.io/blob/develop/README-docker-compose.md) (with its `api` at `http://192.168.168.167:8000`) 1. Start your PostgreSQL and Django containers with `docker compose up -d`. -2. Enter the Django container: `docker compose exec addon_service /bin/bash`. +2. Enter the Django container: `docker compose exec gravyvalet /bin/bash`. 3. Migrate the existing models: `python manage.py migrate`. 4. Visit [http://0.0.0.0:8004/](http://0.0.0.0:8004/). -## Running Tests +## ...run tests To run tests, use the following command: ```bash python manage.py test ``` +(recommend adding `--failfast` when looking for immediate feedback) + +## ...set up external services +start by creating an admin account with +[django's createsuperuser command](https://docs.djangoproject.com/en/4.2/ref/django-admin/#django-admin-createsuperuser): + +```bash +python manage.py createsuperuser +``` + +then log in with that account at `localhost:8004/admin` to manage +external services (including oauth config) and to create other admin users -Development Tips +## ...configure a good environment +see `app/env.py` for details on all environment variables used. + +when run without a `DEBUG` environment variable (note: do NOT run with `DEBUG` in production), +some additional checks are run on the environment: + +- `GRAVYVALET_ENCRYPT_SECRET` is required -- ideally chosen by strong randomness, + with maybe ~128 bits of entropy (e.g. 32 hex digits; 30 d20 rolls; 13 words of a 1000-word vocabulary) + +## ...rotate encryption keys responsibly +don't let your secrets get stale! you can rotate the secret used to derive encryption keys +(as well as the parameters for key derivation -- see `app/env.py` for details) + +1. update environment: + - set `GRAVYVALET_ENCRYPT_SECRET` to a new, long, random string (...no commas, tho) + - add the old secret to `GRAVYVALET_ENCRYPT_SECRET_PRIORS` (comma-separated list) + - (optional) update key-derivation parameters with best-practices du jour +2. run `python manage.py rotate_encryption` to enqueue key-rotation tasks + (on the `gravyvalet_tasks.CHILL` queue by default) +3. once that queue of tasks is complete, update environment again to remove the old secret from + `GRAVYVALET_ENCRYPT_SECRET_PRIORS` + +## ...enable pre-commit hooks Optionally, but recommended: Set up pre-commit hooks that will run formatters and linters on staged files. Install pre-commit using: ```bash @@ -34,7 +78,8 @@ Then, run: pre-commit install --allow-missing-config ``` -Reporting Issues and Questions + +## ...ask questions or report issues If you encounter a bug, have a technical question, or want to request a feature, please don't hesitate to contact us at help@osf.io. While we may respond to questions through other channels, reaching out to us at help@osf.io ensures diff --git a/_TODO.md b/_TODO.md new file mode 100644 index 00000000..419b6d2b --- /dev/null +++ b/_TODO.md @@ -0,0 +1,23 @@ +# TODO: gravyvalet code docs + +### how-to/local_setup_with_osf.md +- with gravyvalet docker-compose.yml +- with osf.io docker-compose.yml +- without docker? + +### how-to/new_imp_interface.md +- defining interface with operations +- required adds to addon_service + +### how-to/migrating_osf_addon_to_imp.md +- implementing imp +- current limitations + +### how-to/new_storage_imp.md +- implementing imp +- required changes to waterbutler? (with mention of ideal "none") + +### how-to/key_rotation.md +- credentials encryption overview +- secret and prior secrets +- scrypt configuration diff --git a/addon_imps/storage/__init__.py b/addon_imps/storage/__init__.py index ea9b7835..f7e37339 100644 --- a/addon_imps/storage/__init__.py +++ b/addon_imps/storage/__init__.py @@ -1 +1,2 @@ -__all__ = () +"""addon_imps.storage: imps that implement a "storage"-like interface +""" diff --git a/addon_service/README.md b/addon_service/README.md new file mode 100644 index 00000000..8e1c0a56 --- /dev/null +++ b/addon_service/README.md @@ -0,0 +1,80 @@ +# addon_service: a django app for the gravyvalet web api + +## network flows + +addon operation invocation thru gravyvalet (as currently implemented with osf) +```mermaid +sequenceDiagram + participant browser + Box *.osf.io + participant gravyvalet + participant osf-api + end + Note over browser: browsing files on osf, say + browser->>gravyvalet: request directory listing (create an addon operation invocation) + gravyvalet->>osf-api: who is this? + osf-api->>gravyvalet: this is who + Note over gravyvalet: gravyvalet asks "who is?" (auth-entic-ation) separate from "may they?" (auth-oriz-ation) + gravyvalet->>osf-api: may they access this directory listing? + alt no + osf-api->>gravyvalet: no + gravyvalet->>browser: no + else yes + osf-api->>gravyvalet: yes + gravyvalet->>external-service: request directory listing + external-service->>gravyvalet: serve directory listing + Note over gravyvalet: listing translated into interoperable format + gravyvalet->>browser: serve directory listing + end +``` + +download a file thru waterbutler, with get_auth and gravyvalet (as currently implemented) +```mermaid +sequenceDiagram + participant browser + Box *.osf.io + participant waterbutler + participant osf-v1 + participant gravyvalet + end + browser->>waterbutler: request file + waterbutler->>osf-v1: get_auth + alt no + osf-v1->>waterbutler: no + waterbutler->>browser: no + else yes + osf-v1->>gravyvalet: request gravy + gravyvalet->>osf-v1: serve gravy + osf-v1->>waterbutler: credentials and config + waterbutler->>external service: request file + external service->>waterbutler: serve file + waterbutler->>browser: serve file + end +``` + +hypothetical world where waterbutler talks to gravyvalet... is this better than get_auth? +```mermaid +sequenceDiagram + participant browser + Box *.osf.io + participant waterbutler + participant gravyvalet + participant osf-api + end + browser->>waterbutler: request file + waterbutler->>gravyvalet: request gravy + gravyvalet->>osf-api: who is this? + osf-api->>gravyvalet: this is who + gravyvalet->>osf-api: may they do what they're asking to? + alt no + osf-api->>gravyvalet: no + gravyvalet->>waterbutler: no + waterbutler->>browser: no + else yes + osf-api->>gravyvalet: yes + gravyvalet->>waterbutler: serve gravy + waterbutler->>external service: request file + external service->>waterbutler: serve file + waterbutler->>browser: serve file + end +``` diff --git a/addon_service/__init__.py b/addon_service/__init__.py index ea9b7835..03c3b19d 100644 --- a/addon_service/__init__.py +++ b/addon_service/__init__.py @@ -1 +1,20 @@ -__all__ = () +""" +.. include:: README.md +""" + +__all__ = ( + "addon_imp", + "addon_operation", + "addon_operation_invocation", + "authentication", + "authorized_storage_account", + "common", + "configured_storage_addon", + "credentials", + "external_storage_service", + "oauth1", + "oauth2", + "resource_reference", + "tasks", + "user_reference", +) diff --git a/addon_service/addon_imp/__init__.py b/addon_service/addon_imp/__init__.py index ea9b7835..66432b29 100644 --- a/addon_service/addon_imp/__init__.py +++ b/addon_service/addon_imp/__init__.py @@ -1 +1,2 @@ -__all__ = () +"""addon_service.addon_imp: for representing static-known addon implementations in the api +""" diff --git a/addon_service/addon_imp/instantiation.py b/addon_service/addon_imp/instantiation.py index a06d9106..58d7af09 100644 --- a/addon_service/addon_imp/instantiation.py +++ b/addon_service/addon_imp/instantiation.py @@ -14,6 +14,10 @@ async def get_storage_addon_instance( account: AuthorizedStorageAccount, config: StorageConfig, ) -> StorageAddonImp: + """create an instance of a `StorageAddonImp` + + (TODO: decide on a common constructor for all `AddonImp`s, remove this) + """ assert issubclass(imp_cls, StorageAddonImp) return imp_cls( config=config, @@ -26,3 +30,7 @@ async def get_storage_addon_instance( get_storage_addon_instance__blocking = async_to_sync(get_storage_addon_instance) +"""create an instance of a `StorageAddonImp` + +(same as `get_storage_addon_instance`, for use in synchronous context +""" diff --git a/addon_service/addon_imp/models.py b/addon_service/addon_imp/models.py index 14473030..2992052d 100644 --- a/addon_service/addon_imp/models.py +++ b/addon_service/addon_imp/models.py @@ -8,10 +8,10 @@ from addon_toolkit import AddonImp -# dataclass wrapper for a concrete subclass of AddonImp which -# meets rest_framework_json_api expectations on a model class @dataclasses.dataclass(frozen=True) class AddonImpModel(StaticDataclassModel): + """each `AddonImpModel` represents a statically defined subclass of `AddonImp`""" + imp_cls: type[AddonImp] ### diff --git a/addon_service/addon_imp/serializers.py b/addon_service/addon_imp/serializers.py index e9e65d01..8159a6c5 100644 --- a/addon_service/addon_imp/serializers.py +++ b/addon_service/addon_imp/serializers.py @@ -12,6 +12,8 @@ class AddonImpSerializer(serializers.Serializer): + """api serializer for the `AddonImpModel` model""" + url = serializers.HyperlinkedIdentityField( view_name=view_names.detail_view(RESOURCE_TYPE) ) diff --git a/addon_service/addon_operation/serializers.py b/addon_service/addon_operation/serializers.py index e18418af..a946e5f6 100644 --- a/addon_service/addon_operation/serializers.py +++ b/addon_service/addon_operation/serializers.py @@ -14,6 +14,8 @@ class AddonOperationSerializer(serializers.Serializer): + """api serializer for the `AddonOperationModel` model""" + url = serializers.HyperlinkedIdentityField( view_name=view_names.detail_view(RESOURCE_TYPE) ) diff --git a/addon_service/addon_operation_invocation/__init__.py b/addon_service/addon_operation_invocation/__init__.py index ea9b7835..6dc34081 100644 --- a/addon_service/addon_operation_invocation/__init__.py +++ b/addon_service/addon_operation_invocation/__init__.py @@ -1 +1,2 @@ -__all__ = () +"""addon_service.addon_operation_invocation: a specific invocation of a gravyvalet addon operation +""" diff --git a/addon_service/addon_operation_invocation/serializers.py b/addon_service/addon_operation_invocation/serializers.py index 3ff1eea1..c3133002 100644 --- a/addon_service/addon_operation_invocation/serializers.py +++ b/addon_service/addon_operation_invocation/serializers.py @@ -20,6 +20,8 @@ class AddonOperationInvocationSerializer(serializers.HyperlinkedModelSerializer): + """api serializer for the `AddonOperationInvocation` model""" + class Meta: model = AddonOperationInvocation fields = [ diff --git a/addon_service/authorized_storage_account/serializers.py b/addon_service/authorized_storage_account/serializers.py index fffa8d2b..2fc9ecc0 100644 --- a/addon_service/authorized_storage_account/serializers.py +++ b/addon_service/authorized_storage_account/serializers.py @@ -31,6 +31,8 @@ class AuthorizedStorageAccountSerializer(serializers.HyperlinkedModelSerializer): + """api serializer for the `AuthorizedStorageAccount` model""" + def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) diff --git a/addon_service/common/__init__.py b/addon_service/common/__init__.py index ea9b7835..5f37e90d 100644 --- a/addon_service/common/__init__.py +++ b/addon_service/common/__init__.py @@ -1 +1,2 @@ -__all__ = () +"""addon_service.common: shared logic among addon_service types +""" diff --git a/addon_service/common/aiohttp_session.py b/addon_service/common/aiohttp_session.py index 5c47d4a4..e6ec186a 100644 --- a/addon_service/common/aiohttp_session.py +++ b/addon_service/common/aiohttp_session.py @@ -16,14 +16,15 @@ async def get_singleton_client_session() -> aiohttp.ClientSession: - if not is_session_valid(): + """return a reusable aiohttp client session (thread-local singleton)""" + if not _is_session_valid(): __SINGLETON_CLIENT_SESSION_STORE.session = aiohttp.ClientSession( cookie_jar=aiohttp.DummyCookieJar(), # ignore all cookies ) return __SINGLETON_CLIENT_SESSION_STORE.session -def is_session_valid() -> bool: +def _is_session_valid() -> bool: return ( hasattr(__SINGLETON_CLIENT_SESSION_STORE, "session") and isinstance(__SINGLETON_CLIENT_SESSION_STORE.session, aiohttp.ClientSession) @@ -32,10 +33,20 @@ def is_session_valid() -> bool: async def close_singleton_client_session() -> None: - if is_session_valid(): + """close the reusable aiohttp client session (thread-local singleton)""" + if _is_session_valid(): await __SINGLETON_CLIENT_SESSION_STORE.close() __SINGLETON_CLIENT_SESSION_STORE.session = None get_singleton_client_session__blocking = async_to_sync(get_singleton_client_session) +"""return a reusable aiohttp client session (thread-local singleton) + +(same as `get_singleton_client_session`, for use in non-async context) +""" + close_singleton_client_session__blocking = async_to_sync(close_singleton_client_session) +"""close the reusable aiohttp client session (thread-local singleton) + +(same as `close_singleton_client_session`, for use in non-async context) +""" diff --git a/addon_service/common/base_model.py b/addon_service/common/base_model.py index a065b505..c153d674 100644 --- a/addon_service/common/base_model.py +++ b/addon_service/common/base_model.py @@ -8,6 +8,8 @@ class AddonsServiceBaseModel(models.Model): + """common base class for all addon_service models""" + id = StrUUIDField(primary_key=True, default=str_uuid4, editable=False) created = models.DateTimeField(editable=False) modified = models.DateTimeField() diff --git a/addon_service/common/credentials_formats.py b/addon_service/common/credentials_formats.py index faaafb0c..aec3c578 100644 --- a/addon_service/common/credentials_formats.py +++ b/addon_service/common/credentials_formats.py @@ -8,6 +8,8 @@ @unique class CredentialsFormats(Enum): + """all available credentials formats""" + UNSPECIFIED = 0 OAUTH2 = 1 ACCESS_KEY_SECRET_KEY = 2 @@ -17,6 +19,7 @@ class CredentialsFormats(Enum): @property def dataclass(self): + """get an `addon_toolkit.credentials.Credentials` subclass for this `CredentialsFormat`""" match self: case CredentialsFormats.OAUTH2: return credentials.AccessTokenCredentials @@ -31,7 +34,11 @@ def dataclass(self): raise ValueError(f"No dataclass support for credentials type {self.name}") @property - def is_direct_from_user(self): + def is_direct_from_user(self) -> bool: + """return True if credentials of this format are provided directly by the user + + (or False if credentials established via oauth or similar) + """ return self in { CredentialsFormats.ACCESS_KEY_SECRET_KEY, CredentialsFormats.USERNAME_PASSWORD, diff --git a/addon_service/common/enum_decorators.py b/addon_service/common/enum_decorators.py index 9287406f..417d9378 100644 --- a/addon_service/common/enum_decorators.py +++ b/addon_service/common/enum_decorators.py @@ -13,6 +13,8 @@ def enum_names_same_as( other_enum: type[enum.Enum], ) -> typing.Callable[[_ThisEnum], _ThisEnum]: + """decorate an enum to guarantee it has the same names as another enum""" + def _enum_decorator(this_enum: _ThisEnum) -> _ThisEnum: _other_names = enum_names(other_enum) _these_names = enum_names(this_enum) diff --git a/addon_service/common/invocation_status.py b/addon_service/common/invocation_status.py index 4d30bdd3..8a747ef1 100644 --- a/addon_service/common/invocation_status.py +++ b/addon_service/common/invocation_status.py @@ -2,7 +2,13 @@ class InvocationStatus(enum.Enum): + """coarse-grained status of an addon operation invocation""" + STARTING = 1 + """the invocation has been recorded and enqueued""" GOING = 2 + """work on the invocation has begun""" SUCCESS = 3 + """the invocation has succeeded and has a result""" ERROR = 128 + """an error occurred""" diff --git a/addon_service/common/network.py b/addon_service/common/network.py index 2a1fc417..e545940e 100644 --- a/addon_service/common/network.py +++ b/addon_service/common/network.py @@ -12,10 +12,9 @@ import aiohttp from asgiref.sync import sync_to_async -from addon_service import models as db from addon_service.common import exceptions from addon_service.common.credentials_formats import CredentialsFormats -from addon_toolkit.constrained_network import ( +from addon_toolkit.constrained_network.http import ( HttpRequestInfo, HttpRequestor, HttpResponseInfo, @@ -23,6 +22,10 @@ from addon_toolkit.iri_utils import Multidict +if typing.TYPE_CHECKING: + from addon_service.models import AuthorizedStorageAccount + + __all__ = ("GravyvaletHttpRequestor",) @@ -52,6 +55,8 @@ async def json_content(self) -> typing.Any: class GravyvaletHttpRequestor(HttpRequestor): + """an `HttpRequestor` implementation using aiohttp""" + # abstract property from HttpRequestor: response_info_cls = _AiohttpResponseInfo @@ -60,7 +65,7 @@ def __init__( *, client_session: aiohttp.ClientSession, prefix_url: str, - account: db.AuthorizedStorageAccount, + account: "AuthorizedStorageAccount", ): _PrivateNetworkInfo(client_session, prefix_url, account).assign(self) @@ -144,7 +149,7 @@ class _PrivateNetworkInfo(_PrivateInfo): # keep network constraints away from imps prefix_url: str - account: db.AuthorizedStorageAccount + account: "AuthorizedStorageAccount" @sync_to_async def get_headers(self) -> Multidict: diff --git a/addon_service/common/osf.py b/addon_service/common/osf.py index 919bfa7e..5add3e58 100644 --- a/addon_service/common/osf.py +++ b/addon_service/common/osf.py @@ -29,6 +29,8 @@ class OSFPermission(enum.StrEnum): + """permission values used by the osf api""" + READ = "read" WRITE = "write" ADMIN = "admin" @@ -44,6 +46,7 @@ def for_capabilities(capabilities: AddonCapabilities) -> "OSFPermission": @async_to_sync async def get_osf_user_uri(request: django_http.HttpRequest) -> str | None: + """get a uri identifying the user making this request""" try: return _get_hmac_verified_user_iri(request) except hmac_utils.RejectedHmac as e: @@ -69,6 +72,7 @@ async def has_osf_permission_on_resource( resource_uri: str, required_permission: OSFPermission, ) -> bool: + """check for a permission on a resource via the osf api""" try: return _has_hmac_verified_osf_permission( request, resource_uri, required_permission diff --git a/addon_service/common/permissions.py b/addon_service/common/permissions.py index 3c482c17..e26ee303 100644 --- a/addon_service/common/permissions.py +++ b/addon_service/common/permissions.py @@ -9,11 +9,15 @@ class IsAuthenticated(permissions.BasePermission): + """allow any logged-in user""" + def has_permission(self, request, view): return request.session.get("user_reference_uri") is not None class SessionUserIsOwner(permissions.BasePermission): + """for object permissions on objects with `owner_uri`""" + def has_object_permission(self, request, view, obj): session_user_uri = request.session.get("user_reference_uri") if session_user_uri: @@ -33,6 +37,8 @@ def has_object_permission(self, request, view, obj): class SessionUserMayConnectAddon(permissions.BasePermission): + """for object permissions on objects with `owner_uri` and `resource_uri`""" + def has_object_permission(self, request, view, obj): _user_uri = request.session.get("user_reference_uri") return ( @@ -48,6 +54,8 @@ def has_object_permission(self, request, view, obj): class SessionUserMayAccessInvocation(permissions.BasePermission): + """for object permissions on `addon_service.models.AddonOperationInvocation`""" + def has_object_permission(self, request, view, obj): _user_uri = request.session.get("user_reference_uri") return bool( @@ -65,6 +73,8 @@ def has_object_permission(self, request, view, obj): class SessionUserMayPerformInvocation(permissions.BasePermission): + """for object permissions on `addon_service.models.AddonOperationInvocation`""" + def has_object_permission(self, request, view, obj): _user_uri = request.session.get("user_reference_uri") _thru_addon = obj.thru_addon @@ -86,6 +96,8 @@ def has_object_permission(self, request, view, obj): class IsValidHMACSignedRequest(permissions.BasePermission): + """allow only requests signed with the known osf hmac key""" + def has_permission(self, request, view): try: hmac_utils.validate_signed_request( diff --git a/addon_service/common/serializer_fields.py b/addon_service/common/serializer_fields.py index 2ca3a892..cecbe2d3 100644 --- a/addon_service/common/serializer_fields.py +++ b/addon_service/common/serializer_fields.py @@ -11,10 +11,14 @@ class ReadOnlyResourceRelatedField( json_api_serializers.ResourceRelatedField, drf_serializers.ReadOnlyField ): + """read-only version of `rest_framework_json_api.serializers.ResourceRelatedField`""" + pass class DataclassRelatedLinkField(SkipDataMixin, ResourceRelatedField): + """related field for use with to-many `StaticDataclassModel` relations""" + def __init__(self, /, dataclass_model, read_only=True, **kwargs): assert dataclasses.is_dataclass(dataclass_model) return super().__init__( @@ -25,6 +29,8 @@ def __init__(self, /, dataclass_model, read_only=True, **kwargs): class DataclassRelatedDataField(ResourceRelatedField): + """related field for use with to-one `StaticDataclassModel` relations""" + def __init__(self, /, dataclass_model, **kwargs): assert dataclasses.is_dataclass(dataclass_model) return super().__init__( diff --git a/addon_service/common/service_types.py b/addon_service/common/service_types.py index 2c54d719..5a4a5b4a 100644 --- a/addon_service/common/service_types.py +++ b/addon_service/common/service_types.py @@ -2,6 +2,20 @@ class ServiceTypes(enum.Flag): + """to what extent can a service be configured with a user's custom url""" + PUBLIC = enum.auto() + """`ServiceTypes.PUBLIC`: the service exists at a specific public domain + + (`api_base_url` must be set on the service and may _not_ be set on the account) + """ HOSTED = enum.auto() + """`ServiceTypes.HOSTED`: the service must be self-hosted + + (`api_base_url` is not set on the service and _must_ be set on the account) + """ HYBRID = PUBLIC | HOSTED + """`ServiceTypes.HYBRID`: the service exists at a specific public domain but may also be self-hosted + + (`api_base_url` should be set on the service but may be overridden on the account) + """ diff --git a/addon_service/common/str_uuid_field.py b/addon_service/common/str_uuid_field.py index 58a40334..b91f8510 100644 --- a/addon_service/common/str_uuid_field.py +++ b/addon_service/common/str_uuid_field.py @@ -10,6 +10,7 @@ def str_uuid4() -> str: + """generate a random UUID (as `str`)""" return str(uuid.uuid4()) @@ -17,12 +18,15 @@ class StrUUIDField(models.UUIDField): """same as UUIDField, but showing the string representation instead of uuid.UUID""" def get_prep_value(self, value): + """parse `str` value as `uuid.UUID`""" _uuid = uuid.UUID(value) if isinstance(value, str) else value return super().get_prep_value(_uuid) def from_db_value(self, value, expression, connection): + """convert `uuid.UUID` value to `str`""" return None if value is None else str(value) def to_python(self, value): + """convert `uuid.UUID` value to `str`""" _uuid_value = super().to_python(value) return None if _uuid_value is None else str(_uuid_value) diff --git a/addon_service/common/validators.py b/addon_service/common/validators.py index 9389c187..5655ffd2 100644 --- a/addon_service/common/validators.py +++ b/addon_service/common/validators.py @@ -1,3 +1,8 @@ +"""reusable validators for static vocabularies + +each raises `django.core.exceptions.ValidationError` for invalid values +""" + import enum from django.core.exceptions import ValidationError @@ -16,24 +21,29 @@ def validate_addon_capability(value): + """validator for `AddonCapabilities` names""" _validate_enum_value(AddonCapabilities, value) def validate_invocation_status(value): + """validator for `InvocationStatus` names""" _validate_enum_value(InvocationStatus, value) def validate_service_type(value): + """validator for `ServiceTypes` names""" _validate_enum_value(ServiceTypes, value) def validate_credentials_format(value): + """validator for `CredentialsFormats` names""" _validate_enum_value( CredentialsFormats, value, excluded_members={CredentialsFormats.UNSPECIFIED} ) def validate_storage_imp_number(value): + """validator for `AddonImpNumbers` integer values""" try: _imp_cls = known_imps.get_imp_by_number(value) except KeyError: diff --git a/addon_service/common/view_names.py b/addon_service/common/view_names.py index 8767c955..3056d739 100644 --- a/addon_service/common/view_names.py +++ b/addon_service/common/view_names.py @@ -1,6 +1,11 @@ +"""helpers for a consistent view naming scheme""" + + def detail_view(resource_type: str): + """detail view for the given resource type""" return f"{resource_type}-detail" def related_view(resource_type: str): + """related view for the given resource type""" return f"{resource_type}-related" diff --git a/addon_service/common/viewsets.py b/addon_service/common/viewsets.py index 8a4f7f8b..4f20b36f 100644 --- a/addon_service/common/viewsets.py +++ b/addon_service/common/viewsets.py @@ -81,6 +81,8 @@ class RetrieveWriteViewSet( drf_mixins.UpdateModelMixin, GenericViewSet, ): + """viewset allowing create, retrieve, update""" + http_method_names = ["get", "post", "patch", "head", "options"] @@ -92,10 +94,14 @@ class RetrieveWriteDeleteViewSet( drf_mixins.DestroyModelMixin, GenericViewSet, ): + """viewset allowing create, retrieve, update, delete""" + http_method_names = ["get", "post", "patch", "delete", "head", "options"] class StaticDataclassViewset(ViewSet, RelatedMixin): + """viewset for read-only access to any `StaticDataclassModel`""" + http_method_names = ["get", "head", "options"] authentication_classes = () # as public as the code they're generated from diff --git a/addon_service/configured_storage_addon/serializers.py b/addon_service/configured_storage_addon/serializers.py index 732e1be1..71d0a7cb 100644 --- a/addon_service/configured_storage_addon/serializers.py +++ b/addon_service/configured_storage_addon/serializers.py @@ -17,6 +17,8 @@ class ConfiguredStorageAddonSerializer(serializers.HyperlinkedModelSerializer): + """api serializer for the `ConfiguredStorageAddon` model""" + root_folder = serializers.CharField(required=False, allow_blank=True) url = serializers.HyperlinkedIdentityField( view_name=view_names.detail_view(RESOURCE_TYPE) diff --git a/addon_service/credentials/__init__.py b/addon_service/credentials/__init__.py index ea9b7835..0b7f33a2 100644 --- a/addon_service/credentials/__init__.py +++ b/addon_service/credentials/__init__.py @@ -1 +1,2 @@ -__all__ = () +"""addon_service.credentials: for securely storing credentials +""" diff --git a/addon_service/credentials/encryption.py b/addon_service/credentials/encryption.py index e597fd15..f2695876 100644 --- a/addon_service/credentials/encryption.py +++ b/addon_service/credentials/encryption.py @@ -135,8 +135,8 @@ def _derive_multifernet_key( if not settings.GRAVYVALET_ENCRYPT_SECRET: raise RuntimeError( "gravyvalet can not keep your secrets without a GRAVYVALET_ENCRYPT_SECRET" - " -- ideally chosen by strong randomness, with around 256 bits of entropy" - " (e.g. 64 hex digits; 60 d20 rolls; 20 words of a 10000-word vocabulary)" + " -- ideally chosen by strong randomness, with maybe ~128 bits of entropy" + " (e.g. 32 hex digits; 30 d20 rolls; 10 words of a 10000-word vocabulary)" ) # https://cryptography.io/en/latest/fernet/#cryptography.fernet.MultiFernet return fernet.MultiFernet( diff --git a/addon_service/external_storage_service/__init__.py b/addon_service/external_storage_service/__init__.py index bbc494c8..5306802c 100644 --- a/addon_service/external_storage_service/__init__.py +++ b/addon_service/external_storage_service/__init__.py @@ -1,4 +1,2 @@ -from .models import ExternalStorageService - - -__all__ = ("ExternalStorageService",) +"""addon_service.external_storage_service: represents a third-party storage service +""" diff --git a/addon_service/external_storage_service/serializers.py b/addon_service/external_storage_service/serializers.py index 858e8fa8..6a284edb 100644 --- a/addon_service/external_storage_service/serializers.py +++ b/addon_service/external_storage_service/serializers.py @@ -13,6 +13,8 @@ class ExternalStorageServiceSerializer(serializers.HyperlinkedModelSerializer): + """api serializer for the `ExternalStorageService` model""" + url = serializers.HyperlinkedIdentityField( view_name=view_names.detail_view(RESOURCE_TYPE) ) diff --git a/addon_service/resource_reference/serializers.py b/addon_service/resource_reference/serializers.py index 036e2397..ae8b3efb 100644 --- a/addon_service/resource_reference/serializers.py +++ b/addon_service/resource_reference/serializers.py @@ -13,6 +13,8 @@ class ResourceReferenceSerializer(serializers.HyperlinkedModelSerializer): + """api serializer for the `ResourceReference` model""" + url = serializers.HyperlinkedIdentityField( view_name=view_names.detail_view(RESOURCE_TYPE) ) diff --git a/addon_service/static/gravyvalet_code_docs/.gitkeep b/addon_service/static/gravyvalet_code_docs/.gitkeep new file mode 100644 index 00000000..cc75e478 --- /dev/null +++ b/addon_service/static/gravyvalet_code_docs/.gitkeep @@ -0,0 +1 @@ +(keep `addon_service/static/gravyvalet_code_docs/` for auto-generated code docs -- see `gravyvalet_code_docs/build.py`) diff --git a/addon_service/tasks/invocation.py b/addon_service/tasks/invocation.py index 1363bb67..5f2c5037 100644 --- a/addon_service/tasks/invocation.py +++ b/addon_service/tasks/invocation.py @@ -1,5 +1,4 @@ import celery -from asgiref.sync import sync_to_async from django.db import transaction from addon_service.addon_imp.instantiation import get_storage_addon_instance__blocking @@ -10,13 +9,13 @@ __all__ = ( - "perform_invocation__async", "perform_invocation__blocking", "perform_invocation__celery", ) def perform_invocation__blocking(invocation: AddonOperationInvocation) -> None: + """perform the given invocation: run an operation thru an addon and handle any errors""" # implemented as a sync function for django transactions with dibs(invocation): # TODO: handle dibs errors try: @@ -45,9 +44,6 @@ def perform_invocation__blocking(invocation: AddonOperationInvocation) -> None: invocation.save() -perform_invocation__async = sync_to_async(perform_invocation__blocking) - - @celery.shared_task(acks_late=True) def perform_invocation__celery(invocation_pk: str) -> None: perform_invocation__blocking(AddonOperationInvocation.objects.get(pk=invocation_pk)) diff --git a/addon_service/user_reference/serializers.py b/addon_service/user_reference/serializers.py index 1af79075..383ec2b7 100644 --- a/addon_service/user_reference/serializers.py +++ b/addon_service/user_reference/serializers.py @@ -14,6 +14,8 @@ class UserReferenceSerializer(serializers.HyperlinkedModelSerializer): + """api serializer for the `UserReference` model""" + url = serializers.HyperlinkedIdentityField( view_name=view_names.detail_view(RESOURCE_TYPE) ) diff --git a/addon_toolkit/README.md b/addon_toolkit/README.md new file mode 100644 index 00000000..7e7d5e19 --- /dev/null +++ b/addon_toolkit/README.md @@ -0,0 +1,3 @@ +# addon_toolkit: a python toolkit for implementing gravyvalet imps and services + +(intended to have no dependencies other than python (3.12) built-ins and standard library) diff --git a/addon_toolkit/__init__.py b/addon_toolkit/__init__.py index 88f85886..cb54d472 100644 --- a/addon_toolkit/__init__.py +++ b/addon_toolkit/__init__.py @@ -1,3 +1,15 @@ +""" +.. include:: README.md +""" + +from . import ( + credentials, + cursor, + declarator, + exceptions, + iri_utils, + json_arguments, +) from .addon_operation_declaration import ( AddonOperationDeclaration, AddonOperationType, @@ -23,4 +35,11 @@ "eventual_operation", "immediate_operation", "redirect_operation", + # whole modules: + "credentials", + "cursor", + "declarator", + "exceptions", + "iri_utils", + "json_arguments", ) diff --git a/addon_toolkit/addon_operation_declaration.py b/addon_toolkit/addon_operation_declaration.py index 87501f42..da46e82a 100644 --- a/addon_toolkit/addon_operation_declaration.py +++ b/addon_toolkit/addon_operation_declaration.py @@ -23,14 +23,19 @@ class AddonOperationType(enum.Enum): - REDIRECT = "redirect" # gravyvalet refers you somewhere helpful - IMMEDIATE = "immediate" # gravyvalet does a simple act, waiting to respond until done (success or problem) - EVENTUAL = "eventual" # gravyvalet starts a potentially long-running act, responding immediately with status + """each addon operation has one of these behaviors""" + + REDIRECT = "redirect" + """gravyvalet refers you somewhere helpful""" + IMMEDIATE = "immediate" + """gravyvalet does a simple act, waiting to respond until done (success or problem)""" + EVENTUAL = "eventual" + """gravyvalet starts a potentially long-running act, responding immediately with status""" @dataclasses.dataclass(frozen=True) class AddonOperationDeclaration: - """dataclass for a declared operation method on an interface + """dataclass for a declared operation method on a `addon_toolkit.AddonInterface` created by decorating a method with one of the "operation" decorators: `@redirect_operation`, `@immediate_operation`, `@eventual_operation` @@ -91,7 +96,7 @@ def return_annotation(self) -> Any: # declarator for all types of operations -- use operation_type-specific decorators below addon_operation = Declarator( declaration_dataclass=AddonOperationDeclaration, - field_for_target="operation_fn", + field_for_subject="operation_fn", ) # decorator for operations that may be performed by a client request (e.g. redirect to waterbutler) diff --git a/addon_toolkit/capabilities.py b/addon_toolkit/capabilities.py index bca00157..da31e46e 100644 --- a/addon_toolkit/capabilities.py +++ b/addon_toolkit/capabilities.py @@ -3,11 +3,17 @@ @enum.unique class AddonCapabilities(enum.Flag): - """the source of truth for recognized names in the "addon capabilities" namespace + """ + each addon operation belongs to one of the capabilities defined here, + used for coarse-grained permissions that may be delegated to collaborators. when you want portability (like an open api), use this enum's member names + >>> AddonCapabilities.ACCESS.name + 'ACCESS' when you want compactness (maybe a database), use this enum's member values + >>> AddonCapabilities.ACCESS.value + 1 """ ACCESS = enum.auto() diff --git a/addon_toolkit/constrained_network/__init__.py b/addon_toolkit/constrained_network/__init__.py index 78d44385..9a7e9d2a 100644 --- a/addon_toolkit/constrained_network/__init__.py +++ b/addon_toolkit/constrained_network/__init__.py @@ -1,16 +1,4 @@ -from .http import ( - HttpRequestInfo, - HttpRequestor, - HttpResponseInfo, -) +from . import http -# TODO: from .simple_aiohttp import SimpleAiohttpRequestor - - -__all__ = ( - "HttpRequestInfo", - "HttpRequestor", - "HttpResponseInfo", - # "SimpleAiohttpRequestor", -) +__all__ = ("http",) diff --git a/addon_toolkit/constrained_network/http.py b/addon_toolkit/constrained_network/http.py index 7a9b8ca7..e84a484d 100644 --- a/addon_toolkit/constrained_network/http.py +++ b/addon_toolkit/constrained_network/http.py @@ -57,6 +57,8 @@ def __call__( class HttpRequestor(typing.Protocol): + """an abstract protocol for sending http requests (allowing different implementations)""" + @property def response_info_cls(self) -> type[HttpResponseInfo]: ... diff --git a/addon_toolkit/credentials.py b/addon_toolkit/credentials.py index ff6bb406..5d9ac29d 100644 --- a/addon_toolkit/credentials.py +++ b/addon_toolkit/credentials.py @@ -4,6 +4,8 @@ @dataclasses.dataclass(frozen=True, kw_only=True) class Credentials(typing.Protocol): + """abstract base for dataclasses representing common shapes of credentials""" + def iter_headers(self) -> typing.Iterator[tuple[str, str]]: yield from () diff --git a/addon_toolkit/cursor.py b/addon_toolkit/cursor.py index d92c09e9..617e8e61 100644 --- a/addon_toolkit/cursor.py +++ b/addon_toolkit/cursor.py @@ -19,6 +19,8 @@ def decode_cursor_dataclass(cursor: str, dataclass_class): class Cursor(Protocol): + """an abstract protocol for pagination cursors""" + @classmethod def from_str(cls, cursor: str): return decode_cursor_dataclass(cursor, cls) diff --git a/addon_toolkit/declarator.py b/addon_toolkit/declarator.py index 416525c4..5b71054b 100644 --- a/addon_toolkit/declarator.py +++ b/addon_toolkit/declarator.py @@ -1,3 +1,37 @@ +"""declarator: "declarative" + "decorator" + +define a dataclass with fields you want declared in your decorator, plus a field +to hold the subject of declaration: +>>> @dataclasses.dataclass +... class TwoPartGreetingDeclaration: +... a: str +... b: str +... on: object + +use that dataclass to define a `Declarator`: +>>> greet = Declarator(TwoPartGreetingDeclaration, field_for_subject='on') + +call the declarator with kwargs for the remaining dataclass fields +and use it as a decorator to create a declaration: +>>> @greet(a='hey', b='hello') +... def _hihi(): +... pass + +use the declarator to access declarations by subject: +>>> greet.get_declaration(_hihi) +TwoPartGreetingDeclaration(a='hey', b='hello', on=) + +use `.with_kwargs` to create aliased decorators with static values: +>>> ora = greet.with_kwargs(b='ora') +>>> @ora(a='kia') +... def _kia_ora(): +... pass + +and find that aliased decoration via the original declarator: +>>> greet.get_declaration(_kia_ora) +TwoPartGreetingDeclaration(a='kia', b='ora', on=) +""" + import dataclasses import weakref from typing import ( @@ -8,52 +42,20 @@ ) -DecoratorTarget = TypeVar("DecoratorTarget") +DecoratorSubject = TypeVar("DecoratorSubject") DeclarationDataclass = TypeVar("DeclarationDataclass") @dataclasses.dataclass class Declarator(Generic[DeclarationDataclass]): - """Declarator: add declarative metadata in python using decorators and dataclasses - - define a dataclass with fields you want declared in your decorator, plus a field - to hold the object of declaration: - >>> @dataclasses.dataclass - ... class TwoPartGreetingDeclaration: - ... a: str - ... b: str - ... on: object - - use that dataclass to define a declarator: - >>> greet = Declarator(TwoPartGreetingDeclaration, field_for_target='on') - - call the declarator with kwargs for the remaining dataclass fields - and use it as a decorator to create a declaration: - >>> @greet(a='hey', b='hello') - ... def _hihi(): - ... pass - - use the declarator to access declarations by object: - >>> greet.get_declaration(_hihi) - TwoPartGreetingDeclaration(a='hey', b='hello', on=) - - use `.with_kwargs` to create aliased decorators with static values: - >>> ora = greet.with_kwargs(b='ora') - >>> @ora(a='kia') - ... def _kia_ora(): - ... pass - - and find that aliased decoration via the original declarator: - >>> greet.get_declaration(_kia_ora) - TwoPartGreetingDeclaration(a='kia', b='ora', on=) - """ + """Declarator: declarative metadata using decorators and dataclasses""" declaration_dataclass: type[DeclarationDataclass] - field_for_target: str + field_for_subject: str static_kwargs: dict[str, Any] | None = None # private storage linking a decorated class or function to data gleaned from its decorator - __declarations_by_target: weakref.WeakKeyDictionary[ + __declarations_by_subject: weakref.WeakKeyDictionary[ object, DeclarationDataclass ] = dataclasses.field( default_factory=weakref.WeakKeyDictionary, @@ -64,45 +66,47 @@ def __post_init__(self) -> None: self.declaration_dataclass ), f"expected dataclass, got {self.declaration_dataclass}" assert any( - _field.name == self.field_for_target + _field.name == self.field_for_subject for _field in dataclasses.fields(self.declaration_dataclass) - ), f'expected field "{self.field_for_target}" on dataclass "{self.declaration_dataclass}"' + ), f'expected field "{self.field_for_subject}" on dataclass "{self.declaration_dataclass}"' def __call__( self, **declaration_dataclass_kwargs - ) -> Callable[[DecoratorTarget], DecoratorTarget]: + ) -> Callable[[DecoratorSubject], DecoratorSubject]: """for using a Declarator as a decorator""" - def _decorator(decorator_target: DecoratorTarget) -> DecoratorTarget: - self.set_declaration(decorator_target, **declaration_dataclass_kwargs) - return decorator_target + def _decorator(decorator_subject: DecoratorSubject) -> DecoratorSubject: + self.set_declaration(decorator_subject, **declaration_dataclass_kwargs) + return decorator_subject return _decorator def with_kwargs(self, **static_kwargs) -> "Declarator": """convenience for decorators that differ only by static field values""" - # note: shared __declarations_by_target + # note: shared __declarations_by_subject return dataclasses.replace(self, static_kwargs=static_kwargs) def set_declaration( - self, declaration_target: DecoratorTarget, **declaration_dataclass_kwargs + self, declaration_subject: DecoratorSubject, **declaration_dataclass_kwargs ) -> None: - """create a declaration associated with the target + """create a declaration associated with the subject has the same effect as using the declarator as a decorator with the given kwargs """ # dataclass validates decorator kwarg names - self.__declarations_by_target[declaration_target] = self.declaration_dataclass( - **declaration_dataclass_kwargs, - **(self.static_kwargs or {}), - **{self.field_for_target: declaration_target}, + self.__declarations_by_subject[declaration_subject] = ( + self.declaration_dataclass( + **declaration_dataclass_kwargs, + **(self.static_kwargs or {}), + **{self.field_for_subject: declaration_subject}, + ) ) - def get_declaration(self, target) -> DeclarationDataclass: + def get_declaration(self, subject) -> DeclarationDataclass: try: - return self.__declarations_by_target[target] + return self.__declarations_by_subject[subject] except KeyError: - raise ValueError(f"no declaration found for {target}") + raise ValueError(f"no declaration found for {subject}") class ClassDeclarator(Declarator): @@ -121,7 +125,7 @@ class ClassDeclarator(Declarator): ... subj: type with shorthand declarator: - >>> semver = ClassDeclarator(SemanticVersionDeclaration, field_for_target='subj') + >>> semver = ClassDeclarator(SemanticVersionDeclaration, field_for_subject='subj') for declarating classes: >>> @semver( @@ -136,7 +140,7 @@ class ClassDeclarator(Declarator): >>> semver.get_declaration(MyLongLivedBaseClass) SemanticVersionDeclaration(major=4, minor=2, patch=9, subj=) - but `get_declaration` recognizes only the exact decorated object, not an instance or subclass: + but `get_declaration` recognizes only the exact decorated subject, not an instance or subclass: >>> semver.get_declaration(MyLongLivedBaseClass()) Traceback (most recent call last): ... diff --git a/addon_toolkit/exceptions.py b/addon_toolkit/exceptions.py index 59cff24d..baa524f2 100644 --- a/addon_toolkit/exceptions.py +++ b/addon_toolkit/exceptions.py @@ -2,7 +2,7 @@ class AddonToolkitException(Exception): - pass + """base class for addon_toolkit exceptions""" ### @@ -10,23 +10,11 @@ class AddonToolkitException(Exception): class NotAnImp(AddonToolkitException): - pass + """expected an AddonImp, but this is not""" -class ImpNotValid(AddonToolkitException): - pass - - -class ImpTooAbstract(ImpNotValid): - pass - - -class ImpHasTooManyJobs(ImpNotValid): - pass - - -class ImpHasNoInterface(ImpNotValid): - pass +class ImpHasNoInterface(AddonToolkitException): + """missing required ADDON_INTERFACE class attribute""" ### @@ -34,15 +22,15 @@ class ImpHasNoInterface(ImpNotValid): class NotAnOperation(AddonToolkitException): - pass + """not an operation""" class OperationNotValid(AddonToolkitException): - pass + """invalid operation declaration""" class OperationNotImplemented(AddonToolkitException): - pass + """operation is declared but not implemented (this may be fine)""" ### @@ -50,20 +38,20 @@ class OperationNotImplemented(AddonToolkitException): class JsonArgumentsError(AddonToolkitException): - pass + """base exception for addon_toolkit.json_arguments""" class TypeNotJsonable(JsonArgumentsError): - pass + """tried using a type annotation that is not easily json-able""" class ValueNotJsonableWithType(JsonArgumentsError): - pass + """got a python value mismatched with the expected type annotation""" class InvalidJsonArgsForSignature(JsonArgumentsError): - pass + """tried using json kwargs with a mismatched call signature""" class JsonValueInvalidForType(JsonArgumentsError): - pass + """got a json value mismatched with a given python type annotation""" diff --git a/addon_toolkit/imp.py b/addon_toolkit/imp.py index 62a231e2..4d44cbb4 100644 --- a/addon_toolkit/imp.py +++ b/addon_toolkit/imp.py @@ -78,6 +78,7 @@ def get_operation_declaration( async def invoke_operation( self, operation: AddonOperationDeclaration, json_kwargs: dict ): + """try to run an operation on this imp""" _operation_method = getattr(self, operation.name) _kwargs = kwargs_from_json(operation.operation_fn, json_kwargs) if not inspect.iscoroutinefunction(_operation_method): @@ -87,6 +88,7 @@ async def invoke_operation( return _result invoke_operation__blocking = async_to_sync(invoke_operation) + """try to run an operation on this imp (and wait until done)""" async def get_external_account_id(self, auth_result_extras: dict[str, str]) -> str: """to be implemented by addons which require an external account id""" diff --git a/addon_toolkit/interfaces/storage.py b/addon_toolkit/interfaces/storage.py index d3224a54..837248a6 100644 --- a/addon_toolkit/interfaces/storage.py +++ b/addon_toolkit/interfaces/storage.py @@ -7,7 +7,7 @@ from addon_toolkit.addon_operation_declaration import immediate_operation from addon_toolkit.capabilities import AddonCapabilities -from addon_toolkit.constrained_network import HttpRequestor +from addon_toolkit.constrained_network.http import HttpRequestor from addon_toolkit.cursor import Cursor from addon_toolkit.imp import AddonImp diff --git a/addon_toolkit/iri_utils.py b/addon_toolkit/iri_utils.py index c8b9230f..5d46d3af 100644 --- a/addon_toolkit/iri_utils.py +++ b/addon_toolkit/iri_utils.py @@ -18,6 +18,7 @@ KeyValuePairs = Iterable[tuple[str, str]] | Mapping[str, str] +"""a type alias allowing multiple ways to convey key-value pairs""" class Multidict(Headers): diff --git a/addon_toolkit/json_arguments.py b/addon_toolkit/json_arguments.py index 168faa65..698f7a01 100644 --- a/addon_toolkit/json_arguments.py +++ b/addon_toolkit/json_arguments.py @@ -1,3 +1,6 @@ +"""build and validate json kwargs from python type annotations +""" + from __future__ import annotations import dataclasses @@ -11,12 +14,12 @@ __all__ = ( + "JsonschemaDocBuilder", + "JsonschemaObjectBuilder", "dataclass_from_json", "json_for_dataclass", "json_for_kwargs", "json_for_typed_value", - "JsonschemaDocBuilder", - "JsonschemaObjectBuilder", "kwargs_from_json", "typed_value_from_json", ) @@ -263,6 +266,7 @@ def kwargs_from_json( annotated_callable: typing.Any, args_from_json: dict, ) -> dict: + """parse json into python kwargs""" _signature = inspect.signature(annotated_callable) _annotations = inspect.get_annotations(annotated_callable) try: @@ -284,6 +288,7 @@ def kwargs_from_json( def dataclass_from_json(dataclass: type, dataclass_json: dict): + """parse json into an instance of the given dataclass""" _kwargs = kwargs_from_json(dataclass, dataclass_json) return dataclass(**_kwargs) @@ -291,6 +296,7 @@ def dataclass_from_json(dataclass: type, dataclass_json: dict): def typed_value_from_json( type_annotation: type, json_value: typing.Any, self_type: type | None = None ) -> typing.Any: + """parse json into a python value of the given type""" _type, _contained_type, _is_optional = _unwrap_type( type_annotation, self_type=self_type ) diff --git a/app/urls.py b/app/urls.py index c0c70a3f..9d8a7df2 100644 --- a/app/urls.py +++ b/app/urls.py @@ -3,9 +3,15 @@ include, path, ) +from django.views.generic.base import RedirectView urlpatterns = [ path("v1/", include("addon_service.urls")), path("admin/", admin.site.urls), + path( + "docs", + RedirectView.as_view(url="/static/gravyvalet_code_docs/index.html"), + name="docs-root", + ), ] diff --git a/docker-compose.yml b/docker-compose.yml index 8d9ac8b9..c3929168 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -5,7 +5,7 @@ services: gravyvalet: build: context: . - target: gv-local + target: gv-dev restart: unless-stopped command: python manage.py runserver 0.0.0.0:8004 environment: &gv_environment diff --git a/gravyvalet_code_docs/__init__.py b/gravyvalet_code_docs/__init__.py new file mode 100644 index 00000000..543a5a20 --- /dev/null +++ b/gravyvalet_code_docs/__init__.py @@ -0,0 +1,2 @@ +"""gravyvalet_code_docs: for building gravyvalet docs from python docstrings +""" diff --git a/gravyvalet_code_docs/build.py b/gravyvalet_code_docs/build.py new file mode 100644 index 00000000..357d73df --- /dev/null +++ b/gravyvalet_code_docs/build.py @@ -0,0 +1,50 @@ +from pathlib import Path + +import pdoc + + +_DOCUMENTED_MODULES = ( + "addon_toolkit", + "addon_toolkit.interfaces.storage", + "addon_imps", + "addon_service", +) + + +def build_docs( + output_directory: str = "addon_service/static/gravyvalet_code_docs/", +) -> None: + pdoc.render.configure( + # mermaid=True, + template_directory=_custom_pdoc_template_dir(), + ) + # include the top-level readme for the index page + pdoc.render.env.globals["gravyvalet_readme"] = _gv_readme() + pdoc.pdoc(*_DOCUMENTED_MODULES, output_directory=Path(output_directory)) + + +def _this_directory() -> Path: + return Path(__file__).resolve().parent + + +def _custom_pdoc_template_dir() -> Path: + """where custom pdoc templates live + + see https://github.com/mitmproxy/pdoc/blob/main/examples/custom-template/README.md + """ + return _this_directory() / "custom_pdoc_templates" + + +def _gv_readme() -> str: + with open(_this_directory().parent / "README.md") as _readme: + return _readme.read() + + +if __name__ == "__main__": + import os + + import django # type: ignore[import-untyped] + + os.environ.setdefault("DJANGO_SETTINGS_MODULE", "app.settings") + django.setup() + build_docs() diff --git a/gravyvalet_code_docs/custom_pdoc_templates/index.html.jinja2 b/gravyvalet_code_docs/custom_pdoc_templates/index.html.jinja2 new file mode 100644 index 00000000..1789d078 --- /dev/null +++ b/gravyvalet_code_docs/custom_pdoc_templates/index.html.jinja2 @@ -0,0 +1,27 @@ +{# gravyvalet_code_docs top-level index.html #} + +{% extends "default/index.html.jinja2" %} + +{% block content %} + {# replaces `content` block in pdoc's default index.html.jinja2 template: https://github.com/mitmproxy/pdoc/blob/main/pdoc/templates/default/index.html.jinja2 #} +
+ {% if search %} + + {% endif %} +
+
+ {{ gravyvalet_readme | to_html | safe }} +
+ {% if search %} + {% include "search.html.jinja2" %} + {% endif %} +{% endblock %} + +{% block nav %} + {% set index = gravyvalet_readme | to_html | attr("toc_html") %} + {% if index %} +

Contents

+ {{ index | safe }} + {% endif %} + {{ super() }} +{% endblock %} diff --git a/requirements/dev-requirements.txt b/requirements/dev-requirements.txt index dc48fcbb..fbec7143 100644 --- a/requirements/dev-requirements.txt +++ b/requirements/dev-requirements.txt @@ -10,3 +10,6 @@ flake8 black==24.* isort pre-commit + +# building static docs +pdoc==14.5.1