Merge pull request #73 from CenterForOpenScience/docs-2024-07

[ENG-5730][ENG-5731] gravyvalet code docs

.gitignore

@@ -2,3 +2,5 @@
 db.sqlite3
 __pycache__
 .venv
+
+addon_service/static/gravyvalet_code_docs/

Dockerfile

@@ -8,18 +8,25 @@ COPY . /code/
 WORKDIR /code
 # END gv-base
 
-# BEGIN gv-local
-FROM gv-base as gv-local
+# BEGIN gv-dev
+FROM gv-base as gv-dev
 # install dev and non-dev dependencies:
 RUN pip3 install --no-cache-dir -r requirements/dev-requirements.txt
 # Start the Django development server
 CMD ["python", "manage.py", "runserver", "0.0.0.0:8004"]
-# END gv-local
+# END gv-dev
+
+# BEGIN gv-docs
+FROM gv-dev as gv-docs
+RUN python -m gravyvalet_code_docs.build
+# END gv-docs
 
 # BEGIN gv-deploy
 FROM gv-base as gv-deploy
 # install non-dev and release-only dependencies:
 RUN pip3 install --no-cache-dir -r requirements/release.txt
+# copy auto-generated static docs (without the dev dependencies that built them)
+COPY --from=gv-docs /code/addon_service/static/gravyvalet_code_docs/ /code/addon_service/static/gravyvalet_code_docs/
 # collect static files into a single directory:
 RUN python manage.py collectstatic --noinput
 # note: no CMD in gv-deploy -- depends on deployment

README.md

@@ -1,26 +1,70 @@
-![Center for Open Science Logo](https://mfr.osf.io/export?url=https://osf.io/download/24697/?direct=%26mode=render&format=2400x2400.jpeg)
+# 🥣 gravyvalet
 
-# OSF Addon Service (GravyValet)
+`gravyvalet` fetches, serves, and holds small ladlefuls of precious bytes.
 
-Welcome to the Open Science Framework's base server for addon integration with our RESTful API (osf.io). This server acts as a gateway between the OSF and external APIs. Authenticated users or machines can access various resources through common file storage and citation management APIs via the OSF. Institutional members can also add their own integrations, tailoring addon usage to their specific communities.
+together with [waterbutler](https://waterbutler.readthedocs.io)
+(which fetches and serves whole streams of bytes, but holds nothing),
+gravyvalet provides an api to support "osf addons",
+whereby you can share controlled access to online accounts
+(e.g. cloud storage) with your collaborators on [osf](https://osf.io).
 
-## Setting up GravyValet Locally
+(NOTE: gravyvalet is still under active development and changes may happen suddenly,
+tho current docs may or may not be available at https://addons.staging.osf.io/docs )
 
+# how to...
+
+## ...set up gravyvalet for local development with osf, using docker
+
+0. have [osf running](https://github.com/CenterForOpenScience/osf.io/blob/develop/README-docker-compose.md) (with its `api` at `http://192.168.168.167:8000`)
 1. Start your PostgreSQL and Django containers with `docker compose up -d`.
-2. Enter the Django container: `docker compose exec addon_service /bin/bash`.
+2. Enter the Django container: `docker compose exec gravyvalet /bin/bash`.
 3. Migrate the existing models: `python manage.py migrate`.
 4. Visit [http://0.0.0.0:8004/](http://0.0.0.0:8004/).
 
-## Running Tests
+## ...run tests
 
 To run tests, use the following command:
 
 ```bash
 python manage.py test
 ```
+(recommend adding `--failfast` when looking for immediate feedback)
+
+## ...set up external services
+start by creating an admin account with
+[django's createsuperuser command](https://docs.djangoproject.com/en/4.2/ref/django-admin/#django-admin-createsuperuser):
+
+```bash
+python manage.py createsuperuser
+```
+
+then log in with that account at `localhost:8004/admin` to manage
+external services (including oauth config) and to create other admin users
 
-Development Tips
 
+## ...configure a good environment
+see `app/env.py` for details on all environment variables used.
+
+when run without a `DEBUG` environment variable (note: do NOT run with `DEBUG` in production),
+some additional checks are run on the environment:
+
+- `GRAVYVALET_ENCRYPT_SECRET` is required -- ideally chosen by strong randomness,
+  with maybe ~128 bits of entropy (e.g. 32 hex digits; 30 d20 rolls; 13 words of a 1000-word vocabulary)
+
+## ...rotate encryption keys responsibly
+don't let your secrets get stale! you can rotate the secret used to derive encryption keys
+(as well as the parameters for key derivation -- see `app/env.py` for details)
+
+1. update environment:
+   - set `GRAVYVALET_ENCRYPT_SECRET` to a new, long, random string (...no commas, tho)
+   - add the old secret to `GRAVYVALET_ENCRYPT_SECRET_PRIORS` (comma-separated list)
+   - (optional) update key-derivation parameters with best-practices du jour
+2. run `python manage.py rotate_encryption` to enqueue key-rotation tasks
+   (on the `gravyvalet_tasks.CHILL` queue by default)
+3. once that queue of tasks is complete, update environment again to remove the old secret from
+   `GRAVYVALET_ENCRYPT_SECRET_PRIORS`
+
+## ...enable pre-commit hooks
 Optionally, but recommended: Set up pre-commit hooks that will run formatters and linters on staged files. Install pre-commit using:
 
 ```bash
@@ -34,7 +78,8 @@ Then, run:
 
 pre-commit install --allow-missing-config
 ```
-Reporting Issues and Questions
+
+## ...ask questions or report issues
 
 If you encounter a bug, have a technical question, or want to request a feature, please don't hesitate to contact us 
 at [email protected]. While we may respond to questions through other channels, reaching out to us at [email protected] ensures 

_TODO.md

@@ -0,0 +1,23 @@
+# TODO: gravyvalet code docs
+
+### how-to/local_setup_with_osf.md
+- with gravyvalet docker-compose.yml
+- with osf.io docker-compose.yml
+- without docker?
+
+### how-to/new_imp_interface.md
+- defining interface with operations
+- required adds to addon_service
+
+### how-to/migrating_osf_addon_to_imp.md
+- implementing imp
+- current limitations
+
+### how-to/new_storage_imp.md
+- implementing imp
+- required changes to waterbutler? (with mention of ideal "none")
+
+### how-to/key_rotation.md
+- credentials encryption overview
+- secret and prior secrets
+- scrypt configuration

addon_imps/storage/__init__.py

@@ -1 +1,2 @@
-__all__ = ()
+"""addon_imps.storage: imps that implement a "storage"-like interface
+"""

addon_service/README.md

@@ -0,0 +1,80 @@
+# addon_service: a django app for the gravyvalet web api
+
+## network flows
+
+addon operation invocation thru gravyvalet (as currently implemented with osf)
+```mermaid
+sequenceDiagram
+    participant browser
+    Box *.osf.io
+        participant gravyvalet
+        participant osf-api
+    end
+    Note over browser: browsing files on osf, say
+    browser->>gravyvalet: request directory listing (create an addon operation invocation)
+    gravyvalet->>osf-api: who is this?
+    osf-api->>gravyvalet: this is who
+    Note over gravyvalet: gravyvalet asks "who is?" (auth-entic-ation) separate from "may they?" (auth-oriz-ation)
+    gravyvalet->>osf-api: may they access this directory listing?
+    alt no
+        osf-api->>gravyvalet: no
+        gravyvalet->>browser: no
+    else yes
+        osf-api->>gravyvalet: yes
+        gravyvalet->>external-service: request directory listing
+        external-service->>gravyvalet: serve directory listing
+        Note over gravyvalet: listing translated into interoperable format
+        gravyvalet->>browser: serve directory listing
+    end
+```
+
+download a file thru waterbutler, with get_auth and gravyvalet (as currently implemented)
+```mermaid
+sequenceDiagram
+    participant browser
+    Box *.osf.io
+        participant waterbutler
+        participant osf-v1
+        participant gravyvalet
+    end
+    browser->>waterbutler: request file
+    waterbutler->>osf-v1: get_auth
+    alt no
+        osf-v1->>waterbutler: no
+        waterbutler->>browser: no
+    else yes
+        osf-v1->>gravyvalet: request gravy
+        gravyvalet->>osf-v1: serve gravy
+        osf-v1->>waterbutler: credentials and config
+        waterbutler->>external service: request file
+        external service->>waterbutler: serve file
+        waterbutler->>browser: serve file
+    end
+```
+
+hypothetical world where waterbutler talks to gravyvalet... is this better than get_auth?
+```mermaid
+sequenceDiagram
+    participant browser
+    Box *.osf.io
+        participant waterbutler
+        participant gravyvalet
+        participant osf-api
+    end
+    browser->>waterbutler: request file
+    waterbutler->>gravyvalet: request gravy
+    gravyvalet->>osf-api: who is this?
+    osf-api->>gravyvalet: this is who
+    gravyvalet->>osf-api: may they do what they're asking to?
+    alt no
+        osf-api->>gravyvalet: no
+        gravyvalet->>waterbutler: no
+        waterbutler->>browser: no
+    else yes
+        osf-api->>gravyvalet: yes
+        gravyvalet->>waterbutler: serve gravy
+        waterbutler->>external service: request file
+        external service->>waterbutler: serve file
+        waterbutler->>browser: serve file
+    end
+```

addon_service/__init__.py

@@ -1 +1,20 @@
-__all__ = ()
+"""
+.. include:: README.md
+"""
+
+__all__ = (
+    "addon_imp",
+    "addon_operation",
+    "addon_operation_invocation",
+    "authentication",
+    "authorized_storage_account",
+    "common",
+    "configured_storage_addon",
+    "credentials",
+    "external_storage_service",
+    "oauth1",
+    "oauth2",
+    "resource_reference",
+    "tasks",
+    "user_reference",
+)

addon_service/addon_imp/__init__.py

@@ -1 +1,2 @@
-__all__ = ()
+"""addon_service.addon_imp: for representing static-known addon implementations in the api
+"""

addon_service/addon_imp/instantiation.py

@@ -14,6 +14,10 @@ async def get_storage_addon_instance(
     account: AuthorizedStorageAccount,
     config: StorageConfig,
 ) -> StorageAddonImp:
+    """create an instance of a `StorageAddonImp`
+
+    (TODO: decide on a common constructor for all `AddonImp`s, remove this)
+    """
     assert issubclass(imp_cls, StorageAddonImp)
     return imp_cls(
         config=config,
@@ -26,3 +30,7 @@ async def get_storage_addon_instance(
 
 
 get_storage_addon_instance__blocking = async_to_sync(get_storage_addon_instance)
+"""create an instance of a `StorageAddonImp`
+
+(same as `get_storage_addon_instance`, for use in synchronous context
+"""

addon_service/addon_imp/models.py

@@ -8,10 +8,10 @@
 from addon_toolkit import AddonImp
 
 
-# dataclass wrapper for a concrete subclass of AddonImp which
-# meets rest_framework_json_api expectations on a model class
 @dataclasses.dataclass(frozen=True)
 class AddonImpModel(StaticDataclassModel):
+    """each `AddonImpModel` represents a statically defined subclass of `AddonImp`"""
+
     imp_cls: type[AddonImp]
 
     ###

addon_service/addon_imp/serializers.py

@@ -12,6 +12,8 @@
 
 
 class AddonImpSerializer(serializers.Serializer):
+    """api serializer for the `AddonImpModel` model"""
+
     url = serializers.HyperlinkedIdentityField(
         view_name=view_names.detail_view(RESOURCE_TYPE)
     )

addon_service/addon_operation/serializers.py

@@ -14,6 +14,8 @@
 
 
 class AddonOperationSerializer(serializers.Serializer):
+    """api serializer for the `AddonOperationModel` model"""
+
     url = serializers.HyperlinkedIdentityField(
         view_name=view_names.detail_view(RESOURCE_TYPE)
     )

addon_service/addon_operation_invocation/__init__.py

@@ -1 +1,2 @@
-__all__ = ()
+"""addon_service.addon_operation_invocation: a specific invocation of a gravyvalet addon operation
+"""

addon_service/addon_operation_invocation/serializers.py

@@ -20,6 +20,8 @@
 
 
 class AddonOperationInvocationSerializer(serializers.HyperlinkedModelSerializer):
+    """api serializer for the `AddonOperationInvocation` model"""
+
     class Meta:
         model = AddonOperationInvocation
         fields = [

addon_service/authorized_storage_account/serializers.py

@@ -31,6 +31,8 @@
 
 
 class AuthorizedStorageAccountSerializer(serializers.HyperlinkedModelSerializer):
+    """api serializer for the `AuthorizedStorageAccount` model"""
+
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
 

addon_service/common/__init__.py

@@ -1 +1,2 @@
-__all__ = ()
+"""addon_service.common: shared logic among addon_service types
+"""

addon_service/common/aiohttp_session.py

@@ -16,14 +16,15 @@
 
 
 async def get_singleton_client_session() -> aiohttp.ClientSession:
-    if not is_session_valid():
+    """return a reusable aiohttp client session (thread-local singleton)"""
+    if not _is_session_valid():
         __SINGLETON_CLIENT_SESSION_STORE.session = aiohttp.ClientSession(
             cookie_jar=aiohttp.DummyCookieJar(),  # ignore all cookies
         )
     return __SINGLETON_CLIENT_SESSION_STORE.session
 
 
-def is_session_valid() -> bool:
+def _is_session_valid() -> bool:
     return (
         hasattr(__SINGLETON_CLIENT_SESSION_STORE, "session")
         and isinstance(__SINGLETON_CLIENT_SESSION_STORE.session, aiohttp.ClientSession)
@@ -32,10 +33,20 @@ def is_session_valid() -> bool:
 
 
 async def close_singleton_client_session() -> None:
-    if is_session_valid():
+    """close the reusable aiohttp client session (thread-local singleton)"""
+    if _is_session_valid():
         await __SINGLETON_CLIENT_SESSION_STORE.close()
         __SINGLETON_CLIENT_SESSION_STORE.session = None
 
 
 get_singleton_client_session__blocking = async_to_sync(get_singleton_client_session)
+"""return a reusable aiohttp client session (thread-local singleton)
+
+(same as `get_singleton_client_session`, for use in non-async context)
+"""
+
 close_singleton_client_session__blocking = async_to_sync(close_singleton_client_session)
+"""close the reusable aiohttp client session (thread-local singleton)
+
+(same as `close_singleton_client_session`, for use in non-async context)
+"""

addon_service/common/base_model.py

@@ -8,6 +8,8 @@
 
 
 class AddonsServiceBaseModel(models.Model):
+    """common base class for all addon_service models"""
+
     id = StrUUIDField(primary_key=True, default=str_uuid4, editable=False)
     created = models.DateTimeField(editable=False)
     modified = models.DateTimeField()