Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File storage interface improvements #256

Draft
wants to merge 41 commits into
base: master
Choose a base branch
from
Draft

File storage interface improvements #256

wants to merge 41 commits into from

Commits on Oct 13, 2020

  1. Entry points for storage backends

    This adds overridable config variables for mapping storage backend names to
    classes (by default drawn from declared entry points) and setting a default
    backend for new files.
    
    It also adds a pyfs storage backend entry point, and makes it the default.
    
    Storage backends should be `django_files_rest.storage:FileStorage` subclasses,
    and declared with a `django_files_rest.storage` entrypoint.
    alexdutton committed Oct 13, 2020
    Configuration menu
    Copy the full SHA
    7dd691d View commit details
    Browse the repository at this point in the history
  2. Add a class for computing checksums, counting bytes and reporting pro…

    …gress
    
    At the moment, the storage backend is responsible for computing checksums and
    reporting the size of files based on the number of bytes read. However, these
    are common concerns across all storage backends, and so this functionality can
    be pulled the other side of the storage backend interface.
    
    This class is therefore intended to wrap any file-like object, and to checksum
    the file as it is read. It also counts bytes and calls the progress callback if
    one is provided.
    
    In a subsequent commit we'll integrate it with FileInstance, and remove the
    functionality from the PyFS storage backend.
    alexdutton committed Oct 13, 2020
    Configuration menu
    Copy the full SHA
    0bf2176 View commit details
    Browse the repository at this point in the history
  3. Add a new extensible storage factory

    This new storage factory uses a `FileInstance.storage_backend` attribute to
    determine which backend to use. It uses the
    `FILES_REST_DEFAULT_STORAGE_BACKEND` config value when the file instance
    doesn't already have a storage backend.
    
    It can be extended to customize how the backend is chosen, and how it is
    initialised.
    
    The file path is chosen based on the `id` of the file instance, but this can be
    customised too.
    
    It's not set as the default storage factory to preserve backwards compatibility.
    alexdutton committed Oct 13, 2020
    Configuration menu
    Copy the full SHA
    7096e97 View commit details
    Browse the repository at this point in the history
  4. _FilesRESTState.storage_factory deprecates old storage factory, can i…

    …nit classes
    
    This property can now instantiate class-based storage factories, i.e. the new
    factory in `invenio_files_rest.storage.factory`.
    
    It's possible the deprecation should be in the pyfs storage factory itself.
    alexdutton committed Oct 13, 2020
    Configuration menu
    Copy the full SHA
    9715111 View commit details
    Browse the repository at this point in the history
  5. Add some type annotations

    These aren't Py3.6-compatible, so will probably need to be removed later, or
    replaced with comment-based annotations.
    alexdutton committed Oct 13, 2020
    Configuration menu
    Copy the full SHA
    7774b7d View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    bca7ee2 View commit details
    Browse the repository at this point in the history
  7. Use the passthrough checksummer

    This will mean we can later remove this functionality from the storage backend.
    alexdutton committed Oct 13, 2020
    Configuration menu
    Copy the full SHA
    f5244b3 View commit details
    Browse the repository at this point in the history
  8. Add a metaclass for FileStorage to return the backend name

    This is exposed as a class property on FileStorage and can be accessed as
    `FileStorage.backend_name`. It returns the associated backend name in the
    application config, which can then be used to find the class.
    alexdutton committed Oct 13, 2020
    Configuration menu
    Copy the full SHA
    c39871a View commit details
    Browse the repository at this point in the history
  9. Simplify the pyfs storage backend implementation

    It no longer needs to do checksumming, and the filepath can be moved to the
    superclass.
    alexdutton committed Oct 13, 2020
    Configuration menu
    Copy the full SHA
    37bd798 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    568cf75 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    9bcd900 View commit details
    Browse the repository at this point in the history
  12. intermediate

    alexdutton committed Oct 13, 2020
    Configuration menu
    Copy the full SHA
    6e94128 View commit details
    Browse the repository at this point in the history
  13. Lots of things. Needs pulling apart before merging.

    All together because this has been very exploratory.
    alexdutton committed Oct 13, 2020
    Configuration menu
    Copy the full SHA
    5899f79 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    2e1a82a View commit details
    Browse the repository at this point in the history
  15. Fix mypy errors

    alexdutton committed Oct 13, 2020
    Configuration menu
    Copy the full SHA
    282e618 View commit details
    Browse the repository at this point in the history
  16. Remove StorageBackend metaclass

    There's always a better option than metaclasses.
    
    In this case, a classmethod, which also simplifies getting the backend name on
    instances (`type(instance).backend_name` → `instance.get_backend_name()`)
    alexdutton committed Oct 13, 2020
    Configuration menu
    Copy the full SHA
    c62cfa0 View commit details
    Browse the repository at this point in the history
  17. Fix typo

    alexdutton committed Oct 13, 2020
    Configuration menu
    Copy the full SHA
    88e5fa8 View commit details
    Browse the repository at this point in the history
  18. Configuration menu
    Copy the full SHA
    705b95e View commit details
    Browse the repository at this point in the history
  19. Improve checksumming

    alexdutton committed Oct 13, 2020
    Configuration menu
    Copy the full SHA
    93bd542 View commit details
    Browse the repository at this point in the history
  20. Configuration menu
    Copy the full SHA
    a8fbd15 View commit details
    Browse the repository at this point in the history
  21. Configuration menu
    Copy the full SHA
    8a25500 View commit details
    Browse the repository at this point in the history
  22. Configuration menu
    Copy the full SHA
    7820f84 View commit details
    Browse the repository at this point in the history
  23. Configuration menu
    Copy the full SHA
    d4e660e View commit details
    Browse the repository at this point in the history
  24. Configuration menu
    Copy the full SHA
    a508e3e View commit details
    Browse the repository at this point in the history
  25. Configuration menu
    Copy the full SHA
    429504f View commit details
    Browse the repository at this point in the history
  26. pydocstyle fixes

    alexdutton committed Oct 13, 2020
    Configuration menu
    Copy the full SHA
    0aeefc4 View commit details
    Browse the repository at this point in the history
  27. Configuration menu
    Copy the full SHA
    8a55a0b View commit details
    Browse the repository at this point in the history
  28. Configuration menu
    Copy the full SHA
    f03154a View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2020

  1. Configuration menu
    Copy the full SHA
    501f35c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    5ac96fe View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    5546899 View commit details
    Browse the repository at this point in the history
  4. More PEP-8 fixes

    alexdutton committed Oct 14, 2020
    Configuration menu
    Copy the full SHA
    6b33fd8 View commit details
    Browse the repository at this point in the history
  5. The last PEP-8 warnings

    alexdutton committed Oct 14, 2020
    Configuration menu
    Copy the full SHA
    da397f8 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    f56a7c3 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    3b0868f View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    985d64d View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    55d059f View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    0fede69 View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2020

  1. Configuration menu
    Copy the full SHA
    dd1a557 View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2020

  1. Configuration menu
    Copy the full SHA
    142660d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    4879f59 View commit details
    Browse the repository at this point in the history