-
Notifications
You must be signed in to change notification settings - Fork 68
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
File storage interface improvements #256
base: master
Are you sure you want to change the base?
Commits on Oct 13, 2020
-
Entry points for storage backends
This adds overridable config variables for mapping storage backend names to classes (by default drawn from declared entry points) and setting a default backend for new files. It also adds a pyfs storage backend entry point, and makes it the default. Storage backends should be `django_files_rest.storage:FileStorage` subclasses, and declared with a `django_files_rest.storage` entrypoint.
Configuration menu - View commit details
-
Copy full SHA for 7dd691d - Browse repository at this point
Copy the full SHA 7dd691dView commit details -
Add a class for computing checksums, counting bytes and reporting pro…
…gress At the moment, the storage backend is responsible for computing checksums and reporting the size of files based on the number of bytes read. However, these are common concerns across all storage backends, and so this functionality can be pulled the other side of the storage backend interface. This class is therefore intended to wrap any file-like object, and to checksum the file as it is read. It also counts bytes and calls the progress callback if one is provided. In a subsequent commit we'll integrate it with FileInstance, and remove the functionality from the PyFS storage backend.
Configuration menu - View commit details
-
Copy full SHA for 0bf2176 - Browse repository at this point
Copy the full SHA 0bf2176View commit details -
Add a new extensible storage factory
This new storage factory uses a `FileInstance.storage_backend` attribute to determine which backend to use. It uses the `FILES_REST_DEFAULT_STORAGE_BACKEND` config value when the file instance doesn't already have a storage backend. It can be extended to customize how the backend is chosen, and how it is initialised. The file path is chosen based on the `id` of the file instance, but this can be customised too. It's not set as the default storage factory to preserve backwards compatibility.
Configuration menu - View commit details
-
Copy full SHA for 7096e97 - Browse repository at this point
Copy the full SHA 7096e97View commit details -
_FilesRESTState.storage_factory deprecates old storage factory, can i…
…nit classes This property can now instantiate class-based storage factories, i.e. the new factory in `invenio_files_rest.storage.factory`. It's possible the deprecation should be in the pyfs storage factory itself.
Configuration menu - View commit details
-
Copy full SHA for 9715111 - Browse repository at this point
Copy the full SHA 9715111View commit details -
These aren't Py3.6-compatible, so will probably need to be removed later, or replaced with comment-based annotations.
Configuration menu - View commit details
-
Copy full SHA for 7774b7d - Browse repository at this point
Copy the full SHA 7774b7dView commit details -
Configuration menu - View commit details
-
Copy full SHA for bca7ee2 - Browse repository at this point
Copy the full SHA bca7ee2View commit details -
Use the passthrough checksummer
This will mean we can later remove this functionality from the storage backend.
Configuration menu - View commit details
-
Copy full SHA for f5244b3 - Browse repository at this point
Copy the full SHA f5244b3View commit details -
Add a metaclass for FileStorage to return the backend name
This is exposed as a class property on FileStorage and can be accessed as `FileStorage.backend_name`. It returns the associated backend name in the application config, which can then be used to find the class.
Configuration menu - View commit details
-
Copy full SHA for c39871a - Browse repository at this point
Copy the full SHA c39871aView commit details -
Simplify the pyfs storage backend implementation
It no longer needs to do checksumming, and the filepath can be moved to the superclass.
Configuration menu - View commit details
-
Copy full SHA for 37bd798 - Browse repository at this point
Copy the full SHA 37bd798View commit details -
Configuration menu - View commit details
-
Copy full SHA for 568cf75 - Browse repository at this point
Copy the full SHA 568cf75View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9bcd900 - Browse repository at this point
Copy the full SHA 9bcd900View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6e94128 - Browse repository at this point
Copy the full SHA 6e94128View commit details -
Lots of things. Needs pulling apart before merging.
All together because this has been very exploratory.
Configuration menu - View commit details
-
Copy full SHA for 5899f79 - Browse repository at this point
Copy the full SHA 5899f79View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2e1a82a - Browse repository at this point
Copy the full SHA 2e1a82aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 282e618 - Browse repository at this point
Copy the full SHA 282e618View commit details -
Remove StorageBackend metaclass
There's always a better option than metaclasses. In this case, a classmethod, which also simplifies getting the backend name on instances (`type(instance).backend_name` → `instance.get_backend_name()`)
Configuration menu - View commit details
-
Copy full SHA for c62cfa0 - Browse repository at this point
Copy the full SHA c62cfa0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 88e5fa8 - Browse repository at this point
Copy the full SHA 88e5fa8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 705b95e - Browse repository at this point
Copy the full SHA 705b95eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 93bd542 - Browse repository at this point
Copy the full SHA 93bd542View commit details -
Configuration menu - View commit details
-
Copy full SHA for a8fbd15 - Browse repository at this point
Copy the full SHA a8fbd15View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8a25500 - Browse repository at this point
Copy the full SHA 8a25500View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7820f84 - Browse repository at this point
Copy the full SHA 7820f84View commit details -
Configuration menu - View commit details
-
Copy full SHA for d4e660e - Browse repository at this point
Copy the full SHA d4e660eView commit details -
Configuration menu - View commit details
-
Copy full SHA for a508e3e - Browse repository at this point
Copy the full SHA a508e3eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 429504f - Browse repository at this point
Copy the full SHA 429504fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0aeefc4 - Browse repository at this point
Copy the full SHA 0aeefc4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8a55a0b - Browse repository at this point
Copy the full SHA 8a55a0bView commit details -
Configuration menu - View commit details
-
Copy full SHA for f03154a - Browse repository at this point
Copy the full SHA f03154aView commit details
Commits on Oct 14, 2020
-
Configuration menu - View commit details
-
Copy full SHA for 501f35c - Browse repository at this point
Copy the full SHA 501f35cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 5ac96fe - Browse repository at this point
Copy the full SHA 5ac96feView commit details -
Configuration menu - View commit details
-
Copy full SHA for 5546899 - Browse repository at this point
Copy the full SHA 5546899View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6b33fd8 - Browse repository at this point
Copy the full SHA 6b33fd8View commit details -
Configuration menu - View commit details
-
Copy full SHA for da397f8 - Browse repository at this point
Copy the full SHA da397f8View commit details -
Configuration menu - View commit details
-
Copy full SHA for f56a7c3 - Browse repository at this point
Copy the full SHA f56a7c3View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3b0868f - Browse repository at this point
Copy the full SHA 3b0868fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 985d64d - Browse repository at this point
Copy the full SHA 985d64dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 55d059f - Browse repository at this point
Copy the full SHA 55d059fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0fede69 - Browse repository at this point
Copy the full SHA 0fede69View commit details
Commits on Oct 15, 2020
-
Configuration menu - View commit details
-
Copy full SHA for dd1a557 - Browse repository at this point
Copy the full SHA dd1a557View commit details
Commits on Nov 20, 2020
-
Configuration menu - View commit details
-
Copy full SHA for 142660d - Browse repository at this point
Copy the full SHA 142660dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4879f59 - Browse repository at this point
Copy the full SHA 4879f59View commit details