-
-
Notifications
You must be signed in to change notification settings - Fork 161
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[MIG] storage_backend_s3: Migration to 16.0 #235
Conversation
- refactor the way to build the url (use a generic base_url). - make more generic the storage backend by moving specific feature in storage file - better name for variable "name" in store and retrieve method use "relative_path" instead - extra amazon S3 storage component in a separated module with test using vcrpy
…t with the specifiation of the type of file binary or base64
* support custom endpoint * refactor bucket handling * re-register vcrpy tests
Every time the selection field was loaded S3 API were called to retrieve regions. This can have a big impact on computing URLs and in any case is useless if we assume that regions do not change every now and then.
a3ea5e6
to
0e1f0ef
Compare
0e1f0ef
to
df8b8e7
Compare
/ocabot migration storage_backend_s3 |
"license": "LGPL-3", | ||
"installable": True, | ||
"external_dependencies": {"python": ["boto3"]}, | ||
"depends": ["storage_backend"], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@abdounasser202 Since the development of the new approach based of fsspecs takes a lot of time, I decided to keep storage_* addons in 16 next to the new fs_* addons even if they cover the same functionalities... It might also be a good idea to facilitate migration from one to another and not block projects that need these features now without being able to wait for the fs_* modules to be finalised.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, @lmignon
then as I'm migrating storage_backend_ftp, it will then be better to have one depending on storage_backend and other one depending on fs_storage to keep both ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, @lmignon then as I'm migrating storage_backend_ftp, it will then be better to have one depending on storage_backend and other one depending on fs_storage to keep both ?
fs_storage
supports natively ftp/ sftp / , S3, azure, DB, ... backends since it uses the fsspec python lib as client layer to communicate with external storage. We don't need additionnal addon by storage we want to support.
@vince-dynapps What's the status of this PR? |
There hasn't been any activity on this pull request in the past 4 months, so it has been marked as stale and it will be closed automatically if no further activity occurs in the next 30 days. |
No description provided.