Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support AWS S3 access #25

Open
zimeon opened this issue Jul 21, 2020 · 3 comments
Open

Support AWS S3 access #25

zimeon opened this issue Jul 21, 2020 · 3 comments
Labels
enhancement New feature or request

Comments

@zimeon
Copy link
Owner

zimeon commented Jul 21, 2020

See pyfilesystem2 branch for work to change over to use PyFilesystem for all file access. This should enable the code to work with regular OS filesystems, S3 and Zipped filesystems among others.

@zimeon zimeon added the enhancement New feature or request label Jul 21, 2020
@zimeon
Copy link
Owner Author

zimeon commented Jul 21, 2020

Have a version running that passes all tests with regular OS filesystem access and using Zipped sets of files (useful for test fixtures with empty dirs that git doesn't support).

Found a gotcha with S3 support via S3FS in that it assumes that there are "directory objects" to help simulate a filesystem (noted in https://fs-s3fs.readthedocs.io/en/latest/#limitations). However, it would be good to be able to validate OCFL objects and storage roots on S3 that do not include ""directory objects". Solution may be to create the S3FS object with strict=False but that doesn't work with the standard open_fs(...) filesystem opener, see PyFilesystem/s3fs#65 (comment)

@zimeon
Copy link
Owner Author

zimeon commented Jul 23, 2020

Maybe it is possible to pass the strict=False into the generic opener by adding a query parameter strict=0 which is parsed and then used in the S3FS version of open_fs https://github.com/PyFilesystem/s3fs/blob/master/fs_s3fs/opener.py#L23-L27

@zimeon
Copy link
Owner Author

zimeon commented Aug 3, 2020

Have merged in pyfilesystem2 branch as version 1.1.0. Needs some more work to tidy the pyfs code, especially the new version of walk that avoids using scandir in ocfl/pyfs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant