FIX-#6778: Read parquet files without file extensions using fastparquet #6789
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ing fastparquet In supporting fastparquet, modin takes the paths provided, globs them, and filters them to only look at files with the .parq or .parquet extension. This commit adds support so that if the path supplied is explicitly a file, it will be included. Signed-off by: Ari Brown <[email protected]>
seydar
requested review from
devin-petersohn,
mvashishtha,
RehanSD,
YarShev,
vnlitvinov,
anmyachev,
dchigarev and
a team
as code owners
3 tasks
…ing fastparquet In supporting fastparquet, modin takes the paths provided, globs them, and filters them to only look at files with the .parq or .parquet extension. This commit adds support so that if the path supplied is explicitly a file, it will be included. This commit also fixes the formatting issues for the `black` linter. Signed-off by: Ari Brown <[email protected]>
|
@seydar awesome PR! Please apply auto linter UPD: |
|
@anmyachev Whoops, sorry about that. I though I ran it, but evidently missed that one. In rectifying it, not sure if I did it right, but I closed this PR and made a new PR with the corrections all in a single commit: #6790 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In supporting fastparquet, modin takes the paths provided, globs them, and filters them to only look at files with the .parq or .parquet extension. This commit adds support so that if the path supplied is explicitly a file, it will be included.
Signed-off by: Ari Brown [email protected]
What do these changes do?
If a a file is passed to
modin.pandas.read_parquet(..., engine='fastparquet')but the file does not have ".parq" or ".parquet" in the name, it will be ignored when determining the number of partitions. This change looks at all of the paths passed, and if the path is a file, then it considers it to be a parquet file.flake8 modin/ asv_bench/benchmarks scripts/doc_checker.pyblack --check modin/ asv_bench/benchmarks scripts/doc_checker.pygit commit -sdocs/development/architecture.rstis up-to-date