Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow partial download of imagesets #52

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Conversation

AbhiDhir
Copy link
Contributor

Overview

  • Allows the option for partial download of imagesets rather than requiring full download \
  • download_full_imagesets is a new optional parameter in config
    • if set to true, behaves how it used to (all imagesets are fully downloaded into cache)
    • if set to false or not present, setSizeFilter must be set to values indicating how many images from each imageset are required
      • This is currently up to the plugin to check for and can be seen in the plugins repo

Notes

  • The current implementation for asynchronous download may be done incorrectly or not as efficiently as possible,
    • I will work more on this by doing time testing
  • This pull request goes along with the request in the ravenml-dataset-plugins repo

@AbhiDhir AbhiDhir self-assigned this Jun 22, 2021
@jacobDeutsch10
Copy link
Contributor

Just tested by making a small dataset and it worked for the most part. The only issue i saw was that it if an image-id doesn't have an associated file for a given prefix/suffix pair it will download an empty file for that image-id and prefix/suffix pair.

in the tfrecord plugin the associated_files are:
associated_files = [ ('meta_', '.json'), ('image_', '.png'), ('image_', '.jpg'), ('image_', '.jpeg'), ('bboxLabels_', '.csv') ]
but since the imageset i used only contains files with for ('meta_', '.json') and ('image_', '.png'). we get empty files for each of the remaining pairs.

I think this could cause some issues if this list was ordered differently

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants