Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data repo currently contains zip files. Upload data files directly to OSF repo (in non-zip file format) #1

Open
evaalonsoortiz opened this issue Dec 4, 2023 · 1 comment
Assignees

Comments

@evaalonsoortiz
Copy link
Contributor

The data repo currently includes zip files. Change this so that the repo contains non-zipped folders and files.

@Kyota-exe
Copy link
Collaborator

I modified the OSF repository for MGH so that it contains the MGH folder and all the individual files directly instead of the zip file. After doing this and modifying the notebook code so it downloads the folder instead of the archive however, I have noticed 3 things:

  1. The time it takes to download the folder has increased considerably: it takes around 24 minutes now, whereas before, it took less than a minute. When we adapt the code so that all sites and subjects are analyzed, this is of course going to increase.
  2. Additionally, from what I understand, there seems to be no way to download an entire folder from OSF through osfclient. As such, I had to clone the entire component. This is not an issue for now since there is only the scan data on each site, but if we add more data to the components in the future, there is no way to only download the scans.
  3. To the best of my knowledge, it is impossible to upload a folder directly to OSF through the web interface (this issue from the OSF GitHub repository confirms this). As such, I used osfclient to upload the folder, which takes ridiculously long, supposedly since it copies each file one at a time. The MGH folder took over 5h to upload. This is not a major issue because we only really need to upload each site once, but it will make modifying the OSF repository data a tedious process.

I believe the cons of storing the scan data as unzipped folders and files far outweigh the benefits, unless there are workarounds for these issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants