Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Downloading data from web should be cached #68

Open
lshep opened this issue May 8, 2020 · 2 comments
Open

Downloading data from web should be cached #68

lshep opened this issue May 8, 2020 · 2 comments
Labels
Intermediate Intermediate level Hackathon project

Comments

@lshep
Copy link
Contributor

lshep commented May 8, 2020

If a package is downloading data from the web it should be cached using BiocFileCache

@lshep lshep added Intermediate Intermediate level Hackathon project Hackathon labels May 8, 2020
@lcolladotor
Copy link
Contributor

Hm.... I think this one is really tricky.

I can see the value in caching, which I appreciate and I should probably do it more in a few of my packages. Now, in some cases, I download the data to a specific directory in a specific structure, so I would need to learn how to copy the data from the BiocFileCache to the location I want it in.

Also, what happens when you are accessing say BigWig files from the web? The idea is to just access a small chunk of the file on the web so you don't have to download the whole thing. I'm assuming that would still be ok.

@lshep
Copy link
Contributor Author

lshep commented May 15, 2020

Downloading data to a users system in a specific location is never really recommended because you never know what RW ability a user will have. We use default caching location of a users system that should be writable.

Depending on the download link I think it should be possible. There is also the ability in BiocFileCache to do processing on a downloaded file and save the result rather than the raw file itself.

@lshep lshep removed the Hackathon label Jun 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Intermediate Intermediate level Hackathon project
Projects
None yet
Development

No branches or pull requests

2 participants