Address issues when downloading large matrices as tsv data. #503
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The browser can fail with an out-of-memory error when trying to download a very large data matrix (in my tests hundreds of gigabytes worth). This patch uses several strategies to increase the download size at which that happens: - Gets access windows one row at a time and only for the range of requested columns. This minimizes tile cache memory needed. - Converts rows to tsv format on the fly, so we don't need to convert the entire matrix to tsv format at once. - Constructs a blob using a vector of the row tsv data. This is more memory efficient than manually building a data URL. These steps help but don't eliminate the problem. (I don't think that's possible purely in browser.) A future patch will a display warning notice to the user for very large download sizes.
If the number of array elements to download exceeds a threshold (currently one million array elements) show a warning to the user that the download may kill the browser due to memory exhaustion. If the user chooses to proceed, a progress bar is displayed. If the warning dialog is displayed, the user can cancel the download. For large downloads there is a noticeable delay between when we have finished all processing (and hide the dialog) and when the browser is ready to save the file. The browser can still crash during this time. I'm not sure if there's anything that we can do about that.
bmbroom
changed the title
jmelott
approved these changes
Jun 13, 2023
marohrdanz
approved these changes
Jun 13, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
These two commits address the memory issues with downloading large TSV files from the NG-CHM (see issue #502).
The first tries to minimize the memory required.
The second shows a warning dialog for large downloads. It's currently set to show at 1 million array elements or more. This is way less than I've ever encountered issues.
The dialog also shows a progress bar during the large download and the user can cancel the download at any time.