Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Address issues when downloading large matrices as tsv data. #503

Merged
merged 2 commits into from
Jun 13, 2023
Merged

Address issues when downloading large matrices as tsv data. #503

merged 2 commits into from
Jun 13, 2023

Commits on Jun 13, 2023

  1. Improve memory efficiency when downloading matrix data. See issue #502. 

    The browser can fail with an out-of-memory error when trying to download a
very large data matrix (in my tests hundreds of gigabytes worth).

This patch uses several strategies to increase the download size at which
that happens:

- Gets access windows one row at a time and only for the range of
  requested columns. This minimizes tile cache memory needed.
- Converts rows to tsv format on the fly, so we don't need to convert
  the entire matrix to tsv format at once.
- Constructs a blob using a vector of the row tsv data.  This is more
  memory efficient than manually building a data URL.

These steps help but don't eliminate the problem.  (I don't think that's
possible purely in browser.)  A future patch will a display warning notice to
the user for very large download sizes.
    @bmbroom
    bmbroom committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    7b9121d View commit details
    Browse the repository at this point in the history

  2. Show warning message prior to large matrix downloads. See issue #502. 

    If the number of array elements to download exceeds a threshold (currently
one million array elements) show a warning to the user that the download
may kill the browser due to memory exhaustion.

If the user chooses to proceed, a progress bar is displayed.

If the warning dialog is displayed, the user can cancel the download.

For large downloads there is a noticeable delay between when we have
finished all processing (and hide the dialog) and when the browser is
ready to save the file.  The browser can still crash during this time.
I'm not sure if there's anything that we can do about that.
    @bmbroom
    bmbroom committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    06acff0 View commit details
    Browse the repository at this point in the history