Improves IO performance for the report readers #104
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
> IO:
* Loop through rows (timesteps) instead of columns (node_ids).
- Changes from tsteps * node_ids reads, to tsteps reads (i.e.,
constant time in terms of IO, regardless of node_ids reqs.).
* Read all node_ids in a buffer every timestep, between min and max
offsets of node_ids requested.
> Memory
* Allocate buffer before passing to HighFive to avoid extra and
unnecessary memory allocations before reading.
* Change data structure of node_ids/offsets from vector to map
for faster search (from n^2 to nlogn).
* In soma reports, assign values directly to return buffer instead
of std::copy/memcpy.
> New features and others:
* Add support for strided reads, allowing to reduce the amount of
timesteps (e.g., 1 by default, 2 for every 2 timesteps, etc.).
* Eliminate duplicated code, avoid calling HDF5 metadata in every
iteration, updated unit tests, and other minor changes.
alkino
previously approved these changes
Oct 9, 2020
|
Really nice way to improve things |
matz-e
previously approved these changes
Oct 9, 2020
|
@jorblancoa and I went through all the comments and suggestions, and integrated everything (specially Jorge, thanks for the effort). Thanks for all the great feedback! If no one has any additional comments, we are ready to close the PR and merge the changes! |
alkino
previously approved these changes
Oct 19, 2020
mgeplf
reviewed
Oct 19, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
IO:
constant time in terms of IO, regardless of node_ids reqs.).
offsets of node_ids requested.
Memory:
unnecessary memory allocations before reading.
for faster search (from n^2 to nlogn).
of std::copy/memcpy.
New features and others:
timesteps (e.g., 1 by default, 2 for every 2 timesteps, etc.).
iteration, updated unit tests, and other minor changes.
Important note
Performance evaluations and other related discussions can be obtained here:
https://bbpteam.epfl.ch/project/issues/browse/REP-60
https://bbpteam.epfl.ch/project/issues/browse/BLPY-217