Skip to content

Commit

Permalink
Query Builder tests for OOM and memory limit (from PR #2053) (#2054)
Browse files Browse the repository at this point in the history
#### Reference Issues/PRs
<!--Example: Fixes #1234. See also #3456.-->

#### What does this implement or fix?

#### Any other comments?

#### Checklist

<details>
  <summary>
Query Builder tests for OOM and memory limit
  </summary>

The PR contains code that introduces memray and pytest-memray libraries.
With their help it is possible to create 2 types of tests:

- memory leaks test - the memory leaks tests will accept one or two
parameters:
- mem leak threshold - a threshold of mem in KB/MB that will trigger
failure if above
- (optional) filter function - the function can be passed to memray to
filterout code that we do not want to be accounted as mem leak. In
future we might get some code that we KNOW has memory leaks but is
outside of our scope. Thus we can use this function to filter it out..
Currently the code accepts that mem leaks under certain treshold will
not lead to failure. If that threshold is vialoted in future this should
mean immediate development investigative task. In the log there will be
information what is the code that triggered that. IMPORTANT: code source
will be if the test is executed against release build - only the size of
leak will be correct. To know the source of the code the test must be
executed against build with debug information

- memory limit test - the memory limit test is about enforcing memory
efficiency of code. Currently I have set CURRENT memory limit as limit
for the this QueryBuilder code. If test fails in future then that would
mean that our code is LESS memory efficient in that code. Why that has
happened needs to be investigated Also, the number currently set accepts
situation AS IS. It does not makes claim that we are memory efficient.
No such study has been done it is outside of the test purpose. If such
thing is needed then an investigative task need to be created
 
- [ ] Have you updated the relevant docstrings, documentation and
copyright notice?
- [ ] Is this contribution tested against [all ArcticDB's
features](../docs/mkdocs/docs/technical/contributing.md)?
- [ ] Do all exceptions introduced raise appropriate [error
messages](https://docs.arcticdb.io/error_messages/)?
 - [ ] Are API changes highlighted in the PR description?
- [ ] Is the PR labelled as enhancement or bug so it appears in
autogenerated release notes?
</details>

<!--
Thanks for contributing a Pull Request to ArcticDB! Please ensure you
have taken a look at:
- ArcticDB's Code of Conduct:
https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md
- ArcticDB's Contribution Licensing:
https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing
-->

---------

Co-authored-by: Georgi Rusev <Georgi Rusev>
Co-authored-by: Vasil Danielov Pashov <[email protected]>
  • Loading branch information
grusev and vasil-pashov authored Jan 10, 2025
1 parent 32341ae commit d71a0bb
Show file tree
Hide file tree
Showing 5 changed files with 555 additions and 15 deletions.
3 changes: 3 additions & 0 deletions environment_unix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -83,3 +83,6 @@ dependencies:
- trustme
- entt
- psutil
- memray
- pytest-memray

2 changes: 1 addition & 1 deletion python/tests/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -395,7 +395,7 @@ def arctic_client_lmdb_map_size_100gb(lmdb_storage) -> Arctic:
return ac

@pytest.fixture
def arctic_library_lmdb(arctic_client_lmdb_map_size_100gb, lib_name) -> Library:
def arctic_library_lmdb_100gb(arctic_client_lmdb_map_size_100gb, lib_name) -> Library:
return arctic_client_lmdb_map_size_100gb.create_library(lib_name)


Expand Down
Loading

0 comments on commit d71a0bb

Please sign in to comment.