Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: multiproc mapper max threads and default numprocess #112

Merged
merged 2 commits into from
Sep 29, 2023
Merged

fix: multiproc mapper max threads and default numprocess #112

merged 2 commits into from
Sep 29, 2023

Conversation

ab93
Copy link
Member

@ab93 ab93 commented Sep 28, 2023 •
edited
Loading

  • fix issue with default number of threads per process
  • set the default number of process to be the value from CPU count
  • fix handler types for async
  • add slots to multiproc mapper to reduce memory footprint by 20%
  • add poetry check to precommit, to ensure toml and lock files are in sync

@codecov
Copy link

codecov bot commented Sep 28, 2023 •
edited
Loading

Codecov Report

Merging #112 (dd8bb22) into main (43892a9) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main     #112   +/-   ##
=======================================
  Coverage   94.83%   94.84%           
=======================================
  Files          29       29           
  Lines        1084     1086    +2     
  Branches       71       71           
=======================================
+ Hits         1028     1030    +2     
  Misses         32       32           
  Partials       24       24
Files Coverage Δ
pynumaflow/mapper/_dtypes.py 93.75% <100.00%> (+0.07%) ⬆️
pynumaflow/mapper/async_server.py 95.83% <100.00%> (ø)
pynumaflow/mapper/multiproc_server.py 95.65% <100.00%> (+0.09%) ⬆️
pynumaflow/reducer/_dtypes.py 99.15% <100.00%> (ø)

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@ab93 ab93 self-assigned this Sep 29, 2023
@ab93 ab93 added the bug Something isn't working label Sep 29, 2023
@ab93 ab93 marked this pull request as ready for review September 29, 2023 19:09
@ab93 ab93 requested a review from kohlisid September 29, 2023 19:09
@ab93 ab93 merged commit c04a279 into main Sep 29, 2023
9 checks passed
@ab93 ab93 deleted the fix-multiproc-threads branch September 29, 2023 23:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vigith vigith vigith approved these changes

@whynowy whynowy Awaiting requested review from whynowy whynowy is a code owner

@kohlisid kohlisid Awaiting requested review from kohlisid

Assignees

@ab93 ab93

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ab93 @vigith