Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added processes arg to executor call #22

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

stevenmikes
Copy link

Using a distributed job submission platform to submit the function evaluation calls to a cluster. It seemed that the number of evaluations run in parallel was lower than what I set it to with the "batch" parameter. Reading through the Python doc for multiprocessing.Pool, the default for the processes argument is the number of cores on the local machine. Setting processes=batch solved the problem, allowing the number of evaluations set to run in parallel.

@paulknysh
Copy link
Owner

paulknysh commented Jun 9, 2020

So, I was thinking about your change - here is the problem. executor has parameter processes only if it's Pool (default executor). But code in general case accepts any executor from user, including custom ones. Those executors might not have processes parameter (or it would be called something else).. Do you agree?

Do you think it would be possible to refactor your change such that it only affects default executor and not all executors? Perhaps by adding processes argument into get_default_executor() function.

@stevenmikes
Copy link
Author

stevenmikes commented Jun 9, 2020 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants