Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make http calls detached from worker thread #111

Open
fbaligand opened this issue Apr 14, 2020 · 0 comments
Open

Make http calls detached from worker thread #111

fbaligand opened this issue Apr 14, 2020 · 0 comments

Comments

@fbaligand
Copy link

Currently, the http call is tied to current worker thread.
Especially, because of this line:

response = client.send(@http_method, url, :body => body, :headers => headers).call

The fact is: http calls are way more slower than filter processing.
Http calls don't require CPU load. It is waiting time for the worker.
And so, http calls dramatically slow down pipeline workers throughput.
Pipeline workers count should be tied to CPU load, to match filter processing needs (BTW, workers count default value is CPU count).

So it would be great that http calls are really asynchronous, with detached threads.
Detached threads count should be equal to pool_max setting.
Thanks to this, pipeline workers wouldn't be impacted by http calls.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant