Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add sidekiq-throttler to sidekiq jobs involving API #256

Open
philippbayer opened this issue Feb 12, 2016 · 4 comments
Open

Add sidekiq-throttler to sidekiq jobs involving API #256

philippbayer opened this issue Feb 12, 2016 · 4 comments

Comments

@philippbayer
Copy link
Member

As in this comment thread: #251 (comment)

Use https://github.com/gevans/sidekiq-throttler for all API jobs to be nicer ✨ 💖 ✨

PLOS has 7200 requests per 24 hours, Mendeley has 500/hour maybe outdated source, Fitbit 150/hour source, SNPedia ???, but something much less

Edit: Looks like sidekiq-throttler only supports Sidekiq 2/3, not 4!

@gedankenstuecke
Copy link
Member

This would address #96 as well I guess? 📕

@philippbayer
Copy link
Member Author

Yes I'd hope so :D

However, sidekiq-throttler looks like it's not supported anymore. sidekiq-rate-limiter looks a bit wonky and unsupported, too. There's a sidekiq4 branch which looks like it works: https://github.com/Docady/sidekiq-rate-limiter/tree/sidekiq-4

@tsujigiri
Copy link
Collaborator

Sidekiq Enterprise has Rate Limiting. No clue how much this would cost... We can ask them, though.

@philippbayer
Copy link
Member Author

One problem is that sidekiq-limit_fetch, which we're currently using, is incompatible with the other rate-limiting gems, since they all patch into the way Sidekiq retrieves jobs from the queue.

One workaround is to remove sidekiq-limit_fetch and use the time-limits for non-API jobs like parsing, too. Yay/Nay?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants