Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to RQ for background jobs #53

Open
5 tasks
jace opened this issue Apr 22, 2016 · 0 comments
Open
5 tasks

Switch to RQ for background jobs #53

jace opened this issue Apr 22, 2016 · 0 comments

Comments

@jace
Copy link
Member

jace commented Apr 22, 2016

Imgee currently uses Celery for background jobs while the rest of HasGeek's code uses RQ. For the sake of maintainability, Imgee should switch to RQ as well.

Imgee currently uses blocking requests when an image is being processed in the background – an earlier iteration returned a placeholder thumbnail until the real one was available. The blocking approach can be implemented with RQ by wrapping RQ with a queue helper:

  1. For every request that needs a background job (thumbnail requested, but one is not in the database), first create a cache key that is deterministic – based on the image id, dimensions and transformations required ("scaled" is only one of many transformations possible)
  2. Consult Redis for whether there is a running RQ job id for this key
  3. If no job, queue the job and save the key to Redis with a TTL of 24 hours
  4. Poll RQ until the job returns a result (should be a redirect URL), sleeping 1 second between polls
  5. Remove Redis cache key and return result to the caller
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant