Workers vs threads

[Nondv]

Hi @msolli !

Started to use your library for background tasks in our service. Loving it so far (reminds me of Ruby's delayed_jobs).

I was wondering if there's a difference between running multiple workers vs one worker with multiple threads.
From briefly looking at the code, there doesn't seem to be much difference but was hoping to get your opinion.

[msolli]

Hi @Nondv! Glad you like it, and sorry for the late reply. It indeed serves the same purpose as, and is in part inspired by delayed_jobs, which I have used many years ago when I was working with Ruby.

Whether to use multiple workers or one worker with multiple threads depends on your existing deployment setup, your parallellization needs, and your willingness to throw hardware (and thus, money) on your background jobs:

• If you already have a fleet of web app servers, you could co-locate the Proletarian workers in the same JVM process. At work we use exactly this approach - there are two web app servers, and each of them has two threads for Proletarian worker jobs, which gives us four parallell worker threads in total (except when deploying, then the number of instances can vary between one and four, depending on the whims of AWS auto-scaling groups). This is operationally very simple - no new infra or hardware, and worker code stays in sync with web app code.
• You could run Proletarian as a separate instance, or instances. Its runtime and deployment lifecycle is then divorced from the rest of the app. You can scale the workers independent of the web apps. If using only one instance (with one or several worker threads), then that is a nice and simple setup, but would give downtime during deploys. Downtime for background tasks might not be critical, but it would translate to a delay in execution of the tasks.

I'd say go for the first option if you have simple IO-bound tasks (like sending emails) that don't need a lot of resources. If you need to scale the workers independently, go for a separate worker fleet of instances.