Research adjusting the prefork setting for Celery and write an ADR proposing adjustments

[ccostino]

After our large batch send test, there might be some additional configuration we can make with Celery and the prefork worker pool to help improve stability, but we want to make sure it doesn't come at the expense of performance; or, if it does, what we can do to mitigate that.

In addition to the Celery documentation itself, there are some other guides (e.g, Celery School - The Prefork Worker Pool) that could be of help in figuring these pieces out.

Implementation Sketch and Acceptance Criteria

• Do some more research into how best to configure Celery's concurrency options, especially with the prefork worker pool
• Identify possible performance trade-offs and how they might be mitigated
• Write up an ADR (Architectural Decision Record) with the proposed changes so that we can discuss them as a team

Security Considerations

• We want to make sure that the site remains stable and resilient during increased usage, especially as we start to scale more.

[ccostino]

We know this is ready to try out, but until we have more memory available to us in cloud.gov to properly test this in staging, we're not able to adequately test this just yet.

[terrazoon]

I tested locally with 1000 message sends and saw no difference with performance. It's hard to really monitor CPU and memory locally since locally the app has access to my whole laptop, but I think this is worth a try on staging. The goal would be to see a significant drop in CPU usage. Right now on staging during a load test it is easy to see CPU usage at 5000% of what is allocated to us. cloud.gov is letting us get away with that at the moment, but at some point in the future we might get throttled if we don't behave better.