Message send performance and stability (issues from Census send)

[ecayer]

We got through our first large send! And learned a lot of things!

This is a collector epic to address improve performance and scalability (specifically, of Notify sending messages to AWS and getting the data back from AWS) based off what we've just learned from Census.

Specific engineering tasks

• Research adjusting the prefork setting for Celery and write an ADR proposing adjustments notifications-api#1432
• Refactor delivery receipts notifications-api#1465
• Fix "Pending Forever" forever notifications-api#1453
• Fix database inserts for notifications notifications-api#1466
• Fix remaining Redis inserts to ensure they have a TTL (expiration) notifications-api#1468
• Check to make sure that when Celery completes a task, that task and its associated data is removed from Redis immediately notifications-api#1487
• Adjust timing in check_job_status notifications-api#1505
• Remove duplicate log entries notifications-api#1488
• Introduce gaps in the sms_send queue notifications-api#1495

Addressing errors/false positives in New Relic

These might be addressed with the above issues, but if they're not we need to dig further into them.

• Investigate builtins:ValueError - not enough values to unpack (expected 3, got 2) - again notifications-api#1467
• Investigate sqlalchemy.exc:NoResultFound, which could be related to the botocore.errorfactory:NoSuchKey errors notifications-api#1396
• Research why botocore.errorfactory:NoSuchKey errors continue to be thrown and what can be done to prevent them notifications-api#1392

[ccostino]

I pulled out some of the UI-facing issues into a new epic here: #2235

[ccostino]

GSA/notifications-api#1508 is created for the follow-on backend-related work that's come out of these more immediate fixes.

[ccostino]

And now #2237 is created for Platform Admin enhancements as well!

[ccostino]

We should now be in a state where we're in the final stretches for this epic with the first round of improvements and fixes! I believe with the most recent merges we could do one or tests to see if there are any final tweaks to be made in order to close out the remaining open items here.