Accommodate slower NewRelic initialization (+goodies)

[jimleroyer]

Summary | Résumé

The New Relic initialization slowed down due to additional package metadata retrieval starting version 8.10.1 (a minor version -- this should not have happened with such release), adding up from 10 to 30 seconds additional time.

What exacerbate the Kubernetes pod initialization with that behavior is that the pod timeout is set to 30 seconds by default. Hence the New Relic initialization gets past this default pod timeout and restart the workers, restarting the overall initialization again and potentially entering a self-triggered crash loop.

Hence a first step to try to smooth that out is to increase the pod timeout, as well as the gunicorn timeout. We are increasing by 5 seconds the graceful period to begin with and by 10 seconds the default timeout for gunicorn (default was 20 seconds for both gunicorn settings).

• Upgrade New Relic library to 9.1.1, a version that added caching to package metadata retrieval. This will not speed up the first initialization with package metadata retrieval but rather subsequent ones with cache retrieval.
• The newrelic.ini file should also pick up proper environment now, as the latter was not properly passed in previously but that is now fixed.
• Installed a few debugging/profiling tools in the vscode devcontainer including kcachegrind for next fun times.
• Installed nice tools such as ripgrep and fzf (with proper completions).

Related Issues | Cartes liées

• https://app.zenhub.com/workspaces/notify-planning-614b3ad91bc2030015ed22f5/issues/gh/cds-snc/notification-planning/1
• https://app.zenhub.com/workspaces/notify-planning-core-6411dfb7c95fb80014e0cab0/issues/gh/cds-snc/notification-planning-core/1

Test instructions | Instructions pour tester la modification

TODO: Fill in test instructions for the reviewer.

Release Instructions | Instructions pour le déploiement

None.

Reviewer checklist | Liste de vérification du réviseur

• This PR does not break existing functionality.
• This PR does not violate GCNotify's privacy policies.
• This PR does not raise new security concerns. Refer to our GC Notify Risk Register document on our Google drive.
• This PR does not significantly alter performance.
• Additional required documentation resulting of these changes is covered (such as the README, setup instructions, a related ADR or the technical documentation).

⚠ If boxes cannot be checked off before merging the PR, they should be moved to the "Release Instructions" section with appropriate steps required to verify before release. For example, changes to celery code may require tests on staging to verify that performance has not been affected.

[ben851]

ITILGTM

[ben851]

I wonder if this will solve our issue with the lambda in dev

[jimleroyer]

I would guess no, unless we specify some special configurations in the newrelic.ini file meant for dev env.

[P0NDER0SA]

ya, this looks ok @jimleroyer