Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: add readiness probe for lms/cms #133

Merged
merged 16 commits into from
Sep 11, 2024
Merged

fix: add readiness probe for lms/cms #133

merged 16 commits into from
Sep 11, 2024

Conversation

Ian2012
Copy link
Contributor

@Ian2012 Ian2012 commented Sep 3, 2024 •
edited
Loading

This PR:

  • Disable uwsgi request logging
  • Disable tracking.log and all.log files for lms and cms pods.
  • Removes the prestop hook in favor of a uwsgi hook call to gracefully terminate workers, which improved the time needed to restart or terminate lms/cms pods while being able to answer pending requests
  • Adds livenessProbe for pods.
  • Adds a soft pod AntiAffinity to prefer scheduling pods across nodes.
  • Setups maxUnavailable to 0 to safely rollout deployments without losing capacity.
  • Setups maxSurge to 2 so as not to overcommit resources temporarily for a deployment.

@Ian2012 Ian2012 marked this pull request as ready for review September 9, 2024 21:35
@Ian2012 Ian2012 requested a review from a team as a code owner September 9, 2024 21:35
@Ian2012 Ian2012 requested a review from jfavellar90 September 9, 2024 21:35
jfavellar90
jfavellar90 reviewed Sep 10, 2024
View reviewed changes
drydock/patches/openedx-common-settings Outdated Show resolved Hide resolved
drydock/templates/drydock/k8s/lifecycle/cms.yml Outdated Show resolved Hide resolved
drydock/templates/drydock/k8s/lifecycle/lms.yml Outdated Show resolved Hide resolved
@Ian2012 Ian2012 merged commit f970e01 into main Sep 11, 2024
4 checks passed
@Ian2012 Ian2012 deleted the cag/hardening branch September 11, 2024 14:39
Ian2012 pushed a commit that referenced this pull request Sep 11, 2024
@Ian2012
fix: add readiness probe for lms/cms (#133)
1026f42 
* fix: add readiness probe for lms/cms

* fix: add readiness probe for lms/cms

* fix: add affinity to spread lms/cms to multiple nodes

* chore: remove readiness probe

* fix: reduce startup probe period seconds

* fix: gracefully kill uwsgi workers

* fix: disable local file loggers

* fix: disable logging

* fix: reduce max unavailable to 0

* fix: add liveness probe for cms and lms

* fix: fail early on tracking logger removal

* chore: remove rolling update options

* fix: restore preStop hook

* fix: use right host for cms livenessProbe

* fix: use lms/cms host only

* chore: restore prestopHook

(cherry picked from commit f970e01)
Ian2012 pushed a commit that referenced this pull request Sep 11, 2024
@Ian2012
fix: add readiness probe for lms/cms (#133)
fac73bf 
* fix: add readiness probe for lms/cms

* fix: add readiness probe for lms/cms

* fix: add affinity to spread lms/cms to multiple nodes

* chore: remove readiness probe

* fix: reduce startup probe period seconds

* fix: gracefully kill uwsgi workers

* fix: disable local file loggers

* fix: disable logging

* fix: reduce max unavailable to 0

* fix: add liveness probe for cms and lms

* fix: fail early on tracking logger removal

* chore: remove rolling update options

* fix: restore preStop hook

* fix: use right host for cms livenessProbe

* fix: use lms/cms host only

* chore: restore prestopHook

(cherry picked from commit f970e01)
Ian2012 pushed a commit that referenced this pull request Sep 11, 2024
fix: add readiness probe for lms/cms (#133) (#136)
07883e4 
* fix: add readiness probe for lms/cms

* fix: add readiness probe for lms/cms

* fix: add affinity to spread lms/cms to multiple nodes

* chore: remove readiness probe

* fix: reduce startup probe period seconds

* fix: gracefully kill uwsgi workers

* fix: disable local file loggers

* fix: disable logging

* fix: reduce max unavailable to 0

* fix: add liveness probe for cms and lms

* fix: fail early on tracking logger removal

* chore: remove rolling update options

* fix: restore preStop hook

* fix: use right host for cms livenessProbe

* fix: use lms/cms host only

* chore: restore prestopHook

(cherry picked from commit f970e01)
Ian2012 pushed a commit that referenced this pull request Sep 11, 2024
fix: add readiness probe for lms/cms (#133) (#137)
917032f 
* fix: add readiness probe for lms/cms

* fix: add readiness probe for lms/cms

* fix: add affinity to spread lms/cms to multiple nodes

* chore: remove readiness probe

* fix: reduce startup probe period seconds

* fix: gracefully kill uwsgi workers

* fix: disable local file loggers

* fix: disable logging

* fix: reduce max unavailable to 0

* fix: add liveness probe for cms and lms

* fix: fail early on tracking logger removal

* chore: remove rolling update options

* fix: restore preStop hook

* fix: use right host for cms livenessProbe

* fix: use lms/cms host only

* chore: restore prestopHook

(cherry picked from commit f970e01)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@MoisesGSalas MoisesGSalas MoisesGSalas approved these changes

@jfavellar90 jfavellar90 jfavellar90 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Ian2012 @jfavellar90 @MoisesGSalas