Improve resilience of posthog integration

[rhysyngsun]

Description/Context

Posthog had an outage which caused calls to their API to timeout.

We worked around this by setting IN_TEST_SUITE=True and FEATURES_DEFAULT=True because we were fortunate enough to have all features enabled. It took some time to evaluate if this was the correct way to address the issue because IN_TEST_SUITE sounds like it could have other undesirable effects.

Plan/Design

• Rename IN_TEST_SUITE to POSTHOG_ENABLED
• Investigate options to tune the timeout/retry settings in posthog
  • The default appears to be 3 retries w/ 15 second timeouts, pushing us well over the 30 second Heroku request timeout.
  • The client itself appears to have settings for these: https://github.com/PostHog/posthog-python/blob/master/posthog/client.py#L32
  • These aren't all exposed through module-level properties, but it looks like we can set default_client
  • See also how the library itself creates a default default_client: https://github.com/PostHog/posthog-python/blob/master/posthog/__init__.py#L441-L455
  • Handle errors from the call to Posthog
• Document procedures for Posthog outages

[pdpinch]

How similar is this to the work @jkachel is doing on mitodl/mit-learn#693

[collinpreston]

@rhysyngsun what does "pushing us well over the 30 second Heroku request timeout." mean?

[rhysyngsun]

Heroku has a 30 second request timeout: https://devcenter.heroku.com/articles/request-timeout

[collinpreston]

This should not be automatically closed. This can be closed once the changes added in mitodl/ol-django#152 are integrated with MITx Online.

[collinpreston]

Blocked by mitodl/ol-django#156

[pdpinch]

Even though we made this change, we had another outage. Unlike the first outage, instead of PostHog timing out, it returned an error. So we need to add better error handling.