-
Notifications
You must be signed in to change notification settings - Fork 44
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Docs Improvement] Update Datadog Entry (#6183)
* Progress * Update doc * Restore config steps * Proofread doc and add TOC entry * Add PR nums to log * Add TOC * Update knowledge_base/Datadog.md Co-authored-by: Timothee Legros <[email protected]> * Rm unnecessary postgres config step * remove DD scripts * Rm datadog script documentation --------- Co-authored-by: Timothee Legros <[email protected]> Co-authored-by: Timothee Legros <[email protected]>
- Loading branch information
3 people
authored
Jan 7, 2024
1 parent
e8dc650
commit 28947d8
Showing
6 changed files
with
64 additions
and
159 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,89 +1,92 @@ | ||
| # Datadog Monitoring on Heroku | ||
| # Datadog | ||
|
|
||
| <!-- Break out to Databog Monitoring readme --> | ||
| We use Datadog for three things: metrics, alerts, and ingesting logs from Heroku. | ||
|
|
||
| `Warning:` Following steps below may lead your app to restart several times, please choose appropriate maintainance window. | ||
| The Datadog dashboard can be located at <https://us5.datadoghq.com>. NB: We use the US5 region; attempts to login and access the dashboard from other regions will silently fail. | ||
|
|
||
| ## Helpful guides | ||
| As of 240102, only the main dashboard can be trusted for accurate data reports. Many of the autogenerated dashboards report data differently, leading to inconsistency and errors. | ||
|
|
||
| - [Generic Datadog Heroku Agent Guide](https://docs.datadoghq.com/agent/basic_agent_usage/heroku/) | ||
| - [Datadtog Postgres Guide]( | ||
| https://docs.datadoghq.com/database_monitoring/guide/heroku-postgres/#pagetitle) | ||
| ## Contents | ||
|
|
||
| ## Prerequisites | ||
| - [Metrics](#metrics) | ||
| - [Alerts](#alerts) | ||
| - [Ingesting & Remapping Heroku Logs](#ingesting--remapping-heroku-logs) | ||
| * [Postgres Metrics](#postgres-metrics) | ||
| * [Redis Metrics](#redis-metrics) | ||
| * [RabbitMQ Metrics](#rabbitmq-metrics) | ||
| - [Configuration Prerequisites](#configuration-prerequisites) | ||
| - [Change Log](#change-log) | ||
|
|
||
| **Setup Environment Variable** | ||
| ## Metrics | ||
|
|
||
| ```bash | ||
| DD_AGENT_MAJOR_VERSION=7 | ||
| DD_API_KEY=<SECRET-DATADOG-ACCOUNT-KEY> | ||
| DD_DYNO_HOST=true | ||
| DD_LOG_LEVEL=WARN | ||
| DD_SITE=us5.datadoghq.com | ||
| DD_ENABLE_HEROKU_POSTGRES=true #postgres specific | ||
| DD_HEROKU_CONF_FOLDER=packages/commonwealth/datadog | ||
| ``` | ||
| In our codebase, we use the [StatsDController](../packages/common-common/src/statsd.ts) to increment counters and add tags. | ||
|
|
||
| **Install Datadog Buildpack** | ||
| This can be done by copying the following URL on Heroku App settings page using `Add Buildpack` button: | ||
| For instance, every time `TokenBalanceCache` successfully fetches, we call `.increment` on the controller and then tag the count accordingly. Tags allow us to easily locate the relevant metrics from Datadog's dashboard and generate graphic visualizations. | ||
|
|
||
| ``` | ||
| https://github.com/DataDog/heroku-buildpack-datadog.git | ||
| ``` | ||
| ## Alerts | ||
|
|
||
| **Postgres Monitoring** | ||
| Every Heroku dyno emits a service heartbeat to Datadog every X seconds (e.g. 30, 60) to say “I’m alive.” If the heartbeat isn’t received, Datadog emits an alert to our #eng-feed-datadog Slack channel. | ||
|
|
||
| - Login to heroku on your local, and use helper script `datadog-db-setup` with target app name as first argument | ||
| - This executes script `setup-datadog-postgres.sh` & run `datadog-postgres.sql` in app database | ||
| - Script creates new datadog database user, schema & other database object required by Datadog. | ||
| ## Ingesting & Remapping Heroku Logs | ||
|
|
||
| ```bash | ||
| cd packages/commonwealth | ||
| yarn datadog-db-setup <app-name> <dd-conf-folder> | ||
| #example | ||
| yarn datadog-db-setup commonwealth-frick packages/commonwealth/datadog | ||
| ``` | ||
| Because we host through Heroku, we cannot set up raw metrics the way we normally might in Datadog. Metrics set up from the Datadog dashboard, using the provided member usage or CPU usage statistics, will be inaccurate — since the metrics reported are taken from the underlying server instance, and not from the relevant dyno container. | ||
|
|
||
| **Datadog Postgres Config** | ||
| To set up dyno-scoped metrics on Datadog: | ||
|
|
||
| - Root folder of our app is same as base folder for monorepo, put Postgres Datadog config in root folder of an Heroku app | ||
| 1. Enable Heroku Labs Dyno Metadata and Heroku Labs Log Runtime metrics. Heroku will log dyno metrics directly into our regular log feed. | ||
|
|
||
| Postgres config file `datadog/conf.d/postgres.yaml` | ||
| Leave file generic, parameters would be replace by `prerun.sh` on runtime if available | ||
| 2. Set up Datadog to parse these logs and extract the data we need from them. Datadog ingests logs from Heroku in raw format and passes them through a pipeline which parses and remaps the log metrics. (Heroku buildpack takes care of this pipeline for us.) To display these metrics on the main dashboard, we need to create a Datadog Measure. A Datadog Measure is a key-value pair consisting of a number and reference string (name). To create a new Measure from the dashboard, select a log from the log feed, open the event attributes tab, select the parsed value from the available event attributes, navigate to “Create Measure,” name it, and click "Add." | ||
|
|
||
| ``` | ||
| init_config: | ||
| For general Heroku documentation, see our dedicated [knowledge base entry](./Heroku.md). | ||
|
|
||
| instances: | ||
| - dbm: true | ||
| host: <YOUR HOSTNAME> | ||
| port: <YOUR PORT> | ||
| username: <YOUR USERNAME> | ||
| password: <YOUR PASSWORD> | ||
| dbname: <YOUR DBNAME> | ||
| ssl: True | ||
| ``` | ||
| ### Postgres Metrics | ||
|
|
||
| ## Verify on Datadog | ||
| To enable Postgres metrics, the environment variable `DD_ENABLE_HEROKU_POSTGRES` must be set to `true` in Heroku. | ||
|
|
||
| Visit `https://us5.datadoghq.com/dashboard/lists` for all available dashboards. | ||
| For a list of available `postgresql` metrics, see <https://docs.datadoghq.com/integrations/postgres/?tab=host#metrics>. Since we use Heroku, not all these metrics will be reported and accessible; the only way to know is to create a metric and find out. | ||
|
|
||
| Available Postgres Dashboards: | ||
| For more information, see Datadog's [Postgres Guide]( | ||
| https://docs.datadoghq.com/database_monitoring/guide/heroku-postgres). | ||
|
|
||
| - `Postgres - Metrics` | ||
| - `Postgres - Overview` | ||
| ### Redis Metrics | ||
|
|
||
| If datadog agent picked up postgres config & setting properly, you will see your app database name in top dropdown on postgres dashabords | ||
| To enable Redis metrics, the environment variable `DD_ENABLE_HEROKU_REDIS` must be set to `true` in Heroku. This will automatically trigger Redis metrics ingestion. | ||
|
|
||
| ## Datadog - Dashboard | ||
| For a list of the metrics available on `redis`, see <https://docs.datadoghq.com/integrations/redisdb/?tab=host#metrics>. Since we use Heroku, not all these metrics will be reported and accessible; the only way to know is to create a metric and find out. | ||
|
|
||
| **Copy Widget(s)** | ||
| ### RabbitMQ Metrics | ||
|
|
||
| - click on any dashboard `Cmd+c` or using `share` icon and select `Copy` option | ||
| - paste to your custom dashboard `Cmd+v` | ||
| To set up the RabbitMQ metrics, navigate to "Resources" from the Heroku dashboard. Then pull up CloudAMQP, navigate to the "Integrations" tab listed in the sidebar. Next, navigate to "Logs & Metrics"; from the Metrics section, add Datadog v2 integration. | ||
|
|
||
| Example - Copy `connections` widget from `Postgres - metrics` and paste it your `main dashboard` | ||
| For a list of the metrics available on `rabbitmq`, see <https://docs.datadoghq.com/integrations/rabbitmq/?tab=host#metrics>. Since we use Heroku, not all these metrics will be reported and accessible; the only way to know is to create a metric and find out. | ||
|
|
||
| For more complete RabbitMQ documentation, see our dedicated [knowledge base entry](./RabbitMQ.md). | ||
|
|
||
| ## Configuration Prerequisites | ||
|
|
||
| These steps detail our Datadog-Heroku setup, in case it needs to be redone. | ||
|
|
||
| `Warning:` Following steps below may lead your app to restart several times; please choose an appropriate maintainance window. | ||
|
|
||
| See also Datadog's [Heroku Agent Guide](https://docs.datadoghq.com/agent/basic_agent_usage/heroku/). | ||
|
|
||
| 1. Set the following environment variables: | ||
|
|
||
| ```bash | ||
| DD_AGENT_MAJOR_VERSION=7 | ||
| DD_API_KEY=<SECRET-DATADOG-ACCOUNT-KEY> | ||
| DD_DYNO_HOST=true | ||
| DD_LOG_LEVEL=WARN | ||
| DD_SITE=us5.datadoghq.com | ||
| DD_ENABLE_HEROKU_POSTGRES=true | ||
| DD_ENABLE_HEROKU_REDIS=true | ||
| ``` | ||
|
|
||
| 2. Install the Datadog buildpack. Copy the following URL on Heroku App settings page using `Add Buildpack` button: <https://github.com/Datadog/heroku-buildpack-datadog.git>. | ||
|
|
||
| 3. Verify on Datadog. Visit <https://us5.datadoghq.com/dashboard/lists> for all available dashboards. If the Datadog agent has successfully picked up the Postgres config, you will see your app database name in the top dropdown of the Postgres dashboards. | ||
|
|
||
| ## Change Log | ||
|
|
||
| - 231031: Split off by Graham Johnson from `commonwealth/README.md`. Flagged for overhaul. | ||
| - 240102: Rewritten by Graham Johnson with assistance of Timothee Legros (#6183). | ||
| - 231031: Split off by Graham Johnson from `commonwealth/README.md` (#5521). Flagged for overhaul. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.