From 28947d800b3cf161b056a2c23c47a2b9590b6295 Mon Sep 17 00:00:00 2001 From: Graham Johnson Date: Sun, 7 Jan 2024 14:37:20 -0500 Subject: [PATCH] [Docs Improvement] Update Datadog Entry (#6183) * Progress * Update doc * Restore config steps * Proofread doc and add TOC entry * Add PR nums to log * Add TOC * Update knowledge_base/Datadog.md Co-authored-by: Timothee Legros <62490329+timolegros@users.noreply.github.com> * Rm unnecessary postgres config step * remove DD scripts * Rm datadog script documentation --------- Co-authored-by: Timothee Legros <62490329+timolegros@users.noreply.github.com> Co-authored-by: Timothee Legros --- knowledge_base/Datadog.md | 123 +++++++++--------- knowledge_base/Package-Scripts.md | 9 -- knowledge_base/_TOC.md | 1 + packages/commonwealth/package.json | 1 - .../commonwealth/scripts/datadog-postgres.sql | 28 ---- .../scripts/setup-datadog-postgres.sh | 61 --------- 6 files changed, 64 insertions(+), 159 deletions(-) delete mode 100644 packages/commonwealth/scripts/datadog-postgres.sql delete mode 100755 packages/commonwealth/scripts/setup-datadog-postgres.sh diff --git a/knowledge_base/Datadog.md b/knowledge_base/Datadog.md index 834551d67ad..a6cd52ebd2a 100644 --- a/knowledge_base/Datadog.md +++ b/knowledge_base/Datadog.md @@ -1,89 +1,92 @@ -# Datadog Monitoring on Heroku +# Datadog - +We use Datadog for three things: metrics, alerts, and ingesting logs from Heroku. -`Warning:` Following steps below may lead your app to restart several times, please choose appropriate maintainance window. +The Datadog dashboard can be located at . NB: We use the US5 region; attempts to login and access the dashboard from other regions will silently fail. -## Helpful guides +As of 240102, only the main dashboard can be trusted for accurate data reports. Many of the autogenerated dashboards report data differently, leading to inconsistency and errors. -- [Generic Datadog Heroku Agent Guide](https://docs.datadoghq.com/agent/basic_agent_usage/heroku/) -- [Datadtog Postgres Guide]( -https://docs.datadoghq.com/database_monitoring/guide/heroku-postgres/#pagetitle) +## Contents -## Prerequisites +- [Metrics](#metrics) +- [Alerts](#alerts) +- [Ingesting & Remapping Heroku Logs](#ingesting--remapping-heroku-logs) + * [Postgres Metrics](#postgres-metrics) + * [Redis Metrics](#redis-metrics) + * [RabbitMQ Metrics](#rabbitmq-metrics) +- [Configuration Prerequisites](#configuration-prerequisites) +- [Change Log](#change-log) -**Setup Environment Variable** +## Metrics -```bash - DD_AGENT_MAJOR_VERSION=7 - DD_API_KEY= - DD_DYNO_HOST=true - DD_LOG_LEVEL=WARN - DD_SITE=us5.datadoghq.com - DD_ENABLE_HEROKU_POSTGRES=true #postgres specific - DD_HEROKU_CONF_FOLDER=packages/commonwealth/datadog -``` +In our codebase, we use the [StatsDController](../packages/common-common/src/statsd.ts) to increment counters and add tags. -**Install Datadog Buildpack** -This can be done by copying the following URL on Heroku App settings page using `Add Buildpack` button: +For instance, every time `TokenBalanceCache` successfully fetches, we call `.increment` on the controller and then tag the count accordingly. Tags allow us to easily locate the relevant metrics from Datadog's dashboard and generate graphic visualizations. -``` -https://github.com/DataDog/heroku-buildpack-datadog.git -``` +## Alerts -**Postgres Monitoring** +Every Heroku dyno emits a service heartbeat to Datadog every X seconds (e.g. 30, 60) to say “I’m alive.” If the heartbeat isn’t received, Datadog emits an alert to our #eng-feed-datadog Slack channel. -- Login to heroku on your local, and use helper script `datadog-db-setup` with target app name as first argument -- This executes script `setup-datadog-postgres.sh` & run `datadog-postgres.sql` in app database -- Script creates new datadog database user, schema & other database object required by Datadog. +## Ingesting & Remapping Heroku Logs -```bash -cd packages/commonwealth -yarn datadog-db-setup -#example -yarn datadog-db-setup commonwealth-frick packages/commonwealth/datadog -``` +Because we host through Heroku, we cannot set up raw metrics the way we normally might in Datadog. Metrics set up from the Datadog dashboard, using the provided member usage or CPU usage statistics, will be inaccurate — since the metrics reported are taken from the underlying server instance, and not from the relevant dyno container. -**Datadog Postgres Config** +To set up dyno-scoped metrics on Datadog: -- Root folder of our app is same as base folder for monorepo, put Postgres Datadog config in root folder of an Heroku app +1. Enable Heroku Labs Dyno Metadata and Heroku Labs Log Runtime metrics. Heroku will log dyno metrics directly into our regular log feed. -Postgres config file `datadog/conf.d/postgres.yaml` -Leave file generic, parameters would be replace by `prerun.sh` on runtime if available +2. Set up Datadog to parse these logs and extract the data we need from them. Datadog ingests logs from Heroku in raw format and passes them through a pipeline which parses and remaps the log metrics. (Heroku buildpack takes care of this pipeline for us.) To display these metrics on the main dashboard, we need to create a Datadog Measure. A Datadog Measure is a key-value pair consisting of a number and reference string (name). To create a new Measure from the dashboard, select a log from the log feed, open the event attributes tab, select the parsed value from the available event attributes, navigate to “Create Measure,” name it, and click "Add." -``` -init_config: +For general Heroku documentation, see our dedicated [knowledge base entry](./Heroku.md). -instances: - - dbm: true - host: - port: - username: - password: - dbname: - ssl: True -``` +### Postgres Metrics -## Verify on Datadog +To enable Postgres metrics, the environment variable `DD_ENABLE_HEROKU_POSTGRES` must be set to `true` in Heroku. -Visit `https://us5.datadoghq.com/dashboard/lists` for all available dashboards. +For a list of available `postgresql` metrics, see . Since we use Heroku, not all these metrics will be reported and accessible; the only way to know is to create a metric and find out. -Available Postgres Dashboards: +For more information, see Datadog's [Postgres Guide]( +https://docs.datadoghq.com/database_monitoring/guide/heroku-postgres). -- `Postgres - Metrics` -- `Postgres - Overview` +### Redis Metrics -If datadog agent picked up postgres config & setting properly, you will see your app database name in top dropdown on postgres dashabords +To enable Redis metrics, the environment variable `DD_ENABLE_HEROKU_REDIS` must be set to `true` in Heroku. This will automatically trigger Redis metrics ingestion. -## Datadog - Dashboard +For a list of the metrics available on `redis`, see . Since we use Heroku, not all these metrics will be reported and accessible; the only way to know is to create a metric and find out. -**Copy Widget(s)** +### RabbitMQ Metrics -- click on any dashboard `Cmd+c` or using `share` icon and select `Copy` option -- paste to your custom dashboard `Cmd+v` +To set up the RabbitMQ metrics, navigate to "Resources" from the Heroku dashboard. Then pull up CloudAMQP, navigate to the "Integrations" tab listed in the sidebar. Next, navigate to "Logs & Metrics"; from the Metrics section, add Datadog v2 integration. -Example - Copy `connections` widget from `Postgres - metrics` and paste it your `main dashboard` +For a list of the metrics available on `rabbitmq`, see . Since we use Heroku, not all these metrics will be reported and accessible; the only way to know is to create a metric and find out. + +For more complete RabbitMQ documentation, see our dedicated [knowledge base entry](./RabbitMQ.md). + +## Configuration Prerequisites + +These steps detail our Datadog-Heroku setup, in case it needs to be redone. + +`Warning:` Following steps below may lead your app to restart several times; please choose an appropriate maintainance window. + +See also Datadog's [Heroku Agent Guide](https://docs.datadoghq.com/agent/basic_agent_usage/heroku/). + +1. Set the following environment variables: + + ```bash + DD_AGENT_MAJOR_VERSION=7 + DD_API_KEY= + DD_DYNO_HOST=true + DD_LOG_LEVEL=WARN + DD_SITE=us5.datadoghq.com + DD_ENABLE_HEROKU_POSTGRES=true + DD_ENABLE_HEROKU_REDIS=true + ``` + +2. Install the Datadog buildpack. Copy the following URL on Heroku App settings page using `Add Buildpack` button: . + +3. Verify on Datadog. Visit for all available dashboards. If the Datadog agent has successfully picked up the Postgres config, you will see your app database name in the top dropdown of the Postgres dashboards. ## Change Log -- 231031: Split off by Graham Johnson from `commonwealth/README.md`. Flagged for overhaul. +- 240102: Rewritten by Graham Johnson with assistance of Timothee Legros (#6183). +- 231031: Split off by Graham Johnson from `commonwealth/README.md` (#5521). Flagged for overhaul. diff --git a/knowledge_base/Package-Scripts.md b/knowledge_base/Package-Scripts.md index fb72db07a19..076839d790a 100644 --- a/knowledge_base/Package-Scripts.md +++ b/knowledge_base/Package-Scripts.md @@ -38,7 +38,6 @@ If you add a script to the package.json, you must add documentation here, descri - [start-android](#start-android) - [start-ios](#start-ios) - [Other Services](#other-services) - - [datadog-db-setup](#datadog-db-setup) - [send-cosmos-notifs](#send-cosmos-notifs) - [send-notification-digest-emails](#send-notification-digest-emails) - [Playwright](#playwright) @@ -325,14 +324,6 @@ Contributor: Dillon Chen # Other services -## datadog-db-setup - -Definition: `chmod u+x scripts/setup-datadog-postgres.sh && ./scripts/setup-datadog-postgres.sh` - -Description: Helper script to complete DataDog Postgres account setup, scripts, and required config. Allows us Heroku database monitoring and stats. See [ReadMe](../packages/commonwealth/README.md) for more information on using DataDog. - -Contributor: Nakul Manchanda - ## send-cosmos-notifs Definition: `ts-node --project tsconfig.json server/cosmosGovNotifications/generateCosmosGovNotifications.ts` diff --git a/knowledge_base/_TOC.md b/knowledge_base/_TOC.md index 9721bc3b982..3c246611d14 100644 --- a/knowledge_base/_TOC.md +++ b/knowledge_base/_TOC.md @@ -25,6 +25,7 @@ ## Services & Tooling - [Analytics (Mixpanel)](Analytics.md) +- [Datadog](Datadog.md) - [Custom Domains](Custom-Domains.md) - [Devnet (Cosmos)](Devnet.md) - [Discobot](Discobot.md) diff --git a/packages/commonwealth/package.json b/packages/commonwealth/package.json index c410228a59a..47da9bb7b50 100644 --- a/packages/commonwealth/package.json +++ b/packages/commonwealth/package.json @@ -12,7 +12,6 @@ "check-types": "tsc --noEmit", "clean-db": "ts-node --project tsconfig.json server/scripts/cleanDb.ts", "create-migration": "npx sequelize migration:generate --name", - "datadog-db-setup": "chmod u+x scripts/setup-datadog-postgres.sh && ./scripts/setup-datadog-postgres.sh", "db-all": "yarn reset-db && yarn load-db && yarn migrate-db", "db-doc": "chmod u+x scripts/gen-mermaid-erd.sh && ./scripts/gen-mermaid-erd.sh > ../../knowledge_base/Database-ERD.md", "dump-db": "pg_dump $(heroku config:get CW_READ_DB -a commonwealth-beta) --verbose --exclude-table-data=\"public.\\\"Subscriptions\\\"\" --exclude-table-data=\"public.\\\"Sessions\\\"\" --exclude-table-data=\"public.\\\"LoginTokens\\\"\" --exclude-table-data=\"public.\\\"Notifications\\\"\" --exclude-table-data=\"public.\\\"Webhooks\\\"\" --exclude-table-data=\"public.\\\"NotificationsRead\\\"\" --no-privileges --no-owner -f latest.dump", diff --git a/packages/commonwealth/scripts/datadog-postgres.sql b/packages/commonwealth/scripts/datadog-postgres.sql deleted file mode 100644 index bb485f8e8ff..00000000000 --- a/packages/commonwealth/scripts/datadog-postgres.sql +++ /dev/null @@ -1,28 +0,0 @@ --- create datadog schema -CREATE SCHEMA IF NOT EXISTS datadog; -GRANT USAGE ON SCHEMA datadog TO datadog; -GRANT USAGE ON SCHEMA public TO datadog; -GRANT pg_monitor TO datadog; -CREATE EXTENSION IF NOT EXISTS pg_stat_statements; - --- create datadog.explain_statement function -CREATE OR REPLACE FUNCTION datadog.explain_statement( - l_query TEXT, - OUT explain JSON -) -RETURNS SETOF JSON AS -$$ -DECLARE -curs REFCURSOR; -plan JSON; - -BEGIN - OPEN curs FOR EXECUTE pg_catalog.concat('EXPLAIN (FORMAT JSON) ', l_query); - FETCH curs INTO plan; - CLOSE curs; - RETURN QUERY SELECT plan; -END; -$$ -LANGUAGE 'plpgsql' -RETURNS NULL ON NULL INPUT -SECURITY DEFINER; \ No newline at end of file diff --git a/packages/commonwealth/scripts/setup-datadog-postgres.sh b/packages/commonwealth/scripts/setup-datadog-postgres.sh deleted file mode 100755 index 353b1d6a2b0..00000000000 --- a/packages/commonwealth/scripts/setup-datadog-postgres.sh +++ /dev/null @@ -1,61 +0,0 @@ -#!/bin/bash - -# set -x -SCRIPT_NAME=$(basename "$0") -SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd) - -HEROKU_APP_NAME=$1 -if [[ -z "$HEROKU_APP_NAME" ]]; then - echo "Must provide Heroku app name as first argument" 1>&2 - exit 1 -fi - -DD_HEROKU_CONF_FOLDER=$2 -if [[ -z "$DD_HEROKU_CONF_FOLDER" ]]; then - echo "Must provide DD_HEROKU_CONF_FOLDER as second argument" 1>&2 - exit 1 -fi - -# heroku DATABASE_URL Add-on name -HEROKU_APP_DATABASE=$(heroku pg:info DATABASE_URL -a ${HEROKU_APP_NAME}| grep -i 'Add-on:' | awk '{print $2}') - -# create datadog credential -heroku pg:credentials:create --name datadog --app ${HEROKU_APP_NAME} - -# run datadog postgres script -heroku pg:psql ${HEROKU_APP_DATABASE} -a ${HEROKU_APP_NAME} < ${SCRIPT_DIR}/datadog-postgres.sql - -# this step will be skipped if datadog addon already exists -# this step will restart app -url_list=$(heroku pg:info -a ${HEROKU_APP_NAME} | head -n 1 | tr ',' '\n' | awk '{gsub(/=== /,""); print}' | awk '{$1=$1; print}') -found=false - -for url_name in $url_list; do - url_value=$(heroku config:get $url_name -a ${HEROKU_APP_NAME}) - if [[ $url_value == postgres://datadog* ]]; then - found=true - break - fi -done - -if ! $found; then - echo "Datadog Add-on do not exists creating one" - heroku addons:attach ${HEROKU_APP_DATABASE} --credential datadog -a ${HEROKU_APP_NAME} -else - echo "Datadog Add-on exists" -fi - -# add new config restart(s) app -enable_heroku_postgres=$(heroku config:get DD_ENABLE_HEROKU_POSTGRES -a ${HEROKU_APP_NAME}) -if [ -z "$enable_heroku_postgres" ]; then - echo "Set DD_ENABLE_HEROKU_POSTGRES to true" - heroku config:set DD_ENABLE_HEROKU_POSTGRES=true -a ${HEROKU_APP_NAME} -fi - -# add new config restart(s) app -enable_heroku_conf_folder=$(heroku config:get DD_HEROKU_CONF_FOLDER -a ${HEROKU_APP_NAME}) -if [ -z "$enable_heroku_conf_folder" ]; then - echo "Set DD_HEROKU_CONF_FOLDER to ${DD_HEROKU_CONF_FOLDER}" - heroku config:set DD_HEROKU_CONF_FOLDER=${DD_HEROKU_CONF_FOLDER} -a ${HEROKU_APP_NAME} -fi -