feat(#81): replace deprecated SQL exporter with non-deprecated one (#100

)

* Replace deprecated SQL exporter with non-deprecated one per #81

* remove comments to clean up yaml file

* default to local cht-couch2pg db values in example

* set example yaml file to actually work with default couch2pg settings

* set min_interval 60s which means calls happening <60s hit cache instead DB directly

* remove config file that shouldn't have been committed

* update git ignore, place holder for prom scrape

* omg get entry command and file name correct

* got prometheus scrape config working \o/

* adding scrape config

* finalize dashboard, update example sql, remove comments from scrape, change gauge -> counter

* add dev restart script, update detail dashboard json

* start to get dev and test environs to work

* normalize passwords for sql and couch2pg

* feat(na): path to new compose file in CI, update readme how to add new sql

* feat(na): fix path to ci test files

* feat(na): update git ignore to include test fake file

* fix(na): add set -e and fake cht to restart script per feedback

.env.example

@@ -16,3 +16,10 @@ PROMETHEUS_VERSION=latest
 # Note: Relative paths do not work with docker-compose v2.x. In that case, simply use an absolute path.
 PROMETHEUS_DATA="./prometheus/data"
 PROMETHEUS_RETENTION_TIME=60d
+
+
+# Use 0.0.0.0 to expose to LAN/Internet, use 127.0.0.1 to
+#    use with reverse proxy (Caddy, Traefik etc)
+SQL_EXPORTER_IP=127.0.0.1
+# 9399 is default from upstream project
+SQL_EXPORTER_PORT=9399

.gitignore

@@ -10,3 +10,4 @@ postgres_exporter.yml
 grafana.ini
 delete-rules.yml
 build
+exporters/postgres/sql_servers.yml

development/README.md

@@ -61,7 +61,7 @@ Metrics loaded from the CHT `/monitoring` endpoint are configured via the json-e
 
 ### Adding a new metric from a Couch2pg Postgres DB
 
-Metrics loaded from a Couch2pg Postgres DB are configured via the postgres-exporter's [`cht-queries.yml`](../exporters/postgres/config/cht-queries.yml) file. New entries into the file can specify the associated Postgres query for loading the metric data. See the [postgres_exporter project](https://github.com/prometheus-community/postgres_exporter) on GitHub for more information.
+Metrics loaded from a Couch2pg Postgres DB are configured via the sql_exporter's [`couch2pg_collector.yml`](../exporters/postgres/couch2pg_collector.yml) file. New queries can be added by copying `couch2pg_collector.yml` into a new file following the `*_collector.yml` nomenclature. For example `reports-by-type_collector.yml` could be created with a new query to gather all CHW submitted reports. See the [sql_exporter project](https://github.com/burningalchemist/sql_exporter) on GitHub for more information.
 
 ### Scraping metrics from a new resource
 
@@ -114,7 +114,7 @@ Copy the example config files:
 
 ```shell
 cp development/fake-cht/example-config/cht-instances.yml cht-instances.yml
-cp development/fake-cht/example-config/postgres* ./exporters/postgres
+cp development/fake-cht/example-config/sql_servers.yml ./exporters/postgres/.
 ```
 
 You will also need to run a few additional commands from the [normal setup process](https://docs.communityhealthtoolkit.org/apps/guides/hosting/monitoring/setup/#setup) to prepare your new instance:
@@ -129,13 +129,13 @@ mkdir -p grafana/data && mkdir  -p prometheus/data
 From the root directory, run:
 
 ```shell
-docker compose -f docker-compose.yml -f exporters/postgres/docker-compose.postgres-exporter.yml -f development/fake-cht/docker-compose.fake-cht.yml up -d
+docker compose -f docker-compose.yml -f exporters/postgres/compose.yml -f development/fake-cht/docker-compose.fake-cht.yml up -d
 ```
 
 The Postgres data will be persisted in a Docker volume. To clear the data when you are finished testing (to allow for a fresh environment on the next run), run your `docker compose down` command with the `-v` flag to delete the volume.
 
 ```shell
-docker compose -f docker-compose.yml -f exporters/postgres/docker-compose.postgres-exporter.yml -f development/fake-cht/docker-compose.fake-cht.yml down -v
+docker compose -f docker-compose.yml -f exporters/postgres/compose.yml -f development/fake-cht/docker-compose.fake-cht.yml down -v
 ```
 
 ### Historical data
@@ -149,7 +149,7 @@ Each test is associated with an `xlsx` file in this directory that contains the
 Start a fresh deployment of cht-watchdog without providing any CHT URL and with the test override:
 
 ```
-docker compose -f docker-compose.yml -f exporters/postgres/docker-compose.postgres-exporter.yml -f development/fake-cht/docker-compose.fake-cht.yml up -d
+docker compose -f docker-compose.yml -f exporters/postgres/compose.yml -f development/fake-cht/docker-compose.fake-cht.yml up -d
 ```
 
 Open the `xlsx` file of the test you want to run. Switch to the `data` sheet and Save As a `csv` file (named `data.csv`) in the `development` directory.

development/fake-cht/docker-compose.fake-cht.yml

@@ -20,8 +20,8 @@ services:
     image: postgres:15
     environment:
       POSTGRES_DB: cht
-      POSTGRES_USER: postgres_root
-      POSTGRES_PASSWORD: postgres_root_password
+      POSTGRES_USER: cht_couch2pg
+      POSTGRES_PASSWORD: cht_couch2pg_password
     volumes:
       - fake-cht-postgres-data:/var/lib/postgresql/data
     networks:

development/fake-cht/example-config/sql_servers.yml

@@ -0,0 +1,29 @@
+# Global settings and defaults.
+global:
+  # Subtracted from Prometheus' scrape_timeout to give us some headroom and prevent Prometheus from
+  # timing out first.
+  scrape_timeout_offset: 500ms
+  # Minimum interval between collector runs: by default (0s) collectors are executed on every scrape.
+  # Medic recommends setting this to 60s so that the endpoint cant be used as a proxy to DOS the upstream Postgres server
+  # see https://github.com/burningalchemist/sql_exporter/issues/447#issue-2144289744 for more info
+  min_interval: 60s
+  # Maximum number of open connections to any one target. Metric queries will run concurrently on
+  # multiple connections.
+  max_connections: 3
+  # Maximum number of idle connections to any one target.
+  max_idle_connections: 3
+  # Maximum amount of time a connection may be reused to any one target. Infinite by default.
+  max_connection_lifetime: 10m
+
+collector_files:
+  - "/etc/sql_exporter/couch2pg_collector.yml"
+
+jobs:
+  - job_name: db_targets
+    collectors: [couch2pg]
+    enable_ping: true
+    static_configs:
+      - targets:
+          # " same as URL in cht-instances.yml      ": 'postgres://USERNAME:PASSWORD@DB_SERVER_IP/DATABASE
+          "fake-cht:8081": 'postgres://cht_couch2pg:cht_couch2pg_password@postgres:5432/cht?sslmode=disable' # //NOSONAR - password is safe to commit
+

development/fake-cht/src/postgres.js

@@ -2,10 +2,10 @@ const { Client } = require('pg');
 const { randomUUID } = require('crypto');
 
 const POSTGRES_CLIENT_CONFIG = {
-  user: 'postgres_root',
+  user: 'cht_couch2pg',
   host: 'postgres',
   database: 'cht',
-  password: 'postgres_root_password',
+  password: 'cht_couch2pg_password',
   port: 5432,
 };
 

development/kill.start.ips.sh

@@ -0,0 +1,60 @@
+#!/bin/bash
+set -e
+echo '
+    ,    /-.
+   ((___/ __>
+   /      }      ---- CHT WATCHDOG RESTART SCRIPT ----
+   \ .--.(
+    \\   \\
+'
+
+echo;echo "Killing...";echo
+docker compose  \
+    -f docker-compose.yml \
+    -f exporters/postgres/compose.yml \
+    -f development/fake-cht/docker-compose.fake-cht.yml \
+    kill
+
+echo;echo "Starting...";echo
+docker compose  \
+    -f docker-compose.yml \
+    -f exporters/postgres/compose.yml \
+    -f development/fake-cht/docker-compose.fake-cht.yml \
+    up -d \
+    --remove-orphans
+
+# shellcheck disable=SC2046
+ips=$(docker inspect $(docker ps -q ) \
+  --format='{{ printf "%-50s" .Name}} {{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}'  \
+  | tr "\/" " " | tr -s " " )
+
+# shellcheck disable=SC2046
+readarray -t ipsArray <<< $(echo -e "${ips}")
+
+final=""
+# loop it
+for ipLine in "${ipsArray[@]}"; do
+  ip=$(echo "$ipLine"  | cut -f3 -d" ")
+  name=$(echo "$ipLine"  |cut -f2 -d" ")
+  case $name in
+    cht-watchdog-extra_sql_exporter-1)
+      portPath=":9187/metrics";;
+    cht-watchdog-grafana-1)
+      portPath=":3000";;
+    cht-watchdog-postgres-exporter-1)
+      portPath=":9187/metrics";;
+    cht-watchdog-json-exporter-1)
+      portPath=":7979/metrics";;
+    cht-watchdog-prometheus-1) 
+      portPath=":9090/targets?search=";;
+    cht-watchdog-sql_exporter-1)
+      portPath=":9399/metrics";;
+    *)
+      portPath="";;
+  esac
+  final+="${name}  http://${ip}${portPath}\n"
+done
+
+echo;echo "Services:";echo
+echo -e "${final}"  | column -t
+echo

development/tests/integration/utils/config.js

@@ -12,12 +12,8 @@ const initConfigFiles = async () => {
     `${ROOT_PATH}/cht-instances.yml`
   );
   await copyIfNotExists(
-    `${ROOT_PATH}/development/fake-cht/example-config/postgres-instances.yml`,
-    `${ROOT_PATH}/exporters/postgres/postgres-instances.yml`
-  );
-  await copyIfNotExists(
-    `${ROOT_PATH}/development/fake-cht/example-config/postgres_exporter.yml`,
-    `${ROOT_PATH}/exporters/postgres/postgres_exporter.yml`
+    `${ROOT_PATH}/development/fake-cht/example-config/sql_servers.yml`,
+    `${ROOT_PATH}/exporters/postgres/sql_servers.yml`
   );
   await copyIfNotExists(
     `${ROOT_PATH}/grafana/grafana.example.ini`,

development/tests/integration/utils/docker.js

@@ -7,14 +7,14 @@ const startWatchdog = async () => {
   await exec(`docker compose \
     -f ${ROOT_PATH}/docker-compose.yml \
     -f ${ROOT_PATH}/development/docker-compose.test-data.yml \
-    -f ${ROOT_PATH}/exporters/postgres/docker-compose.postgres-exporter.yml \
+    -f ${ROOT_PATH}/exporters/postgres/compose.yml \
     up -d`);
 };
 
 const stopWatchdog = async () => {
   await exec(`docker compose \
     -f ${ROOT_PATH}/docker-compose.yml \
-    -f ${ROOT_PATH}/exporters/postgres/docker-compose.postgres-exporter.yml \
+    -f ${ROOT_PATH}/exporters/postgres/compose.yml \
     down -v`);
 };
 

docker-compose.yml

@@ -1,5 +1,3 @@
-version: "3.9"
-
 volumes:
   prometheus-data:
     driver: local

exporters/postgres/compose.yml

@@ -0,0 +1,17 @@
+services:
+
+# todo: uncomment and get this working
+  prometheus:
+    volumes:
+      - ./exporters/postgres/scrape.yml:/etc/prometheus/scrape_configs/cht-postgres.yml:ro
+
+  sql_exporter:
+    restart: unless-stopped
+    image: burningalchemist/sql_exporter:latest
+    command: --config.file=/etc/sql_exporter/sql_servers.yml
+    volumes:
+      - "./exporters/postgres/:/etc/sql_exporter/"
+    ports:
+      - "${SQL_EXPORTER_IP:-127.0.0.1}:${SQL_EXPORTER_PORT:-9399}:9399"
+    networks:
+      - cht-watchdog-net

exporters/postgres/couch2pg_collector.yml

@@ -0,0 +1,22 @@
+collector_name: couch2pg
+
+queries:
+  - query_name: couch2pg-query
+    query: |
+      SELECT
+        substring(seq for position('-' in seq) - 1) as sequence,
+        substring(source from position('/' in source) + 1) as db
+      FROM
+        couchdb_progress
+      WHERE
+        source like '%/%' and
+        seq like '%-%'
+
+metrics:
+  - metric_name: couch2pg_progress_sequence
+    type: counter
+    help: 'couch2pg backlog.'
+    key_labels:
+      - db
+    values: [sequence]
+    query_ref: couch2pg-query

exporters/postgres/scrape.yml

@@ -0,0 +1,4 @@
+scrape_configs:
+  - job_name: sql_exporter
+    static_configs:
+      - targets: ['sql_exporter:9399']

exporters/postgres/sql_servers_example.yml

@@ -0,0 +1,29 @@
+# Global settings and defaults.
+global:
+  # Subtracted from Prometheus' scrape_timeout to give us some headroom and prevent Prometheus from
+  # timing out first.
+  scrape_timeout_offset: 500ms
+  # Minimum interval between collector runs: by default (0s) collectors are executed on every scrape.
+  # Medic recommends setting this to 60s so that the endpoint cant be used as a proxy to DOS the upstream Postgres server
+  # see https://github.com/burningalchemist/sql_exporter/issues/447#issue-2144289744 for more info
+  min_interval: 60s
+  # Maximum number of open connections to any one target. Metric queries will run concurrently on
+  # multiple connections.
+  max_connections: 3
+  # Maximum number of idle connections to any one target.
+  max_idle_connections: 3
+  # Maximum amount of time a connection may be reused to any one target. Infinite by default.
+  max_connection_lifetime: 10m
+
+collector_files:
+  - "/etc/sql_exporter/couch2pg_collector.yml"
+
+jobs:
+  - job_name: db_targets
+    collectors: [couch2pg]
+    enable_ping: true
+    static_configs:
+      - targets:
+          # " same as URL in cht-instances.yml      ": 'postgres://USERNAME:PASSWORD@DB_SERVER_IP/DATABASE
+          "172-17-0-1.local-ip.medicmobile.org:10464": 'postgres://cht_couch2pg:[email protected]:5432/cht?sslmode=disable' # //NOSONAR - password is safe to commit
+