Skip to content

v1.0.0 - Production! Acoustic batched processing, metrics
Compare
Choose a tag to compare
@jwhitlock jwhitlock released this 09 Jun 20:39
· 673 commits to main since this release
v1.0.0
This tag was signed with the committer’s verified signature.
jwhitlock John Whitlock
GPG key ID: 082C735D154FB750
Learn about vigilant mode.
d905ddc
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

This release updates how the Acoustic synchronization job processes large backlogs, and adds metrics.

Tag v0.8.3 has been running in production without Salesforce for a few weeks, so we're bumping the version number to 1.0.0. Scripts used during the final import have been updated in this release.

Acoustic Synchronization Job

  • Pending updates are now processed in batches, rather than all pending updates. This avoids long processing runtime without feedback. The default is 20 updates per batch.
  • Prometheus metrics are pushed to the pushgateway, if configured.

Deployments

  • Two new environment variables to tune the Acoustic Synchronization Job:
    • ACOUSTIC_BATCH_LIMIT - set the number of updates per batch
    • PROMETHEUS_PUSHGATEWAY_URL - set the URL of the Prometheus push gateway
  • New metrics are available, if configured:
    • ctms_background_acoustic_request_total - Total count of acoustic requests by method and status
    • ctms_background_acoustic_requests_duration - Histogram of requests processing time by method (in seconds)
    • ctms_background_acoustic_sync_total - Total count of contacts synced to acoustic
    • ctms_background_acoustic_sync_retries - Gauge of pending records with >0 retries to acoustic
    • ctms_background_acoustic_sync_backlog - Gauge of the number of contacts in the sync backlog. Not counting over-retried records.

Other Changes

  • The import script scripts/importers/setup.sql and scripts/importers/finish.sql includes updates for the final import, such as index dropping and creation, case-insensitive duplicate email dropping, and newsletter source column cleanup.
Assets 2
Loading