⚠️ End of Support (EoS) Notice

The Heroku SignalFx Collector and SignalFx Smart Agent have reached End of Support.

The Splunk Distribution of OpenTelemetry Collector for Heroku is the successor.

Heroku SignalFx Collector

A Golang Heroku app that collects metrics from apps and reports them to SignalFx. The Heroku SignalFx Collector collects the following metrics -

• Application metrics
• Heroku Dyno metrics
• Custom metrics

Note: Application and Heroku Dyno metrics will be prefixed by heroku. in SignalFx.

Deploy on Heroku

Deploy the SignalFx Collector on Heroku using:

Setup

At a high level, the collector runs as a HTTPS Log Drain app on Heroku. See here for more information.

Deploying Heroku SignalFx Collector

Clone this repo

git clone [email protected]:signalfx/heroku-signalfx-collector.git

Create a Heroku app

cd heroku-signalfx-collector/
heroku create

# Add Golang buildpack
heroku buildpacks:add heroku/go

Configure the collector

Use the following environment variables to configure the collector

Environment Variable	Description	Example
SFX_TOKEN	SignalFx access token of the org to which data needs to be sent (required)	somevalidtoken
SFX_INGEST_URL	Ingest URL to which data needs to be sent (required if SFX_REALM is not set)	https://ingest.us0.signalfx.com
SFX_REALM	SignalFx realm to which data is to be sent (required if SFX_INGEST_URL is not set)	us0, us1, us2, eu0, ap0
SFX_METRICS_TO_EXCLUDE	Comma separated metric names that the collector should not emit	metric_name1,metric_name2,metric_name3
SFX_DIMENSION_PAIRS_TO_EXCLUDE	Comma separated dimension key value pairs that the collector should not emit	key1=val1,key2=val2
SFX_REPORTING_INTERVAL	Reporting interval of the collector in seconds. Default value is 10 seconds	20
SFX_INTERNAL_METRICS	Whether or not to report internal metrics (set to true by default)	false

Configure Heroku App to send logs to SignalFx Collector

Enable Heroku log run-time metrics

heroku labs:enable log-runtime-metrics

See here for more information.

Configure Heroku App to expose Dyno metadata

heroku labs:enable runtime-dyno-metadata

This metadata is required by internal metrics to report accurate dimensions for fields like app_name and dyno_id.

See here for more information.

Follow the steps below so that an app sends logs to the collector

heroku drains:add "https://<SFX_COLLECTOR_APP_NAME>.herokuapp.com?app_name=<APP_NAME>"

Note: The above command expects SFX_COLLECTOR_APP_NAME to be the name of SignalFx Collector app and APP_NAME to be the Heroku app to monitored. Use heroku apps:info | grep '===' | cut -d' ' -f2 to set the app name. Passing in the app_name parameter is required. app_name is added as dimension to datapoints being by the collector.

To add custom dimensions to all datapoints being emitted by the collector, pass in query parameters while configuring the log drain.

heroku drains:add "https://<SFX_COLLECTOR_APP_NAME>.herokuapp.com?app_name=<APP_NAME>&dim1=key1&dim2=key2"

For example, configuring a log drain in the manner above will add dim1=key1 and dim2=key2 dimensions to all datapoints being emitted by the collector.

Deploy the collector to Heroku

git push heroku master
heroku ps:scale web=1

Sending in custom metrics

This app looks for logs of the following format. This is the standard format in which Heroku exposes default metrics. Custom metrics defined in the following format can be collected and emitted by the app.

\<(?P<pri>\d+)\>(?P<version>1) (?P<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{6})?\+\d{2}:\d{2}) (?P<hostname>[a-z0-9\-\_\.]+) (?P<appname>[a-z0-9\.-]+) (?P<procid>[a-z0-9\-\_\.]+) (?P<msgid>\-) (?P<message>.*)$

IMPORTANT: All the fields in the above expect for the the last (message) field are generated by the Heroku platform. message field will contain application logs. This is where metrics and dimensions should be defined.

An example custom metric log that the collector can detect:

gauge#quota_used=20 cumulative#response_bytes=100 sfxdimension#service=backend sfxdimension#client=sfx_app

The above log message from an application would be interpreted as 2 datapoints, from quota_used (a gauge with value 20) and response_bytes (a cumulative counter with value 100) metrics, and both datapoints with service=backend and client_sfx_app dimensions.

Use the following keywords to (identify key-value pairs as metrics or dimensions) construct custom metrics

• gauge# - gauge type
• counter# - counter type
• cumulative# - cumulative counter type
• sfxdimension# - dimension

Internal Metrics

The collector reports internal metrics by default. Below is a list of internal metrics.

Metric Name	Description
sfx_heroku.total_drain_requests	Number of drain requests received by the collector
sfx_heroku.tracked_metrics	Number of metrics collected per metric type. Metric types are determined by the dimension called type (i.e., cumulative_counter, counter, gauge).

Note: These metrics are collected by default and can be turned off by setting SFX_INTERNAL_METRICS to false.