This tool serves as a proxy for applications aiming to dispatch messages to Slack nicely. While many applications adopt a "fire and forget" approach for notifications, this method remains effective only up to a certain scale. When operating at larger scales, there's a risk of reaching Slack's rate limits. It's advisable to maintain a rate of one message per second (as per Slack documentation). Slack does provide a burst capacity. However, the specifics of this burst capacity remain undetermined.
For those managing multiple applications, a practical approach is to create a separate Slack app or webhook for each application. That method is preferred. However, if you are operating a singular system, like Alertmanager or Grafana, and wish to avoid the complexities of providing individual teams with:
- Their distinct webhooks
- Unique notification channel keys
- Possible additional routing configurations
Then, this tool is your ideal solution.
Many applications posting messages to Slack either overlook Slack's rate limits or neglect to process Slack's response. By incorporating a rate limiter and a queue, we can not only ensure compliance with Slack's guidelines but also enhance the likelihood of our messages being successfully delivered.
By being a 1:1 forwarding proxy, you simply POST to this application instead, and it will get forwarded to Slack.
Furthermore, by adding observability, we can have a much clearer picture of:
- Requests per second
- To which channel?
- Are there failures and at what rate?
These type of insights are currently not possible to know via Slack, and only via different methods if your applications are instrumented that way (which they often aren't)
We don't try to 'mock' the Slack API. We make a fair assumption that the message you post to the proxy is already tested and meets the API spec. In other words, if you got a new (custom) application where you are testing the API, I would highly recommend you do that to Slack directly. Once you have 'battletested' your implementation, you then simply convert the URL to this proxy and gain out of the box retries and rate limit behaviour with included metrics.
The slackproxy service provides several metrics to monitor and gauge the performance of the proxy, especially regarding how it handles requests. The metrics are exposed using the Prometheus client library.
-
Requests Received Total
- Metric:
slackproxy_requests_recieved_total - Description: The total number of requests received by the proxy.
- Labels:
channel
- Metric:
-
Requests Failed Total
- Metric:
slackproxy_requests_failed_total - Description: The total number of requests that failed.
- Labels:
channel
- Metric:
-
Requests Retried Total
- Metric:
slackproxy_requests_retried_total - Description: The total number of requests retried by the proxy.
- Labels:
channel
- Metric:
-
Requests Succeeded Total
- Metric:
slackproxy_requests_succeeded_total - Description: The total number of requests that succeeded.
- Labels:
channel
- Metric:
-
Requests Not Processed
- Metric:
slackproxy_requests_not_processed_total - Description: The total number of requests not processed by the proxy.
- Labels:
channel
- Metric:
-
Queue Size
- Metric:
slackproxy_queue_size - Description: The current size of the proxy's queue.
- Metric:
Monitor the queue size with the slackproxy_queue_size metric. This isn't a persistent queue. If the application crashes abruptly, the queue is lost. However, during a clean application shutdown, the queue processes, given adequate time. If, for instance, there's a prolonged Slack outage or if you face an outage, the queue might be lost. While the queue size is configurable, remember that the processing rate is a maximum of 1 message per second. If the queue consistently reaches its limit, consider horizontal scaling.
When the error channel_not_found appears, rather than retrying, ANY request to post to the said channel is placed on a 'DoNotProcess' list for 15 minutes. This minimizes unnecessary Slack calls. Monitor this behavior with the slackproxy_requests_not_processed_total metric.
Permanent errors are logged in detail, including the complete POST request. Concurrently, the slackproxy_requests_failed_total metric is incremented.
- Currently, we do not use the original header bearer token. It is required you setup this application with a slack webhook. I personally think that's fine/good. Open for suggestions..
- Implement our own bearer token auth method. This way the application can run protected (at the moment anyone can POST against this application and it will send the message to Slack) - open for other suggestions..
- Build + Docker image
- Code check
- How to run multiple replicas with each their own API key?
- Add some basic sanity check if the basics are part of the request (channel, some body, etc)
This manifest is required when making an application that can:
- Use a single token
- Post to any (public) channel
- Change it's name
display_information:
name: your-name
features:
bot_user:
display_name: your-default-display-name
always_online: false
oauth_config:
scopes:
user:
- chat:write
bot:
- chat:write
- chat:write.customize
- chat:write.public
settings:
org_deploy_enabled: false
socket_mode_enabled: false
token_rotation_enabled: false
--token: Bearer token for the Slack API.- Example:
--token=YOUR_BEARER_TOKEN
- Example:
I would recommend not altering these values until you have a good understanding how it performs for your workload
-
--maxRetries: Maximum number of retries for posting a message.- Default:
3 - Example:
--maxRetries=5
- Default:
-
--initialBackoffMs: Initial backoff in milliseconds for retries.- Default:
1000 - Example:
--initialBackoffMs=2000
- Default:
-
--slackURL: Slack Post Message API URL.- Default:
https://slack.com/api/chat.postMessage - Example:
--slackURL=https://api.slack.com/your-endpoint
- Default:
-
--queueSize: Maximum number of messages in the queue.- Default:
100 - Example:
--queueSize=200
- Default:
-
--burst: Maximum number of burst messages to allow.- Default:
3 - Example:
--burst=2
- Default:
-
--metricsPort: Port used for the/metricsendpoint- Default:
:9090 - Example:
--metricsPort :9090
- Default:
-
--applicationPort: Port used for the application endpoint (where you send your requests to)- Default:
:8080 - Example:
--applicationPort :8080
- Default:
-
--channelOverride: Override on sending all messages to this defined channel. This is useful for debugging or if you want to force to use a single channel- Default: ``
- Example:
--channelOverride #debug-notifications
-
--slackRequestRate: Request rate for slack requests in milliseconds.- Default:
1000 - Example:
--slackRequestRate=500
- Default: