Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error counts for grouped exception notifications are updated inconsistently, causing duplicate alerts #534

Open
maivy opened this issue Jun 16, 2023 · 0 comments

Comments

@maivy
Copy link

maivy commented Jun 16, 2023

Steps to reproduce

  1. Use Redis as a cache to group notifications and deploy app onto Heroku using Heroku-20 Stack. (This bug does not appear during local development.)
  2. Trigger an exception 8 times within the default grouping period of 5 minutes. See that notifications are triggered appriopriately.
  3. Wait for the default grouping period of 5 minutes to pass.
  4. Do step 2 again. See that notifications are not triggered appropriately. The second alert is duplicated.
Step 2 Step 4
before-exception-notifications after-exception-notifications

Expected behavior

Notifications should always be triggered appropriately and not duplicated.

Actual behavior

Sometimes, notifications are duplicated. When investigating the Redis cache, it looks like ExceptionNotifier sometimes switches from updating and using the error count from the message-based key to the error count from the backtrace-based key even though it is supposed to just hit the message-based key consistently.

For example, in step 4, alerts are triggered in the following way:

  1. Trigger first exception:
    message-based key has error count of 1
    backtrace-based key has error count of 1
    First alert is triggered
  2. Trigger second exception
    message-based key has error count of 2
    backtrace-based key has error count of 1
    Second alert is triggered with "(2 times)" in title
  3. Trigger third exception
    message-based key has error count of 3
    backtrace-based key has error count of 1
    No alert is triggered
  4. Trigger fourth exception
    message-based key has error count of 3
    backtrace-based key has error count of 2
    Third alert is triggered with "(2 times)" in title
  5. Trigger fifth exception
    message-based key has error count of 3
    backtrace-based key has error count of 3
    No alert is triggered
  6. Trigger sixth exception
    message-based key has error count of 4
    backtrace-based key has error count of 3
    Fourth alert is triggered with "(4 times)" in title

System configuration

Rails version: 6.1.7.3

Ruby version: 3.0.5

Other configurations:
Sidekiq: 6.5.9
Redis: >= 4.5.0, < 5 (we get Redis from importing Sidekiq)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant