Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove immediate flush on reload/restart #3419

Closed
wants to merge 6 commits into from
Closed

Remove immediate flush on reload/restart #3419

wants to merge 6 commits into from

Commits on Oct 12, 2023

  1. Remove immediate flush on reload/restart 

    This commit changes Alertmanager so it no longer flushes aggregation
groups on configuration reload or restart of Alertmanager as this
behavior causes a number of issues:

1. Alertmanager will send notifications for inhibited alerts if the
   inhibited alert is sent to Alertmanager before the inhibiting alert
   following a restart

2. Reloading Alertmanager via /-/reload can cause incomplete flushes
   of aggregation groups (prometheus#3407)

A potential issue with this change is that following a reload or restart
of Alertmanager, alerts that were waiting for group_wait will have to
wait from the beginning of group_wait again. If group_wait is large
then notifications could take longer to send then expected.

Signed-off-by: George Robinson <[email protected]>
    @grobinson-grafana
    grobinson-grafana committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    20d1839 View commit details
    Browse the repository at this point in the history

  2. Fix lint 

    Signed-off-by: George Robinson <[email protected]>
    @grobinson-grafana
    grobinson-grafana committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    22e2390 View commit details
    Browse the repository at this point in the history

  3. Fix lint again 

    Signed-off-by: George Robinson <[email protected]>
    @grobinson-grafana
    grobinson-grafana committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    a1b4eee View commit details
    Browse the repository at this point in the history

  4. Fix tests 

    Signed-off-by: George Robinson <[email protected]>
    @grobinson-grafana
    grobinson-grafana committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    5d2478b View commit details
    Browse the repository at this point in the history

  5. Remove mutex as timer cannot be reset from multiple goroutines 

    Signed-off-by: George Robinson <[email protected]>
    @grobinson-grafana
    grobinson-grafana committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    9849fe5 View commit details
    Browse the repository at this point in the history

  6. Update comments in tests 

    Signed-off-by: George Robinson <[email protected]>
    @grobinson-grafana
    grobinson-grafana committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    0266390 View commit details
    Browse the repository at this point in the history