Support UTF-8 label matchers: Use compat package in Alertmanager server #3567

grobinson-grafana · 2023-10-20T16:03:21Z

This pull request adds use of the compat package in Alertmanager server that will allow users to switch between the new matchers/parse parser and the old pkg/labels parser. The new matchers/parse parser uses a fallback mechanism where if the input cannot be parsed in the new parser it then attempts to use the old parser. If an input is parsed in the old parser but not the new parser then a warning log is emitted.

api/v2/api.go

grobinson-grafana · 2023-11-14T17:22:15Z

silence/silence.go

+}
+
+// InitFromFlags initializes the validation function from the flagger.
+func InitFromFlags(l log.Logger, f featurecontrol.Flagger) {


I found an interesting issue here! If you create a silence with UTF-8 matchers and then restart Alertmanager with the "classic-matchers-parsing" feature flag, you can no longer edit or expire the silence because the validation function returns an error. However, the silence will still expire once it's expiration time has elapsed.

I've attempted to fix that with this commit here.

Great Find! I'll take a look once we have the other changes in.

grobinson-grafana · 2023-11-16T16:03:02Z

@gotjosh let me know when you have some time and we can review this together?

gotjosh

Great job so far, please take a look at my comments.

api/v2/api.go

gotjosh · 2023-11-21T16:11:24Z

cli/alert_query.go

@@ -82,7 +83,7 @@ func (a *alertQueryCmd) queryAlerts(ctx context.Context, _ *kingpin.ParseContext
 		m := a.matcherGroups[0]
 		_, err := compat.Matcher(m)
 		if err != nil {
-			a.matcherGroups[0] = fmt.Sprintf("alertname=%s", m)
+			a.matcherGroups[0] = fmt.Sprintf("alertname=%s", strconv.Quote(m))


This is not related - just something we missed from before, right? We had before silence_add, silence_query and alert_add so I guess it makes sense.

Yes, correct! I noticed it as the changes I made in this PR caused a test to fail.

gotjosh · 2023-11-21T16:35:33Z

test/with_api_v2/acceptance/silence_test.go

+
+	_, err := am.Client().Silence.PostSilences(silenceParams)
+	require.NoError(t, err)
+}


Can we add a test for the case that you found - where you create an utf-8 silence, restart the server with the classic flag and try to operate on it?

I'll take a look. I might need to make some larger changes to acceptance.go to support restarting clusters, as it seems at present stopping a cluster deletes all it's data.

test/with_api_v2/acceptance/api_test.go

gotjosh · 2023-11-21T16:52:53Z

test/with_api_v2/acceptance/silence_test.go

+	at := NewAcceptanceTest(t, &AcceptanceOpts{
+		Tolerance: 150 * time.Millisecond,
+	})


I'm confused - and running this test in isolation seems to agree with me. Where exactly do we make this test accept Unicode characters?

Remember the default mode of Alertmanager (unless you use classic-mode) is UTF-8, with fallback to classic mode. Makes me think I should rename utf8-mode to utf8-strict-mode.

Thanks! I completely forgot about this haha.

types/types.go

gotjosh · 2023-11-21T17:13:45Z

silence/silence.go

+}
+
+// InitFromFlags initializes the validation function from the flagger.
+func InitFromFlags(l log.Logger, f featurecontrol.Flagger) {


Great Find! I'll take a look once we have the other changes in.

silence/silence.go

cmd/alertmanager/main.go

gotjosh

Great job! I only have 2 files left to review - silence and silce_test.

api/api.go

api/v2/api.go

gotjosh · 2023-11-22T15:59:35Z

test/with_api_v2/acceptance/utf8_test.go

+	// Can get same alert from the API.
+	resp, err := am.Client().Alert.GetAlerts(nil)
+	require.NoError(t, err)
+	require.Len(t, resp.Payload, 1)
+	require.Equal(t, labels, resp.Payload[0].Labels)
+
+	// Can filter alerts on UTF-8 labels.
+	getAlertParams := alert.NewGetAlertsParams()
+	getAlertParams = getAlertParams.WithFilter([]string{"00=b", "Σ=c", "\"\\xf0\\x9f\\x99\\x82\"=dΘ"})
+	resp, err = am.Client().Alert.GetAlerts(getAlertParams)
+	require.NoError(t, err)
+	require.Len(t, resp.Payload, 1)
+	require.Equal(t, labels, resp.Payload[0].Labels)
+
+	// Can get same alert in alert group from the API.
+	alertGroupResp, err := am.Client().Alertgroup.GetAlertGroups(nil)
+	require.NoError(t, err)
+	require.Len(t, alertGroupResp.Payload, 1)
+	require.Len(t, alertGroupResp.Payload[0].Alerts, 1)
+	require.Equal(t, labels, alertGroupResp.Payload[0].Alerts[0].Labels)
+
+	// Can filter alertGroups on UTF-8 labels.
+	getAlertGroupsParams := alertgroup.NewGetAlertGroupsParams()
+	getAlertGroupsParams.Filter = []string{"00=b", "Σ=c", "\"\\xf0\\x9f\\x99\\x82\"=dΘ"}
+	alertGroupResp, err = am.Client().Alertgroup.GetAlertGroups(getAlertGroupsParams)
+	require.NoError(t, err)
+	require.Len(t, alertGroupResp.Payload, 1)
+	require.Len(t, alertGroupResp.Payload[0].Alerts, 1)
+	require.Equal(t, labels, alertGroupResp.Payload[0].Alerts[0].Labels)


Great job! ❤️

gotjosh

LGTM but please take a look at my comment wrt the silences validation.

I think we're missing three things at this point:

Make the CI pass
The test to catch the removal of validation in expire
The avoidance of copy paste code.

silence/silence.go

gotjosh · 2023-11-23T09:55:27Z

silence/silence.go

+func validateUTF8Matcher(m *pb.Matcher) error {
+	if !utf8.ValidString(m.Name) {
+		return fmt.Errorf("invalid label name %q", m.Name)
+	}
+	switch m.Type {
+	case pb.Matcher_EQUAL, pb.Matcher_NOT_EQUAL:
+		if !utf8.ValidString(m.Pattern) {
+			return fmt.Errorf("invalid label value %q", m.Pattern)
+		}
+	case pb.Matcher_REGEXP, pb.Matcher_NOT_REGEXP:
+		if !utf8.ValidString(m.Pattern) {
+			return fmt.Errorf("invalid regular expression %q", m.Pattern)
+		}
+		if _, err := regexp.Compile(m.Pattern); err != nil {
+			return fmt.Errorf("invalid regular expression %q: %s", m.Pattern, err)
+		}
+	default:
+		return fmt.Errorf("unknown matcher type %q", m.Type)
+	}
+	return nil
+}
+


I have my reservations when it comes to this code - to me, this feels wrong. We're accepting matches as user input from the silences API in a structured format:

"matchers": [ { "name": "string", "value": "string", "isRegex": true, "isEqual": true } ],

In my opinion, this validation should happen at an API level and ideally make the input matchers go through the parser, by taking each parser and constructing a string that would you give you the struct directly.

My main worry is that right now validation for entities is spreadout in multiple layers which makes it difficult to track who validates what.

Not something that we need to solve right now, but it's worth thinking about.

gotjosh

LGTM - please take a look at my comments but overall I have no further blocking questions.

Great work and thank you very much for your contribution.

silence/silence.go

gotjosh · 2023-11-24T09:09:31Z

I've merged #2469, can you please rebase so that I can also merge this one?

Signed-off-by: George Robinson <[email protected]>

This commit adds isDelete to setSilence to skip validation when expiring silences. This prevents an issue where a silence that is no longer valid (i.e. the rules have changed in a newer version of Alertmanager) cannot be expired via the API or UI. Signed-off-by: George Robinson <[email protected]>

Signed-off-by: George Robinson <[email protected]>

grobinson-grafana · 2023-11-24T09:49:21Z

I've merged #2469, can you please rebase so that I can also merge this one?

Done! 🙂

gotjosh

LGTM

* [CHANGE] Deprecate and remove api/v1/ #2970 * [CHANGE] Remove unused feature flags #3676 * [CHANGE] Newlines in smtp password file are now ignored #3681 * [CHANGE] Change compat metrics to counters #3686 * [CHANGE] Do not register compat metrics in amtool #3713 * [CHANGE] Remove metrics from compat package #3714 * [CHANGE] Mark muted alerts #3793 * [FEATURE] Add metric for inhibit rules #3681 * [FEATURE] Support UTF-8 label matchers #3453, #3507, #3523, #3483, #3567, #3568, #3569, #3571, #3595, #3604, #3619, #3658, #3659, #3662, #3668, 3572 * [FEATURE] Add counter to track alerts dropped outside of time_intervals #3565 * [FEATURE] Add date and tz functions to templates #3812 * [FEATURE] Add limits for silences #3852 * [FEATURE] Add time helpers for templates #3863 * [FEATURE] Add auto GOMAXPROCS #3837 * [FEATURE] Add auto GOMEMLIMIT #3895 * [FEATURE] Add Jira receiver integration #3590 * [ENHANCEMENT] Add the receiver name to notification metrics #3045 * [ENHANCEMENT] Add the route ID to uuid #3372 * [ENHANCEMENT] Add duration to the notify success message #3559 * [ENHANCEMENT] Implement webhook_url_file for discord and msteams #3555 * [ENHANCEMENT] Add debug logs for muted alerts #3558 * [ENHANCEMENT] API: Allow the Silences API to use their own 400 response #3610 * [ENHANCEMENT] Add summary to msteams notification #3616 * [ENHANCEMENT] Add context reasons to notifications failed counter #3631 * [ENHANCEMENT] Add optional native histogram support to latency metrics #3737 * [ENHANCEMENT] Enable setting ThreadId for Telegram notifications #3638 * [ENHANCEMENT] Allow webex roomID from template #3801 * [BUGFIX] Add missing integrations to notify metrics #3480 * [BUGFIX] Add missing ttl in pushhover #3474 * [BUGFIX] Fix scheme required for webhook url in amtool #3409 * [BUGFIX] Remove duplicate integration from metrics #3516 * [BUGFIX] Reflect Discord's max length message limits #3597 * [BUGFIX] Fix nil error in warn logs about incompatible matchers #3683 * [BUGFIX] Fix a small number of inconsistencies in compat package logging #3718 * [BUGFIX] Fix log line in featurecontrol #3719 * [BUGFIX] Fix panic in acceptance tests #3592 * [BUGFIX] Fix flaky test TestClusterJoinAndReconnect/TestTLSConnection #3722 * [BUGFIX] Fix crash on errors when url_file is used #3800 * [BUGFIX] Fix race condition in dispatch.go #3826 * [BUGFIX] Fix race conditions in the memory alerts store #3648 * [BUGFIX] Hide config.SecretURL when the URL is incorrect. #3887 * [BUGFIX] Fix invalid silence causes incomplete updates #3898 * [BUGFIX] Fix leaking of Silences matcherCache entries #3930 * [BUGFIX] Close SMTP submission correctly to handle errors #4006 Signed-off-by: SuperQ <[email protected]>

grobinson-grafana mentioned this pull request Oct 20, 2023

Epic: Support UTF-8 characters in Alertmanager label matchers #3486

Closed

4 tasks

grobinson-grafana changed the title ~~Use compat package in Alertmanager server~~ Support UTF-8 label matchers: Use compat package in Alertmanager server Oct 20, 2023

grobinson-grafana marked this pull request as draft October 20, 2023 16:04

grobinson-grafana force-pushed the grobinson/use-adapter-alertmanager-server branch from cd3904e to 67c0042 Compare November 1, 2023 10:28

grobinson-grafana commented Nov 1, 2023

View reviewed changes

api/v2/api.go Outdated Show resolved Hide resolved

grobinson-grafana marked this pull request as ready for review November 2, 2023 17:13

grobinson-grafana marked this pull request as draft November 3, 2023 20:02

grobinson-grafana mentioned this pull request Nov 6, 2023

Update Alertmanager vendor grafana/mimir#6560

Merged

3 tasks

grobinson-grafana force-pushed the grobinson/use-adapter-alertmanager-server branch 3 times, most recently from df1e680 to 3f30b33 Compare November 14, 2023 17:16

grobinson-grafana commented Nov 14, 2023

View reviewed changes

grobinson-grafana force-pushed the grobinson/use-adapter-alertmanager-server branch from 9ec6b89 to 7261a7f Compare November 15, 2023 13:14

grobinson-grafana marked this pull request as ready for review November 15, 2023 13:17

grobinson-grafana force-pushed the grobinson/use-adapter-alertmanager-server branch from 64ac0a0 to bfb4b78 Compare November 21, 2023 11:27

gotjosh reviewed Nov 21, 2023

View reviewed changes

gotjosh mentioned this pull request Nov 21, 2023

Deprecate and remove api/v1/ #2970

Merged

3 tasks

grobinson-grafana force-pushed the grobinson/use-adapter-alertmanager-server branch 2 times, most recently from 2c60bec to ee2e8da Compare November 22, 2023 14:42

grobinson-grafana requested a review from gotjosh November 22, 2023 15:12

gotjosh reviewed Nov 22, 2023

View reviewed changes

gotjosh reviewed Nov 23, 2023

View reviewed changes

grobinson-grafana force-pushed the grobinson/use-adapter-alertmanager-server branch from 423ddc7 to 422e80f Compare November 23, 2023 10:34

grobinson-grafana requested a review from gotjosh November 23, 2023 16:08

grobinson-grafana force-pushed the grobinson/use-adapter-alertmanager-server branch from f2529d0 to a33c000 Compare November 23, 2023 16:35

gotjosh approved these changes Nov 23, 2023

View reviewed changes

silence/silence.go Show resolved Hide resolved

grobinson-grafana added 2 commits November 24, 2023 09:21

Use compat package in Alertmanager server

3f96a45

Signed-off-by: George Robinson <[email protected]>

Fix tests

48368ab

Signed-off-by: George Robinson <[email protected]>

grobinson-grafana added 22 commits November 24, 2023 09:21

Add acceptance tests for creating silences

8022958

Signed-off-by: George Robinson <[email protected]>

Fix lint

976b713

Signed-off-by: George Robinson <[email protected]>

Fix use of renamed feature flags

fa2bca6

Signed-off-by: George Robinson <[email protected]>

Inject feature flags into APIs and Silencer

a38aeb4

Signed-off-by: George Robinson <[email protected]>

Fix tests

fc1a2db

Signed-off-by: George Robinson <[email protected]>

Use parseFilters

6916407

Signed-off-by: George Robinson <[email protected]>

Rename UTF-8 mode to UTF-8 strict mode

7c5b767

Signed-off-by: George Robinson <[email protected]>

Move UTF-8 tests to separate file

38a5bcd

Signed-off-by: George Robinson <[email protected]>

Add tests for filtering alerts and silences

cda2a30

Signed-off-by: George Robinson <[email protected]>

Fix license

fb98219

Signed-off-by: George Robinson <[email protected]>

Fix lint

5ab3d73

Signed-off-by: George Robinson <[email protected]>

Add test for filtering alert groups

9fa29ad

Signed-off-by: George Robinson <[email protected]>

Fix lint again

9da9da6

Signed-off-by: George Robinson <[email protected]>

Fix tests

e3c7ba8

Signed-off-by: George Robinson <[email protected]>

Fix comments

51187ea

Signed-off-by: George Robinson <[email protected]>

Feedback

8c86375

Signed-off-by: George Robinson <[email protected]>

Fix

d5aebdf

Signed-off-by: George Robinson <[email protected]>

Rename to skipValidate

969ccc3

Signed-off-by: George Robinson <[email protected]>

Feedback

7fa308e

Signed-off-by: George Robinson <[email protected]>

Fix rebase mistake

c3655b4

Signed-off-by: George Robinson <[email protected]>

Fix deletion of v1

318b822

Signed-off-by: George Robinson <[email protected]>

grobinson-grafana force-pushed the grobinson/use-adapter-alertmanager-server branch from 62b5ade to 318b822 Compare November 24, 2023 09:29

gotjosh approved these changes Nov 24, 2023

View reviewed changes

gotjosh merged commit 70bd5da into prometheus:main Nov 24, 2023
11 checks passed

alanprot mentioned this pull request Apr 2, 2024

Chore: Update Prometheus cortexproject/cortex#5841

Merged

grobinson-grafana deleted the grobinson/use-adapter-alertmanager-server branch April 16, 2024 14:44

SuperQ mentioned this pull request Oct 16, 2024

Release v0.28.0-rc.0 #4072

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support UTF-8 label matchers: Use compat package in Alertmanager server #3567

Support UTF-8 label matchers: Use compat package in Alertmanager server #3567

grobinson-grafana commented Oct 20, 2023 •

edited

Loading

grobinson-grafana Nov 14, 2023

grobinson-grafana Nov 15, 2023

gotjosh Nov 21, 2023

grobinson-grafana commented Nov 16, 2023

gotjosh left a comment

gotjosh Nov 21, 2023

grobinson-grafana Nov 21, 2023

gotjosh Nov 21, 2023

grobinson-grafana Nov 22, 2023

gotjosh Nov 21, 2023

grobinson-grafana Nov 22, 2023

gotjosh Nov 22, 2023

gotjosh Nov 21, 2023

gotjosh left a comment

gotjosh Nov 22, 2023

gotjosh left a comment

gotjosh Nov 23, 2023

gotjosh left a comment

gotjosh commented Nov 24, 2023

grobinson-grafana commented Nov 24, 2023

gotjosh left a comment

Support UTF-8 label matchers: Use compat package in Alertmanager server #3567

Support UTF-8 label matchers: Use compat package in Alertmanager server #3567

Conversation

grobinson-grafana commented Oct 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grobinson-grafana commented Nov 16, 2023

gotjosh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gotjosh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gotjosh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gotjosh left a comment

Choose a reason for hiding this comment

gotjosh commented Nov 24, 2023

grobinson-grafana commented Nov 24, 2023

gotjosh left a comment

Choose a reason for hiding this comment

grobinson-grafana commented Oct 20, 2023 •

edited

Loading