Support count event per query_key

[w-HA]

Dears,

Good news with this new elastalert2

Is it possible to have the count of event per query_key ( compound_key too )
There is the num_events but not the exact the count ( 20 in my example ) and sometimes not enough

here is my rules with alerta (work fine )
I tried with top_cout_keys but it didn't work with coumpound_key

# Index to search, wildcard supported
index: production_logs

# rule name, must be unique
name: ProductionLogs | Erreur java Exception Majeur

# Type of alert
type: frequency
timeframe:
 minutes: 5
num_events: 20

# A list of elasticsearch filters used to find events
filter:
 - query:
     query_string:
           query: tags:as AND javaexception:"java.sql.SQLException"

query_key: ["host","application","applog","doctype","javaclass","javaexception","req_alerta"]

# Anti spam
realert:
  minutes: 5

# Alert is sent when a match is found
alert:
- alerta
alerta_api_url: "http://XXXXXXXXXXXXXXXXXXX/api/alert"
alerta_customer: "%(application)s"
alerta_environment: "%(application)s"
alerta_service: ["%(application)s"]
alerta_resource: "%(applog)s %(doctype)s %(javaexception)s"
alerta_event: "%(req_alerta)s major"
alerta_text: "%(applog)s %(doctype)s %(javaclass)s %(req_alerta)s"
alerta_severity: "major"
alerta_tags: ["%(javaexception)s","exploitation","logs"]
alerta_value: "%(host)s %(javaclass)s %(javaexception)s"
alerta_api_key: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXx'
alerta_timeout: 259200
alerta_use_match_timestamp: true

w-HA

[jertel]

Moving to discussion since it doesn't meet the criteria for an issue.

[nsano-rururu]

Although it is use_term_query, when I look at the issue of yelp / elastalert, it seems that it works normally when there is one query_key, and it does not work as expected with multiple query_keys.
Yelp/elastalert#3026
Yelp/elastalert#2155

[nsano-rururu]

top_count_keys does not work with multiple query_key due to a unexpected leading space in filter
Yelp/elastalert#2874

[nsano-rururu]

elastalert 0.1.34 fixes a bug in top_count_keys when using compound query_key, but that process may still be buggy.

[nsano-rururu]

There are multiple query_key bugs in percentage_match as well.
Yelp/elastalert#1281

[nsano-rururu]

I don't feel that the current elastalert 2 can meet your needs.

[w-HA]

dears,

i have just tested new approach with metric_aggregation.

It works very well and i have the count per multiple query_key

I sent an alert to alerta only when value count > 50

metric_agg_key: req_alerta.keyword
metric_agg_type: value_count
query_key:

• host.keyword
• application.keyword
• applog.keyword
• doctype.keyword
• javaclass.keyword
• req_alerta.keyword
  max_threshold: 50

Alert is sent when a match is found

alert:

• alerta
  alerta_api_url: "http://XXXXXXXXXXXXXXXX/api/alert"
  alerta_customer: "%(application.keyword)s"
  alerta_environment: "%(application.keyword)s"
  alerta_service: ["%(application.keyword)s"]
  alerta_resource: "%(host.keyword)s"
  alerta_event: "%(applog.keyword)s %(doctype.keyword)s %(javaclass.keyword)s %(req_alerta.keyword)s"
  alerta_text: "%(applog.keyword)s %(doctype.keyword)s %(javaclass.keyword)s %(req_alerta.keyword)s"
  alerta_severity: "major"
  alerta_tags: ["exploitation","logs"]
  alerta_value: "%(metric_req_alerta.keyword_value_count)s"
  alerta_api_key: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
  alerta_timeout: 259200
  alerta_use_match_timestamp: true

w-HA

[w-HA]

Hi

I test now with
max_threshold: 50

Alert if number of event > 50

min threshold : 20

Alert if number < 20

but i would like to trigger the alert if number of events is > 20 and < 50
So i have in my rule
max_threshold:20
min_thershold:50

It seems it doesn't work (or not possible to do that )
the trigger is fired even if number of events < 20

Thanks for your help

w-HA

[nsano-rururu]

Should I compare max and min, and if max is large, drop it with a gill?

[nsano-rururu]

May be true with 1 <25
https://github.com/jertel/elastalert2/blob/master/elastalert/ruletypes.py#L1135

if 'min_threshold' in self.rules and metric_value < self.rules['min_threshold']:

[nsano-rururu]

When both max_threshold and min_threshold are specified, it is necessary to make the first judgment, but it is judged separately in the current implementation, so is it strange?

>>> (1 < 25) and (1 > 5)
False
>>> (1 > 5)
False
>>> (1 < 25)
True

[nsano-rururu]

Current

        if metric_value is None:
            return False
        if 'max_threshold' in self.rules and metric_value > self.rules['max_threshold']:
            return True
        if 'min_threshold' in self.rules and metric_value < self.rules['min_threshold']:
            return True

Modified image

        if metric_value is None:
            return False
        elif 'max_threshold' in self.rules and 'min_threshold' in self.rules:
            if (metric_value < self.rules['min_threshold']) and (metric_value > self.rules['max_threshold']):
                return True
            else
                return False
        elif 'min_threshold' in self.rules and max_threshold' not in self.rules and metric_value < self.rules['min_threshold']:
            return True
        elif 'min_threshold' not in self.rules and max_threshold' in self.rules and metric_value < self.rules['min_threshold']:
            return True

[jertel]

There are two scenarios that need to be supported to accommodate the original post:

1. value < min_threshold OR value > max_threshold
2. value > max_threshold AND value < min_threshold

The current logic supports scenario 1. To support both 1 and 2 it would need to look something like this:

    def crossed_thresholds(self, metric_value):
        if metric_value is None:
            return False

        if ('max_threshold' in self.rules and 'min_threshold' in self.rules and 
            self.rules['max_threshold'] < self.rules['min_threshold'] and
            metric_value > self.rules['max_threshold'] and
            metric_value < self.rules['min_threshold']):
            # Value is within a specified range that meets the threshold criteria
            return True

        if 'max_threshold' in self.rules and metric_value > self.rules['max_threshold']:
            return True
        if 'min_threshold' in self.rules and metric_value < self.rules['min_threshold']:
            return True
        return False

There are other ways to implement this but those would require additional configuration inputs, such as invert_thresholds: True, or something similar, instead of detecting the inversion from the min_threshold and max_threshold relationship.

[w-HA]

hi

i have just tested nsano rururu solution with some little syntax errors but il works fine !!!

if metric_value is None:
return False
elif 'max_threshold' in self.rules and 'min_threshold' in self.rules:
if (metric_value < self.rules['min_threshold']) and (metric_value > self.rules['max_threshold']):
return True
else :
return False
elif 'min_threshold' in self.rules and 'max_threshold' not in self.rules and metric_value < self.rules['min_threshold']:
return True
elif 'min_threshold' not in self.rules and 'max_threshold' in self.rules and metric_value > self.rules['max_threshold']:
return True

now i have
Threshold violation, value_count:req_alerta.keyword 19 (min: 49 max : 10) OK
and
Threshold violation, value_count:req_alerta.keyword 65 (min: None max : 50) OK

it s very good solution side alerta to manage minor and major alert

Thanks a lot !!!!
w-HA

[nsano-rururu]

In this survey, I noticed that there is a setting called percentiles in the metric_agg_type of Spike Aggregation and Metric Aggregation, and there is a percentile_range related to that setting, although it is not described in the document.
Yelp/elastalert#2713

[nsano-rururu]

Percentage Match is likely to have similar problems.

https://github.com/jertel/elastalert2/blob/master/elastalert/ruletypes.py#L1292

    def percentage_violation(self, match_percentage):
        if 'max_percentage' in self.rules and match_percentage > self.rules['max_percentage']:
            return True
        if 'min_percentage' in self.rules and match_percentage < self.rules['min_percentage']:
            return True
        return False

Metric Aggregation and Percentage Match seem to be features created by the same person in 2016. This issue is a bug from the time of implementation.
Yelp/elastalert#846