Frequency rule with realert using number as field type in query key returns wrong number of match

[oTwoWin]

Hello !

First of all, thank you for continuing the project for all of those years. We use it daily and it's very useful that you are taking care of it.

I recently discovered a wrong behavior with the version 2.18.0 of Elastalert on Docker. We used it for Security purpose so I won't be able to share much information but here is what I can share.

We are basically monitoring suspicious network actions from some source IP and each action match a signature that has an unique ID.

Our rule is quiet basic:

alert:
- elastalert_modules.SplunkHECAlerter.SplunkHEC
description: ''
filter:
- query:
    query_string:
      query: (alert_category:("Web\ Application\ Attack"))
index: someindex*
name: somerulename
num_events: 2
priority: 3
query_key:
  - alert_id
  - src_ip_addr
realert:
  minutes: 10
timeframe:
  minutes: 10
type: frequency

alert_id is a numeric type and src_ip_addr is a keyword type.

We tested some scenarios to validate our detection and I was surprised to find that there is a mismatch of the number of matches that the rule returns:

elastalert_status - {'rule_name': 'somerulename', 'endtime': datetime.datetime(2024, 7, 11, 8, 21, 54, 120892, tzinfo=tzutc()), 'starttime': datetime.datetime(2024, 7, 10, 8, 21, 54, 120892, tzinfo=tzutc()), 'matches': 1, 'hits': 17, '@timestamp': datetime.datetime(2024, 7, 11, 8, 21, 54, 805250, tzinfo=tzutc()), 'time_taken': 0.33525657653808594}

In this case, only one match was returned even if several "src_ip_addr" were different.

However, if I change alert_id to an other keyword type field instead that is present and have the same value over all hits, it returns 8 matches (which is the correct behavior as it hasn't happened on the 10 minutes realert timeframe):

elastalert_status - {'rule_name': 'somerulename', 'endtime': datetime.datetime(2024, 7, 11, 8, 2, 57, 835178, tzinfo=tzutc()), 'starttime': datetime.datetime(2024, 7, 10, 8, 2, 57, 835178, tzinfo=tzutc()), 'matches': 8, 'hits': 17, '@timestamp': datetime.datetime(2024, 7, 11, 8, 2, 58, 492246, tzinfo=tzutc()), 'time_taken': 0.327101469039917}

I confirmed the behavior by replacing the alert_id field by another numeric type field and the result is only 1 match again.

In the documentation, there is no mention that the query_key fields should be keyword typed in order to work with realert.
Is it a bug or an expected behavior ?

Let me know if you need more information.

Thanks in advance for your help.

Best regards,

Daniel

[oTwoWin]

Hello !

After testing a few other scenarios, I finally found out what it was rather an issue of comprehension.

I didn't fully understand the query_key logic in this scenario.

In this case, I wanted to detect a source making multiple attempts and then silence based on both fields. But not multiple attempts by source IP AND signature.

It wasn't behaving as it was logically thought in the first place so that's why the confusion about it. But it works with numeric field type.

If someone steps into this, you may want to check thoroughly the documentation !

Best regards,

Daniel