Count total hits for today instead of 24 hours or 1 day

[Mridul0902]

I have few elastalert2 rules where I need to count the hits that are true for specific filter for entire day i.e from 00:00 to now. I have tried to use "timestamp" range filter like we use in sentinl.

Checking the mails that I am receiving I think it is not working correctly.

Sample rule file

  Errors-Today-Cumulative-Above-25: |-
    ---
    name: "[Cumulative errors][ElastAlert]"
    type: frequency
    index: "search-*"
    is_enabled: true
    use_count_query: true
    scan_entire_timeframe: true
    num_events: 26 
    realert:
      minutes: 60 
    timeframe:
      days: 1 
    filter:
    - range:
        "@timestamp":
          gte: "now/d"    
    - term:
        message.keyword: "Got exception"
    - term:
        provider.keyword: "abc"
    - bool:
        must_not:
          - term:
              error: "Invalid ID"        
    alert:
    - "email"
    email:
    - "[email protected]"     
    smtp_host: test.mail.server
    smtp_port: 25
    smtp_ssl: false
    from_addr: "[email protected]"
    email_format: html
    alert_text_type: alert_text_jinja
    alert_text: |
      <p>Hello Oncall,</p><div><p>A spike in the number of <b style=\"color:red;\">Cumulative errors from SMS to Indigo</b> have been detected for today from 00:00 to until now.<br />The normal threshold is : <b style=\"color:green;\">25 </b>Cases.<br />It has breached the threshold at : <b style=\"color:red;\">{{num_hits}}</b> Cases currently.</p></div>
  

EXPECTED BEHAVIOUR:

It should check for hit count every 60 minutes. If the count is greater than 25 for today, we should receive a mail.

CURRENT BEHAVIOUR:
It is not exactly scanning for today to trigger mail. It is scanning for some random timeframe.

Example:

If at 02:47 AM, i got 95 hits for this filter, I should not receive any mail before 02:47 AM and should receive hourly mails after this time with with the same value of 95.

Then lets say if I again got 39 hits at 07:50 AM, all the mails received in between 02:47 & 07:50 should have count of 95 hits. All the mails received after 07:50 AM should have count of 134 hits (95+39)

---

For today, I only received 3 mails between 10:55 AM to 12:35 PM with count of 39 hits. Not sure what timeframe it actually scanned.

Is this case possible to resolve with elastalert? Please help.

[jertel]

Similar to the other discussion, enable debug and grab the raw query being sent to Elasticsearch. I'm curious if your specification of a range filter could be interferring with ElastAlert 2's own time range filter.