Unable to access @timestamp in http_post2_payload

[akusei]

I'm not sure if it's me doing something wrong but I'm having a real hard time getting the value @timestamp in the http_post2_payload. We're migrating from elasticsearch's SIEM alerts to elastalert2 (which is the much better option in my opinion) and have a need to maintain some legacy fields in the alert sent to our endpoint webhook using the HTTP POST2 plugin. Reading the documentation about alerting in general kind of leads me to believe there's a jinja2 template field called _data which I can use to access dot notation fields or a field with invalid characters like @timestamp. I was mistaken and _data is not available to the post2 plugin.

I may be doing this the wrong way so please let me know if I am. How am I meant to access fields like @timestamp from the post2 alert payload? I can't do {{ @timestamp }} as it is invalid, and {{ _data['@timestamp'] }} is undefined. I also tried setting jinja_root_name but that didn't work either.

I did come up with a workaround, shown below, but it's really smelly. So I'm hoping someone has a better way or maybe this could become an enhancement/bug?

alert:
  - post2:
      http_post2_url: https://localhost/webhook
      http_post2_headers:
        authorization: $api_key$
      http_post2_all_values: true
      # won't work because http_post2_raw_fields doesn't expect a dict as a value
      # http_post2_raw_fields:
      #   signal:
      #     original_time: "@timestamp"
      http_post2_payload:
        signal:
          # using self._TemplateReference__context as a workaround to access dot notation or elastic @ variables
          original_time: "{{ self._TemplateReference__context['@timestamp'] }}"
          rule:
            name: $name$
            description: $description$
            severity: low
        kibana:
          alert:
            ancestors:
              - id: '{{ _id }}'
                index: '{{ _index }}'

EDIT: I forgot to mention that I'm using the latest docker image jertel/elastalert2

[jertel]

Try this:

http_post2_payload:
  body: "Time: {{sometime}}"
http_post2_raw_fields:
  sometime: "@timestamp"

[akusei]

This results in the following json sent to the webhook

{"body": "Time: ", "timestamp_value": "2023-01-31T07:30:09Z"}

Because of limitation in the webhook, we must have the listed format in my example above. It doesn't appear possible with your suggestion. I modified it to try and this is what I came up with:

http_post2_raw_fields:
  timestamp_value: "@timestamp"
http_post2_payload:
  signal: '{"original_time":"{{ timestamp_value }}","rule":{"name":$name$,"description":$description$,"severity":"low"}}'
  kibana: '{"alert":{"ancestors":[{"id":"{{_id}}",{"index":"{{_index}}"'

But this results in the following, which won't work (note that it also doesn't resolve the $field$ tokens):

{"signal": "{\"original_time\":\"\",\"rule\":{\"name\":$name$,\"description\":$description$,\"severity\":\"low\"}}", "kibana": "{\"alert\":{\"ancestors\":[{\"id\":\"KI28BoYBB46vIqlModJd\",{\"index\":\"functionbeat-7.4.0-000019\"", "timestamp_value": "2023-01-31T07:30:09Z"}

I also tried your suggestion with my existing example and it doesn't resolve {{ timestamp_value }}:

http_post2_raw_fields:
  timestamp_value: "@timestamp"
http_post2_payload:
  signal:
    original_time: "{{ timestamp_value }}"
    rule:
      name: $name$
      description: $description$
      severity: low
  kibana:
    alert:
      ancestors:
        - id: '{{ _id }}'
          index: '{{ _index }}'

Another issue I have with this method is the extra field added to the root of the document sent with the alert. Not a big deal but it's extra bytes going over the network that aren't needed and the elastic docs are large to begin with.

To be specific, the output must look like this (although it can be minified):

{
    "signal": {
        "original_time": "2023-01-31T07:30:09Z",
        "rule": {
            "name": "rule name",
            "description": "rule desc",
            "severity": "low"
        }
    },
    "kibana": {
        "alert": {
            "ancestors": [
                {
                    "id": "KI28BoYBB46vIqlModJd",
                    "index": "index-name"
                }
            ]
        }
    }
}

EDIT: I didn't notice it before but it looks like the example you provided doesn't work either. Using the exact example you provided, I get the following. Which has the extra sometime field containing the timestamp but did not resolve it in the payload with {{sometime}}. I pretty printed for clarity:

{
    "body": "Time: ",
    "sometime": "2023-01-31T07:30:09Z"
}

This is the alert config I used to test:

alert:
  - post2:
      http_post2_url: http://localhost/webhook
      http_post2_headers:
        authorization: $api_key$
      http_post2_all_values: false
      http_post2_raw_fields:
        sometime: "@timestamp"
      http_post2_payload:
        body: "Time: {{sometime}}"

[jertel]

You're dealing with a Jinja limitation. Jinja does not allow non-identifier characters in the variable name. So the "@" is not valid. I understand now that the http_post2_raw_fields idea won't work for you. An option you have is to write a small Enhancement that takes the "@timestamp" key and inserts it into a new dict inside of the match dict. Currently the match dict looks like this to the Jinja template renderer library:

{
  "@timestamp": "2022-10-23T01:02:03Z",
  "_id": "xyz",
  "_index": "myindex",
  ...
}

But if your enhancement creates a temporary dict within that dict, like the following:

{
  "@timestamp": "2022-10-23T01:02:03Z",
  "_id": "xyz",
  "_index": "myindex",
  ...
  "custom_dict": {
    "@timestamp": "2022-10-23T01:02:03Z"
  }
}

you can now indirectly access it, including the non-identifier symbol "@", in your Jinja post template as follows:

http_post2_payload:
  signal:
    original_time: "{{ custom_dict['@timestamp'] }}"
 ...

And you wouldn't need the http_post2_raw_fields setting.

[akusei]

What you're describing here is exactly the same behavior for the base alerter. In the base alerter, all fields are accessible by a temporary field named _data. If that conflicts with existing field names you can configure jinja_root_name to override that default. Would adding **{self.rule['jinja_root_name']: match} to the http post2 template render be acceptable? Not that this is the solution or how it should be implemented but I've tested this and it works:

httppost2.py line 43-50

def alert(self, matches):
    """ Each match will trigger a POST to the specified endpoint(s). """
    for match in matches:
        match_js_esc = _escape_all_values(match)
        payload = match if self.post_all_values else {}
        payload_template = Template(json.dumps(self.post_payload))
        # old line:
        # payload_res = json.loads(payload_template.render(**match_js_esc))
        # new line:
        payload_res = json.loads(payload_template.render(**match_js_esc, **{self.rule['jinja_root_name']: match}))
        payload = {**payload, **payload_res}

With the above change, this now becomes possible:

alert:
  - post2:
      jinja_root_name: _my_custom_field
      http_post2_url: https://localhost/webhook
      http_post2_headers:
        authorization: $api_key$
      http_post2_all_values: false
      http_post2_payload:
        signal:
          original_time: "{{ _my_custom_field['@timestamp'] }}"
          rule:
            name: $name$
            description: $description$
            severity: low
        kibana:
          alert:
            ancestors:
              - id: '{{ _id }}'
                index: '{{ _index }}'

as well as

alert:
  - post2:
      http_post2_url: https://localhost/webhook
      http_post2_headers:
        authorization: $api_key$
      http_post2_all_values: false
      http_post2_payload:
        signal:
          original_time: "{{ _data['@timestamp'] }}"
          rule:
            name: $name$
            description: $description$
            severity: low
        kibana:
          alert:
            ancestors:
              - id: '{{ _id }}'
                index: '{{ _index }}'

It also keeps in line with what the base alerter is already doing

[jertel]

Yes, I would support that approach, if you're interested in submitting a PR. Have you considered passing match_js_esc instead of match as the key value? Seems like for consistency we'd want the escaped version, but I haven't studied the escape code in depth.

I planned to release the next version this week but I can hold off until you get time to make that contribution.

[akusei]

That sounds good. I'm pretty busy this week but I'll work to get a PR done as well as test match_js_esc; I agree it should be the escaped version

[akusei]

@jertel I've noticed that because the rule payload is serialized to a json string, double quotes get escaped. If you have the following payload

http_post2_payload:
  body: '{{ _data["key"] }}'

It gets dumped to a json string as

{"body":"{{ _data[\"key\"] }}"}

Which causes jinja to raise a TemplateSyntaxError. Yaml supports using either body: "{{ _data['key'] }}" or body: '{{ _data["key"] }}' without escaping but json doesn't. I'll add this to documentation but knowing this, would you like to have an exception raised with text suggesting quotes might be incorrect in this circumstance?

A TemplateSyntaxError that doesn't contain "invalid char" and other exceptions will still be raised as normal

[jertel]

It sounds like a useful improvement. Would non-Jinja double quotes still be allowed? Ex:

http_post2_payload:
  body: 'Alert for "foo" triggered'

[akusei]

Correct, double quotes outside of Jinja templates should still be allowed but I'll test today to make sure

[akusei]

There's another option to handle escaped data in the post payload. I kept going back and forth with some ideas because I don't necessarily like the idea of requiring a specific subset of yaml escaping or quotes to account for json rest calls being made with the post2 plugin. This would be a big change but I think I can do it while maintaining existing functionality.

Currently only YAML is supported in the http_post2_payload field but that field gets converted to json when sending. Json and Yaml are fairly interchangeable but in this case it causes some weirdness with needing to escape certain characters or even using different quotes to account for these oddities. What do you think about changing the payload to a string value containing the json to send? For example: this works great:

http_post2_payload: |
  {
    "posted_name": "toto",
    "args_name": "{{_data["some_field"]}}",
    "another_name": "{{_data['some_field']}",
    "no_data_field": "{{some_field}}",
    "field_{{some_field}}": "some value"
  }

This would also still work, but would require the use of specific quotes:

http_post2_payload:
  posted_name: toto
  args_name: "{{_data['some_field']}}"
  no_data_field: "{{some_field}}"
  field_{{some_field}}: some value
  # cannot do the following as it will error
  # another_name: '{{_data["some_field"]}}'
  # likewise, these will fail as well
  # another_name: "{{_data["some_field"]}}"
  # another_name: "{{_data[\"some_field\"]}}"

Errors from syntax issues can still be raised as well. It also handles escaping json values appropriately. What are your thoughts here? If you like this idea, would you like the same treatment of headers?

EDIT: another benefit to this is the ability to use jinja control flow like multiline if statements and for loops. For example, this now becomes possible

data

{"dict_variable":{"field1":"value1","field2":"value2"}}

rule

http_post2_payload: |
  {
    "posted_name": "toto"
    {% for k,v in dict_variable.items() %}
    ,"{{ k }}": "{{ v }}"
    {% endfor %}
    ,"last_field": "value"
  }

output

{"posted_name":"toto","field1":"value1","field2":"value2","last_field":"value"}

I made a copy of every test in httppost2_test.py and changed them to use a yaml doc as the rule with a json string as the payload, everything works as expected, including the existing untouched tests

[jertel]

If the change won't break existing user's rules, yet adds more capability to the project, then that seems like a good change.