Certain @fields are reserved fields and should not be removed.

[guyboertje]

We need an enhancement to the remove_field code that should check if a field is reserved, ignore that field and log warn this.

Removing the @timestamp field will cause undefined method for nil class when trying to use it later.

[liketic]

@suyograo Can I take this?

[andrewvc]

Why is it that @timestamp should not be removable? Is this actually a bug?

While I understand that is the behavior today, is that intentional?

I think it's fine for us to decide that certain fields are reserved, but I'd like to understand the history of those fields before we flag them specially.

In Logstash 2.2.2 for instance

bin/logstash -e 'input { generator {} } filter { mutate { remove => "@timestamp" } } output { stdout { codec => json_lines } }'

ran successfully.

As did

bin/logstash -e 'input { generator {} } filter { mutate { remove => "@metadata" } } output { stdout { codec => json_lines } }'

This feels more like accidental behavior to me as a result of moving the Event class to Java.

[guyboertje]

I believe it is the elasticsearch output mechanism that breaks on @timestamp removal.

[AbdulHaseebHussain]

What should the next steps be in regards to this issue?

[guyboertje]

@andrewvc

Dynamic values that use time format String interpolations fail without a timestamp.
Like when setting a time based index name or a day_of_week field.

The Java code expects that the timestamp is never null. https://github.com/elastic/logstash/blob/master/logstash-core/src/main/java/org/logstash/StringInterpolation.java#L44

[guyboertje]

Like this one. logstash-plugins/logstash-output-elasticsearch#739

[AbdulHaseebHussain]

Maybe this restriction of preventing deletion of reserved fields could be restricted to the timestamp field.

[andrewvc]

My point is that the java code is not necessarily correct. I think the conversation we should be having is what behaviour would be the best UX. What would be best in line with the principle of least surprise?

…

On Feb 26, 2018 1:41 PM, "AbdulHaseebHussain" ***@***.***> wrote: Maybe this restriction of preventing deletion of reserved fields could be restricted to the timestamp field. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6728 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIBY4Ytod72jKRZ4C6FtX9gGqRCoFRSks5tYyUFgaJpZM4MEioL> .