Unexpected error when @timestamp field missing #739

gmoskovicz · 2018-02-22T15:53:06Z

Version: Logstash 5.x and Logstash 6.x
Operating System: Any
Config File (if you have sensitive info, please remove it):

input {
  generator {}
}
filter {
  mutate {
    rename => {
      "@timestamp" => "timestamp"
    }
  }
}

output {
  elasticsearch {
    user => elastic
    password => xxxxx
  }
}

Sample Data:

Any data

Steps to Reproduce:

Run logstash with that configuration.

Output:

5.x

[2018-02-22T13:49:42,425][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: timestamp field is missing>, :backtrace=>["org/logstash/ext/JrubyEventExtLibrary.java:205:in sprintf'", "/Users/Gabriel/Documents/ElasticSearch/logstash-5.6.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:169:in event_action_params'", "/Users/Gabriel/Documents/ElasticSearch/logstash-5.6.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:44:in event_action_tuple'", "/Users/Gabriel/Documents/ElasticSearch/logstash-5.6.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:38:in multi_receive'", "org/jruby/RubyArray.java:2414:in map'", "/Users/Gabriel/Documents/ElasticSearch/logstash-5.6.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:38:in multi_receive'", "/Users/Gabriel/Documents/ElasticSearch/logstash-5.6.2/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13:in multi_receive'", "/Users/Gabriel/Documents/ElasticSearch/logstash-5.6.2/logstash-core/lib/logstash/output_delegator.rb:49:in multi_receive'", "/Users/Gabriel/Documents/ElasticSearch/logstash-5.6.2/logstash-core/lib/logstash/pipeline.rb:436:in output_batch'", "org/jruby/RubyHash.java:1342:in each'", "/Users/Gabriel/Documents/ElasticSearch/logstash-5.6.2/logstash-core/lib/logstash/pipeline.rb:435:in output_batch'", "/Users/Gabriel/Documents/ElasticSearch/logstash-5.6.2/logstash-core/lib/logstash/pipeline.rb:381:in worker_loop'", "/Users/Gabriel/Documents/ElasticSearch/logstash-5.6.2/logstash-core/lib/logstash/pipeline.rb:342:in `start_workers'"]}

6.x

[2018-02-22T13:50:52,960][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: timestamp field is missing>, :backtrace=>["org/logstash/ext/JrubyEventExtLibrary.java:168:in sprintf'", "/Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:169:in event_action_params'", "/Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:39:in event_action_tuple'", "/Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:34:in block in multi_receive'", "org/jruby/RubyArray.java:2486:in map'", "/Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:34:in multi_receive'", "/Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13:in multi_receive'", "/Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/logstash-core/lib/logstash/output_delegator.rb:50:in multi_receive'", "/Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/logstash-core/lib/logstash/pipeline.rb:487:in block in output_batch'", "org/jruby/RubyHash.java:1343:in each'", "/Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/logstash-core/lib/logstash/pipeline.rb:486:in output_batch'", "/Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/logstash-core/lib/logstash/pipeline.rb:438:in worker_loop'", "/Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/logstash-core/lib/logstash/pipeline.rb:393:in `block in start_workers'"]}

LogStash::Error: timestamp field is missing
                 sprintf at org/logstash/ext/JrubyEventExtLibrary.java:168
     event_action_params at /Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:169
      event_action_tuple at /Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:39
  block in multi_receive at /Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:34
                     map at org/jruby/RubyArray.java:2486
           multi_receive at /Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:34
           multi_receive at /Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13
           multi_receive at /Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/logstash-core/lib/logstash/output_delegator.rb:50
   block in output_batch at /Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/logstash-core/lib/logstash/pipeline.rb:487
                    each at org/jruby/RubyHash.java:1343
            output_batch at /Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/logstash-core/lib/logstash/pipeline.rb:486
             worker_loop at /Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/logstash-core/lib/logstash/pipeline.rb:438
  block in start_workers at /Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/logstash-core/lib/logstash/pipeline.rb:393

Should this be a better error rather than FATAL? Given that this could happen just with a specific document (if you have conditionals), best is to fail that event rather than a FATAL error?

The text was updated successfully, but these errors were encountered:

guyboertje · 2018-02-22T16:35:27Z

The @timestamp is missing so the default index string can't be built

  mod.config :index, :validate => :string, :default => "logstash-%{+YYYY.MM.dd}"

KevSex · 2018-05-01T15:46:52Z

Has there been any fix for this? Just ran into this FATAL error in 6.2.3

Had to create the following workaround in hope that this stops it from crashing logstash in future.

filter {
  if ![@timestamp] {
    drop { }
  }
}

Note that if the index doesn't require a timestamp to resolve, events are let through. Closes logstash-plugins#739.

webmat · 2018-05-25T19:15:32Z

Fix for this incoming: #777.

webmat · 2018-05-30T19:04:52Z

@gmoskovicz In case you're interested, discussions on my fix for this issue took a turn. Discussion will be happening here #779 :-)

Perhaps you can chime in with what you think our customers would prefer?

Jmainguy · 2018-06-28T17:58:56Z

@KevSex did that actually stop it from crashing for you? I tried adding that in /etc/logstash/conf.d/ files but didnt seem to change anything, logstash keeps flapping for me.

EDIT:

Ignore me, we had a bad mutate that was removing the timestamp field later, womp.

vacri · 2018-09-13T06:31:34Z

Still a problem in Logstash 6.4.

I believe the problem for us came from using the documented Nginx snippet at https://www.elastic.co/guide/en/logstash/current/logstash-config-for-filebeat-modules.html#parsing-nginx

For the nginx access log, the 'read_timestamp' is an added field
For the nginx error log, the 'read_timestamp' is renamed from @timestamp, removing @timestamp

milesgillham · 2018-11-06T00:12:25Z

@vacri - the fault is in the example they give, you have correctly pointed out the problem which is caused by the reference to the timestamp in the output:

"%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"

So to fix it replace this:

mutate { rename => { "@timestamp" => "read_timestamp" } }

with:

mutate { add_field => { "read_timestamp" => "%{@timestamp}" } }

ES needs to update their example, ideally. :-)

gmoskovicz added bug discuss labels Feb 22, 2018

guyboertje mentioned this issue Feb 22, 2018

Certain @fields are reserved fields and should not be removed. elastic/logstash#6728

Open

webmat self-assigned this May 25, 2018

webmat pushed a commit to webmat/logstash-output-elasticsearch that referenced this issue May 25, 2018

Use the helper to filter out messages missing a timestamp...

2492c0c

Note that if the index doesn't require a timestamp to resolve, events are let through. Closes logstash-plugins#739.

webmat mentioned this issue May 25, 2018

Survive attempts at indexing events that don't have a @timestamp #777

Closed

webmat added the crashes label May 25, 2018

webmat mentioned this issue May 29, 2018

How should output-elasticsearch deal with a missing @timestamp? #779

Open

webmat removed their assignment Sep 13, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unexpected error when @timestamp field missing #739

Unexpected error when @timestamp field missing #739

gmoskovicz commented Feb 22, 2018 •

edited

Loading

guyboertje commented Feb 22, 2018

KevSex commented May 1, 2018

webmat commented May 25, 2018

webmat commented May 30, 2018

Jmainguy commented Jun 28, 2018 •

edited

Loading

vacri commented Sep 13, 2018

milesgillham commented Nov 6, 2018

Unexpected error when @timestamp field missing #739

Unexpected error when @timestamp field missing #739

Comments

gmoskovicz commented Feb 22, 2018 • edited Loading

5.x

6.x

guyboertje commented Feb 22, 2018

KevSex commented May 1, 2018

webmat commented May 25, 2018

webmat commented May 30, 2018

Jmainguy commented Jun 28, 2018 • edited Loading

vacri commented Sep 13, 2018

milesgillham commented Nov 6, 2018

gmoskovicz commented Feb 22, 2018 •

edited

Loading

Jmainguy commented Jun 28, 2018 •

edited

Loading