Throw NoSuchFileException exception when start for the 1st time

[ebuildy]

• Version: 6.2.4
• Operating System: Using official Docker OSS image

Using a simple multi pipelines:

logstash.yml

dead_letter_queue.enable: true
path.dead_letter_queue: "/usr/share/logstash/dead_letter_queue"

pipelines.yml

- pipeline.id: main
  path.config: "./pipeline/dev_es_strict_mapping.conf"
  pipeline.workers: 1
- pipeline.id: failover
  path.config: "./pipeline/failover.conf"
  pipeline.workers: 1
  dead_letter_queue.enable: false

failover.conf

input
{
  dead_letter_queue
  {
      path => "/usr/share/logstash/dead_letter_queue"
      commit_offsets => true
      pipeline_id => "main"
  }
}

filter
{
  mutate
  {
      add_field => { "_tag" => "mapping_error" }
  }
}

output
{
   stdout
   {
       codec => rubydebug { metadata => true }
   }

   elasticsearch
   {
     hosts => ["elasticsearch:9200"]
     index => "failover-v1"
     manage_template => false
   }
}

At 1st start:

Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2018-05-28T11:25:56,460][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
[2018-05-28T11:25:56,515][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-05-28T11:25:56,556][ERROR][logstash.pipeline        ] Error registering plugin {:pipeline_id=>"failover", :plugin=>"<LogStash::Inputs::DeadLetterQueue path=>\"/usr/share/logstash/dead_letter_queue\", commit_offsets=>true, pipeline_id=>\"main\", id=>\"3fc95e0607478ea9cc5a5768a63439c2610379b27e98a4d310718caa6f4c86e9\", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>\"plain_9eaf1a38-de8a-4330-8082-b42e3cee0ab1\", enable_metric=>true, charset=>\"UTF-8\">>", :error=>"/usr/share/logstash/dead_letter_queue/main", :thread=>"#<Thread:0x5202013f@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}
[2018-05-28T11:25:57,537][ERROR][logstash.pipeline        ] Pipeline aborted due to error {:pipeline_id=>"failover", :exception=>java.nio.file.NoSuchFileException: /usr/share/logstash/dead_letter_queue/main, :backtrace=>["sun.nio.fs.UnixException.translateToIOException(sun/nio/fs/UnixException.java:86)", "sun.nio.fs.UnixException.asIOException(sun/nio/fs/UnixException.java:111)", "sun.nio.fs.LinuxWatchService$Poller.implRegister(sun/nio/fs/LinuxWatchService.java:246)", "sun.nio.fs.AbstractPoller.processRequests(sun/nio/fs/AbstractPoller.java:260)", "sun.nio.fs.LinuxWatchService$Poller.run(sun/nio/fs/LinuxWatchService.java:364)", "java.lang.Thread.run(java/lang/Thread.java:748)"], :thread=>"#<Thread:0x5202013f@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}
[2018-05-28T11:25:57,551][ERROR][logstash.agent           ] Failed to execute action {:id=>:failover, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: LogStash::PipelineAction::Create/pipeline_id:failover, action_result: false", :backtrace=>nil}

I can see the folder is created, at 2nd start, it's working well. Look like logstash didnt initialize the data folder before running the pipeline.

[keshrisohit]

+1

[ijardillier]

+1

[tumbledwyer]

+1

[FabienLamonarca]

+1