Cannot get Connection from Pool

[gingerwizard]

• Version: 5.5.1

• Operating System: OSX

• Steps to Reproduce:
  This issue manifests itself when the plugin is under high load with more than 1 LS worker configured. Assuming each document results in an ES query (these typically execute in a few ms) the plugin with perodically log:

[2017-09-13T14:39:37,075][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"threats", :query=>{"query"=>{"percolate"=>{"field"=>"query", "document_type"=>"threat", "document"=>{"entity"=>["MacBook-Pro.local", "fe80::12c3:7bff:fede:dc18", "fe80::83f:d39:1b5d:a2d4"]}}}, "size"=>10}, :event=>2017-03-30T22:51:21.628Z MacBook-Pro.local %{message}, :error=>#<Elasticsearch::Transport::Transport::Error: Cannot get new connection from pool.>}

With a single worker this issue does not occur. I will assemble a reproducible config and test case.

[gingerwizard]

Steps to reproduce:

1. Ensure ES is installed > 5.5.x
2. Create a simple index - ensure the doc has @timestamp field

PUT test/test/1
{
  "message":"test",
  "@timestamp":"2017-09-10T23:00:00.000Z"
}

3. Configure LS as follows

input {
    stdin {
    }
}

filter {
        elasticsearch {
              hosts => ["localhost:9200"]
              query => "test"
              index => "test"
              fields => [ ["message", "match"] ]
              user => "elastic"
              password => "changeme"
        }
}


output {
     stdout { codec => dots }
}

4. Ensure workers in LS is > 1 (default if more than 1 core)
5. Pipe any large file though LS i.e.

cat /tmp/nice_big_file | ./logstash -f ../conf/es_filter_issue/logstash.conf

[ph]

@gingerwizard

We have discussed this morning and like I've said it appear to be a thread safety issues in the elasticsearch-ruby client that is show under heavy load.

We have two plan for this plugin:

Long term
We are planning to rewrite the plugin using the same client library that we have developed for the elasticsearch output or use the java native client.

Short term
Meanwhile we will implement a fix to work around the thread safety issue, we will create a new connection object for every worker thread. This will waste a bit of resource but it will make sure that every thread have their own view of the world.

[ph]

@gingerwizard I have created #77, and it appears to fix the issue you were having. With your use case I can reproduce the error when I apply the PR the error is gone.

I have created #78 as a long term goal.