Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow the use of the size parameter inside a query #142

Open
Sergi-GC opened this issue Dec 1, 2020 · 0 comments
Open

Allow the use of the size parameter inside a query #142

Sergi-GC opened this issue Dec 1, 2020 · 0 comments

Comments

@Sergi-GC
Copy link

Sergi-GC commented Dec 1, 2020

For all general issues, please provide the following details for fast resolution:

  • Version: 7.9.1
  • Config File (if you have sensitive info, please remove it):
input {
  elasticsearch {
    cloud_id => "<deployment_endpoint>"
    cloud_auth => "<username>:<password>"
    index => "<filebeat-*>"
    schedule => "*/5 * * * *"
    query => '{ "size": 10, "query": { "bool": { "must_not": [ { "query_string": { "query": "(starting)", "default_field": "status" } } ], "filter": [ { "bool": { "must_not": { "exists": { "field": "event_ts" } } } } ] } } }'
    docinfo => true
  }
}
  • Use case: Replacement for the Elapsed filter plugin ingesting Kafka logs
  • Sample Data: Kafka logs
  • Workaround: Reducing the number of hits with a range filter or by limiting the number of indices to query...
  • Steps to Reproduce:

While using the Elasticsearch input plugin to query against a set of indices, it seems not possible to limit the number of documents returned by the query.

The size parameter of the Elasticsearch input plugin is used with scroll queries to limit the number of items per page in the scroll, not the total number of documents.

And the size parameter inside the query is ignored and the query always returns the default 10.000 documents.

This was reported in the discuss forum with a simple example:
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html

I'd like to be able to use the size parameter inside the query field to limit the number of documents returned.

Related: #85

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant