Index template loading from beats

[tahaderouiche]

Beats documentation contains a lot of references about index templates.

Although this is a feature of beats, when trying to customise these, it generally adds additional overhead to have custom templates on beats agents. It will also require additional permissions on the beat agent user when security is enabled to load templates.

A more sensible approach is to recommend users to centrally manage these index templates. Through Elasticsearch API or other means in the future.

The docs can reflect that. Provide guidance around best practices on managing/customising index templates.

This is also valid for ILM and Kibana dashboards as well.

Having a central way to manage them makes more sense when you are running with a large number of beats, and it would be good to reflect this in docs.

[Leaf-Lin]

When working on documentation improvement, can you also make sure to stress the "danger" of having setup.template.enabled: true and setup.template.overwrite: true?

These setting could potentially cause elasticsearch updating templates every time when there's a document get ingested and that's a huge load to elasticsearch cluster if there are large number of templates to be updated. This is frequently seen as a cause of unstable cluster.

Here are few places the settings were mentioned:

• https://www.elastic.co/guide/en/beats/filebeat/current/configuration-template.html
• https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-template.html#load-template-manually

Worth adding the same comment into the reference yml file.
Presumably, other beats will need to have the same attention too.

[simitt]

@Leaf-Lin templates only get loaded to ES when the beat is started or the setup command is run. The setup.template.* settings do not have an impact during document ingestion.

[Leaf-Lin]

Ah. sorry, I probably got those settings confused somehow. What I was referring to was elastic/elasticsearch#57662 which has been fixed in elastic/elasticsearch#57851. This was caused by some template update settings in beats.

[DaveCTurner]

Even though each Beat only triggers a create-template request at startup, we nonetheless encounter quite a few clusters that are completely overwhelmed by an excess of create-template requests thanks to a large number of Beats all (presumably) starting up at the same time. We've made some changes on the Elasticsearch side to improve our handling in this area, and more improvements on the way, but it'd be even better to avoid having to handle all these requests in the first place. So I'm +1 on documenting the risks of setting setup.template.overwrite: true on all your Beats.

[inqueue]

@Leaf-Lin templates only get loaded to ES when the beat is started or the setup command is run. The setup.template.* settings do not have an impact during document ingestion.

This is not the case when Beats is receiving a back pressure response from the cluster and setup.template.overwrite: false. For example:

Metricbeat detects the template:

2021-02-01T14:23:36.300-0500    INFO    template/load.go:89     Template metricbeat-7.8.1 already exists and will not be overwritten.
2021-02-01T14:23:36.300-0500    INFO    [index-management]      idxmgmt/std.go:298      Loaded index template.
2021-02-01T14:23:36.300-0500    INFO    [publisher_pipeline_output]     pipeline/output.go:152  Connection to backoff(elasticsearch(https://host:9200)) established

Then encounters a 429:

2021-02-01T14:24:41.551-0500    ERROR   [elasticsearch] elasticsearch/client.go:223     failed to perform any bulk index operations: 429 Too Many Requests: {"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [4141563072/3.8gb], which is larger than the limit of [4080218931/3.7gb], real usage: [4141422960/3.8gb], new bytes reserved: [140112/136.8kb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=2583644394/2.4gb, accounting=0/0b]","bytes_wanted":4141563072,"bytes_limit":4080218931,"durability":"TRANSIENT"}],"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [4141563072/3.8gb], which is larger than the limit of [4080218931/3.7gb], real usage: [4141422960/3.8gb], new bytes reserved: [140112/136.8kb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=2583644394/2.4gb, accounting=0/0b]","bytes_wanted":4141563072,"bytes_limit":4080218931,"durability":"TRANSIENT"},"status":429}

Now the publisher is disconnected. Retry:

2021-02-01T14:24:43.234-0500    INFO    [publisher_pipeline_output]     pipeline/output.go:144  Connecting to backoff(elasticsearch(https://host:9200))
2021-02-01T14:24:43.234-0500    INFO    [publisher]     pipeline/retry.go:221   retryer: send unwait signal to consumer
2021-02-01T14:24:43.234-0500    INFO    [publisher]     pipeline/retry.go:225     done
2021-02-01T14:24:43.363-0500    INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 7.8.1

License check GET /_license:

2021-02-01T14:24:56.334-0500    INFO    [license]       licenser/es_callback.go:51      Elasticsearch license: Platinum
2021-02-01T14:24:56.559-0500    INFO    template/load.go:109    Try loading template metricbeat-7.8.1 to Elasticsearch

Try to load the template:

2021-02-01T14:24:56.559-0500    INFO    template/load.go:109    Try loading template metricbeat-7.8.1 to Elasticsearch
...<monitoring messages removed>...
2021-02-01T14:26:30.537-0500    ERROR   [publisher_pipeline_output]     pipeline/output.go:155  Failed to connect to backoff(elasticsearch(https://host:9200)): Connection marked as failed because the onConnect callback failed: error loading template: could not load template. Elasticsearch returned: couldn't load template: Put https://host:9200/_template/metricbeat-7.8.1: net/http: request canceled (Client.Timeout exceeded while awaiting headers). Response body: . Template is: {

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!