[Filebeat] setup check gives spurious ILM error

[faec]

• Version: Filebeat 7.2.0, 7.6.0
• Operating System: confirmed on MacOS
• Discuss Forum URL: https://discuss.elastic.co/t/error-index-management-filebeat-to-logstash/218849
• Steps to Reproduce: Run filebeat setup -e with the following configuration:

  setup.ilm.enabled: false
  
  filebeat.inputs:
  - type: log
    paths: ["/var/log/*.log"]
  
  output.logstash:
    hosts: ["192.168.1.104:5044"]
  

This gives the result 2020-02-14T12:45:57.733-0500 ERROR instance/beat.go:921 Exiting: Index management requested but the Elasticsearch output is not configured/enabled even though ILM is disabled.

This doesn't appear to trigger on a real Filebeat run (filebeat -e), only on setup.

This seems possibly related to #14589 but is still reproducible at head.

[ma-hei]

Just getting started with this code base, but if this is a good first issue, I'd like to look into it.

[faec]

@ma-hei It looks like no one's working on it yet so you're welcome to :-) I haven't looked into the specifics but a good place to start is probably Beat.Setup in libbeat/cmd/instance/beat.go

[ma-hei]

@faec This is what's happening here:
When running ./filebeat setup -e, then Beat.Setup will try to (a) load the index template in Elasticsearch and to (b) setup the Kibana dashboard (among other things). For both (a) and (b), filebeat needs to communicate with Elasticsearch.
Now in filebeat.yml, Filebeat can be configured to send output to Elasticsearch directly or to any other instance (for example Logstash). In Beat.Setup, the error is thrown if the output is configured to be any instance that is not Elasticsearch:
if outCfg.Name() != "elasticsearch" { return fmt.Errorf("Index management requested but the Elasticsearch output is not configured/enabled") }
The method should be changed here, in order to allow the setup to work when the output is not Elasticsearch.
The question is: when the output is not set to Elasticsearch, how should we allow the user to specify the Elasticsearch URL? In case the output is set to Elasticsearch, the Setup method gets this URL directly from the output.elasticsearch section in the config file (which is of course not present, if the output is for example logstash).
I can think of two ways of allowing the user to specify this URL:

1. Allow the user to specify the elasticsearch URL in filebeat.yml (even if output is not sent to Elasticsearch directly)
2. Add a parameter to filebeat setup that allows to specify the Elasticsearch URL (for example filebeat setup -e -elasticsearch.host=localhost:9200 or similar).

What is your opinion on this (Are you the right person for this question? If not, who should be consulted here?)?

[faec]

Hmm -- to be honest, looking at this closer I'm no longer sure if it's a bug per se so much as a confusing error message / incomplete documentation.

Even if ILM is disabled, the other things this command does (like setup index templates) also require an Elasticsearch backend to make sense. I think some parts of the command can theoretically be run in isolation (e.g. --dashboards might only require Kibana?) but it doesn't seem like the primary use.

I'd like the connection to the Filebeat configuration to be clearer -- some parts of the documentation allude to running setup with ILM disabled, but I think it might mean disabled in the Elasticsearch configuration rather than Filebeat? But I think failing with an error here is probably correct, the error should just be more clear about what's being attempted and why it needs an Elasticsearch endpoint.

[ma-hei]

Right. Maybe it would make sense to explain that the output is configured for logstash instead. The setup command can be run successfully by temporarily disabling the logstash output and by passing in the Elasticsearch host, using the -E option. We could give the user this as a hint. So I suggest the error message:
This command attempts index management which requires an Elasticsearch host. In this case output is configured for Logstash. When output is not configured for Elasticsearch directly, this command can be run with the -E option in order to temporarily disable the configured output and provide the Elasticsearch host. Example: "filebeat setup -E output.logstash.enabled=false -E output.elasticsearch.hosts=["localhost:9200"]"
Feel free to modify this error message and I will create a PR.

[jgelsey]

Having just struggled with this error for the last 3 days, I feel it would be awesome to clarify the error message this way.

[RadoNko]

Update: 9th February 2021 and still getting the very same bug/error message, cant move past filebeat setup -e step.

[pplanel]

Hi guys, do we have an update on this issue?

[arun7000]

2021-07-01T21:15:13.254+0530 WARN beater/filebeat.go:178 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-07-01T21:15:13.254+0530 ERROR instance/beat.go:989 Exiting: Index management requested but the Elasticsearch output is not configured/enabled
Exiting: Index management requested but the Elasticsearch output is not configured/enabled

same issue in version filebeat-7.13.2-windows-x86_64, windows 10 os

[Mekk]

Just use some other setting, please. Like management.elasticsearch or sth similar.

filebeat setup happens (and should happen) to be executed during upgrades or other admin operations, having to artificially reconfigure for the sake of this op is troublesome.

Of course in case management.elasticearch is not defined, and output.elasticsearch is, falling back is reasonable and natural. But setup with filebeat passing data to logstash isn't that rare…

[Mekk]

BTW: there already exists similar setting: monitoring.elasticsearch (setup where to put monitoring data, independent from output setup)

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[SteffBauer]

I just came across the problem too. An update would be very helpful 👍🏼

[Goetterfleisch1337]

:+1

[shaahrokh]

Do we have an update on this issue?

[Mekk]

Doesn't seem so. In case sb. decides to look at it, after additional look I still recommend my earlier suggestion:

• introduce possibility to use separate config block like management.elasticsearch or (maybe better) setup.elasticsearch to configure how filebeat setup should access elasticsearch
• (shortcut for simple configs) if this block is missing, try also output.elasticsearch (or merge settings, merging would be more consistent with monitoring)
• patch the error message (so it suggests setup.elasticsearch)

Rationale for syntax and approach : this is exactly how similar problem for monitoring is solved (there is monitoring.elasticsearch block one can use to configure ES access for monitoring data, described here, possibly some code could also be reused).

PS Separate setup.elasticsearch may even make sense in elasticsearch output case („setup” account needs more elevated permissions).

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!