Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] data-prepper sink fails to create templates when number_of_shards is passed #4960

Open
egarbi opened this issue Sep 19, 2024 · 1 comment
Labels
bug Something isn't working follow up

Comments

@egarbi
Copy link

egarbi commented Sep 19, 2024

Describe the bug
I have data-prepper creating templates and sending data with no issue
This is the config

          sink:
            - opensearch:
                hosts: ["$${OS_HOST}"]
                username: $${OS_USER}
                password: $${OS_PASS}
                index_type: trace-analytics-raw
                template_type: index-template
                max_retries: 10

My cluster sets the number of shards=5 as the default value for new indices being rolled over.
So, I tried adding an extra line to change that settings as stated here in the docs

number_of_shards: 1

But as soon I restart the service it starts to fail with the following error

Request failed: [x_content_parse_exception] [1:1118] [index_template] unknown field [settings]

To Reproduce
Steps to reproduce the behavior:

  1. Set a sink to use OpenSearch as shown above using number_of_shards: N
  2. Restart the service

Expected behavior
Templates should be created using the right sharding number.

Environment (please complete the following information):

  • Amazon OpenSearch service
  • Version [2.11]
@egarbi egarbi added bug Something isn't working untriaged labels Sep 19, 2024
@dlvenable dlvenable changed the title [BUG?] data-prepper sink fails to create templates when number_of_shards is passed [BUG] data-prepper sink fails to create templates when number_of_shards is passed Oct 1, 2024
@dlvenable
Copy link
Member

@egarbi , Did you try using template_type: v1? This should work for setting the number of shards.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working follow up
Projects
Development

No branches or pull requests

2 participants