Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

routed_sender size property overwrites sender_api size property #374

Open
ciorg opened this issue Dec 1, 2020 · 0 comments
Open

routed_sender size property overwrites sender_api size property #374

ciorg opened this issue Dec 1, 2020 · 0 comments
Labels
bug Something isn't working

Comments

@ciorg
Copy link
Member

ciorg commented Dec 1, 2020

The size setting in the elasticsearch_sender_api is overwritten by the routed_sender size.

Here's the jobs settings that I was using:

   {
            "_op": "routed_sender",
            "api_name": "elasticsearch_sender_api",
            "routing": {
                "**": "ES-CLUSTER"
            }
        }
    ],
    "apis": [
        {
            "_name": "elasticsearch_sender_api",
            "index": "INDEX-NAME",
            "type": "_doc",
            "size": 20000
        }
    ],

Both processors chunk the incoming data array based on the size to pass it on to the next step. The routed_sender uses a promise map with a concurrency setting, while the elasticsearch_sender uses Promise.all to process the array chunks.

The routed_sender's default size is 500, and concurrency is 10.

I believe the elasticsearch_sender should use its respective size settings.

Looks like it's happening on the api.create -

client = await this.api.create(

if the config were removed from the create action like this:

client = await this.api.create(
                config.connection,
                {
                    tryFn: this.tryFn,
                    logger: this.logger
                }
            );

I'm not sure how the settings in the apis array are passed in. Looked around but didn't find it. But it looks like any fields shared between the api configs and the routed_sender would be overwritten by the routed_sender configs.

How would this affect other sender_apis, not just es? Seems like they should all use their own settings.

@ciorg ciorg added the bug Something isn't working label Dec 1, 2020
@ciorg ciorg changed the title routed_sender size setting conflict with doe routed_sender size setting conflict with sender_api Dec 1, 2020
@ciorg ciorg changed the title routed_sender size setting conflict with sender_api routed_sender size property overwrites sender_api size property Dec 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant