allow workers added to cluster after start of a query to contribute to running the query

[tooptoop4]

scenario:

1. cluster is started with 2 workers
2. select * from table where col = 'x' query starts running, (2 workers are contributing to this query)
3. cluster is scaled up to 5 workers
4. query from step 2 is still running but still only using the resources of the initial 2 workers, ideally it can make use of all 5 workers as this query takes 20mins to run.

In my case I am talking about simple queries that return GBs of data like:

query type 1:
select * from table;
or
query type 2:
select * from table where col = 'x';

I see these operators being used in query type 2:
ExchangeOperator
TaskOutputOperator
ScanFilterAndProjectOperator

current limited behavior:
"New workers will only be utilized for table scans (and only if the splits are not already assigned). The machines used for hash partitions, i.e. aggregation and joins, are fixed when the query starts."

[sopel39]

New workers will execute table scans from existing queries. However, they won't participate in (final) aggregations or joins

[tooptoop4]

do u know which version that was added in? new workers are not being assigned for me in query type 2. i think the TaskOutputOperator is taking the most time
json2wrk.txt

[sopel39]

This have been part of Presto for long time

[dain]

I think this has been in since the first public release of Presto (like 7 years ago). This feature was a side effect of doing streaming discovery of splits, which we added because we couldn't load all splits into memory for large tables. As splits are "discovered", we assign them to the currently available machines.

Should we close this issue?

[tooptoop4]

it doesn't work though. query json shows 5 workers did not participate

[electrum]

It depends how many splits are produced for the table scan. If a split is assigned to a worker, the split must execute there, even if that worker is busy and a new worker shows up later.

You can try decreasing node-scheduler.max-splits-per-node and node-scheduler.max-pending-splits-per-task so that splits are scheduled closer to when they are needed. Note that this can increase query latency, since workers will need to wait for the coordinator to notice they are under capacity and schedule more work.

[hashhar]

With FTE this is closer to reality.