Database aware partition scheme

[wangxiaoying]

Currently, we evenly partition the given partition column in order to split a big query to several small ones. But it has some issues:

• User need to specify an integer column for partition manually
• The chosen column might be suboptimal (e.g., it does not have an index while another column has)
• Data might not be uniformly distributed on the partitioned column, which lead to imbalanced small queries and thus affect the performance improvement

We could tackle these issues by pre-fetching metadata like schema, constraints and statistics from DB and automatically partition the query in an optimal way.

Reply to share your ideas!

[bloukanov]

@wangxiaoying can the requirement to supply an integer column be removed? and if the user chooses partitioning, the package splits according to the number of rows it discovers via a count(*)? the user can still provide the number of partitions (cores) they would like

[wangxiaoying]

Hi @bloukanov , thanks for the feedback! It will also be challenging since the partition strategy might be case by case according to the statistics we could get, and also will be unknown if the query is very complex. But indeed it will be beneficial if we can partition the query internally with some statistics collected from db instead of letting user specify a partition column.

[bloukanov]

got it!