To support incremental scan ranges deployment.

[dirtysalt]

Enhancement

In practice we have found that it can take a lot of time on the FE to get all the files that need to be scanned, and then send them down to the BE for execution. This is fine for native table, but not ideal for hive/iceberg/hudi/deltalake.

Ideally, the FE should start executing on the BE after scanning some of the files, while the FE continues to scan the rest of the files. This way, the whole execution process can be executed in parallel on both the FE and the BE, thus shortening the execution time.

Arch diagram:

two new session variables:

• enable_connector_incremental_scan_ranges=true (if to enable incremental scan ranges deployment)
• connector_incremental_scan_ranges_size=50 (if enabled, how many scan ranges delivered each round)

To achieve this goal, the following things need to be done.

• the FE should support incremental acquisition of scanned files.
  • [Refactor] reduce interfaces to getRemoteFiles #49230
  • [Refactor] refactor getRemoteFiles into a async way #49352
  • [Refactor] add hive/hudi connector scan range source #49451
• incremental deployment of files to the BE should be supported on the FE.
  • [Enhancement] support incremental scan ranges deployment at FE side #50189
• the BE should support incremental execution of files.
  • [Enhancement] support incremental scan ranges deployment at BE side #50254