Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[enhance](mtmv)External partition prune #44415 #44567 #44673 #44767

Merged
merged 4 commits into from
Nov 29, 2024

Conversation

zddr
Copy link
Contributor

@zddr zddr commented Nov 29, 2024

zddr added 3 commits November 29, 2024 10:30
…ad of partitionId (apache#44415)

The partition ID of external data sources is meaningless, and some data
sources only have partition names, so the return result of partition
pruning is replaced with name instead of ID
…44567)

Previously, external partition cropping only supported Hive. If you want
to support other types of tables, you need to understand the internal
processing logic of partition pruning.

This PR abstracts the logic of partition pruning, and other tables can
be implemented by simply covering a few methods of externalTable

[opt](planner) Unified external partition prune interface
…resh and partition pruning (apache#44673)

- Add `MvccTable` to represent a table that supports querying specified
version data
- Add the `MvccSnapshot` interface to store snapshot information of mvcc
at a certain moment in time
- Add the `MvccSnapshot` parameter to the method of the
`MTMVRelatedTableIf `interface to retrieve data of a specified version
- Partition pruning related methods combined with the `MvccSnapshot`
parameter are used to obtain partition information for a specified
version
- Load the snapshot information of mvccTable at the beginning of the
query plan and store it in StatementContext

Unified external table interface supporting partition refresh and
partition pruning
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zddr
Copy link
Contributor Author

zddr commented Nov 29, 2024

run buildall

@morningman morningman changed the title [enhance](mtmv)External partition prune [enhance](mtmv)External partition prune #44415 #44567 #44673 Nov 29, 2024
@zddr
Copy link
Contributor Author

zddr commented Nov 29, 2024

run buildall

@yiguolei yiguolei merged commit e03517e into apache:branch-2.1 Nov 29, 2024
18 of 19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants