Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

branch-3.0: [fix](hudi) upgrade hudi to 0.15.0 (#44267)(#44995) #45054

Merged
merged 2 commits into from
Dec 5, 2024
Merged

branch-3.0: [fix](hudi) upgrade hudi to 0.15.0 (#44267)(#44995) #45054

merged 2 commits into from
Dec 5, 2024

Conversation

suxiaogang223
Copy link
Contributor

@suxiaogang223 suxiaogang223 commented Dec 5, 2024 •
edited by morningman
Loading

cherry-pick (#44267)(#44995)

@suxiaogang223
[fix](hudi) upgrade hudi to 0.15.0 (apache#44267)
29f54a7 
### What problem does this PR solve?

1. upgrade hudi to 0.15.0.
2. impl new hudi jni reader based on hudi-hadoop-mr 
3. add session variable `hudi_jni_scanner` to choose which hudi jni
reader to use, "hadoop" means HadoopHudiJniReader, "spark" means old
HudiJniReader, default value is "hadoop"
4. support session variable `force_jni_scanner` for hudi
5. add more cases for hudi p2

### Release note
[opt](hudi) upgrade hudi to 0.15 and support hadoop jni reader
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@suxiaogang223
[fix](hudi) support reading hudi read optimized table with orc format (…
8cad0e4 
…apache#44995)

### What problem does this PR solve?
Problem Summary:
When reading the hudi ro table, it will be pushed back from jni to the
native reader. However, this process will default the file format to
parquet, and does not consider the situation that the hudi table is
stored in orc format.

1. support reading hudi read optimized table with orc format
2. fix explain results of hudiScanNode when force_jni_reader=true
3. add cases about  timestamp with different timezones
@suxiaogang223 suxiaogang223 changed the title branch-3.0: [fix](hudi) upgrade hudi to 0.15.0 (#44267) branch-3.0: [fix](hudi) upgrade hudi to 0.15.0 (#44267)(#44995) Dec 5, 2024
@suxiaogang223
Copy link
Contributor Author

run buildall

1 similar comment
@suxiaogang223
Copy link
Contributor Author

run buildall

@morningman morningman merged commit d6482e1 into apache:branch-3.0 Dec 5, 2024
19 of 21 checks passed
@suxiaogang223 suxiaogang223 deleted the fix_hudi branch December 12, 2024 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CalvinKirs CalvinKirs Awaiting requested review from CalvinKirs CalvinKirs is a code owner

@morningman morningman Awaiting requested review from morningman morningman is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@suxiaogang223 @doris-robot @morningman