Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature][Manager][Sort] Resource adaptive adjustment for Hudi #7072

Closed
2 tasks done
Tracked by #6781
featzhang opened this issue Dec 26, 2022 · 1 comment · May be fixed by #7077
Closed
2 tasks done
Tracked by #6781

[Feature][Manager][Sort] Resource adaptive adjustment for Hudi #7072

featzhang opened this issue Dec 26, 2022 · 1 comment · May be fixed by #7077
Labels
stage/stale Issues or PRs that had no activity for a long time type/feature

Comments

@featzhang
Copy link
Member

Description

Hudi flink jobs often have unreasonable resource allocation. Too much allocation will lead to a waste of resources, and too little will lead to back pressure or OOM.

When allocating resources, you first need to determine the concurrency of the source side to ensure that there is no data backlog upstream when reading. Here is a general configuration situation, such as partitioning by day, with about 15 billion data per day, and about 50 concurrent configurations. Other data volumes can be converted appropriately.

After determining the concurrency on the source side, you can configure the concurrency of write according to the ratio of 1:1.5 or 1:2.

Use case

No response

Are you willing to submit PR?

  • Yes, I am willing to submit a PR!

Code of Conduct
@featzhang featzhang mentioned this issue Dec 26, 2022
Closed
16 tasks
@featzhang featzhang changed the title [Feature][Manager] Resource adaptive adjustment for Hudi [Feature][Manager][Sort] Resource adaptive adjustment for Hudi Dec 27, 2022
@featzhang featzhang mentioned this issue Dec 27, 2022
Draft
3 tasks
@github-actions
Copy link

This issue is stale because it has been open for 60 days with no activity.

@github-actions github-actions bot added the stage/stale Issues or PRs that had no activity for a long time label Feb 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
stage/stale Issues or PRs that had no activity for a long time type/feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[INLONG-7072][Manager][Sort] Resource adaptive adjustment for Hudi featzhang/inlong
2 participants
@featzhang @dockerzhang