Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Enhancement] add more profile and metrics to measure runtime bloom filter size #46360

Merged
merged 3 commits into from
May 29, 2024
Merged

[Enhancement] add more profile and metrics to measure runtime bloom filter size #46360

merged 3 commits into from
May 29, 2024

Conversation

silverbullet233
Copy link
Contributor

@silverbullet233 silverbullet233 commented May 28, 2024 •
edited
Loading

Why I'm doing:

What I'm doing:

Major changes:

  1. add new metrics to query profile to observe the memory occupied by partial runtime bloom filter.
  2. add new be system metrics to observe the number of rf events in the queue and the bloom filter memory consumption in the queue.

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function
  • This is a backport pr

Bugfix cherry-pick branch check:

  • I have checked the version labels which the pr will be auto-backported to the target branch
    • 3.3
    • 3.2
    • 3.1
    • 3.0
    • 2.5

@silverbullet233 silverbullet233 requested review from a team as code owners May 28, 2024 08:39
@silverbullet233
fix
e74dc5d 
Signed-off-by: silverbullet233 <[email protected]>
satanson
satanson previously approved these changes May 28, 2024
View reviewed changes
stdpain
stdpain previously approved these changes May 28, 2024
View reviewed changes
ZiheLiu
ZiheLiu previously approved these changes May 28, 2024
View reviewed changes
@silverbullet233
fix format
8f9e5f9 
Signed-off-by: silverbullet233 <[email protected]>
@silverbullet233 silverbullet233 dismissed stale reviews from ZiheLiu, stdpain, and satanson via 8f9e5f9 May 29, 2024 06:22
@github-actions GitHub Actions
Copy link

[FE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

@github-actions GitHub Actions
Copy link

[BE Incremental Coverage Report]

✅ pass : 72 / 76 (94.74%)

file detail

path covered_line new_line coverage not_covered_line_detail
🔵 be/src/runtime/runtime_filter_worker.h 18 21 85.71% [141, 142, 144]
🔵 be/src/util/system_metrics.cpp 23 24 95.83% [688]
🔵 be/src/exec/pipeline/hashjoin/hash_join_build_operator.cpp 7 7 100.00% []
🔵 be/src/runtime/runtime_filter_worker.cpp 18 18 100.00% []
🔵 be/src/exec/hash_joiner.cpp 1 1 100.00% []
🔵 be/src/exprs/runtime_filter.h 5 5 100.00% []

@silverbullet233 silverbullet233 enabled auto-merge (squash) May 29, 2024 08:34
kevincai
kevincai approved these changes May 29, 2024
View reviewed changes
be/src/util/system_metrics.cpp
for (int i = 0; i < EventType::MAX_COUNT; i++) {
auto* metrics = new RuntimeFilterMetrics();
const auto& type = EventTypeToString((EventType)i);
#define REGISTER_RUNTIME_FILTER_METRIC(name) \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe better move this macro out of the function body, favor for code readability.

be/src/util/system_metrics.cpp
@@ -135,6 +143,9 @@ SystemMetrics::~SystemMetrics() {
if (_line_ptr != nullptr) {
free(_line_ptr);
}
for (auto& it : _runtime_filter_metrics) {
delete it.second;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

may consider using std::unique_ptr to manage the pointers instead of using the raw one. so the resource deallocation will be as easy as _runtime_filter_metrics.clear()

@silverbullet233 silverbullet233 merged commit 16ec9c1 into StarRocks:main May 29, 2024
46 checks passed
@silverbullet233 silverbullet233 deleted the rf_metrics branch May 29, 2024 08:44
@github-actions github-actions bot added the 3.3 label May 29, 2024
@silverbullet233
Copy link
Contributor Author

@mergify backport branch-2.5 branch-3.0 branch-3.1 branch-3.2 branch-3.3

@mergify Mergify
Copy link
Contributor

mergify bot commented May 29, 2024 •
edited
Loading

backport branch-2.5 branch-3.0 branch-3.1 branch-3.2 branch-3.3

✅ Backports have been created

mergify bot pushed a commit that referenced this pull request May 29, 2024
@silverbullet233 @mergify
[Enhancement] add more profile and metrics to measure runtime bloom f…
81476c6 
…ilter size (#46360)

Signed-off-by: silverbullet233 <[email protected]>
(cherry picked from commit 16ec9c1)

# Conflicts:
#	be/src/exec/pipeline/hashjoin/hash_join_build_operator.cpp
#	be/src/runtime/runtime_filter_worker.cpp
#	be/src/runtime/runtime_filter_worker.h
This was referenced May 29, 2024
Closed
Closed
mergify bot pushed a commit that referenced this pull request May 29, 2024
@silverbullet233 @mergify
[Enhancement] add more profile and metrics to measure runtime bloom f…
53e670b 
…ilter size (#46360)

Signed-off-by: silverbullet233 <[email protected]>
(cherry picked from commit 16ec9c1)

# Conflicts:
#	be/src/exec/pipeline/hashjoin/hash_join_build_operator.cpp
#	be/src/runtime/runtime_filter_worker.cpp
#	be/src/runtime/runtime_filter_worker.h
mergify bot pushed a commit that referenced this pull request May 29, 2024
@silverbullet233 @mergify
[Enhancement] add more profile and metrics to measure runtime bloom f…
453581c 
…ilter size (#46360)

Signed-off-by: silverbullet233 <[email protected]>
(cherry picked from commit 16ec9c1)

# Conflicts:
#	be/src/runtime/runtime_filter_worker.cpp
#	be/src/runtime/runtime_filter_worker.h
This was referenced May 29, 2024
Closed
Closed
silverbullet233 added a commit to silverbullet233/starrocks that referenced this pull request May 30, 2024
@silverbullet233
[Enhancement] add more profile and metrics to measure runtime bloom f…
b922254 
…ilter size (StarRocks#46360)

Signed-off-by: silverbullet233 <[email protected]>
@silverbullet233 silverbullet233 mentioned this pull request May 30, 2024
Merged
24 tasks
silverbullet233 added a commit to silverbullet233/starrocks that referenced this pull request May 30, 2024
@silverbullet233
[Enhancement] add more profile and metrics to measure runtime bloom f…
69ae917 
…ilter size (StarRocks#46360)

Signed-off-by: silverbullet233 <[email protected]>
@silverbullet233 silverbullet233 mentioned this pull request May 30, 2024
Merged
24 tasks
silverbullet233 added a commit to silverbullet233/starrocks that referenced this pull request May 30, 2024
@silverbullet233
[Enhancement] add more profile and metrics to measure runtime bloom f…
99ca170 
…ilter size (StarRocks#46360)

Signed-off-by: silverbullet233 <[email protected]>
@silverbullet233 silverbullet233 mentioned this pull request May 30, 2024
Merged
24 tasks
silverbullet233 added a commit to silverbullet233/starrocks that referenced this pull request May 30, 2024
@silverbullet233
[Enhancement] add more profile and metrics to measure runtime bloom f…
d48d1a3 
…ilter size (StarRocks#46360)

Signed-off-by: silverbullet233 <[email protected]>
@silverbullet233 silverbullet233 mentioned this pull request May 30, 2024
Merged
24 tasks
silverbullet233 added a commit to silverbullet233/starrocks that referenced this pull request May 30, 2024
@silverbullet233
[Enhancement] add more profile and metrics to measure runtime bloom f…
6682a67 
…ilter size (StarRocks#46360)

Signed-off-by: silverbullet233 <[email protected]>
@silverbullet233 silverbullet233 mentioned this pull request May 30, 2024
Merged
24 tasks
silverbullet233 added a commit to silverbullet233/starrocks that referenced this pull request May 30, 2024
@silverbullet233
[Enhancement] add more profile and metrics to measure runtime bloom f…
278ea71 
…ilter size (StarRocks#46360)

Signed-off-by: silverbullet233 <[email protected]>
silverbullet233 added a commit to silverbullet233/starrocks that referenced this pull request May 30, 2024
@silverbullet233
[Enhancement] add more profile and metrics to measure runtime bloom f…
5157bc4 
…ilter size (StarRocks#46360)

Signed-off-by: silverbullet233 <[email protected]>
silverbullet233 added a commit to silverbullet233/starrocks that referenced this pull request May 30, 2024
@silverbullet233
[Enhancement] add more profile and metrics to measure runtime bloom f…
a5197c1 
…ilter size (StarRocks#46360)

Signed-off-by: silverbullet233 <[email protected]>
silverbullet233 added a commit that referenced this pull request May 30, 2024
@silverbullet233
[Enhancement] add more profile and metrics to measure runtime bloom f…
9e6ac31 
…ilter size(backport #46360) (#46438)

Signed-off-by: silverbullet233 <[email protected]>
silverbullet233 added a commit that referenced this pull request May 31, 2024
@silverbullet233
[Enhancement] add more profile and metrics to measure runtime bloom f…
bc107d6 
…ilter(backport #46360) (#46441)

Signed-off-by: silverbullet233 <[email protected]>
silverbullet233 added a commit that referenced this pull request May 31, 2024
@silverbullet233
[Enhancement] add more profile and metrics to measure runtime bloom f…
1e86d11 
…ilter size (backport #46360) (#46445)

Signed-off-by: silverbullet233 <[email protected]>
silverbullet233 added a commit that referenced this pull request May 31, 2024
@silverbullet233
[Enhancement] add more profile and metrics to measure runtime bloom f…
f35a5ec 
…ilter size(backport #46360) (#46453)

Signed-off-by: silverbullet233 <[email protected]>
silverbullet233 added a commit that referenced this pull request Jun 7, 2024
@silverbullet233
[Enhancement] add more profile and metrics to measure runtime bloom f…
d696eb8 
…ilter size (backport #46360) (#46430)

Signed-off-by: silverbullet233 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@starrocks-cr starrocks-cr starrocks-cr left review comments

@kevincai kevincai kevincai approved these changes

@satanson satanson satanson approved these changes

@ZiheLiu ZiheLiu ZiheLiu approved these changes

@stdpain stdpain stdpain left review comments

Assignees

@silverbullet233 silverbullet233

Labels
2.5-merged 2.5 3.0-merged 3.0 3.1-merged 3.1 3.2-merged 3.2 3.3-merged 3.3
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@silverbullet233 @kevincai @satanson @ZiheLiu @stdpain