Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Unify load profile through analyze profile & support session/table profile collect granularity #47548

Merged
merged 1 commit into from
Sep 6, 2024

Conversation

meegoo
Copy link
Contributor

@meegoo meegoo commented Jun 26, 2024

Why I'm doing:

  • support table granularity for profile collect
  • support profile collect when stream/routine load failure
  • unify profile analyze interface

What I'm doing:

mysql> create table t(k int, v int);
Query OK, 0 rows affected (0.11 sec)

mysql> alter table t set('enable_load_profile'='true');
Query OK, 0 rows affected (0.02 sec)

mysql> analyze profile from '1a462e3a-dcc7-94f4-e7ad-d272b485439f';
+-----------------------------------------------------------+
| Explain String                                            |
+-----------------------------------------------------------+
| Load:                                                     |
|   Summary:                                                |
|      - Query ID: 1a462e3a-dcc7-94f4-e7ad-d272b485439f     |
|      - Start Time: 2024-06-27 01:14:35                    |
|      - End Time: 2024-06-27 01:14:47                      |
|      - Query Type: Load                                   |
|      - Load Type: STREAM_LOAD                             |
|      - Query State: Finished                              |
|      - StarRocks Version: unify_load_profile-a79910b      |
|      - Sql Statement                                      |
|      - Default Db: d                                      |
|      - Total: 11s515ms                                    |
|   Execution:                                              |
|     Fragment 0:                                           |
|        - Address: 172.26.95.141:51686                     |
|        - InstanceId: 1a462e3a-dcc7-94f4-e7ad-d272b48543a0 |
|        - TxnID: 8009                                      |
|        - ReplicatedStorage: true                          |
|        - AutomaticPartition: false                        |
|        - AutomaticBucketSize: 4294967296                  |
|        - InstanceAllocatedMemoryUsage: 1.370 GB           |
|        - InstanceDeallocatedMemoryUsage: 860.124 MB       |
|        - InstancePeakMemoryUsage: 15.092 MB               |
|        - MemoryLimit: -1.000 B                            |
|        - RowsProduced: 55.325M (55324672)                 |
|          - AllocAutoIncrementTime: 1.286ms                |
|          - BytesRead: 0.000 B                             |
|          - CloseWaitTime: 0ns                             |
|          - IOTaskExecTime: 0ns                            |
|          - IOTaskWaitTime: 0ns                            |
|          - IndexNum: 1                                    |
|          - NumDiskAccess: 0                               |
|          - OpenTime: 2.942ms                              |
|          - PeakMemoryUsage: 0.000 B                       |
|          - PrepareDataTime: 1s386ms                       |
|            - ConvertChunkTime: 0ns                        |
|            - ValidateDataTime: 0ns                        |
|          - RowsFiltered: 0                                |
|          - RowsRead: 0                                    |
|          - RowsReturned: 55.325M (55324672)               |
|          - RowsReturnedRate: 69.136M (69135610) /sec      |
|          - RpcClientSideTime: 9s521ms                     |
|          - RpcServerSideTime: 0ns                         |
|          - RpcServerWaitFlushTime: 0ns                    |
|          - ScanTime: 8s815ms                              |
|          - ScannerQueueCounter: 199                       |
|          - ScannerQueueTime: 11s126ms                     |
|          - ScannerThreadsInvoluntaryContextSwitches: 0    |
|          - ScannerThreadsTotalWallClockTime: 0ns          |
|            - MaterializeTupleTime(*): 0ns                 |
|            - ScannerThreadsSysTime: 0ns                   |
|            - ScannerThreadsUserTime: 0ns                  |
|          - ScannerThreadsVoluntaryContextSwitches: 0      |
|          - SendDataTime: 8s728ms                          |
|            - PackChunkTime: 1s634ms                       |
|            - SendRpcTime: 0ns                             |
|              - CompressTime: 0ns                          |
|              - SerializeChunkTime: 0ns                    |
|            - WaitResponseTime: 2s238ms                    |
|          - TotalRawReadTime(*): 0ns                       |
|          - TotalReadThroughput: 0.000 B/sec               |
|       DataSource:                                         |
|          - DataSourceType: FileDataSource                 |
|          - FileScanner:                                   |
|            - CastChunkTime: 0ns                           |
|            - CreateChunkTime: 0ns                         |
|            - FileReadCount: 0                             |
|            - FileReadTime: 0ns                            |
|            - FillTime: 0ns                                |
|            - MaterializeTime: 0ns                         |
|            - ReadTime: 0ns                                |
|          - ScannerTotalTime: 0ns                          |
+-----------------------------------------------------------+
72 rows in set (0.02 sec)

Fixes #47069

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function
  • This is a backport pr

Bugfix cherry-pick branch check:

  • I have checked the version labels which the pr will be auto-backported to the target branch
    • 3.3
    • 3.2
    • 3.1
    • 3.0
    • 2.5

@meegoo meegoo requested review from a team as code owners June 26, 2024 17:42
@github-actions github-actions bot added the documentation Improvements or additions to documentation label Jun 26, 2024
@wanpengfei-git wanpengfei-git requested review from a team June 26, 2024 17:43
@mergify mergify bot assigned meegoo Jun 26, 2024
Copy link

sonarcloud bot commented Jun 26, 2024

@wyb wyb requested review from banmoy and jaogoy June 28, 2024 02:30
@meegoo meegoo force-pushed the unify_load_profile branch 6 times, most recently from be0d1ed to 92ce983 Compare August 8, 2024 02:27
@meegoo meegoo force-pushed the unify_load_profile branch 2 times, most recently from c959e51 to 3de0aac Compare August 14, 2024 17:14
@meegoo meegoo force-pushed the unify_load_profile branch 6 times, most recently from 446148c to c76f6ac Compare September 5, 2024 04:12
@meegoo meegoo enabled auto-merge (squash) September 5, 2024 04:12
…n/table profile collect granularity

Signed-off-by: meegoo <[email protected]>
Copy link

sonarcloud bot commented Sep 5, 2024

Copy link

github-actions bot commented Sep 5, 2024

[BE Incremental Coverage Report]

pass : 0 / 0 (0%)

Copy link

github-actions bot commented Sep 5, 2024

[Java-Extensions Incremental Coverage Report]

pass : 0 / 0 (0%)

Copy link

github-actions bot commented Sep 5, 2024

[FE Incremental Coverage Report]

pass : 96 / 112 (85.71%)

file detail

path covered_line new_line coverage not_covered_line_detail
🔵 com/starrocks/qe/StmtExecutor.java 0 2 00.00% [1337, 1338]
🔵 com/starrocks/alter/SchemaChangeHandler.java 3 4 75.00% [2106]
🔵 com/starrocks/planner/StreamLoadPlanner.java 7 9 77.78% [287, 289]
🔵 com/starrocks/load/streamload/StreamLoadTask.java 15 19 78.95% [841, 1078, 1136, 1515]
🔵 com/starrocks/catalog/OlapTable.java 14 17 82.35% [2561, 2566, 3570]
🔵 com/starrocks/server/LocalMetastore.java 12 14 85.71% [4240, 4241]
🔵 com/starrocks/service/FrontendServiceImpl.java 7 8 87.50% [1656]
🔵 com/starrocks/sql/ExplainAnalyzer.java 7 8 87.50% [99]
🔵 com/starrocks/server/OlapTableFactory.java 2 2 100.00% []
🔵 com/starrocks/qe/scheduler/dag/FragmentInstanceExecState.java 2 2 100.00% []
🔵 com/starrocks/load/loadv2/LoadJob.java 7 7 100.00% []
🔵 com/starrocks/common/Config.java 2 2 100.00% []
🔵 com/starrocks/sql/analyzer/AlterTableClauseAnalyzer.java 2 2 100.00% []
🔵 com/starrocks/catalog/TableProperty.java 8 8 100.00% []
🔵 com/starrocks/alter/AlterJobExecutor.java 2 2 100.00% []
🔵 com/starrocks/common/util/PropertyAnalyzer.java 4 4 100.00% []
🔵 com/starrocks/persist/EditLog.java 2 2 100.00% []

@meegoo meegoo disabled auto-merge September 5, 2024 17:25
@meegoo meegoo enabled auto-merge (squash) September 5, 2024 17:25
@meegoo meegoo merged commit 2a31794 into StarRocks:main Sep 6, 2024
50 of 53 checks passed
@wanpengfei-git wanpengfei-git mentioned this pull request Sep 6, 2024
1 task
HangyuanLiu pushed a commit to HangyuanLiu/starrocks that referenced this pull request Sep 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation META-REVIEW PROTO-REVIEW version:3.4
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Unify Load Profile
6 participants