Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improment]log more detail when query hits policy #44685

Merged
merged 1 commit into from
Nov 28, 2024

Conversation

wangbo
Copy link
Contributor

@wangbo wangbo commented Nov 27, 2024

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
Add resource usage value when query hits policy.

  1. create a policy as follow.
mysql [information_schema]>select * from workload_policy;
+-------+--------------+--------------------------------------------+--------------+----------+---------+---------+----------------+
| ID    | NAME         | CONDITION                                  | ACTION       | PRIORITY | ENABLED | VERSION | WORKLOAD_GROUP |
+-------+--------------+--------------------------------------------+--------------+----------+---------+---------+----------------+
| 12137 | cancel_query | query_time > 0;be_scan_bytes > 10000000000 | cancel_query |        0 |       1 |       0 |                |
+-------+--------------+--------------------------------------------+--------------+----------+---------+---------+----------------+
1 row in set (0.02 sec)
  1. when query hits the policy, error message could contain real scanbytes and query time.
mysql [information_schema]>select * from hits.hits;
ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR]query d003973bb06e416e-a40ca810ddc82519 cancelled by workload policy: cancel_query, id:12137, query_time:26650(0), scan_bytes:10084786176(10000000000)

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@wangbo
Copy link
Contributor Author

wangbo commented Nov 27, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.33% (9975/26024)
Line Coverage: 29.43% (83488/283639)
Region Coverage: 28.59% (42988/150365)
Branch Coverage: 25.19% (21833/86674)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0a60b150b537459d713adbe9cbda79860c108dfd_0a60b150b537459d713adbe9cbda79860c108dfd/report/index.html

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 27, 2024
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 39847 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0a60b150b537459d713adbe9cbda79860c108dfd, data reload: false

------ Round 1 ----------------------------------
q1	17578	7455	7240	7240
q2	2040	173	167	167
q3	10693	1102	1143	1102
q4	10585	770	692	692
q5	7630	2734	2737	2734
q6	238	158	150	150
q7	996	632	613	613
q8	9242	1815	1861	1815
q9	6505	6333	6396	6333
q10	7030	2292	2344	2292
q11	462	254	259	254
q12	424	222	223	222
q13	17777	3006	2989	2989
q14	246	208	217	208
q15	590	523	511	511
q16	663	594	598	594
q17	981	500	582	500
q18	7314	6769	6760	6760
q19	1334	999	994	994
q20	465	184	176	176
q21	4187	3407	3194	3194
q22	392	319	307	307
Total cold run time: 107372 ms
Total hot run time: 39847 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7296	7276	7343	7276
q2	325	227	223	223
q3	2907	2800	2945	2800
q4	2149	1840	1894	1840
q5	5655	5665	5687	5665
q6	223	146	145	145
q7	2264	1827	1831	1827
q8	3415	3577	3491	3491
q9	8868	8903	8897	8897
q10	3597	3543	3528	3528
q11	604	508	520	508
q12	850	596	599	596
q13	9550	3390	3254	3254
q14	311	279	275	275
q15	566	512	525	512
q16	694	647	654	647
q17	1831	1614	1609	1609
q18	8407	7738	7673	7673
q19	2857	1604	1581	1581
q20	2139	1935	1853	1853
q21	5686	5366	5370	5366
q22	638	560	563	560
Total cold run time: 70832 ms
Total hot run time: 60126 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196751 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0a60b150b537459d713adbe9cbda79860c108dfd, data reload: false

query1	1315	946	971	946
query2	6241	2165	2080	2080
query3	11019	4180	4327	4180
query4	67935	28676	23493	23493
query5	4986	460	445	445
query6	407	186	197	186
query7	5563	298	287	287
query8	313	223	241	223
query9	8815	2711	2697	2697
query10	460	251	258	251
query11	17184	15375	15972	15375
query12	155	107	100	100
query13	1516	414	436	414
query14	9729	7194	6726	6726
query15	215	180	188	180
query16	7172	474	471	471
query17	1287	587	606	587
query18	1928	312	306	306
query19	205	155	157	155
query20	117	111	109	109
query21	209	105	115	105
query22	4711	4548	4463	4463
query23	34645	34315	34256	34256
query24	5672	2512	2533	2512
query25	507	403	395	395
query26	637	152	147	147
query27	1930	291	288	288
query28	4334	2446	2467	2446
query29	695	426	468	426
query30	212	150	145	145
query31	1004	850	842	842
query32	69	53	62	53
query33	430	289	282	282
query34	949	513	501	501
query35	887	754	761	754
query36	1082	958	966	958
query37	125	77	81	77
query38	4461	4387	4491	4387
query39	1538	1481	1489	1481
query40	213	102	99	99
query41	45	43	44	43
query42	114	101	103	101
query43	547	511	507	507
query44	1177	835	824	824
query45	186	165	187	165
query46	1135	696	724	696
query47	2020	1947	1921	1921
query48	421	331	325	325
query49	718	399	376	376
query50	845	400	393	393
query51	7424	7225	7171	7171
query52	98	89	88	88
query53	248	180	180	180
query54	517	399	396	396
query55	81	76	73	73
query56	269	231	234	231
query57	1311	1179	1161	1161
query58	220	213	221	213
query59	3197	3008	3079	3008
query60	274	247	242	242
query61	107	109	111	109
query62	791	677	672	672
query63	209	197	189	189
query64	1408	667	646	646
query65	3254	3256	3260	3256
query66	713	300	309	300
query67	15858	15715	15557	15557
query68	3867	581	552	552
query69	418	253	252	252
query70	1222	1147	1112	1112
query71	354	255	242	242
query72	6538	4050	4003	4003
query73	750	367	376	367
query74	10351	9119	9075	9075
query75	3393	2671	2661	2661
query76	1826	1123	1031	1031
query77	476	276	276	276
query78	10498	9413	9326	9326
query79	1488	600	584	584
query80	877	446	436	436
query81	509	238	230	230
query82	1269	118	117	117
query83	270	151	155	151
query84	275	73	72	72
query85	884	305	374	305
query86	343	301	308	301
query87	4601	4647	4627	4627
query88	3580	2227	2221	2221
query89	417	291	291	291
query90	2049	188	191	188
query91	133	100	103	100
query92	67	49	51	49
query93	1877	525	543	525
query94	821	303	302	302
query95	346	249	252	249
query96	616	277	277	277
query97	2811	2647	2668	2647
query98	213	200	208	200
query99	1607	1352	1311	1311
Total cold run time: 318805 ms
Total hot run time: 196751 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.6 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0a60b150b537459d713adbe9cbda79860c108dfd, data reload: false

query1	0.04	0.03	0.02
query2	0.08	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.11	0.10
query5	0.41	0.43	0.42
query6	1.18	0.67	0.65
query7	0.02	0.01	0.01
query8	0.04	0.04	0.02
query9	0.59	0.50	0.50
query10	0.54	0.55	0.56
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.60	0.61	0.61
query14	2.74	2.85	2.84
query15	0.90	0.82	0.82
query16	0.37	0.37	0.39
query17	1.01	1.00	1.04
query18	0.22	0.21	0.20
query19	1.99	1.84	2.04
query20	0.01	0.01	0.02
query21	15.35	0.59	0.57
query22	2.45	2.50	2.21
query23	17.04	0.91	0.83
query24	3.35	0.59	0.46
query25	0.29	0.18	0.10
query26	0.44	0.14	0.12
query27	0.04	0.05	0.04
query28	11.63	1.11	1.07
query29	12.54	3.20	3.21
query30	0.25	0.06	0.06
query31	2.86	0.39	0.37
query32	3.28	0.46	0.48
query33	3.06	3.05	3.09
query34	16.88	4.52	4.49
query35	4.53	4.52	4.54
query36	0.66	0.50	0.47
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.04	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 108.01 s
Total hot run time: 32.6 s

Copy link
Contributor

@xinyiZzz xinyiZzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wangbo wangbo merged commit 9b9bd9e into apache:master Nov 28, 2024
23 of 30 checks passed
wangbo added a commit to wangbo/incubator-doris that referenced this pull request Dec 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged reviewed workload-schedule
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants