Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](cluster key) support segcompaction #45599

Merged
merged 2 commits into from
Dec 27, 2024

Conversation

mymeiyi
Copy link
Contributor

@mymeiyi mymeiyi commented Dec 18, 2024

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Copy link
Contributor

sh-checker report

To get the full details, please check in the job output.

shellcheck errors

'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.



shfmt errors
'shfmt ' found no issues.

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

Copy link
Contributor

sh-checker report

To get the full details, please check in the job output.

shellcheck errors

'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.



shfmt errors
'shfmt ' found no issues.

@mymeiyi
Copy link
Contributor Author

mymeiyi commented Dec 18, 2024

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@mymeiyi mymeiyi closed this Dec 18, 2024
@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.88% (10133/26065)
Line Coverage: 29.78% (85150/285914)
Region Coverage: 28.83% (43680/151493)
Branch Coverage: 25.38% (22193/87458)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8b3cc236250ac901ae9a86de0d27cf76051473e8_8b3cc236250ac901ae9a86de0d27cf76051473e8/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 40442 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8b3cc236250ac901ae9a86de0d27cf76051473e8, data reload: false

------ Round 1 ----------------------------------
q1	17609	7411	7287	7287
q2	2046	171	163	163
q3	10578	1072	1157	1072
q4	10730	777	743	743
q5	8207	2726	2702	2702
q6	239	145	142	142
q7	994	625	606	606
q8	9292	1843	1876	1843
q9	6655	6425	6407	6407
q10	7034	2283	2308	2283
q11	466	263	272	263
q12	443	224	228	224
q13	17858	3000	3036	3000
q14	238	211	205	205
q15	560	518	495	495
q16	695	596	584	584
q17	967	557	581	557
q18	7296	7072	7432	7072
q19	2624	1075	1138	1075
q20	480	205	192	192
q21	4163	3354	3200	3200
q22	394	331	327	327
Total cold run time: 109568 ms
Total hot run time: 40442 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7344	7322	7294	7294
q2	330	231	229	229
q3	3090	2978	2977	2977
q4	2135	1927	1822	1822
q5	5663	5705	5648	5648
q6	224	146	149	146
q7	2222	1833	1817	1817
q8	3337	3520	3480	3480
q9	8829	8928	8917	8917
q10	3643	3530	3490	3490
q11	611	502	498	498
q12	826	625	619	619
q13	16499	3086	3123	3086
q14	322	288	273	273
q15	569	513	511	511
q16	691	665	653	653
q17	1883	1637	1596	1596
q18	8356	7747	7560	7560
q19	2125	1578	1638	1578
q20	2098	1873	1906	1873
q21	5627	5497	5378	5378
q22	612	581	600	581
Total cold run time: 77036 ms
Total hot run time: 60026 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195556 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8b3cc236250ac901ae9a86de0d27cf76051473e8, data reload: false

query1	1280	991	940	940
query2	6226	2266	2168	2168
query3	11026	4777	4574	4574
query4	33279	23420	23291	23291
query5	4032	490	443	443
query6	279	196	180	180
query7	3994	294	301	294
query8	294	229	223	223
query9	9518	2692	2714	2692
query10	482	269	244	244
query11	18127	15243	15174	15174
query12	148	103	98	98
query13	1550	411	401	401
query14	10110	7242	6994	6994
query15	247	194	200	194
query16	8304	457	450	450
query17	1533	599	578	578
query18	2183	314	290	290
query19	325	160	157	157
query20	125	109	108	108
query21	208	108	113	108
query22	4880	4673	4396	4396
query23	34411	33484	33439	33439
query24	10448	2592	2494	2494
query25	640	405	389	389
query26	1196	151	154	151
query27	2402	328	338	328
query28	7482	2470	2480	2470
query29	875	430	455	430
query30	266	155	150	150
query31	1065	853	843	843
query32	93	57	54	54
query33	786	299	290	290
query34	975	525	512	512
query35	898	762	760	760
query36	1132	965	981	965
query37	135	77	76	76
query38	4407	4084	4225	4084
query39	1500	1456	1463	1456
query40	213	106	101	101
query41	45	42	44	42
query42	109	103	102	102
query43	536	477	493	477
query44	1313	846	849	846
query45	198	177	166	166
query46	1190	727	706	706
query47	1993	1935	1928	1928
query48	432	332	328	328
query49	928	407	407	407
query50	879	406	404	404
query51	7386	7168	7221	7168
query52	113	94	99	94
query53	259	189	184	184
query54	1117	422	425	422
query55	87	82	86	82
query56	275	269	273	269
query57	1263	1169	1162	1162
query58	251	247	227	227
query59	3311	3236	3071	3071
query60	282	239	244	239
query61	107	111	116	111
query62	928	688	683	683
query63	231	202	195	195
query64	4001	660	642	642
query65	3247	3196	3167	3167
query66	826	298	301	298
query67	15929	15478	15550	15478
query68	4934	557	546	546
query69	483	252	259	252
query70	1175	1121	1140	1121
query71	476	250	248	248
query72	7033	4218	4131	4131
query73	788	357	367	357
query74	10236	8828	8855	8828
query75	3683	2642	2641	2641
query76	3731	1062	1118	1062
query77	564	270	269	269
query78	10267	9401	9585	9401
query79	2225	593	602	593
query80	770	523	434	434
query81	516	243	240	240
query82	1072	126	115	115
query83	192	142	140	140
query84	230	69	67	67
query85	1571	300	298	298
query86	473	309	302	302
query87	4498	4565	4283	4283
query88	3719	2215	2191	2191
query89	425	292	282	282
query90	2044	191	192	191
query91	138	103	101	101
query92	63	51	53	51
query93	2285	537	538	537
query94	912	282	277	277
query95	350	253	250	250
query96	610	282	283	282
query97	2881	2725	2677	2677
query98	222	189	196	189
query99	1561	1316	1313	1313
Total cold run time: 304437 ms
Total hot run time: 195556 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.91 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8b3cc236250ac901ae9a86de0d27cf76051473e8, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.05	0.03
query3	0.24	0.07	0.07
query4	1.62	0.11	0.11
query5	0.42	0.40	0.43
query6	1.17	0.66	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.50	0.51
query10	0.56	0.58	0.57
query11	0.14	0.10	0.10
query12	0.13	0.11	0.11
query13	0.60	0.61	0.60
query14	2.86	2.76	2.74
query15	0.89	0.82	0.83
query16	0.39	0.39	0.39
query17	1.06	1.07	1.04
query18	0.22	0.21	0.21
query19	1.97	1.88	2.02
query20	0.01	0.01	0.01
query21	15.36	0.59	0.59
query22	2.52	2.64	2.75
query23	16.83	1.16	0.90
query24	3.39	2.05	1.25
query25	0.13	0.20	0.09
query26	0.58	0.14	0.14
query27	0.05	0.05	0.05
query28	9.66	1.09	1.06
query29	12.57	3.28	3.27
query30	0.26	0.06	0.06
query31	2.86	0.38	0.39
query32	3.26	0.46	0.46
query33	3.04	3.02	3.20
query34	17.11	4.47	4.54
query35	4.44	4.44	4.54
query36	0.66	0.47	0.51
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.03	0.02
query40	0.16	0.13	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.23 s
Total hot run time: 33.91 s

@mymeiyi mymeiyi reopened this Dec 19, 2024
Copy link
Contributor

sh-checker report

To get the full details, please check in the job output.

shellcheck errors

'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.



shfmt errors
'shfmt ' found no issues.

Copy link
Contributor

sh-checker report

To get the full details, please check in the job output.

shellcheck errors

'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.



shfmt errors
'shfmt ' found no issues.

Copy link
Contributor

sh-checker report

To get the full details, please check in the job output.

shellcheck errors

'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.



shfmt errors
'shfmt ' found no issues.

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@mymeiyi
Copy link
Contributor Author

mymeiyi commented Dec 19, 2024

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39674 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0dfe4d9ff08b4114bbdc209208090fd1a5f4605b, data reload: false

------ Round 1 ----------------------------------
q1	17599	7702	7306	7306
q2	2049	173	163	163
q3	10618	1084	1152	1084
q4	10562	725	749	725
q5	7626	2722	2662	2662
q6	242	149	147	147
q7	986	635	595	595
q8	9240	1894	1930	1894
q9	6671	6406	6337	6337
q10	6935	2298	2298	2298
q11	485	279	270	270
q12	427	219	222	219
q13	17762	3053	2979	2979
q14	246	204	205	204
q15	568	498	495	495
q16	652	608	593	593
q17	992	496	514	496
q18	7319	6586	6629	6586
q19	1346	994	884	884
q20	494	189	182	182
q21	4030	3381	3248	3248
q22	373	324	307	307
Total cold run time: 107222 ms
Total hot run time: 39674 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7227	7243	7234	7234
q2	328	235	239	235
q3	2919	2815	2953	2815
q4	2096	1883	1838	1838
q5	5698	5667	5663	5663
q6	230	145	146	145
q7	2248	1804	1805	1804
q8	3451	3548	3522	3522
q9	8826	9099	9018	9018
q10	3603	3572	3580	3572
q11	623	506	512	506
q12	836	579	610	579
q13	12013	3216	3188	3188
q14	320	269	277	269
q15	557	520	506	506
q16	688	653	651	651
q17	1898	1623	1625	1623
q18	8383	7839	7729	7729
q19	3303	1466	1516	1466
q20	2090	1947	1861	1861
q21	5526	5540	5516	5516
q22	651	590	589	589
Total cold run time: 73514 ms
Total hot run time: 60329 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195432 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0dfe4d9ff08b4114bbdc209208090fd1a5f4605b, data reload: false

query1	1289	944	934	934
query2	6245	2351	2254	2254
query3	11100	4807	4930	4807
query4	32884	23281	23387	23281
query5	4231	475	470	470
query6	306	196	192	192
query7	4004	302	313	302
query8	301	244	248	244
query9	9340	2631	2632	2631
query10	464	236	238	236
query11	17922	15075	15161	15075
query12	152	105	101	101
query13	1570	412	393	393
query14	8943	6623	7051	6623
query15	275	201	190	190
query16	8204	428	470	428
query17	1600	573	567	567
query18	2106	318	339	318
query19	226	166	169	166
query20	119	120	130	120
query21	220	116	107	107
query22	4933	4688	4602	4602
query23	35347	33783	33685	33685
query24	11693	2414	2534	2414
query25	666	392	397	392
query26	1816	156	146	146
query27	2981	334	331	331
query28	7831	2463	2455	2455
query29	1040	416	424	416
query30	227	149	145	145
query31	1073	831	834	831
query32	97	58	55	55
query33	786	291	304	291
query34	968	521	537	521
query35	882	767	745	745
query36	1113	961	985	961
query37	182	72	78	72
query38	4330	4106	4303	4106
query39	1539	1501	1510	1501
query40	279	104	105	104
query41	50	49	47	47
query42	125	107	103	103
query43	551	514	510	510
query44	1282	847	803	803
query45	197	168	162	162
query46	1184	702	721	702
query47	2031	1986	1945	1945
query48	431	330	318	318
query49	1105	388	386	386
query50	837	409	392	392
query51	7325	7257	7148	7148
query52	107	90	94	90
query53	268	185	179	179
query54	1019	409	408	408
query55	87	79	81	79
query56	261	237	233	233
query57	1348	1161	1172	1161
query58	242	230	227	227
query59	3399	3253	3075	3075
query60	278	235	239	235
query61	105	111	111	111
query62	855	670	666	666
query63	225	182	194	182
query64	4873	678	693	678
query65	3246	3198	3216	3198
query66	1019	295	291	291
query67	15911	15543	15480	15480
query68	5379	537	540	537
query69	450	241	252	241
query70	1185	1146	1156	1146
query71	444	250	244	244
query72	7037	4080	4084	4080
query73	778	356	356	356
query74	9980	8771	8759	8759
query75	3714	2572	2611	2572
query76	3631	1049	1106	1049
query77	643	286	281	281
query78	10206	9425	9335	9335
query79	1865	614	598	598
query80	917	428	425	425
query81	536	247	224	224
query82	1096	118	123	118
query83	263	139	203	139
query84	230	69	72	69
query85	1530	337	302	302
query86	473	305	293	293
query87	4610	4365	4424	4365
query88	3767	2162	2138	2138
query89	428	290	299	290
query90	1983	240	187	187
query91	143	105	105	105
query92	66	49	59	49
query93	2600	537	536	536
query94	743	286	292	286
query95	354	251	254	251
query96	619	275	276	275
query97	2841	2668	2662	2662
query98	225	203	196	196
query99	1590	1322	1303	1303
Total cold run time: 307971 ms
Total hot run time: 195432 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.84 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0dfe4d9ff08b4114bbdc209208090fd1a5f4605b, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.03
query3	0.23	0.07	0.07
query4	1.62	0.11	0.11
query5	0.42	0.40	0.43
query6	1.14	0.65	0.65
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.50	0.51
query10	0.54	0.57	0.56
query11	0.15	0.10	0.11
query12	0.14	0.11	0.10
query13	0.60	0.61	0.60
query14	2.73	2.85	2.90
query15	0.89	0.82	0.84
query16	0.39	0.38	0.38
query17	1.06	1.06	1.02
query18	0.23	0.21	0.21
query19	1.97	1.76	2.01
query20	0.01	0.01	0.02
query21	15.36	0.60	0.61
query22	2.11	2.17	2.11
query23	17.16	0.96	0.87
query24	3.05	1.70	0.78
query25	0.25	0.08	0.11
query26	0.55	0.13	0.14
query27	0.03	0.05	0.05
query28	10.21	1.09	1.08
query29	12.58	3.24	3.26
query30	0.25	0.07	0.07
query31	2.83	0.41	0.39
query32	3.25	0.47	0.46
query33	3.17	3.10	3.07
query34	17.18	4.44	4.54
query35	4.53	4.45	4.52
query36	0.67	0.47	0.47
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.51 s
Total hot run time: 32.84 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.88% (10134/26066)
Line Coverage: 29.78% (85177/285990)
Region Coverage: 28.84% (43695/151529)
Branch Coverage: 25.37% (22199/87484)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0dfe4d9ff08b4114bbdc209208090fd1a5f4605b_0dfe4d9ff08b4114bbdc209208090fd1a5f4605b/report/index.html

@mymeiyi
Copy link
Contributor Author

mymeiyi commented Dec 19, 2024

run p0

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 23, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhannngchen zhannngchen merged commit fe0724c into apache:master Dec 27, 2024
26 of 30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants