Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](auto-partition) Fix auto partition load failure in multi replica #36344

Merged
merged 1 commit into from
Jun 20, 2024
Merged

[fix](auto-partition) Fix auto partition load failure in multi replica #36344

merged 1 commit into from
Jun 20, 2024

Conversation

zclllyybb
Copy link
Contributor

@zclllyybb zclllyybb commented Jun 15, 2024 •
edited
Loading

Proposed changes

Issue Number: close #xxx

One sender may incremental_open tablets_channel many times. but only close one time. so don't count num_senders duplicated for one sender instance.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zclllyybb
Copy link
Contributor Author

run compile

@zclllyybb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39756 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 22238235956ba8c5dad935bc9579d39f807a0af1, data reload: false

------ Round 1 ----------------------------------
q1	17622	4320	4265	4265
q2	2021	191	198	191
q3	10443	1206	1181	1181
q4	10201	839	825	825
q5	7463	2683	2621	2621
q6	222	140	139	139
q7	966	611	592	592
q8	9223	2047	2103	2047
q9	8835	6502	6446	6446
q10	8934	3767	3718	3718
q11	455	230	236	230
q12	441	237	230	230
q13	18820	3001	2983	2983
q14	255	227	224	224
q15	515	503	480	480
q16	524	369	370	369
q17	974	709	639	639
q18	7911	7404	7292	7292
q19	8025	1475	1482	1475
q20	640	310	313	310
q21	4837	3163	3964	3163
q22	395	338	336	336
Total cold run time: 119722 ms
Total hot run time: 39756 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4455	4289	4233	4233
q2	367	259	269	259
q3	3068	2858	2911	2858
q4	1992	1766	1768	1766
q5	5527	5484	5477	5477
q6	222	139	134	134
q7	2248	1813	1836	1813
q8	3245	3416	3411	3411
q9	8666	8832	8770	8770
q10	4149	3733	3787	3733
q11	606	526	494	494
q12	810	628	631	628
q13	17102	3176	3190	3176
q14	323	264	276	264
q15	522	493	479	479
q16	484	436	447	436
q17	1844	1527	1505	1505
q18	8166	7861	7910	7861
q19	1878	1647	1741	1647
q20	3066	1892	1854	1854
q21	9751	4888	4717	4717
q22	654	556	576	556
Total cold run time: 79145 ms
Total hot run time: 56071 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170256 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 22238235956ba8c5dad935bc9579d39f807a0af1, data reload: false

query1	901	373	375	373
query2	6457	2375	2262	2262
query3	6629	206	210	206
query4	18703	17282	17227	17227
query5	3726	471	467	467
query6	239	166	159	159
query7	4590	299	294	294
query8	320	292	276	276
query9	8564	2389	2369	2369
query10	575	315	287	287
query11	10396	9960	9929	9929
query12	122	91	83	83
query13	1651	365	355	355
query14	10316	7660	6276	6276
query15	234	178	186	178
query16	7795	279	274	274
query17	1804	564	530	530
query18	1965	283	288	283
query19	199	149	153	149
query20	93	82	84	82
query21	215	132	126	126
query22	4274	4064	3951	3951
query23	33433	33632	33742	33632
query24	10657	2929	2824	2824
query25	589	386	372	372
query26	713	157	162	157
query27	2339	331	325	325
query28	6013	2090	2101	2090
query29	900	658	610	610
query30	251	161	162	161
query31	964	788	743	743
query32	105	55	54	54
query33	657	281	282	281
query34	899	490	482	482
query35	750	642	623	623
query36	1129	973	988	973
query37	149	78	83	78
query38	2968	2838	2778	2778
query39	929	835	817	817
query40	227	131	130	130
query41	58	56	50	50
query42	106	104	108	104
query43	598	565	549	549
query44	1096	731	727	727
query45	187	158	162	158
query46	1080	733	723	723
query47	1868	1769	1770	1769
query48	373	292	296	292
query49	841	409	418	409
query50	772	396	395	395
query51	6699	6724	6643	6643
query52	104	94	95	94
query53	354	300	299	299
query54	845	449	444	444
query55	74	74	73	73
query56	280	261	253	253
query57	1133	1062	1094	1062
query58	240	260	266	260
query59	3408	3052	2966	2966
query60	298	278	286	278
query61	94	88	89	88
query62	576	431	458	431
query63	325	291	289	289
query64	8549	2249	1815	1815
query65	3185	3160	3150	3150
query66	765	337	336	336
query67	15544	15138	15029	15029
query68	5640	540	550	540
query69	576	475	443	443
query70	1218	1147	1108	1108
query71	441	292	284	284
query72	7472	5791	2894	2894
query73	764	337	335	335
query74	5753	5506	5472	5472
query75	3663	2683	2687	2683
query76	3190	900	913	900
query77	602	312	314	312
query78	10401	9763	9833	9763
query79	5313	520	516	516
query80	1934	502	487	487
query81	586	217	223	217
query82	1502	108	102	102
query83	323	182	163	163
query84	267	92	85	85
query85	1387	286	264	264
query86	473	334	367	334
query87	3228	3024	3066	3024
query88	4426	2380	2349	2349
query89	491	380	378	378
query90	1776	191	190	190
query91	127	97	98	97
query92	61	48	48	48
query93	5619	511	503	503
query94	1186	190	185	185
query95	401	309	324	309
query96	602	263	266	263
query97	3285	3056	3045	3045
query98	211	188	188	188
query99	1326	823	827	823
Total cold run time: 277136 ms
Total hot run time: 170256 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.43 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 22238235956ba8c5dad935bc9579d39f807a0af1, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.04
query3	0.23	0.05	0.04
query4	1.66	0.07	0.08
query5	0.49	0.50	0.50
query6	1.13	0.74	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.49	0.49
query10	0.54	0.53	0.54
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.59	0.58	0.60
query14	0.76	0.78	0.78
query15	0.83	0.81	0.82
query16	0.35	0.37	0.37
query17	0.96	0.97	0.97
query18	0.22	0.25	0.25
query19	1.78	1.70	1.73
query20	0.01	0.03	0.01
query21	15.42	0.67	0.65
query22	4.02	7.49	1.70
query23	18.26	1.34	1.28
query24	2.09	0.24	0.22
query25	0.16	0.09	0.08
query26	0.26	0.18	0.17
query27	0.09	0.07	0.08
query28	13.20	1.01	1.00
query29	12.67	3.31	3.25
query30	0.26	0.07	0.06
query31	2.84	0.40	0.39
query32	3.25	0.49	0.47
query33	2.85	2.92	2.92
query34	17.28	4.48	4.48
query35	4.50	4.49	4.48
query36	0.65	0.46	0.48
query37	0.19	0.15	0.15
query38	0.15	0.14	0.15
query39	0.04	0.03	0.04
query40	0.20	0.15	0.14
query41	0.09	0.05	0.05
query42	0.05	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.15 s
Total hot run time: 30.43 s

@zclllyybb
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39939 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d5db8160436d98087aa1ca9d83a8daaa9416257a, data reload: false

------ Round 1 ----------------------------------
q1	17639	4716	4353	4353
q2	2026	187	187	187
q3	10543	1185	1073	1073
q4	10243	848	844	844
q5	7469	2703	2669	2669
q6	220	138	144	138
q7	958	606	602	602
q8	9225	2086	2112	2086
q9	9123	6540	6489	6489
q10	8988	3766	3768	3766
q11	454	229	230	229
q12	424	235	228	228
q13	17768	2985	2964	2964
q14	273	210	220	210
q15	537	464	474	464
q16	527	375	387	375
q17	977	667	762	667
q18	8192	7601	7377	7377
q19	3763	1403	1415	1403
q20	671	307	330	307
q21	4892	3167	3892	3167
q22	394	343	341	341
Total cold run time: 115306 ms
Total hot run time: 39939 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4481	4237	4278	4237
q2	372	267	269	267
q3	2996	2713	2774	2713
q4	1888	1576	1582	1576
q5	5259	5281	5280	5280
q6	216	126	125	125
q7	2155	1716	1699	1699
q8	3221	3352	3339	3339
q9	8411	8327	8338	8327
q10	3867	3655	3625	3625
q11	577	489	497	489
q12	774	591	574	574
q13	17464	2987	2967	2967
q14	285	268	268	268
q15	517	465	473	465
q16	471	406	405	405
q17	1774	1466	1475	1466
q18	7649	7562	7373	7373
q19	1742	1636	1606	1606
q20	1975	1761	1770	1761
q21	4845	4647	4754	4647
q22	619	529	530	529
Total cold run time: 71558 ms
Total hot run time: 53738 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173764 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d5db8160436d98087aa1ca9d83a8daaa9416257a, data reload: false

query1	938	379	366	366
query2	6475	2477	2277	2277
query3	6660	204	209	204
query4	19422	17447	17447	17447
query5	4120	470	439	439
query6	248	165	156	156
query7	4581	288	291	288
query8	328	292	283	283
query9	8451	2415	2399	2399
query10	589	291	268	268
query11	10708	10194	10207	10194
query12	134	85	86	85
query13	1643	354	364	354
query14	10168	7618	7359	7359
query15	235	191	185	185
query16	7819	265	266	265
query17	1907	536	548	536
query18	1948	272	267	267
query19	200	160	155	155
query20	99	82	83	82
query21	213	129	123	123
query22	4560	4292	4215	4215
query23	33858	33104	32983	32983
query24	11118	2852	2845	2845
query25	623	350	353	350
query26	1167	148	153	148
query27	2434	313	323	313
query28	6887	2059	2045	2045
query29	877	632	608	608
query30	286	177	151	151
query31	960	734	755	734
query32	92	52	56	52
query33	765	287	277	277
query34	966	466	464	464
query35	740	607	605	605
query36	1103	938	935	935
query37	145	69	68	68
query38	2857	2735	2747	2735
query39	840	784	801	784
query40	205	123	123	123
query41	55	52	53	52
query42	116	98	99	98
query43	559	564	558	558
query44	1173	729	734	729
query45	195	166	166	166
query46	1081	759	728	728
query47	1902	1775	1770	1770
query48	375	299	301	299
query49	1114	396	397	396
query50	755	385	401	385
query51	6819	6642	6659	6642
query52	106	91	101	91
query53	354	286	283	283
query54	957	426	439	426
query55	74	71	72	71
query56	273	251	274	251
query57	1156	1077	1046	1046
query58	243	247	237	237
query59	3296	3138	3061	3061
query60	304	265	267	265
query61	124	108	94	94
query62	653	458	464	458
query63	314	285	291	285
query64	8901	2223	1782	1782
query65	3170	3118	3114	3114
query66	824	344	353	344
query67	15278	15186	15224	15186
query68	4542	578	533	533
query69	459	293	316	293
query70	1186	1141	1121	1121
query71	361	270	268	268
query72	7068	5557	5528	5528
query73	731	325	332	325
query74	6017	5517	5531	5517
query75	3341	2618	2674	2618
query76	2327	910	910	910
query77	467	287	290	287
query78	10337	10023	9716	9716
query79	2726	516	502	502
query80	1214	458	451	451
query81	599	222	222	222
query82	738	104	100	100
query83	232	169	171	169
query84	237	84	84	84
query85	1539	276	280	276
query86	516	329	311	311
query87	3294	3124	3075	3075
query88	4106	2392	2417	2392
query89	472	379	380	379
query90	1747	190	187	187
query91	126	98	95	95
query92	59	51	49	49
query93	1686	509	488	488
query94	1167	185	183	183
query95	397	307	318	307
query96	577	268	265	265
query97	3219	3123	3079	3079
query98	213	203	196	196
query99	1332	865	854	854
Total cold run time: 270572 ms
Total hot run time: 173764 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.73 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d5db8160436d98087aa1ca9d83a8daaa9416257a, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.66	0.08	0.08
query5	0.52	0.47	0.51
query6	1.13	0.74	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.54	0.50	0.49
query10	0.54	0.55	0.53
query11	0.15	0.11	0.12
query12	0.14	0.11	0.12
query13	0.59	0.58	0.60
query14	0.77	0.79	0.76
query15	0.83	0.81	0.80
query16	0.38	0.37	0.37
query17	0.95	1.02	0.96
query18	0.21	0.22	0.25
query19	1.79	1.72	1.66
query20	0.01	0.01	0.00
query21	15.40	0.66	0.65
query22	4.43	7.85	1.30
query23	18.32	1.31	1.22
query24	2.00	0.22	0.24
query25	0.15	0.08	0.09
query26	0.26	0.17	0.18
query27	0.08	0.08	0.08
query28	13.19	1.02	0.98
query29	12.60	3.27	3.26
query30	0.27	0.07	0.06
query31	2.86	0.39	0.39
query32	3.27	0.47	0.47
query33	2.91	2.86	2.91
query34	17.04	4.43	4.44
query35	4.54	4.46	4.51
query36	0.65	0.48	0.45
query37	0.19	0.16	0.15
query38	0.16	0.14	0.14
query39	0.04	0.04	0.03
query40	0.18	0.15	0.15
query41	0.10	0.05	0.05
query42	0.06	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.37 s
Total hot run time: 29.73 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.46% (9001/24684)
Line Coverage: 28.01% (73796/263467)
Region Coverage: 27.49% (38324/139402)
Branch Coverage: 24.20% (19543/80740)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d5db8160436d98087aa1ca9d83a8daaa9416257a_d5db8160436d98087aa1ca9d83a8daaa9416257a/report/index.html

kaijchen
kaijchen approved these changes Jun 20, 2024
View reviewed changes
Copy link
Contributor

@kaijchen kaijchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

HappenLee
HappenLee approved these changes Jun 20, 2024
View reviewed changes
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 20, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@BiteTheDDDDt BiteTheDDDDt merged commit 99a3d89 into apache:master Jun 20, 2024
26 of 30 checks passed
zclllyybb added a commit to zclllyybb/doris that referenced this pull request Jun 20, 2024
@zclllyybb
[fix](auto-partition) Fix auto partition load failure in multi replica (
fc81e53 
apache#36344)

One sender may incremental_open `tablets_channel` many times. but only
close one time. so don't count num_senders duplicated for one sender
instance.
@zclllyybb zclllyybb mentioned this pull request Jun 20, 2024
Merged
morningman pushed a commit that referenced this pull request Jun 20, 2024
@zclllyybb @kaijchen
[branch-2.1](auto-partition) Fix auto partition load failure in multi…
bd47d5a 
… replica (#36586)

this pr
1. picked #35630, which was reverted #36098 before.
2. picked #36344 from master

these two pr fixed existing bug about auto partition load.

---------

Co-authored-by: Kaijie Chen <[email protected]>
iszhangpch pushed a commit to iszhangpch/doris-p that referenced this pull request Jun 21, 2024
@zclllyybb
[fix](auto-partition) Fix auto partition load failure in multi replica (
e2dc339 
apache#36344)

## Proposed changes
One sender may incremental_open `tablets_channel` many times. but only
close one time. so don't count num_senders duplicated for one sender
instance.
dataroaring pushed a commit that referenced this pull request Jun 21, 2024
@zclllyybb @dataroaring
[fix](auto-partition) Fix auto partition load failure in multi replica (
d7ffe4f 
#36344)

## Proposed changes
One sender may incremental_open `tablets_channel` many times. but only
close one time. so don't count num_senders duplicated for one sender
instance.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@HappenLee HappenLee HappenLee approved these changes

@kaijchen kaijchen kaijchen approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.4-merged dev/3.0.0-merged meta-change reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@zclllyybb @doris-robot @kaijchen @HappenLee @morningman @BiteTheDDDDt @dataroaring