Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve](restore) Split creating replica task by table id #42251

Closed
wants to merge 3 commits into from

Conversation

w41ter
Copy link
Contributor

@w41ter w41ter commented Oct 22, 2024

No description provided.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@w41ter
Copy link
Contributor Author

w41ter commented Oct 22, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49113 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 301603b1dd29138f472ffb02eaaaa84447060540, data reload: false

------ Round 1 ----------------------------------
q1	17955	4478	4389	4389
q2	2035	161	152	152
q3	10252	1889	1933	1889
q4	10099	1236	1314	1236
q5	8471	3889	3974	3889
q6	226	150	124	124
q7	2024	1606	1579	1579
q8	9517	2717	2711	2711
q9	13174	9945	9767	9767
q10	8613	3489	3509	3489
q11	421	253	259	253
q12	479	309	304	304
q13	18384	3973	3980	3973
q14	358	323	321	321
q15	516	474	458	458
q16	556	452	462	452
q17	1130	1019	955	955
q18	7169	6870	6950	6870
q19	1718	1553	1552	1552
q20	512	322	333	322
q21	4410	4136	4044	4044
q22	495	391	384	384
Total cold run time: 118514 ms
Total hot run time: 49113 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4325	4329	4364	4329
q2	322	228	231	228
q3	4174	4133	4158	4133
q4	2762	2737	2749	2737
q5	7106	7129	7044	7044
q6	241	121	124	121
q7	3267	2856	2896	2856
q8	4365	4508	4472	4472
q9	13779	13523	13622	13523
q10	4248	4251	4240	4240
q11	758	686	691	686
q12	1033	834	838	834
q13	6913	3724	3737	3724
q14	441	419	425	419
q15	510	459	462	459
q16	645	600	606	600
q17	3868	3907	3828	3828
q18	8880	8852	8926	8852
q19	1729	1650	1662	1650
q20	2353	2101	2144	2101
q21	8497	8499	8402	8402
q22	1029	985	938	938
Total cold run time: 81245 ms
Total hot run time: 76176 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 212598 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 301603b1dd29138f472ffb02eaaaa84447060540, data reload: false

query1	931	396	409	396
query2	6548	2344	2150	2150
query3	6924	201	200	200
query4	23519	21614	21746	21614
query5	19728	6524	6449	6449
query6	277	221	231	221
query7	4324	306	324	306
query8	259	238	263	238
query9	3101	2649	2590	2590
query10	456	302	292	292
query11	16082	14949	15373	14949
query12	130	76	73	73
query13	1034	457	440	440
query14	16553	13538	13394	13394
query15	386	226	237	226
query16	6485	290	264	264
query17	1756	915	896	896
query18	900	329	307	307
query19	211	155	142	142
query20	97	98	100	98
query21	187	100	92	92
query22	5243	5064	4990	4990
query23	34466	33590	33859	33590
query24	7977	6377	6381	6377
query25	535	430	422	422
query26	1276	164	160	160
query27	2485	294	295	294
query28	6129	2228	2209	2209
query29	2966	2696	2640	2640
query30	243	173	168	168
query31	946	735	736	735
query32	70	61	59	59
query33	453	264	253	253
query34	864	491	464	464
query35	1110	909	900	900
query36	1371	1290	1167	1167
query37	171	66	65	65
query38	3035	2925	2911	2911
query39	1377	1330	1335	1330
query40	310	100	102	100
query41	41	37	38	37
query42	85	91	93	91
query43	586	556	577	556
query44	1184	714	722	714
query45	243	225	230	225
query46	1229	956	970	956
query47	1907	1740	1757	1740
query48	511	417	401	401
query49	652	382	380	380
query50	860	597	605	597
query51	4768	4662	4633	4633
query52	91	94	84	84
query53	234	185	186	185
query54	2660	2471	2521	2471
query55	86	74	85	74
query56	224	205	218	205
query57	1274	1144	1206	1144
query58	223	216	208	208
query59	3417	3236	3228	3228
query60	216	205	211	205
query61	100	93	94	93
query62	833	499	458	458
query63	202	179	178	178
query64	5892	1634	1492	1492
query65	3627	3546	3548	3546
query66	637	406	406	406
query67	17752	15962	15381	15381
query68	9447	637	629	629
query69	506	266	267	266
query70	1570	1440	1401	1401
query71	404	309	301	301
query72	6774	4802	4753	4753
query73	751	307	315	307
query74	6298	5828	5918	5828
query75	5244	3711	3687	3687
query76	5275	1132	1193	1132
query77	911	251	273	251
query78	12593	11852	11692	11692
query79	8522	644	641	641
query80	1807	386	391	386
query81	481	237	235	235
query82	1643	96	91	91
query83	169	131	130	130
query84	254	69	70	69
query85	897	313	316	313
query86	337	284	292	284
query87	3267	2959	3061	2959
query88	4768	2263	2265	2263
query89	489	286	300	286
query90	1937	202	204	202
query91	159	126	136	126
query92	59	50	52	50
query93	7227	591	552	552
query94	725	211	204	204
query95	1997	1966	1930	1930
query96	650	322	317	317
query97	6428	6433	6437	6433
query98	210	207	217	207
query99	3072	814	877	814
Total cold run time: 325320 ms
Total hot run time: 212598 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.05 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 301603b1dd29138f472ffb02eaaaa84447060540, data reload: false

query1	0.02	0.03	0.02
query2	0.07	0.02	0.02
query3	0.24	0.04	0.05
query4	1.78	0.07	0.06
query5	0.54	0.54	0.52
query6	1.23	0.62	0.63
query7	0.02	0.01	0.01
query8	0.04	0.02	0.03
query9	0.55	0.49	0.49
query10	0.55	0.52	0.55
query11	0.12	0.08	0.08
query12	0.12	0.08	0.09
query13	0.61	0.61	0.62
query14	0.79	0.78	0.79
query15	0.78	0.76	0.76
query16	0.39	0.37	0.36
query17	1.01	1.03	1.04
query18	0.23	0.26	0.26
query19	1.91	1.84	1.86
query20	0.01	0.00	0.00
query21	15.47	0.56	0.55
query22	2.11	2.50	1.50
query23	17.05	0.95	0.98
query24	4.36	1.38	1.60
query25	0.35	0.09	0.06
query26	0.62	0.16	0.15
query27	0.04	0.04	0.04
query28	8.16	0.75	0.72
query29	12.77	2.32	2.32
query30	0.57	0.48	0.57
query31	2.80	0.39	0.37
query32	3.37	0.49	0.50
query33	3.11	3.08	3.13
query34	15.26	4.76	4.77
query35	4.85	4.84	4.82
query36	1.09	1.00	1.04
query37	0.06	0.04	0.05
query38	0.04	0.02	0.02
query39	0.02	0.02	0.02
query40	0.15	0.14	0.15
query41	0.06	0.01	0.01
query42	0.02	0.01	0.01
query43	0.03	0.02	0.02
Total cold run time: 103.37 s
Total hot run time: 31.05 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 301603b1dd29138f472ffb02eaaaa84447060540 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.8 seconds inserted 10000000 Rows, about 458K ops/s

@w41ter w41ter closed this Oct 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants