Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](auto bucket) fix auto buckets calc using the first k partition #41675 #41821

Merged
merged 1 commit into from
Oct 15, 2024
Merged

[fix](auto bucket) fix auto buckets calc using the first k partition #41675 #41821

merged 1 commit into from
Oct 15, 2024

Conversation

yujun777
Copy link
Collaborator

cherry pick from #41675

@yujun777
[fix](auto bucket) fix auto buckets calc using the first k partition (a…
e657fd0 
…pache#41675)

If the first k (at most 7) partition data size is ascending, the result
will be partion_size[k-1] + ema(first k partitons delta).

This is a bug, should use the last k partitions, but not the first k
partitions to calculate.
@yujun777
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@doris-robot
Copy link

TPC-H: Total hot run time: 49221 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e657fd02097dca08c9151fcdc4c03fe73a5ff059, data reload: false

------ Round 1 ----------------------------------
q1	18219	4353	4355	4353
q2	2079	154	146	146
q3	10268	1936	1938	1936
q4	10348	1264	1332	1264
q5	8537	3911	3863	3863
q6	232	121	125	121
q7	2045	1607	1624	1607
q8	9295	2729	2709	2709
q9	10012	9743	9808	9743
q10	8663	3569	3538	3538
q11	427	237	268	237
q12	476	304	296	296
q13	18376	3999	4069	3999
q14	353	322	326	322
q15	525	459	464	459
q16	558	469	459	459
q17	1146	950	949	949
q18	7298	6843	6851	6843
q19	1793	1571	1538	1538
q20	544	315	286	286
q21	4504	4182	4150	4150
q22	500	403	408	403
Total cold run time: 116198 ms
Total hot run time: 49221 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4352	4359	4383	4359
q2	333	228	224	224
q3	4185	4148	4195	4148
q4	2817	2827	2844	2827
q5	7455	7350	7362	7350
q6	243	129	122	122
q7	3355	2930	2887	2887
q8	4361	4518	4523	4518
q9	13779	13524	13481	13481
q10	4240	4235	4268	4235
q11	723	682	685	682
q12	1029	860	831	831
q13	7096	3768	3750	3750
q14	466	417	411	411
q15	505	473	456	456
q16	622	584	597	584
q17	3854	3758	3813	3758
q18	8704	8679	8773	8679
q19	1721	1691	1673	1673
q20	2391	2108	2106	2106
q21	8514	8495	8326	8326
q22	1001	937	925	925
Total cold run time: 81746 ms
Total hot run time: 76332 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 211987 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e657fd02097dca08c9151fcdc4c03fe73a5ff059, data reload: false

query1	936	395	411	395
query2	6529	2174	2126	2126
query3	6927	199	195	195
query4	23109	21896	21386	21386
query5	19741	6538	6493	6493
query6	284	221	248	221
query7	4339	299	298	298
query8	303	242	227	227
query9	3067	2678	2587	2587
query10	465	303	283	283
query11	15662	15156	14799	14799
query12	119	75	75	75
query13	1044	430	421	421
query14	17639	13320	13416	13320
query15	370	218	220	218
query16	5807	292	260	260
query17	1694	938	919	919
query18	898	317	313	313
query19	208	160	149	149
query20	110	96	109	96
query21	191	98	97	97
query22	5199	5044	4962	4962
query23	34079	33429	33513	33429
query24	7613	6312	6235	6235
query25	530	414	421	414
query26	1344	158	158	158
query27	2357	291	291	291
query28	6073	2286	2231	2231
query29	3011	2695	2739	2695
query30	237	170	164	164
query31	975	773	742	742
query32	63	63	56	56
query33	451	268	260	260
query34	884	467	471	467
query35	1145	927	919	919
query36	1257	1121	1243	1121
query37	95	58	61	58
query38	3032	2928	2884	2884
query39	1370	1310	1307	1307
query40	305	95	94	94
query41	38	37	37	37
query42	80	88	87	87
query43	601	617	580	580
query44	1191	721	719	719
query45	241	231	223	223
query46	1233	962	955	955
query47	1817	1817	1738	1738
query48	494	409	413	409
query49	645	374	368	368
query50	869	624	612	612
query51	4742	4617	4708	4617
query52	84	88	87	87
query53	247	189	188	188
query54	2665	2441	2488	2441
query55	97	90	89	89
query56	239	228	212	212
query57	1351	1199	1155	1155
query58	214	204	207	204
query59	3511	3325	3180	3180
query60	220	206	205	205
query61	95	93	92	92
query62	776	435	503	435
query63	202	180	173	173
query64	3595	1568	1504	1504
query65	3606	3543	3561	3543
query66	795	417	405	405
query67	15475	15776	16873	15776
query68	6412	626	623	623
query69	473	252	281	252
query70	1503	1469	1392	1392
query71	369	312	315	312
query72	6828	4839	4762	4762
query73	735	319	321	319
query74	6222	5804	5809	5804
query75	4476	3709	3748	3709
query76	3463	1160	1196	1160
query77	541	256	243	243
query78	12556	11546	12457	11546
query79	10506	629	623	623
query80	2821	386	376	376
query81	515	231	234	231
query82	1478	94	95	94
query83	182	136	128	128
query84	252	73	69	69
query85	1454	313	317	313
query86	353	296	281	281
query87	3214	3009	3000	3000
query88	4907	2279	2269	2269
query89	476	282	283	282
query90	1773	214	201	201
query91	165	123	131	123
query92	57	48	51	48
query93	6084	598	529	529
query94	857	209	203	203
query95	1946	1901	1939	1901
query96	647	326	325	325
query97	6516	6494	6416	6416
query98	229	210	193	193
query99	2631	962	861	861
Total cold run time: 315226 ms
Total hot run time: 211987 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.92 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e657fd02097dca08c9151fcdc4c03fe73a5ff059, data reload: false

query1	0.02	0.03	0.03
query2	0.07	0.02	0.02
query3	0.24	0.05	0.06
query4	1.77	0.10	0.07
query5	0.54	0.53	0.53
query6	1.33	0.63	0.61
query7	0.01	0.01	0.01
query8	0.04	0.02	0.02
query9	0.55	0.49	0.48
query10	0.54	0.54	0.56
query11	0.12	0.08	0.09
query12	0.12	0.09	0.09
query13	0.63	0.62	0.63
query14	0.79	0.80	0.81
query15	0.80	0.78	0.78
query16	0.39	0.37	0.37
query17	1.03	1.02	1.02
query18	0.21	0.27	0.25
query19	1.86	1.89	1.90
query20	0.01	0.02	0.01
query21	15.69	0.57	0.57
query22	1.86	1.80	1.62
query23	17.27	1.03	0.98
query24	4.67	1.06	1.28
query25	0.29	0.07	0.06
query26	0.70	0.17	0.14
query27	0.04	0.05	0.04
query28	8.13	0.75	0.74
query29	12.63	2.35	2.13
query30	0.65	0.53	0.48
query31	2.81	0.39	0.38
query32	3.38	0.50	0.50
query33	3.06	3.07	3.06
query34	15.27	4.79	4.81
query35	4.85	4.86	4.84
query36	1.06	1.02	1.02
query37	0.06	0.04	0.04
query38	0.04	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.07	0.01	0.01
query42	0.02	0.02	0.01
query43	0.02	0.01	0.01
Total cold run time: 103.82 s
Total hot run time: 30.92 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit e657fd02097dca08c9151fcdc4c03fe73a5ff059 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          57 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.6 seconds inserted 10000000 Rows, about 462K ops/s

@dataroaring dataroaring merged commit bd2d64f into apache:branch-2.0 Oct 15, 2024
22 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@yujun777 @doris-robot @dataroaring