Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve](schema-change) support nested type with varchar type to support length growing #46639

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

amorynan
Copy link
Contributor

@amorynan amorynan commented Jan 8, 2025

What problem does this PR solve?

The array|map|struct schema-change behavior supports moditfy to change the length of the varchar type in the current column type.
before

mysql> alter table t_sc MODIFY COLUMN s struct<col:varchar(20)>;
ERROR 1105 (HY000): errCode = 2, detailMessage = Can not change struct<col:varchar(10)> to `__doris_shadow_s` struct<col:varchar(20)> NULL

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 8, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@amorynan
Copy link
Contributor Author

amorynan commented Jan 8, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32732 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e0114317ab12b807006efd68c448e369582820bb, data reload: false

------ Round 1 ----------------------------------
q1	17629	6170	6041	6041
q2	2059	303	172	172
q3	10430	1225	734	734
q4	10207	873	436	436
q5	7734	2205	1952	1952
q6	201	178	150	150
q7	891	759	624	624
q8	9253	1350	1145	1145
q9	5285	4902	4889	4889
q10	6821	2314	1872	1872
q11	466	280	252	252
q12	347	357	221	221
q13	17850	3780	3230	3230
q14	244	229	213	213
q15	571	514	522	514
q16	611	620	596	596
q17	587	866	338	338
q18	6950	6498	6366	6366
q19	2835	969	559	559
q20	299	308	188	188
q21	2872	2136	1932	1932
q22	364	341	308	308
Total cold run time: 104506 ms
Total hot run time: 32732 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6386	6224	6251	6224
q2	241	340	230	230
q3	2218	2633	2341	2341
q4	1400	1812	1369	1369
q5	4334	4720	4818	4720
q6	185	188	147	147
q7	2056	2012	1864	1864
q8	2607	2779	2707	2707
q9	7272	7247	7355	7247
q10	3080	3337	2834	2834
q11	567	514	489	489
q12	648	730	575	575
q13	3543	3883	3251	3251
q14	284	324	292	292
q15	575	506	509	506
q16	645	688	668	668
q17	1236	1727	1257	1257
q18	7538	7538	7281	7281
q19	832	1043	1144	1043
q20	2011	2061	1911	1911
q21	5723	5262	4937	4937
q22	648	615	602	602
Total cold run time: 54029 ms
Total hot run time: 52495 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195928 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e0114317ab12b807006efd68c448e369582820bb, data reload: false

query1	1299	937	886	886
query2	6349	2366	2362	2362
query3	11128	4837	4874	4837
query4	32656	23497	23838	23497
query5	5728	574	457	457
query6	285	203	195	195
query7	3998	499	308	308
query8	304	255	237	237
query9	9684	2744	2716	2716
query10	547	325	266	266
query11	18222	15168	15111	15111
query12	173	115	113	113
query13	1586	539	406	406
query14	9448	7300	7701	7300
query15	229	204	195	195
query16	7691	642	446	446
query17	1536	758	593	593
query18	1442	391	323	323
query19	211	187	162	162
query20	121	120	112	112
query21	212	123	106	106
query22	4628	4641	4401	4401
query23	34186	33093	33111	33093
query24	6359	2273	2295	2273
query25	535	444	390	390
query26	714	277	152	152
query27	2150	465	340	340
query28	4626	2490	2492	2490
query29	587	561	458	458
query30	221	198	153	153
query31	957	920	821	821
query32	80	61	58	58
query33	472	349	324	324
query34	771	891	538	538
query35	829	866	772	772
query36	1020	1059	964	964
query37	122	107	76	76
query38	4217	4125	4368	4125
query39	1499	1448	1489	1448
query40	218	117	105	105
query41	47	44	46	44
query42	119	111	104	104
query43	530	531	518	518
query44	1360	860	855	855
query45	187	183	178	178
query46	890	1068	672	672
query47	1919	1871	1836	1836
query48	399	460	344	344
query49	824	512	418	418
query50	652	715	406	406
query51	7059	7147	6967	6967
query52	105	102	93	93
query53	226	269	197	197
query54	495	501	419	419
query55	86	82	79	79
query56	259	274	242	242
query57	1183	1229	1135	1135
query58	241	236	225	225
query59	3159	3246	3167	3167
query60	288	267	272	267
query61	110	142	107	107
query62	825	807	726	726
query63	228	196	185	185
query64	2992	1020	669	669
query65	3313	3231	3237	3231
query66	692	404	326	326
query67	16128	15587	15374	15374
query68	8463	697	517	517
query69	516	296	251	251
query70	1215	1082	1097	1082
query71	469	290	257	257
query72	6215	3896	3896	3896
query73	644	764	361	361
query74	9758	8886	8928	8886
query75	3918	3155	2679	2679
query76	3615	1194	755	755
query77	678	362	267	267
query78	10144	9849	9407	9407
query79	3737	816	588	588
query80	737	527	444	444
query81	500	259	235	235
query82	621	155	125	125
query83	176	231	142	142
query84	245	87	70	70
query85	745	383	317	317
query86	353	318	300	300
query87	4601	4547	4343	4343
query88	5043	2212	2208	2208
query89	415	322	293	293
query90	1919	192	184	184
query91	141	137	108	108
query92	69	59	54	54
query93	2215	929	515	515
query94	624	410	298	298
query95	336	270	257	257
query96	502	616	281	281
query97	2928	2927	2736	2736
query98	231	203	202	202
query99	1581	1489	1477	1477
Total cold run time: 294710 ms
Total hot run time: 195928 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.5 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e0114317ab12b807006efd68c448e369582820bb, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.04
query3	0.23	0.07	0.07
query4	1.60	0.10	0.11
query5	0.40	0.43	0.42
query6	1.15	0.66	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.60	0.51	0.49
query10	0.55	0.55	0.55
query11	0.15	0.10	0.10
query12	0.15	0.12	0.11
query13	0.61	0.61	0.60
query14	2.72	2.70	2.76
query15	0.91	0.82	0.82
query16	0.40	0.38	0.37
query17	1.08	1.05	1.06
query18	0.22	0.20	0.20
query19	1.90	1.83	1.98
query20	0.02	0.01	0.01
query21	15.34	0.95	0.56
query22	0.76	0.82	0.65
query23	15.29	1.48	0.52
query24	3.09	1.07	1.53
query25	0.16	0.25	0.14
query26	0.23	0.15	0.13
query27	0.05	0.05	0.07
query28	13.83	1.50	1.05
query29	12.57	3.96	3.31
query30	0.25	0.09	0.07
query31	2.83	0.58	0.39
query32	3.23	0.55	0.46
query33	3.08	3.14	3.12
query34	16.87	5.16	4.51
query35	4.56	4.50	4.63
query36	0.64	0.52	0.47
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.03	0.02
query40	0.16	0.15	0.14
query41	0.09	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 106.12 s
Total hot run time: 31.5 s

@amorynan amorynan changed the title [imprve](schema-change) support nested type with varchar type to support length growing [improve](schema-change) support nested type with varchar type to support length growing Jan 9, 2025
}


sql """ alter table ${tableName} modify column c_a array<varchar(20)> """
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

plz add a case as such like:

alter table ${tableName} modify column c_a array<varchar(20)> after c_s

cuz modify type+reorder will do link schema change rather than light schema change
we shall test this case additionaly

@amorynan
Copy link
Contributor Author

amorynan commented Jan 9, 2025

run buildall

// This method defines the char type or complex type nested char type
// to support the schema-change behavior of length growth.
public boolean isSupportSchemaChangeForCharType(Type other) throws TypeException {
if ((this.getPrimitiveType() == PrimitiveType.VARCHAR && other.getPrimitiveType() == PrimitiveType.VARCHAR) || (
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seperate this if to a function , and check PrimitiveType.VARCHAR/ isComplexType in seperate branch

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done!

@doris-robot
Copy link

TPC-H: Total hot run time: 32793 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b69c3457d164bab98068f576b31d9e51074a8b58, data reload: false

------ Round 1 ----------------------------------
q1	17600	6167	6027	6027
q2	2053	314	168	168
q3	10418	1297	766	766
q4	10230	898	450	450
q5	7843	2218	1980	1980
q6	210	181	142	142
q7	899	759	600	600
q8	9240	1367	1180	1180
q9	5334	4993	5007	4993
q10	6769	2309	1866	1866
q11	478	283	258	258
q12	350	377	225	225
q13	17766	3687	3091	3091
q14	235	242	220	220
q15	578	514	501	501
q16	644	623	592	592
q17	568	867	323	323
q18	6995	6615	6396	6396
q19	2023	964	544	544
q20	309	316	183	183
q21	2962	2273	1980	1980
q22	368	340	308	308
Total cold run time: 103872 ms
Total hot run time: 32793 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6313	6190	6197	6190
q2	242	326	233	233
q3	2252	2697	2319	2319
q4	1417	1897	1353	1353
q5	4314	4733	4785	4733
q6	189	180	148	148
q7	2054	1985	1787	1787
q8	2645	2847	2717	2717
q9	7368	7395	7337	7337
q10	3097	3321	2705	2705
q11	589	508	518	508
q12	686	801	620	620
q13	3489	3851	3254	3254
q14	308	291	265	265
q15	563	520	500	500
q16	656	694	649	649
q17	1229	1745	1265	1265
q18	7879	7462	7495	7462
q19	792	1142	1073	1073
q20	1982	2052	2003	2003
q21	5643	5036	5140	5036
q22	634	633	596	596
Total cold run time: 54341 ms
Total hot run time: 52753 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195459 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b69c3457d164bab98068f576b31d9e51074a8b58, data reload: false

query1	1308	947	931	931
query2	6301	2347	2387	2347
query3	11136	4799	4940	4799
query4	32958	23590	23295	23295
query5	3595	614	468	468
query6	282	204	191	191
query7	3991	485	299	299
query8	297	230	220	220
query9	9486	2706	2715	2706
query10	470	311	248	248
query11	17929	15238	15031	15031
query12	158	111	107	107
query13	1558	559	419	419
query14	10713	6898	7176	6898
query15	257	212	184	184
query16	7902	616	456	456
query17	1554	788	586	586
query18	2054	419	330	330
query19	204	196	172	172
query20	130	152	115	115
query21	209	122	104	104
query22	4693	4598	4536	4536
query23	34250	33412	33466	33412
query24	6558	2336	2328	2328
query25	458	456	389	389
query26	724	287	153	153
query27	2060	479	343	343
query28	5387	2509	2497	2497
query29	561	573	437	437
query30	207	180	150	150
query31	962	880	814	814
query32	70	57	59	57
query33	504	350	291	291
query34	764	877	522	522
query35	806	838	748	748
query36	1036	1086	963	963
query37	120	99	76	76
query38	4220	4389	4063	4063
query39	1536	1495	1432	1432
query40	199	119	103	103
query41	53	47	47	47
query42	120	106	109	106
query43	520	541	503	503
query44	1355	845	842	842
query45	182	176	163	163
query46	883	1066	659	659
query47	1898	1861	1875	1861
query48	394	409	330	330
query49	725	500	389	389
query50	625	678	406	406
query51	7149	6959	7126	6959
query52	103	97	94	94
query53	229	255	198	198
query54	498	525	416	416
query55	84	77	83	77
query56	264	264	245	245
query57	1203	1213	1142	1142
query58	238	246	238	238
query59	3217	3286	3239	3239
query60	271	255	251	251
query61	110	109	111	109
query62	872	803	736	736
query63	237	188	184	184
query64	2781	1065	671	671
query65	3259	3302	3236	3236
query66	1064	401	327	327
query67	16412	15838	15452	15452
query68	8121	709	533	533
query69	468	294	256	256
query70	1191	1132	1100	1100
query71	435	276	250	250
query72	6470	3947	3886	3886
query73	672	751	365	365
query74	9876	9031	8885	8885
query75	4122	3136	2650	2650
query76	3555	1184	768	768
query77	746	369	287	287
query78	10263	10260	9332	9332
query79	3201	798	593	593
query80	673	544	428	428
query81	464	279	227	227
query82	618	156	126	126
query83	164	161	152	152
query84	241	95	75	75
query85	786	368	296	296
query86	349	311	290	290
query87	4280	4387	4452	4387
query88	4358	2187	2128	2128
query89	417	326	293	293
query90	1870	186	190	186
query91	138	175	106	106
query92	62	58	51	51
query93	1438	837	550	550
query94	648	383	297	297
query95	334	263	254	254
query96	488	609	281	281
query97	2900	2928	2780	2780
query98	220	207	200	200
query99	1539	1530	1347	1347
Total cold run time: 293460 ms
Total hot run time: 195459 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.3 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b69c3457d164bab98068f576b31d9e51074a8b58, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.10	0.10
query5	0.42	0.42	0.40
query6	1.15	0.66	0.66
query7	0.02	0.01	0.02
query8	0.04	0.03	0.04
query9	0.59	0.51	0.50
query10	0.56	0.56	0.56
query11	0.14	0.10	0.11
query12	0.15	0.11	0.12
query13	0.62	0.60	0.61
query14	2.75	2.72	2.73
query15	0.89	0.83	0.83
query16	0.38	0.39	0.37
query17	1.08	1.05	1.08
query18	0.23	0.21	0.22
query19	1.93	1.79	2.06
query20	0.01	0.01	0.02
query21	15.37	0.97	0.56
query22	0.76	0.76	0.69
query23	15.26	1.40	0.52
query24	3.10	1.92	1.74
query25	0.14	0.34	0.13
query26	0.18	0.14	0.14
query27	0.05	0.05	0.04
query28	14.64	1.49	1.06
query29	12.58	4.04	3.36
query30	0.25	0.09	0.07
query31	2.82	0.59	0.38
query32	3.23	0.55	0.46
query33	3.08	3.07	3.24
query34	17.00	5.20	4.56
query35	4.56	4.54	4.59
query36	0.64	0.50	0.49
query37	0.11	0.06	0.06
query38	0.04	0.03	0.04
query39	0.04	0.03	0.03
query40	0.17	0.14	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 107.1 s
Total hot run time: 32.3 s

@amorynan
Copy link
Contributor Author

amorynan commented Jan 9, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32535 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6295442e7a6fde91445fd6d3a5b9c464712751dd, data reload: false

------ Round 1 ----------------------------------
q1	17720	6167	6017	6017
q2	2046	307	178	178
q3	10404	1245	763	763
q4	10289	858	432	432
q5	8714	2165	1974	1974
q6	205	178	147	147
q7	910	780	605	605
q8	9243	1352	1142	1142
q9	5218	4918	4866	4866
q10	6761	2308	1845	1845
q11	486	277	270	270
q12	342	383	216	216
q13	17759	3595	3092	3092
q14	234	230	215	215
q15	561	513	505	505
q16	639	618	576	576
q17	574	853	338	338
q18	6889	6502	6364	6364
q19	1908	946	510	510
q20	305	324	195	195
q21	2863	2194	1984	1984
q22	360	333	301	301
Total cold run time: 104430 ms
Total hot run time: 32535 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6323	6194	6225	6194
q2	248	333	235	235
q3	2260	2696	2325	2325
q4	1393	1749	1386	1386
q5	4326	4744	4799	4744
q6	186	181	145	145
q7	2113	1990	1807	1807
q8	2595	2802	2733	2733
q9	7234	7142	7209	7142
q10	3045	3293	2723	2723
q11	584	513	496	496
q12	738	784	647	647
q13	3462	3850	3214	3214
q14	284	313	297	297
q15	575	518	502	502
q16	668	675	645	645
q17	1241	1759	1276	1276
q18	7873	7543	7390	7390
q19	795	1145	1065	1065
q20	1951	2045	2013	2013
q21	5701	5101	5069	5069
q22	603	599	616	599
Total cold run time: 54198 ms
Total hot run time: 52647 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195776 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6295442e7a6fde91445fd6d3a5b9c464712751dd, data reload: false

query1	1297	950	913	913
query2	6437	2359	2327	2327
query3	10982	4592	4444	4444
query4	33415	23805	23248	23248
query5	4950	611	459	459
query6	289	193	183	183
query7	3988	479	320	320
query8	297	252	237	237
query9	9468	2710	2725	2710
query10	476	308	258	258
query11	18025	15256	15129	15129
query12	169	110	103	103
query13	1586	521	398	398
query14	10660	7735	7642	7642
query15	237	194	188	188
query16	7884	652	485	485
query17	1507	761	627	627
query18	2106	422	307	307
query19	219	185	157	157
query20	119	111	113	111
query21	199	147	110	110
query22	4685	4512	4399	4399
query23	33984	33430	33394	33394
query24	6158	2322	2431	2322
query25	469	451	386	386
query26	733	279	149	149
query27	1954	471	339	339
query28	5031	2488	2460	2460
query29	633	551	427	427
query30	225	184	153	153
query31	956	863	821	821
query32	74	57	58	57
query33	477	347	296	296
query34	798	857	507	507
query35	782	836	717	717
query36	1018	1038	961	961
query37	124	95	76	76
query38	4052	4175	4089	4089
query39	1517	1471	1441	1441
query40	204	116	98	98
query41	52	46	47	46
query42	125	102	103	102
query43	507	536	506	506
query44	1346	839	861	839
query45	180	178	171	171
query46	880	1071	648	648
query47	1942	1905	1837	1837
query48	383	411	338	338
query49	726	505	390	390
query50	681	670	403	403
query51	7189	7124	7095	7095
query52	102	99	94	94
query53	235	253	183	183
query54	495	486	436	436
query55	94	84	83	83
query56	246	256	256	256
query57	1228	1206	1129	1129
query58	242	226	229	226
query59	3212	3235	3162	3162
query60	304	268	260	260
query61	116	127	110	110
query62	869	786	719	719
query63	229	186	187	186
query64	2965	1046	666	666
query65	3247	3356	3235	3235
query66	776	398	306	306
query67	16275	15708	15425	15425
query68	7184	703	534	534
query69	509	299	249	249
query70	1187	1149	1114	1114
query71	447	282	257	257
query72	6272	3926	4055	3926
query73	656	764	358	358
query74	10302	9279	9014	9014
query75	4089	3104	2626	2626
query76	3732	1176	765	765
query77	774	371	267	267
query78	10032	10004	9250	9250
query79	3486	826	610	610
query80	634	508	431	431
query81	479	273	226	226
query82	667	157	122	122
query83	160	161	142	142
query84	289	83	78	78
query85	898	354	296	296
query86	364	311	303	303
query87	4306	4545	4358	4358
query88	4450	2189	2160	2160
query89	415	316	303	303
query90	1794	190	196	190
query91	137	131	106	106
query92	69	56	57	56
query93	2073	834	539	539
query94	656	400	293	293
query95	338	268	254	254
query96	483	604	286	286
query97	2886	2910	2789	2789
query98	215	200	200	200
query99	1604	1479	1375	1375
Total cold run time: 294417 ms
Total hot run time: 195776 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.75 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6295442e7a6fde91445fd6d3a5b9c464712751dd, data reload: false

query1	0.03	0.03	0.03
query2	0.09	0.03	0.04
query3	0.24	0.07	0.06
query4	1.62	0.10	0.10
query5	0.42	0.42	0.40
query6	1.16	0.66	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.59	0.49	0.51
query10	0.56	0.58	0.55
query11	0.15	0.10	0.11
query12	0.15	0.12	0.11
query13	0.61	0.61	0.60
query14	2.81	2.71	2.80
query15	0.91	0.83	0.82
query16	0.36	0.39	0.38
query17	1.04	1.02	1.07
query18	0.23	0.21	0.21
query19	1.99	1.87	1.92
query20	0.01	0.01	0.01
query21	15.36	0.91	0.58
query22	0.77	0.73	0.71
query23	15.35	1.40	0.61
query24	2.96	1.25	1.40
query25	0.21	0.15	0.11
query26	0.25	0.15	0.14
query27	0.06	0.05	0.04
query28	14.12	1.56	1.04
query29	12.62	3.95	3.30
query30	0.25	0.09	0.06
query31	2.82	0.60	0.38
query32	3.23	0.55	0.48
query33	3.02	3.12	3.10
query34	16.85	5.12	4.50
query35	4.51	4.45	4.48
query36	0.66	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.07	0.02	0.02
query42	0.04	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.54 s
Total hot run time: 31.75 s

@amorynan
Copy link
Contributor Author

run buildall

@amorynan amorynan requested a review from eldenmoon January 10, 2025 03:36
@doris-robot
Copy link

TPC-H: Total hot run time: 32838 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6bf5fc1769cd78fb03c74bc5ca0aef80b75f2b07, data reload: false

------ Round 1 ----------------------------------
q1	17599	6243	6051	6051
q2	2046	302	171	171
q3	10411	1249	734	734
q4	10204	879	446	446
q5	7537	2159	1962	1962
q6	211	180	149	149
q7	879	754	619	619
q8	9248	1371	1158	1158
q9	5209	4873	4950	4873
q10	6734	2305	1872	1872
q11	476	285	284	284
q12	349	354	223	223
q13	17785	3662	3097	3097
q14	240	245	214	214
q15	561	501	495	495
q16	635	600	584	584
q17	591	846	339	339
q18	7231	6521	6552	6521
q19	1695	945	537	537
q20	326	337	194	194
q21	2760	2262	2007	2007
q22	373	331	308	308
Total cold run time: 103100 ms
Total hot run time: 32838 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6352	6213	6241	6213
q2	238	328	246	246
q3	2255	2620	2302	2302
q4	1428	1781	1392	1392
q5	4321	4728	4773	4728
q6	181	178	146	146
q7	2094	1999	1852	1852
q8	2596	2779	2660	2660
q9	7326	7194	7265	7194
q10	3051	3276	2825	2825
q11	578	536	490	490
q12	704	755	585	585
q13	3451	3856	3289	3289
q14	299	308	278	278
q15	574	524	493	493
q16	648	687	646	646
q17	1210	1706	1244	1244
q18	7816	7414	7050	7050
q19	801	843	1054	843
q20	1949	1967	1793	1793
q21	5414	5053	4971	4971
q22	613	644	599	599
Total cold run time: 53899 ms
Total hot run time: 51839 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187817 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6bf5fc1769cd78fb03c74bc5ca0aef80b75f2b07, data reload: false

query1	972	368	383	368
query2	6522	2294	2286	2286
query3	6709	219	223	219
query4	33391	23339	23172	23172
query5	4379	625	449	449
query6	294	207	229	207
query7	4628	484	317	317
query8	286	231	226	226
query9	9411	2707	2726	2707
query10	476	323	273	273
query11	18155	15225	14995	14995
query12	147	103	105	103
query13	1643	511	376	376
query14	9229	6929	6485	6485
query15	229	185	181	181
query16	7893	554	436	436
query17	1544	711	567	567
query18	1928	391	292	292
query19	200	186	181	181
query20	118	108	110	108
query21	209	117	100	100
query22	4385	4232	4307	4232
query23	34242	32955	33396	32955
query24	6512	2362	2210	2210
query25	473	442	388	388
query26	1221	268	156	156
query27	2326	446	340	340
query28	5648	2466	2432	2432
query29	687	593	446	446
query30	241	188	158	158
query31	993	861	758	758
query32	72	66	61	61
query33	521	361	313	313
query34	751	814	513	513
query35	796	818	729	729
query36	995	1013	959	959
query37	129	109	80	80
query38	4182	3953	4068	3953
query39	1488	1404	1423	1404
query40	210	112	101	101
query41	52	54	50	50
query42	114	106	102	102
query43	505	509	490	490
query44	1356	830	825	825
query45	199	174	163	163
query46	858	1027	640	640
query47	1836	1839	1772	1772
query48	392	399	316	316
query49	769	479	374	374
query50	610	668	388	388
query51	6819	7107	6911	6911
query52	102	97	90	90
query53	227	255	178	178
query54	480	485	406	406
query55	77	77	81	77
query56	261	274	238	238
query57	1192	1149	1066	1066
query58	255	230	260	230
query59	2972	3253	3265	3253
query60	288	259	252	252
query61	121	113	124	113
query62	855	787	679	679
query63	242	185	189	185
query64	4031	1002	645	645
query65	3201	3143	3169	3143
query66	1066	419	303	303
query67	15897	15533	15440	15440
query68	7243	717	526	526
query69	466	284	249	249
query70	1216	1138	1052	1052
query71	408	283	249	249
query72	6477	3874	4020	3874
query73	659	760	367	367
query74	10176	8800	8836	8800
query75	3526	3157	2670	2670
query76	3323	1172	759	759
query77	603	356	280	280
query78	10146	10028	9308	9308
query79	3854	785	603	603
query80	736	521	528	521
query81	507	270	240	240
query82	674	147	116	116
query83	189	181	153	153
query84	240	96	76	76
query85	776	351	306	306
query86	400	313	293	293
query87	4535	4435	4217	4217
query88	4833	2188	2157	2157
query89	403	325	303	303
query90	1875	184	189	184
query91	133	137	111	111
query92	65	57	57	57
query93	2235	834	535	535
query94	699	385	295	295
query95	344	265	254	254
query96	493	609	286	286
query97	2848	2949	2850	2850
query98	231	203	193	193
query99	1450	1543	1368	1368
Total cold run time: 290775 ms
Total hot run time: 187817 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.63 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6bf5fc1769cd78fb03c74bc5ca0aef80b75f2b07, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.03
query3	0.24	0.06	0.06
query4	1.62	0.11	0.11
query5	0.41	0.42	0.40
query6	1.17	0.65	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.59	0.48	0.49
query10	0.56	0.56	0.55
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.61	0.59
query14	2.75	2.79	2.76
query15	0.89	0.83	0.81
query16	0.38	0.39	0.37
query17	0.98	1.01	1.08
query18	0.24	0.22	0.22
query19	1.88	1.77	1.93
query20	0.01	0.01	0.01
query21	15.36	0.99	0.58
query22	0.76	0.88	0.77
query23	15.12	1.50	0.61
query24	3.03	1.20	1.61
query25	0.18	0.18	0.13
query26	0.24	0.14	0.12
query27	0.06	0.05	0.06
query28	13.97	1.52	1.06
query29	12.65	3.98	3.24
query30	0.25	0.11	0.07
query31	2.81	0.60	0.39
query32	3.22	0.55	0.45
query33	3.06	3.08	3.17
query34	16.63	5.19	4.49
query35	4.57	4.51	4.52
query36	0.75	0.47	0.47
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.18	0.13	0.14
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 105.92 s
Total hot run time: 31.63 s

eldenmoon
eldenmoon previously approved these changes Jan 10, 2025
Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 10, 2025
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@amorynan
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jan 13, 2025
@doris-robot
Copy link

TPC-H: Total hot run time: 32562 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8d480bf8dc137d9b7a5ad041ebb1ff4e12497038, data reload: false

------ Round 1 ----------------------------------
q1	17588	6177	6135	6135
q2	2056	297	174	174
q3	10414	1229	694	694
q4	10213	855	426	426
q5	7525	2179	1972	1972
q6	205	175	147	147
q7	895	746	614	614
q8	9239	1320	1129	1129
q9	5199	4861	4857	4857
q10	6737	2287	1843	1843
q11	472	284	275	275
q12	341	351	215	215
q13	17784	3689	3064	3064
q14	236	227	204	204
q15	550	498	501	498
q16	641	607	580	580
q17	560	840	319	319
q18	7112	6496	6366	6366
q19	1198	943	541	541
q20	323	327	190	190
q21	2757	2198	2009	2009
q22	371	339	310	310
Total cold run time: 102416 ms
Total hot run time: 32562 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6166	6170	6227	6170
q2	242	331	237	237
q3	2220	2667	2323	2323
q4	1379	1775	1345	1345
q5	4328	4784	4643	4643
q6	195	177	144	144
q7	2082	1933	1816	1816
q8	2624	2773	2628	2628
q9	7174	7073	7099	7073
q10	3047	3314	2807	2807
q11	592	513	515	513
q12	694	752	602	602
q13	3496	3866	3217	3217
q14	284	306	278	278
q15	552	513	505	505
q16	664	675	636	636
q17	1217	1709	1272	1272
q18	7699	7418	7026	7026
q19	780	848	1107	848
q20	1892	2000	1829	1829
q21	5373	5099	4847	4847
q22	582	595	596	595
Total cold run time: 53282 ms
Total hot run time: 51354 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187509 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8d480bf8dc137d9b7a5ad041ebb1ff4e12497038, data reload: false

query1	978	388	362	362
query2	6513	2341	2416	2341
query3	6713	215	216	215
query4	33479	23234	22988	22988
query5	4381	619	486	486
query6	289	209	193	193
query7	4619	480	307	307
query8	301	226	217	217
query9	9687	2683	2670	2670
query10	464	308	264	264
query11	17840	15374	15183	15183
query12	164	103	103	103
query13	1633	502	376	376
query14	10398	6573	7739	6573
query15	256	198	197	197
query16	8014	602	483	483
query17	1632	770	565	565
query18	1257	374	307	307
query19	233	183	162	162
query20	115	109	118	109
query21	215	133	107	107
query22	4216	4489	4240	4240
query23	33851	33532	33102	33102
query24	6389	2291	2219	2219
query25	503	444	397	397
query26	1169	263	153	153
query27	2010	446	324	324
query28	5092	2423	2426	2423
query29	589	517	412	412
query30	236	181	154	154
query31	964	852	796	796
query32	91	66	67	66
query33	511	352	293	293
query34	757	858	495	495
query35	779	809	723	723
query36	989	1028	926	926
query37	136	101	80	80
query38	4082	4254	3953	3953
query39	1441	1412	1429	1412
query40	209	116	101	101
query41	52	53	50	50
query42	123	106	110	106
query43	496	514	482	482
query44	1296	795	814	795
query45	176	167	169	167
query46	863	1023	633	633
query47	1816	1854	1784	1784
query48	373	379	304	304
query49	780	493	402	402
query50	615	645	393	393
query51	6861	6934	6860	6860
query52	101	99	92	92
query53	222	247	192	192
query54	478	490	414	414
query55	85	79	78	78
query56	256	265	260	260
query57	1150	1140	1093	1093
query58	243	227	243	227
query59	2938	3164	2918	2918
query60	287	278	256	256
query61	120	114	113	113
query62	907	777	719	719
query63	228	184	188	184
query64	4181	1034	650	650
query65	3232	3114	3124	3114
query66	1029	417	316	316
query67	15665	15683	15368	15368
query68	8391	696	511	511
query69	471	280	262	262
query70	1227	1102	1145	1102
query71	438	277	252	252
query72	6114	3846	3895	3846
query73	665	744	355	355
query74	10688	8895	8774	8774
query75	4611	3162	2672	2672
query76	3930	1182	781	781
query77	786	471	273	273
query78	9986	10068	9350	9350
query79	3625	835	597	597
query80	719	528	424	424
query81	472	271	240	240
query82	410	157	121	121
query83	195	170	161	161
query84	286	92	70	70
query85	837	343	294	294
query86	360	320	311	311
query87	4441	4435	4174	4174
query88	3065	2186	2125	2125
query89	431	320	288	288
query90	1944	197	197	197
query91	149	132	106	106
query92	68	57	56	56
query93	1975	856	527	527
query94	661	383	288	288
query95	341	264	251	251
query96	481	603	288	288
query97	2833	2936	2797	2797
query98	224	197	197	197
query99	1573	1474	1372	1372
Total cold run time: 290165 ms
Total hot run time: 187509 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.14 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8d480bf8dc137d9b7a5ad041ebb1ff4e12497038, data reload: false

query1	0.04	0.04	0.04
query2	0.06	0.04	0.03
query3	0.24	0.07	0.07
query4	1.60	0.11	0.11
query5	0.45	0.39	0.42
query6	1.15	0.68	0.64
query7	0.03	0.02	0.02
query8	0.04	0.02	0.03
query9	0.57	0.54	0.49
query10	0.54	0.55	0.54
query11	0.14	0.10	0.10
query12	0.13	0.11	0.10
query13	0.61	0.61	0.59
query14	2.86	2.85	2.78
query15	0.90	0.81	0.82
query16	0.37	0.37	0.39
query17	1.00	0.98	1.01
query18	0.23	0.22	0.22
query19	1.93	1.96	1.80
query20	0.02	0.01	0.01
query21	15.37	1.01	0.60
query22	0.76	0.86	0.59
query23	15.34	1.37	0.52
query24	3.66	1.06	0.82
query25	0.21	0.09	0.21
query26	0.30	0.16	0.14
query27	0.05	0.05	0.04
query28	13.28	1.48	1.05
query29	12.69	4.11	3.39
query30	0.26	0.09	0.06
query31	2.82	0.58	0.39
query32	3.23	0.54	0.49
query33	3.17	3.12	3.17
query34	16.69	5.10	4.50
query35	4.54	4.44	4.50
query36	0.79	0.48	0.48
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.04	0.02	0.03
query40	0.17	0.14	0.12
query41	0.09	0.03	0.03
query42	0.04	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.57 s
Total hot run time: 31.14 s

@amorynan
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32692 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6be45648cd585277631280a7b4096146ef96c124, data reload: false

------ Round 1 ----------------------------------
q1	17556	6116	6004	6004
q2	2053	304	174	174
q3	10618	1223	731	731
q4	10264	869	428	428
q5	8581	2207	1942	1942
q6	212	181	152	152
q7	898	779	596	596
q8	9277	1353	1125	1125
q9	5207	4999	4979	4979
q10	6854	2341	1890	1890
q11	473	292	250	250
q12	342	360	229	229
q13	18192	3622	3042	3042
q14	237	236	211	211
q15	548	495	508	495
q16	612	627	606	606
q17	567	847	313	313
q18	6879	6524	6513	6513
q19	1669	948	557	557
q20	306	314	183	183
q21	2795	2128	1959	1959
q22	373	336	313	313
Total cold run time: 104513 ms
Total hot run time: 32692 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6237	6238	6187	6187
q2	231	329	231	231
q3	2222	2659	2325	2325
q4	1388	1751	1368	1368
q5	4324	4770	4783	4770
q6	184	175	140	140
q7	2029	2020	1788	1788
q8	2611	2811	2666	2666
q9	7296	7258	7206	7206
q10	3041	3306	2690	2690
q11	583	533	506	506
q12	696	744	652	652
q13	3587	3771	3242	3242
q14	311	303	282	282
q15	572	518	508	508
q16	660	686	652	652
q17	1209	1713	1253	1253
q18	7731	7511	7290	7290
q19	789	1194	1016	1016
q20	1937	2041	1944	1944
q21	5743	5082	5048	5048
q22	632	613	598	598
Total cold run time: 54013 ms
Total hot run time: 52362 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 194958 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6be45648cd585277631280a7b4096146ef96c124, data reload: false

query1	1328	1006	939	939
query2	6088	2297	2270	2270
query3	10951	4646	4689	4646
query4	32940	23503	23606	23503
query5	5300	616	444	444
query6	309	183	192	183
query7	3991	490	322	322
query8	293	222	209	209
query9	9471	2666	2667	2666
query10	487	313	246	246
query11	18072	15333	15004	15004
query12	156	110	105	105
query13	1559	520	396	396
query14	10334	6947	6758	6758
query15	240	215	194	194
query16	7955	620	461	461
query17	1541	787	572	572
query18	2081	407	304	304
query19	197	180	156	156
query20	121	113	111	111
query21	226	128	103	103
query22	4580	4623	4349	4349
query23	33991	33880	33640	33640
query24	6439	2274	2217	2217
query25	472	441	379	379
query26	734	264	152	152
query27	2015	468	326	326
query28	5605	2461	2452	2452
query29	543	542	427	427
query30	210	185	160	160
query31	975	870	820	820
query32	73	61	58	58
query33	548	363	298	298
query34	762	874	507	507
query35	815	804	750	750
query36	992	1035	952	952
query37	122	95	73	73
query38	4045	4035	4172	4035
query39	1561	1463	1461	1461
query40	213	116	102	102
query41	53	53	49	49
query42	130	104	105	104
query43	514	532	487	487
query44	1328	809	811	809
query45	177	175	166	166
query46	876	1065	668	668
query47	1915	1917	1868	1868
query48	377	401	324	324
query49	725	496	404	404
query50	643	697	401	401
query51	7079	7095	6971	6971
query52	103	105	93	93
query53	224	255	183	183
query54	488	500	429	429
query55	84	82	77	77
query56	269	269	280	269
query57	1212	1206	1167	1167
query58	263	246	253	246
query59	3183	3158	3137	3137
query60	274	277	252	252
query61	114	120	118	118
query62	883	839	749	749
query63	226	187	187	187
query64	3198	1004	662	662
query65	3335	3216	3233	3216
query66	773	423	333	333
query67	16489	15730	15523	15523
query68	7561	702	521	521
query69	500	276	254	254
query70	1199	1134	1118	1118
query71	455	283	272	272
query72	6446	3829	3845	3829
query73	661	743	351	351
query74	10746	9301	8780	8780
query75	4106	3142	2643	2643
query76	3724	1197	766	766
query77	796	394	277	277
query78	10803	9917	9325	9325
query79	3665	783	576	576
query80	685	564	537	537
query81	502	279	234	234
query82	615	156	117	117
query83	163	169	149	149
query84	245	94	73	73
query85	766	444	297	297
query86	367	297	295	295
query87	4480	4367	4380	4367
query88	4581	2143	2137	2137
query89	405	307	298	298
query90	1841	194	199	194
query91	132	134	108	108
query92	64	55	52	52
query93	1142	735	540	540
query94	640	388	272	272
query95	329	266	258	258
query96	480	615	276	276
query97	2835	2969	2805	2805
query98	223	213	196	196
query99	1612	1515	1399	1399
Total cold run time: 296039 ms
Total hot run time: 194958 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.68 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6be45648cd585277631280a7b4096146ef96c124, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.03
query3	0.24	0.06	0.07
query4	1.62	0.10	0.11
query5	0.41	0.42	0.42
query6	1.18	0.64	0.65
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.59	0.49	0.51
query10	0.56	0.56	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.12
query13	0.61	0.60	0.59
query14	2.72	2.74	2.75
query15	0.89	0.83	0.82
query16	0.39	0.39	0.39
query17	0.98	1.08	1.06
query18	0.23	0.21	0.20
query19	1.83	1.75	2.01
query20	0.02	0.01	0.02
query21	15.38	0.94	0.60
query22	0.75	0.74	0.66
query23	15.33	1.44	0.54
query24	2.69	1.24	1.32
query25	0.20	0.17	0.21
query26	0.23	0.14	0.15
query27	0.05	0.05	0.05
query28	14.17	1.51	1.04
query29	12.60	4.11	3.42
query30	0.25	0.10	0.07
query31	2.81	0.59	0.39
query32	3.24	0.55	0.46
query33	3.14	3.14	3.13
query34	16.49	5.11	4.44
query35	4.53	4.41	4.47
query36	0.81	0.48	0.51
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 105.87 s
Total hot run time: 31.68 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants