Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve](column) implement insert_many_from function for all column #41623

Merged
merged 3 commits into from
Oct 14, 2024

Conversation

zhangstar333
Copy link
Contributor

Proposed changes

the cross join will call insert_many_from function to combine left table column as result,
implement it's will reduce much virtual function call.

mysql [ssb]>select count(c_custkey) from (select customer.c_custkey from dates cross join customer)t;
+------------------+
| count(c_custkey) |
+------------------+
|       7668000000 |
+------------------+
1 row in set (5.56 sec)
mysql [ssb]>select count(c_custkey) from (select customer.c_custkey from dates cross join customer)t;
+------------------+
| count(c_custkey) |
+------------------+
|       7668000000 |
+------------------+
1 row in set (3.29 sec)

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zhangstar333
Copy link
Contributor Author

run buildall

Copy link
Contributor

github-actions bot commented Oct 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40942 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bc0fc89de1601f5205d5ec214986345c87f0abd1, data reload: false

------ Round 1 ----------------------------------
q1	17613	7452	7275	7275
q2	2033	298	276	276
q3	12212	1072	1192	1072
q4	10518	709	824	709
q5	7792	2871	2849	2849
q6	236	158	155	155
q7	995	632	612	612
q8	9357	1948	1996	1948
q9	6627	6467	6398	6398
q10	6944	2306	2323	2306
q11	440	248	259	248
q12	417	230	227	227
q13	17801	2997	3001	2997
q14	237	209	217	209
q15	582	520	524	520
q16	647	585	578	578
q17	986	600	543	543
q18	7480	6727	6646	6646
q19	1328	985	875	875
q20	502	211	200	200
q21	3978	3332	3310	3310
q22	1124	1012	989	989
Total cold run time: 109849 ms
Total hot run time: 40942 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7285	7194	7193	7193
q2	328	230	234	230
q3	3074	2909	2970	2909
q4	2042	1868	1794	1794
q5	5817	5760	5727	5727
q6	232	153	156	153
q7	2245	1877	1826	1826
q8	3395	3606	3451	3451
q9	8928	8951	8868	8868
q10	3621	3572	3544	3544
q11	604	515	505	505
q12	862	661	657	657
q13	9793	3168	3169	3168
q14	327	277	279	277
q15	587	544	540	540
q16	694	662	648	648
q17	1869	1624	1609	1609
q18	8491	7670	7668	7668
q19	1756	1438	1491	1438
q20	2146	1882	1921	1882
q21	5771	5619	5503	5503
q22	1146	1043	1092	1043
Total cold run time: 71013 ms
Total hot run time: 60633 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192305 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bc0fc89de1601f5205d5ec214986345c87f0abd1, data reload: false

query1	946	399	407	399
query2	6264	2047	2035	2035
query3	8690	194	200	194
query4	34365	23454	23446	23446
query5	3523	486	474	474
query6	288	171	176	171
query7	4187	305	303	303
query8	283	225	231	225
query9	9094	2695	2698	2695
query10	460	279	274	274
query11	17814	15118	15223	15118
query12	154	99	105	99
query13	1585	487	452	452
query14	9117	7544	7621	7544
query15	254	172	185	172
query16	7406	472	487	472
query17	1639	577	624	577
query18	2006	347	322	322
query19	356	164	155	155
query20	124	116	111	111
query21	216	109	110	109
query22	4617	4775	4682	4682
query23	35160	34364	33982	33982
query24	10745	2843	2820	2820
query25	589	415	407	407
query26	716	159	168	159
query27	2022	305	299	299
query28	6347	2458	2431	2431
query29	706	459	428	428
query30	260	154	151	151
query31	1040	826	823	823
query32	106	56	53	53
query33	668	305	309	305
query34	893	530	521	521
query35	872	734	733	733
query36	1113	996	953	953
query37	152	87	85	85
query38	4050	4000	3895	3895
query39	1493	1414	1427	1414
query40	207	98	100	98
query41	47	44	46	44
query42	123	96	92	92
query43	539	503	507	503
query44	1178	825	816	816
query45	196	171	166	166
query46	1131	739	716	716
query47	1995	1884	1852	1852
query48	438	347	341	341
query49	903	415	401	401
query50	845	411	443	411
query51	7006	6813	6835	6813
query52	100	88	84	84
query53	252	178	175	175
query54	1247	465	480	465
query55	84	80	77	77
query56	285	253	247	247
query57	1274	1148	1161	1148
query58	228	237	229	229
query59	3242	3006	3096	3006
query60	294	264	265	264
query61	105	100	101	100
query62	874	675	693	675
query63	222	196	188	188
query64	3651	658	610	610
query65	3246	3174	3178	3174
query66	837	327	307	307
query67	15862	15547	15639	15547
query68	4415	580	559	559
query69	513	290	304	290
query70	1220	1055	1180	1055
query71	355	279	276	276
query72	7472	3937	3941	3937
query73	769	348	345	345
query74	9971	8947	9015	8947
query75	3485	2757	2665	2665
query76	2959	851	889	851
query77	594	304	297	297
query78	10395	9587	9476	9476
query79	1500	597	604	597
query80	2526	464	448	448
query81	595	237	236	236
query82	754	136	145	136
query83	306	132	140	132
query84	277	82	80	80
query85	1457	288	296	288
query86	439	305	307	305
query87	4412	4471	4288	4288
query88	3549	2468	2404	2404
query89	413	291	295	291
query90	2052	190	186	186
query91	139	113	115	113
query92	64	46	49	46
query93	1067	542	543	542
query94	1121	299	290	290
query95	354	245	255	245
query96	601	285	284	284
query97	3234	3167	3156	3156
query98	226	207	199	199
query99	1552	1304	1307	1304
Total cold run time: 295517 ms
Total hot run time: 192305 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.8 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bc0fc89de1601f5205d5ec214986345c87f0abd1, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.02	0.03
query3	0.23	0.06	0.06
query4	1.65	0.10	0.10
query5	0.54	0.52	0.51
query6	1.13	0.73	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.56	0.49	0.48
query10	0.55	0.53	0.54
query11	0.14	0.10	0.11
query12	0.13	0.12	0.11
query13	0.60	0.59	0.60
query14	2.81	2.85	2.90
query15	0.90	0.82	0.82
query16	0.39	0.40	0.38
query17	1.12	1.05	1.05
query18	0.19	0.19	0.20
query19	1.98	1.89	2.01
query20	0.01	0.01	0.01
query21	15.36	0.60	0.59
query22	2.84	2.80	1.99
query23	17.16	0.94	0.82
query24	2.66	1.53	0.76
query25	0.20	0.10	0.10
query26	0.48	0.14	0.13
query27	0.04	0.05	0.03
query28	10.83	1.11	1.06
query29	12.55	3.27	3.24
query30	0.25	0.06	0.06
query31	2.87	0.41	0.37
query32	3.26	0.48	0.47
query33	3.00	3.05	3.08
query34	17.02	4.45	4.43
query35	4.50	4.43	4.50
query36	0.67	0.51	0.49
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.03	0.02
query40	0.15	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.02	0.02
Total cold run time: 107.23 s
Total hot run time: 32.8 s

@zhangstar333
Copy link
Contributor Author

run buildall

1 similar comment
@zhangstar333
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@zhangstar333
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.40% (9662/25834)
Line Coverage: 28.68% (80180/279521)
Region Coverage: 28.11% (41443/147434)
Branch Coverage: 24.71% (21105/85406)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f3a4f93464258ed3ed4227bd70a376a30eec0076_f3a4f93464258ed3ed4227bd70a376a30eec0076/report/index.html

@zhangstar333
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.39% (9661/25836)
Line Coverage: 28.67% (80142/279548)
Region Coverage: 28.09% (41429/147479)
Branch Coverage: 24.69% (21096/85430)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f3a4f93464258ed3ed4227bd70a376a30eec0076_f3a4f93464258ed3ed4227bd70a376a30eec0076/report/index.html

@zhangstar333
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.40% (9661/25835)
Line Coverage: 28.67% (80156/279575)
Region Coverage: 28.09% (41436/147487)
Branch Coverage: 24.70% (21104/85436)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f3a4f93464258ed3ed4227bd70a376a30eec0076_f3a4f93464258ed3ed4227bd70a376a30eec0076/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 41288 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f3a4f93464258ed3ed4227bd70a376a30eec0076, data reload: false

------ Round 1 ----------------------------------
q1	17879	7540	7339	7339
q2	2031	274	280	274
q3	12295	1055	1229	1055
q4	10546	804	745	745
q5	7796	2860	2825	2825
q6	241	152	150	150
q7	1027	631	616	616
q8	9347	1979	1909	1909
q9	6575	6477	6440	6440
q10	6976	2317	2318	2317
q11	445	260	247	247
q12	424	228	225	225
q13	17778	3025	2998	2998
q14	252	222	217	217
q15	572	525	528	525
q16	633	581	589	581
q17	983	567	573	567
q18	7518	6761	6662	6662
q19	1353	1044	1044	1044
q20	496	197	198	197
q21	4106	3336	3364	3336
q22	1121	1036	1019	1019
Total cold run time: 110394 ms
Total hot run time: 41288 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7259	7234	7259	7234
q2	328	244	248	244
q3	3094	2984	3026	2984
q4	2144	1819	1804	1804
q5	5790	5742	5795	5742
q6	239	142	149	142
q7	2283	1846	1769	1769
q8	3400	3548	3510	3510
q9	9003	9030	8882	8882
q10	3617	3568	3554	3554
q11	615	505	507	505
q12	865	621	636	621
q13	9404	3244	3162	3162
q14	327	287	272	272
q15	568	530	525	525
q16	693	629	671	629
q17	1852	1644	1624	1624
q18	8450	7619	7650	7619
q19	1731	1446	1620	1446
q20	2168	1872	1898	1872
q21	5650	5525	5535	5525
q22	1165	1056	1085	1056
Total cold run time: 70645 ms
Total hot run time: 60721 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192453 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f3a4f93464258ed3ed4227bd70a376a30eec0076, data reload: false

query1	901	396	407	396
query2	6230	2074	2014	2014
query3	8690	210	203	203
query4	34286	23454	23468	23454
query5	3315	476	487	476
query6	269	168	165	165
query7	4199	306	306	306
query8	280	220	226	220
query9	9427	2696	2707	2696
query10	467	282	276	276
query11	17696	15313	15282	15282
query12	154	104	96	96
query13	1581	455	454	454
query14	8915	7499	7423	7423
query15	226	166	167	166
query16	7869	479	469	469
query17	1632	609	586	586
query18	2101	356	320	320
query19	247	163	159	159
query20	121	111	111	111
query21	215	112	108	108
query22	4645	4555	4358	4358
query23	35049	34024	34201	34024
query24	11263	2789	2957	2789
query25	653	422	419	419
query26	1342	163	168	163
query27	2669	306	301	301
query28	7767	2449	2440	2440
query29	905	438	439	438
query30	266	164	153	153
query31	1035	794	818	794
query32	97	53	58	53
query33	761	317	317	317
query34	898	500	503	500
query35	877	731	724	724
query36	1100	961	978	961
query37	152	97	85	85
query38	3935	3881	3863	3863
query39	1478	1528	1433	1433
query40	262	98	103	98
query41	47	45	45	45
query42	115	102	102	102
query43	536	503	495	495
query44	1300	845	826	826
query45	200	165	164	164
query46	1153	732	749	732
query47	1885	1848	1836	1836
query48	438	347	347	347
query49	959	416	411	411
query50	810	418	424	418
query51	7193	7054	6955	6955
query52	99	86	86	86
query53	257	186	185	185
query54	1214	482	491	482
query55	81	78	77	77
query56	275	273	274	273
query57	1306	1162	1152	1152
query58	225	261	266	261
query59	3055	3119	3036	3036
query60	299	258	278	258
query61	111	105	102	102
query62	861	672	667	667
query63	219	192	190	190
query64	4918	635	596	596
query65	3463	3229	3182	3182
query66	1407	309	310	309
query67	15943	15791	15534	15534
query68	4487	607	571	571
query69	498	297	308	297
query70	1140	1107	1123	1107
query71	372	279	274	274
query72	7316	3972	3959	3959
query73	772	346	351	346
query74	10182	9024	9029	9024
query75	3411	2744	2630	2630
query76	2973	879	908	879
query77	447	313	294	294
query78	10411	9633	9556	9556
query79	1184	612	596	596
query80	2394	438	474	438
query81	581	241	241	241
query82	746	146	135	135
query83	294	139	140	139
query84	280	78	78	78
query85	1516	296	292	292
query86	397	254	310	254
query87	4400	4332	4361	4332
query88	3625	2470	2435	2435
query89	410	295	287	287
query90	2254	184	185	184
query91	146	110	106	106
query92	61	50	50	50
query93	1071	544	557	544
query94	1175	293	279	279
query95	352	255	256	255
query96	617	291	279	279
query97	3233	3137	3098	3098
query98	217	197	192	192
query99	1528	1296	1288	1288
Total cold run time: 300834 ms
Total hot run time: 192453 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.6 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f3a4f93464258ed3ed4227bd70a376a30eec0076, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.03
query3	0.22	0.06	0.06
query4	1.67	0.11	0.10
query5	0.52	0.49	0.51
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.55	0.50	0.47
query10	0.54	0.55	0.53
query11	0.14	0.12	0.11
query12	0.13	0.11	0.11
query13	0.61	0.60	0.61
query14	2.72	2.72	2.72
query15	0.91	0.82	0.84
query16	0.38	0.37	0.37
query17	1.02	1.04	1.00
query18	0.20	0.20	0.20
query19	1.88	1.87	1.92
query20	0.01	0.01	0.01
query21	15.37	0.59	0.57
query22	3.06	2.46	1.72
query23	17.00	1.04	0.91
query24	2.93	1.88	2.00
query25	0.24	0.16	0.09
query26	0.53	0.14	0.13
query27	0.04	0.04	0.05
query28	9.40	1.10	1.07
query29	12.55	3.25	3.27
query30	0.24	0.06	0.06
query31	2.88	0.39	0.37
query32	3.29	0.46	0.45
query33	3.00	3.03	2.99
query34	17.22	4.49	4.51
query35	4.51	4.52	4.52
query36	0.66	0.52	0.49
query37	0.08	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.03	0.02	0.02
Total cold run time: 106.16 s
Total hot run time: 33.6 s

Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Oct 11, 2024
Copy link
Contributor

PR approved by anyone and no changes requested.

@zhangstar333 zhangstar333 merged commit 0cb16b9 into apache:master Oct 14, 2024
24 of 27 checks passed
qzsee pushed a commit to qzsee/incubator-doris that referenced this pull request Oct 16, 2024
…pache#41623)

## Proposed changes

the cross join will call insert_many_from function to combine left table
column as result,
implement it's will reduce much virtual function call.
```
mysql [ssb]>select count(c_custkey) from (select customer.c_custkey from dates cross join customer)t;
+------------------+
| count(c_custkey) |
+------------------+
|       7668000000 |
+------------------+
1 row in set (5.56 sec)
```

```
mysql [ssb]>select count(c_custkey) from (select customer.c_custkey from dates cross join customer)t;
+------------------+
| count(c_custkey) |
+------------------+
|       7668000000 |
+------------------+
1 row in set (3.29 sec)
```

<!--Describe your changes.-->
zhangstar333 added a commit to zhangstar333/incubator-doris that referenced this pull request Nov 21, 2024
…pache#41623)

## Proposed changes

the cross join will call insert_many_from function to combine left table
column as result,
implement it's will reduce much virtual function call.
```
mysql [ssb]>select count(c_custkey) from (select customer.c_custkey from dates cross join customer)t;
+------------------+
| count(c_custkey) |
+------------------+
|       7668000000 |
+------------------+
1 row in set (5.56 sec)
```

```
mysql [ssb]>select count(c_custkey) from (select customer.c_custkey from dates cross join customer)t;
+------------------+
| count(c_custkey) |
+------------------+
|       7668000000 |
+------------------+
1 row in set (3.29 sec)
```

<!--Describe your changes.-->
zhangstar333 added a commit to zhangstar333/incubator-doris that referenced this pull request Nov 21, 2024
…pache#41623)

## Proposed changes

the cross join will call insert_many_from function to combine left table
column as result,
implement it's will reduce much virtual function call.
```
mysql [ssb]>select count(c_custkey) from (select customer.c_custkey from dates cross join customer)t;
+------------------+
| count(c_custkey) |
+------------------+
|       7668000000 |
+------------------+
1 row in set (5.56 sec)
```

```
mysql [ssb]>select count(c_custkey) from (select customer.c_custkey from dates cross join customer)t;
+------------------+
| count(c_custkey) |
+------------------+
|       7668000000 |
+------------------+
1 row in set (3.29 sec)
```

<!--Describe your changes.-->
zhangstar333 added a commit that referenced this pull request Nov 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.4-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants