[fix](Outfile) Fix the data type mapping for complex types in Doris to the ORC and Parquet file formats. #44041

BePPPower · 2024-11-15T10:02:27Z

What problem does this PR solve?

Problem Summary:

As before, the behavior of exporting of complex data types in Doris is as follows:

	orc type	parquet type	csv
bitmap	string	Not Supported	Not Supported
quantile_state	Not Supported	Not Supported	Not Supported
hll	string	string	invisible string
jsonb	Not Supported	string	string
variant	Not Supported	string	string

What's more, there are some issues when exporting complex data types to the ORC file format.

This PR does two things:

Fix the problem with exporting complex data types from Doris.
Support exporting these three complex types to both the ORC and the Parquet file format.

	orc type	parquet type	csv
bitmap	binary	binary	"NULL"
quantile_state	binary	binary	"NULL"
hll	binary	binary	"NULL"
jsonb	string	string	string
variant	string	string	string

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes. [fix](outfile) Fix the data type mapping for complex types in Doris to the ORC and Parquet file formats. doris-website#1370

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-11-15T10:02:32Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

BePPPower · 2024-11-15T10:02:38Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-11-15T10:08:04Z

be/src/vec/data_types/serde/data_type_quantilestate_serde.h

@@ -17,6 +17,7 @@

 #pragma once

+#include <arrow/array/builder_binary.h>


warning: 'arrow/array/builder_binary.h' file not found [clang-diagnostic-error]

#include <arrow/array/builder_binary.h> ^

doris-robot · 2024-11-15T10:41:53Z

TeamCity be ut coverage result:
Function Coverage: 38.00% (9901/26057)
Line Coverage: 29.16% (82698/283600)
Region Coverage: 28.29% (42481/150152)
Branch Coverage: 24.87% (21541/86622)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ee5770b3619e2c1a7754f00864c5eb11e750b1f3_ee5770b3619e2c1a7754f00864c5eb11e750b1f3/report/index.html

BePPPower · 2024-11-19T12:31:25Z

run buildall

BePPPower · 2024-11-20T07:30:23Z

run buildall

doris-robot · 2024-11-20T08:50:15Z

TeamCity be ut coverage result:
Function Coverage: 38.03% (9902/26039)
Line Coverage: 29.20% (82819/283663)
Region Coverage: 28.33% (42550/150175)
Branch Coverage: 24.89% (21565/86646)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b113fc1e1319fd688b4d8dbad4fcb094b40f3510_b113fc1e1319fd688b4d8dbad4fcb094b40f3510/report/index.html

BePPPower · 2024-11-20T14:24:36Z

run buildall

doris-robot · 2024-11-20T15:01:26Z

TeamCity be ut coverage result:
Function Coverage: 38.02% (9900/26039)
Line Coverage: 29.20% (82820/283663)
Region Coverage: 28.32% (42536/150175)
Branch Coverage: 24.88% (21556/86646)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b113fc1e1319fd688b4d8dbad4fcb094b40f3510_b113fc1e1319fd688b4d8dbad4fcb094b40f3510/report/index.html

doris-robot · 2024-11-20T15:04:26Z

TPC-H: Total hot run time: 39950 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b113fc1e1319fd688b4d8dbad4fcb094b40f3510, data reload: false

------ Round 1 ----------------------------------
q1	17575	7423	7285	7285
q2	2044	177	169	169
q3	10542	1042	1096	1042
q4	10217	736	763	736
q5	7593	2731	2735	2731
q6	239	148	146	146
q7	982	614	599	599
q8	9251	1822	1914	1822
q9	6641	6339	6423	6339
q10	7051	2309	2329	2309
q11	459	266	251	251
q12	422	222	220	220
q13	18031	3052	3099	3052
q14	243	207	218	207
q15	572	530	508	508
q16	664	584	599	584
q17	988	536	594	536
q18	7335	6781	6790	6781
q19	1333	978	1067	978
q20	487	183	180	180
q21	4015	3253	3162	3162
q22	371	325	313	313
Total cold run time: 107055 ms
Total hot run time: 39950 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7350	7362	7359	7359
q2	324	228	230	228
q3	2905	2863	3024	2863
q4	2032	1839	1829	1829
q5	5728	5725	5707	5707
q6	227	142	147	142
q7	2262	1801	1876	1801
q8	3428	3652	3550	3550
q9	9124	9165	9236	9165
q10	3658	3646	3594	3594
q11	614	544	536	536
q12	861	650	664	650
q13	16719	3231	3201	3201
q14	310	283	283	283
q15	579	520	532	520
q16	693	677	661	661
q17	1903	1648	1654	1648
q18	8624	7965	7886	7886
q19	1745	1588	1505	1505
q20	2146	1891	1873	1873
q21	5588	5468	5263	5263
q22	639	589	592	589
Total cold run time: 77459 ms
Total hot run time: 60853 ms

doris-robot · 2024-11-20T15:17:05Z

TPC-DS: Total hot run time: 196086 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b113fc1e1319fd688b4d8dbad4fcb094b40f3510, data reload: false

query1	1264	967	944	944
query2	6260	2141	2066	2066
query3	10812	4113	4005	4005
query4	67622	29043	23985	23985
query5	4960	466	454	454
query6	411	187	195	187
query7	5553	289	285	285
query8	305	211	234	211
query9	8725	2678	2655	2655
query10	446	248	248	248
query11	17458	15245	15971	15245
query12	152	108	107	107
query13	1510	450	460	450
query14	10710	7825	7357	7357
query15	211	187	197	187
query16	7056	480	482	480
query17	1079	584	592	584
query18	1777	295	294	294
query19	219	155	161	155
query20	116	111	109	109
query21	202	104	100	100
query22	4755	4258	4389	4258
query23	34590	34004	34167	34004
query24	5397	2483	2468	2468
query25	482	390	398	390
query26	658	148	153	148
query27	1711	281	284	281
query28	4328	2475	2396	2396
query29	670	424	413	413
query30	209	147	146	146
query31	1011	802	829	802
query32	68	61	55	55
query33	389	268	279	268
query34	930	518	498	498
query35	846	735	706	706
query36	1064	938	953	938
query37	129	95	78	78
query38	4340	4145	4213	4145
query39	1454	1440	1407	1407
query40	196	101	96	96
query41	44	42	41	41
query42	106	96	99	96
query43	532	513	512	512
query44	1201	829	808	808
query45	179	163	167	163
query46	1145	708	709	708
query47	1940	1854	1838	1838
query48	402	305	322	305
query49	714	386	398	386
query50	815	387	401	387
query51	7292	7184	7194	7184
query52	97	92	95	92
query53	262	177	179	177
query54	524	399	388	388
query55	77	74	79	74
query56	252	247	243	243
query57	1292	1173	1148	1148
query58	226	210	230	210
query59	3297	3041	2930	2930
query60	270	248	253	248
query61	107	140	106	106
query62	803	680	689	680
query63	219	188	187	187
query64	1335	655	627	627
query65	3260	3206	3301	3206
query66	701	314	365	314
query67	15921	15922	15664	15664
query68	4067	565	555	555
query69	430	255	275	255
query70	1173	1094	1156	1094
query71	366	258	249	249
query72	6460	4028	3882	3882
query73	768	354	356	354
query74	10200	8989	9012	8989
query75	3401	2700	2697	2697
query76	2036	986	1036	986
query77	508	297	304	297
query78	10599	9475	9370	9370
query79	2360	598	608	598
query80	1407	423	457	423
query81	532	227	236	227
query82	1265	122	119	119
query83	180	155	152	152
query84	295	74	67	67
query85	975	304	357	304
query86	409	297	287	287
query87	4730	4525	4608	4525
query88	3395	2221	2169	2169
query89	430	286	302	286
query90	1971	184	182	182
query91	137	104	103	103
query92	65	49	49	49
query93	3272	548	539	539
query94	808	283	273	273
query95	345	249	250	249
query96	622	284	289	284
query97	2858	2730	2718	2718
query98	215	200	203	200
query99	1632	1314	1277	1277
Total cold run time: 320999 ms
Total hot run time: 196086 ms

doris-robot · 2024-11-20T15:22:34Z

ClickBench: Total hot run time: 33.37 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b113fc1e1319fd688b4d8dbad4fcb094b40f3510, data reload: false

query1	0.04	0.03	0.02
query2	0.06	0.05	0.04
query3	0.23	0.08	0.07
query4	1.62	0.10	0.10
query5	0.43	0.41	0.38
query6	1.13	0.65	0.65
query7	0.02	0.02	0.02
query8	0.03	0.03	0.03
query9	0.58	0.51	0.49
query10	0.55	0.54	0.57
query11	0.15	0.11	0.10
query12	0.14	0.11	0.11
query13	0.61	0.59	0.60
query14	2.71	2.81	2.76
query15	0.89	0.82	0.83
query16	0.38	0.40	0.37
query17	1.06	1.06	1.03
query18	0.20	0.19	0.21
query19	1.98	1.86	1.96
query20	0.02	0.01	0.01
query21	15.37	0.63	0.60
query22	2.33	1.62	2.07
query23	16.95	1.08	0.82
query24	2.92	2.07	2.04
query25	0.22	0.08	0.08
query26	0.64	0.14	0.14
query27	0.05	0.05	0.04
query28	8.95	1.11	1.07
query29	12.57	3.22	3.24
query30	0.26	0.06	0.07
query31	2.88	0.38	0.38
query32	3.27	0.47	0.48
query33	2.99	3.01	3.04
query34	17.06	4.48	4.41
query35	4.51	4.47	4.50
query36	0.66	0.48	0.47
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 104.93 s
Total hot run time: 33.37 s

BePPPower · 2024-11-21T01:56:09Z

run buildall

doris-robot · 2024-11-21T02:33:31Z

TeamCity be ut coverage result:
Function Coverage: 38.03% (9902/26039)
Line Coverage: 29.20% (82843/283663)
Region Coverage: 28.33% (42541/150175)
Branch Coverage: 24.89% (21566/86646)
Coverage Report: http://coverage.selectdb-in.cc/coverage/21abf3dbe6a856b1e962e565300f44b98efa32a0_21abf3dbe6a856b1e962e565300f44b98efa32a0/report/index.html

doris-robot · 2024-11-21T02:36:08Z

TPC-H: Total hot run time: 40632 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 21abf3dbe6a856b1e962e565300f44b98efa32a0, data reload: false

------ Round 1 ----------------------------------
q1	17608	7477	7284	7284
q2	2044	171	158	158
q3	10823	1081	1161	1081
q4	10514	734	712	712
q5	7967	2783	2694	2694
q6	248	153	151	151
q7	981	642	623	623
q8	9243	1795	1939	1795
q9	6594	6414	6474	6414
q10	7052	2254	2292	2254
q11	470	272	253	253
q12	418	225	225	225
q13	18106	3120	3073	3073
q14	244	212	211	211
q15	569	548	520	520
q16	647	585	595	585
q17	985	699	616	616
q18	8593	7207	7313	7207
q19	2541	1053	1059	1053
q20	746	187	183	183
q21	4023	3226	3229	3226
q22	384	314	316	314
Total cold run time: 110800 ms
Total hot run time: 40632 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7295	7283	7250	7250
q2	327	229	226	226
q3	3051	2976	2954	2954
q4	2116	1815	1813	1813
q5	5604	5632	5725	5632
q6	234	151	146	146
q7	2241	1822	1795	1795
q8	3400	3532	3392	3392
q9	8938	8883	8790	8790
q10	3631	3531	3572	3531
q11	582	517	516	516
q12	843	618	612	612
q13	16871	3265	3276	3265
q14	302	291	274	274
q15	574	524	542	524
q16	714	657	641	641
q17	1851	1630	1608	1608
q18	8266	7820	7708	7708
q19	1689	1616	1684	1616
q20	2135	1905	1922	1905
q21	5730	5376	5432	5376
q22	638	572	565	565
Total cold run time: 77032 ms
Total hot run time: 60139 ms

doris-robot · 2024-11-21T02:48:45Z

TPC-DS: Total hot run time: 195881 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 21abf3dbe6a856b1e962e565300f44b98efa32a0, data reload: false

query1	1234	759	813	759
query2	6249	2174	2203	2174
query3	10831	4034	3816	3816
query4	67412	29506	23627	23627
query5	4858	474	450	450
query6	408	179	184	179
query7	5633	303	294	294
query8	320	231	229	229
query9	9111	2677	2666	2666
query10	457	243	248	243
query11	17519	15262	15952	15262
query12	157	119	117	117
query13	1587	426	452	426
query14	10502	7431	7521	7431
query15	198	178	177	177
query16	6997	471	427	427
query17	1078	545	547	545
query18	1821	282	293	282
query19	197	151	145	145
query20	110	105	107	105
query21	211	100	101	100
query22	4536	4286	4488	4286
query23	34422	34301	33907	33907
query24	5435	2537	2485	2485
query25	476	382	374	374
query26	636	151	146	146
query27	1720	292	288	288
query28	4221	2465	2432	2432
query29	651	414	429	414
query30	207	147	142	142
query31	988	804	811	804
query32	66	54	54	54
query33	391	280	276	276
query34	902	500	512	500
query35	825	712	713	712
query36	1110	947	983	947
query37	114	74	75	74
query38	4346	4372	4175	4175
query39	1451	1447	1445	1445
query40	193	99	100	99
query41	46	42	45	42
query42	108	95	95	95
query43	562	503	504	503
query44	1144	809	800	800
query45	194	159	163	159
query46	1119	727	674	674
query47	1966	1843	1873	1843
query48	413	305	317	305
query49	716	392	380	380
query50	818	376	397	376
query51	7326	7116	7165	7116
query52	105	88	92	88
query53	252	173	178	173
query54	524	408	386	386
query55	78	71	85	71
query56	247	241	239	239
query57	1265	1182	1165	1165
query58	225	222	217	217
query59	3408	3427	3113	3113
query60	271	258	241	241
query61	129	111	111	111
query62	784	676	666	666
query63	213	185	183	183
query64	1385	665	711	665
query65	3302	3233	3211	3211
query66	721	313	318	313
query67	16167	16067	15717	15717
query68	3876	584	566	566
query69	429	258	253	253
query70	1213	1148	1139	1139
query71	350	249	252	249
query72	6405	4147	4074	4074
query73	765	357	350	350
query74	10316	8912	9012	8912
query75	3421	2720	2688	2688
query76	1852	1060	1008	1008
query77	464	281	282	281
query78	10529	9498	9462	9462
query79	1812	587	604	587
query80	1101	430	436	430
query81	523	233	234	233
query82	1321	122	120	120
query83	262	151	156	151
query84	287	68	71	68
query85	903	296	332	296
query86	332	307	301	301
query87	4776	4556	4605	4556
query88	3409	2235	2155	2155
query89	420	290	299	290
query90	2016	186	191	186
query91	140	102	104	102
query92	69	51	52	51
query93	1977	528	536	528
query94	834	299	266	266
query95	351	246	248	246
query96	627	274	274	274
query97	2913	2675	2702	2675
query98	210	201	208	201
query99	1705	1317	1355	1317
Total cold run time: 318676 ms
Total hot run time: 195881 ms

doris-robot · 2024-11-21T02:54:15Z

ClickBench: Total hot run time: 33.54 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 21abf3dbe6a856b1e962e565300f44b98efa32a0, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.08	0.07
query4	1.62	0.10	0.10
query5	0.41	0.42	0.39
query6	1.14	0.65	0.66
query7	0.03	0.02	0.02
query8	0.04	0.04	0.03
query9	0.60	0.48	0.50
query10	0.55	0.56	0.56
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.61	0.61
query14	2.83	2.74	2.74
query15	0.90	0.83	0.83
query16	0.38	0.39	0.39
query17	1.05	1.00	1.05
query18	0.20	0.20	0.20
query19	1.98	1.87	1.98
query20	0.02	0.01	0.01
query21	15.37	0.61	0.60
query22	2.49	2.45	1.77
query23	17.32	0.85	0.68
query24	3.29	2.10	1.88
query25	0.18	0.21	0.18
query26	0.48	0.15	0.14
query27	0.05	0.04	0.05
query28	8.80	1.10	1.08
query29	12.58	3.28	3.26
query30	0.24	0.06	0.06
query31	2.88	0.38	0.39
query32	3.27	0.47	0.46
query33	3.00	3.01	3.05
query34	17.01	4.50	4.49
query35	4.61	4.55	4.49
query36	0.64	0.49	0.49
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.02
query40	0.15	0.12	0.12
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.02	0.02
Total cold run time: 105.65 s
Total hot run time: 33.54 s

morningman · 2024-11-21T06:28:27Z

be/src/vec/data_types/serde/data_type_hll_serde.cpp

@@ -48,28 +49,17 @@ Status DataTypeHLLSerDe::serialize_column_to_json(const IColumn& column, int64_t
 Status DataTypeHLLSerDe::serialize_one_cell_to_json(const IColumn& column, int64_t row_num,
                                                    BufferWritable& bw,
                                                    FormatOptions& options) const {
-    if (!options._output_object_data) {


only export use this option _output_object_data?

BePPPower · 2024-11-25T06:30:57Z

run buildall

doris-robot · 2024-11-25T08:15:21Z

TPC-H: Total hot run time: 40135 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f50c44b5bb8701c5a50b57490a44f7ce07cb40e3, data reload: false

------ Round 1 ----------------------------------
q1	17649	7535	7258	7258
q2	2046	171	181	171
q3	10865	1131	1251	1131
q4	10385	752	692	692
q5	7631	2771	2784	2771
q6	242	151	147	147
q7	980	639	602	602
q8	9256	1884	1935	1884
q9	6685	6437	6421	6421
q10	7019	2322	2325	2322
q11	468	253	266	253
q12	424	219	223	219
q13	17771	3017	3043	3017
q14	243	220	215	215
q15	582	526	536	526
q16	679	598	588	588
q17	993	598	607	598
q18	7461	6677	6688	6677
q19	1337	1020	1074	1020
q20	470	181	185	181
q21	4070	3132	3149	3132
q22	375	310	311	310
Total cold run time: 107631 ms
Total hot run time: 40135 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7251	7274	7236	7236
q2	325	232	230	230
q3	2966	2845	2946	2845
q4	2104	1827	1883	1827
q5	5692	5707	5696	5696
q6	213	140	143	140
q7	2173	1820	1757	1757
q8	3440	3559	3603	3559
q9	8929	8910	9014	8910
q10	3580	3577	3580	3577
q11	603	507	504	504
q12	833	613	606	606
q13	11631	3261	3343	3261
q14	303	265	272	265
q15	584	524	521	521
q16	698	650	633	633
q17	1883	1651	1623	1623
q18	8281	7881	7687	7687
q19	1719	1544	1573	1544
q20	2154	1901	1875	1875
q21	5665	5411	5468	5411
q22	628	599	576	576
Total cold run time: 71655 ms
Total hot run time: 60283 ms

doris-robot · 2024-11-25T08:15:57Z

TeamCity be ut coverage result:
Function Coverage: 38.31% (9981/26051)
Line Coverage: 29.43% (83536/283884)
Region Coverage: 28.57% (42990/150478)
Branch Coverage: 25.16% (21843/86802)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f50c44b5bb8701c5a50b57490a44f7ce07cb40e3_f50c44b5bb8701c5a50b57490a44f7ce07cb40e3/report/index.html

doris-robot · 2024-11-25T08:28:00Z

TPC-DS: Total hot run time: 196626 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f50c44b5bb8701c5a50b57490a44f7ce07cb40e3, data reload: false

query1	1270	931	905	905
query2	6253	2079	2027	2027
query3	10790	3887	4100	3887
query4	67610	28398	23430	23430
query5	4912	439	430	430
query6	388	168	164	164
query7	5535	293	296	293
query8	320	233	237	233
query9	8855	2608	2601	2601
query10	427	251	247	247
query11	17194	15295	15816	15295
query12	147	101	103	101
query13	1469	425	439	425
query14	10942	7728	7577	7577
query15	213	203	191	191
query16	7218	487	500	487
query17	1069	582	592	582
query18	1904	309	315	309
query19	199	168	160	160
query20	127	110	114	110
query21	209	103	108	103
query22	4754	4673	4446	4446
query23	34950	34489	34381	34381
query24	5643	2628	2539	2539
query25	501	388	383	383
query26	664	150	152	150
query27	2050	280	294	280
query28	4485	2474	2452	2452
query29	689	442	432	432
query30	223	154	158	154
query31	1006	826	861	826
query32	70	53	53	53
query33	436	293	303	293
query34	937	510	514	510
query35	885	754	747	747
query36	1098	962	979	962
query37	125	73	73	73
query38	4589	4316	4385	4316
query39	1552	1502	1469	1469
query40	205	121	97	97
query41	46	46	46	46
query42	111	96	97	96
query43	540	518	489	489
query44	1162	828	819	819
query45	183	168	170	168
query46	1172	710	719	710
query47	2043	1879	1926	1879
query48	402	321	319	319
query49	733	403	385	385
query50	838	386	392	386
query51	7463	7189	7086	7086
query52	100	85	90	85
query53	259	175	178	175
query54	499	396	379	379
query55	78	75	75	75
query56	245	234	244	234
query57	1293	1165	1140	1140
query58	216	223	208	208
query59	3119	2981	3085	2981
query60	272	263	236	236
query61	110	109	115	109
query62	813	666	663	663
query63	215	187	192	187
query64	1484	658	674	658
query65	3308	3225	3203	3203
query66	717	301	322	301
query67	16176	15538	15525	15525
query68	3791	580	557	557
query69	440	259	254	254
query70	1140	1164	1150	1150
query71	366	242	241	241
query72	6427	3967	4156	3967
query73	759	349	348	348
query74	10160	9021	8914	8914
query75	3386	2678	2699	2678
query76	1828	1026	1024	1024
query77	503	272	262	262
query78	10585	9444	9411	9411
query79	1620	596	608	596
query80	870	431	425	425
query81	504	244	228	228
query82	1296	124	113	113
query83	253	146	156	146
query84	286	78	66	66
query85	921	297	343	297
query86	339	311	301	301
query87	4684	4678	4687	4678
query88	3288	2188	2154	2154
query89	421	292	306	292
query90	2017	194	180	180
query91	140	103	104	103
query92	63	50	50	50
query93	1902	538	537	537
query94	789	292	294	292
query95	342	246	247	246
query96	621	280	284	280
query97	2950	2678	2693	2678
query98	225	196	193	193
query99	1581	1319	1303	1303
Total cold run time: 320007 ms
Total hot run time: 196626 ms

doris-robot · 2024-11-25T08:33:29Z

ClickBench: Total hot run time: 32.94 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f50c44b5bb8701c5a50b57490a44f7ce07cb40e3, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.03
query3	0.24	0.07	0.07
query4	1.62	0.10	0.10
query5	0.43	0.40	0.43
query6	1.14	0.66	0.65
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.58	0.51	0.51
query10	0.54	0.55	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.60	0.59	0.59
query14	2.81	2.85	2.74
query15	0.89	0.83	0.83
query16	0.39	0.39	0.40
query17	0.97	1.06	1.08
query18	0.23	0.21	0.19
query19	1.94	1.85	2.03
query20	0.01	0.01	0.01
query21	15.52	0.57	0.57
query22	3.12	2.49	1.83
query23	17.04	0.96	0.81
query24	2.55	1.32	1.91
query25	0.31	0.15	0.05
query26	0.56	0.13	0.13
query27	0.04	0.04	0.04
query28	9.96	1.11	1.08
query29	12.57	3.21	3.18
query30	0.25	0.07	0.06
query31	2.86	0.38	0.39
query32	3.29	0.46	0.47
query33	2.98	3.02	3.01
query34	16.82	4.49	4.46
query35	4.51	4.56	4.51
query36	0.67	0.50	0.50
query37	0.08	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.03	0.02
query40	0.17	0.14	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.35 s
Total hot run time: 32.94 s

BePPPower · 2024-11-26T04:00:00Z

run buildall

doris-robot · 2024-11-26T05:34:33Z

TeamCity be ut coverage result:
Function Coverage: 38.37% (9979/26007)
Line Coverage: 29.45% (83494/283519)
Region Coverage: 28.58% (42983/150396)
Branch Coverage: 25.18% (21828/86680)
Coverage Report: http://coverage.selectdb-in.cc/coverage/005a47c98156457559d99b12be67abfe0fea80cf_005a47c98156457559d99b12be67abfe0fea80cf/report/index.html

doris-robot · 2024-11-26T05:35:29Z

TPC-H: Total hot run time: 40156 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 005a47c98156457559d99b12be67abfe0fea80cf, data reload: false

------ Round 1 ----------------------------------
q1	17882	7734	7273	7273
q2	2051	177	167	167
q3	10659	1096	1145	1096
q4	10358	720	805	720
q5	7642	2821	2718	2718
q6	231	147	152	147
q7	973	626	608	608
q8	9260	1860	1952	1860
q9	6668	6463	6415	6415
q10	6992	2276	2357	2276
q11	450	275	262	262
q12	417	215	223	215
q13	17762	3072	3100	3072
q14	243	214	211	211
q15	581	519	516	516
q16	666	595	595	595
q17	987	499	638	499
q18	7571	6636	6777	6636
q19	1334	1039	1075	1039
q20	479	183	196	183
q21	4022	3344	3337	3337
q22	387	314	311	311
Total cold run time: 107615 ms
Total hot run time: 40156 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7236	7273	7232	7232
q2	323	233	231	231
q3	2892	2912	2937	2912
q4	2104	1823	1843	1823
q5	5659	5717	5644	5644
q6	226	146	149	146
q7	2289	1841	1809	1809
q8	3451	3583	3587	3583
q9	8893	8856	8956	8856
q10	3612	3590	3544	3544
q11	604	516	507	507
q12	826	602	598	598
q13	11244	3253	3248	3248
q14	307	276	268	268
q15	581	519	521	519
q16	696	641	670	641
q17	1852	1660	1642	1642
q18	8412	7989	7765	7765
q19	3442	1626	1561	1561
q20	2130	1872	1866	1866
q21	5682	5456	5353	5353
q22	634	574	617	574
Total cold run time: 73095 ms
Total hot run time: 60322 ms

doris-robot · 2024-11-26T05:48:10Z

TPC-DS: Total hot run time: 196842 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 005a47c98156457559d99b12be67abfe0fea80cf, data reload: false

query1	1282	937	958	937
query2	6237	2092	2073	2073
query3	10769	3990	3944	3944
query4	67428	29000	23482	23482
query5	5075	465	453	453
query6	410	189	192	189
query7	5688	309	290	290
query8	316	238	237	237
query9	9393	2666	2648	2648
query10	472	267	270	267
query11	17529	15281	15765	15281
query12	150	105	109	105
query13	1570	411	424	411
query14	10799	7695	7292	7292
query15	234	188	213	188
query16	7228	455	434	434
query17	1223	570	602	570
query18	1809	314	326	314
query19	224	161	168	161
query20	125	114	118	114
query21	216	125	108	108
query22	4785	4364	4654	4364
query23	34693	34203	35006	34203
query24	5503	2505	2601	2505
query25	501	397	403	397
query26	672	153	148	148
query27	1844	291	283	283
query28	4358	2499	2459	2459
query29	701	422	436	422
query30	232	150	149	149
query31	1004	857	847	847
query32	69	57	60	57
query33	470	294	302	294
query34	950	523	519	519
query35	858	753	740	740
query36	1124	967	963	963
query37	128	85	74	74
query38	4495	4417	4357	4357
query39	1512	1599	1490	1490
query40	215	106	104	104
query41	45	42	44	42
query42	109	97	97	97
query43	538	505	520	505
query44	1245	844	848	844
query45	192	177	173	173
query46	1176	703	735	703
query47	2046	1939	1925	1925
query48	440	328	342	328
query49	754	420	419	419
query50	868	402	419	402
query51	7436	7188	7188	7188
query52	96	86	86	86
query53	274	183	185	183
query54	514	397	394	394
query55	79	77	75	75
query56	270	250	242	242
query57	1281	1170	1139	1139
query58	221	218	216	216
query59	3314	3108	2990	2990
query60	264	250	239	239
query61	115	112	138	112
query62	801	689	669	669
query63	210	187	202	187
query64	1448	679	652	652
query65	3351	3208	3192	3192
query66	705	302	322	302
query67	16224	15778	15702	15702
query68	4099	552	569	552
query69	414	267	258	258
query70	1167	1072	1135	1072
query71	345	262	252	252
query72	6195	4016	4003	4003
query73	784	365	365	365
query74	10234	9103	9091	9091
query75	3371	2668	2670	2668
query76	1980	1095	1115	1095
query77	484	278	272	272
query78	10470	9375	9298	9298
query79	1498	598	581	581
query80	871	433	439	433
query81	517	234	224	224
query82	1265	119	116	116
query83	260	179	148	148
query84	275	69	73	69
query85	892	303	376	303
query86	340	294	298	294
query87	4778	4512	4724	4512
query88	3603	2227	2197	2197
query89	417	294	303	294
query90	1994	188	193	188
query91	137	103	103	103
query92	69	52	51	51
query93	1873	541	544	541
query94	788	310	288	288
query95	359	257	253	253
query96	605	297	272	272
query97	2845	2686	2696	2686
query98	221	191	187	187
query99	1600	1340	1297	1297
Total cold run time: 321057 ms
Total hot run time: 196842 ms

doris-robot · 2024-11-26T05:53:39Z

ClickBench: Total hot run time: 32.21 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 005a47c98156457559d99b12be67abfe0fea80cf, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.22	0.08	0.06
query4	1.63	0.10	0.10
query5	0.43	0.44	0.41
query6	1.18	0.68	0.64
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.59	0.50	0.52
query10	0.56	0.54	0.55
query11	0.14	0.10	0.10
query12	0.13	0.12	0.12
query13	0.61	0.62	0.60
query14	2.72	2.86	2.87
query15	0.90	0.83	0.82
query16	0.38	0.39	0.38
query17	1.04	1.06	1.03
query18	0.23	0.21	0.21
query19	1.89	1.86	1.97
query20	0.01	0.01	0.01
query21	15.39	0.56	0.58
query22	2.82	1.95	1.51
query23	17.06	0.92	0.85
query24	2.62	1.16	0.66
query25	0.11	0.21	0.19
query26	0.38	0.13	0.15
query27	0.05	0.05	0.04
query28	11.15	1.10	1.08
query29	12.59	3.25	3.25
query30	0.25	0.06	0.06
query31	2.86	0.37	0.38
query32	3.30	0.48	0.47
query33	3.02	3.10	2.99
query34	17.16	4.47	4.44
query35	4.45	4.50	4.50
query36	0.65	0.48	0.47
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.16	0.13	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 107.17 s
Total hot run time: 32.21 s

BePPPower · 2024-11-27T02:16:05Z

run p0

BePPPower · 2024-11-27T03:19:05Z

run p0

BePPPower · 2024-11-27T09:24:46Z

run buildall

doris-robot · 2024-11-27T11:00:41Z

TeamCity be ut coverage result:
Function Coverage: 38.35% (9977/26016)
Line Coverage: 29.44% (83500/283670)
Region Coverage: 28.57% (42989/150472)
Branch Coverage: 25.18% (21838/86730)
Coverage Report: http://coverage.selectdb-in.cc/coverage/74a1c2c2854594bad66b5b52613b687d082c45ae_74a1c2c2854594bad66b5b52613b687d082c45ae/report/index.html

doris-robot · 2024-11-27T11:09:01Z

TPC-H: Total hot run time: 40023 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 74a1c2c2854594bad66b5b52613b687d082c45ae, data reload: false

------ Round 1 ----------------------------------
q1	17742	7564	7316	7316
q2	2067	182	190	182
q3	10875	1107	1231	1107
q4	10413	722	829	722
q5	7603	2765	2761	2761
q6	239	148	149	148
q7	972	640	612	612
q8	9243	1866	1992	1866
q9	6668	6391	6333	6333
q10	6974	2307	2299	2299
q11	473	261	255	255
q12	424	215	214	214
q13	17784	3022	3055	3022
q14	248	217	210	210
q15	575	536	514	514
q16	697	587	593	587
q17	995	560	542	542
q18	7389	6678	6698	6678
q19	1327	997	1077	997
q20	469	184	182	182
q21	4017	3163	3164	3163
q22	374	315	313	313
Total cold run time: 107568 ms
Total hot run time: 40023 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7332	7240	7254	7240
q2	329	229	230	229
q3	2938	2862	2932	2862
q4	2080	1845	1907	1845
q5	5657	5665	5695	5665
q6	224	145	141	141
q7	2265	1772	1825	1772
q8	3460	3572	3498	3498
q9	8802	8960	8867	8867
q10	3617	3540	3585	3540
q11	606	521	522	521
q12	818	630	612	612
q13	11566	3310	3222	3222
q14	308	267	265	265
q15	573	522	530	522
q16	706	641	634	634
q17	1861	1626	1633	1626
q18	8458	7923	7740	7740
q19	3275	1651	1442	1442
q20	2103	1855	1887	1855
q21	5501	5431	5501	5431
q22	650	567	566	566
Total cold run time: 73129 ms
Total hot run time: 60095 ms

doris-robot · 2024-11-27T11:21:41Z

TPC-DS: Total hot run time: 196145 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 74a1c2c2854594bad66b5b52613b687d082c45ae, data reload: false

query1	1289	952	962	952
query2	6253	2118	2014	2014
query3	10942	4124	4065	4065
query4	67303	29319	23660	23660
query5	5236	479	440	440
query6	430	191	201	191
query7	5823	297	288	288
query8	322	213	208	208
query9	9262	2731	2706	2706
query10	456	245	238	238
query11	17749	15240	15797	15240
query12	158	105	98	98
query13	1587	425	440	425
query14	11185	7658	7145	7145
query15	211	187	182	182
query16	7171	467	476	467
query17	1314	593	587	587
query18	1762	323	307	307
query19	219	160	155	155
query20	124	122	114	114
query21	214	119	112	112
query22	4604	4595	4325	4325
query23	34796	34264	34536	34264
query24	5304	2450	2451	2450
query25	512	398	397	397
query26	639	148	144	144
query27	1789	282	292	282
query28	4335	2500	2506	2500
query29	668	418	445	418
query30	216	154	148	148
query31	1012	859	837	837
query32	65	52	57	52
query33	417	276	278	276
query34	947	514	566	514
query35	889	759	766	759
query36	1095	905	971	905
query37	120	75	73	73
query38	4449	4416	4548	4416
query39	1519	1453	1468	1453
query40	196	98	98	98
query41	45	41	47	41
query42	111	98	100	98
query43	545	501	500	500
query44	1173	828	826	826
query45	185	163	168	163
query46	1154	686	684	684
query47	2034	1910	1938	1910
query48	429	324	309	309
query49	778	390	384	384
query50	869	407	392	392
query51	7349	7267	7090	7090
query52	97	90	86	86
query53	249	181	176	176
query54	499	406	383	383
query55	78	72	74	72
query56	249	235	241	235
query57	1311	1201	1142	1142
query58	243	215	208	208
query59	3123	3140	2866	2866
query60	268	248	239	239
query61	111	109	105	105
query62	763	677	655	655
query63	214	184	188	184
query64	1362	654	641	641
query65	3288	3207	3167	3167
query66	635	300	310	300
query67	15925	15667	15567	15567
query68	4137	568	566	566
query69	423	256	252	252
query70	1175	1041	1096	1041
query71	338	252	243	243
query72	6587	3991	3987	3987
query73	756	359	366	359
query74	10143	9115	8939	8939
query75	3383	2668	2675	2668
query76	1888	1082	1086	1082
query77	492	271	277	271
query78	10463	9369	9403	9369
query79	2131	593	592	592
query80	1260	416	417	416
query81	526	236	224	224
query82	826	122	120	120
query83	258	149	148	148
query84	262	71	69	69
query85	1001	290	289	289
query86	385	297	293	293
query87	4755	4516	4537	4516
query88	3825	2223	2196	2196
query89	411	297	283	283
query90	1907	187	180	180
query91	137	104	101	101
query92	65	48	48	48
query93	2078	540	546	540
query94	791	293	300	293
query95	349	249	248	248
query96	614	269	282	269
query97	2892	2703	2690	2690
query98	225	197	196	196
query99	1579	1329	1336	1329
Total cold run time: 321731 ms
Total hot run time: 196145 ms

doris-robot · 2024-11-27T11:27:10Z

ClickBench: Total hot run time: 32 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 74a1c2c2854594bad66b5b52613b687d082c45ae, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.04
query3	0.23	0.08	0.07
query4	1.61	0.10	0.10
query5	0.41	0.41	0.42
query6	1.17	0.66	0.65
query7	0.02	0.01	0.01
query8	0.03	0.04	0.03
query9	0.59	0.52	0.51
query10	0.55	0.56	0.56
query11	0.13	0.11	0.11
query12	0.13	0.11	0.11
query13	0.61	0.59	0.60
query14	2.82	2.75	2.76
query15	0.90	0.82	0.82
query16	0.39	0.37	0.38
query17	1.01	1.03	0.96
query18	0.22	0.21	0.21
query19	1.99	1.74	1.94
query20	0.01	0.01	0.01
query21	15.36	0.58	0.58
query22	2.20	2.14	2.48
query23	17.30	0.87	0.71
query24	2.83	0.28	0.93
query25	0.19	0.11	0.09
query26	0.44	0.15	0.14
query27	0.06	0.05	0.04
query28	11.44	1.09	1.06
query29	12.58	3.26	3.30
query30	0.25	0.06	0.06
query31	2.84	0.38	0.39
query32	3.27	0.46	0.46
query33	2.99	3.01	3.03
query34	17.14	4.44	4.50
query35	4.49	4.53	4.52
query36	0.65	0.50	0.48
query37	0.09	0.05	0.05
query38	0.04	0.03	0.03
query39	0.03	0.02	0.03
query40	0.16	0.12	0.12
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 107.43 s
Total hot run time: 32 s

github-actions · 2024-11-28T15:03:55Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-11-28T15:03:57Z

PR approved by anyone and no changes requested.

…s to the ORC and Parquet file formats (#45122) picked from #44041

github-actions bot reviewed Nov 15, 2024

View reviewed changes

BePPPower force-pushed the fixOutfileComplexTypes branch from 4d80aa1 to 6990720 Compare November 18, 2024 09:32

BePPPower changed the title ~~[fix](Outfile) export the complex types of Doris to binary type of orc and parquet file format~~ [fix](Outfile) Fix the data type mapping for complex types in Doris to the ORC and Parquet file formats. Nov 18, 2024

BePPPower marked this pull request as ready for review November 18, 2024 10:10

BePPPower force-pushed the fixOutfileComplexTypes branch from 3790342 to c48f8dc Compare November 19, 2024 12:31

morningman added dev/3.0.x dev/2.1.x kind/behavior-changed labels Nov 21, 2024

morningman reviewed Nov 21, 2024

View reviewed changes

BePPPower force-pushed the fixOutfileComplexTypes branch from f50c44b to 005a47c Compare November 26, 2024 03:59

fix test

74a1c2c

morningman approved these changes Nov 28, 2024

View reviewed changes

CalvinKirs approved these changes Nov 28, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 28, 2024

github-actions bot added the reviewed label Nov 28, 2024

morningman merged commit 5641c2d into apache:master Nov 28, 2024
23 of 25 checks passed

github-actions bot added dev/3.0.x-conflict dev/2.1.x-conflict labels Nov 28, 2024

morningman removed dev/2.1.x dev/2.1.x-conflict labels Dec 6, 2024

BePPPower mentioned this pull request Dec 6, 2024

[fix](branch-3.0) Fix the data type mapping for complex types in Doris to the ORC and Parquet file formats #45122

Merged

16 tasks

morningman pushed a commit that referenced this pull request Dec 8, 2024

[fix](branch-3.0) Fix the data type mapping for complex types in Dori…

fcd2ee7

…s to the ORC and Parquet file formats (#45122) picked from #44041

morningman added dev/3.0.4-merged and removed dev/3.0.x dev/3.0.x-conflict labels Dec 8, 2024

		@@ -17,6 +17,7 @@

		#pragma once

		#include <arrow/array/builder_binary.h>

[fix](Outfile) Fix the data type mapping for complex types in Doris to the ORC and Parquet file formats. #44041

[fix](Outfile) Fix the data type mapping for complex types in Doris to the ORC and Parquet file formats. #44041

Conversation

BePPPower commented Nov 15, 2024 • edited Loading

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

doris-robot commented Nov 15, 2024

BePPPower commented Nov 15, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Nov 15, 2024

Choose a reason for hiding this comment

doris-robot commented Nov 15, 2024

BePPPower commented Nov 19, 2024

BePPPower commented Nov 20, 2024

doris-robot commented Nov 20, 2024

BePPPower commented Nov 20, 2024

doris-robot commented Nov 20, 2024

doris-robot commented Nov 20, 2024

doris-robot commented Nov 20, 2024

doris-robot commented Nov 20, 2024

BePPPower commented Nov 21, 2024

doris-robot commented Nov 21, 2024

doris-robot commented Nov 21, 2024

doris-robot commented Nov 21, 2024

doris-robot commented Nov 21, 2024

morningman Nov 21, 2024

Choose a reason for hiding this comment

BePPPower commented Nov 25, 2024

doris-robot commented Nov 25, 2024

doris-robot commented Nov 25, 2024

doris-robot commented Nov 25, 2024

doris-robot commented Nov 25, 2024

BePPPower commented Nov 26, 2024

doris-robot commented Nov 26, 2024

doris-robot commented Nov 26, 2024

doris-robot commented Nov 26, 2024

doris-robot commented Nov 26, 2024

BePPPower commented Nov 27, 2024

BePPPower commented Nov 27, 2024

BePPPower commented Nov 27, 2024

doris-robot commented Nov 27, 2024

doris-robot commented Nov 27, 2024

doris-robot commented Nov 27, 2024

doris-robot commented Nov 27, 2024

github-actions bot commented Nov 28, 2024

github-actions bot commented Nov 28, 2024

BePPPower commented Nov 15, 2024 •

edited

Loading