[fix](statistics)Control memory use for sample partition column and key column. #46534

Jibing-Li · 2025-01-07T07:47:27Z

What problem does this PR solve?

When doing sample analyze for partition column and key column, BE may encounter OOM problem. The reason is, partition column need to choose at least one tablet in each partition to calculate the NDV and couldn't use limit in the SQL, so when the table has large number of partitions and each tablet in each partition is quite large, the sample SQL may try to read too many data which will cause BE OOM.
Similarly, key column couldn't use limit as well, so when one tablet is very large, it also could cause OOM.

This pr is try to solve this problem.
For partition columns, when the selected tablets contain more than 1000000000 (one billion) rows, we use ndv() function to read up to 5 partitions to get the NDV value of this 5 partitions, say the ndv is n. Suppose the row count in the 5 partitions is r, and the row count of tje table is R, the table NDV would be n * R / r.
ndv() function use hll, so it only use a small amount of memory.

For key columns, when the selected tablets contain more than 1000000000 rows, we use limit 1000000000 to control the rows to read.

Reading 1000000000 rows would use at most 8GB memory in BE, which is acceptable.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-01-07T07:47:32Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Jibing-Li · 2025-01-07T07:50:36Z

run buildall

doris-robot · 2025-01-07T09:05:53Z

TPC-H: Total hot run time: 32583 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5654640029037abe2d8d0020f3ba6f0e5ad35ec5, data reload: false

------ Round 1 ----------------------------------
q1	17588	6118	6023	6023
q2	2047	298	163	163
q3	10432	1218	758	758
q4	10261	860	434	434
q5	8568	2179	2001	2001
q6	207	181	144	144
q7	887	753	604	604
q8	9233	1377	1178	1178
q9	5233	4896	4954	4896
q10	6733	2324	1865	1865
q11	487	294	259	259
q12	339	356	216	216
q13	17747	3646	3085	3085
q14	245	240	218	218
q15	568	501	496	496
q16	626	613	585	585
q17	576	864	333	333
q18	6870	6435	6315	6315
q19	1899	967	550	550
q20	307	313	196	196
q21	2859	2149	1963	1963
q22	357	336	301	301
Total cold run time: 104069 ms
Total hot run time: 32583 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6270	6197	6196	6196
q2	233	324	229	229
q3	2239	2658	2299	2299
q4	1378	1857	1394	1394
q5	4331	4766	4757	4757
q6	183	178	145	145
q7	2042	1947	1800	1800
q8	2634	2803	2727	2727
q9	7318	7317	7367	7317
q10	3078	3340	2854	2854
q11	592	501	488	488
q12	636	709	578	578
q13	3420	3846	3216	3216
q14	298	295	276	276
q15	575	535	498	498
q16	651	716	658	658
q17	1233	1750	1249	1249
q18	7575	7446	7240	7240
q19	813	1076	1136	1076
q20	2036	2043	1881	1881
q21	5733	5348	4904	4904
q22	592	631	571	571
Total cold run time: 53860 ms
Total hot run time: 52353 ms

doris-robot · 2025-01-07T09:18:14Z

TPC-DS: Total hot run time: 197364 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5654640029037abe2d8d0020f3ba6f0e5ad35ec5, data reload: false

query1	1281	939	933	933
query2	6317	2366	2266	2266
query3	10971	4791	4830	4791
query4	32863	23634	23295	23295
query5	3980	640	447	447
query6	291	198	179	179
query7	3986	486	317	317
query8	292	236	226	226
query9	9395	2734	2731	2731
query10	464	311	242	242
query11	17885	15328	15362	15328
query12	151	107	100	100
query13	1566	506	398	398
query14	9880	7357	7266	7266
query15	260	197	195	195
query16	8269	570	420	420
query17	1589	776	600	600
query18	2127	406	327	327
query19	207	225	206	206
query20	121	116	120	116
query21	210	126	118	118
query22	4678	4726	4615	4615
query23	34749	33771	33619	33619
query24	6370	2260	2357	2260
query25	479	460	391	391
query26	781	282	156	156
query27	2222	461	346	346
query28	5711	2512	2492	2492
query29	666	557	433	433
query30	213	179	150	150
query31	986	937	837	837
query32	87	58	57	57
query33	492	353	322	322
query34	767	857	520	520
query35	829	833	785	785
query36	1050	1052	992	992
query37	122	101	74	74
query38	4342	4326	4208	4208
query39	1558	1462	1494	1462
query40	200	116	104	104
query41	45	44	49	44
query42	129	106	100	100
query43	536	523	489	489
query44	1352	824	823	823
query45	187	174	168	168
query46	896	1069	661	661
query47	1984	2053	1962	1962
query48	383	417	339	339
query49	734	503	418	418
query50	629	660	417	417
query51	7093	7120	6978	6978
query52	107	101	95	95
query53	230	265	187	187
query54	498	504	428	428
query55	88	84	79	79
query56	275	302	277	277
query57	1269	1237	1187	1187
query58	231	231	220	220
query59	3163	3338	3172	3172
query60	267	264	256	256
query61	109	106	112	106
query62	873	835	744	744
query63	242	194	189	189
query64	3702	1033	670	670
query65	3371	3234	3321	3234
query66	806	418	311	311
query67	16532	15949	15527	15527
query68	8477	711	513	513
query69	492	297	254	254
query70	1202	1170	1155	1155
query71	435	309	265	265
query72	6339	3904	3842	3842
query73	654	751	370	370
query74	10450	9152	9028	9028
query75	3969	3171	2680	2680
query76	3680	1181	777	777
query77	756	353	284	284
query78	10216	10064	9523	9523
query79	3688	831	580	580
query80	757	515	438	438
query81	494	265	235	235
query82	615	156	120	120
query83	157	165	139	139
query84	243	94	74	74
query85	783	425	317	317
query86	403	320	294	294
query87	4412	4420	4704	4420
query88	4616	2232	2214	2214
query89	474	339	300	300
query90	1785	191	190	190
query91	143	138	105	105
query92	70	55	53	53
query93	2172	870	535	535
query94	651	382	262	262
query95	341	292	263	263
query96	480	615	284	284
query97	2928	2902	2827	2827
query98	220	203	199	199
query99	1682	1554	1419	1419
Total cold run time: 297922 ms
Total hot run time: 197364 ms

doris-robot · 2025-01-07T09:23:33Z

ClickBench: Total hot run time: 30.77 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5654640029037abe2d8d0020f3ba6f0e5ad35ec5, data reload: false

query1	0.03	0.06	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.11	0.10
query5	0.42	0.43	0.41
query6	1.14	0.66	0.64
query7	0.03	0.02	0.01
query8	0.04	0.03	0.04
query9	0.59	0.50	0.49
query10	0.54	0.55	0.55
query11	0.14	0.11	0.10
query12	0.14	0.12	0.11
query13	0.62	0.59	0.61
query14	2.73	2.83	2.71
query15	0.89	0.82	0.82
query16	0.38	0.39	0.39
query17	1.07	1.05	1.05
query18	0.24	0.20	0.21
query19	1.96	1.78	1.98
query20	0.02	0.01	0.02
query21	15.36	0.91	0.57
query22	0.74	0.81	0.69
query23	15.24	1.43	0.54
query24	2.96	0.37	0.84
query25	0.21	0.22	0.06
query26	0.32	0.14	0.14
query27	0.05	0.05	0.05
query28	13.27	1.58	1.07
query29	12.63	4.07	3.38
query30	0.26	0.09	0.06
query31	2.83	0.61	0.39
query32	3.23	0.53	0.46
query33	3.08	3.18	3.10
query34	16.87	5.10	4.52
query35	4.54	4.48	4.46
query36	0.83	0.48	0.50
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 105.81 s
Total hot run time: 30.77 s

Jibing-Li · 2025-01-07T15:59:17Z

run buildall

doris-robot · 2025-01-07T16:37:26Z

TPC-H: Total hot run time: 32498 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 03002b907e85d614ca8d4769aaa9aba47aba07a2, data reload: false

------ Round 1 ----------------------------------
q1	17598	6121	6045	6045
q2	2042	307	176	176
q3	10418	1207	742	742
q4	10233	875	447	447
q5	7693	2189	1963	1963
q6	204	179	144	144
q7	904	754	603	603
q8	9255	1332	1144	1144
q9	5206	4801	4849	4801
q10	6745	2287	1870	1870
q11	481	277	259	259
q12	346	351	214	214
q13	17763	3706	3083	3083
q14	220	223	217	217
q15	549	493	497	493
q16	647	612	601	601
q17	563	837	317	317
q18	6904	6504	6367	6367
q19	1854	987	554	554
q20	321	317	198	198
q21	2829	2292	1955	1955
q22	364	339	305	305
Total cold run time: 103139 ms
Total hot run time: 32498 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6241	6211	6214	6211
q2	236	330	230	230
q3	2219	2633	2298	2298
q4	1393	1807	1351	1351
q5	4357	4752	4748	4748
q6	182	178	141	141
q7	2071	1933	1798	1798
q8	2613	2876	2689	2689
q9	7311	7257	7281	7257
q10	3070	3325	2862	2862
q11	590	518	480	480
q12	638	728	564	564
q13	3482	3902	3262	3262
q14	311	305	277	277
q15	563	507	503	503
q16	652	677	662	662
q17	1219	1723	1279	1279
q18	7639	7523	7457	7457
q19	821	1004	1145	1004
q20	2072	2055	1886	1886
q21	5643	5170	4831	4831
q22	606	612	583	583
Total cold run time: 53929 ms
Total hot run time: 52373 ms

doris-robot · 2025-01-07T16:49:45Z

TPC-DS: Total hot run time: 196034 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 03002b907e85d614ca8d4769aaa9aba47aba07a2, data reload: false

query1	1317	953	911	911
query2	6403	2285	2280	2280
query3	10989	4777	4443	4443
query4	33100	23750	23262	23262
query5	3697	602	437	437
query6	270	195	188	188
query7	3972	485	311	311
query8	306	246	238	238
query9	9545	2845	2820	2820
query10	476	320	260	260
query11	18321	15747	15242	15242
query12	164	103	103	103
query13	1568	549	436	436
query14	11411	6601	7287	6601
query15	256	211	184	184
query16	7901	636	451	451
query17	1573	766	562	562
query18	2081	426	329	329
query19	226	209	162	162
query20	117	118	108	108
query21	201	126	117	117
query22	4614	4689	4471	4471
query23	34227	33444	33900	33444
query24	6270	2300	2269	2269
query25	474	462	397	397
query26	718	275	153	153
query27	2266	460	327	327
query28	5456	2538	2489	2489
query29	625	555	446	446
query30	221	185	158	158
query31	983	940	829	829
query32	74	61	60	60
query33	493	366	321	321
query34	790	860	528	528
query35	822	872	742	742
query36	1039	1083	972	972
query37	111	103	84	84
query38	4139	4220	4444	4220
query39	1506	1460	1478	1460
query40	208	113	100	100
query41	45	43	47	43
query42	124	108	107	107
query43	534	529	486	486
query44	1324	871	837	837
query45	184	177	165	165
query46	893	1094	659	659
query47	1969	1985	1912	1912
query48	396	432	341	341
query49	696	482	412	412
query50	649	663	413	413
query51	7102	7069	6959	6959
query52	103	104	96	96
query53	231	260	198	198
query54	486	490	408	408
query55	83	86	84	84
query56	255	282	246	246
query57	1217	1248	1152	1152
query58	236	242	243	242
query59	3358	3337	3232	3232
query60	294	292	294	292
query61	134	128	140	128
query62	875	811	790	790
query63	230	199	202	199
query64	2784	1023	636	636
query65	3315	3293	3195	3195
query66	775	399	313	313
query67	16419	15966	15406	15406
query68	9443	701	532	532
query69	460	286	255	255
query70	1170	1163	1107	1107
query71	423	284	262	262
query72	6222	3839	3798	3798
query73	665	759	374	374
query74	10097	9188	8912	8912
query75	4441	3119	2654	2654
query76	4661	1169	752	752
query77	833	379	286	286
query78	10841	11593	9924	9924
query79	2618	738	600	600
query80	684	535	431	431
query81	471	272	238	238
query82	242	140	210	140
query83	194	168	151	151
query84	277	84	70	70
query85	727	353	300	300
query86	348	307	285	285
query87	4618	4433	4434	4433
query88	4117	2252	2249	2249
query89	408	324	294	294
query90	1863	184	183	183
query91	133	134	114	114
query92	69	54	51	51
query93	957	812	532	532
query94	655	376	284	284
query95	339	264	249	249
query96	490	598	277	277
query97	2900	2982	2805	2805
query98	224	199	196	196
query99	1707	1571	1430	1430
Total cold run time: 296938 ms
Total hot run time: 196034 ms

doris-robot · 2025-01-07T16:55:05Z

ClickBench: Total hot run time: 30.86 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 03002b907e85d614ca8d4769aaa9aba47aba07a2, data reload: false

query1	0.04	0.03	0.05
query2	0.07	0.04	0.03
query3	0.23	0.07	0.07
query4	1.63	0.10	0.10
query5	0.44	0.41	0.41
query6	1.16	0.65	0.65
query7	0.02	0.01	0.02
query8	0.04	0.03	0.04
query9	0.60	0.51	0.52
query10	0.55	0.57	0.57
query11	0.15	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.83	2.80	2.86
query15	0.89	0.82	0.81
query16	0.39	0.38	0.37
query17	1.06	1.05	1.06
query18	0.23	0.21	0.21
query19	1.98	1.72	2.03
query20	0.01	0.02	0.01
query21	15.37	0.93	0.56
query22	0.76	0.73	0.68
query23	15.35	1.50	0.63
query24	3.14	0.46	1.73
query25	0.17	0.26	0.10
query26	0.30	0.15	0.13
query27	0.08	0.05	0.04
query28	13.73	1.52	1.04
query29	12.57	3.86	3.26
query30	0.25	0.08	0.06
query31	2.83	0.59	0.39
query32	3.23	0.54	0.44
query33	3.16	3.19	3.19
query34	16.49	5.10	4.48
query35	4.61	4.47	4.42
query36	0.83	0.49	0.48
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.03
query40	0.18	0.13	0.12
query41	0.09	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.47 s
Total hot run time: 30.86 s

Jibing-Li · 2025-01-07T17:11:39Z

run external

fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java

Jibing-Li · 2025-01-08T11:41:55Z

run buildall

github-actions · 2025-01-08T11:51:53Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-01-08T11:51:54Z

PR approved by anyone and no changes requested.

doris-robot · 2025-01-08T13:30:01Z

TPC-H: Total hot run time: 33036 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 701042fdba4af34131b7f1c75a547f1e42f30c8c, data reload: false

------ Round 1 ----------------------------------
q1	17704	6176	6084	6084
q2	2054	320	170	170
q3	10693	1256	746	746
q4	10225	881	444	444
q5	7705	2209	2004	2004
q6	211	184	151	151
q7	889	760	598	598
q8	9256	1370	1229	1229
q9	5231	4944	4893	4893
q10	6749	2287	1866	1866
q11	485	286	263	263
q12	355	370	233	233
q13	17790	3707	3130	3130
q14	234	232	207	207
q15	636	498	494	494
q16	619	624	607	607
q17	574	859	324	324
q18	7264	6654	6458	6458
q19	2217	985	564	564
q20	309	320	191	191
q21	2930	2259	2064	2064
q22	363	339	316	316
Total cold run time: 104493 ms
Total hot run time: 33036 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6389	6204	6237	6204
q2	236	323	233	233
q3	2222	2655	2332	2332
q4	1428	1883	1367	1367
q5	4324	4744	4908	4744
q6	186	189	148	148
q7	2080	2009	1829	1829
q8	2645	2855	2703	2703
q9	7289	7329	7328	7328
q10	3103	3261	2742	2742
q11	598	528	524	524
q12	687	741	626	626
q13	3493	3886	3206	3206
q14	288	305	266	266
q15	593	525	517	517
q16	639	702	646	646
q17	1232	1743	1258	1258
q18	7979	7530	7328	7328
q19	844	1172	1135	1135
q20	1963	2057	1889	1889
q21	5713	5154	5156	5154
q22	619	627	619	619
Total cold run time: 54550 ms
Total hot run time: 52798 ms

doris-robot · 2025-01-08T13:42:17Z

TPC-DS: Total hot run time: 194290 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 701042fdba4af34131b7f1c75a547f1e42f30c8c, data reload: false

query1	1313	983	906	906
query2	6307	2429	2324	2324
query3	11084	4857	4795	4795
query4	32716	23658	23181	23181
query5	3728	638	452	452
query6	286	208	203	203
query7	3982	503	313	313
query8	315	246	233	233
query9	9520	2754	2746	2746
query10	447	317	265	265
query11	17913	15188	14922	14922
query12	169	106	103	103
query13	1577	559	410	410
query14	10507	7637	6930	6930
query15	254	218	188	188
query16	8530	629	476	476
query17	1641	732	589	589
query18	2046	418	306	306
query19	214	203	157	157
query20	119	113	108	108
query21	204	121	100	100
query22	4699	4601	4524	4524
query23	33997	33223	33401	33223
query24	6395	2333	2323	2323
query25	471	446	405	405
query26	782	285	157	157
query27	2102	483	340	340
query28	5773	2529	2514	2514
query29	586	565	423	423
query30	214	192	153	153
query31	940	896	817	817
query32	68	58	62	58
query33	477	381	287	287
query34	787	885	512	512
query35	819	859	751	751
query36	1025	1024	959	959
query37	130	106	74	74
query38	4163	4230	4096	4096
query39	1490	1480	1460	1460
query40	207	120	108	108
query41	49	43	44	43
query42	121	107	104	104
query43	546	535	505	505
query44	1369	837	841	837
query45	183	181	165	165
query46	912	1086	667	667
query47	1855	1873	1848	1848
query48	392	417	354	354
query49	720	502	407	407
query50	692	652	383	383
query51	7116	7013	6905	6905
query52	99	106	100	100
query53	237	267	185	185
query54	484	496	396	396
query55	84	79	80	79
query56	267	250	235	235
query57	1199	1193	1167	1167
query58	251	243	223	223
query59	3151	3292	3149	3149
query60	279	266	254	254
query61	114	100	102	100
query62	872	786	724	724
query63	229	195	202	195
query64	2836	1098	644	644
query65	3320	3237	3177	3177
query66	795	402	302	302
query67	16118	15623	15359	15359
query68	9108	702	515	515
query69	489	285	252	252
query70	1198	1166	1134	1134
query71	434	280	253	253
query72	6474	3821	3858	3821
query73	659	755	364	364
query74	9865	9181	8560	8560
query75	4629	3159	2663	2663
query76	4620	1181	776	776
query77	774	356	272	272
query78	10057	10176	9266	9266
query79	3609	824	580	580
query80	712	513	438	438
query81	476	283	237	237
query82	631	161	118	118
query83	201	160	148	148
query84	283	94	78	78
query85	768	356	302	302
query86	347	297	299	297
query87	4502	4376	4273	4273
query88	4417	2205	2179	2179
query89	430	327	302	302
query90	1842	192	192	192
query91	130	136	140	136
query92	67	55	50	50
query93	1442	842	525	525
query94	648	391	279	279
query95	334	278	254	254
query96	487	609	276	276
query97	2874	2895	2812	2812
query98	224	195	196	195
query99	1604	1486	1345	1345
Total cold run time: 296622 ms
Total hot run time: 194290 ms

doris-robot · 2025-01-08T13:47:36Z

ClickBench: Total hot run time: 31.1 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 701042fdba4af34131b7f1c75a547f1e42f30c8c, data reload: false

query1	0.03	0.03	0.04
query2	0.06	0.04	0.03
query3	0.24	0.06	0.07
query4	1.62	0.11	0.11
query5	0.42	0.41	0.41
query6	1.16	0.64	0.65
query7	0.02	0.01	0.02
query8	0.05	0.03	0.03
query9	0.60	0.50	0.52
query10	0.55	0.57	0.55
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.60	0.59	0.59
query14	2.70	2.77	2.73
query15	0.88	0.81	0.83
query16	0.39	0.39	0.38
query17	1.00	1.02	1.03
query18	0.21	0.20	0.20
query19	1.96	1.85	1.97
query20	0.01	0.01	0.02
query21	15.36	0.98	0.57
query22	0.74	0.81	0.61
query23	15.30	1.45	0.51
query24	2.88	0.92	1.25
query25	0.11	0.14	0.08
query26	0.36	0.15	0.14
query27	0.07	0.07	0.05
query28	13.33	1.57	1.04
query29	12.55	3.92	3.26
query30	0.24	0.09	0.06
query31	2.82	0.60	0.37
query32	3.23	0.54	0.46
query33	3.07	3.12	3.11
query34	16.80	5.17	4.47
query35	4.54	4.52	4.51
query36	0.65	0.49	0.48
query37	0.10	0.06	0.05
query38	0.04	0.03	0.04
query39	0.04	0.02	0.02
query40	0.17	0.13	0.12
query41	0.09	0.03	0.02
query42	0.04	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.35 s
Total hot run time: 31.1 s

Refactor doSample function.

Jibing-Li · 2025-01-08T16:08:50Z

run buildall

doris-robot · 2025-01-08T16:56:32Z

TPC-H: Total hot run time: 32564 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c059119629bee5adba6d22da7b304236b5b8eac4, data reload: false

------ Round 1 ----------------------------------
q1	17571	6149	6007	6007
q2	2044	286	164	164
q3	10442	1214	751	751
q4	10238	868	440	440
q5	7932	2147	1953	1953
q6	200	184	150	150
q7	895	748	593	593
q8	9232	1311	1156	1156
q9	5187	4902	4808	4808
q10	6772	2291	1848	1848
q11	485	271	268	268
q12	345	362	218	218
q13	17787	3633	3065	3065
q14	246	245	224	224
q15	560	512	504	504
q16	661	617	584	584
q17	581	849	338	338
q18	7183	6710	6412	6412
q19	1716	988	566	566
q20	307	306	198	198
q21	2947	2174	2003	2003
q22	370	344	314	314
Total cold run time: 103701 ms
Total hot run time: 32564 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6468	6245	6244	6244
q2	237	323	233	233
q3	2268	2640	2315	2315
q4	1421	1830	1359	1359
q5	4288	4772	4768	4768
q6	188	178	140	140
q7	2080	1939	1809	1809
q8	2611	2852	2683	2683
q9	7291	7264	7187	7187
q10	3068	3308	2751	2751
q11	590	512	519	512
q12	680	771	653	653
q13	3526	3805	3215	3215
q14	287	311	296	296
q15	566	512	507	507
q16	662	693	653	653
q17	1234	1721	1235	1235
q18	7772	7421	7266	7266
q19	834	1068	1119	1068
q20	2016	2098	1939	1939
q21	5635	5164	5003	5003
q22	625	653	567	567
Total cold run time: 54347 ms
Total hot run time: 52403 ms

doris-robot · 2025-01-08T17:08:46Z

TPC-DS: Total hot run time: 195261 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c059119629bee5adba6d22da7b304236b5b8eac4, data reload: false

query1	1294	949	904	904
query2	6416	2372	2284	2284
query3	11102	4710	4968	4710
query4	32848	23938	23307	23307
query5	4823	623	455	455
query6	287	213	194	194
query7	4006	494	301	301
query8	313	266	242	242
query9	9606	2630	2632	2630
query10	521	333	256	256
query11	18023	15233	15244	15233
query12	164	107	106	106
query13	1589	545	414	414
query14	10521	7159	7406	7159
query15	226	207	189	189
query16	7985	642	459	459
query17	1551	775	559	559
query18	2124	418	315	315
query19	198	198	161	161
query20	120	115	120	115
query21	213	126	106	106
query22	4730	4548	4382	4382
query23	34072	33221	33158	33158
query24	6309	2350	2261	2261
query25	476	457	395	395
query26	895	285	150	150
query27	2008	462	326	326
query28	5695	2470	2438	2438
query29	686	562	413	413
query30	204	183	144	144
query31	936	909	834	834
query32	75	64	60	60
query33	510	354	305	305
query34	803	831	517	517
query35	783	814	723	723
query36	1005	1006	946	946
query37	122	95	81	81
query38	4088	4130	4230	4130
query39	1535	1470	1471	1470
query40	199	125	110	110
query41	48	43	43	43
query42	120	106	102	102
query43	505	536	486	486
query44	1282	824	808	808
query45	177	174	171	171
query46	894	1064	659	659
query47	1876	1878	1852	1852
query48	395	420	314	314
query49	713	500	425	425
query50	647	677	388	388
query51	6926	7117	6894	6894
query52	108	103	92	92
query53	225	252	182	182
query54	480	496	415	415
query55	88	85	75	75
query56	269	256	248	248
query57	1219	1237	1136	1136
query58	238	227	229	227
query59	3167	3423	3153	3153
query60	273	273	262	262
query61	113	109	109	109
query62	842	802	740	740
query63	252	190	198	190
query64	3396	1050	671	671
query65	3313	3211	3236	3211
query66	801	398	307	307
query67	16563	15766	15588	15588
query68	8799	701	517	517
query69	477	295	259	259
query70	1166	1082	1141	1082
query71	439	287	264	264
query72	6475	3927	3850	3850
query73	644	737	349	349
query74	9824	8993	8974	8974
query75	4121	3142	2661	2661
query76	3684	1165	756	756
query77	767	354	269	269
query78	10306	9876	9351	9351
query79	3986	798	579	579
query80	730	515	436	436
query81	478	270	221	221
query82	589	151	130	130
query83	184	159	148	148
query84	285	89	69	69
query85	886	355	299	299
query86	367	314	300	300
query87	4390	4506	4477	4477
query88	3842	2159	2118	2118
query89	433	309	286	286
query90	1955	187	186	186
query91	131	145	104	104
query92	67	55	56	55
query93	1694	797	535	535
query94	658	401	275	275
query95	334	256	254	254
query96	464	602	274	274
query97	2816	2977	2898	2898
query98	215	197	201	197
query99	1620	1483	1398	1398
Total cold run time: 296817 ms
Total hot run time: 195261 ms

doris-robot · 2025-01-08T17:14:07Z

ClickBench: Total hot run time: 31.59 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c059119629bee5adba6d22da7b304236b5b8eac4, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.03
query3	0.23	0.07	0.07
query4	1.61	0.11	0.11
query5	0.41	0.41	0.41
query6	1.13	0.65	0.64
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.50	0.50
query10	0.56	0.57	0.54
query11	0.14	0.10	0.10
query12	0.13	0.12	0.11
query13	0.61	0.60	0.59
query14	2.68	2.78	2.75
query15	0.90	0.82	0.82
query16	0.38	0.39	0.39
query17	0.99	1.00	1.08
query18	0.23	0.21	0.20
query19	1.90	1.88	1.97
query20	0.02	0.01	0.02
query21	15.36	0.94	0.58
query22	0.76	0.78	0.60
query23	15.38	1.38	0.56
query24	2.96	2.24	1.12
query25	0.15	0.13	0.15
query26	0.37	0.15	0.14
query27	0.09	0.07	0.04
query28	13.78	1.49	1.04
query29	12.63	3.95	3.27
query30	0.25	0.09	0.07
query31	2.83	0.60	0.37
query32	3.24	0.53	0.46
query33	3.15	3.16	3.13
query34	16.70	5.17	4.57
query35	4.54	4.50	4.53
query36	0.65	0.52	0.47
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.14
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.02 s
Total hot run time: 31.59 s

github-actions · 2025-01-09T03:16:59Z

PR approved by at least one committer and no changes requested.

zfr9527

LGTM

Jibing-Li marked this pull request as ready for review January 7, 2025 07:50

Jibing-Li force-pushed the oom branch from 5654640 to 03002b9 Compare January 7, 2025 15:58

morrySnow added dev/2.1.x dev/3.0.x labels Jan 8, 2025

morrySnow reviewed Jan 8, 2025

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java Outdated Show resolved Hide resolved

Jibing-Li force-pushed the oom branch 4 times, most recently from 65ec2c6 to 701042f Compare January 8, 2025 11:38

morrySnow previously approved these changes Jan 8, 2025

View reviewed changes

morrySnow removed dev/2.1.x dev/3.0.x labels Jan 8, 2025

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 8, 2025

github-actions bot added the reviewed label Jan 8, 2025

Jibing-Li dismissed morrySnow’s stale review via 7421297 January 8, 2025 15:03

Jibing-Li force-pushed the oom branch from 701042f to 7421297 Compare January 8, 2025 15:03

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jan 8, 2025

Jibing-Li force-pushed the oom branch from 7421297 to fb52299 Compare January 8, 2025 15:31

Control memory use for sample partition column and key column.

c059119

Refactor doSample function.

Jibing-Li force-pushed the oom branch from fb52299 to c059119 Compare January 8, 2025 15:51

morrySnow approved these changes Jan 9, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 9, 2025

zfr9527 approved these changes Jan 9, 2025

View reviewed changes

Jibing-Li merged commit 56a61a9 into apache:master Jan 9, 2025
28 of 30 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix](statistics)Control memory use for sample partition column and key column. #46534

[fix](statistics)Control memory use for sample partition column and key column. #46534

Jibing-Li commented Jan 7, 2025 •

edited

Loading

Thearas commented Jan 7, 2025

Jibing-Li commented Jan 7, 2025

doris-robot commented Jan 7, 2025

doris-robot commented Jan 7, 2025

doris-robot commented Jan 7, 2025

Jibing-Li commented Jan 7, 2025

doris-robot commented Jan 7, 2025

doris-robot commented Jan 7, 2025

doris-robot commented Jan 7, 2025

Jibing-Li commented Jan 7, 2025

Jibing-Li commented Jan 8, 2025

github-actions bot commented Jan 8, 2025

github-actions bot commented Jan 8, 2025

doris-robot commented Jan 8, 2025

doris-robot commented Jan 8, 2025

doris-robot commented Jan 8, 2025

Jibing-Li commented Jan 8, 2025

doris-robot commented Jan 8, 2025

doris-robot commented Jan 8, 2025

doris-robot commented Jan 8, 2025

github-actions bot commented Jan 9, 2025

zfr9527 left a comment

[fix](statistics)Control memory use for sample partition column and key column. #46534

[fix](statistics)Control memory use for sample partition column and key column. #46534

Conversation

Jibing-Li commented Jan 7, 2025 • edited Loading

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Thearas commented Jan 7, 2025

Jibing-Li commented Jan 7, 2025

doris-robot commented Jan 7, 2025

doris-robot commented Jan 7, 2025

doris-robot commented Jan 7, 2025

Jibing-Li commented Jan 7, 2025

doris-robot commented Jan 7, 2025

doris-robot commented Jan 7, 2025

doris-robot commented Jan 7, 2025

Jibing-Li commented Jan 7, 2025

Jibing-Li commented Jan 8, 2025

github-actions bot commented Jan 8, 2025

github-actions bot commented Jan 8, 2025

doris-robot commented Jan 8, 2025

doris-robot commented Jan 8, 2025

doris-robot commented Jan 8, 2025

Jibing-Li commented Jan 8, 2025

doris-robot commented Jan 8, 2025

doris-robot commented Jan 8, 2025

doris-robot commented Jan 8, 2025

github-actions bot commented Jan 9, 2025

zfr9527 left a comment

Choose a reason for hiding this comment

Jibing-Li commented Jan 7, 2025 •

edited

Loading