Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. #42399

Merged
merged 1 commit into from
Dec 11, 2024
Merged

[improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. #42399

merged 1 commit into from
Dec 11, 2024

Conversation

Jibing-Li
Copy link
Contributor

@Jibing-Li Jibing-Li commented Oct 24, 2024 •
edited
Loading

Support auto analyze columns that haven't been analyzed for a long time. Add a very low priority job queue for auto analyze to process this kind of columns.

The purpose of this change is to make sure all tables could be auto analyzed within a certain time. In the earlier Doris versions, users often encounter this kind of issues:
User load some new data to a large table everyday, but the change rate (percentage of new data) is very low, because there is a large size of old data. In this case, auto analyze for this table will not be triggered for a very long time, because the default trigger threshold of auto analyze is 40% (more than 40% of the data in a table is changed since last analyze). This will probably cause a bad plan because min/max/ndv statistics are outdated.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41185 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ccc3ae1721f8241c4724680b176424edc833a678, data reload: false

------ Round 1 ----------------------------------
q1	17571	7553	7303	7303
q2	2059	157	154	154
q3	10578	1135	1141	1135
q4	10244	851	880	851
q5	7762	3077	3048	3048
q6	241	152	154	152
q7	1007	603	593	593
q8	9363	1850	1968	1850
q9	6656	6442	6438	6438
q10	7019	2393	2410	2393
q11	454	242	249	242
q12	414	217	221	217
q13	17782	2988	2999	2988
q14	249	211	223	211
q15	563	533	514	514
q16	639	589	603	589
q17	965	533	634	533
q18	7492	6665	6835	6665
q19	1364	932	990	932
q20	499	184	184	184
q21	4031	3253	3172	3172
q22	1098	1025	1021	1021
Total cold run time: 108050 ms
Total hot run time: 41185 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7312	7262	7265	7262
q2	325	236	226	226
q3	2859	2782	2787	2782
q4	1951	1758	1643	1643
q5	5413	5450	5425	5425
q6	227	142	142	142
q7	2106	1736	1697	1697
q8	3232	3409	3425	3409
q9	8558	8542	8571	8542
q10	3480	3448	3418	3418
q11	579	477	483	477
q12	843	587	589	587
q13	7556	3049	3022	3022
q14	295	271	265	265
q15	570	512	515	512
q16	696	645	630	630
q17	1784	1587	1585	1585
q18	7891	7349	7590	7349
q19	1694	1440	1466	1440
q20	2045	1815	1829	1815
q21	5404	5282	5248	5248
q22	1115	1042	1016	1016
Total cold run time: 65935 ms
Total hot run time: 58492 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192221 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ccc3ae1721f8241c4724680b176424edc833a678, data reload: false

query1	980	378	369	369
query2	6526	2075	2083	2075
query3	6790	211	221	211
query4	34379	23614	23499	23499
query5	4383	474	482	474
query6	266	175	184	175
query7	4607	298	293	293
query8	281	240	236	236
query9	9544	2715	2728	2715
query10	480	299	275	275
query11	18080	15327	15216	15216
query12	158	103	101	101
query13	1693	439	405	405
query14	10416	7134	7260	7134
query15	283	174	186	174
query16	8120	458	461	458
query17	1665	578	557	557
query18	2139	297	297	297
query19	364	151	150	150
query20	119	108	106	106
query21	218	107	103	103
query22	4664	4463	4400	4400
query23	34931	34183	33882	33882
query24	11191	2759	2779	2759
query25	679	403	416	403
query26	1377	160	160	160
query27	2833	281	288	281
query28	7949	2464	2460	2460
query29	930	423	428	423
query30	338	166	161	161
query31	1043	830	829	829
query32	99	59	57	57
query33	776	290	301	290
query34	940	516	524	516
query35	1067	889	908	889
query36	1080	969	961	961
query37	158	98	92	92
query38	4389	4291	4366	4291
query39	1465	1416	1403	1403
query40	287	102	99	99
query41	50	46	47	46
query42	128	101	98	98
query43	540	485	486	485
query44	1297	814	807	807
query45	199	165	168	165
query46	1144	690	685	685
query47	1950	1838	1857	1838
query48	427	319	318	318
query49	1173	444	473	444
query50	804	381	393	381
query51	7149	6989	6950	6950
query52	101	92	90	90
query53	254	186	180	180
query54	1240	444	429	429
query55	79	80	81	80
query56	288	277	289	277
query57	1362	1179	1163	1163
query58	260	245	290	245
query59	3218	3040	3148	3040
query60	303	260	262	260
query61	104	101	100	100
query62	877	667	681	667
query63	216	190	183	183
query64	5308	643	607	607
query65	3271	3267	3224	3224
query66	1454	315	320	315
query67	16122	15748	15731	15731
query68	4857	569	550	550
query69	454	317	304	304
query70	1131	1153	1111	1111
query71	337	277	275	275
query72	6361	3975	3918	3918
query73	776	355	360	355
query74	10073	8962	8953	8953
query75	3428	2709	2704	2704
query76	2810	890	916	890
query77	415	295	310	295
query78	10658	9643	9510	9510
query79	1477	592	601	592
query80	1748	444	437	437
query81	574	242	254	242
query82	1220	138	151	138
query83	252	146	136	136
query84	263	67	69	67
query85	1371	300	284	284
query86	387	303	309	303
query87	4750	4638	4693	4638
query88	3538	2212	2191	2191
query89	396	288	292	288
query90	2046	189	184	184
query91	134	100	105	100
query92	71	50	49	49
query93	1035	528	533	528
query94	1101	295	302	295
query95	358	251	246	246
query96	616	278	283	278
query97	2888	2678	2694	2678
query98	207	195	206	195
query99	1649	1300	1300	1300
Total cold run time: 304386 ms
Total hot run time: 192221 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.48 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ccc3ae1721f8241c4724680b176424edc833a678, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.03	0.03
query3	0.22	0.06	0.06
query4	1.65	0.10	0.10
query5	0.42	0.41	0.40
query6	1.17	0.66	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.50	0.52
query10	0.56	0.56	0.55
query11	0.13	0.10	0.11
query12	0.15	0.11	0.11
query13	0.61	0.59	0.59
query14	2.68	2.80	2.77
query15	0.90	0.84	0.83
query16	0.40	0.40	0.39
query17	1.01	1.02	1.05
query18	0.23	0.22	0.22
query19	1.98	1.87	2.02
query20	0.01	0.01	0.01
query21	15.36	0.58	0.58
query22	2.69	1.94	2.05
query23	17.05	1.02	0.72
query24	3.17	1.08	0.83
query25	0.26	0.08	0.11
query26	0.38	0.14	0.13
query27	0.04	0.04	0.04
query28	10.86	1.09	1.07
query29	12.57	3.30	3.24
query30	0.24	0.06	0.06
query31	2.87	0.39	0.37
query32	3.26	0.46	0.45
query33	3.00	3.02	3.03
query34	16.91	4.48	4.42
query35	4.47	4.54	4.49
query36	0.67	0.47	0.51
query37	0.08	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.03
query40	0.17	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.12 s
Total hot run time: 32.48 s

@Jibing-Li Jibing-Li force-pushed the 24hours branch 3 times, most recently from b941eae to 7adbd1f Compare October 25, 2024 07:06
@Jibing-Li Jibing-Li marked this pull request as ready for review October 25, 2024 07:07
@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41053 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1b57e04b68aee2b2894c876929c834d47b38e26b, data reload: false

------ Round 1 ----------------------------------
q1	17600	7387	7268	7268
q2	2013	276	271	271
q3	11993	1059	1211	1059
q4	10571	860	904	860
q5	7741	3107	3043	3043
q6	233	150	145	145
q7	1010	607	588	588
q8	9366	1966	1993	1966
q9	6533	6436	6423	6423
q10	7036	2483	2399	2399
q11	443	249	249	249
q12	398	214	210	210
q13	17767	2986	2989	2986
q14	255	205	219	205
q15	561	520	509	509
q16	646	584	592	584
q17	954	526	560	526
q18	7260	6693	6607	6607
q19	1341	1003	966	966
q20	492	184	180	180
q21	3944	3280	2992	2992
q22	1119	1029	1017	1017
Total cold run time: 109276 ms
Total hot run time: 41053 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7244	7224	7243	7224
q2	324	234	240	234
q3	2997	3000	2923	2923
q4	2077	1854	1791	1791
q5	5774	5742	5794	5742
q6	231	144	139	139
q7	2224	1847	1783	1783
q8	3378	3533	3443	3443
q9	8882	8924	8861	8861
q10	3593	3567	3588	3567
q11	581	492	476	476
q12	865	641	592	592
q13	9083	3167	3193	3167
q14	311	283	266	266
q15	583	519	519	519
q16	689	643	649	643
q17	1840	1611	1601	1601
q18	8298	7870	7786	7786
q19	1712	1629	1630	1629
q20	2149	1883	1863	1863
q21	5618	5489	5476	5476
q22	1179	1045	1034	1034
Total cold run time: 69632 ms
Total hot run time: 60759 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192416 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1b57e04b68aee2b2894c876929c834d47b38e26b, data reload: false

query1	866	390	401	390
query2	6245	2050	2027	2027
query3	8686	189	197	189
query4	35193	23663	23685	23663
query5	4831	479	433	433
query6	282	196	168	168
query7	4204	289	285	285
query8	285	237	244	237
query9	9684	2652	2651	2651
query10	496	247	241	241
query11	18138	15264	15215	15215
query12	149	107	102	102
query13	1573	454	400	400
query14	9666	7418	7425	7418
query15	249	176	179	176
query16	7927	455	488	455
query17	1535	598	612	598
query18	2143	317	305	305
query19	358	158	160	158
query20	127	114	113	113
query21	205	106	107	106
query22	5033	4486	4630	4486
query23	34878	34186	34115	34115
query24	10698	2839	2801	2801
query25	594	406	404	404
query26	718	159	168	159
query27	2026	280	282	280
query28	6222	2423	2382	2382
query29	739	414	423	414
query30	254	169	165	165
query31	1135	781	807	781
query32	104	55	57	55
query33	670	284	273	273
query34	908	509	510	509
query35	1052	898	863	863
query36	1121	935	952	935
query37	117	72	79	72
query38	4426	4199	4259	4199
query39	1484	1418	1452	1418
query40	200	99	101	99
query41	50	45	46	45
query42	112	96	93	93
query43	530	490	481	481
query44	1164	820	800	800
query45	183	175	169	169
query46	1125	721	700	700
query47	1941	1858	1859	1858
query48	403	323	339	323
query49	904	411	398	398
query50	814	393	394	393
query51	6995	7003	6891	6891
query52	109	87	85	85
query53	254	180	179	179
query54	1171	418	404	404
query55	79	76	76	76
query56	271	273	248	248
query57	1304	1217	1183	1183
query58	239	192	203	192
query59	3281	2802	3098	2802
query60	278	244	243	243
query61	100	104	107	104
query62	858	670	686	670
query63	214	185	178	178
query64	3851	626	615	615
query65	3273	3214	3215	3214
query66	862	304	305	304
query67	16041	15841	15900	15841
query68	4442	553	563	553
query69	429	255	260	255
query70	1177	1109	1137	1109
query71	330	264	263	263
query72	6311	4097	3918	3918
query73	774	364	360	360
query74	10315	9100	9035	9035
query75	3432	2646	2687	2646
query76	2825	1102	995	995
query77	382	267	271	267
query78	10614	9620	9583	9583
query79	2129	576	588	576
query80	1056	419	418	418
query81	558	241	242	241
query82	634	117	113	113
query83	240	133	132	132
query84	244	76	71	71
query85	1289	300	286	286
query86	433	287	286	286
query87	4890	4773	4764	4764
query88	3620	2183	2175	2175
query89	399	291	289	289
query90	2045	185	183	183
query91	135	101	98	98
query92	59	48	48	48
query93	2373	536	544	536
query94	936	291	255	255
query95	356	245	244	244
query96	620	283	289	283
query97	2848	2772	2710	2710
query98	214	205	201	201
query99	1564	1306	1294	1294
Total cold run time: 299300 ms
Total hot run time: 192416 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.42 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1b57e04b68aee2b2894c876929c834d47b38e26b, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.07
query4	1.64	0.10	0.10
query5	0.42	0.39	0.40
query6	1.14	0.65	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.50	0.50
query10	0.56	0.55	0.54
query11	0.15	0.10	0.10
query12	0.13	0.11	0.11
query13	0.60	0.59	0.59
query14	2.88	2.81	2.73
query15	0.89	0.82	0.84
query16	0.39	0.39	0.40
query17	1.06	1.04	1.04
query18	0.20	0.20	0.20
query19	1.90	1.88	2.00
query20	0.02	0.01	0.01
query21	15.35	0.57	0.58
query22	2.84	3.25	1.73
query23	17.06	0.99	0.85
query24	3.29	1.24	0.96
query25	0.13	0.25	0.04
query26	0.52	0.14	0.13
query27	0.05	0.04	0.03
query28	10.27	1.11	1.07
query29	12.56	3.30	3.29
query30	0.25	0.06	0.06
query31	2.86	0.38	0.37
query32	3.28	0.47	0.45
query33	2.98	3.01	3.00
query34	17.12	4.49	4.45
query35	4.44	4.46	4.47
query36	0.66	0.47	0.49
query37	0.08	0.06	0.05
query38	0.04	0.04	0.03
query39	0.04	0.02	0.02
query40	0.17	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 107.08 s
Total hot run time: 32.42 s

@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41620 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a89d592b48e8286fa1d33f582b1486a07e71c877, data reload: false

------ Round 1 ----------------------------------
q1	17576	7548	7297	7297
q2	2055	159	170	159
q3	10568	1172	1232	1172
q4	10228	820	829	820
q5	7747	3125	3069	3069
q6	236	148	146	146
q7	1028	610	612	610
q8	9356	1996	2063	1996
q9	6688	6518	6505	6505
q10	7070	2462	2406	2406
q11	452	251	257	251
q12	412	219	213	213
q13	17786	3027	3013	3013
q14	241	205	222	205
q15	569	545	516	516
q16	642	589	584	584
q17	988	569	582	569
q18	7534	6822	6617	6617
q19	1336	990	1085	990
q20	473	187	184	184
q21	4046	3388	3279	3279
q22	1126	1019	1027	1019
Total cold run time: 108157 ms
Total hot run time: 41620 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7280	7276	7282	7276
q2	322	224	227	224
q3	2952	2835	2793	2793
q4	1925	1747	1753	1747
q5	5491	5524	5558	5524
q6	224	136	139	136
q7	2081	1751	1708	1708
q8	3248	3426	3468	3426
q9	8603	8631	8634	8631
q10	3509	3493	3447	3447
q11	603	505	507	505
q12	787	571	591	571
q13	11688	3020	3027	3020
q14	288	264	256	256
q15	571	515	505	505
q16	685	638	624	624
q17	1847	1596	1602	1596
q18	7873	7556	7421	7421
q19	1675	1524	1546	1524
q20	2053	1834	1830	1830
q21	5452	5180	5345	5180
q22	1108	1048	1007	1007
Total cold run time: 70265 ms
Total hot run time: 58951 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191814 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a89d592b48e8286fa1d33f582b1486a07e71c877, data reload: false

query1	965	385	392	385
query2	6506	2095	2072	2072
query3	6780	222	215	215
query4	33931	23887	23634	23634
query5	4353	455	448	448
query6	274	196	176	176
query7	4615	312	336	312
query8	313	218	230	218
query9	9498	2816	2803	2803
query10	480	255	244	244
query11	18142	15434	15177	15177
query12	151	101	101	101
query13	1672	449	432	432
query14	10314	6440	6387	6387
query15	243	180	178	178
query16	7991	444	468	444
query17	1604	595	556	556
query18	2135	309	309	309
query19	245	144	143	143
query20	117	109	112	109
query21	207	102	102	102
query22	4625	4165	4242	4165
query23	35055	34099	34249	34099
query24	11564	2734	2789	2734
query25	696	398	393	393
query26	1832	162	163	162
query27	2907	291	301	291
query28	8360	2644	2620	2620
query29	1049	431	426	426
query30	334	161	150	150
query31	1039	801	811	801
query32	96	59	60	59
query33	778	290	287	287
query34	977	512	521	512
query35	908	738	730	730
query36	1093	946	968	946
query37	202	75	83	75
query38	4388	4344	4344	4344
query39	1477	1432	1424	1424
query40	290	107	106	106
query41	52	46	46	46
query42	112	98	103	98
query43	537	502	491	491
query44	1351	910	892	892
query45	185	164	165	164
query46	1143	714	704	704
query47	1949	1825	1844	1825
query48	416	329	323	323
query49	1301	404	396	396
query50	810	397	391	391
query51	7453	7267	7158	7158
query52	104	94	89	89
query53	257	187	181	181
query54	1149	427	439	427
query55	81	80	84	80
query56	270	268	251	251
query57	1319	1220	1156	1156
query58	230	195	212	195
query59	3356	3236	3103	3103
query60	275	242	252	242
query61	111	100	107	100
query62	884	674	692	674
query63	218	185	183	183
query64	5211	682	640	640
query65	3363	3246	3189	3189
query66	1416	300	302	300
query67	16465	15735	15605	15605
query68	4716	662	648	648
query69	439	247	251	247
query70	1316	1118	1108	1108
query71	419	259	285	259
query72	6406	3980	4006	3980
query73	774	351	355	351
query74	10361	8993	9004	8993
query75	3462	2662	2700	2662
query76	2885	1020	1079	1020
query77	393	289	280	280
query78	10338	9408	9394	9394
query79	1784	623	648	623
query80	1096	467	438	438
query81	548	237	238	237
query82	926	117	124	117
query83	214	135	135	135
query84	237	71	67	67
query85	1245	322	297	297
query86	362	313	290	290
query87	4845	4692	4674	4674
query88	3258	2175	2132	2132
query89	404	291	289	289
query90	1994	182	183	182
query91	132	101	100	100
query92	54	50	49	49
query93	1286	598	590	590
query94	911	293	296	293
query95	345	250	247	247
query96	615	282	280	280
query97	2899	2672	2698	2672
query98	211	197	195	195
query99	1552	1295	1293	1293
Total cold run time: 304651 ms
Total hot run time: 191814 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.97 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a89d592b48e8286fa1d33f582b1486a07e71c877, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.41	0.42	0.43
query6	1.14	0.66	0.64
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.57	0.49	0.49
query10	0.55	0.55	0.54
query11	0.13	0.10	0.10
query12	0.13	0.11	0.10
query13	0.61	0.60	0.60
query14	2.72	2.73	2.76
query15	0.89	0.82	0.82
query16	0.39	0.39	0.37
query17	0.99	1.00	1.04
query18	0.20	0.19	0.20
query19	1.94	1.89	1.95
query20	0.01	0.01	0.01
query21	15.37	0.60	0.59
query22	2.28	1.89	1.44
query23	17.00	1.05	0.64
query24	3.12	0.99	1.27
query25	0.45	0.20	0.03
query26	0.25	0.14	0.15
query27	0.04	0.05	0.04
query28	10.62	1.09	1.06
query29	12.57	3.25	3.23
query30	0.24	0.06	0.05
query31	2.87	0.40	0.39
query32	3.27	0.47	0.46
query33	2.96	3.03	3.05
query34	17.20	4.45	4.44
query35	4.48	4.54	4.49
query36	0.68	0.48	0.49
query37	0.10	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.03
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 106.6 s
Total hot run time: 31.97 s

@Jibing-Li
Copy link
Contributor Author

run cloud_p0

@Jibing-Li Jibing-Li requested a review from morrySnow November 11, 2024 03:51
@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39765 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 990467635c439b25faca646c03206c93bb16e9f7, data reload: false

------ Round 1 ----------------------------------
q1	17589	7521	7317	7317
q2	2058	176	165	165
q3	10603	1080	1248	1080
q4	10576	703	698	698
q5	7592	2794	2713	2713
q6	243	147	144	144
q7	981	619	607	607
q8	9260	1878	1926	1878
q9	6630	6380	6588	6380
q10	7046	2283	2328	2283
q11	459	261	263	261
q12	427	215	212	212
q13	17772	2989	3016	2989
q14	252	207	218	207
q15	567	534	526	526
q16	698	577	573	573
q17	976	561	534	534
q18	7271	6639	6805	6639
q19	1347	1043	913	913
q20	457	182	179	179
q21	4254	3278	3152	3152
q22	373	320	315	315
Total cold run time: 107431 ms
Total hot run time: 39765 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7260	7299	7330	7299
q2	330	229	228	228
q3	2931	2849	2982	2849
q4	2047	1800	1874	1800
q5	5677	5679	5686	5679
q6	236	143	145	143
q7	2260	1840	1848	1840
q8	3481	3554	3545	3545
q9	8895	8898	8900	8898
q10	3605	3592	3584	3584
q11	599	515	512	512
q12	834	634	634	634
q13	13320	3215	3232	3215
q14	305	281	279	279
q15	575	528	544	528
q16	734	647	659	647
q17	1839	1661	1609	1609
q18	8393	7891	7635	7635
q19	1729	1623	1536	1536
q20	2123	1882	1887	1882
q21	5699	5375	5397	5375
q22	664	562	577	562
Total cold run time: 73536 ms
Total hot run time: 60279 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195861 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 990467635c439b25faca646c03206c93bb16e9f7, data reload: false

query1	1248	944	968	944
query2	6268	2212	2224	2212
query3	10801	4003	3912	3912
query4	67599	28166	23662	23662
query5	4973	463	444	444
query6	398	183	188	183
query7	5518	293	281	281
query8	295	218	220	218
query9	8430	2644	2631	2631
query10	446	240	233	233
query11	17393	15250	15772	15250
query12	150	99	106	99
query13	1465	429	428	428
query14	11101	6903	7358	6903
query15	219	181	193	181
query16	6986	474	451	451
query17	1016	600	546	546
query18	1795	282	284	282
query19	196	153	149	149
query20	114	109	111	109
query21	198	105	99	99
query22	4640	4418	4392	4392
query23	35464	34129	34075	34075
query24	5371	2400	2453	2400
query25	488	374	375	374
query26	631	173	153	153
query27	1663	281	283	281
query28	4223	2472	2401	2401
query29	645	405	405	405
query30	210	145	144	144
query31	989	761	823	761
query32	66	54	57	54
query33	436	278	283	278
query34	930	489	511	489
query35	862	715	736	715
query36	1094	952	975	952
query37	118	76	70	70
query38	4374	4254	4197	4197
query39	1466	1447	1445	1445
query40	199	99	99	99
query41	44	42	42	42
query42	108	98	92	92
query43	541	512	510	510
query44	1162	783	815	783
query45	183	164	160	160
query46	1129	685	698	685
query47	1950	1845	1810	1810
query48	415	304	321	304
query49	737	401	400	400
query50	814	397	410	397
query51	7353	7070	7206	7070
query52	98	84	92	84
query53	258	179	178	178
query54	517	391	389	389
query55	80	77	80	77
query56	247	233	236	233
query57	1290	1184	1173	1173
query58	228	231	217	217
query59	3366	3091	3200	3091
query60	277	266	249	249
query61	137	142	135	135
query62	792	670	686	670
query63	216	195	183	183
query64	1500	784	747	747
query65	3283	3254	3234	3234
query66	720	325	377	325
query67	15981	15721	15665	15665
query68	3549	575	566	566
query69	426	252	255	252
query70	1203	1152	1138	1138
query71	389	242	254	242
query72	6223	4098	4022	4022
query73	779	367	360	360
query74	9889	8967	9010	8967
query75	3402	2671	2667	2667
query76	1871	1056	1015	1015
query77	518	300	289	289
query78	10585	9614	9410	9410
query79	1459	608	603	603
query80	836	497	437	437
query81	528	230	229	229
query82	1307	120	113	113
query83	212	154	202	154
query84	290	72	74	72
query85	894	297	307	297
query86	338	308	308	308
query87	4781	4636	4556	4556
query88	3541	2199	2171	2171
query89	435	295	296	295
query90	1996	190	188	188
query91	136	101	105	101
query92	65	50	51	50
query93	1964	548	545	545
query94	755	307	287	287
query95	357	252	250	250
query96	631	284	295	284
query97	2918	2709	2690	2690
query98	212	200	200	200
query99	1630	1311	1298	1298
Total cold run time: 318023 ms
Total hot run time: 195861 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.67 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 990467635c439b25faca646c03206c93bb16e9f7, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.03	0.04
query3	0.24	0.08	0.07
query4	1.61	0.10	0.10
query5	0.42	0.41	0.39
query6	1.15	0.66	0.67
query7	0.02	0.01	0.01
query8	0.04	0.02	0.03
query9	0.59	0.50	0.50
query10	0.55	0.55	0.56
query11	0.14	0.11	0.10
query12	0.13	0.11	0.11
query13	0.61	0.60	0.60
query14	2.85	2.75	2.76
query15	0.90	0.84	0.82
query16	0.38	0.39	0.40
query17	1.03	1.08	1.06
query18	0.21	0.20	0.20
query19	1.98	1.88	1.95
query20	0.01	0.01	0.01
query21	15.49	0.59	0.56
query22	2.29	2.36	2.21
query23	17.11	1.11	0.81
query24	2.98	1.15	0.58
query25	0.30	0.11	0.10
query26	0.38	0.14	0.13
query27	0.06	0.04	0.04
query28	10.90	1.11	1.08
query29	12.64	3.24	3.18
query30	0.25	0.07	0.07
query31	2.87	0.40	0.37
query32	3.29	0.47	0.47
query33	2.93	3.04	3.09
query34	16.80	4.48	4.47
query35	4.48	4.49	4.53
query36	0.68	0.50	0.48
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.04	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.92 s
Total hot run time: 32.67 s

@morningman morningman self-assigned this Nov 21, 2024
@doris-robot
Copy link

TPC-DS: Total hot run time: 197674 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4fde56054bffbc8e3900284c2a25b03486e2c177, data reload: false

query1	1273	963	986	963
query2	6234	2133	2043	2043
query3	10959	4411	4411	4411
query4	67346	28623	23490	23490
query5	4959	465	471	465
query6	424	187	184	184
query7	5587	313	299	299
query8	323	240	236	236
query9	9253	2739	2719	2719
query10	442	255	238	238
query11	17374	15265	15795	15265
query12	144	113	107	107
query13	1546	425	414	414
query14	10909	7349	7269	7269
query15	215	183	208	183
query16	7514	506	482	482
query17	1066	590	596	590
query18	1657	310	315	310
query19	209	161	158	158
query20	121	114	111	111
query21	221	101	105	101
query22	4727	4473	4678	4473
query23	34882	34832	34480	34480
query24	5406	2597	2465	2465
query25	489	391	389	389
query26	666	154	156	154
query27	1903	300	294	294
query28	4463	2502	2463	2463
query29	681	471	411	411
query30	205	153	154	153
query31	1023	832	828	828
query32	64	53	53	53
query33	446	287	288	287
query34	938	515	547	515
query35	851	782	764	764
query36	1108	969	959	959
query37	123	71	75	71
query38	4518	4439	4616	4439
query39	1521	1459	1459	1459
query40	203	106	97	97
query41	45	43	42	42
query42	109	102	102	102
query43	543	485	481	481
query44	1208	846	849	846
query45	193	168	166	166
query46	1166	737	725	725
query47	2074	1978	1918	1918
query48	434	324	328	324
query49	730	391	394	391
query50	849	388	416	388
query51	7341	7096	7183	7096
query52	98	90	86	86
query53	259	185	183	183
query54	507	406	388	388
query55	80	78	79	78
query56	249	248	250	248
query57	1253	1094	1122	1094
query58	214	206	210	206
query59	3262	3005	2946	2946
query60	264	243	249	243
query61	109	150	102	102
query62	773	679	662	662
query63	215	184	188	184
query64	1398	679	629	629
query65	3305	3177	3221	3177
query66	757	297	288	288
query67	16335	15642	15681	15642
query68	4004	589	561	561
query69	431	266	256	256
query70	1206	1176	1155	1155
query71	359	252	246	246
query72	6458	4088	4119	4088
query73	771	368	358	358
query74	9509	9283	9052	9052
query75	3398	2635	2660	2635
query76	1780	1117	1089	1089
query77	473	270	264	264
query78	10409	9520	9422	9422
query79	2291	620	621	620
query80	1439	433	429	429
query81	509	231	235	231
query82	1287	122	111	111
query83	165	157	138	138
query84	290	74	74	74
query85	1005	304	289	289
query86	413	280	298	280
query87	4700	4666	4604	4604
query88	3424	2233	2178	2178
query89	422	295	304	295
query90	1962	189	187	187
query91	142	103	101	101
query92	63	50	50	50
query93	2860	548	547	547
query94	881	296	293	293
query95	348	266	249	249
query96	625	278	276	276
query97	2860	2670	2678	2670
query98	216	198	194	194
query99	1635	1308	1305	1305
Total cold run time: 322098 ms
Total hot run time: 197674 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.3 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4fde56054bffbc8e3900284c2a25b03486e2c177, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.03
query3	0.23	0.07	0.07
query4	1.63	0.10	0.10
query5	0.46	0.39	0.40
query6	1.14	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.04	0.04
query9	0.57	0.53	0.50
query10	0.55	0.54	0.54
query11	0.14	0.10	0.11
query12	0.14	0.11	0.12
query13	0.61	0.61	0.62
query14	2.81	2.74	2.74
query15	0.90	0.83	0.83
query16	0.39	0.38	0.38
query17	1.00	1.10	1.05
query18	0.23	0.21	0.22
query19	2.00	1.73	2.03
query20	0.01	0.01	0.01
query21	15.41	0.59	0.55
query22	2.70	1.88	2.40
query23	17.03	1.18	0.82
query24	2.48	1.75	2.03
query25	0.42	0.19	0.08
query26	0.50	0.14	0.14
query27	0.04	0.05	0.04
query28	9.93	1.11	1.08
query29	12.56	3.24	3.25
query30	0.25	0.06	0.07
query31	2.88	0.39	0.39
query32	3.23	0.48	0.47
query33	3.00	3.06	3.04
query34	17.05	4.41	4.46
query35	4.51	4.43	4.49
query36	0.69	0.50	0.51
query37	0.09	0.06	0.05
query38	0.05	0.03	0.03
query39	0.03	0.02	0.03
query40	0.18	0.12	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.14 s
Total hot run time: 33.3 s

@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39880 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9cb6827dd47fde8bd54e1ecd5dd2aa86454a5a66, data reload: false

------ Round 1 ----------------------------------
q1	17589	7436	7277	7277
q2	2035	178	165	165
q3	10612	1094	1173	1094
q4	10547	729	745	729
q5	7617	2679	2651	2651
q6	245	147	143	143
q7	1018	627	605	605
q8	9248	1885	1916	1885
q9	6581	6413	6446	6413
q10	7021	2289	2299	2289
q11	465	265	261	261
q12	421	223	231	223
q13	17790	3024	3081	3024
q14	232	220	215	215
q15	583	532	521	521
q16	675	615	600	600
q17	980	518	572	518
q18	7122	6728	6637	6637
q19	1334	938	976	938
q20	460	185	181	181
q21	4006	3196	3220	3196
q22	375	320	315	315
Total cold run time: 106956 ms
Total hot run time: 39880 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7247	7228	7263	7228
q2	325	233	232	232
q3	2932	2818	2969	2818
q4	2097	1847	1824	1824
q5	5629	5640	5660	5640
q6	223	145	141	141
q7	2269	1790	1803	1790
q8	3384	3566	3482	3482
q9	8973	9030	8959	8959
q10	3588	3558	3566	3558
q11	604	509	519	509
q12	800	607	618	607
q13	12028	3215	3226	3215
q14	306	272	289	272
q15	564	543	549	543
q16	699	652	640	640
q17	1868	1651	1630	1630
q18	8294	7725	7373	7373
q19	1689	1570	1556	1556
q20	2105	1906	1919	1906
q21	5559	5485	5544	5485
q22	674	569	569	569
Total cold run time: 71857 ms
Total hot run time: 59977 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197233 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9cb6827dd47fde8bd54e1ecd5dd2aa86454a5a66, data reload: false

query1	1246	945	943	943
query2	6248	2031	2022	2022
query3	10960	4428	4343	4343
query4	67549	29160	23412	23412
query5	4894	481	462	462
query6	425	196	194	194
query7	5552	310	295	295
query8	326	244	237	237
query9	8741	2736	2711	2711
query10	452	243	241	241
query11	17202	15379	15878	15379
query12	157	103	105	103
query13	1458	443	411	411
query14	10924	7772	7358	7358
query15	215	186	197	186
query16	7147	499	464	464
query17	1140	623	559	559
query18	1782	303	311	303
query19	205	152	157	152
query20	117	113	115	113
query21	208	104	105	104
query22	4717	4458	4559	4458
query23	34697	34462	34636	34462
query24	5533	2478	2494	2478
query25	493	386	385	385
query26	656	157	161	157
query27	1804	281	284	281
query28	4461	2509	2493	2493
query29	673	415	430	415
query30	224	155	157	155
query31	1003	809	852	809
query32	68	56	65	56
query33	446	301	315	301
query34	968	520	548	520
query35	866	775	764	764
query36	1115	966	988	966
query37	124	83	80	80
query38	4487	4382	4392	4382
query39	1530	1477	1482	1477
query40	210	98	97	97
query41	44	41	45	41
query42	116	102	97	97
query43	535	502	504	502
query44	1245	858	870	858
query45	198	171	168	168
query46	1166	753	717	717
query47	2029	1942	1926	1926
query48	432	322	319	319
query49	735	399	402	399
query50	860	399	409	399
query51	7378	7171	7070	7070
query52	99	87	85	85
query53	263	186	180	180
query54	508	396	397	396
query55	79	77	78	77
query56	244	236	253	236
query57	1267	1142	1118	1118
query58	226	232	227	227
query59	3087	2973	2934	2934
query60	306	283	263	263
query61	137	128	128	128
query62	812	661	664	661
query63	213	229	184	184
query64	1354	645	636	636
query65	3256	3213	3207	3207
query66	701	298	297	297
query67	15815	15583	15685	15583
query68	4007	580	578	578
query69	427	267	245	245
query70	1172	1146	1165	1146
query71	359	252	248	248
query72	6367	4074	3746	3746
query73	761	359	362	359
query74	10425	9123	8916	8916
query75	3505	2717	2659	2659
query76	1782	1223	1068	1068
query77	487	268	270	268
query78	10623	9492	9457	9457
query79	2214	611	618	611
query80	1372	452	431	431
query81	504	235	236	235
query82	1242	120	125	120
query83	171	139	142	139
query84	279	72	71	71
query85	960	299	292	292
query86	415	310	298	298
query87	4939	4541	4622	4541
query88	3604	2235	2192	2192
query89	434	319	298	298
query90	1635	194	188	188
query91	138	104	100	100
query92	102	50	52	50
query93	2909	554	542	542
query94	698	291	289	289
query95	346	263	255	255
query96	640	284	282	282
query97	2860	2680	2667	2667
query98	223	200	201	200
query99	1608	1306	1320	1306
Total cold run time: 321336 ms
Total hot run time: 197233 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.81 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9cb6827dd47fde8bd54e1ecd5dd2aa86454a5a66, data reload: false

query1	0.03	0.02	0.04
query2	0.07	0.03	0.04
query3	0.23	0.07	0.08
query4	1.62	0.10	0.10
query5	0.42	0.42	0.39
query6	1.18	0.67	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.51	0.48
query10	0.55	0.59	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.12
query13	0.61	0.60	0.59
query14	2.72	2.73	2.80
query15	0.90	0.83	0.82
query16	0.38	0.37	0.39
query17	0.99	1.05	0.98
query18	0.23	0.20	0.21
query19	1.84	1.86	1.97
query20	0.02	0.01	0.01
query21	15.38	0.58	0.58
query22	2.59	2.56	1.94
query23	16.94	1.03	0.87
query24	3.27	1.25	1.05
query25	0.21	0.19	0.19
query26	0.51	0.13	0.13
query27	0.04	0.04	0.04
query28	10.46	1.10	1.07
query29	12.57	3.20	3.20
query30	0.24	0.07	0.06
query31	2.88	0.38	0.38
query32	3.26	0.45	0.46
query33	2.99	3.10	3.04
query34	16.99	4.43	4.44
query35	4.46	4.49	4.49
query36	0.66	0.50	0.48
query37	0.10	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.03
query40	0.17	0.12	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 106.66 s
Total hot run time: 32.81 s

@Jibing-Li
Copy link
Contributor Author

run p0

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 10, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@Jibing-Li Jibing-Li merged commit 1341e93 into apache:master Dec 11, 2024
28 of 30 checks passed
github-actions bot pushed a commit that referenced this pull request Dec 11, 2024
@Jibing-Li
[improvement](statistics)Support auto analyze columns that haven't be…
a2602bb 
…en analyzed for a long time. (#42399)

Support auto analyze columns that haven't been analyzed for a long time.
Add a very low priority job queue for auto analyze to process this kind
of columns.

The purpose of this change is to make sure all tables could be auto
analyzed within a certain time. In the earlier Doris versions, users
often encounter this kind of issues:
User load some new data to a large table everyday, but the change rate
(percentage of new data) is very low, because there is a large size of
old data. In this case, auto analyze for this table will not be
triggered for a very long time, because the default trigger threshold of
auto analyze is 40% (more than 40% of the data in a table is changed
since last analyze). This will probably cause a bad plan because
min/max/ndv statistics are outdated.
@github-actions github-actions bot mentioned this pull request Dec 11, 2024
Merged
Jibing-Li added a commit that referenced this pull request Dec 11, 2024
@github-actions @Jibing-Li
branch-3.0: [improvement](statistics)Support auto analyze columns tha…
bf0d060 
…t haven't been analyzed for a long time. #42399 (#45281)

Cherry-picked from #42399

Co-authored-by: James <[email protected]>
@Jibing-Li Jibing-Li mentioned this pull request Dec 11, 2024
Merged
Jibing-Li added a commit that referenced this pull request Dec 11, 2024
@Jibing-Li
[feature](statistics)Support auto analyze columns that haven't been a…
36695e8 
…nalyzed for a long time. #42399 (#45250)

backport: #42399
@Jibing-Li Jibing-Li deleted the 24hours branch December 13, 2024 02:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morrySnow morrySnow morrySnow approved these changes

@morningman morningman morningman approved these changes

@dataroaring dataroaring Awaiting requested review from dataroaring

@CalvinKirs CalvinKirs Awaiting requested review from CalvinKirs

Assignees

@morningman morningman

Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.4-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Jibing-Li @doris-robot @morningman @morrySnow