Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](statistics)Fix clear stale mv row count bug. (#41653) #41805

Merged
merged 1 commit into from
Oct 14, 2024
Merged

[fix](statistics)Fix clear stale mv row count bug. (#41653) #41805

merged 1 commit into from
Oct 14, 2024

Conversation

Jibing-Li
Copy link
Contributor

backport: #41653

@Jibing-Li
[fix](statistics)Fix clear stale mv row count bug. (apache#41653)
9538619 
When drop an mv, the row count for that mv should be removed in memory
as well. Before we use OlapTable.getIndexIds to get all index ids, but
this function always returns null. This pr is to fix this bug, use
getIndexIdList instead.
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Jibing-Li Jibing-Li marked this pull request as ready for review October 14, 2024 09:45
@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 48779 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9538619981a56b128041e09f8a2f4fdd72109ff1, data reload: false

------ Round 1 ----------------------------------
q1	17792	4332	4305	4305
q2	2030	192	152	152
q3	10359	1892	1914	1892
q4	10362	1240	1273	1240
q5	8833	3828	3913	3828
q6	228	120	121	120
q7	2014	1627	1588	1588
q8	9307	2732	2688	2688
q9	10190	9738	9761	9738
q10	8607	3518	3495	3495
q11	420	235	251	235
q12	464	297	294	294
q13	18352	3975	4011	3975
q14	345	318	333	318
q15	513	474	459	459
q16	531	465	446	446
q17	1132	919	897	897
q18	7270	6799	6840	6799
q19	1671	1624	1545	1545
q20	520	297	310	297
q21	4416	4136	4071	4071
q22	494	403	397	397
Total cold run time: 115850 ms
Total hot run time: 48779 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4308	4255	4324	4255
q2	319	220	222	220
q3	4127	4164	4157	4157
q4	2728	2747	2741	2741
q5	7175	7064	7114	7064
q6	239	117	117	117
q7	3227	2819	2874	2819
q8	4330	4439	4471	4439
q9	13739	13607	13491	13491
q10	4202	4239	4262	4239
q11	761	680	671	671
q12	1027	830	853	830
q13	6511	3768	3739	3739
q14	456	431	410	410
q15	504	464	445	445
q16	619	596	587	587
q17	3798	3828	3881	3828
q18	8733	8706	8817	8706
q19	1693	1643	1701	1643
q20	2331	2117	2105	2105
q21	8445	8429	8362	8362
q22	1030	912	928	912
Total cold run time: 80302 ms
Total hot run time: 75780 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 211753 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9538619981a56b128041e09f8a2f4fdd72109ff1, data reload: false

query1	942	391	412	391
query2	6541	2171	1958	1958
query3	6918	206	203	203
query4	23195	21356	21353	21353
query5	19729	6548	6520	6520
query6	294	211	231	211
query7	4332	294	303	294
query8	253	281	238	238
query9	3083	2685	2609	2609
query10	460	302	295	295
query11	15748	15258	15001	15001
query12	125	76	74	74
query13	1021	437	439	437
query14	17178	13621	13740	13621
query15	357	221	232	221
query16	5916	284	267	267
query17	1735	912	907	907
query18	894	318	317	317
query19	211	155	158	155
query20	98	93	101	93
query21	189	101	91	91
query22	5115	4932	5034	4932
query23	34260	33270	33421	33270
query24	7847	6261	6319	6261
query25	521	421	436	421
query26	1330	162	158	158
query27	2428	299	295	295
query28	6101	2255	2216	2216
query29	2836	2739	2758	2739
query30	240	167	163	163
query31	944	745	743	743
query32	73	61	60	60
query33	456	262	262	262
query34	889	482	479	479
query35	1124	929	861	861
query36	1344	1148	1109	1109
query37	175	56	63	56
query38	3048	2938	2915	2915
query39	1390	1320	1330	1320
query40	308	95	91	91
query41	39	40	37	37
query42	91	93	87	87
query43	630	619	617	617
query44	1152	721	725	721
query45	248	231	232	231
query46	1221	949	973	949
query47	1815	2129	1697	1697
query48	526	412	404	404
query49	650	386	368	368
query50	853	624	591	591
query51	4703	4643	4714	4643
query52	93	82	88	82
query53	240	186	196	186
query54	2656	2476	2464	2464
query55	88	80	85	80
query56	222	193	195	193
query57	1244	1161	1143	1143
query58	225	208	209	208
query59	3418	3229	3096	3096
query60	222	220	193	193
query61	95	98	94	94
query62	825	453	458	453
query63	205	184	177	177
query64	3531	1570	1500	1500
query65	3626	3556	3547	3547
query66	783	427	406	406
query67	15494	16607	15196	15196
query68	9828	648	647	647
query69	488	266	270	266
query70	1523	1644	1317	1317
query71	409	307	314	307
query72	6797	4932	4797	4797
query73	749	315	331	315
query74	6267	5799	5822	5799
query75	5246	3734	3717	3717
query76	5606	1171	1193	1171
query77	970	269	255	255
query78	12117	11794	11633	11633
query79	7323	650	646	646
query80	1379	400	384	384
query81	491	234	238	234
query82	1526	101	95	95
query83	172	135	135	135
query84	256	73	70	70
query85	890	318	327	318
query86	341	305	291	291
query87	3267	3026	3020	3020
query88	4351	2293	2300	2293
query89	483	284	294	284
query90	1989	210	213	210
query91	161	131	137	131
query92	62	53	54	53
query93	6291	562	530	530
query94	708	215	206	206
query95	2096	1932	1914	1914
query96	646	317	338	317
query97	6530	6349	6355	6349
query98	223	206	203	203
query99	2794	876	914	876
Total cold run time: 316522 ms
Total hot run time: 211753 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.85 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9538619981a56b128041e09f8a2f4fdd72109ff1, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.03	0.03
query3	0.25	0.05	0.06
query4	1.79	0.06	0.07
query5	0.54	0.52	0.51
query6	1.23	0.62	0.61
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.52	0.47	0.48
query10	0.51	0.52	0.53
query11	0.12	0.08	0.08
query12	0.12	0.08	0.09
query13	0.62	0.61	0.61
query14	0.80	0.81	0.79
query15	0.76	0.76	0.74
query16	0.36	0.36	0.36
query17	1.03	0.98	1.00
query18	0.23	0.25	0.23
query19	1.93	1.86	1.85
query20	0.02	0.01	0.01
query21	15.45	0.56	0.57
query22	2.32	1.95	1.96
query23	17.39	0.96	0.91
query24	6.50	0.90	1.07
query25	0.37	0.08	0.04
query26	0.70	0.15	0.14
query27	0.04	0.04	0.03
query28	6.71	0.76	0.77
query29	12.64	2.18	2.30
query30	0.59	0.52	0.54
query31	2.79	0.39	0.38
query32	3.40	0.50	0.49
query33	3.12	3.07	3.06
query34	15.26	4.78	4.79
query35	4.86	4.87	4.88
query36	1.06	1.00	1.02
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.02	0.02
query40	0.15	0.13	0.15
query41	0.06	0.01	0.01
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 104.52 s
Total hot run time: 30.85 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 9538619981a56b128041e09f8a2f4fdd72109ff1 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          57 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       22.0 seconds inserted 10000000 Rows, about 454K ops/s

@Jibing-Li Jibing-Li merged commit 033220e into apache:branch-2.0 Oct 14, 2024
22 of 24 checks passed
@Jibing-Li Jibing-Li deleted the indexrow2.0 branch October 15, 2024 01:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
kind/test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Jibing-Li @doris-robot