Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improve](routine-load) reduce routine load job transaction info storage in memory #30903

Merged
merged 1 commit into from
Feb 6, 2024
Merged

[improve](routine-load) reduce routine load job transaction info storage in memory #30903

merged 1 commit into from
Feb 6, 2024

Conversation

sollhui
Copy link
Contributor

@sollhui sollhui commented Feb 6, 2024

Proposed changes

Some transaction info of routine load job is invalid, which do not need storage, it will reduce memory using.

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@sollhui
Copy link
Contributor Author

sollhui commented Feb 6, 2024

run buildall

@sollhui sollhui changed the title [improve](routine-load) reduce routine load job transaction info storage [improve](routine-load) reduce routine load job transaction info storage in memory Feb 6, 2024
CalvinKirs
CalvinKirs previously approved these changes Feb 6, 2024
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 6, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 6, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 6, 2024

PR approved by anyone and no changes requested.

@sollhui
Copy link
Contributor Author

sollhui commented Feb 6, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Feb 6, 2024
JNSimba
JNSimba approved these changes Feb 6, 2024
View reviewed changes
Copy link
Member

@JNSimba JNSimba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 6, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 6, 2024

PR approved by at least one committer and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 37140 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 46a4d697ce4ca28226c59d5a271aa5fc7ee5c3bc, data reload: false

------ Round 1 ----------------------------------
q1	17610	4626	4409	4409
q2	2044	149	135	135
q3	10613	967	963	963
q4	4951	756	759	756
q5	7729	2825	2755	2755
q6	186	122	122	122
q7	1164	743	728	728
q8	9299	2024	2020	2020
q9	7239	6350	6329	6329
q10	8130	2479	2426	2426
q11	414	199	208	199
q12	742	293	304	293
q13	18004	3364	3330	3330
q14	268	255	257	255
q15	517	501	493	493
q16	463	410	413	410
q17	938	618	529	529
q18	6857	5939	6023	5939
q19	1566	1378	1318	1318
q20	595	355	340	340
q21	7020	3135	3100	3100
q22	812	308	291	291
Total cold run time: 107161 ms
Total hot run time: 37140 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4412	4436	4438	4436
q2	333	230	236	230
q3	3003	2792	2815	2792
q4	1870	1640	1693	1640
q5	5174	5250	5191	5191
q6	191	118	116	116
q7	2175	1764	1752	1752
q8	3137	3225	3314	3225
q9	8384	8345	8335	8335
q10	5816	3557	3516	3516
q11	535	447	456	447
q12	745	558	579	558
q13	6955	3142	3098	3098
q14	272	252	262	252
q15	533	493	495	493
q16	495	483	471	471
q17	1860	1729	1669	1669
q18	7999	7735	7515	7515
q19	8843	1535	1469	1469
q20	2119	1933	1908	1908
q21	4770	4638	4467	4467
q22	556	481	441	441
Total cold run time: 70177 ms
Total hot run time: 54021 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 181372 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 46a4d697ce4ca28226c59d5a271aa5fc7ee5c3bc, data reload: false

query1	938	345	338	338
query2	6534	2046	1923	1923
query3	6702	220	208	208
query4	31838	21937	21991	21937
query5	4295	430	413	413
query6	259	165	173	165
query7	4602	293	276	276
query8	260	173	179	173
query9	9070	2280	2263	2263
query10	435	207	206	206
query11	18169	15332	15288	15288
query12	133	77	77	77
query13	1616	417	424	417
query14	9306	6935	6856	6856
query15	250	181	180	180
query16	8103	261	249	249
query17	1841	536	495	495
query18	2099	266	274	266
query19	227	147	146	146
query20	83	77	79	77
query21	187	135	144	135
query22	4816	4545	4558	4545
query23	31075	30076	29964	29964
query24	10947	2756	2766	2756
query25	578	342	340	340
query26	1407	144	147	144
query27	2927	300	304	300
query28	7734	1817	1819	1817
query29	899	600	601	600
query30	279	135	140	135
query31	899	710	694	694
query32	89	59	58	58
query33	733	219	223	219
query34	1112	464	486	464
query35	865	766	771	766
query36	1025	910	897	897
query37	112	57	60	57
query38	3204	3166	3152	3152
query39	1293	1256	1252	1252
query40	273	95	101	95
query41	38	36	38	36
query42	98	95	90	90
query43	536	484	484	484
query44	1094	687	697	687
query45	194	186	182	182
query46	1060	695	664	664
query47	1606	1397	1511	1397
query48	429	342	366	342
query49	1138	290	278	278
query50	761	372	386	372
query51	5258	5161	5216	5161
query52	102	88	86	86
query53	341	267	259	259
query54	268	208	223	208
query55	81	74	78	74
query56	224	199	201	199
query57	1017	874	906	874
query58	204	173	180	173
query59	2502	2480	2311	2311
query60	239	210	211	210
query61	84	85	83	83
query62	673	353	404	353
query63	298	280	279	279
query64	6054	3669	3508	3508
query65	3238	3250	3222	3222
query66	1051	310	313	310
query67	14229	14097	13912	13912
query68	4038	528	545	528
query69	472	340	331	331
query70	1231	1270	1207	1207
query71	326	244	268	244
query72	5957	2826	2657	2657
query73	709	320	322	320
query74	6643	6261	6175	6175
query75	3002	2362	2311	2311
query76	2571	911	965	911
query77	362	233	232	232
query78	9198	8749	8589	8589
query79	4418	483	493	483
query80	1838	362	360	360
query81	517	211	197	197
query82	1132	89	79	79
query83	241	128	132	128
query84	285	86	76	76
query85	2416	336	323	323
query86	488	294	276	276
query87	3383	3214	3182	3182
query88	4458	2336	2339	2336
query89	460	364	351	351
query90	2041	168	164	164
query91	149	120	119	119
query92	59	48	45	45
query93	5370	501	480	480
query94	1305	176	174	174
query95	8119	7769	7938	7769
query96	599	269	283	269
query97	4232	4091	4122	4091
query98	225	196	190	190
query99	1093	699	678	678
Total cold run time: 297368 ms
Total hot run time: 181372 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.19 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 46a4d697ce4ca28226c59d5a271aa5fc7ee5c3bc, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.02	0.03
query3	0.22	0.06	0.06
query4	1.67	0.10	0.10
query5	0.54	0.52	0.52
query6	1.23	0.64	0.63
query7	0.01	0.01	0.02
query8	0.03	0.03	0.03
query9	0.56	0.51	0.48
query10	0.55	0.54	0.56
query11	0.12	0.08	0.08
query12	0.10	0.08	0.09
query13	0.61	0.60	0.61
query14	0.79	0.80	0.81
query15	0.80	0.78	0.78
query16	0.39	0.39	0.39
query17	1.02	1.03	1.02
query18	0.22	0.26	0.22
query19	1.89	1.81	1.78
query20	0.02	0.01	0.01
query21	15.43	0.59	0.57
query22	2.25	2.71	1.37
query23	17.38	0.81	0.80
query24	2.47	0.57	1.25
query25	0.34	0.22	0.08
query26	0.50	0.13	0.14
query27	0.04	0.05	0.04
query28	11.81	0.84	0.83
query29	12.52	3.27	3.15
query30	0.60	0.56	0.54
query31	2.79	0.35	0.34
query32	3.36	0.48	0.48
query33	3.21	3.22	3.23
query34	15.83	4.28	4.31
query35	4.33	4.29	4.32
query36	1.08	1.06	1.06
query37	0.06	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.01	0.01
query40	0.17	0.13	0.14
query41	0.07	0.02	0.01
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 105.21 s
Total hot run time: 30.19 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 46a4d697ce4ca28226c59d5a271aa5fc7ee5c3bc with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.4 seconds inserted 10000000 Rows, about 746K ops/s

@xiaokang xiaokang added the usercase Important user case type label label Feb 6, 2024
@yiguolei yiguolei merged commit c0e3287 into apache:master Feb 6, 2024
28 of 30 checks passed
@sollhui sollhui mentioned this pull request Feb 7, 2024
Merged
xiaokang pushed a commit that referenced this pull request Feb 7, 2024
@sollhui
[branch-2.0](multi-table-load) reduce routine load job transaction in…
b7f513e 
…fo storage in memory #30903 (#30967)
mymeiyi pushed a commit to mymeiyi/doris that referenced this pull request Feb 19, 2024
@sollhui @mymeiyi
[improve](routine-load) reduce routine load job transaction info stor…
baf50fa 
…age in memory apache#30903
@sollhui sollhui mentioned this pull request Feb 19, 2024
Merged
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@sollhui
[branch-2.0](multi-table-load) reduce routine load job transaction in…
f3b9c3f 
…fo storage in memory apache#30903 (apache#30967)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CalvinKirs CalvinKirs CalvinKirs approved these changes

@JNSimba JNSimba JNSimba approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.5-merged reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@sollhui @doris-robot @JNSimba @CalvinKirs @xiaokang @yiguolei