Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

branch-3.0: [feat](clone) Speed clone tablet via batch small file downloading #45061 #45191

Merged
merged 1 commit into from
Dec 10, 2024
Merged

branch-3.0: [feat](clone) Speed clone tablet via batch small file downloading #45061 #45191

merged 1 commit into from
Dec 10, 2024

Conversation

github-actions[bot]
Copy link
Contributor

@github-actions github-actions bot commented Dec 9, 2024

Cherry-picked from #45061

@w41ter
[feat](clone) Speed clone tablet via batch small file downloading (#4…
4fe1f93 
…5061)

Downloading small files is too slow and might cause the clone tablet
task to time out. This PR supports a batch downloading API to speed up
the downloading of small files.

Before

```
succeed to copy tablet 10088, total file size: 19256126 B, cost: 78674 ms, rate: 0.244758 MB/s
```

After

```
succeed to copy tablet 30157, total files: 20006, total file size: 19311624 B, cost: 4016 ms, rate: 4.80867 MB/s
```
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Dec 9, 2024
@dataroaring dataroaring reopened this Dec 9, 2024
@doris-robot
Copy link

run buildall

github-actions[bot]
github-actions bot commented Dec 9, 2024
View reviewed changes
Copy link
Contributor Author

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/http/http_channel.cpp
@@ -20,8 +20,8 @@
#include <event2/buffer.h>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'event2/buffer.h' file not found [clang-diagnostic-error]

#include <event2/buffer.h>
         ^

@doris-robot
Copy link

TPC-H: Total hot run time: 40719 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4fe1f93fba61bf9fef3114fca5b561bdd3ded91f, data reload: false

------ Round 1 ----------------------------------
q1	17617	7488	7328	7328
q2	2057	176	193	176
q3	10671	1058	1136	1058
q4	10562	836	759	759
q5	7767	2809	2811	2809
q6	235	146	146	146
q7	983	617	597	597
q8	9368	1940	2032	1940
q9	6609	6399	6426	6399
q10	7007	2308	2308	2308
q11	461	258	266	258
q12	414	214	206	206
q13	17803	2978	2957	2957
q14	237	211	205	205
q15	573	518	526	518
q16	666	620	601	601
q17	989	546	554	546
q18	7321	6610	6545	6545
q19	3152	1113	1002	1002
q20	492	205	193	193
q21	3962	3312	3197	3197
q22	1073	990	971	971
Total cold run time: 110019 ms
Total hot run time: 40719 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7447	7419	7231	7231
q2	325	247	226	226
q3	3058	2935	2862	2862
q4	2102	1786	1730	1730
q5	5659	5734	5743	5734
q6	226	142	139	139
q7	2235	1786	1754	1754
q8	3417	3522	3357	3357
q9	8919	8908	8922	8908
q10	3592	3511	3508	3508
q11	589	501	514	501
q12	819	645	609	609
q13	16442	3146	3157	3146
q14	319	271	275	271
q15	571	535	528	528
q16	705	680	669	669
q17	1892	1630	1601	1601
q18	8198	7759	7621	7621
q19	3543	1656	1592	1592
q20	2104	1868	1867	1867
q21	10115	5140	5359	5140
q22	1135	1001	1013	1001
Total cold run time: 83412 ms
Total hot run time: 59995 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196496 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4fe1f93fba61bf9fef3114fca5b561bdd3ded91f, data reload: false

query1	1313	934	921	921
query2	6234	2052	2101	2052
query3	10822	4209	3987	3987
query4	66966	29005	23415	23415
query5	5375	462	470	462
query6	455	175	170	170
query7	5673	312	307	307
query8	317	227	230	227
query9	9402	2707	2669	2669
query10	519	296	280	280
query11	17846	15196	15750	15196
query12	160	106	105	105
query13	1548	451	434	434
query14	10672	7133	7472	7133
query15	227	180	185	180
query16	7306	519	481	481
query17	1116	569	592	569
query18	1886	358	335	335
query19	226	168	170	168
query20	124	109	120	109
query21	213	107	120	107
query22	4721	4685	4657	4657
query23	34747	34158	35065	34158
query24	6257	2917	2931	2917
query25	536	403	407	403
query26	685	180	170	170
query27	1890	305	305	305
query28	4439	2535	2525	2525
query29	728	458	436	436
query30	252	163	166	163
query31	1024	813	835	813
query32	71	55	56	55
query33	442	284	272	272
query34	922	516	508	508
query35	871	777	730	730
query36	1114	954	930	930
query37	129	69	75	69
query38	4089	4130	4043	4043
query39	1529	1481	1499	1481
query40	220	101	98	98
query41	49	48	48	48
query42	105	97	95	95
query43	527	492	490	490
query44	1229	824	822	822
query45	188	167	171	167
query46	1168	745	714	714
query47	1971	1871	1876	1871
query48	493	393	393	393
query49	755	391	395	391
query50	855	425	430	425
query51	7310	7283	7047	7047
query52	104	91	92	91
query53	269	192	190	190
query54	571	464	459	459
query55	76	73	73	73
query56	263	243	248	243
query57	1220	1119	1098	1098
query58	220	227	207	207
query59	3193	2985	2848	2848
query60	282	264	271	264
query61	130	130	126	126
query62	778	633	668	633
query63	213	191	197	191
query64	1804	725	627	627
query65	3273	3187	3168	3168
query66	640	292	309	292
query67	15799	15355	15316	15316
query68	5023	568	555	555
query69	410	256	255	255
query70	1140	1134	1132	1132
query71	392	250	262	250
query72	6393	3894	3980	3894
query73	779	342	340	340
query74	10071	8955	8961	8955
query75	3316	2614	2651	2614
query76	2213	1021	1046	1021
query77	487	269	264	264
query78	10968	9598	9437	9437
query79	10299	582	596	582
query80	2287	428	432	428
query81	560	242	233	233
query82	1137	117	119	117
query83	304	149	137	137
query84	289	88	75	75
query85	1677	317	288	288
query86	497	314	298	298
query87	4423	4270	4369	4270
query88	5368	2422	2390	2390
query89	535	294	284	284
query90	2112	183	186	183
query91	185	145	148	145
query92	67	48	49	48
query93	7025	533	542	533
query94	1048	285	298	285
query95	345	257	248	248
query96	639	281	287	281
query97	3330	3128	3133	3128
query98	210	204	205	204
query99	1628	1306	1293	1293
Total cold run time: 341708 ms
Total hot run time: 196496 ms

@w41ter w41ter merged commit 4100a75 into branch-3.0 Dec 10, 2024
21 of 24 checks passed
@w41ter w41ter deleted the auto-pick-45061-branch-3.0 branch December 10, 2024 04:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@doris-robot @w41ter @dataroaring