Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix] let backup work on azure #46411

Merged
merged 1 commit into from
Jan 6, 2025
Merged

[fix] let backup work on azure #46411

merged 1 commit into from
Jan 6, 2025

Conversation

dataroaring
Copy link
Contributor

  1. use https for azure
  2. region is unnecessary for gcp.
  3. fe pass provider to be
  4. be listblob endless loop
  5. fe globlist use hierarchy api
  6. azure type should be s3 in fe
  7. add azure file system adaptor in gson

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring
Copy link
Contributor Author

run buildall

@dataroaring dataroaring force-pushed the azure_backup_glob branch 3 times, most recently from 399449a to 59a6e56 Compare January 4, 2025 05:18
@dataroaring
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32821 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 59a6e562f29fe9780fc9375a4f04960637e50551, data reload: false

------ Round 1 ----------------------------------
q1	17579	6130	6090	6090
q2	2049	308	163	163
q3	10835	1321	759	759
q4	10256	866	453	453
q5	7653	2196	2014	2014
q6	210	181	147	147
q7	906	764	601	601
q8	9366	1508	1170	1170
q9	5149	4902	4878	4878
q10	6738	2310	1878	1878
q11	482	296	262	262
q12	355	357	235	235
q13	17777	3631	3120	3120
q14	228	240	208	208
q15	566	499	490	490
q16	634	646	604	604
q17	591	855	319	319
q18	6988	6389	6431	6389
q19	4023	972	547	547
q20	298	311	182	182
q21	2732	2155	2004	2004
q22	357	334	308	308
Total cold run time: 105772 ms
Total hot run time: 32821 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6424	6292	6308	6292
q2	238	325	238	238
q3	2227	2654	2283	2283
q4	1407	1813	1347	1347
q5	4360	4755	4919	4755
q6	191	175	144	144
q7	2093	1927	1845	1845
q8	2670	2808	2757	2757
q9	7314	7294	7282	7282
q10	3077	3334	2759	2759
q11	560	506	484	484
q12	644	788	627	627
q13	3519	3876	3262	3262
q14	291	315	273	273
q15	564	511	501	501
q16	645	713	644	644
q17	1198	1739	1284	1284
q18	7821	7452	7235	7235
q19	878	1048	1162	1048
q20	1965	2052	1906	1906
q21	5778	5279	4914	4914
q22	598	603	617	603
Total cold run time: 54462 ms
Total hot run time: 52483 ms

gavinchou
gavinchou previously approved these changes Jan 4, 2025
View reviewed changes
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 4, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jan 4, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 4, 2025

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.89% (10129/26048)
Line Coverage: 29.92% (85682/286401)
Region Coverage: 29.02% (43728/150669)
Branch Coverage: 25.55% (22312/87314)
Coverage Report: http://coverage.selectdb-in.cc/coverage/59a6e562f29fe9780fc9375a4f04960637e50551_59a6e562f29fe9780fc9375a4f04960637e50551/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 196746 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 59a6e562f29fe9780fc9375a4f04960637e50551, data reload: false

query1	1316	926	920	920
query2	6303	2402	2294	2294
query3	10988	4769	4681	4681
query4	32915	24122	23559	23559
query5	4264	628	462	462
query6	285	215	201	201
query7	3991	499	299	299
query8	300	254	247	247
query9	9370	2610	2614	2610
query10	451	315	250	250
query11	18250	15385	15095	15095
query12	161	112	107	107
query13	1584	519	401	401
query14	10748	6999	7295	6999
query15	263	209	190	190
query16	8116	609	518	518
query17	1545	771	567	567
query18	2158	412	317	317
query19	201	213	156	156
query20	123	109	119	109
query21	213	127	112	112
query22	4513	4742	4429	4429
query23	34047	33685	33568	33568
query24	6441	2351	2374	2351
query25	514	451	394	394
query26	817	284	163	163
query27	2248	473	337	337
query28	5551	2461	2449	2449
query29	659	563	451	451
query30	218	186	157	157
query31	1021	944	861	861
query32	117	57	58	57
query33	487	370	295	295
query34	769	863	506	506
query35	819	827	762	762
query36	1022	1040	933	933
query37	125	108	79	79
query38	4457	4317	4465	4317
query39	1545	1523	1486	1486
query40	207	119	105	105
query41	48	74	42	42
query42	114	98	107	98
query43	509	551	505	505
query44	1340	825	822	822
query45	179	177	178	177
query46	921	1068	669	669
query47	2004	1998	1941	1941
query48	408	417	328	328
query49	739	503	410	410
query50	676	667	388	388
query51	7158	7210	7151	7151
query52	108	98	101	98
query53	229	257	191	191
query54	477	496	419	419
query55	84	81	78	78
query56	250	243	258	243
query57	1278	1254	1170	1170
query58	251	235	230	230
query59	3249	3309	3181	3181
query60	301	295	269	269
query61	160	112	111	111
query62	863	804	757	757
query63	221	188	184	184
query64	3672	1027	673	673
query65	3307	3301	3209	3209
query66	955	412	307	307
query67	16261	15915	15525	15525
query68	8911	761	514	514
query69	467	297	245	245
query70	1179	1119	1115	1115
query71	439	312	257	257
query72	6310	3845	3697	3697
query73	649	740	359	359
query74	10276	9238	9165	9165
query75	3887	3167	2626	2626
query76	3616	1201	768	768
query77	764	358	282	282
query78	10224	10185	9487	9487
query79	4150	831	575	575
query80	839	516	413	413
query81	491	263	219	219
query82	621	152	123	123
query83	185	162	144	144
query84	296	96	72	72
query85	720	350	295	295
query86	406	320	314	314
query87	4424	4562	4454	4454
query88	4638	2151	2146	2146
query89	421	330	294	294
query90	1784	181	183	181
query91	132	149	106	106
query92	64	56	57	56
query93	2171	891	534	534
query94	647	402	289	289
query95	335	263	253	253
query96	498	611	278	278
query97	2956	2955	2847	2847
query98	219	212	192	192
query99	1707	1534	1476	1476
Total cold run time: 299310 ms
Total hot run time: 196746 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.25 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 59a6e562f29fe9780fc9375a4f04960637e50551, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.07
query4	1.61	0.10	0.11
query5	0.43	0.44	0.40
query6	1.17	0.64	0.65
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.60	0.50	0.50
query10	0.55	0.56	0.55
query11	0.15	0.10	0.10
query12	0.14	0.12	0.11
query13	0.62	0.60	0.61
query14	2.71	2.74	2.73
query15	0.89	0.82	0.83
query16	0.38	0.38	0.39
query17	1.07	1.05	1.06
query18	0.22	0.21	0.21
query19	1.88	1.78	1.96
query20	0.01	0.01	0.02
query21	15.38	0.91	0.56
query22	0.74	0.84	0.70
query23	15.21	1.42	0.63
query24	3.35	0.71	1.57
query25	0.14	0.12	0.12
query26	0.35	0.14	0.14
query27	0.07	0.05	0.05
query28	13.95	1.56	1.05
query29	12.61	3.97	3.33
query30	0.24	0.09	0.07
query31	2.83	0.59	0.38
query32	3.22	0.54	0.45
query33	3.12	3.17	3.13
query34	16.68	5.09	4.50
query35	4.53	4.45	4.45
query36	0.64	0.50	0.49
query37	0.10	0.06	0.06
query38	0.04	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.14	0.14
query41	0.07	0.03	0.02
query42	0.04	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 106.4 s
Total hot run time: 31.25 s

[fix] let backup work on azure
d59742d 
1. use https for azure
2. region is unnecessary for gcp.
3. fe pass provider to be
4. be listblob endless loop
5. fe globlist use hierarchy api
6. azure type should be s3 in fe
7. add azure file system adaptor in gson
@dataroaring
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jan 5, 2025
@doris-robot
Copy link

TPC-H: Total hot run time: 32642 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d59742db242c16497dcef324f19259a660348476, data reload: false

------ Round 1 ----------------------------------
q1	17614	6165	6027	6027
q2	2051	295	173	173
q3	10625	1229	734	734
q4	10237	864	420	420
q5	7734	2174	1972	1972
q6	203	177	143	143
q7	886	745	607	607
q8	9239	1356	1145	1145
q9	5284	4899	5026	4899
q10	6910	2363	1883	1883
q11	501	282	256	256
q12	352	391	224	224
q13	17767	3670	3058	3058
q14	227	239	210	210
q15	560	503	496	496
q16	634	622	598	598
q17	558	843	331	331
q18	6841	6431	6484	6431
q19	2053	981	565	565
q20	291	317	187	187
q21	2872	2225	1982	1982
q22	350	325	301	301
Total cold run time: 103789 ms
Total hot run time: 32642 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6314	6230	6256	6230
q2	233	325	235	235
q3	2237	2634	2335	2335
q4	1387	1822	1366	1366
q5	4351	4713	4828	4713
q6	189	177	141	141
q7	2051	1952	1808	1808
q8	2637	2803	2700	2700
q9	7330	7241	7300	7241
q10	3107	3336	2850	2850
q11	589	505	494	494
q12	624	719	575	575
q13	3504	3909	3206	3206
q14	275	311	285	285
q15	554	524	503	503
q16	657	676	651	651
q17	1234	1722	1248	1248
q18	7626	7418	7464	7418
q19	866	1207	1101	1101
q20	2022	2005	1919	1919
q21	5655	5179	4832	4832
q22	607	609	617	609
Total cold run time: 54049 ms
Total hot run time: 52460 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.89% (10129/26048)
Line Coverage: 29.93% (85721/286406)
Region Coverage: 29.02% (43730/150672)
Branch Coverage: 25.57% (22325/87316)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d59742db242c16497dcef324f19259a660348476_d59742db242c16497dcef324f19259a660348476/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 196909 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d59742db242c16497dcef324f19259a660348476, data reload: false

query1	1305	930	915	915
query2	6465	2232	2311	2232
query3	11035	4627	4670	4627
query4	32782	24112	23890	23890
query5	4278	596	436	436
query6	278	190	178	178
query7	3983	489	306	306
query8	288	238	242	238
query9	9287	2609	2607	2607
query10	468	298	241	241
query11	18057	15652	15236	15236
query12	157	106	97	97
query13	1585	534	398	398
query14	9782	7165	7371	7165
query15	267	211	185	185
query16	8065	694	489	489
query17	1603	795	597	597
query18	2142	406	315	315
query19	190	181	157	157
query20	121	115	109	109
query21	205	125	110	110
query22	4967	4663	4864	4663
query23	34933	33276	34154	33276
query24	7412	2328	2311	2311
query25	462	458	421	421
query26	782	281	150	150
query27	2195	473	338	338
query28	5411	2456	2427	2427
query29	536	557	414	414
query30	206	197	150	150
query31	970	927	833	833
query32	86	63	63	63
query33	503	344	308	308
query34	742	863	501	501
query35	817	828	775	775
query36	1032	1049	933	933
query37	120	106	70	70
query38	4210	4127	4179	4127
query39	1533	1490	1447	1447
query40	206	120	126	120
query41	45	45	42	42
query42	123	103	101	101
query43	521	523	489	489
query44	1309	810	810	810
query45	178	175	170	170
query46	862	1049	648	648
query47	2041	2029	1973	1973
query48	374	416	331	331
query49	709	487	401	401
query50	649	670	397	397
query51	7319	7156	7203	7156
query52	108	100	96	96
query53	242	251	184	184
query54	491	492	428	428
query55	84	80	79	79
query56	266	268	276	268
query57	1266	1236	1187	1187
query58	242	222	239	222
query59	3258	3242	3191	3191
query60	289	259	243	243
query61	121	103	110	103
query62	866	805	790	790
query63	236	199	186	186
query64	3640	1015	640	640
query65	3379	3277	3228	3228
query66	813	434	319	319
query67	16170	15833	15676	15676
query68	9009	703	529	529
query69	484	299	256	256
query70	1216	1097	1194	1097
query71	416	281	250	250
query72	6169	3804	3883	3804
query73	630	752	349	349
query74	10407	9079	9021	9021
query75	3762	3144	2650	2650
query76	3407	1181	779	779
query77	760	360	274	274
query78	10383	10202	9478	9478
query79	3076	793	598	598
query80	704	512	419	419
query81	515	266	228	228
query82	624	141	118	118
query83	162	159	146	146
query84	242	92	70	70
query85	780	348	313	313
query86	405	324	301	301
query87	4590	4392	4467	4392
query88	5197	2159	2253	2159
query89	411	324	291	291
query90	1867	185	188	185
query91	134	134	107	107
query92	66	53	53	53
query93	2805	849	525	525
query94	615	396	287	287
query95	339	268	247	247
query96	491	611	279	279
query97	2917	2937	2830	2830
query98	218	206	193	193
query99	1477	1542	1468	1468
Total cold run time: 299345 ms
Total hot run time: 196909 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d59742db242c16497dcef324f19259a660348476, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.04
query3	0.23	0.07	0.06
query4	1.62	0.10	0.10
query5	0.43	0.43	0.42
query6	1.14	0.65	0.64
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.58	0.50	0.50
query10	0.56	0.55	0.56
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.80	2.87	2.81
query15	0.88	0.82	0.82
query16	0.36	0.38	0.37
query17	1.05	1.07	1.05
query18	0.22	0.21	0.21
query19	1.95	1.83	1.91
query20	0.02	0.01	0.01
query21	15.36	0.86	0.59
query22	0.75	0.82	0.65
query23	15.26	1.45	0.62
query24	3.28	1.45	1.53
query25	0.15	0.22	0.06
query26	0.27	0.14	0.13
query27	0.05	0.06	0.04
query28	14.32	1.48	1.04
query29	12.54	3.97	3.26
query30	0.25	0.08	0.06
query31	2.84	0.59	0.37
query32	3.23	0.54	0.47
query33	3.17	3.25	3.19
query34	16.83	5.11	4.52
query35	4.62	4.49	4.47
query36	0.63	0.52	0.48
query37	0.10	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.14
query41	0.08	0.02	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.97 s
Total hot run time: 32 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 6, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 6, 2025

PR approved by at least one committer and no changes requested.

@gavinchou gavinchou merged commit 824c357 into apache:master Jan 6, 2025
24 of 27 checks passed
github-actions bot pushed a commit that referenced this pull request Jan 6, 2025
@dataroaring
[fix] let backup work on azure (#46411)
f0450af 
1. use https for azure
2. region is unnecessary for gcp.
3. fe pass provider to be
4. be listblob endless loop
5. fe globlist use hierarchy api
6. azure type should be s3 in fe
7. add azure file system adaptor in gson
@github-actions github-actions bot mentioned this pull request Jan 6, 2025
Merged
dataroaring added a commit that referenced this pull request Jan 10, 2025
@github-actions @dataroaring
branch-3.0: [fix] let backup work on azure #46411 (#46445)
9b375be 
Cherry-picked from #46411

Co-authored-by: Yongqiang YANG <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gavinchou gavinchou gavinchou approved these changes

@kaijchen kaijchen kaijchen approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.4-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@dataroaring @hello-stephen @doris-robot @kaijchen @gavinchou