Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](meta-cache) fix refreshOnlyCatalogCache when use_meta_cache = false #44363

Merged
merged 3 commits into from
Nov 22, 2024
Merged

[fix](meta-cache) fix refreshOnlyCatalogCache when use_meta_cache = false #44363

merged 3 commits into from
Nov 22, 2024

Conversation

suxiaogang223
Copy link
Contributor

@suxiaogang223 suxiaogang223 commented Nov 20, 2024 •
edited by morningman
Loading

What problem does this PR solve?

Bug: When hive catalog set use_meta_cache=false, refresh catalog cannot update the database list.
Fix: Set initialized = false in refreshOnlyCatalogCache().

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@suxiaogang223
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40137 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d08e68f725392a6e6b88cb70de9e32fbc17a7a92, data reload: false

------ Round 1 ----------------------------------
q1	17563	7399	7223	7223
q2	2044	181	169	169
q3	10652	1078	1139	1078
q4	10217	727	793	727
q5	7607	2682	2673	2673
q6	239	148	146	146
q7	987	623	599	599
q8	9251	1864	1900	1864
q9	6655	6473	6523	6473
q10	7834	2341	2307	2307
q11	464	260	256	256
q12	434	220	221	220
q13	17830	3041	3061	3041
q14	243	223	215	215
q15	569	523	522	522
q16	650	589	585	585
q17	975	547	573	547
q18	7366	6837	6855	6837
q19	1333	1010	1121	1010
q20	477	181	179	179
q21	4014	3155	3194	3155
q22	374	320	311	311
Total cold run time: 107778 ms
Total hot run time: 40137 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7349	7349	7315	7315
q2	331	230	230	230
q3	3106	3027	3054	3027
q4	2164	2015	1943	1943
q5	5783	5833	5840	5833
q6	241	145	149	145
q7	2333	1897	1868	1868
q8	3446	3653	3493	3493
q9	9039	8906	8891	8891
q10	3576	3550	3554	3550
q11	594	500	522	500
q12	854	636	600	600
q13	16461	3198	3274	3198
q14	302	277	275	275
q15	570	533	530	530
q16	711	672	656	656
q17	1849	1620	1618	1618
q18	8436	7925	7707	7707
q19	1730	1526	1474	1474
q20	2137	1866	1842	1842
q21	5644	5511	5458	5458
q22	638	548	566	548
Total cold run time: 77294 ms
Total hot run time: 60701 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197933 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d08e68f725392a6e6b88cb70de9e32fbc17a7a92, data reload: false

query1	1282	924	901	901
query2	6275	2140	2043	2043
query3	10771	3975	3926	3926
query4	67401	30799	23882	23882
query5	5022	453	444	444
query6	321	194	180	180
query7	4902	314	294	294
query8	297	229	237	229
query9	6400	2680	2660	2660
query10	409	250	254	250
query11	15788	15277	15920	15277
query12	167	106	105	105
query13	1048	446	421	421
query14	10209	7433	7510	7433
query15	221	196	203	196
query16	7392	442	496	442
query17	1119	579	571	571
query18	1910	294	290	290
query19	198	151	150	150
query20	116	119	109	109
query21	206	103	106	103
query22	4947	4774	4611	4611
query23	34536	34377	34464	34377
query24	5749	2562	2481	2481
query25	492	401	379	379
query26	640	149	148	148
query27	1882	283	293	283
query28	4474	2472	2446	2446
query29	688	443	433	433
query30	222	145	144	144
query31	1003	823	841	823
query32	67	61	57	57
query33	457	288	292	288
query34	931	517	542	517
query35	826	726	738	726
query36	1082	954	947	947
query37	151	85	70	70
query38	4534	4389	4533	4389
query39	1508	1460	1480	1460
query40	203	95	98	95
query41	45	43	42	42
query42	104	97	104	97
query43	536	482	499	482
query44	1173	840	840	840
query45	196	172	171	171
query46	1160	751	732	732
query47	2061	1957	1952	1952
query48	421	316	313	313
query49	714	397	405	397
query50	854	385	401	385
query51	7613	7168	7146	7146
query52	95	91	87	87
query53	253	174	174	174
query54	505	409	406	406
query55	80	71	77	71
query56	271	235	229	229
query57	1325	1142	1172	1142
query58	223	222	214	214
query59	3339	3064	3086	3064
query60	272	241	244	241
query61	104	110	106	106
query62	825	679	683	679
query63	213	191	190	190
query64	1342	666	693	666
query65	3266	3202	3246	3202
query66	698	312	303	303
query67	16064	16067	15856	15856
query68	3781	581	550	550
query69	473	259	253	253
query70	1191	1141	1121	1121
query71	444	244	244	244
query72	6693	4087	3994	3994
query73	781	364	362	362
query74	10167	8984	9071	8984
query75	3448	2642	2680	2642
query76	2473	1144	1143	1143
query77	584	269	286	269
query78	10610	9560	9400	9400
query79	1561	588	591	588
query80	949	436	497	436
query81	533	234	232	232
query82	240	121	116	116
query83	196	150	152	150
query84	274	72	71	71
query85	1017	303	300	300
query86	383	271	309	271
query87	4728	4691	4628	4628
query88	3910	2230	2213	2213
query89	408	288	302	288
query90	2252	185	188	185
query91	138	112	105	105
query92	69	50	49	49
query93	2436	544	543	543
query94	780	295	294	294
query95	343	245	247	245
query96	638	277	277	277
query97	2872	2684	2750	2684
query98	224	198	201	198
query99	1695	1343	1331	1331
Total cold run time: 315889 ms
Total hot run time: 197933 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d08e68f725392a6e6b88cb70de9e32fbc17a7a92, data reload: false

query1	0.03	0.04	0.05
query2	0.08	0.03	0.03
query3	0.24	0.07	0.08
query4	1.61	0.10	0.11
query5	0.43	0.42	0.41
query6	1.16	0.66	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.50	0.50
query10	0.56	0.57	0.56
query11	0.14	0.11	0.10
query12	0.14	0.12	0.12
query13	0.62	0.62	0.59
query14	2.84	2.73	2.80
query15	0.89	0.83	0.82
query16	0.39	0.39	0.39
query17	1.06	1.06	1.07
query18	0.22	0.20	0.21
query19	1.93	1.82	1.97
query20	0.02	0.01	0.01
query21	15.36	0.61	0.59
query22	2.68	2.71	1.35
query23	17.09	0.87	0.84
query24	2.55	0.79	1.52
query25	0.37	0.14	0.06
query26	0.31	0.13	0.13
query27	0.04	0.04	0.04
query28	10.85	1.09	1.08
query29	12.56	3.32	3.33
query30	0.25	0.06	0.06
query31	2.87	0.38	0.38
query32	3.29	0.46	0.47
query33	3.00	3.01	3.04
query34	16.85	4.44	4.41
query35	4.51	4.44	4.52
query36	0.68	0.49	0.50
query37	0.09	0.07	0.06
query38	0.04	0.04	0.03
query39	0.03	0.03	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.73 s
Total hot run time: 32 s

morningman
morningman approved these changes Nov 22, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman added usercase Important user case type label and removed usercase Important user case type label labels Nov 22, 2024
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 22, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morningman morningman merged commit 9c1f6ab into apache:master Nov 22, 2024
35 of 37 checks passed
github-actions bot pushed a commit that referenced this pull request Nov 22, 2024
@suxiaogang223
[fix](meta-cache) fix refreshOnlyCatalogCache when use_meta_cache = f…
68c8b87 
…alse (#44363)

### What problem does this PR solve?
Bug: When hive catalog set use_meta_cache=false, refresh catalog cannot
update the database list.
Fix: Set initialized = false in `refreshOnlyCatalogCache()`.
@github-actions github-actions bot mentioned this pull request Nov 22, 2024
Merged
github-actions bot pushed a commit that referenced this pull request Nov 22, 2024
@suxiaogang223
[fix](meta-cache) fix refreshOnlyCatalogCache when use_meta_cache = f…
5c639e3 
…alse (#44363)

### What problem does this PR solve?
Bug: When hive catalog set use_meta_cache=false, refresh catalog cannot
update the database list.
Fix: Set initialized = false in `refreshOnlyCatalogCache()`.
@github-actions github-actions bot mentioned this pull request Nov 22, 2024
Merged
@morningman morningman added the usercase Important user case type label label Nov 22, 2024
yiguolei pushed a commit that referenced this pull request Nov 22, 2024
@github-actions @suxiaogang223
branch-2.1: [fix](meta-cache) fix refreshOnlyCatalogCache when use_me…
ec06804 
…ta_cache = false #44363 (#44461)

Cherry-picked from #44363

Co-authored-by: Socrates <[email protected]>
@suxiaogang223 suxiaogang223 deleted the fix_refresh_catalog branch November 29, 2024 03:47
dataroaring pushed a commit that referenced this pull request Nov 29, 2024
@github-actions @suxiaogang223
branch-3.0: [fix](meta-cache) fix refreshOnlyCatalogCache when use_me…
0c51bc6 
…ta_cache = false #44363 (#44458)

Cherry-picked from #44363

Co-authored-by: Socrates <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hubgeter hubgeter hubgeter approved these changes

@wuwenchi wuwenchi wuwenchi approved these changes

@morningman morningman morningman approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.4-merged reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@suxiaogang223 @doris-robot @morningman @wuwenchi @hubgeter @yiguolei @dataroaring