Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feat](binlog) Support drop rollup binlog #44677

Merged
merged 1 commit into from
Nov 28, 2024
Merged

[feat](binlog) Support drop rollup binlog #44677

merged 1 commit into from
Nov 28, 2024

Conversation

w41ter
Copy link
Contributor

@w41ter w41ter commented Nov 27, 2024 •
edited
Loading

What problem does this PR solve?

Issue Number: ref #43435

Related PR: selectdb/ccr-syncer#269

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@w41ter
Copy link
Contributor Author

w41ter commented Nov 27, 2024

run buildall

@w41ter w41ter force-pushed the support_drop_rollup_binlog branch 3 times, most recently from 544aa73 to 0ea9a11 Compare November 27, 2024 09:21
@w41ter w41ter force-pushed the support_drop_rollup_binlog branch from 0ea9a11 to 351cfc4 Compare November 27, 2024 09:34
dataroaring
dataroaring approved these changes Nov 27, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring
Copy link
Contributor

run buildall

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 27, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 39910 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 351cfc4822399918590c4effc67aad4b31c2eb62, data reload: false

------ Round 1 ----------------------------------
q1	17843	7692	7243	7243
q2	2057	174	167	167
q3	10661	1104	1228	1104
q4	10574	751	707	707
q5	7600	2761	2791	2761
q6	240	153	146	146
q7	981	628	596	596
q8	9244	1887	1927	1887
q9	6547	6376	6378	6376
q10	6998	2314	2330	2314
q11	470	261	263	261
q12	423	212	216	212
q13	17769	3032	3067	3032
q14	235	213	213	213
q15	563	545	514	514
q16	652	574	601	574
q17	975	564	554	554
q18	7470	6719	6571	6571
q19	1340	1052	1077	1052
q20	507	184	178	178
q21	4134	3139	3191	3139
q22	388	322	309	309
Total cold run time: 107671 ms
Total hot run time: 39910 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7304	7238	7272	7238
q2	326	229	225	225
q3	2919	2858	2938	2858
q4	2050	1788	1830	1788
q5	5721	5667	5681	5667
q6	225	146	143	143
q7	2241	1795	1817	1795
q8	3438	3557	3495	3495
q9	8877	8920	8991	8920
q10	3578	3565	3561	3561
q11	592	513	511	511
q12	828	590	593	590
q13	10512	3246	3224	3224
q14	319	270	278	270
q15	582	516	518	516
q16	672	658	636	636
q17	1846	1641	1656	1641
q18	8318	7724	7786	7724
q19	1696	1589	1589	1589
q20	2118	1927	1866	1866
q21	5528	5399	5400	5399
q22	646	625	576	576
Total cold run time: 70336 ms
Total hot run time: 60232 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197428 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 351cfc4822399918590c4effc67aad4b31c2eb62, data reload: false

query1	1263	959	961	959
query2	6216	2075	2079	2075
query3	10882	3957	3980	3957
query4	67478	27213	23728	23728
query5	4965	448	468	448
query6	421	185	185	185
query7	5519	302	293	293
query8	312	238	242	238
query9	8803	2708	2691	2691
query10	435	259	252	252
query11	17214	15278	15997	15278
query12	159	99	108	99
query13	1448	444	416	416
query14	10604	7402	7374	7374
query15	218	185	188	185
query16	7211	510	534	510
query17	1091	627	584	584
query18	1903	320	314	314
query19	210	160	158	158
query20	124	112	115	112
query21	215	104	101	101
query22	4704	4615	4258	4258
query23	35095	34430	34638	34430
query24	5494	2494	2491	2491
query25	488	394	402	394
query26	647	159	156	156
query27	1832	290	319	290
query28	4025	2490	2466	2466
query29	674	429	417	417
query30	219	149	152	149
query31	1018	837	845	837
query32	71	52	56	52
query33	445	310	300	300
query34	940	516	531	516
query35	874	769	776	769
query36	1085	952	965	952
query37	125	76	73	73
query38	4473	4500	4467	4467
query39	1544	1471	1490	1471
query40	214	97	98	97
query41	44	42	42	42
query42	138	100	99	99
query43	556	524	502	502
query44	1214	847	803	803
query45	190	177	174	174
query46	1171	722	716	716
query47	2045	1914	1951	1914
query48	415	311	330	311
query49	736	408	400	400
query50	821	415	395	395
query51	7423	7384	7133	7133
query52	101	88	87	87
query53	261	178	184	178
query54	513	397	418	397
query55	101	73	74	73
query56	261	240	256	240
query57	1323	1195	1123	1123
query58	215	235	213	213
query59	3349	3006	3154	3006
query60	272	241	249	241
query61	110	108	112	108
query62	785	668	674	668
query63	214	194	192	192
query64	1379	668	635	635
query65	3321	3212	3180	3180
query66	702	308	301	301
query67	15966	15647	15786	15647
query68	4163	560	549	549
query69	431	259	262	259
query70	1212	1125	1158	1125
query71	356	253	269	253
query72	6542	4045	3963	3963
query73	761	361	367	361
query74	10299	9024	9059	9024
query75	3428	2672	2676	2672
query76	1836	1051	1062	1051
query77	465	375	285	285
query78	10490	9381	9412	9381
query79	2106	599	611	599
query80	1397	432	440	432
query81	538	234	222	222
query82	1295	121	121	121
query83	196	151	146	146
query84	278	70	71	70
query85	1073	305	293	293
query86	400	302	315	302
query87	5092	4677	4666	4666
query88	3668	2241	2210	2210
query89	417	283	302	283
query90	1952	190	189	189
query91	135	103	103	103
query92	63	49	49	49
query93	2817	541	541	541
query94	854	302	299	299
query95	352	248	250	248
query96	643	278	277	277
query97	2855	2678	2625	2625
query98	211	194	199	194
query99	2102	1316	1364	1316
Total cold run time: 322711 ms
Total hot run time: 197428 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.95 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 351cfc4822399918590c4effc67aad4b31c2eb62, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.67	0.08	0.08
query5	0.44	0.39	0.40
query6	1.17	0.67	0.66
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.56	0.50	0.50
query10	0.55	0.55	0.56
query11	0.16	0.12	0.11
query12	0.16	0.12	0.12
query13	0.62	0.61	0.61
query14	2.84	2.77	2.85
query15	0.92	0.86	0.86
query16	0.38	0.39	0.38
query17	1.05	1.05	1.03
query18	0.18	0.18	0.18
query19	1.98	1.90	2.01
query20	0.01	0.02	0.01
query21	15.37	0.66	0.66
query22	4.13	6.38	2.53
query23	18.26	1.39	1.33
query24	2.19	0.23	0.22
query25	0.16	0.09	0.08
query26	0.27	0.17	0.18
query27	0.08	0.08	0.09
query28	13.26	1.15	1.13
query29	12.59	3.38	3.36
query30	0.24	0.05	0.06
query31	2.86	0.41	0.39
query32	3.25	0.49	0.49
query33	3.02	3.00	3.10
query34	16.84	4.52	4.49
query35	4.60	4.56	4.56
query36	0.67	0.49	0.48
query37	0.19	0.17	0.16
query38	0.16	0.15	0.15
query39	0.05	0.05	0.04
query40	0.16	0.13	0.12
query41	0.10	0.06	0.05
query42	0.06	0.06	0.05
query43	0.05	0.04	0.04
Total cold run time: 111.66 s
Total hot run time: 33.95 s

@w41ter w41ter merged commit c4288a0 into apache:master Nov 28, 2024
24 of 27 checks passed
@w41ter w41ter deleted the support_drop_rollup_binlog branch November 28, 2024 04:01
github-actions bot pushed a commit that referenced this pull request Nov 28, 2024
@w41ter
[feat](binlog) Support drop rollup binlog (#44677)
fc5375b 
ref #43435

Related PR: selectdb/ccr-syncer#269
@github-actions github-actions bot mentioned this pull request Nov 28, 2024
Merged
w41ter added a commit that referenced this pull request Nov 28, 2024
@github-actions @w41ter
branch-3.0: [feat](binlog) Support drop rollup binlog #44677 (#44714)
956c235 
Cherry-picked from #44677

Co-authored-by: walter <[email protected]>
@w41ter w41ter mentioned this pull request Nov 28, 2024
Merged
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Nov 28, 2024
@w41ter
[feat](binlog) Support drop rollup binlog (apache#44677)
e30bf48 
ref apache#43435

Related PR: selectdb/ccr-syncer#269
w41ter added a commit that referenced this pull request Nov 29, 2024
@w41ter
[feat](binlog) Support drop rollup binlog #44677 (#44743)
fc942c4 
cherry pick from #44677
@w41ter w41ter mentioned this pull request Dec 6, 2024
Merged
16 tasks
w41ter added a commit that referenced this pull request Dec 6, 2024
@w41ter
[fix](editlog) Fix replay BatchDropInfo (#45077)
6eca023 
introduced by #44677

The new field `indexNameMap` does not exist in the former version of the
persisted metadata, and should be skipped during replaying.
github-actions bot pushed a commit that referenced this pull request Dec 6, 2024
@w41ter
[fix](editlog) Fix replay BatchDropInfo (#45077)
c661d2f 
introduced by #44677

The new field `indexNameMap` does not exist in the former version of the
persisted metadata, and should be skipped during replaying.
github-actions bot pushed a commit that referenced this pull request Dec 6, 2024
@w41ter
[fix](editlog) Fix replay BatchDropInfo (#45077)
982bff8 
introduced by #44677

The new field `indexNameMap` does not exist in the former version of the
persisted metadata, and should be skipped during replaying.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@gavinchou gavinchou gavinchou approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.4-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@w41ter @doris-robot @dataroaring @gavinchou