Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[enhance](regression-test) fix hudi incremental query bug and add hudi p2 cases #44003

Merged
merged 17 commits into from
Nov 22, 2024
Merged

[enhance](regression-test) fix hudi incremental query bug and add hudi p2 cases #44003

merged 17 commits into from
Nov 22, 2024

Conversation

suxiaogang223
Copy link
Contributor

@suxiaogang223 suxiaogang223 commented Nov 15, 2024 •
edited by morningman
Loading

What problem does this PR solve?

Problem Summary:
fix hudi incremental query bug and add hudi p2 cases

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@github-actions github-actions bot added the lfs-detected! Warning Label for use when LFS is detected in the commits of a Pull Request label Nov 17, 2024
@github-actions GitHub Actions
Copy link
Contributor

Possible file(s) that should be tracked in LFS detected: 🚨

The following file(s) exceeds the file size limit: 1048576 bytes, as set in the .yml configuration files:

  • docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hudi_tables/hudi_tables.tar.gz

Consider using git-lfs to manage large files.

@github-actions github-actions bot removed the lfs-detected! Warning Label for use when LFS is detected in the commits of a Pull Request label Nov 17, 2024
@morningman
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 45272 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 362f5fe10ec51a68dc3098c8b859fc8dcaae14ef, data reload: false

------ Round 1 ----------------------------------
q1	17660	7673	7326	7326
q2	2209	1166	1161	1161
q3	10039	1168	1159	1159
q4	10268	759	752	752
q5	7594	2695	2645	2645
q6	237	151	147	147
q7	1001	630	619	619
q8	9381	2359	2367	2359
q9	6560	6343	6421	6343
q10	7042	2351	2342	2342
q11	473	280	271	271
q12	404	219	221	219
q13	17794	3050	3126	3050
q14	248	237	228	228
q15	574	544	527	527
q16	668	595	581	581
q17	992	484	518	484
q18	7339	6916	6841	6841
q19	1396	1005	1074	1005
q20	2977	2750	2686	2686
q21	4089	3192	3281	3192
q22	1388	1347	1335	1335
Total cold run time: 110333 ms
Total hot run time: 45272 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7350	7353	7748	7353
q2	335	234	240	234
q3	3168	3104	2991	2991
q4	2080	1820	1794	1794
q5	5675	5773	5827	5773
q6	222	145	141	141
q7	2265	1896	1747	1747
q8	3320	3575	3564	3564
q9	8947	8876	9035	8876
q10	3674	3652	3561	3561
q11	598	511	519	511
q12	840	615	683	615
q13	9925	3302	3339	3302
q14	318	282	289	282
q15	573	529	501	501
q16	679	660	637	637
q17	1884	1629	1604	1604
q18	8334	7741	7963	7741
q19	1698	1522	1604	1522
q20	2116	1890	1931	1890
q21	5674	5404	5586	5404
q22	621	562	556	556
Total cold run time: 70296 ms
Total hot run time: 60599 ms

@suxiaogang223 suxiaogang223 changed the title [enhance](regression-test) add hudi cases [enhance](regression-test) fix hudi incremental query bug and add hudi p2 cases Nov 21, 2024
@suxiaogang223
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39925 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e546e3d025d0b50da6f65523b35af2aca995aaf9, data reload: false

------ Round 1 ----------------------------------
q1	17607	8161	7243	7243
q2	2050	172	171	171
q3	10637	1085	1159	1085
q4	10564	743	708	708
q5	7591	2695	2657	2657
q6	236	149	149	149
q7	975	629	610	610
q8	9228	1868	1906	1868
q9	6508	6388	6394	6388
q10	6988	2276	2313	2276
q11	455	266	267	266
q12	406	220	219	219
q13	17753	3032	3042	3032
q14	263	220	213	213
q15	587	523	519	519
q16	625	594	608	594
q17	957	572	557	557
q18	7178	6756	6781	6756
q19	1334	1030	992	992
q20	478	184	179	179
q21	3967	3174	3132	3132
q22	371	318	311	311
Total cold run time: 106758 ms
Total hot run time: 39925 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7253	7298	7273	7273
q2	326	225	228	225
q3	2886	2748	2918	2748
q4	2096	1812	1863	1812
q5	5677	5669	5690	5669
q6	226	145	144	144
q7	2243	1826	1828	1826
q8	3421	3555	3478	3478
q9	8894	8990	9029	8990
q10	3635	3538	3531	3531
q11	610	505	510	505
q12	830	607	601	601
q13	12018	3300	3210	3210
q14	303	287	269	269
q15	576	536	524	524
q16	698	669	650	650
q17	1880	1613	1589	1589
q18	8381	7794	7593	7593
q19	1823	1590	1525	1525
q20	2156	1909	1898	1898
q21	5449	5440	5415	5415
q22	640	577	595	577
Total cold run time: 72021 ms
Total hot run time: 60052 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198236 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e546e3d025d0b50da6f65523b35af2aca995aaf9, data reload: false

query1	1245	950	923	923
query2	6254	2125	2085	2085
query3	10920	4079	4024	4024
query4	68140	28149	23668	23668
query5	4864	446	457	446
query6	395	168	170	168
query7	5532	295	291	291
query8	315	215	214	214
query9	8610	2660	2647	2647
query10	431	267	256	256
query11	17187	16728	15884	15884
query12	165	106	110	106
query13	1490	435	432	432
query14	10874	7514	7358	7358
query15	216	190	202	190
query16	7186	473	424	424
query17	1423	598	587	587
query18	1893	314	307	307
query19	207	181	154	154
query20	110	110	113	110
query21	217	101	102	101
query22	4828	4608	4402	4402
query23	34705	34918	34899	34899
query24	5435	2458	2534	2458
query25	496	400	396	396
query26	655	150	152	150
query27	2149	290	291	290
query28	4559	2461	2426	2426
query29	694	420	426	420
query30	218	150	149	149
query31	1057	856	839	839
query32	67	57	59	57
query33	414	288	300	288
query34	971	519	517	517
query35	829	740	739	739
query36	1109	956	960	956
query37	117	77	76	76
query38	4544	4366	4430	4366
query39	1640	1474	1456	1456
query40	212	98	95	95
query41	45	41	51	41
query42	109	95	97	95
query43	531	512	506	506
query44	1241	843	835	835
query45	208	167	164	164
query46	1136	689	707	689
query47	2079	1886	1935	1886
query48	425	338	314	314
query49	731	410	399	399
query50	844	401	403	401
query51	7387	7345	7069	7069
query52	100	85	92	85
query53	248	183	173	173
query54	509	408	385	385
query55	73	76	82	76
query56	243	251	220	220
query57	1298	1171	1155	1155
query58	216	204	224	204
query59	3242	3069	2875	2875
query60	269	246	242	242
query61	106	114	117	114
query62	768	681	674	674
query63	211	186	184	184
query64	1365	666	640	640
query65	3291	3226	3218	3218
query66	702	297	309	297
query67	15996	15836	15835	15835
query68	4111	578	560	560
query69	432	248	304	248
query70	1162	1161	1077	1077
query71	349	258	250	250
query72	6337	4054	4035	4035
query73	754	370	367	367
query74	10344	9019	9158	9019
query75	3397	2678	2674	2674
query76	1898	1088	1036	1036
query77	503	265	267	265
query78	10811	9466	9402	9402
query79	2246	588	599	588
query80	1342	416	426	416
query81	535	231	233	231
query82	1181	121	120	120
query83	271	144	146	144
query84	275	82	68	68
query85	1040	325	297	297
query86	435	300	309	300
query87	4726	4624	4653	4624
query88	3634	2233	2204	2204
query89	424	292	289	289
query90	1959	187	186	186
query91	138	100	104	100
query92	69	47	49	47
query93	2743	557	543	543
query94	853	294	299	294
query95	350	248	256	248
query96	628	276	278	276
query97	2893	2697	2694	2694
query98	221	192	195	192
query99	1749	1312	1322	1312
Total cold run time: 323556 ms
Total hot run time: 198236 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.2 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e546e3d025d0b50da6f65523b35af2aca995aaf9, data reload: false

query1	0.03	0.03	0.02
query2	0.06	0.03	0.03
query3	0.24	0.07	0.07
query4	1.63	0.10	0.10
query5	0.42	0.41	0.42
query6	1.17	0.66	0.66
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.60	0.51	0.51
query10	0.54	0.54	0.55
query11	0.15	0.10	0.10
query12	0.15	0.11	0.11
query13	0.61	0.60	0.60
query14	2.85	2.89	2.83
query15	0.91	0.83	0.83
query16	0.39	0.39	0.38
query17	1.07	1.07	1.05
query18	0.23	0.22	0.21
query19	1.88	1.81	1.97
query20	0.02	0.01	0.01
query21	15.38	0.59	0.56
query22	2.75	2.45	2.03
query23	17.07	1.11	0.79
query24	3.16	1.14	1.22
query25	0.28	0.18	0.08
query26	0.46	0.13	0.14
query27	0.03	0.04	0.04
query28	10.62	1.10	1.07
query29	12.53	3.32	3.32
query30	0.25	0.06	0.06
query31	2.88	0.39	0.39
query32	3.27	0.49	0.46
query33	3.14	2.99	3.13
query34	17.27	4.52	4.52
query35	4.61	4.56	4.52
query36	0.65	0.50	0.50
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.03
query40	0.16	0.13	0.12
query41	0.09	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 107.84 s
Total hot run time: 33.2 s

morningman
morningman approved these changes Nov 22, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 22, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morningman morningman merged commit 9fc6a5e into apache:master Nov 22, 2024
34 of 36 checks passed
github-actions bot pushed a commit that referenced this pull request Nov 22, 2024
@suxiaogang223
[enhance](regression-test) fix hudi incremental query bug and add hud…
2a326cd 
…i p2 cases (#44003)

### What problem does this PR solve?

Problem Summary:
fix hudi incremental query bug and add hudi p2 cases
@github-actions github-actions bot mentioned this pull request Nov 22, 2024
Merged
github-actions bot pushed a commit that referenced this pull request Nov 22, 2024
@suxiaogang223
[enhance](regression-test) fix hudi incremental query bug and add hud…
52617d3 
…i p2 cases (#44003)

### What problem does this PR solve?

Problem Summary:
fix hudi incremental query bug and add hudi p2 cases
@github-actions github-actions bot mentioned this pull request Nov 22, 2024
Merged
yiguolei pushed a commit that referenced this pull request Nov 22, 2024
@github-actions @suxiaogang223
branch-2.1: [enhance](regression-test) fix hudi incremental query bug…
33389ce 
… and add hudi p2 cases #44003 (#44460)

Cherry-picked from #44003

Co-authored-by: Socrates <[email protected]>
@suxiaogang223 suxiaogang223 deleted the hudi_p2 branch November 29, 2024 03:47
dataroaring pushed a commit that referenced this pull request Nov 29, 2024
@github-actions @suxiaogang223
branch-3.0: [enhance](regression-test) fix hudi incremental query bug…
16f36aa 
… and add hudi p2 cases #44003 (#44457)

Cherry-picked from #44003

Co-authored-by: Socrates <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hubgeter hubgeter hubgeter approved these changes

@CalvinKirs CalvinKirs CalvinKirs approved these changes

@morningman morningman morningman approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.4-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@suxiaogang223 @doris-robot @morningman @CalvinKirs @hubgeter @yiguolei @dataroaring