Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement] (nereids)implement showCreateTableCommand in nereids #43137

Merged
merged 3 commits into from
Nov 20, 2024

Conversation

msridhar78
Copy link
Contributor

@msridhar78 msridhar78 commented Nov 2, 2024

Issue Number: close #42736

implement showCreateTableCommand in nereids

Changed Session Variables:

VarName CurrentValue DefaultValue
character_set_results NULL utf8mb4
sql_mode STRICT_TRANS_TABLES
variable_version 0 300
experimental_enable_nereids_distribute_planner false true

2024-11-18 11:17:06.052 INFO [suite-thread-1] (Suite.groovy:387) - Execute sql: DROP TABLE IF EXISTS tb_show_create_table
2024-11-18 11:17:06.070 INFO [suite-thread-1] (ScriptContext.groovy:120) - Run test_show_create_table_nereids in /root/doris/doris/regression-test/suites/nereids_p0/show/test_show_create_table_nereids.groovy succeed
2024-11-18 11:17:25.342 INFO [main] (RegressionTest.groovy:281) - Start to run single scripts
2024-11-18 11:18:43.673 INFO [main] (RegressionTest.groovy:410) - Success suites:
/root/doris/doris/regression-test/suites/nereids_p0/show/test_show_create_table_nereids.groovy: group=query,arrow_flight_sql,p0, name=test_show_create_table_nereids
2024-11-18 11:18:43.691 INFO [main] (RegressionTest.groovy:492) - All suites success.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@msridhar78
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41366 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 48e559fa81241953fe0f24720712fd5fdac90f09, data reload: false

------ Round 1 ----------------------------------
q1	17585	7438	7295	7295
q2	2047	165	172	165
q3	10595	1087	1209	1087
q4	10541	850	921	850
q5	7747	3114	3079	3079
q6	240	145	146	145
q7	1032	629	621	621
q8	9361	1989	1974	1974
q9	6549	6441	6467	6441
q10	7064	2440	2406	2406
q11	476	250	255	250
q12	408	224	214	214
q13	17768	3016	3013	3013
q14	247	212	210	210
q15	584	521	509	509
q16	665	571	604	571
q17	969	567	475	475
q18	7208	6771	6761	6761
q19	1333	1085	979	979
q20	476	184	180	180
q21	3994	3150	3142	3142
q22	1103	999	1016	999
Total cold run time: 107992 ms
Total hot run time: 41366 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7298	7248	7259	7248
q2	321	230	235	230
q3	3025	2935	2933	2933
q4	2149	1896	1801	1801
q5	5739	5778	5782	5778
q6	221	144	141	141
q7	2294	1843	1785	1785
q8	3390	3515	3494	3494
q9	8985	8941	8933	8933
q10	3610	3559	3592	3559
q11	596	513	522	513
q12	830	636	634	634
q13	11395	3205	3205	3205
q14	302	282	283	282
q15	554	504	513	504
q16	691	658	655	655
q17	1854	1635	1630	1630
q18	8148	7852	7752	7752
q19	1691	1622	1568	1568
q20	2089	1852	1876	1852
q21	5510	5531	5431	5431
q22	1108	1079	1013	1013
Total cold run time: 71800 ms
Total hot run time: 60941 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196696 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 48e559fa81241953fe0f24720712fd5fdac90f09, data reload: false

query1	1210	900	892	892
query2	6230	2190	2114	2114
query3	10802	3915	3936	3915
query4	67669	29013	23677	23677
query5	4976	464	450	450
query6	407	180	191	180
query7	5614	298	290	290
query8	314	230	241	230
query9	9081	2753	2740	2740
query10	432	267	276	267
query11	17400	15405	15925	15405
query12	154	104	104	104
query13	1553	446	426	426
query14	9925	7935	6935	6935
query15	208	179	179	179
query16	7047	455	470	455
query17	987	534	545	534
query18	1791	287	290	287
query19	190	146	145	145
query20	125	112	113	112
query21	193	99	100	99
query22	4524	4410	4443	4410
query23	34603	34054	34284	34054
query24	5998	2821	2752	2752
query25	486	374	376	374
query26	650	154	154	154
query27	1690	287	286	286
query28	4283	2453	2391	2391
query29	668	415	411	411
query30	225	149	152	149
query31	973	798	813	798
query32	64	55	57	55
query33	458	262	263	262
query34	900	506	503	503
query35	881	748	735	735
query36	1073	930	958	930
query37	119	76	70	70
query38	4346	4295	4281	4281
query39	1475	1446	1426	1426
query40	200	99	99	99
query41	46	45	46	45
query42	110	98	98	98
query43	551	498	511	498
query44	1163	820	826	820
query45	187	168	169	168
query46	1123	703	706	703
query47	1946	1813	1886	1813
query48	413	321	339	321
query49	742	403	410	403
query50	813	396	416	396
query51	7266	7090	7208	7090
query52	104	91	90	90
query53	255	186	187	186
query54	543	420	412	412
query55	83	76	76	76
query56	258	254	256	254
query57	1286	1184	1147	1147
query58	223	208	237	208
query59	3404	3130	3152	3130
query60	284	264	256	256
query61	128	127	124	124
query62	795	706	682	682
query63	212	202	189	189
query64	1461	742	697	697
query65	3293	3215	3261	3215
query66	716	314	319	314
query67	15964	15624	15597	15597
query68	3010	588	578	578
query69	411	269	263	263
query70	1202	1152	1179	1152
query71	357	259	257	257
query72	6174	4109	4052	4052
query73	762	348	350	348
query74	10106	8960	9159	8960
query75	3403	2696	2682	2682
query76	1564	1017	1133	1017
query77	470	294	271	271
query78	10445	9477	9468	9468
query79	1373	602	585	585
query80	864	419	444	419
query81	527	241	243	241
query82	1306	120	118	118
query83	226	150	142	142
query84	281	69	66	66
query85	892	302	295	295
query86	348	306	304	304
query87	4913	4655	4756	4655
query88	3730	2225	2173	2173
query89	413	291	297	291
query90	1993	186	189	186
query91	131	99	100	99
query92	60	49	50	49
query93	1939	549	535	535
query94	784	306	257	257
query95	350	248	245	245
query96	678	281	286	281
query97	2871	2717	2701	2701
query98	214	204	196	196
query99	1615	1302	1331	1302
Total cold run time: 316705 ms
Total hot run time: 196696 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.35 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 48e559fa81241953fe0f24720712fd5fdac90f09, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.39	0.43	0.41
query6	1.15	0.65	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.50	0.51
query10	0.55	0.55	0.53
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.59
query14	2.88	2.74	2.78
query15	0.89	0.82	0.84
query16	0.39	0.40	0.37
query17	1.06	1.05	1.01
query18	0.19	0.19	0.20
query19	1.96	1.80	2.00
query20	0.01	0.01	0.01
query21	15.36	0.60	0.58
query22	2.80	1.76	2.06
query23	16.95	0.99	0.80
query24	2.52	1.82	0.94
query25	0.36	0.07	0.05
query26	0.48	0.15	0.14
query27	0.05	0.05	0.04
query28	10.49	1.12	1.06
query29	12.53	3.26	3.23
query30	0.24	0.06	0.06
query31	2.88	0.38	0.38
query32	3.28	0.46	0.46
query33	2.98	3.01	3.01
query34	17.06	4.45	4.47
query35	4.49	4.51	4.49
query36	0.67	0.48	0.48
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.15	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.03	0.03
query43	0.03	0.03	0.03
Total cold run time: 106.57 s
Total hot run time: 32.35 s

}

@Override
public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ShowCreateTableStmt's analyze method needs to be re-implemented in nereids. For example, add a private void validate(ConnectContext ctx) throws UserException method to do the job, and call it at the beginning of run method.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done - please review

@msridhar78 msridhar78 force-pushed the show_create_table branch 4 times, most recently from 84e8c15 to 4f8cbc3 Compare November 10, 2024 17:59
public ShowResultSet doRun(ConnectContext ctx, StmtExecutor executor) throws Exception {
validate(ctx);
// If dbName is not provided, use the default database from the context
String effectiveDbName = (tblNameInfo.getDb() == null || tblNameInfo.getDb().isEmpty())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please refer to handleShowCreateTable() in ShowExecutor.java. The old code just needs some minor changes, it's easier to reivew. thanks.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done - please review


// Fetch the catalog, database, and table metadata
Env env = Env.getCurrentEnv();
DatabaseIf db = env.getCatalogMgr().getCatalogOrAnalysisException(effectiveCatalog)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
DatabaseIf db = env.getCatalogMgr().getCatalogOrAnalysisException(effectiveCatalog)
DatabaseIf db = env.getCatalogMgr().getCatalogOrAnalysisException(tblNameInfo.getCtl())
.getDbOrMetaException(tblNameInfo.getDb());

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done - pls review

public ShowResultSet doRun(ConnectContext ctx, StmtExecutor executor) throws Exception {
validate(ctx);
// If dbName is not provided, use the default database from the context
String effectiveDbName = (tblNameInfo.getDb() == null || tblNameInfo.getDb().isEmpty())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
String effectiveDbName = (tblNameInfo.getDb() == null || tblNameInfo.getDb().isEmpty())

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done - pls review

? ctx.getDatabase() : tblNameInfo.getDb();

// If catalog is not provided, use the default catalog (e.g., internal)
String effectiveCatalog = (tblNameInfo.getCtl() == null || tblNameInfo.getCtl().isEmpty())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
String effectiveCatalog = (tblNameInfo.getCtl() == null || tblNameInfo.getCtl().isEmpty())

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@msridhar78
Copy link
Contributor Author

run buildall

@LiBinfeng-01
Copy link
Collaborator

pls resolve conflicts and regression cases fail

@msridhar78
Copy link
Contributor Author

run buildall

@msridhar78
Copy link
Contributor Author

run buildall

1 similar comment
@msridhar78
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 45237 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2f31750fdef597ac746f72e128d42ead322bb91f, data reload: false

------ Round 1 ----------------------------------
q1	17602	7458	7331	7331
q2	2242	1169	1168	1168
q3	9967	1158	1147	1147
q4	10237	722	764	722
q5	7582	2707	2752	2707
q6	232	150	148	148
q7	984	628	611	611
q8	9374	2358	2365	2358
q9	6532	6432	6453	6432
q10	7004	2337	2307	2307
q11	464	265	258	258
q12	494	217	211	211
q13	17775	3074	3029	3029
q14	245	213	228	213
q15	589	524	524	524
q16	684	582	606	582
q17	986	597	563	563
q18	7159	6715	6662	6662
q19	1314	990	988	988
q20	2907	2719	2734	2719
q21	4001	3393	3217	3217
q22	1401	1361	1340	1340
Total cold run time: 109775 ms
Total hot run time: 45237 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7321	7258	7435	7258
q2	345	245	243	243
q3	3110	3078	3047	3047
q4	2062	1856	1779	1779
q5	5636	5683	5733	5683
q6	221	143	142	142
q7	2249	1865	1778	1778
q8	3345	3533	3509	3509
q9	8947	8891	8906	8891
q10	3636	3578	3611	3578
q11	607	525	501	501
q12	837	665	636	636
q13	13153	3302	3313	3302
q14	327	285	270	270
q15	568	510	525	510
q16	685	645	671	645
q17	1837	1634	1593	1593
q18	8253	7737	7850	7737
q19	1681	1499	1442	1442
q20	2119	1880	1905	1880
q21	5551	5492	5553	5492
q22	640	560	546	546
Total cold run time: 73130 ms
Total hot run time: 60462 ms

@msridhar78
Copy link
Contributor Author

run performance

@msridhar78
Copy link
Contributor Author

run p0

@doris-robot
Copy link

TPC-H: Total hot run time: 44763 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2f31750fdef597ac746f72e128d42ead322bb91f, data reload: false

------ Round 1 ----------------------------------
q1	17575	7443	7262	7262
q2	2234	1170	1178	1170
q3	9991	1162	1151	1151
q4	10215	772	656	656
q5	7574	2714	2781	2714
q6	239	156	146	146
q7	991	633	606	606
q8	9374	2367	2350	2350
q9	6591	6399	6403	6399
q10	7003	2346	2291	2291
q11	466	263	269	263
q12	427	224	217	217
q13	17773	3039	3045	3039
q14	239	208	208	208
q15	587	536	517	517
q16	673	598	596	596
q17	973	563	558	558
q18	7235	6730	6577	6577
q19	1346	1020	991	991
q20	2880	2683	2703	2683
q21	4102	3237	3032	3032
q22	1400	1337	1353	1337
Total cold run time: 109888 ms
Total hot run time: 44763 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7255	7221	7385	7221
q2	330	241	245	241
q3	3115	3012	2999	2999
q4	2048	1785	1833	1785
q5	5620	5683	5763	5683
q6	222	137	138	137
q7	2191	1873	1811	1811
q8	3360	3542	3490	3490
q9	8850	8939	8826	8826
q10	3640	3594	3546	3546
q11	593	521	497	497
q12	804	638	629	629
q13	11723	3287	3241	3241
q14	303	276	269	269
q15	567	523	530	523
q16	665	659	646	646
q17	1845	1635	1634	1634
q18	8311	7681	7723	7681
q19	1719	1535	1585	1535
q20	2110	1847	1897	1847
q21	5546	5478	5375	5375
q22	636	553	560	553
Total cold run time: 71453 ms
Total hot run time: 60169 ms

@msridhar78
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 45402 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 59aea7f574673d9de70a2a6137a574f8c2a24565, data reload: false

------ Round 1 ----------------------------------
q1	17587	7443	7357	7357
q2	2295	1169	1184	1169
q3	9984	1216	1176	1176
q4	10232	789	734	734
q5	7592	2758	2740	2740
q6	242	158	157	157
q7	995	649	629	629
q8	9373	2377	2374	2374
q9	6589	6441	6495	6441
q10	6998	2296	2323	2296
q11	470	271	259	259
q12	428	228	221	221
q13	17795	3048	3058	3048
q14	238	211	216	211
q15	584	526	513	513
q16	655	588	605	588
q17	995	536	598	536
q18	7657	6870	6813	6813
q19	1340	1051	991	991
q20	2948	2693	2667	2667
q21	4121	3146	3161	3146
q22	1414	1336	1352	1336
Total cold run time: 110532 ms
Total hot run time: 45402 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7307	7247	7318	7247
q2	335	234	239	234
q3	3086	2987	2947	2947
q4	2086	1878	1851	1851
q5	5720	5658	5508	5508
q6	225	142	143	142
q7	2181	1736	1733	1733
q8	3302	3474	3402	3402
q9	8595	8594	8644	8594
q10	3510	3481	3432	3432
q11	590	507	499	499
q12	771	593	584	584
q13	10555	3016	3091	3016
q14	293	266	258	258
q15	551	508	512	508
q16	673	647	616	616
q17	1806	1605	1578	1578
q18	7974	7586	7474	7474
q19	1690	1643	1649	1643
q20	2047	1826	1827	1826
q21	5438	5254	5178	5178
q22	629	564	551	551
Total cold run time: 69364 ms
Total hot run time: 58821 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.94 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 59aea7f574673d9de70a2a6137a574f8c2a24565, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.63	0.10	0.10
query5	0.43	0.44	0.43
query6	1.15	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.51	0.50
query10	0.55	0.56	0.57
query11	0.14	0.10	0.10
query12	0.15	0.12	0.11
query13	0.62	0.60	0.60
query14	2.86	2.72	2.73
query15	0.91	0.83	0.83
query16	0.40	0.38	0.37
query17	1.04	1.06	1.06
query18	0.23	0.21	0.22
query19	1.87	2.03	1.77
query20	0.02	0.01	0.01
query21	15.36	0.61	0.59
query22	3.01	2.59	1.92
query23	16.99	0.99	0.76
query24	3.33	1.17	1.98
query25	0.24	0.08	0.16
query26	0.59	0.13	0.14
query27	0.05	0.06	0.04
query28	9.71	1.11	1.07
query29	12.57	3.28	3.24
query30	0.24	0.06	0.06
query31	2.86	0.38	0.38
query32	3.26	0.45	0.46
query33	3.00	3.08	3.16
query34	16.96	4.48	4.45
query35	4.48	4.58	4.54
query36	0.67	0.49	0.48
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.16	0.12	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 106.77 s
Total hot run time: 32.94 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 20, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@starocean999 starocean999 merged commit 50a89d5 into apache:master Nov 20, 2024
26 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Enhancement] (nereids)implement showCreateTableCommand in nereids
4 participants