Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feat](nereids) support ShowTabletStorageFormat command in nereids #43295

Merged

Conversation

rijeshkp
Copy link
Contributor

@rijeshkp rijeshkp commented Nov 5, 2024

What problem does this PR solve?

Issue Number: close #42783

Related PR: #xxx

Problem Summary:

Check List (For Committer)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No colde files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.
  • Release note

    None

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@rijeshkp rijeshkp force-pushed the ShowTableStorageFormatToNereids branch from a1fb1da to f3729c0 Compare November 6, 2024 02:50
@rijeshkp rijeshkp marked this pull request as ready for review November 6, 2024 02:55
@rijeshkp
Copy link
Contributor Author

rijeshkp commented Nov 6, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41191 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c36ee4bab9227ccc21763754c22a01c4542ac89f, data reload: false

------ Round 1 ----------------------------------
q1	17619	7893	7337	7337
q2	2030	166	160	160
q3	10686	1071	1145	1071
q4	10548	884	820	820
q5	7723	3095	3026	3026
q6	236	145	145	145
q7	1003	607	596	596
q8	9350	1987	1963	1963
q9	6601	6460	6484	6460
q10	7127	2399	2429	2399
q11	474	263	261	261
q12	416	215	210	210
q13	17799	3048	3029	3029
q14	246	223	213	213
q15	572	528	524	524
q16	657	572	580	572
q17	973	532	497	497
q18	7247	6755	6826	6755
q19	1334	1096	921	921
q20	446	184	181	181
q21	3972	3172	3061	3061
q22	1120	999	990	990
Total cold run time: 108179 ms
Total hot run time: 41191 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7292	7253	7219	7219
q2	318	230	228	228
q3	3020	2949	2974	2949
q4	2076	1957	1816	1816
q5	5711	5761	5799	5761
q6	233	138	143	138
q7	2229	1833	1824	1824
q8	3429	3501	3453	3453
q9	8982	8945	8877	8877
q10	3616	3551	3565	3551
q11	600	518	508	508
q12	820	642	610	610
q13	9972	3211	3191	3191
q14	324	290	297	290
q15	577	530	515	515
q16	677	634	636	634
q17	1856	1618	1598	1598
q18	8250	7881	7697	7697
q19	1715	1655	1610	1610
q20	2143	1863	1893	1863
q21	5651	5485	5553	5485
q22	1187	1103	1068	1068
Total cold run time: 70678 ms
Total hot run time: 60885 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197783 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c36ee4bab9227ccc21763754c22a01c4542ac89f, data reload: false

query1	1183	916	958	916
query2	6233	2080	2028	2028
query3	10892	4161	4157	4157
query4	67427	28534	24122	24122
query5	5053	468	423	423
query6	389	161	169	161
query7	5640	282	282	282
query8	296	207	213	207
query9	9015	2611	2604	2604
query10	473	261	262	261
query11	17523	15373	16006	15373
query12	154	100	104	100
query13	1579	428	442	428
query14	10447	7545	7323	7323
query15	219	198	179	179
query16	7124	527	510	510
query17	1046	562	569	562
query18	1845	301	308	301
query19	195	155	153	153
query20	120	111	108	108
query21	204	102	103	102
query22	4740	4466	4422	4422
query23	34907	34492	34354	34354
query24	6052	2827	2714	2714
query25	519	402	396	396
query26	658	160	158	158
query27	1706	295	284	284
query28	3873	2430	2407	2407
query29	694	430	433	430
query30	234	160	154	154
query31	958	817	802	802
query32	66	53	57	53
query33	411	267	274	267
query34	909	520	504	504
query35	849	744	729	729
query36	1064	959	973	959
query37	116	76	73	73
query38	4313	4365	4282	4282
query39	1446	1429	1417	1417
query40	205	100	101	100
query41	46	45	45	45
query42	108	94	97	94
query43	522	486	486	486
query44	1163	812	817	812
query45	182	167	168	167
query46	1139	707	693	693
query47	1981	1882	1880	1880
query48	422	322	318	318
query49	750	426	391	391
query50	804	393	398	393
query51	7218	7213	7176	7176
query52	99	89	84	84
query53	253	177	183	177
query54	520	442	382	382
query55	83	71	77	71
query56	246	237	235	235
query57	1281	1176	1143	1143
query58	224	219	205	205
query59	3166	3050	3104	3050
query60	271	244	241	241
query61	104	101	100	100
query62	793	685	671	671
query63	208	184	185	184
query64	1364	614	594	594
query65	3353	3224	3270	3224
query66	738	322	299	299
query67	16136	15862	15619	15619
query68	3233	567	589	567
query69	411	255	255	255
query70	1174	1158	1148	1148
query71	359	247	251	247
query72	6100	4069	4049	4049
query73	760	355	352	352
query74	10020	9029	9024	9024
query75	3425	2679	2675	2675
query76	1570	992	1123	992
query77	495	283	282	282
query78	10460	9503	9809	9503
query79	1349	593	595	593
query80	813	424	424	424
query81	512	237	241	237
query82	1316	116	121	116
query83	234	135	143	135
query84	282	74	65	65
query85	875	296	293	293
query86	340	298	307	298
query87	4912	4731	4695	4695
query88	3355	2210	2157	2157
query89	416	293	287	287
query90	1969	187	181	181
query91	133	99	100	99
query92	60	47	49	47
query93	1375	535	530	530
query94	775	295	294	294
query95	343	243	251	243
query96	613	271	278	271
query97	2922	2723	2695	2695
query98	211	197	207	197
query99	1787	1304	1293	1293
Total cold run time: 316550 ms
Total hot run time: 197783 ms

if (result == null) {
throw new AnalysisException("get tablet data from backend: " + be.getId() + "error.");
}
if (isVerbose()) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove isVerbose() function, access verbose member directly

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

@Override
public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
if (!Env.getCurrentEnv().getAccessManager().checkGlobalPriv(ConnectContext.get(), PrivPredicate.ADMIN)
&& !Env.getCurrentEnv().getAccessManager().checkGlobalPriv(ConnectContext.get(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

!Env.getCurrentEnv().getAccessManager().checkGlobalPriv(ConnectContext.get(), PrivPredicate.OPERATOR) is not in ShowTabletStorageFormatStmt's analyze() method, why need it here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the check. Thanks for pointing it out. Since this is admin only, the operator should not be allowed.

/**
* admin show tablet storage format command
*/
public class ShowTabletStorageFormatCommand extends Command implements NoForward {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a new method checkSupportedInCloudMode() is added in Command.java by https://github.com/apache/doris/pull/43271/files, ShowTabletStorageFormatCommand need implement it with same logic like ShowTabletStorageFormatStmt, please update after the pr is merged

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. Update after the PR is merged.

@starocean999 starocean999 changed the title Fixes [Feat](nereids) support ShowTabletStorageFormat command in nereids Nov 6, 2024
*/
public class ShowTabletStorageFormatCommand extends Command implements NoForward {
public static final Logger LOG = LogManager.getLogger(ShowTabletStorageFormatCommand.class);
private boolean verbose;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
private boolean verbose;
private final boolean verbose;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@rijeshkp rijeshkp force-pushed the ShowTableStorageFormatToNereids branch from c36ee4b to 53323be Compare November 6, 2024 14:31
@rijeshkp
Copy link
Contributor Author

rijeshkp commented Nov 6, 2024

run buildall

@rijeshkp
Copy link
Contributor Author

rijeshkp commented Nov 6, 2024

Output Screenshot

image

@doris-robot
Copy link

TPC-H: Total hot run time: 41189 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 53323bec0ca782af43271b824c6c6778e334d40c, data reload: false

------ Round 1 ----------------------------------
q1	17570	8011	7395	7395
q2	2042	171	176	171
q3	10757	1087	1123	1087
q4	10567	810	875	810
q5	7739	3079	3067	3067
q6	231	148	151	148
q7	1005	609	604	604
q8	9371	1977	2008	1977
q9	6685	6453	6438	6438
q10	7093	2441	2412	2412
q11	468	260	259	259
q12	425	215	221	215
q13	17811	2994	3004	2994
q14	257	220	212	212
q15	586	526	518	518
q16	681	591	607	591
q17	969	603	487	487
q18	7300	6595	6647	6595
q19	1324	1036	995	995
q20	462	186	174	174
q21	3930	3197	3061	3061
q22	1122	1007	979	979
Total cold run time: 108395 ms
Total hot run time: 41189 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7290	7228	7305	7228
q2	347	254	244	244
q3	2995	2924	2995	2924
q4	2093	1867	1778	1778
q5	5750	5780	5784	5780
q6	237	147	147	147
q7	2270	1882	1778	1778
q8	3389	3574	3502	3502
q9	8977	8954	8914	8914
q10	3579	3586	3548	3548
q11	605	500	511	500
q12	800	654	648	648
q13	10980	3237	3223	3223
q14	302	274	275	274
q15	616	544	550	544
q16	678	648	638	638
q17	1845	1632	1592	1592
q18	8389	8319	7961	7961
q19	1692	1506	1636	1506
q20	2095	1909	1873	1873
q21	5582	5400	5347	5347
q22	1161	1078	1044	1044
Total cold run time: 71672 ms
Total hot run time: 60993 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196541 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 53323bec0ca782af43271b824c6c6778e334d40c, data reload: false

query1	2256	2057	2076	2057
query2	6215	2046	2041	2041
query3	15513	12013	248	248
query4	33063	23750	23948	23750
query5	3686	456	442	442
query6	271	176	175	175
query7	3983	294	293	293
query8	291	230	233	230
query9	9434	2715	2703	2703
query10	461	249	246	246
query11	18021	15262	15518	15262
query12	155	107	103	103
query13	1579	417	419	417
query14	10265	7395	7266	7266
query15	262	197	196	196
query16	8047	487	518	487
query17	1536	590	618	590
query18	2474	625	663	625
query19	338	186	187	186
query20	128	120	123	120
query21	217	106	102	102
query22	4650	4494	4522	4494
query23	35413	34517	34687	34517
query24	11345	3382	3418	3382
query25	544	400	405	400
query26	733	182	186	182
query27	1923	283	289	283
query28	6598	2436	2441	2436
query29	672	424	428	424
query30	394	311	298	298
query31	1026	783	821	783
query32	99	56	58	56
query33	691	289	286	286
query34	917	516	517	516
query35	882	753	731	731
query36	1106	953	967	953
query37	123	78	73	73
query38	4426	4309	4368	4309
query39	1464	1428	1420	1420
query40	201	105	105	105
query41	48	49	48	48
query42	110	101	99	99
query43	523	501	501	501
query44	1226	813	813	813
query45	185	163	167	163
query46	1154	725	708	708
query47	1923	1827	1841	1827
query48	417	327	320	320
query49	906	409	405	405
query50	823	403	400	400
query51	7225	6999	6989	6989
query52	101	93	88	88
query53	264	186	186	186
query54	1133	434	402	402
query55	83	77	77	77
query56	256	233	241	233
query57	1262	1163	1123	1123
query58	229	206	198	198
query59	3246	3192	3180	3180
query60	273	253	247	247
query61	132	119	112	112
query62	896	664	660	660
query63	212	195	191	191
query64	3791	645	607	607
query65	3350	3241	3259	3241
query66	878	312	321	312
query67	15975	15863	15742	15742
query68	4411	585	581	581
query69	435	248	255	248
query70	1198	1153	1148	1148
query71	350	261	248	248
query72	6029	4054	4047	4047
query73	800	355	353	353
query74	10373	9204	9076	9076
query75	3455	2712	2683	2683
query76	2795	1110	1151	1110
query77	391	277	288	277
query78	11181	9916	9452	9452
query79	2495	621	619	619
query80	936	423	441	423
query81	576	240	241	240
query82	733	118	115	115
query83	245	166	163	163
query84	238	75	70	70
query85	815	306	301	301
query86	429	299	296	296
query87	4948	4695	4711	4695
query88	3902	2152	2142	2142
query89	420	289	294	289
query90	2068	191	188	188
query91	139	115	102	102
query92	57	50	49	49
query93	2260	546	549	546
query94	850	299	303	299
query95	346	254	258	254
query96	635	282	276	276
query97	2879	2704	2741	2704
query98	234	200	191	191
query99	1725	1297	1305	1297
Total cold run time: 306371 ms
Total hot run time: 196541 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.84 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 53323bec0ca782af43271b824c6c6778e334d40c, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.02
query3	0.23	0.06	0.06
query4	1.65	0.10	0.11
query5	0.42	0.40	0.39
query6	1.16	0.65	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.49	0.51
query10	0.54	0.55	0.56
query11	0.14	0.10	0.11
query12	0.14	0.12	0.10
query13	0.61	0.60	0.59
query14	2.71	2.74	2.85
query15	0.87	0.83	0.83
query16	0.39	0.38	0.36
query17	1.06	0.98	0.99
query18	0.20	0.19	0.20
query19	1.83	1.83	1.85
query20	0.02	0.01	0.01
query21	15.36	0.59	0.59
query22	2.88	1.78	1.66
query23	17.00	0.92	0.83
query24	3.04	1.94	1.38
query25	0.29	0.22	0.12
query26	0.42	0.14	0.13
query27	0.04	0.04	0.05
query28	10.07	1.09	1.06
query29	12.81	3.24	3.26
query30	0.25	0.07	0.06
query31	2.87	0.38	0.37
query32	3.27	0.46	0.46
query33	2.94	3.05	3.05
query34	17.17	4.47	4.51
query35	4.50	4.55	4.53
query36	0.64	0.49	0.50
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.73 s
Total hot run time: 32.84 s

@GentleCold
Copy link
Contributor

GentleCold commented Nov 10, 2024

pr's related issue is SHOW TABLET STORAGE FORMAT VERBOSE?, but the commits add the ADMIN SHOW TABLET STORAGE FORMAT VERBOSE? linked here: #42849

@starocean999
Copy link
Contributor

please rebase latest master and resolve the conflicts

@rijeshkp rijeshkp force-pushed the ShowTableStorageFormatToNereids branch from 53323be to b2c1ca3 Compare November 17, 2024 13:33
@rijeshkp rijeshkp force-pushed the ShowTableStorageFormatToNereids branch 4 times, most recently from 1321101 to 730a7ce Compare November 17, 2024 16:15
@rijeshkp rijeshkp force-pushed the ShowTableStorageFormatToNereids branch from 730a7ce to 1a4c1b9 Compare November 17, 2024 16:16
@rijeshkp
Copy link
Contributor Author

run buildall

Copy link
Contributor

github-actions bot commented Dec 6, 2024

PR approved by anyone and no changes requested.

@rijeshkp
Copy link
Contributor Author

rijeshkp commented Dec 9, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 9, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 40271 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a75528a3540f0edc54cc80b38363bb030c929c71, data reload: false

------ Round 1 ----------------------------------
q1	17841	7563	7312	7312
q2	2055	176	167	167
q3	10630	1191	1172	1172
q4	10230	766	710	710
q5	7620	2796	2709	2709
q6	238	151	153	151
q7	1008	624	606	606
q8	9247	1918	2012	1918
q9	6685	6460	6543	6460
q10	7037	2339	2313	2313
q11	468	267	257	257
q12	422	223	216	216
q13	17795	3023	3022	3022
q14	241	202	220	202
q15	577	545	538	538
q16	657	591	583	583
q17	1007	546	555	546
q18	7701	6725	6802	6725
q19	1330	1080	948	948
q20	486	179	183	179
q21	4495	3224	3361	3224
q22	389	313	323	313
Total cold run time: 108159 ms
Total hot run time: 40271 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7317	7430	7312	7312
q2	336	229	234	229
q3	2925	2790	2815	2790
q4	1973	1737	1736	1736
q5	5397	5411	5429	5411
q6	220	145	145	145
q7	2084	1678	1732	1678
q8	3266	3445	3455	3445
q9	8682	8725	8645	8645
q10	3506	3470	3448	3448
q11	593	488	491	488
q12	796	561	577	561
q13	11443	3042	2978	2978
q14	290	266	263	263
q15	573	528	514	514
q16	668	626	634	626
q17	1834	1591	1579	1579
q18	7903	7394	7273	7273
q19	1671	1583	1549	1549
q20	2029	1817	1811	1811
q21	5314	5237	5231	5231
q22	637	592	603	592
Total cold run time: 69457 ms
Total hot run time: 58304 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191137 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a75528a3540f0edc54cc80b38363bb030c929c71, data reload: false

query1	966	395	384	384
query2	6515	2100	1990	1990
query3	6705	210	220	210
query4	34017	23627	23635	23627
query5	4342	475	456	456
query6	268	176	175	175
query7	4646	296	300	296
query8	291	249	219	219
query9	9652	2690	2698	2690
query10	467	250	253	250
query11	18111	15462	15296	15296
query12	152	109	101	101
query13	1650	405	394	394
query14	9672	6859	7030	6859
query15	267	175	184	175
query16	7470	436	480	436
query17	1653	589	582	582
query18	2010	306	300	300
query19	401	160	153	153
query20	122	112	116	112
query21	214	106	105	105
query22	4583	4276	4300	4276
query23	35692	34200	34301	34200
query24	10499	2523	2479	2479
query25	635	376	371	371
query26	1423	150	152	150
query27	2569	281	280	280
query28	7558	2431	2448	2431
query29	894	410	407	407
query30	293	177	156	156
query31	1024	821	829	821
query32	98	56	58	56
query33	768	297	291	291
query34	970	515	509	509
query35	850	760	730	730
query36	1111	954	932	932
query37	127	71	77	71
query38	4336	4191	4299	4191
query39	1473	1442	1426	1426
query40	280	102	100	100
query41	46	43	43	43
query42	112	98	97	97
query43	530	497	492	492
query44	1229	806	798	798
query45	179	170	161	161
query46	1166	735	722	722
query47	1925	1844	1831	1831
query48	408	298	320	298
query49	1109	385	385	385
query50	891	382	406	382
query51	7214	7186	7121	7121
query52	104	93	89	89
query53	258	180	183	180
query54	1277	418	400	400
query55	81	77	80	77
query56	258	237	256	237
query57	1233	1126	1106	1106
query58	231	206	218	206
query59	3211	2883	2931	2883
query60	270	243	247	243
query61	120	106	116	106
query62	839	670	671	670
query63	212	194	189	189
query64	4437	663	643	643
query65	3316	3212	3254	3212
query66	797	312	342	312
query67	16023	15513	15429	15429
query68	5054	552	601	552
query69	416	249	257	249
query70	1168	1134	1119	1119
query71	309	246	249	246
query72	6512	4166	4116	4116
query73	768	360	357	357
query74	10104	9063	9029	9029
query75	3455	2643	2659	2643
query76	2948	1092	996	996
query77	477	286	273	273
query78	10338	9494	9530	9494
query79	2301	596	595	595
query80	1219	421	429	421
query81	554	236	230	230
query82	706	125	111	111
query83	244	192	159	159
query84	235	70	69	69
query85	1649	316	292	292
query86	487	303	296	296
query87	4634	4783	4593	4593
query88	3933	2230	2179	2179
query89	409	299	303	299
query90	1911	189	188	188
query91	133	101	101	101
query92	71	53	53	53
query93	1591	537	541	537
query94	732	289	295	289
query95	356	254	252	252
query96	620	282	278	278
query97	2859	2642	2693	2642
query98	230	187	192	187
query99	1540	1327	1293	1293
Total cold run time: 300213 ms
Total hot run time: 191137 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.99 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a75528a3540f0edc54cc80b38363bb030c929c71, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.08	0.06
query4	1.63	0.10	0.10
query5	0.43	0.44	0.40
query6	1.16	0.65	0.66
query7	0.02	0.01	0.01
query8	0.05	0.04	0.03
query9	0.60	0.51	0.49
query10	0.56	0.53	0.58
query11	0.15	0.10	0.10
query12	0.14	0.11	0.10
query13	0.61	0.60	0.59
query14	2.84	2.71	2.84
query15	0.90	0.83	0.83
query16	0.37	0.38	0.38
query17	1.02	1.01	0.98
query18	0.22	0.22	0.22
query19	1.92	1.81	1.96
query20	0.01	0.01	0.01
query21	15.37	0.62	0.57
query22	2.46	1.88	1.79
query23	17.20	0.84	0.67
query24	2.96	0.73	2.01
query25	0.20	0.04	0.16
query26	0.61	0.14	0.14
query27	0.05	0.05	0.04
query28	10.19	1.10	1.07
query29	12.53	3.25	3.27
query30	0.26	0.07	0.06
query31	2.85	0.39	0.39
query32	3.26	0.46	0.46
query33	3.03	3.00	3.05
query34	17.07	4.50	4.45
query35	4.48	4.48	4.47
query36	0.68	0.48	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.15	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.64 s
Total hot run time: 31.99 s

@rijeshkp rijeshkp force-pushed the ShowTableStorageFormatToNereids branch from a75528a to abae223 Compare December 9, 2024 07:15
@rijeshkp
Copy link
Contributor Author

rijeshkp commented Dec 9, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40380 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit abae22306c7037c5f196c6a661694e912930b3c3, data reload: false

------ Round 1 ----------------------------------
q1	17593	7443	7314	7314
q2	2056	176	167	167
q3	10624	1178	1164	1164
q4	10455	748	725	725
q5	7618	2821	2785	2785
q6	240	149	151	149
q7	1016	632	636	632
q8	9266	1895	1955	1895
q9	6694	6560	6485	6485
q10	7052	2319	2345	2319
q11	483	268	259	259
q12	412	221	224	221
q13	17772	3024	3056	3024
q14	238	212	204	204
q15	558	538	532	532
q16	684	578	575	575
q17	1002	544	571	544
q18	7281	6768	6578	6578
q19	1334	1049	1048	1048
q20	475	182	188	182
q21	4228	3268	3287	3268
q22	376	310	318	310
Total cold run time: 107457 ms
Total hot run time: 40380 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7269	7240	7239	7239
q2	330	239	240	239
q3	2876	2967	3094	2967
q4	2103	1853	1799	1799
q5	5661	5734	5712	5712
q6	220	136	141	136
q7	2172	1810	1885	1810
q8	3421	3854	3511	3511
q9	9045	9142	9075	9075
q10	3666	3572	3548	3548
q11	606	509	516	509
q12	797	641	616	616
q13	11334	3183	3271	3183
q14	296	279	273	273
q15	583	519	511	511
q16	656	636	655	636
q17	1836	1582	1578	1578
q18	7875	7540	7397	7397
q19	1649	1493	1541	1493
q20	2060	1855	1802	1802
q21	5544	5310	5335	5310
q22	638	611	570	570
Total cold run time: 70637 ms
Total hot run time: 59914 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190820 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit abae22306c7037c5f196c6a661694e912930b3c3, data reload: false

query1	957	387	375	375
query2	6512	2115	1973	1973
query3	6704	222	213	213
query4	33534	23734	23622	23622
query5	4378	481	456	456
query6	272	213	194	194
query7	4619	307	311	307
query8	302	246	246	246
query9	9174	2762	2736	2736
query10	462	259	268	259
query11	18087	15316	15187	15187
query12	149	102	103	102
query13	1656	407	423	407
query14	10076	6534	7339	6534
query15	271	184	184	184
query16	8420	484	415	415
query17	1708	578	564	564
query18	2175	302	335	302
query19	358	150	143	143
query20	131	110	108	108
query21	205	103	106	103
query22	4597	4215	4361	4215
query23	35177	34170	34653	34170
query24	11555	2504	2561	2504
query25	669	397	382	382
query26	1867	153	152	152
query27	2918	282	289	282
query28	8208	2463	2469	2463
query29	1043	411	400	400
query30	302	151	156	151
query31	1064	822	810	810
query32	86	56	56	56
query33	770	309	296	296
query34	963	507	539	507
query35	887	753	736	736
query36	1104	975	956	956
query37	292	73	76	73
query38	4376	4249	4253	4249
query39	1466	1418	1423	1418
query40	285	104	98	98
query41	45	44	44	44
query42	106	97	98	97
query43	535	491	488	488
query44	1280	802	807	802
query45	183	168	168	168
query46	1157	707	708	707
query47	1918	1890	1837	1837
query48	409	320	322	320
query49	1300	402	392	392
query50	838	393	384	384
query51	7312	7007	7101	7007
query52	101	94	88	88
query53	258	184	183	183
query54	1185	422	413	413
query55	81	79	78	78
query56	272	252	245	245
query57	1226	1095	1103	1095
query58	232	210	219	210
query59	3203	3233	2852	2852
query60	270	230	241	230
query61	111	111	106	106
query62	886	673	663	663
query63	216	191	184	184
query64	5137	666	639	639
query65	3261	3193	3300	3193
query66	1445	315	314	314
query67	15877	15512	15512	15512
query68	5410	554	563	554
query69	407	248	247	247
query70	1174	1119	1112	1112
query71	326	262	240	240
query72	6396	4122	4042	4042
query73	770	376	373	373
query74	10343	9132	9001	9001
query75	3436	2704	2713	2704
query76	3131	1098	1068	1068
query77	517	280	274	274
query78	10391	9472	9479	9472
query79	1430	600	595	595
query80	1107	447	433	433
query81	526	226	239	226
query82	949	128	123	123
query83	248	148	145	145
query84	231	71	67	67
query85	1310	313	339	313
query86	364	308	298	298
query87	4694	4522	4531	4522
query88	3347	2282	2201	2201
query89	404	291	295	291
query90	2070	194	189	189
query91	141	114	103	103
query92	66	58	52	52
query93	1097	543	543	543
query94	1133	306	278	278
query95	374	258	249	249
query96	619	281	285	281
query97	2927	2723	2690	2690
query98	219	200	202	200
query99	1575	1335	1344	1335
Total cold run time: 303755 ms
Total hot run time: 190820 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.93 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit abae22306c7037c5f196c6a661694e912930b3c3, data reload: false

query1	0.04	0.04	0.03
query2	0.09	0.03	0.04
query3	0.23	0.08	0.07
query4	1.62	0.10	0.10
query5	0.40	0.40	0.41
query6	1.16	0.66	0.64
query7	0.02	0.01	0.02
query8	0.04	0.03	0.04
query9	0.57	0.52	0.50
query10	0.55	0.57	0.56
query11	0.15	0.10	0.11
query12	0.15	0.12	0.12
query13	0.60	0.60	0.60
query14	2.82	2.76	2.76
query15	0.91	0.83	0.81
query16	0.39	0.38	0.38
query17	0.96	1.00	0.97
query18	0.22	0.22	0.21
query19	1.91	1.78	1.86
query20	0.02	0.01	0.02
query21	15.38	0.58	0.58
query22	2.96	2.16	1.53
query23	17.20	0.87	0.72
query24	3.20	0.70	1.77
query25	0.28	0.08	0.18
query26	0.40	0.14	0.14
query27	0.04	0.06	0.04
query28	10.33	1.12	1.08
query29	12.51	3.30	3.24
query30	0.24	0.06	0.06
query31	2.85	0.39	0.39
query32	3.25	0.47	0.46
query33	3.04	3.15	3.04
query34	16.92	4.47	4.51
query35	4.58	4.50	4.57
query36	0.66	0.48	0.49
query37	0.10	0.06	0.06
query38	0.06	0.03	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.13
query41	0.07	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.02	0.02
Total cold run time: 107.19 s
Total hot run time: 31.93 s

@rijeshkp
Copy link
Contributor Author

rijeshkp commented Dec 9, 2024

run doris_compile

@rijeshkp
Copy link
Contributor Author

rijeshkp commented Dec 9, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39667 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 115dd25b173b26a0a6a72e650cb3501cd58ae619, data reload: false

------ Round 1 ----------------------------------
q1	17635	7392	7264	7264
q2	2041	181	171	171
q3	10584	1064	1203	1064
q4	10555	767	732	732
q5	7636	2659	2600	2600
q6	238	144	146	144
q7	977	624	630	624
q8	9251	1831	1928	1831
q9	6618	6504	6417	6417
q10	7032	2258	2305	2258
q11	457	263	262	262
q12	409	220	226	220
q13	17776	3052	3010	3010
q14	245	207	213	207
q15	569	540	523	523
q16	655	592	597	592
q17	966	522	558	522
q18	7264	6634	6754	6634
q19	1331	1056	956	956
q20	470	180	184	180
q21	3990	3138	3217	3138
q22	388	318	324	318
Total cold run time: 107087 ms
Total hot run time: 39667 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7221	7259	7216	7216
q2	329	238	233	233
q3	2917	2805	2958	2805
q4	2040	1818	1770	1770
q5	5709	5653	5611	5611
q6	226	141	141	141
q7	2217	1821	1833	1821
q8	3354	3544	3450	3450
q9	8962	9057	8965	8965
q10	3613	3575	3581	3575
q11	599	507	495	495
q12	830	642	616	616
q13	11882	3310	3263	3263
q14	308	276	270	270
q15	580	533	529	529
q16	695	667	648	648
q17	1858	1618	1584	1584
q18	8302	7776	7647	7647
q19	1680	1484	1553	1484
q20	2168	1892	1874	1874
q21	5573	5384	5434	5384
q22	661	592	556	556
Total cold run time: 71724 ms
Total hot run time: 59937 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197033 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 115dd25b173b26a0a6a72e650cb3501cd58ae619, data reload: false

query1	1234	969	946	946
query2	6258	2081	2019	2019
query3	10957	4386	4438	4386
query4	33192	23546	23668	23546
query5	3597	481	462	462
query6	295	184	178	178
query7	3981	296	313	296
query8	307	232	234	232
query9	9536	2718	2711	2711
query10	445	237	255	237
query11	17896	15276	15343	15276
query12	157	107	104	104
query13	1577	407	396	396
query14	9698	6843	7089	6843
query15	251	192	205	192
query16	8045	469	468	468
query17	1591	590	583	583
query18	2141	310	299	299
query19	340	145	146	145
query20	123	115	119	115
query21	207	104	107	104
query22	4868	4490	4617	4490
query23	35265	34410	34642	34410
query24	10492	2510	2479	2479
query25	608	415	402	402
query26	1241	157	147	147
query27	2517	291	290	290
query28	7361	2485	2471	2471
query29	828	423	397	397
query30	245	154	151	151
query31	1052	888	834	834
query32	99	56	51	51
query33	760	292	298	292
query34	1021	495	515	495
query35	901	783	773	773
query36	1131	968	974	968
query37	159	75	76	75
query38	4560	4363	4432	4363
query39	1523	1464	1500	1464
query40	206	98	98	98
query41	44	41	43	41
query42	114	97	100	97
query43	526	480	492	480
query44	1242	851	849	849
query45	187	171	169	169
query46	1188	739	738	738
query47	2014	1957	1950	1950
query48	428	321	330	321
query49	910	379	416	379
query50	840	392	393	392
query51	7412	7149	7278	7149
query52	107	98	91	91
query53	271	192	187	187
query54	1109	422	411	411
query55	83	76	80	76
query56	254	241	234	234
query57	1277	1178	1171	1171
query58	231	213	211	211
query59	3241	2886	2908	2886
query60	266	240	231	231
query61	105	101	113	101
query62	877	663	683	663
query63	210	187	183	183
query64	4017	647	665	647
query65	3262	3180	3195	3180
query66	844	307	306	306
query67	16389	15737	15628	15628
query68	4809	572	567	567
query69	422	258	259	258
query70	1209	1152	1167	1152
query71	345	262	288	262
query72	6334	4048	4006	4006
query73	760	347	359	347
query74	10475	9217	8989	8989
query75	3442	2673	2687	2673
query76	2715	1035	1037	1035
query77	374	275	258	258
query78	10397	9578	9515	9515
query79	1763	596	606	596
query80	1190	426	429	426
query81	560	235	228	228
query82	811	124	117	117
query83	237	136	144	136
query84	227	65	68	65
query85	1436	287	290	287
query86	459	302	283	283
query87	4772	4623	4573	4573
query88	3826	2189	2163	2163
query89	429	289	281	281
query90	2119	185	182	182
query91	131	98	103	98
query92	59	49	48	48
query93	2285	546	539	539
query94	1036	293	294	293
query95	344	254	253	253
query96	623	273	290	273
query97	2872	2683	2672	2672
query98	221	198	191	191
query99	1551	1308	1325	1308
Total cold run time: 302346 ms
Total hot run time: 197033 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.22 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 115dd25b173b26a0a6a72e650cb3501cd58ae619, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.10	0.11
query5	0.43	0.43	0.41
query6	1.15	0.66	0.66
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.61	0.49	0.48
query10	0.55	0.55	0.57
query11	0.14	0.10	0.10
query12	0.14	0.11	0.12
query13	0.60	0.60	0.61
query14	2.72	2.75	2.85
query15	0.91	0.83	0.83
query16	0.37	0.40	0.37
query17	1.02	1.08	1.06
query18	0.23	0.20	0.20
query19	1.90	1.83	1.96
query20	0.01	0.01	0.01
query21	15.36	0.61	0.57
query22	2.46	1.98	1.62
query23	17.18	0.88	0.80
query24	3.20	0.72	1.54
query25	0.30	0.14	0.23
query26	0.33	0.14	0.14
query27	0.03	0.05	0.05
query28	10.35	1.09	1.07
query29	12.54	3.27	3.26
query30	0.24	0.06	0.06
query31	2.86	0.40	0.37
query32	3.26	0.49	0.46
query33	3.00	3.08	3.03
query34	17.23	4.44	4.50
query35	4.61	4.48	4.52
query36	0.66	0.47	0.48
query37	0.08	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.03	0.02
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.86 s
Total hot run time: 32.22 s

@rijeshkp rijeshkp force-pushed the ShowTableStorageFormatToNereids branch from 115dd25 to 1c51a0f Compare December 9, 2024 13:48
@rijeshkp
Copy link
Contributor Author

rijeshkp commented Dec 9, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40065 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1c51a0f0dece36a95fac6f54626e193d75ce155d, data reload: false

------ Round 1 ----------------------------------
q1	17609	7440	7273	7273
q2	2044	184	187	184
q3	10577	1066	1224	1066
q4	10586	721	794	721
q5	7621	2707	2722	2707
q6	235	147	148	147
q7	983	644	599	599
q8	9231	1814	1944	1814
q9	6676	6510	6473	6473
q10	7056	2313	2338	2313
q11	472	282	271	271
q12	452	227	240	227
q13	17767	3069	3080	3069
q14	239	208	206	206
q15	590	553	518	518
q16	651	594	591	591
q17	968	507	573	507
q18	7372	6716	6674	6674
q19	1345	1002	970	970
q20	477	186	188	186
q21	4049	3327	3227	3227
q22	398	322	324	322
Total cold run time: 107398 ms
Total hot run time: 40065 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7199	7342	7204	7204
q2	333	229	238	229
q3	2915	2827	2968	2827
q4	2072	1801	1839	1801
q5	5665	5717	5654	5654
q6	224	145	141	141
q7	2237	1841	1787	1787
q8	3360	3560	3488	3488
q9	9021	9084	9000	9000
q10	3590	3571	3583	3571
q11	607	502	499	499
q12	828	614	634	614
q13	9935	3290	3234	3234
q14	300	291	271	271
q15	567	522	528	522
q16	709	645	643	643
q17	1848	1628	1615	1615
q18	8238	7671	7679	7671
q19	1700	1567	1627	1567
q20	2147	1863	1866	1863
q21	5617	5418	5511	5418
q22	662	577	585	577
Total cold run time: 69774 ms
Total hot run time: 60196 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198371 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1c51a0f0dece36a95fac6f54626e193d75ce155d, data reload: false

query1	1253	989	929	929
query2	6233	2096	2061	2061
query3	10950	4463	4421	4421
query4	67105	28842	23913	23913
query5	4955	476	463	463
query6	407	195	197	195
query7	5474	308	293	293
query8	309	226	227	226
query9	8638	2721	2732	2721
query10	438	251	250	250
query11	17105	15358	15889	15358
query12	160	108	108	108
query13	1460	449	447	447
query14	10728	8044	7450	7450
query15	224	200	201	200
query16	7090	469	437	437
query17	1244	587	598	587
query18	1764	329	314	314
query19	202	173	163	163
query20	127	118	120	118
query21	208	110	103	103
query22	4590	4475	4379	4379
query23	35490	34188	34515	34188
query24	5401	2641	2561	2561
query25	492	405	389	389
query26	652	158	154	154
query27	1830	300	291	291
query28	4311	2492	2478	2478
query29	668	434	425	425
query30	209	153	161	153
query31	1000	856	831	831
query32	75	56	54	54
query33	420	293	300	293
query34	945	519	528	519
query35	881	793	811	793
query36	1085	962	986	962
query37	122	74	77	74
query38	4656	4599	4417	4417
query39	1499	1494	1506	1494
query40	209	105	107	105
query41	46	42	42	42
query42	116	104	97	97
query43	546	499	507	499
query44	1226	865	875	865
query45	199	177	169	169
query46	1212	740	753	740
query47	2019	1968	1858	1858
query48	431	328	324	324
query49	719	393	417	393
query50	852	393	394	393
query51	7347	7109	7196	7109
query52	102	86	86	86
query53	262	178	180	178
query54	515	406	409	406
query55	81	74	74	74
query56	256	247	241	241
query57	1243	1123	1125	1123
query58	215	229	232	229
query59	3186	3186	3161	3161
query60	270	238	254	238
query61	113	108	107	107
query62	784	669	667	667
query63	215	188	187	187
query64	1369	722	634	634
query65	3303	3218	3203	3203
query66	639	303	305	303
query67	15993	15706	15521	15521
query68	4043	643	562	562
query69	419	261	267	261
query70	1243	1173	1072	1072
query71	345	259	242	242
query72	6486	4196	4021	4021
query73	787	358	364	358
query74	10195	9099	9067	9067
query75	3449	2745	2700	2700
query76	1889	1133	1045	1045
query77	472	275	295	275
query78	10459	9515	9441	9441
query79	2307	624	628	624
query80	1375	448	438	438
query81	510	230	230	230
query82	1323	124	119	119
query83	182	151	147	147
query84	280	74	69	69
query85	1027	298	302	298
query86	415	308	315	308
query87	4752	4575	4711	4575
query88	3719	2206	2199	2199
query89	425	317	297	297
query90	1863	189	189	189
query91	136	101	104	101
query92	65	50	49	49
query93	2841	554	545	545
query94	763	293	283	283
query95	425	249	250	249
query96	629	279	283	279
query97	2831	2692	2677	2677
query98	234	194	197	194
query99	1594	1293	1304	1293
Total cold run time: 321351 ms
Total hot run time: 198371 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.15 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1c51a0f0dece36a95fac6f54626e193d75ce155d, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.62	0.10	0.11
query5	0.42	0.41	0.41
query6	1.15	0.66	0.67
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.57	0.51	0.50
query10	0.55	0.56	0.54
query11	0.14	0.11	0.11
query12	0.14	0.12	0.12
query13	0.62	0.60	0.61
query14	2.81	2.73	2.82
query15	0.91	0.84	0.82
query16	0.37	0.39	0.39
query17	1.05	1.04	0.98
query18	0.22	0.20	0.20
query19	1.99	1.85	2.05
query20	0.01	0.02	0.01
query21	15.38	0.58	0.57
query22	2.55	3.30	2.28
query23	16.92	1.24	0.81
query24	3.49	1.26	0.30
query25	0.27	0.10	0.10
query26	0.43	0.14	0.13
query27	0.06	0.04	0.04
query28	11.01	1.09	1.06
query29	12.56	3.18	3.17
query30	0.25	0.07	0.07
query31	2.87	0.39	0.39
query32	3.25	0.46	0.46
query33	2.97	3.00	3.30
query34	16.93	4.40	4.46
query35	4.50	4.43	4.48
query36	0.67	0.49	0.47
query37	0.09	0.07	0.06
query38	0.05	0.03	0.04
query39	0.04	0.02	0.03
query40	0.16	0.13	0.14
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.02
Total cold run time: 107.58 s
Total hot run time: 32.15 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 10, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@starocean999 starocean999 merged commit ed11252 into apache:master Dec 10, 2024
25 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Enhancement] (nereids)implement showTabletStorageFormatCommand in nereids
5 participants