Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[only test] #44427

Closed
Closed

Conversation

Mryange
Copy link
Contributor

@Mryange Mryange commented Nov 21, 2024

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@@ -211,8 +208,6 @@ Status ExchangeSinkBuffer::add_block(BroadcastTransmitInfo&& request) {
Status ExchangeSinkBuffer::_send_rpc(InstanceLoId id) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function '_send_rpc' has cognitive complexity of 71 (threshold 50) [readability-function-cognitive-complexity]

Status ExchangeSinkBuffer::_send_rpc(InstanceLoId id) {
                           ^
Additional context

be/src/pipeline/exec/exchange_sink_buffer.cpp:214: +1, including nesting penalty of 0, nesting level increased to 1

    if (_is_finishing) {
    ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:218: +1, including nesting penalty of 0, nesting level increased to 1

    if (_instance_to_receiver_eof[id]) {
    ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:222: +1, including nesting penalty of 0, nesting level increased to 1

    if (!q.empty()) {
    ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:230: +2, including nesting penalty of 1, nesting level increased to 2

        if (request.block && !request.block->column_metas().empty()) {
        ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:230: +1

        if (request.block && !request.block->column_metas().empty()) {
                          ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:233: +2, including nesting penalty of 1, nesting level increased to 2

        if (!request.exec_status.ok()) {
        ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:239: +2, including nesting penalty of 1, nesting level increased to 2

        if (config::execution_ignore_eovercrowded) {
        ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:242: nesting level increased to 2

        send_callback->addFailedHandler([&, weak_task_ctx = weak_task_exec_ctx()](
                                        ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:245: +3, including nesting penalty of 2, nesting level increased to 3

            if (task_lock == nullptr) {
            ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:254: nesting level increased to 2

        send_callback->addSuccessHandler([&, weak_task_ctx = weak_task_exec_ctx()](
                                         ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:259: +3, including nesting penalty of 2, nesting level increased to 3

            if (task_lock == nullptr) {
            ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:270: +3, including nesting penalty of 2, nesting level increased to 3

            if (s.is<ErrorCode::END_OF_FILE>()) {
            ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:272: +1, nesting level increased to 3

            } else if (!s.ok()) {
                   ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:276: +1, nesting level increased to 3

            } else if (eos) {
                   ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:281: +3, including nesting penalty of 2, nesting level increased to 3

                if (!s) {
                ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:292: +2, including nesting penalty of 1, nesting level increased to 2

            if (enable_http_send_block(*brpc_request)) {
            ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:293: +3, including nesting penalty of 2, nesting level increased to 3

                RETURN_IF_ERROR(transmit_block_httpv2(_context->exec_env(),
                ^

be/src/common/status.h:632: expanded from macro 'RETURN_IF_ERROR'

    do {                                \
    ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:293: +4, including nesting penalty of 3, nesting level increased to 4

                RETURN_IF_ERROR(transmit_block_httpv2(_context->exec_env(),
                ^

be/src/common/status.h:634: expanded from macro 'RETURN_IF_ERROR'

        if (UNLIKELY(!_status_.ok())) { \
        ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:296: +1, nesting level increased to 2

            } else {
              ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:301: +2, including nesting penalty of 1, nesting level increased to 2

        if (request.block) {
        ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:308: +2, including nesting penalty of 1, nesting level increased to 2

        if (_total_queue_size <= _queue_capacity) {
        ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:313: +1, nesting level increased to 1

    } else if (!broadcast_q.empty()) {
           ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:321: +2, including nesting penalty of 1, nesting level increased to 2

        if (request.block_holder->get_block() &&
        ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:321: +1

        if (request.block_holder->get_block() &&
                                              ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:327: +2, including nesting penalty of 1, nesting level increased to 2

        if (config::execution_ignore_eovercrowded) {
        ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:330: nesting level increased to 2

        send_callback->addFailedHandler([&, weak_task_ctx = weak_task_exec_ctx()](
                                        ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:333: +3, including nesting penalty of 2, nesting level increased to 3

            if (task_lock == nullptr) {
            ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:342: nesting level increased to 2

        send_callback->addSuccessHandler([&, weak_task_ctx = weak_task_exec_ctx()](
                                         ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:347: +3, including nesting penalty of 2, nesting level increased to 3

            if (task_lock == nullptr) {
            ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:358: +3, including nesting penalty of 2, nesting level increased to 3

            if (s.is<ErrorCode::END_OF_FILE>()) {
            ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:360: +1, nesting level increased to 3

            } else if (!s.ok()) {
                   ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:364: +1, nesting level increased to 3

            } else if (eos) {
                   ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:369: +3, including nesting penalty of 2, nesting level increased to 3

                if (!s) {
                ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:380: +2, including nesting penalty of 1, nesting level increased to 2

            if (enable_http_send_block(*brpc_request)) {
            ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:381: +3, including nesting penalty of 2, nesting level increased to 3

                RETURN_IF_ERROR(transmit_block_httpv2(_context->exec_env(),
                ^

be/src/common/status.h:632: expanded from macro 'RETURN_IF_ERROR'

    do {                                \
    ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:381: +4, including nesting penalty of 3, nesting level increased to 4

                RETURN_IF_ERROR(transmit_block_httpv2(_context->exec_env(),
                ^

be/src/common/status.h:634: expanded from macro 'RETURN_IF_ERROR'

        if (UNLIKELY(!_status_.ok())) { \
        ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:384: +1, nesting level increased to 2

            } else {
              ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:389: +2, including nesting penalty of 1, nesting level increased to 2

        if (request.block_holder->get_block()) {
        ^

be/src/pipeline/exec/exchange_sink_buffer.cpp:393: +1, nesting level increased to 1

    } else {
      ^

@@ -18,6 +18,7 @@
#pragma once

#include <gen_cpp/Types_types.h>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'gen_cpp/Types_types.h' file not found [clang-diagnostic-error]

#include <gen_cpp/Types_types.h>
         ^

@Mryange
Copy link
Contributor Author

Mryange commented Nov 21, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41714 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 239bebbe2b29e9f59b4db528f8cd318762baae70, data reload: false

------ Round 1 ----------------------------------
q1	17609	7499	7284	7284
q2	2050	188	176	176
q3	10527	1176	1186	1176
q4	10232	761	684	684
q5	7715	2925	2861	2861
q6	241	157	148	148
q7	995	635	628	628
q8	9243	1903	1966	1903
q9	7190	6908	6929	6908
q10	7044	2351	2396	2351
q11	479	281	272	272
q12	410	219	218	218
q13	17751	3412	3379	3379
q14	277	252	243	243
q15	959	913	915	913
q16	663	616	594	594
q17	990	575	565	565
q18	7548	6800	6796	6796
q19	1343	1092	963	963
q20	489	191	184	184
q21	4101	3201	3104	3104
q22	429	364	366	364
Total cold run time: 108285 ms
Total hot run time: 41714 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7292	7278	7288	7278
q2	330	233	240	233
q3	3184	3054	3032	3032
q4	2102	1866	1889	1866
q5	5883	5864	5782	5782
q6	223	145	143	143
q7	2275	1851	1840	1840
q8	3398	3586	3610	3586
q9	9523	9550	9616	9550
q10	3686	3628	3654	3628
q11	604	512	511	511
q12	841	651	662	651
q13	10528	3628	3541	3541
q14	345	319	299	299
q15	968	919	922	919
q16	702	659	663	659
q17	1858	1638	1653	1638
q18	8538	7608	7539	7539
q19	1693	1653	1572	1572
q20	2603	2368	2374	2368
q21	5383	5303	5123	5123
q22	929	856	834	834
Total cold run time: 72888 ms
Total hot run time: 62592 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.03% (9903/26041)
Line Coverage: 29.22% (82869/283611)
Region Coverage: 28.34% (42547/150111)
Branch Coverage: 24.90% (21570/86622)
Coverage Report: http://coverage.selectdb-in.cc/coverage/239bebbe2b29e9f59b4db528f8cd318762baae70_239bebbe2b29e9f59b4db528f8cd318762baae70/report/index.html

@Mryange Mryange force-pushed the poc-rpc-single-eof-dev-11-21-22-22 branch from 239bebb to 5cda382 Compare November 22, 2024 09:19
@Mryange
Copy link
Contributor Author

Mryange commented Nov 22, 2024

run buildall

@Mryange Mryange force-pushed the poc-rpc-single-eof-dev-11-21-22-22 branch from 5cda382 to e9c0636 Compare November 22, 2024 09:50
@Mryange
Copy link
Contributor Author

Mryange commented Nov 22, 2024

run buildall

1 similar comment
@Mryange
Copy link
Contributor Author

Mryange commented Nov 22, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.29% (9977/26055)
Line Coverage: 29.42% (83501/283840)
Region Coverage: 28.57% (42963/150353)
Branch Coverage: 25.17% (21832/86752)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e9c063636b9b1d2e57c97f92e6dd12f6eda09920_e9c063636b9b1d2e57c97f92e6dd12f6eda09920/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 39565 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e9c063636b9b1d2e57c97f92e6dd12f6eda09920, data reload: false

------ Round 1 ----------------------------------
q1	17573	7444	7237	7237
q2	2038	170	170	170
q3	10623	1085	1187	1085
q4	10577	775	671	671
q5	7605	2763	2741	2741
q6	237	153	147	147
q7	961	620	605	605
q8	9250	1850	1933	1850
q9	6589	6361	6416	6361
q10	7011	2332	2334	2332
q11	466	265	257	257
q12	423	219	219	219
q13	17777	3020	3055	3020
q14	249	209	217	209
q15	591	541	521	521
q16	667	573	578	573
q17	1019	581	508	508
q18	7252	6554	6712	6554
q19	1326	1050	968	968
q20	500	178	186	178
q21	3941	3173	3037	3037
q22	381	322	322	322
Total cold run time: 107056 ms
Total hot run time: 39565 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7244	7263	7251	7251
q2	326	226	230	226
q3	2894	2804	2943	2804
q4	2028	1801	1805	1801
q5	5708	5667	5656	5656
q6	220	143	142	142
q7	2270	1785	1878	1785
q8	3370	3577	3502	3502
q9	8995	8925	8915	8915
q10	3605	3577	3593	3577
q11	598	511	511	511
q12	872	632	608	608
q13	11216	3317	3217	3217
q14	304	266	294	266
q15	578	531	523	523
q16	692	665	652	652
q17	1848	1641	1627	1627
q18	8338	7879	7776	7776
q19	2869	1608	1515	1515
q20	2114	1880	1837	1837
q21	5480	5407	5554	5407
q22	673	580	572	572
Total cold run time: 72242 ms
Total hot run time: 60170 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197988 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e9c063636b9b1d2e57c97f92e6dd12f6eda09920, data reload: false

query1	1269	925	943	925
query2	6259	2080	2124	2080
query3	10792	4092	4084	4084
query4	67448	28364	23487	23487
query5	4947	441	447	441
query6	408	179	186	179
query7	5563	311	295	295
query8	311	229	216	216
query9	8242	2646	2629	2629
query10	451	247	240	240
query11	17311	15302	16009	15302
query12	163	107	103	103
query13	1486	434	434	434
query14	10342	7730	7662	7662
query15	206	191	205	191
query16	7265	507	471	471
query17	1039	570	568	568
query18	2133	322	291	291
query19	198	157	152	152
query20	131	114	144	114
query21	200	105	105	105
query22	4807	4560	4629	4560
query23	34887	34907	34365	34365
query24	5490	2499	2520	2499
query25	504	394	408	394
query26	639	154	154	154
query27	1821	289	289	289
query28	4586	2473	2462	2462
query29	670	460	462	460
query30	213	152	150	150
query31	1038	841	840	840
query32	65	57	62	57
query33	431	278	274	274
query34	959	540	505	505
query35	846	730	739	730
query36	1113	963	980	963
query37	133	81	77	77
query38	4571	4386	4490	4386
query39	1520	1487	1475	1475
query40	214	101	99	99
query41	46	42	44	42
query42	116	105	100	100
query43	558	530	512	512
query44	1177	863	855	855
query45	190	168	171	168
query46	1166	709	713	709
query47	2077	1920	1967	1920
query48	419	320	329	320
query49	720	392	400	392
query50	863	399	406	399
query51	7421	7289	7142	7142
query52	96	88	88	88
query53	247	180	178	178
query54	497	397	391	391
query55	82	76	75	75
query56	245	239	229	229
query57	1313	1165	1158	1158
query58	212	211	217	211
query59	3172	3173	3159	3159
query60	287	235	243	235
query61	109	107	105	105
query62	801	657	696	657
query63	214	185	183	183
query64	1367	774	724	724
query65	3263	3275	3234	3234
query66	729	327	319	319
query67	16130	15745	15799	15745
query68	4035	620	585	585
query69	426	257	261	257
query70	1245	1110	1121	1110
query71	359	254	250	250
query72	6196	3954	3931	3931
query73	763	357	359	357
query74	10252	8990	9039	8990
query75	3402	2706	2653	2653
query76	1776	1123	1043	1043
query77	466	277	306	277
query78	10952	9601	9485	9485
query79	2928	619	631	619
query80	1205	443	449	443
query81	538	234	236	234
query82	990	122	115	115
query83	270	154	146	146
query84	294	75	73	73
query85	953	297	291	291
query86	469	301	286	286
query87	4667	4666	4566	4566
query88	4297	2281	2221	2221
query89	419	291	290	290
query90	2026	182	184	182
query91	134	103	100	100
query92	69	49	48	48
query93	2879	546	543	543
query94	902	278	267	267
query95	349	241	242	241
query96	622	277	275	275
query97	2873	2674	2665	2665
query98	213	194	197	194
query99	1673	1293	1333	1293
Total cold run time: 322830 ms
Total hot run time: 197988 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.38 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e9c063636b9b1d2e57c97f92e6dd12f6eda09920, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.07	0.06
query4	1.62	0.11	0.10
query5	0.43	0.42	0.41
query6	1.17	0.67	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.60	0.50	0.51
query10	0.56	0.55	0.56
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.73	2.82	2.82
query15	0.93	0.82	0.83
query16	0.38	0.41	0.38
query17	1.07	1.02	1.06
query18	0.22	0.20	0.21
query19	1.95	1.89	1.94
query20	0.01	0.02	0.01
query21	15.40	0.57	0.58
query22	3.02	1.92	1.50
query23	17.10	0.86	0.87
query24	2.99	1.09	0.85
query25	0.17	0.09	0.16
query26	0.59	0.13	0.13
query27	0.05	0.04	0.04
query28	10.75	1.12	1.08
query29	12.55	3.31	3.30
query30	0.25	0.06	0.06
query31	2.86	0.38	0.37
query32	3.27	0.47	0.48
query33	2.99	3.05	3.05
query34	17.04	4.48	4.43
query35	4.53	4.43	4.43
query36	0.69	0.48	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.67 s
Total hot run time: 32.38 s

@Mryange Mryange closed this Dec 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants