[fix](delete) Fix potential delete job stuck util timeout if exception happend in FE DeleteJob execution (#41672)

[TangSiyang2001]

Proposed changes

pick: #41672

Fail task should also count down for the count down latch to prevent job stuck.

[doris-robot]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

[TangSiyang2001]

run buildall

[doris-robot]

TPC-H: Total hot run time: 48870 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0bf9647047ae8bb8962ae8470c964492a04aab3f, data reload: false

------ Round 1 ----------------------------------
q1	18130	4352	4373	4352
q2	2080	153	148	148
q3	10367	1911	1932	1911
q4	10317	1229	1300	1229
q5	8471	3896	3906	3896
q6	231	121	122	121
q7	2056	1623	1576	1576
q8	9273	2726	2703	2703
q9	10270	9754	9762	9754
q10	8645	3507	3472	3472
q11	424	243	253	243
q12	459	295	300	295
q13	18367	3961	4020	3961
q14	348	329	322	322
q15	510	475	456	456
q16	551	461	450	450
q17	1128	977	959	959
q18	7206	6787	6873	6787
q19	1688	1589	1543	1543
q20	539	318	286	286
q21	4368	4071	4010	4010
q22	505	396	406	396
Total cold run time: 115933 ms
Total hot run time: 48870 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4333	4288	4259	4259
q2	327	223	219	219
q3	4191	4166	4136	4136
q4	2759	2735	2742	2735
q5	7168	7089	7106	7089
q6	238	118	116	116
q7	3228	2892	2838	2838
q8	4335	4440	4461	4440
q9	13689	13581	13565	13565
q10	4328	4292	4232	4232
q11	739	709	686	686
q12	1026	864	865	864
q13	7041	3719	3750	3719
q14	448	421	419	419
q15	488	459	466	459
q16	606	583	599	583
q17	3806	3907	3825	3825
q18	8818	8710	8839	8710
q19	1721	1690	1639	1639
q20	2396	2114	2101	2101
q21	8492	8439	8428	8428
q22	1060	927	913	913
Total cold run time: 81237 ms
Total hot run time: 75975 ms

[doris-robot]

TPC-DS: Total hot run time: 211695 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0bf9647047ae8bb8962ae8470c964492a04aab3f, data reload: false

query1	929	392	409	392
query2	6552	2253	2261	2253
query3	6922	207	203	203
query4	23121	21256	21151	21151
query5	19993	6517	6573	6517
query6	282	221	239	221
query7	4327	303	312	303
query8	249	234	239	234
query9	3113	2704	2613	2613
query10	458	323	336	323
query11	16192	15111	15434	15111
query12	131	75	73	73
query13	1034	446	464	446
query14	18428	13427	13513	13427
query15	378	235	255	235
query16	6707	295	267	267
query17	1935	976	958	958
query18	1467	354	352	352
query19	513	170	179	170
query20	120	115	120	115
query21	1131	125	145	125
query22	5759	5102	4831	4831
query23	34469	33486	33429	33429
query24	6904	6369	6325	6325
query25	504	412	424	412
query26	1087	159	160	159
query27	2261	291	288	288
query28	6086	2285	2234	2234
query29	2843	2675	2690	2675
query30	243	167	168	167
query31	948	735	751	735
query32	73	59	56	56
query33	452	257	258	257
query34	855	485	472	472
query35	1157	901	924	901
query36	1497	1169	1297	1169
query37	94	58	58	58
query38	3056	2886	2850	2850
query39	1373	1318	1330	1318
query40	310	96	94	94
query41	39	38	37	37
query42	85	90	82	82
query43	560	534	634	534
query44	1179	733	727	727
query45	242	227	229	227
query46	1225	939	973	939
query47	1926	1915	1872	1872
query48	495	426	412	412
query49	636	366	381	366
query50	855	624	576	576
query51	4743	4667	4682	4667
query52	88	75	81	75
query53	238	185	187	185
query54	2675	2444	2452	2444
query55	93	86	89	86
query56	233	194	200	194
query57	1261	1155	1095	1095
query58	218	212	205	205
query59	3518	3471	3204	3204
query60	215	210	215	210
query61	97	94	104	94
query62	815	451	471	451
query63	194	172	173	172
query64	3355	1576	1467	1467
query65	3590	3528	3540	3528
query66	812	414	436	414
query67	15396	16313	15022	15022
query68	9496	653	675	653
query69	497	272	259	259
query70	1764	1691	1324	1324
query71	419	312	309	309
query72	6813	4778	4802	4778
query73	763	332	320	320
query74	6333	5783	5795	5783
query75	5016	3685	3690	3685
query76	5117	1118	1147	1118
query77	845	251	248	248
query78	12509	11483	11717	11483
query79	8092	625	638	625
query80	1797	385	377	377
query81	489	242	240	240
query82	1659	92	96	92
query83	169	136	134	134
query84	258	70	69	69
query85	910	323	314	314
query86	335	292	296	292
query87	3203	3004	3034	3004
query88	4786	2312	2314	2312
query89	512	295	306	295
query90	2092	207	213	207
query91	155	127	131	127
query92	55	49	51	49
query93	6895	540	541	540
query94	779	211	212	211
query95	1985	1967	1942	1942
query96	644	328	321	321
query97	6381	6335	6321	6321
query98	228	205	204	204
query99	3028	855	887	855
Total cold run time: 322760 ms
Total hot run time: 211695 ms

[doris-robot]

ClickBench: Total hot run time: 31.12 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0bf9647047ae8bb8962ae8470c964492a04aab3f, data reload: false

query1	0.02	0.03	0.02
query2	0.07	0.03	0.03
query3	0.25	0.04	0.04
query4	1.80	0.07	0.07
query5	0.54	0.53	0.52
query6	1.28	0.61	0.63
query7	0.01	0.01	0.01
query8	0.04	0.03	0.02
query9	0.51	0.49	0.47
query10	0.55	0.52	0.52
query11	0.12	0.09	0.09
query12	0.12	0.09	0.08
query13	0.62	0.60	0.62
query14	0.80	0.77	0.81
query15	0.77	0.76	0.75
query16	0.37	0.39	0.36
query17	1.03	1.02	1.03
query18	0.23	0.26	0.24
query19	1.91	1.77	1.84
query20	0.02	0.01	0.01
query21	15.48	0.56	0.56
query22	2.07	2.41	1.86
query23	17.33	0.98	0.97
query24	4.56	1.64	1.14
query25	0.39	0.09	0.04
query26	0.53	0.16	0.15
query27	0.04	0.04	0.03
query28	8.19	0.74	0.71
query29	12.76	2.31	2.31
query30	0.62	0.50	0.53
query31	2.83	0.39	0.37
query32	3.38	0.50	0.49
query33	3.11	3.08	3.09
query34	15.25	4.82	4.80
query35	4.86	4.85	4.84
query36	1.04	1.02	1.02
query37	0.06	0.04	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.15	0.15	0.14
query41	0.06	0.02	0.01
query42	0.02	0.01	0.01
query43	0.03	0.02	0.02
Total cold run time: 103.87 s
Total hot run time: 31.12 s

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 0bf9647047ae8bb8962ae8470c964492a04aab3f with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          57 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s