Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug](function)fix json_object function check null nums error as return bool #44321

Merged
merged 1 commit into from
Nov 21, 2024

Conversation

zhangstar333
Copy link
Contributor

@zhangstar333 zhangstar333 commented Nov 20, 2024

What problem does this PR solve?

Related PR: #https://github.com/apache/doris/pull/34591/files

Problem Summary:
before the SQL report error, as it error change the simd::count_zero_num function return value static as bool,
so it run into the if check of (not_null_num < size).
but actual it's maybe return int type.

now:

mysql [(none)]>select json_object ( CONCAT('k',t.number%30926%3000 + 0),CONCAT('k',t.number%30926%3000 + 0,t.number%1000000) ) from numbers("number" = "2") t order by 1;
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| json_object(concat('k', cast((((number % 30926) % 3000) + 0) as VARCHAR(65533))), concat('k', cast((((number % 30926) % 3000) + 0) as VARCHAR(65533)), cast((number % 1000000) as VARCHAR(65533))), '66') |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| {"k0":"k00"}                                                                                                                                                                                              |
| {"k1":"k11"}                                                                                                                                                                                              |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
2 rows in set (0.04 sec)

before:

mysql [(none)]>select json_object
    -> (
    -> CONCAT('k',t.number%30926%3000 + 0),CONCAT('k',t.number%30926%3000 + 0,t.number%1000000)
    -> )
    -> from numbers("number" = "2") t;
ERROR 1105 (HY000): errCode = 2, detailMessage = (10.16.10.8)[INTERNAL_ERROR]function json_object can not input null value , JSON documents may not contain NULL member names.
mysql [(none)]>

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

Copy link
Contributor

PR approved by anyone and no changes requested.

@zhangstar333
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.03% (9902/26035)
Line Coverage: 29.22% (82841/283514)
Region Coverage: 28.35% (42539/150075)
Branch Coverage: 24.91% (21565/86588)
Coverage Report: http://coverage.selectdb-in.cc/coverage/162b7538e914810a77717a4eabc1b1428990b366_162b7538e914810a77717a4eabc1b1428990b366/report/index.html

Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 21, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 40204 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 162b7538e914810a77717a4eabc1b1428990b366, data reload: false

------ Round 1 ----------------------------------
q1	17730	7490	7314	7314
q2	2052	178	183	178
q3	10856	1096	1150	1096
q4	10395	795	720	720
q5	7631	2796	2772	2772
q6	239	149	147	147
q7	999	631	608	608
q8	9264	1909	1971	1909
q9	6619	6486	6449	6449
q10	6972	2316	2338	2316
q11	462	262	258	258
q12	426	222	222	222
q13	17775	3122	3066	3066
q14	248	210	210	210
q15	586	528	535	528
q16	679	585	588	585
q17	1004	590	478	478
q18	7521	6741	6647	6647
q19	1337	1054	1000	1000
q20	495	180	189	180
q21	4064	3213	3334	3213
q22	388	319	308	308
Total cold run time: 107742 ms
Total hot run time: 40204 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7263	7245	7288	7245
q2	336	226	228	226
q3	2923	2976	2951	2951
q4	2063	1855	1834	1834
q5	5651	5715	5750	5715
q6	226	139	147	139
q7	2275	1815	1796	1796
q8	3438	3546	3561	3546
q9	8834	8956	8939	8939
q10	3609	3565	3551	3551
q11	639	508	506	506
q12	786	603	635	603
q13	12410	3188	3190	3188
q14	305	267	272	267
q15	583	527	520	520
q16	680	656	656	656
q17	1850	1639	1640	1639
q18	8338	7836	7531	7531
q19	1717	1664	1548	1548
q20	2126	1895	1883	1883
q21	5644	5520	5469	5469
q22	656	581	572	572
Total cold run time: 72352 ms
Total hot run time: 60324 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195814 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 162b7538e914810a77717a4eabc1b1428990b366, data reload: false

query1	1244	950	914	914
query2	6245	2125	2103	2103
query3	10984	4003	4007	4003
query4	67770	29018	23596	23596
query5	4937	460	473	460
query6	415	181	204	181
query7	5537	305	295	295
query8	318	232	232	232
query9	8812	2705	2684	2684
query10	457	256	258	256
query11	17208	15286	15940	15286
query12	152	102	108	102
query13	1486	425	435	425
query14	10542	7724	6885	6885
query15	215	196	183	183
query16	7103	481	459	459
query17	997	554	542	542
query18	1736	291	282	282
query19	190	142	147	142
query20	114	113	107	107
query21	197	101	99	99
query22	4582	4385	4462	4385
query23	34897	34211	34087	34087
query24	5425	2394	2456	2394
query25	493	409	403	403
query26	648	183	156	156
query27	1657	281	287	281
query28	4226	2439	2409	2409
query29	650	406	409	406
query30	208	144	142	142
query31	1016	813	811	811
query32	66	57	57	57
query33	385	283	285	283
query34	898	526	513	513
query35	833	752	729	729
query36	1090	966	957	957
query37	122	76	74	74
query38	4390	4270	4267	4267
query39	1454	1436	1458	1436
query40	193	99	110	99
query41	45	48	47	47
query42	101	102	100	100
query43	528	490	475	475
query44	1185	824	822	822
query45	178	170	163	163
query46	1121	690	714	690
query47	1937	1857	1891	1857
query48	433	307	318	307
query49	714	394	392	392
query50	829	383	409	383
query51	7244	7153	7193	7153
query52	99	89	88	88
query53	264	176	179	176
query54	544	403	403	403
query55	78	76	85	76
query56	249	237	251	237
query57	1298	1156	1122	1122
query58	224	215	217	215
query59	3183	2989	2925	2925
query60	262	248	259	248
query61	115	111	108	108
query62	778	685	669	669
query63	219	193	186	186
query64	1446	667	632	632
query65	3286	3204	3231	3204
query66	699	308	335	308
query67	15878	15520	15621	15520
query68	2882	572	578	572
query69	412	260	260	260
query70	1209	1142	1158	1142
query71	366	245	257	245
query72	5990	4125	4139	4125
query73	785	374	386	374
query74	9618	8980	9339	8980
query75	3408	2705	2773	2705
query76	1492	1124	971	971
query77	466	276	275	275
query78	10550	9384	9392	9384
query79	2394	609	609	609
query80	1391	432	447	432
query81	528	240	249	240
query82	1334	120	122	120
query83	186	151	153	151
query84	275	71	72	71
query85	990	299	297	297
query86	428	298	282	282
query87	4749	4651	4585	4585
query88	3845	2263	2220	2220
query89	419	292	293	292
query90	1972	186	190	186
query91	133	103	105	103
query92	64	48	49	48
query93	2797	530	539	530
query94	883	296	296	296
query95	358	251	253	251
query96	647	272	284	272
query97	2930	2739	2659	2659
query98	211	199	205	199
query99	1859	1319	1307	1307
Total cold run time: 318400 ms
Total hot run time: 195814 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.82 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 162b7538e914810a77717a4eabc1b1428990b366, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.03	0.04
query3	0.23	0.08	0.06
query4	1.61	0.10	0.11
query5	0.45	0.38	0.41
query6	1.16	0.66	0.66
query7	0.01	0.02	0.02
query8	0.04	0.04	0.03
query9	0.57	0.52	0.49
query10	0.53	0.53	0.56
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.60	0.60
query14	2.86	2.76	2.84
query15	0.90	0.84	0.84
query16	0.38	0.36	0.40
query17	1.06	1.07	0.98
query18	0.20	0.20	0.20
query19	1.91	1.86	1.99
query20	0.01	0.01	0.02
query21	15.39	0.58	0.58
query22	3.07	2.98	2.05
query23	17.01	0.95	0.89
query24	3.54	1.79	0.96
query25	0.17	0.14	0.08
query26	0.54	0.13	0.15
query27	0.05	0.04	0.04
query28	9.90	1.09	1.07
query29	12.56	3.24	3.22
query30	0.24	0.06	0.07
query31	2.86	0.40	0.38
query32	3.27	0.47	0.47
query33	2.97	3.08	3.00
query34	17.02	4.48	4.52
query35	4.59	4.48	4.58
query36	0.68	0.50	0.48
query37	0.09	0.07	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.26 s
Total hot run time: 32.82 s

@zhangstar333 zhangstar333 merged commit 3f86819 into apache:master Nov 21, 2024
30 of 33 checks passed
github-actions bot pushed a commit that referenced this pull request Nov 21, 2024
…rn bool (#44321)

### What problem does this PR solve?

Related PR: #https://github.com/apache/doris/pull/34591/files

Problem Summary:
before the SQL report error, as it error change the simd::count_zero_num
function return value static as bool,
so it run into the if check of (not_null_num < size).
but actual it's maybe return int type.


now:
```
mysql [(none)]>select json_object ( CONCAT('k',t.number%30926%3000 + 0),CONCAT('k',t.number%30926%3000 + 0,t.number%1000000) ) from numbers("number" = "2") t order by 1;
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| json_object(concat('k', cast((((number % 30926) % 3000) + 0) as VARCHAR(65533))), concat('k', cast((((number % 30926) % 3000) + 0) as VARCHAR(65533)), cast((number % 1000000) as VARCHAR(65533))), '66') |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| {"k0":"k00"}                                                                                                                                                                                              |
| {"k1":"k11"}                                                                                                                                                                                              |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
2 rows in set (0.04 sec)
```


before:
```
mysql [(none)]>select json_object
    -> (
    -> CONCAT('k',t.number%30926%3000 + 0),CONCAT('k',t.number%30926%3000 + 0,t.number%1000000)
    -> )
    -> from numbers("number" = "2") t;
ERROR 1105 (HY000): errCode = 2, detailMessage = (10.16.10.8)[INTERNAL_ERROR]function json_object can not input null value , JSON documents may not contain NULL member names.
mysql [(none)]>

```
github-actions bot pushed a commit that referenced this pull request Nov 21, 2024
…rn bool (#44321)

### What problem does this PR solve?

Related PR: #https://github.com/apache/doris/pull/34591/files

Problem Summary:
before the SQL report error, as it error change the simd::count_zero_num
function return value static as bool,
so it run into the if check of (not_null_num < size).
but actual it's maybe return int type.


now:
```
mysql [(none)]>select json_object ( CONCAT('k',t.number%30926%3000 + 0),CONCAT('k',t.number%30926%3000 + 0,t.number%1000000) ) from numbers("number" = "2") t order by 1;
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| json_object(concat('k', cast((((number % 30926) % 3000) + 0) as VARCHAR(65533))), concat('k', cast((((number % 30926) % 3000) + 0) as VARCHAR(65533)), cast((number % 1000000) as VARCHAR(65533))), '66') |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| {"k0":"k00"}                                                                                                                                                                                              |
| {"k1":"k11"}                                                                                                                                                                                              |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
2 rows in set (0.04 sec)
```


before:
```
mysql [(none)]>select json_object
    -> (
    -> CONCAT('k',t.number%30926%3000 + 0),CONCAT('k',t.number%30926%3000 + 0,t.number%1000000)
    -> )
    -> from numbers("number" = "2") t;
ERROR 1105 (HY000): errCode = 2, detailMessage = (10.16.10.8)[INTERNAL_ERROR]function json_object can not input null value , JSON documents may not contain NULL member names.
mysql [(none)]>

```
yiguolei pushed a commit that referenced this pull request Nov 22, 2024
…rror as return bool #44321 (#44381)

Cherry-picked from #44321

Co-authored-by: zhangstar333 <[email protected]>
dataroaring pushed a commit that referenced this pull request Nov 26, 2024
…rror as return bool #44321 (#44380)

Cherry-picked from #44321

Co-authored-by: zhangstar333 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.4-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants