Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Add function get_query_dump #48105

Merged
merged 2 commits into from
Jul 11, 2024

Conversation

ZiheLiu
Copy link
Contributor

@ZiheLiu ZiheLiu commented Jul 10, 2024

Why I'm doing:

Add function get_query_dump to get query dump more easily.

select get_query_dump(c_int_2_seq) from t1;
(1064, 'Getting analyzing error. Detail message: Meta function get_query_dump does not support non-constant arguments.')

select get_query_dump("select count(1) from t1") ;

+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| {"statement":"select * from t1","table_meta":{"d2.t1":"CREATE TABLE `t1` (\n  `k1` bigint(20) NULL COMMENT \"\",\n  `c_int_1_seq` bigint(20) NULL COMMENT \"\",\n  `c_int_2_seq` varchar(65533) NULL COMMENT \"\"\n) ENGINE=OLAP \nDUPLICATE KEY(`k1`)\nDISTRIBUTED BY HASH(`k1`) BUCKETS 32 \nPROPERTIES (\n\"compression\" = \"LZ4\",\n\"fast_schema_evolution\" = \"true\",\n\"replicated_storage\" = \"true\",\n\"replication_num\" = \"1\"\n);"},"table_row_count":{"d2.t1":{"t1":1280000}},"column_statisti... |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

with 
    query_dump as (select get_query_dump(concat('select *', ' from t1')) as x),
    w1 as (
        select 'statement' as k, get_json_string(x, '$.statement') as not_empty from query_dump
        UNION ALL
        select 'table_meta' as k, get_json_string(x, '$.table_meta')  as not_empty from query_dump
        UNION ALL
        select 'table_row_count' as k, get_json_string(x, '$.table_row_count')  as not_empty from query_dump
        UNION ALL
        select 'column_statistics' as k, get_json_string(x, '$.column_statistics')  as not_empty from query_dump
    ) 
select * from w1 order by k;

+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| k                 | not_empty                                                                                                                                                                                                                                                                                                                                                                                                 |
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| column_statistics | {"d2.t1": {"c_int_1_seq": "[1.0, 1280000.0, 0.0, 8.0, 1273013.0] ESTIMATE", "c_int_2_seq": "[-Infinity, Infinity, 0.0, 11.13195, 1268234.0] ESTIMATE", "k1": "[1.0, 1280000.0, 0.0, 8.0, 1273013.0] ESTIMATE"}}                                                                                                                                                                                           |
| statement         | select * from t1                                                                                                                                                                                                                                                                                                                                                                                          |
| table_meta        | {"d2.t1": "CREATE TABLE `t1` (\n  `k1` bigint(20) NULL COMMENT \"\",\n  `c_int_1_seq` bigint(20) NULL COMMENT \"\",\n  `c_int_2_seq` varchar(65533) NULL COMMENT \"\"\n) ENGINE=OLAP \nDUPLICATE KEY(`k1`)\nDISTRIBUTED BY HASH(`k1`) BUCKETS 32 \nPROPERTIES (\n\"compression\" = \"LZ4\",\n\"fast_schema_evolution\" = \"true\",\n\"replicated_storage\" = \"true\",\n\"replication_num\" = \"1\"\n);"} |
| table_row_count   | {"d2.t1": {"t1": 1280000}}                                                                                                                                                                                                                                                                                                                                                                                |
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

What I'm doing:

  1. Extract QueryDumper used by the HTTP API QueryDumperAction and the function get_query_dump.
  2. Implement constant meta function get_query_dump only in FE.
  3. Report error Meta function xxx does not support non-constant arguments when meta functions accept non-constant arguments.
  4. The user privilege of get_query_dump is the same as the HTTP API /api/query_dump.

Close #48039

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function
  • This is a backport pr

Bugfix cherry-pick branch check:

  • I have checked the version labels which the pr will be auto-backported to the target branch
    • 3.3
    • 3.2
    • 3.1
    • 3.0
    • 2.5

s
Signed-off-by: zihe.liu <[email protected]>
@ZiheLiu ZiheLiu requested review from a team as code owners July 10, 2024 06:32
s
Signed-off-by: zihe.liu <[email protected]>
Copy link

sonarcloud bot commented Jul 10, 2024

Copy link

[FE Incremental Coverage Report]

pass : 70 / 73 (95.89%)

file detail

path covered_line new_line coverage not_covered_line_detail
🔵 com/starrocks/sql/optimizer/function/MetaFunctions.java 5 6 83.33% [357]
🔵 com/starrocks/sql/optimizer/dump/QueryDumper.java 42 44 95.45% [32, 92]
🔵 com/starrocks/http/rest/QueryDumpAction.java 8 8 100.00% []
🔵 com/starrocks/sql/optimizer/rewrite/ScalarOperatorEvaluator.java 3 3 100.00% []
🔵 com/starrocks/sql/optimizer/rewrite/scalar/FoldConstantsRule.java 7 7 100.00% []
🔵 com/starrocks/catalog/Function.java 5 5 100.00% []

Copy link

[BE Incremental Coverage Report]

pass : 0 / 0 (0%)

@kangkaisen kangkaisen enabled auto-merge (squash) July 11, 2024 02:22
@kangkaisen kangkaisen merged commit d42320f into StarRocks:main Jul 11, 2024
71 checks passed
@wanpengfei-git wanpengfei-git mentioned this pull request Jul 11, 2024
2 tasks
@ZiheLiu
Copy link
Contributor Author

ZiheLiu commented Jul 11, 2024

@mergify backport branch-2.5 branch-3.0 branch-3.1 branch-3.2 branch-3.3

Copy link
Contributor

mergify bot commented Jul 11, 2024

backport branch-2.5 branch-3.0 branch-3.1 branch-3.2 branch-3.3

✅ Backports have been created

mergify bot pushed a commit that referenced this pull request Jul 11, 2024
Signed-off-by: zihe.liu <[email protected]>
(cherry picked from commit d42320f)

# Conflicts:
#	fe/fe-core/src/main/java/com/starrocks/catalog/Function.java
#	fe/fe-core/src/main/java/com/starrocks/http/rest/QueryDumpAction.java
#	fe/fe-core/src/main/java/com/starrocks/sql/optimizer/function/MetaFunctions.java
#	fe/fe-core/src/main/java/com/starrocks/sql/optimizer/rewrite/scalar/FoldConstantsRule.java
#	fe/fe-core/src/test/java/com/starrocks/http/StarRocksHttpTestCase.java
#	fe/fe-core/src/test/java/com/starrocks/sql/plan/ConstantExpressionTest.java
mergify bot pushed a commit that referenced this pull request Jul 11, 2024
Signed-off-by: zihe.liu <[email protected]>
(cherry picked from commit d42320f)

# Conflicts:
#	fe/fe-core/src/main/java/com/starrocks/catalog/Function.java
#	fe/fe-core/src/main/java/com/starrocks/http/rest/QueryDumpAction.java
#	fe/fe-core/src/main/java/com/starrocks/sql/optimizer/function/MetaFunctions.java
#	fe/fe-core/src/test/java/com/starrocks/http/StarRocksHttpTestCase.java
mergify bot pushed a commit that referenced this pull request Jul 11, 2024
Signed-off-by: zihe.liu <[email protected]>
(cherry picked from commit d42320f)

# Conflicts:
#	fe/fe-core/src/main/java/com/starrocks/catalog/Function.java
#	fe/fe-core/src/main/java/com/starrocks/http/rest/QueryDumpAction.java
#	fe/fe-core/src/main/java/com/starrocks/sql/optimizer/function/MetaFunctions.java
#	fe/fe-core/src/test/java/com/starrocks/http/StarRocksHttpTestCase.java
mergify bot pushed a commit that referenced this pull request Jul 11, 2024
Signed-off-by: zihe.liu <[email protected]>
(cherry picked from commit d42320f)

# Conflicts:
#	fe/fe-core/src/main/java/com/starrocks/catalog/Function.java
#	fe/fe-core/src/main/java/com/starrocks/http/rest/QueryDumpAction.java
#	fe/fe-core/src/main/java/com/starrocks/sql/optimizer/function/MetaFunctions.java
#	fe/fe-core/src/test/java/com/starrocks/http/StarRocksHttpTestCase.java
mergify bot pushed a commit that referenced this pull request Jul 11, 2024
Signed-off-by: zihe.liu <[email protected]>
(cherry picked from commit d42320f)
@ZiheLiu
Copy link
Contributor Author

ZiheLiu commented Jul 11, 2024

ignore backport check: 3.1.14

@ZiheLiu
Copy link
Contributor Author

ZiheLiu commented Jul 11, 2024

@mergify ignore backport check: 3.1.14

Copy link
Contributor

mergify bot commented Jul 11, 2024

ignore backport check: 3.1.14

❌ Sorry but I didn't understand the command. Please consult the commands documentation 📚.

@ZiheLiu
Copy link
Contributor Author

ZiheLiu commented Jul 11, 2024

ignore backport check: 3.3.1

wanpengfei-git pushed a commit that referenced this pull request Jul 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Function] get_query_dump
7 participants