[Feature] Bypass sdk read and write decoupling #45918

Jay-ju · 2024-05-20T07:33:18Z

Why I'm doing:

What I'm doing:

You can learn more about the background from pr's #38466.

This PR is mainly used to solve the problem that when BE directly reads and writes SR file format on S3, the code cannot be encapsulated into SDK form. This PR is mainly to decouple some code implementation and provide some compilation scripts.

build format lib

BUILD_TYPE=Debug ./build.sh --format-lib  -j32

will install directory:

-- Installing: /data00/code/community/starrocks/be/output/format-lib/
-- Installing: /data00/code/community/starrocks/be/output/format-lib/libstarrocks_format.so
Skip Building Java Extensions
***************************************
Successfully build StarRocks  √ Format Lib ; StartTime:2024-05-20 09:52:29, EndTime:2024-05-20 09:58:21, TotalTime:352s

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

be/src/fs/fs_s3.cpp

build.sh

be/src/storage/lake/rowset.cpp

be/src/fs/CMakeLists.txt

be/src/io/s3_input_stream.cpp

tracymacding · 2024-06-27T21:59:41Z

be/src/runtime/descriptors.cpp

+            (*tbl)->_slot_with_column_name_map[tdesc.id] = slot_d;
+        }
+
+        if (!slot_d->col_name().empty()) {


duplicated？

tracymacding · 2024-06-27T22:06:58Z

be/src/runtime/fragment_mgr.cpp

@@ -873,7 +872,7 @@ Status FragmentMgr::exec_external_plan_fragment(const TScanOpenParams& params, c
            LOG(WARNING) << "tuple descriptor is null. id: " << slot_ref.tuple_id;
            return Status::InvalidArgument("tuple descriptor is null");
        }
-        auto* slot_desc = desc_tbl->get_slot_descriptor(slot_ref.slot_id);
+        auto* slot_desc = desc_tbl->get_slot_descriptor_with_column(slot_ref.slot_id);


why change to this new api?

when spark connector using filter push down, be can't recognize slot column name

tracymacding · 2024-06-27T22:10:31Z

be/src/starrocks_format/starrocks_lib.h

+namespace starrocks::lake {
+
+extern lake::TabletManager* _lake_tablet_manager;
+// extern lake::UpdateManager* _lake_update_manager;


remove this?

tracymacding · 2024-06-27T22:11:22Z

be/src/storage/CMakeLists.txt

+
+if (NOT BUILD_FORMAT_LIB)
+    list(APPEND STORAGE_FILES  lake/starlet_location_provider.cpp)
+    message(STATUS "include starlet_location_provider.cpp when build Storage.")


remove this?

tracymacding · 2024-06-27T22:15:00Z

be/src/storage/lake/tablet.cpp

@@ -103,7 +104,13 @@ const std::shared_ptr<const TabletSchema> Tablet::tablet_schema() const {
 }

 StatusOr<std::shared_ptr<const TabletSchema>> Tablet::get_schema() {
-    return _mgr->get_tablet_schema(_id, &_version_hint);
+    if (_tablet_schema) {


when will _table_schema not be nullptr?

here Tablet Construct Function has been modified

tracymacding · 2024-06-27T22:17:46Z

be/src/storage/lake/tablet_manager.cpp

@@ -239,11 +244,17 @@ StatusOr<TabletMetadataPtr> TabletManager::get_tablet_metadata(int64_t tablet_id
 }

 StatusOr<TabletMetadataPtr> TabletManager::get_tablet_metadata(const string& path, bool fill_cache) {
+    std::shared_ptr<FileSystem> fs;
+    return get_tablet_metadata(fs, path, fill_cache);


why pass a invalid fs pointer as a parameter?

be/CMakeLists.txt

build.sh

github-actions · 2024-09-19T05:31:13Z

[FE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

Signed-off-by: jukejian <[email protected]> Signed-off-by: plotor <[email protected]>

Signed-off-by: Binglin Chang <[email protected]>

sonarcloud · 2024-09-20T08:55:43Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

github-actions · 2024-09-20T11:15:13Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-09-20T11:17:01Z

[BE Incremental Coverage Report]

❌ fail : 211 / 320 (65.94%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	be/src/runtime/fragment_mgr.cpp	0	1	00.00%	[878]
🔵	be/src/fs/fs_s3.cpp	0	68	00.00%	[140, 141, 143, 251, 252, 253, 264, 267, 288, 289, 308, 309, 310, 313, 314, 315, 317, 319, 320, 321, 323, 388, 389, 391, 392, 395, 396, 397, 399, 407, 408, 409, 410, 411, 418, 419, 420, 421, 423, 424, 425, 426, 428, 431, 434, 435, 450, 454, 542, 543, 544, 545, 546, 547, 548, 549, 550, 551, 553, 563, 564, 565, 576, 577, 578, 592, 593, 594]
🔵	be/src/io/s3_input_stream.h	0	6	00.00%	[30, 32, 33, 34, 35, 37]
🔵	be/src/runtime/descriptors.cpp	2	6	33.33%	[820, 822, 824, 827]
🔵	be/src/storage/lake/general_tablet_writer.cpp	12	22	54.55%	[76, 100, 101, 271, 272, 273, 275, 277, 298, 299]
🔵	be/src/storage/lake/pk_tablet_writer.cpp	4	6	66.67%	[67, 68]
🔵	be/src/storage/rowset/scalar_column_iterator.cpp	2	3	66.67%	[427]
🔵	be/src/fs/fs.cpp	4	5	80.00%	[93]
🔵	be/src/storage/protobuf_file.cpp	17	20	85.00%	[60, 77, 155]
🔵	be/src/storage/lake/rowset.cpp	33	38	86.84%	[177, 438, 460, 461, 472]
🔵	be/src/storage/lake/lake_delvec_loader.cpp	14	16	87.50%	[28, 31]
🔵	be/src/storage/lake/tablet_manager.cpp	15	17	88.24%	[72, 73]
🔵	be/src/storage/lake/tablet.cpp	12	13	92.31%	[109]
🔵	be/src/io/s3_input_stream.cpp	42	45	93.33%	[70, 75, 111]
🔵	be/src/types/timestamp_value.cpp	7	7	100.00%	[]
🔵	be/src/storage/lake/meta_file.cpp	9	9	100.00%	[]
🔵	be/src/storage/lake/lake_delvec_loader.h	5	5	100.00%	[]
🔵	be/src/storage/rowset/indexed_column_reader.cpp	2	2	100.00%	[]
🔵	be/src/runtime/exec_env.cpp	1	1	100.00%	[]
🔵	be/src/storage/lake/horizontal_compaction_task.cpp	2	2	100.00%	[]
🔵	be/src/storage/lake/vertical_compaction_task.cpp	2	2	100.00%	[]
🔵	be/src/storage/lake/tablet_reader.cpp	2	2	100.00%	[]
🔵	be/src/storage/lake/update_manager.cpp	9	9	100.00%	[]
🔵	be/src/storage/lake/tablet_manager.h	1	1	100.00%	[]
🔵	be/src/storage/rowset/segment.cpp	1	1	100.00%	[]
🔵	be/src/storage/lake/lake_primary_key_compaction_conflict_resolver.h	1	1	100.00%	[]
🔵	be/src/storage/protobuf_file.h	1	1	100.00%	[]
🔵	be/src/storage/lake/tablet.h	4	4	100.00%	[]
🔵	be/src/fs/fs.h	2	2	100.00%	[]
🔵	be/src/storage/lake/lake_primary_key_compaction_conflict_resolver.cpp	3	3	100.00%	[]
🔵	be/src/fs/hdfs/fs_hdfs.cpp	2	2	100.00%	[]

Jay-ju requested review from a team as code owners May 20, 2024 07:33

starrocks-cr bot reviewed May 20, 2024

View reviewed changes

be/src/fs/fs_s3.cpp Show resolved Hide resolved

starrocks-cr bot reviewed May 20, 2024

View reviewed changes

build.sh Show resolved Hide resolved

starrocks-cr bot reviewed May 20, 2024

View reviewed changes

be/src/storage/lake/rowset.cpp Show resolved Hide resolved

mergify bot assigned Jay-ju May 20, 2024

Jay-ju force-pushed the bypass_sdk_read_and_write_decoupling branch 8 times, most recently from e0c2c85 to 61d5aad Compare May 27, 2024 02:09

wyb requested review from decster, kevincai and wyb June 19, 2024 05:10

Jay-ju force-pushed the bypass_sdk_read_and_write_decoupling branch 6 times, most recently from d74163f to 18ed0a7 Compare June 24, 2024 08:07

tracymacding reviewed Jun 27, 2024

View reviewed changes

Jay-ju force-pushed the bypass_sdk_read_and_write_decoupling branch 10 times, most recently from a4e1b17 to 0452052 Compare August 19, 2024 05:51

Jay-ju force-pushed the bypass_sdk_read_and_write_decoupling branch from 0452052 to 2c84e0d Compare August 22, 2024 06:19

kevincai reviewed Aug 22, 2024

View reviewed changes

be/CMakeLists.txt Outdated Show resolved Hide resolved

kevincai reviewed Aug 22, 2024

View reviewed changes

build.sh Outdated Show resolved Hide resolved

Jay-ju force-pushed the bypass_sdk_read_and_write_decoupling branch 6 times, most recently from f1ebd72 to 45c4d64 Compare August 26, 2024 05:58

plotor force-pushed the bypass_sdk_read_and_write_decoupling branch 4 times, most recently from fc747df to 635f43f Compare September 19, 2024 04:45

[Enhancement] Decoupling read and write for BypassLoad SDK.

92db223

Signed-off-by: jukejian <[email protected]> Signed-off-by: plotor <[email protected]>

plotor force-pushed the bypass_sdk_read_and_write_decoupling branch from 635f43f to 92db223 Compare September 19, 2024 08:53

Merge branch 'main' into bypass_sdk_read_and_write_decoupling

00ded6a

Signed-off-by: Binglin Chang <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Bypass sdk read and write decoupling #45918

[Feature] Bypass sdk read and write decoupling #45918

Jay-ju commented May 20, 2024 •

edited by kevincai

Loading

tracymacding Jun 27, 2024

Jay-ju Jun 28, 2024

tracymacding Jun 27, 2024

Jay-ju Jun 28, 2024

tracymacding Jun 27, 2024

tracymacding Jun 27, 2024

tracymacding Jun 27, 2024

Jay-ju Jun 28, 2024 •

edited

Loading

tracymacding Jun 27, 2024

Jay-ju Jun 28, 2024

github-actions bot commented Sep 19, 2024

sonarcloud bot commented Sep 20, 2024

github-actions bot commented Sep 20, 2024

github-actions bot commented Sep 20, 2024

[Feature] Bypass sdk read and write decoupling #45918

Are you sure you want to change the base?

[Feature] Bypass sdk read and write decoupling #45918

Conversation

Jay-ju commented May 20, 2024 • edited by kevincai Loading

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Jay-ju Jun 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Sep 19, 2024

[FE Incremental Coverage Report]

sonarcloud bot commented Sep 20, 2024

Quality Gate passed

github-actions bot commented Sep 20, 2024

[Java-Extensions Incremental Coverage Report]

github-actions bot commented Sep 20, 2024

[BE Incremental Coverage Report]

file detail

Jay-ju commented May 20, 2024 •

edited by kevincai

Loading

Jay-ju Jun 28, 2024 •

edited

Loading