[Enhancement] Skip tablet schema in rowset meta during ingestion. #50873

sevev · 2024-09-09T13:31:50Z

Why I'm doing:

Since version 3.2, Rowset has been designed to save its own schema, and the complete tablet schema is stored in the metadata. This can lead to the following issues:

When the import frequency is very high, and a large number of Rowsets are generated, the metadata in RocksDB grows, particularly in non-primary key (non-PK) table scenarios. This is because, in non-PK tables, each update to the tablet metadata rewrites the historical Rowset metadata, leading to a large amount of obsolete data in RocksDB.
When the tablet has a very large number of columns (e.g., 10,000 columns), the time taken to persist the Rowset metadata increases, especially when the imported data volume is small.

These two issues can eventually reduce the efficiency of real-time imports.

What I'm doing:

This PR attempts to solve the issue of reduced import efficiency caused by metadata bloat.

One feasible solution is to store the tablet schema only once for all Rowsets that share the same schema. Instead of saving the complete schema in each Rowset's metadata, a reference or marker would be saved in each Rowset’s metadata to point to the corresponding tablet schema.

However, the issue with this solution is compatibility. In previous versions, each Rowset generated its corresponding schema based on its own metadata. If the system is upgraded and then rollback to an older version, the older version would not be able to locate the schema using a reference or marker. This would lead to the generation of incorrect schemas, as the previous versions expect the full schema to be included in each Rowset's metadata.

So I choose a more conservative solution, and the main changes are as follows:

Skip the schema in Rowset meta during import if the Rowset's schema is identical to the latest tablet schema.
Update the RowsetMeta that does not store schema when updating the tablet schema.

If the BE exits at any given time and restarts, those Rowsets that have not saved their own schema will be initialized using the tablet's current schema. Since the Rowset meta without schemas is updated each time the tablet schema is modified, it ensures that after the BE restarts, every Rowset can find its corresponding schema.

Moreover, this logic is backward compatible with older versions, so even after an upgrade and subsequent downgrade, the BE will still be able to retrieve the correct schema.

Compared to imports, DDL operations can be considered low-frequency tasks. As a result, in most cases, the Rowset meta generated during imports will not carry the schema, which helps alleviate metadata bloat.

However, there can still be some bad cases. For example, in non-PK tables, during the period between an alter operation and the deletion of outdated Rowset meta, if the number of outdated Rowsets is particularly large, the system will still rewrite all outdated Rowsets each time the tablet meta is saved. This can still lead to a decline in import performance.

To solve this issue, we need to resolve the problem of storing multiple copies of the same schema. I think we can first support downgrading and then resolve this issue, allowing for an iteration based on this PR.

Below is a test based on this PR:
a table with 200 columns, one bucket, writing one row of data at a time, with 10 concurrent threads, executed 1,000 times.

Branch	Table type	Total cost time
main-8f128b	Duplicate	1043.77(s)
this pr	Duplicate	178.49(s)
main-8f128b	Primary	188.46(s)
this pr	Primary	186.68(s)

Fixes #issue

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

sevev · 2024-09-24T08:41:10Z

@mergify rebase

Signed-off-by: sevev <[email protected]>

mergify · 2024-09-24T08:41:38Z

rebase

✅ Branch has been successfully rebased

Signed-off-by: zhangqiang <[email protected]>

Signed-off-by: sevev <[email protected]>

be/src/storage/tablet.cpp

Signed-off-by: sevev <[email protected]>

github-actions · 2024-10-12T04:32:16Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-10-12T04:32:21Z

[FE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-10-12T04:42:06Z

[BE Incremental Coverage Report]

✅ pass : 166 / 187 (88.77%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	be/src/storage/data_dir.cpp	0	2	00.00%	[402, 421]
🔵	be/src/storage/base_tablet.h	0	1	00.00%	[112]
🔵	be/src/storage/txn_manager.cpp	23	31	74.19%	[272, 297, 298, 307, 308, 632, 633, 634]
🔵	be/src/storage/tablet.cpp	29	32	90.62%	[632, 633, 634]
🔵	be/src/storage/tablet_updates.cpp	49	54	90.74%	[2636, 5682, 5691, 5702, 5706]
🔵	be/src/storage/tablet_meta.cpp	27	29	93.10%	[329, 337]
🔵	be/src/storage/compaction_task.h	1	1	100.00%	[]
🔵	be/src/storage/tablet_meta_manager.cpp	10	10	100.00%	[]
🔵	be/src/storage/olap_common.h	6	6	100.00%	[]
🔵	be/src/storage/rowset/rowset_meta_manager.cpp	5	5	100.00%	[]
🔵	be/src/storage/tablet.h	2	2	100.00%	[]
🔵	be/src/storage/rowset/rowset_meta.h	14	14	100.00%	[]

github-actions · 2024-10-12T09:09:24Z

@Mergifyio backport branch-3.3

github-actions · 2024-10-12T09:09:26Z

@Mergifyio backport branch-3.2

mergify · 2024-10-12T09:09:32Z

backport branch-3.3

✅ Backports have been created

#51842 [Enhancement] Skip tablet schema in rowset meta during ingestion. (backport #50873) has been created for branch branch-3.3 but encountered conflicts

mergify · 2024-10-12T09:09:34Z

backport branch-3.2

✅ Backports have been created

#51843 [Enhancement] Skip tablet schema in rowset meta during ingestion. (backport #50873) has been created for branch branch-3.2 but encountered conflicts

…0873) ## Why I'm doing: Since version 3.2, Rowset has been designed to save its own schema, and the complete tablet schema is stored in the metadata. This can lead to the following issues: 1. **When the import frequency is very high**, and a large number of Rowsets are generated, the metadata in RocksDB grows, particularly in non-primary key (non-PK) table scenarios. This is because, in non-PK tables, each update to the tablet metadata rewrites the historical Rowset metadata, leading to a large amount of obsolete data in RocksDB. 2. **When the tablet has a very large number of columns (e.g., 10,000 columns)**, the time taken to persist the Rowset metadata increases, especially when the imported data volume is small. These two issues can eventually reduce the efficiency of real-time imports. ## What I'm doing: This PR attempts to solve the issue of reduced import efficiency caused by metadata bloat. One feasible solution is to store the tablet schema only once for all Rowsets that share the same schema. Instead of saving the complete schema in each Rowset's metadata, a reference or marker would be saved in each Rowset’s metadata to point to the corresponding tablet schema. However, the issue with this solution is compatibility. In previous versions, each Rowset generated its corresponding schema based on its own metadata. If the system is upgraded and then rollback to an older version, the older version would not be able to locate the schema using a reference or marker. This would lead to the generation of incorrect schemas, as the previous versions expect the full schema to be included in each Rowset's metadata. So I choose a more conservative solution, and the main changes are as follows: 1. **Skip the schema in Rowset meta during import if the Rowset's schema is identical to the latest tablet schema.** 2. **Update the RowsetMeta that does not store schema when updating the tablet schema.** If the BE exits at any given time and restarts, those Rowsets that have not saved their own schema will be initialized using the tablet's current schema. Since the Rowset meta without schemas is updated each time the tablet schema is modified, it ensures that after the BE restarts, every Rowset can find its corresponding schema. Moreover, this logic is backward compatible with older versions, so even after an upgrade and subsequent downgrade, the BE will still be able to retrieve the correct schema. Compared to imports, DDL operations can be considered low-frequency tasks. As a result, in most cases, the Rowset meta generated during imports will not carry the schema, which helps alleviate metadata bloat. However, there can still be some bad cases. For example, in non-PK tables, during the period between an alter operation and the deletion of outdated Rowset meta, if the number of outdated Rowsets is particularly large, the system will still rewrite all outdated Rowsets each time the tablet meta is saved. This can still lead to a decline in import performance. To solve this issue, we need to resolve the problem of storing multiple copies of the same schema. I think we can first support downgrading and then resolve this issue, allowing for an iteration based on this PR. Below is a test based on this PR: a table with 200 columns, one bucket, writing one row of data at a time, with 10 concurrent threads, executed 1,000 times. | Branch | Table type |Total cost time | |----------|----------|----------| | main-8f128b | Duplicate | 1043.77(s) | | this pr | Duplicate | 178.49(s) | | main-8f128b | Primary | 188.46(s) | | this pr | Primary | 186.68(s) | Signed-off-by: sevev <[email protected]> Signed-off-by: zhangqiang <[email protected]> (cherry picked from commit 3005729) # Conflicts: # be/src/common/config.h # be/src/storage/tablet.h # be/src/storage/tablet_meta.h

…0873) ## Why I'm doing: Since version 3.2, Rowset has been designed to save its own schema, and the complete tablet schema is stored in the metadata. This can lead to the following issues: 1. **When the import frequency is very high**, and a large number of Rowsets are generated, the metadata in RocksDB grows, particularly in non-primary key (non-PK) table scenarios. This is because, in non-PK tables, each update to the tablet metadata rewrites the historical Rowset metadata, leading to a large amount of obsolete data in RocksDB. 2. **When the tablet has a very large number of columns (e.g., 10,000 columns)**, the time taken to persist the Rowset metadata increases, especially when the imported data volume is small. These two issues can eventually reduce the efficiency of real-time imports. ## What I'm doing: This PR attempts to solve the issue of reduced import efficiency caused by metadata bloat. One feasible solution is to store the tablet schema only once for all Rowsets that share the same schema. Instead of saving the complete schema in each Rowset's metadata, a reference or marker would be saved in each Rowset’s metadata to point to the corresponding tablet schema. However, the issue with this solution is compatibility. In previous versions, each Rowset generated its corresponding schema based on its own metadata. If the system is upgraded and then rollback to an older version, the older version would not be able to locate the schema using a reference or marker. This would lead to the generation of incorrect schemas, as the previous versions expect the full schema to be included in each Rowset's metadata. So I choose a more conservative solution, and the main changes are as follows: 1. **Skip the schema in Rowset meta during import if the Rowset's schema is identical to the latest tablet schema.** 2. **Update the RowsetMeta that does not store schema when updating the tablet schema.** If the BE exits at any given time and restarts, those Rowsets that have not saved their own schema will be initialized using the tablet's current schema. Since the Rowset meta without schemas is updated each time the tablet schema is modified, it ensures that after the BE restarts, every Rowset can find its corresponding schema. Moreover, this logic is backward compatible with older versions, so even after an upgrade and subsequent downgrade, the BE will still be able to retrieve the correct schema. Compared to imports, DDL operations can be considered low-frequency tasks. As a result, in most cases, the Rowset meta generated during imports will not carry the schema, which helps alleviate metadata bloat. However, there can still be some bad cases. For example, in non-PK tables, during the period between an alter operation and the deletion of outdated Rowset meta, if the number of outdated Rowsets is particularly large, the system will still rewrite all outdated Rowsets each time the tablet meta is saved. This can still lead to a decline in import performance. To solve this issue, we need to resolve the problem of storing multiple copies of the same schema. I think we can first support downgrading and then resolve this issue, allowing for an iteration based on this PR. Below is a test based on this PR: a table with 200 columns, one bucket, writing one row of data at a time, with 10 concurrent threads, executed 1,000 times. | Branch | Table type |Total cost time | |----------|----------|----------| | main-8f128b | Duplicate | 1043.77(s) | | this pr | Duplicate | 178.49(s) | | main-8f128b | Primary | 188.46(s) | | this pr | Primary | 186.68(s) | Signed-off-by: sevev <[email protected]> Signed-off-by: zhangqiang <[email protected]> (cherry picked from commit 3005729) # Conflicts: # be/src/common/config.h # be/src/storage/compaction_task.h # be/src/storage/tablet.cpp # be/src/storage/tablet.h # be/src/storage/tablet_meta.cpp # be/src/storage/tablet_meta.h # be/src/storage/txn_manager.cpp

yingtingdong · 2024-10-12T12:33:33Z

ignore backport check:3.2.12

…ckport #50873) (#51842) Signed-off-by: sevev <[email protected]> Co-authored-by: zhangqiang <[email protected]>

…ckport #50873) (#51843) Signed-off-by: sevev <[email protected]> Co-authored-by: zhangqiang <[email protected]>

…arRocks#50873) ## Why I'm doing: Since version 3.2, Rowset has been designed to save its own schema, and the complete tablet schema is stored in the metadata. This can lead to the following issues: 1. **When the import frequency is very high**, and a large number of Rowsets are generated, the metadata in RocksDB grows, particularly in non-primary key (non-PK) table scenarios. This is because, in non-PK tables, each update to the tablet metadata rewrites the historical Rowset metadata, leading to a large amount of obsolete data in RocksDB. 2. **When the tablet has a very large number of columns (e.g., 10,000 columns)**, the time taken to persist the Rowset metadata increases, especially when the imported data volume is small. These two issues can eventually reduce the efficiency of real-time imports. ## What I'm doing: This PR attempts to solve the issue of reduced import efficiency caused by metadata bloat. One feasible solution is to store the tablet schema only once for all Rowsets that share the same schema. Instead of saving the complete schema in each Rowset's metadata, a reference or marker would be saved in each Rowset’s metadata to point to the corresponding tablet schema. However, the issue with this solution is compatibility. In previous versions, each Rowset generated its corresponding schema based on its own metadata. If the system is upgraded and then rollback to an older version, the older version would not be able to locate the schema using a reference or marker. This would lead to the generation of incorrect schemas, as the previous versions expect the full schema to be included in each Rowset's metadata. So I choose a more conservative solution, and the main changes are as follows: 1. **Skip the schema in Rowset meta during import if the Rowset's schema is identical to the latest tablet schema.** 2. **Update the RowsetMeta that does not store schema when updating the tablet schema.** If the BE exits at any given time and restarts, those Rowsets that have not saved their own schema will be initialized using the tablet's current schema. Since the Rowset meta without schemas is updated each time the tablet schema is modified, it ensures that after the BE restarts, every Rowset can find its corresponding schema. Moreover, this logic is backward compatible with older versions, so even after an upgrade and subsequent downgrade, the BE will still be able to retrieve the correct schema. Compared to imports, DDL operations can be considered low-frequency tasks. As a result, in most cases, the Rowset meta generated during imports will not carry the schema, which helps alleviate metadata bloat. However, there can still be some bad cases. For example, in non-PK tables, during the period between an alter operation and the deletion of outdated Rowset meta, if the number of outdated Rowsets is particularly large, the system will still rewrite all outdated Rowsets each time the tablet meta is saved. This can still lead to a decline in import performance. To solve this issue, we need to resolve the problem of storing multiple copies of the same schema. I think we can first support downgrading and then resolve this issue, allowing for an iteration based on this PR. Below is a test based on this PR: a table with 200 columns, one bucket, writing one row of data at a time, with 10 concurrent threads, executed 1,000 times. | Branch | Table type |Total cost time | |----------|----------|----------| | main-8f128b | Duplicate | 1043.77(s) | | this pr | Duplicate | 178.49(s) | | main-8f128b | Primary | 188.46(s) | | this pr | Primary | 186.68(s) | Signed-off-by: sevev <[email protected]> Signed-off-by: zhangqiang <[email protected]>

…arRocks#50873) ## Why I'm doing: Since version 3.2, Rowset has been designed to save its own schema, and the complete tablet schema is stored in the metadata. This can lead to the following issues: 1. **When the import frequency is very high**, and a large number of Rowsets are generated, the metadata in RocksDB grows, particularly in non-primary key (non-PK) table scenarios. This is because, in non-PK tables, each update to the tablet metadata rewrites the historical Rowset metadata, leading to a large amount of obsolete data in RocksDB. 2. **When the tablet has a very large number of columns (e.g., 10,000 columns)**, the time taken to persist the Rowset metadata increases, especially when the imported data volume is small. These two issues can eventually reduce the efficiency of real-time imports. ## What I'm doing: This PR attempts to solve the issue of reduced import efficiency caused by metadata bloat. One feasible solution is to store the tablet schema only once for all Rowsets that share the same schema. Instead of saving the complete schema in each Rowset's metadata, a reference or marker would be saved in each Rowset’s metadata to point to the corresponding tablet schema. However, the issue with this solution is compatibility. In previous versions, each Rowset generated its corresponding schema based on its own metadata. If the system is upgraded and then rollback to an older version, the older version would not be able to locate the schema using a reference or marker. This would lead to the generation of incorrect schemas, as the previous versions expect the full schema to be included in each Rowset's metadata. So I choose a more conservative solution, and the main changes are as follows: 1. **Skip the schema in Rowset meta during import if the Rowset's schema is identical to the latest tablet schema.** 2. **Update the RowsetMeta that does not store schema when updating the tablet schema.** If the BE exits at any given time and restarts, those Rowsets that have not saved their own schema will be initialized using the tablet's current schema. Since the Rowset meta without schemas is updated each time the tablet schema is modified, it ensures that after the BE restarts, every Rowset can find its corresponding schema. Moreover, this logic is backward compatible with older versions, so even after an upgrade and subsequent downgrade, the BE will still be able to retrieve the correct schema. Compared to imports, DDL operations can be considered low-frequency tasks. As a result, in most cases, the Rowset meta generated during imports will not carry the schema, which helps alleviate metadata bloat. However, there can still be some bad cases. For example, in non-PK tables, during the period between an alter operation and the deletion of outdated Rowset meta, if the number of outdated Rowsets is particularly large, the system will still rewrite all outdated Rowsets each time the tablet meta is saved. This can still lead to a decline in import performance. To solve this issue, we need to resolve the problem of storing multiple copies of the same schema. I think we can first support downgrading and then resolve this issue, allowing for an iteration based on this PR. Below is a test based on this PR: a table with 200 columns, one bucket, writing one row of data at a time, with 10 concurrent threads, executed 1,000 times. | Branch | Table type |Total cost time | |----------|----------|----------| | main-8f128b | Duplicate | 1043.77(s) | | this pr | Duplicate | 178.49(s) | | main-8f128b | Primary | 188.46(s) | | this pr | Primary | 186.68(s) | Signed-off-by: sevev <[email protected]> Signed-off-by: zhangqiang <[email protected]> Signed-off-by: zhiminr.ren <[email protected]>

sevev requested review from a team as code owners September 9, 2024 13:31

github-actions bot added the title needs [type] label Sep 9, 2024

mergify bot assigned sevev Sep 9, 2024

sevev changed the title ~~[WIP][Enhancement] Skip tablet schema in rowset meta during ingestion.~~ [Enhancement] Skip tablet schema in rowset meta during ingestion. Sep 19, 2024

github-actions bot added 3.3 3.2 and removed title needs [type] labels Sep 19, 2024

sevev force-pushed the load_schema_remove branch from 5e36a40 to 000a707 Compare September 24, 2024 02:13

decster previously approved these changes Sep 24, 2024

View reviewed changes

sevev added 7 commits September 24, 2024 08:41

backup commit

4122822

Signed-off-by: sevev <[email protected]>

skip tablet schema in rowset meta during ingestion

a85aff6

Signed-off-by: sevev <[email protected]>

skip tablet schema during compaction

40ba852

Signed-off-by: sevev <[email protected]>

fix bug

dfff4a6

Signed-off-by: sevev <[email protected]>

update

c891f9b

Signed-off-by: sevev <[email protected]>

add ut

7f29104

Signed-off-by: sevev <[email protected]>

update ut

16f0b0d

Signed-off-by: sevev <[email protected]>

sevev force-pushed the load_schema_remove branch from 000a707 to 16f0b0d Compare September 24, 2024 08:41

Merge branch 'main' into load_schema_remove

e31a038

Signed-off-by: zhangqiang <[email protected]>

sevev dismissed decster’s stale review via e31a038 September 24, 2024 10:55

remove expired annoation

9166538

Signed-off-by: sevev <[email protected]>

wyb reviewed Sep 27, 2024

View reviewed changes

be/src/storage/tablet.cpp Show resolved Hide resolved

luohaha self-requested a review October 3, 2024 02:32

StarRocks deleted a comment from mergify bot Oct 3, 2024

wyb previously approved these changes Oct 3, 2024

View reviewed changes

luohaha previously approved these changes Oct 3, 2024

View reviewed changes

sevev dismissed luohaha’s stale review via 4e5244c October 8, 2024 08:28

sevev force-pushed the load_schema_remove branch from 1620da2 to 8f45131 Compare October 12, 2024 01:39

luohaha previously approved these changes Oct 12, 2024

View reviewed changes

sevev dismissed luohaha’s stale review via 644a450 October 12, 2024 02:32

sevev force-pushed the load_schema_remove branch from 8f45131 to 644a450 Compare October 12, 2024 02:32

update

4af09ac

Signed-off-by: sevev <[email protected]>

sevev force-pushed the load_schema_remove branch from 644a450 to 4af09ac Compare October 12, 2024 02:35

wyb approved these changes Oct 12, 2024

View reviewed changes

decster approved these changes Oct 12, 2024

View reviewed changes

kevincai approved these changes Oct 12, 2024

View reviewed changes

sevev merged commit 3005729 into StarRocks:main Oct 12, 2024
50 checks passed

github-actions bot removed the 3.3 label Oct 12, 2024

github-actions bot removed the 3.2 label Oct 12, 2024

This was referenced Oct 12, 2024

[Enhancement] Skip tablet schema in rowset meta during ingestion. (backport #50873) #51842

Merged

[Enhancement] Skip tablet schema in rowset meta during ingestion. (backport #50873) #51843

Merged

wanpengfei-git pushed a commit that referenced this pull request Oct 16, 2024

[Enhancement] Skip tablet schema in rowset meta during ingestion. (ba…

53d68da

…ckport #50873) (#51842) Signed-off-by: sevev <[email protected]> Co-authored-by: zhangqiang <[email protected]>

github-actions bot added the 3.3-merged label Oct 16, 2024

wanpengfei-git pushed a commit that referenced this pull request Oct 24, 2024

[Enhancement] Skip tablet schema in rowset meta during ingestion. (ba…

a6264c9

…ckport #50873) (#51843) Signed-off-by: sevev <[email protected]> Co-authored-by: zhangqiang <[email protected]>

github-actions bot added the 3.2-merged label Oct 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] Skip tablet schema in rowset meta during ingestion. #50873

[Enhancement] Skip tablet schema in rowset meta during ingestion. #50873

sevev commented Sep 9, 2024 •

edited

Loading

sevev commented Sep 24, 2024

mergify bot commented Sep 24, 2024

github-actions bot commented Oct 12, 2024

github-actions bot commented Oct 12, 2024

github-actions bot commented Oct 12, 2024

github-actions bot commented Oct 12, 2024

github-actions bot commented Oct 12, 2024

mergify bot commented Oct 12, 2024 •

edited

Loading

mergify bot commented Oct 12, 2024 •

edited

Loading

yingtingdong commented Oct 12, 2024

[Enhancement] Skip tablet schema in rowset meta during ingestion. #50873

[Enhancement] Skip tablet schema in rowset meta during ingestion. #50873

Conversation

sevev commented Sep 9, 2024 • edited Loading

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

sevev commented Sep 24, 2024

mergify bot commented Sep 24, 2024

✅ Branch has been successfully rebased

github-actions bot commented Oct 12, 2024

[Java-Extensions Incremental Coverage Report]

github-actions bot commented Oct 12, 2024

[FE Incremental Coverage Report]

github-actions bot commented Oct 12, 2024

[BE Incremental Coverage Report]

file detail

github-actions bot commented Oct 12, 2024

github-actions bot commented Oct 12, 2024

mergify bot commented Oct 12, 2024 • edited Loading

✅ Backports have been created

mergify bot commented Oct 12, 2024 • edited Loading

✅ Backports have been created

yingtingdong commented Oct 12, 2024

sevev commented Sep 9, 2024 •

edited

Loading

mergify bot commented Oct 12, 2024 •

edited

Loading

mergify bot commented Oct 12, 2024 •

edited

Loading