[Enhancement] Support Gson subtype rollback #50471
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![starrocks-cr[bot]](https://avatars.githubusercontent.com/in/404334?s=60&v=4){kind=small}
| return (T) object.getValue(); | ||
| } | ||
| return GsonUtils.GSON.fromJson(jsonReader, typeOfT); | ||
| } | ||
|
|
||
| @Override | ||
| public void close() throws IOException { |
There was a problem hiding this comment.
The most risky bug in this code is:
The casting in the readJson method may lead to a ClassCastException.
You can modify the code like this:
@Override
public <T> T readJson(Class<T> classOfT) throws IOException, SRMetaBlockEOFException {
Object object = readJson((Type) classOfT);
if(!Primitives.wrap(classOfT).isInstance(object)) {
throw new JsonSyntaxException("Expected type " + classOfT.getName() + " but got "
+ (object == null ? "null" : object.getClass().getName()));
}
return Primitives.wrap(classOfT).cast(object);
}
| } | ||
| } | ||
| } | ||
|
|
||
| @Override | ||
| public void close() throws IOException, SRMetaBlockException { | ||
| if (header == null) { |
There was a problem hiding this comment.
The most risky bug in this code is:
Potential ClassCastException in the readJson(Class<T> classOfT) method.
You can modify the code like this:
@Override
public <T> T readJson(Class<T> classOfT) throws IOException, SRMetaBlockEOFException {
T object = GsonUtils.GSON.fromJson(genJsonReader(readJsonBytes()), classOfT);
return Primitives.wrap(classOfT).cast(object);
}
|
|
||
| // Initialize the Authorizer class in advance during the loading phase | ||
| // to prevent loading errors and lack of permissions. | ||
| Authorizer.getInstance(); | ||
| } | ||
|
|
||
| public void saveV2(ImageWriter imageWriter) throws IOException { |
There was a problem hiding this comment.
The most risky bug in this code is:
Potentially throwing IOException within the lambda functions of readMapEntry, which may not be adequately handled and could be suppressed.
You can modify the code like this:
public void loadV2(SRMetaBlockReader reader) throws IOException, SRMetaBlockException, SRMetaBlockEOFException {
// 1 json for myself
AuthorizationMgr ret = reader.readJson(AuthorizationMgr.class);
ret.globalStateMgr = globalStateMgr;
ret.provider = Objects.requireNonNullElseGet(provider, DefaultAuthorizationProvider::new);
ret.initBuiltinRolesAndUsers();
try {
LOG.info("loading users");
reader.readMapEntry(UserIdentity.class, UserPrivilegeCollectionV2.class,
(MapEntryConsumer<UserIdentity, UserPrivilegeCollectionV2>) (userIdentity, collection) -> {
if (userIdentity.equals(UserIdentity.ROOT)) {
try {
UserPrivilegeCollectionV2 rootUserPrivCollection =
ret.getUserPrivilegeCollectionUnlocked(UserIdentity.ROOT);
collection.grantRoles(rootUserPrivCollection.getAllRoles());
collection.setDefaultRoleIds(rootUserPrivCollection.getDefaultRoleIds());
collection.typeToPrivilegeEntryList = rootUserPrivCollection.typeToPrivilegeEntryList;
} catch (PrivilegeException e) {
throw new RuntimeException("failed to load users in AuthorizationManager!", e);
}
}
ret.userToPrivilegeCollection.put(userIdentity, collection);
});
LOG.info("loading roles");
reader.readMapEntry(Long.class, RolePrivilegeCollectionV2.class,
(MapEntryConsumer<Long, RolePrivilegeCollectionV2>) (roleId, collection) -> {
// Use hard-code PrivilegeCollection in the memory as the built-in role permission.
// The reason why need to replay from the image here
// is because the associated information of the role-id is stored in the image.
if (PrivilegeBuiltinConstants.IMMUTABLE_BUILT_IN_ROLE_IDS.contains(roleId)) {
RolePrivilegeCollectionV2 builtInRolePrivilegeCollection =
ret.roleIdToPrivilegeCollection.get(roleId);
collection.typeToPrivilegeEntryList = builtInRolePrivilegeCollection.typeToPrivilegeEntryList;
}
ret.roleIdToPrivilegeCollection.put(roleId, collection);
});
LOG.info("loaded {} users, {} roles",
ret.userToPrivilegeCollection.size(), ret.roleIdToPrivilegeCollection.size());
// mark data is loaded
isLoaded = true;
roleNameToId = ret.roleNameToId;
pluginId = ret.pluginId;
pluginVersion = ret.pluginVersion;
userToPrivilegeCollection = ret.userToPrivilegeCollection;
roleIdToPrivilegeCollection = ret.roleIdToPrivilegeCollection;
// Initialize the Authorizer class in advance during the loading phase
// to prevent loading errors and lack of permissions.
Authorizer.getInstance();
} catch (RuntimeException e) {
throw new IOException("Failed to load AuthorizationManager!", e);
}
}
kevincai
reviewed
Sep 1, 2024
fe/fe-core/src/main/java/com/starrocks/persist/gson/RuntimeTypeAdapterFactory.java
Outdated
Show resolved
Hide resolved
gengjun-git
force-pushed
the
gson_sub_type_compatibility
branch
from
ab2763a
to
e1f9c28
Compare
kevincai
reviewed
Sep 4, 2024
fe/fe-core/src/test/java/com/starrocks/persist/metablock/SRMetaBlockV1Test.java
Outdated
Show resolved
Hide resolved
|
@Mergifyio rebase |
✅ Branch has been successfully rebased |
gengjun-git
force-pushed
the
gson_sub_type_compatibility
branch
from
d29c044
to
e064051
Compare
wyb
reviewed
Sep 5, 2024
fe/fe-core/src/main/java/com/starrocks/persist/metablock/SRMetaBlockReader.java
Outdated
Show resolved
Hide resolved
fe/fe-core/src/main/java/com/starrocks/persist/metablock/SRMetaBlockReaderV1.java
Outdated
Show resolved
Hide resolved
kevincai
previously approved these changes
Sep 5, 2024
|
|
yes, ignore the unknown meta data block when rollback |
gengjun-git
force-pushed
the
gson_sub_type_compatibility
branch
from
e064051
to
9e2523b
Compare
|
@Mergifyio rebase |
✅ Branch has been successfully rebased |
gengjun-git
force-pushed
the
gson_sub_type_compatibility
branch
from
9e2523b
to
8a9fe83
Compare
wyb
reviewed
Sep 10, 2024
fe/fe-core/src/main/java/com/starrocks/persist/gson/GsonUtils.java
Outdated
Show resolved
Hide resolved
wyb
previously approved these changes
Sep 10, 2024
nshangyiming
previously approved these changes
Sep 13, 2024
|
@Mergifyio rebase |
✅ Branch has been successfully rebased |
gengjun-git
force-pushed
the
gson_sub_type_compatibility
branch
from
9c6a614
to
e0e5f01
Compare
Signed-off-by: gengjun-git <[email protected]>
Signed-off-by: gengjun-git <[email protected]>
gengjun-git
force-pushed
the
gson_sub_type_compatibility
branch
from
e0e5f01
to
94c747a
Compare
|
nshangyiming
approved these changes
Sep 19, 2024
[Java-Extensions Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
[FE Incremental Coverage Report]✅ pass : 266 / 314 (84.71%) file detail
|
[BE Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
wyb
approved these changes
Sep 19, 2024
kevincai
approved these changes
Sep 19, 2024
Seaven
approved these changes
Sep 19, 2024
|
@Mergifyio backport branch-3.3 |
✅ Backports have been created
|
mergify bot
pushed a commit
that referenced
this pull request
Sep 19, 2024
SubtypeNotFoundException will be thrown when there is a undefined class when deserialize a subclass. The `readCollection` and `readMapEntry` will ignore the SubtypeNotFoundException if ignore_unknown_subtype is true. Signed-off-by: gengjun-git <[email protected]> (cherry picked from commit 790a2b0) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/journal/JournalEntity.java # fe/fe-core/src/main/java/com/starrocks/qe/VariableMgr.java # fe/fe-core/src/test/java/com/starrocks/plugin/PluginMgrTest.java
42 tasks
wanpengfei-git
pushed a commit
that referenced
this pull request
Sep 20, 2024
Signed-off-by: gengjun-git <[email protected]> Co-authored-by: gengjun-git <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why I'm doing:
What I'm doing:
SubtypeNotFoundException will be thrown when there is a undefined class when deserialize a subclass. The
readCollectionandreadMapEntrywill ignore the SubtypeNotFoundException if ignore_unknown_subtype is true.Fixes #issue
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check: