-
Notifications
You must be signed in to change notification settings - Fork 51
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(database-change): supports the execution of sql by downloading files from a separate bucket #3436
base: feat/obcloud_202409
Are you sure you want to change the base?
fix(database-change): supports the execution of sql by downloading files from a separate bucket #3436
Conversation
…le_download' into ziyan_24v3_feat_sql_plan_task_file_download
@@ -29,6 +29,7 @@ | |||
import org.springframework.jdbc.core.ConnectionCallback; | |||
import org.springframework.jdbc.core.StatementCallback; | |||
|
|||
import com.amazonaws.util.CollectionUtils; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use apache commons instead.
|
||
if (CollectionUtils.isNullOrEmpty(parameters.getSqlObjectIds())) { | ||
String sqlContent = parameters.getSqlContent(); | ||
log.info("SQL content={}", sqlContent); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sqlContent may too long for log output
log.info("SQL content={}", sqlContent); | ||
|
||
List<OffsetString> offsetStringList = processor.split(buffer, sqlContent); | ||
result.setTotalStatements(offsetStringList.size()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
result.setTotalStatements may set out of getSplitSql method, responsibility should isolated
log.info("successfully read content from cloud storage, objectId={}", objectId); | ||
String sqlContent = new String(bytes, StandardCharsets.UTF_8); | ||
List<OffsetString> offsetStringList = processor.split(buffer, sqlContent); | ||
splitSqlList.addAll(offsetStringList); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
read all sql statement into memory may cost a lot of memory,
the database change task already contains stream read/split logic, sql plan may follow the same flow.
What type of PR is this?
type-bug
What this PR does / why we need it:
sql plan task supports the execution of sql by downloading files from a separate bucket
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Additional documentation e.g., usage docs, etc.: