Added a feature to support migrating archive files from Aliyun OSS #6
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
daixba
reviewed
Apr 24, 2022
| @@ -277,12 +277,14 @@ func (c *S3Client) ListObjects(ctx context.Context, continuationToken, prefix *s | |||
|
|
|||
| for _, obj := range output.Contents { | |||
| // log.Printf("key=%s size=%d", *obj.Key, obj.Size) | |||
| if obj.StorageClass == "GLACIER" || obj.StorageClass == "DEEP_ARCHIVE" { | |||
There was a problem hiding this comment.
这个改动会影响现有s3 的吗? 本来过滤了Glacier,给取消了
daixba
reviewed
Apr 24, 2022
| log.Printf("Unable to load credentials - %s\n", err.Error()) | ||
| } | ||
| srcCred := getCredentials(ctx, cfg.SrcCredential, cfg.SrcInCurrentAccount, sm) | ||
| aliyun_endpoint := "https://oss-" + cfg.SrcRegion + ".aliyuncs.com" |
There was a problem hiding this comment.
golang 不是这种 命名规范, 驼峰, 不带下划线
|
|
||
| // Restore Aliyun Glacier Object | ||
| func RestoreObject(obj *Object, oss_bucket *oss.Bucket) error { | ||
| err := oss_bucket.RestoreObject(obj.Key) |
There was a problem hiding this comment.
这个功能可以事先做吗? 意思就是说, 可以先批量恢复后再 调跑dth 吗
| //fmt.Println("======restore object======") | ||
| if obj.StorageClass == "GLACIER" { | ||
| fmt.Println("----------Start restoring object.----------") | ||
| err := RestoreObject(obj, oss_bucket) |
There was a problem hiding this comment.
不好意思，回复晚了。同时，非常感谢答复！
- 这个改动会影响现有s3 的吗? 本来过滤了Glacier,给取消了
//会影响，目前这个只是针对特定的客户和场景。 - 这个功能可以事先做吗? 意思就是说, 可以先批量恢复后再 调跑dth 吗
- 因为程序里会过滤掉Glacier和Deep Archive，如果不注释过滤Glacier和Deep Archive的代码，则需要修改Storage Class，把Glacier改成Standard或者IA
- 如果是批量恢复，则需要把过滤Glacier和Deep Archive的代码注释掉。因为已经做了批量恢复，不再需要过滤Glacier
- 只是如果迁移周期比较长，那么第一种方式会增加很多成本，第二种方式需要反复解冻，也会增加成本。
- 上述两种方式都有额外步骤需要客户操作且迁移到S3后都是Standard存储类型
- 参考1：归档类型Object：对于归档类型的Object，解冻状态默认持续24小时，24小时内再次调用RestoreObject接口则解冻状态会自动延长24小时，一次解冻流程内可有效调用7次RestoreObject接口达到最长7天的解冻持续时间。您也可以通过传入解冻天数，一次调用RestoreObject接口指定最长7天的解冻持续时间。
- 参考2： 数据取回费用是0.06元/GB
- 所有glacier 都能调用吗?
//目前只是测试了阿里云，且我理解是需要对dth支持的每个公有云对象存储额外增加RestoreObject函数。
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #, if available:
Current version of dthcli doesn't support migrating archive files.
Description of changes:
Mainly added two functions which are InitializaOss() and RestoreObject().
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.