Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multipart Copy #80

Open
thiagorfernandes opened this issue May 15, 2018 · 2 comments · May be fixed by #106
Open

Multipart Copy #80

thiagorfernandes opened this issue May 15, 2018 · 2 comments · May be fixed by #106

Comments

@thiagorfernandes
Copy link

thiagorfernandes commented May 15, 2018 •
edited
Loading

Is the Multipart Copy working? I'm getting the following response when I try to copy a 6 GB file from a bucket to another:

ERROR: ErrorInvalidArgument
Message: The specified header is not valid in this context
Extra Details:
ArgumentName: x-amz-metadata-directive
ArgumentValue: REPLACE
RequestId: B9A7B3C41FB12F55
HostId: oTuaiCPM5pCw99BQ1RDzSMjSTRUXhCGKgkYvwa6KrhmGp8yM8l70+9UwvXXbH2RN9cilgPyHPRE=
@JustinKyleJames
Copy link

JustinKyleJames commented Sep 22, 2020 •
edited
Loading

I am also getting the same error when trying this. In my case the source and destination were in the same bucket.

@JustinKyleJames JustinKyleJames mentioned this issue Dec 7, 2020
Closed
2 tasks
@JustinKyleJames
Copy link

JustinKyleJames commented Dec 7, 2020 •
edited
Loading

I believe the fix should be to not add this if we are doing an UploadPartCopy but only if we are doing a CopyObject.

See:

https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html

vs

https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html

libs3/src/request.c

Lines 392 to 412 in 287e4be

if (params->httpRequestType == HttpRequestTypeCOPY) {
// Add the x-amz-copy-source header
if (params->copySourceBucketName && params->copySourceBucketName[0]
&& params->copySourceKey && params->copySourceKey[0]) {
char bucketKey[S3_MAX_METADATA_SIZE];
snprintf(bucketKey, sizeof(bucketKey), "/%s/%s",
params->copySourceBucketName, params->copySourceKey);
append_amz_header(values, 0, "x-amz-copy-source", bucketKey);
}
// If byteCount != 0 then we're just copying a range, add header
if (params->byteCount > 0) {
char byteRange[S3_MAX_METADATA_SIZE];
snprintf(byteRange, sizeof(byteRange), "bytes=%zd-%zd",
params->startByte, params->startByte + params->byteCount);
append_amz_header(values, 0, "x-amz-copy-source-range", byteRange);
}
// And the x-amz-metadata-directive header
if (properties) {
append_amz_header(values, 0, "x-amz-metadata-directive", "REPLACE");
}
}

I believe it should be:

    if (params->httpRequestType == HttpRequestTypeCOPY) {
        // Add the x-amz-copy-source header
        if (params->copySourceBucketName && params->copySourceBucketName[0]
            && params->copySourceKey && params->copySourceKey[0]) {
            char bucketKey[S3_MAX_METADATA_SIZE];
            snprintf(bucketKey, sizeof(bucketKey), "/%s/%s",
                     params->copySourceBucketName, params->copySourceKey);
            append_amz_header(values, 0, "x-amz-copy-source", bucketKey);
        }
        // If byteCount != 0 then we're just copying a range, add header
        if (params->byteCount > 0) { 
            char byteRange[S3_MAX_METADATA_SIZE];
            snprintf(byteRange, sizeof(byteRange), "bytes=%zd-%zd",
                     params->startByte, params->startByte + params->byteCount - 1);
            append_amz_header(values, 0, "x-amz-copy-source-range", byteRange);
        } else {
            // And the x-amz-metadata-directive header
            if (properties) {
                append_amz_header(values, 0, "x-amz-metadata-directive", "REPLACE");
            }
        }
    }

trel pushed a commit to irods/libs3 that referenced this issue Dec 9, 2020
@JustinKyleJames @trel
Only send x-amz-metadata-directive on CopyObject
054a76a 
Fixes bji#80

See:
  https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html
  and
  https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html
@trel trel linked a pull request Dec 9, 2020 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Only send x-amz-metadata-directive on CopyObject irods/libs3
2 participants
@JustinKyleJames @thiagorfernandes