'mc pipe' with '--enc-c=' fails with 'Unsupported header 'x-amz-checksum-algorithm' received for this API call.' when sending to Backblaze/Wasabi.

[trackroute]

Expected behavior

I expected mc pipe with --enc-c to successfully upload to Backblaze or Wasabi (the two providers I've tested with).

My use case: I'm piping the output of zfs send to mc pipe to backup zfs snapshots and wish to use a SSE-C key.

Actual behavior

When using mc pipe with --enc-c to backup to Backblaze or Wasabi I'm receiving the following error: mc: <ERROR> Unable to write to one or more targets. Unsupported header 'x-amz-checksum-algorithm' received for this API call.

Note: mc cp works fine, and mc pipe without --enc-c also works fine, but mc pipe with --enc-c returns the above error when backing up to Backblaze or Wasabi.

Steps to reproduce the behavior

1. Create a random test file to try upload:
   dd if=/dev/urandom of=/tmp/rand bs=100K count=1

2. Try to upload the file:
   cat /tmp/rand | mc pipe backblaze/{TARGET_BUCKET}/rand --enc-c='backblaze/{TARGET_BUCKET}/=0000000000000000000000000000000000000000000'

The above example just uses '0000000000000000000000000000000000000000000' as the base64 encoded SSE-C key for simplicity sake. Obviously replace {TARGET_BUCKET} with your bucket name and replace 'backblaze' with your configured alias.

mc --version

mc version RELEASE.2024-06-29T19-08-46Z (commit-id=f5b325d9d399e09b4d21384081e397a52f6ca89d)
Runtime: go1.22.4 linux/amd64
Copyright (c) 2015-2024 MinIO, Inc.
License GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html

System information

Debian 12, Linux (x86_64)

Thanks.

[klauspost]

We don't provide workarounds for incomplete S3 implementation.

[trackroute]

Well, I've just tested with AWS S3 and the minio play server and it isn't working with either of those either. So your dismissal of this as being due to incomplete S3 implementation appears premature.

play.min.io:
cat rand | ./mc pipe play/testbucket/rand --enc-c='play/testbucket/=0000000000000000000000000000000000000000000'
mc: <ERROR> Unable to write to one or more targets. Requests specifying Server Side Encryption with Customer provided keys must provide an appropriate secret key.

AWS S3:
cat rand | mc pipe s3/{BUCKET}/rand --enc-c='s3/{BUCKET}/=0000000000000000000000000000000000000000000'
mc: <ERROR> Unable to write to one or more targets. The object was stored using a form of Server Side Encryption. Correct SSE-C request parameters are required for this request when specifying checksums for each part.

[klauspost]

Yeah, that seems more like a bug. Seems like the CompleteMultipartUpload doesn't include the SSE-C key:

 POST /trace.json?uploadId=njks15ONbwS5Ahg9iqtDtGJg3CvxtZzyG2KeijeI8bDPAfmx7PwcMDY4HZuGGk.SlQ2ynP4yHi2bAW0ztavh6c_Mp_WQ6g76VHvRhXalGKjAnYs7DnUd_phMDSIluIki HTTP/1.1
Host: xxxx.s3.dualstack.us-east-1.amazonaws.com
User-Agent: MinIO (windows; amd64) minio-go/v7.0.72 mc/DEVELOPMENT.GOGET
Content-Length: 224
Accept-Encoding: zstd,gzip
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20240702/us-east-1/s3/aws4_request, SignedHeaders=content-type;host;x-amz-checksum-crc32c;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
Content-Type: application/octet-stream
X-Amz-Checksum-Crc32c: FTkqvw==
X-Amz-Content-Sha256: 8a66d1f9e9630ee765e40e0a60a74dd52c0155ac6451077dc491e5cd122dac3f
X-Amz-Date: 20240702T162253Z

mc: <DEBUG> HTTP/1.1 400 Bad Request
Connection: close
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 02 Jul 2024 16:22:48 GMT
Server: AmazonS3
X-Amz-Id-2: 2ALv2+/Ccm8OUE91MCd7/Jg5wwjwSkraCHM5sgi1skH9n45bBaEDeOmIaay8+exyfzUChWnD1ZM=
X-Amz-Request-Id: ZDE903WVN6CAK6GX

166
<Error><Code>InvalidRequest</Code><Message>The object was stored using a form of Server Side Encryption. Correct SSE-C request parameters are required for this request when specifying checksums for each part.</Message><RequestId>ZDE903WVN6CAK6GX</RequestId><HostId>2ALv2+/Ccm8OUE91MCd7/Jg5wwjwSkraCHM5sgi1skH9n45bBaEDeOmIaay8+exyfzUChWnD1ZM=</HostId></Error>

[klauspost]

I will take a look at https://github.com/minio/minio-go

[klauspost]

See minio/minio-go#1980

[trackroute]

That was fast work 🥇 , thanks so much 👍 .