403 SignatureDoesNotMatch Forbidden in newer releases

[Jayd603]

I was running s3proxy containers behind haproxy and after updating to the latest version from an older version (current latest tag on dockerhub), something broke.

403 SignatureDoesNotMatch Forbidden {}
[s3proxy] D 11-21 18:52:34.837 S3Proxy-Jetty-14 o.gaul.s3proxy.S3ProxyHandler:301 |::] request: Request(GET http://somehost.realdomain.net/)@2439a92f

I have -e S3PROXY_CORS_ALLOW_ALL="true" and -e S3PROXY_ENDPOINT="http://127.0.0.1:30565" (changing endpoint to https makes no difference)

https is terminated on haproxy and it connects to an s3proxy running on localhost:

The actual request URL from the client is https, but it does show as http in the s3proxy logs, perhaps that is it? ..or is the non-standard port included in the signature calculation now?

I will try to debug this further myself too.

UPDATE:
I went back 5 months to "sha-4175022" docker image and it works (2.2.0 i believe). The latest docker image from 2 days ago does not work. Not sure what changed between 2.2 and 2.3 or maybe it is just because it is a dev build with some broken code pushes.

[Jayd603]

FYI - i also tested a docker pull from a few weeks back and that also didn't work for me.... just letting you know and seeing if you have an idea of what is breaking. I'm using s3cmd client. The working container I have from 5 months ago I believe has the jclouds updates to improve performance so that was a big one so I'm in no hurry to update to newer anyway.

When you guys slap the latest tag on the 2.3.0 docker image I will give it a try so we know 2.3.0 broke something (at least with my use case)

[Jayd603]

Tested latest 2.4.0, same thing. I suspect this broke things for others out there since all i'm doing is proxying to s3proxies on different ports

[gaul]

You can try experimenting with -DLOG_LEVEL=trace to determine what's going on. I'm not going to set up haproxy myself though. S3Proxy uses Ceph's comprehensive s3-tests so I suspect something is incorrect in your haproxy configuration that doesn't forward the authorization header or rewrites the URL in some way. I would not assume that a previous S3Proxy version authentication behavior was correct.

[Jayd603]

I tried it but it doesn't seem to give specifics:

[s3proxy] D 11-23 18:49:36.860 S3Proxy-Jetty-19 o.gaul.s3proxy.S3ProxyHandler:302 |::] request: Request(GET http://dyniback.s3.domain.net/)@7633f90
[s3proxy] T 11-23 18:49:36.862 S3Proxy-Jetty-19 o.gaul.s3proxy.S3ProxyHandler:327 |::] header: Authorization: AWS4-HMAC-SHA256 Credential=dyniback/20241123/US/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=4c51afa1cdb1c00f3e410662a19991dc923c1760ff4f2c4c8316c743ccc33b49
[s3proxy] T 11-23 18:49:36.862 S3Proxy-Jetty-19 o.gaul.s3proxy.S3ProxyHandler:327 |::] header: x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
[s3proxy] T 11-23 18:49:36.862 S3Proxy-Jetty-19 o.gaul.s3proxy.S3ProxyHandler:327 |::] header: content-length: 0
[s3proxy] T 11-23 18:49:36.862 S3Proxy-Jetty-19 o.gaul.s3proxy.S3ProxyHandler:327 |::] header: x-amz-date: 20241123T184936Z
[s3proxy] T 11-23 18:49:36.862 S3Proxy-Jetty-19 o.gaul.s3proxy.S3ProxyHandler:327 |::] header: host: dyniback.s3.domain.net
[s3proxy] T 11-23 18:49:36.862 S3Proxy-Jetty-19 o.gaul.s3proxy.S3ProxyHandler:327 |::] header: Accept-Encoding: identity
[s3proxy] D 11-23 18:49:36.882 S3Proxy-Jetty-19 o.gaul.s3proxy.S3ProxyHandler:3002 |::] sendSimpleErrorResponse: 403 SignatureDoesNotMatch Forbidden {}

[Jayd603]

The only thing I can see is I am going from https to http. if that is part of the signature it will fail and the only solution will be to configure ssl within s3proxy.

Is there anything that shows exactly what is used to make up the signature?

I configured s3proxy with ssl and still sig does not match.

[Jayd603]

The plot thickens, I can't actually make s3proxy work at all on a non-standard port, even without HAProxy:

docker run -d --restart=unless-stopped --net=host --name=s3-backups -v /mnt/s3/backups:/mnt/s3/backups -e S3PROXY_AUTHORIZATION=aws-v2-or-v4 -e S3PROXY_IDENTITY=backups -e S3PROXY_CREDENTIAL=JEuhCv3U -e S3PROXY_ENDPOINT="http://127.0.0.1:30565" -e S3PROXY_CORS_ALLOW_ALL="true" -e S3PROXY_SERVICE_PATH="/backups" -e JCLOUDS_PROVIDER="filesystem" -e S3PROXY_V4_MAX_NON_CHUNKED_REQ_SIZE="310000000" -e JCLOUDS_FILESYSTEM_BASEDIR="/mnt/s3/backups" -e LOG_LEVEL="trace" andrewgaul/s3proxy:2.4.0

Trying to connect using 127.0.0.1:30565 from s3cmd - signature does not match. Adding a domain name to my hosts file didn't make a different either. What else changed?

NOTE: The above config works with the 5 month old container just fine.

I read somewhere that the signature calculation is different when using a non-standard port, is that part what is broken? Apparently the :80/:443 is stripped but when using a non-standard port, the port number is used in the calculation.

Actually, I can't make s3proxy work at all with the latest container. Nothing works. Even using port 80. I did try the latest aws cli too.

[Jayd603]

docker run -d --restart=unless-stopped --net=host -m 500m --name=s3-test -e S3PROXY_AUTHORIZATION=aws-v2-or-v4 -e S3PROXY_IDENTITY=test -e S3PROXY_CREDENTIAL=test -e S3PROXY_ENDPOINT="http://127.0.0.1:8080" -e S3PROXY_CORS_ALLOW_ALL="true" -e LOG_LEVEL="trace" andrewgaul/s3proxy:2.4.0

that is working config. so something i'm passing is breaking the signature calc possibly, will try to narrow it down

[Jayd603]

Looks like I solved the puzzle finally - S3PROXY_SERVICE_PATH was breaking things. Removed it and it worked.

Oh and, it works fine with HAProxy