Unable to deploy for 1st time itself - Upload to S3 bucket fails due to 'Error: Forbidden. Invalid according to Policy: Policy expired.'

[agnihotrish]

Version

5.34.8

Operating System

Windows

Browser

Chrome

What are the steps to reproduce this bug?

Run 'yarn webiny deploy'

What is the expected behavior?

Successfully deploy on S3 buckets

What do you see instead?

webiny info: Running "hook-after-deploy" hook...
webiny info: Uploading React application...
webiny error: Failed to upload static/js/496.ed4465dd.js.LICENSE.txt
Error: Forbidden

AccessDeniedInvalid according to Policy: Policy expired.DECPRFFJQT9SPPQ5oPqj5I92LA4E1uZYk3eyqcX3m+IzzAuTXDQIc1mhvsD06mLjmDGZjSv5Ot9ICkwJkMaoZnbgMOQ=
at C:\Users\anugr\Documents\Expo\demo-webiny2\node_modules@webiny\pulumi-aws\utils\uploadFolderToS3.js:154:21
at processTicksAndRejections (node:internal/process/task_queues:95:5)
webiny error: Failed to upload asset-manifest.json
Error: Forbidden

AccessDeniedInvalid according to Policy: Policy expired.DECXG21F2FKTNYY8Ydslfrj11Ma5sgJZDWSGD6t6wD8WD1h5z0KYsjBicpLcimQ1WX07yVwSBfZ4qCqcdjBoRSf2T1s=
at C:\Users\anugr\Documents\Expo\demo-webiny2\node_modules@webiny\pulumi-aws\utils\uploadFolderToS3.js:154:21
at processTicksAndRejections (node:internal/process/task_queues:95:5)
webiny error: Failed to upload index.html
Error: Forbidden

AccessDeniedInvalid according to Policy: Policy expired.DECXHPA2VT7ME49AmNX4jzlBn9XHf1yeSEVClejpK/K+sjG/Q84ddH0v1IiHljucx+CYc6P9ItGtOhfa4AVMzowxejo=
at C:\Users\anugr\Documents\Expo\demo-webiny2\node_modules@webiny\pulumi-aws\utils\uploadFolderToS3.js:154:21
at processTicksAndRejections (node:internal/process/task_queues:95:5)
webiny error: Failed to upload static/js/runtime-main.c8075e5d.js
Error: Forbidden

AccessDeniedInvalid according to Policy: Policy expired.DECJ7SB2HQW6S5RZtUqeY7DFqCSNsWUobQS2dKRmVVww4Wn8WMvRoUYaeNoJ5Rq307EXkZ89BDF2WuvrjeRiHz91Kjg=
at C:\Users\anugr\Documents\Expo\demo-webiny2\node_modules@webiny\pulumi-aws\utils\uploadFolderToS3.js:154:21
at processTicksAndRejections (node:internal/process/task_queues:95:5)
webiny error: Failed to upload static/js/main.2ab6e29a.js.LICENSE.txt
Error: Forbidden

AccessDeniedInvalid according to Policy: Policy expired.DECG3G8JV38XZR2Mux6tfwky/wnMfrv/6MKgew3V6xGs/KyMooYSSDZ+Bk296a1yHr14ROO7nEC/I4m515qoKukxiKY=
at C:\Users\anugr\Documents\Expo\demo-webiny2\node_modules@webiny\pulumi-aws\utils\uploadFolderToS3.js:154:21
at processTicksAndRejections (node:internal/process/task_queues:95:5)
webiny error: Failed to upload static/css/main.6647481e.css
Error: Forbidden

AccessDeniedInvalid according to Policy: Policy expired.DECH47S0QC0Q3QDEmZFQ4TYlAP4l+BD2rYC+OYakQpmMkAdEQFAbjR9rkmG3ZQpb952XS0073AkpbvfILe0ZuJqkJAg=
at C:\Users\anugr\Documents\Expo\demo-webiny2\node_modules@webiny\pulumi-aws\utils\uploadFolderToS3.js:154:21
at processTicksAndRejections (node:internal/process/task_queues:95:5)
webiny error: Failed to upload static/js/496.ed4465dd.js
Error: Forbidden

AccessDeniedInvalid according to Policy: Policy expired.DECJYK1YW7JWN5DAjVlqz7YGX2tdO+lS3iSSoxQG5Mmkr/Zm37nJHk9HiVcOQGFLcr4+hEi5blXU8tnSl117Hhp6oVU=
at C:\Users\anugr\Documents\Expo\demo-webiny2\node_modules@webiny\pulumi-aws\utils\uploadFolderToS3.js:154:21
at processTicksAndRejections (node:internal/process/task_queues:95:5)
webiny error: Failed to upload static/js/main.2ab6e29a.js
Error: Forbidden

AccessDeniedInvalid according to Policy: Policy expired.DECGBQAX7B1P6XC84l/llpRIod+ocKTWBTBFJzoNSSpPu0jp9hglCSPKNl2d6bKko9DPIE6wlLGkEKvRvoe+TLQiYtw=
at C:\Users\anugr\Documents\Expo\demo-webiny2\node_modules@webiny\pulumi-aws\utils\uploadFolderToS3.js:154:21
at processTicksAndRejections (node:internal/process/task_queues:95:5)

Additional information

Buckets are created successfully on S3 but they have 0 objects inside them. All other deployment steps were completed successfully as well.

Possible solution

No response

[jonnysmith1981]

Was there any movement on this? I get exactly the same problem. All the rest of the AWS stack deployed fine, but the uploading of the web apps to the various S3 buckets has failed with the same error.

[Pavel910]

@jonnysmith1981 do you get the same issues with the latest version of Webiny? The latest is 5.39.3.
5.34.8 is essentially a dead version, because AWS no longer supports runtimes older than Node 16. The latest Webiny uses Node 18.

[jonnysmith1981]

@Pavel910 just checked the Webiny version and it's v5.39.3. Using node 18.19.0 locally

[Pavel910]

@jonnysmith1981 the AWS credentials you use to deploy a project, are those "full access" credentials? If not, have you used our Cloudformation template to setup the deployment user?

The process is described here: https://www.webiny.com/docs/infrastructure/aws/configure-aws-credentials

And the exact template can be found here: https://www.webiny.com/docs/infrastructure/aws/configure-aws-credentials#deploy-webiny-project-aws-cloud-formation-template

[jonnysmith1981]

I have gone through and re-setup my user as per the instructions:

I still get the same error: "Invalid according to Policy: Policy expired".

When i try to manually copy one of the files for the admin site to the s3 bucket from the command line, that works fine:

So my user definitely has PutObject access on that bucket

[jonnysmith1981]

More info....

I dug through CloudTrail and can see that on the initial deployment of Webiny, all the S3 bucket setup events produced "MalformedPolicy" errors. The error message is: Invalid principal in policy

The principal in the policy is:

I have confirmed that the GUID is the same as the one referenced in the origin on the cloudfront distribution, so i have no idea why it says the principal is invalid.

[Pavel910]

Buckets are created with Pulumi, and if there were errors, those errors would show at deploy time, and the deploy would abort. Also, the errors would show in Pulumi state files. We've never encountered an issue like this.

Just guessing here, could the upload of files after deploy throw errors because of the recent time change? Maybe your machine doesn't have the time synced correctly? This could explain why you have all the access on the bucket, and can upload manually, but not programmatically.

[jonnysmith1981]

Ok, looks like the MalformedPolicy error is a red-herring as the policy does actually exist on the bucket despite the error.

Investigating the time difference is interesting. If i do curl http://s3.amazonaws.com -v then the S3 server time reports 09:32:04 GMT. Obviously we in the UK are now in BST so my machine reports 10:32:04.

I find it strange that AWS servers cannot cope with different timezones, especially since the AWS CLI S3 upload works.

[swapnilmmane]

Hi @jonnysmith1981, would it be possible for you to deploy a fresh Webiny project?

Also, considering the error, it seems that the issue is not related to Webiny code, as I recently deployed a project and it worked as expected for you.

I request that we move this conversation to our official community Slack, where we will have a wider support available for you. You can join the Webiny community Slack through this link: https://www.webiny.com/slack

[jonnysmith1981]

I know its not good practise to add to closed issues, however I have just managed to get the admin site deployed.

I was checking the various pulumi-aws/utils scripts that are called, and in the getPresignedPost.js there is this:

const s3Params = {

    Key: fields.key,

    Expires: 20,                  <----------************

    Bucket: bucket,

    Conditions: [["content-length-range", 0, 26214400], {

      acl

    }],

    Fields: fields

  };

If I change the Expires value to 600 (10 minutes) then the upload part of the deployment works fine.

I'm not entirely comfortable with changing this value as it feels like more of a hack than an actual "fix".

[Pavel910]

@jonnysmith1981 no worries, thanks for coming back with an update!
Yeah so this definitely is something related to time. We can increase this number to be several minutes, I just wonder if a smaller increment would work for you, or is 10 minutes the smallest number that does the trick 🤔

[jonnysmith1981]

Its a difficult one to judge as 20 seconds wasn't long enough, but 30 seconds might be OK for me and my laptop.

[Pavel910]

Look, stick to your hack for now, and we'll try to make it configurable. Worst case scenario, we'll add an ENV variable to control that value.