How to Set Up an Amazon S3 Bucket for Use with OpenFarm

About the Openfarm Setup

OpenFarm uses Amazon S3 for cloud storage of media files such as profile and guide images.

OpenFarm has a server-side permission policy that Rails and PaperClip use when deleting / updating images and a “client-side” policy for users to dump unprocessed media into.

NOTE: You should not need to set up AWS to do image uploading on your local instance. On development environments OpenFarm defaults to the standard Rails upload methods. If you want to set up AWS to work on that process, please proceed.

Why Have We Done It This Way?

This setup is slightly more complicated than a direct upload setup, but creates a number of benefits such as:

• It frees web workers from the resource-intensive task of accepting large file uploads.
• It allows OpenFarm to be hosted on services like Heroku, where there is a 30 second request timeout.
• It allows the API to accept URLs of files hosted elsewhere (not just S3, but services like Imgur/Wordpress, etc.)
• By POSTing URLs instead of files, we can defer the process of downloading/cropping/resizing images to a background worker which creates a better experience for API and Website users.
• We can pay a third party (currently S3) to manage the file storage infrastructure, which frees up developers to work on more pertinent tasks and features.

A Step-by-Step Guide to Client-Side Setup for S3

1. Create a new S3 bucket. (Search Google if you don’t know how to do this.)
2. In the Bucket settings panel, select Permissions > Add CORS Configuration.
   
3. Replace the sample configuration by pasting in the following configuration:

   <?xml version="1.0" encoding="UTF-8"?>
   <CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
   <CORSRule>
   <AllowedOrigin>*</AllowedOrigin>
   <AllowedMethod>POST</AllowedMethod>
   <AllowedMethod>PUT</AllowedMethod>
   <AllowedHeader>*</AllowedHeader>
   </CORSRule>
   </CORSConfiguration>

   
4. Click Save. Click Close.
5. Create an IAM user for the S3 bucket. Select my account > Security Credentials in the top right of the page, then click Get Started with IAM Users in the pop-up dialog.
   
6. Click Create New Users. Enter at least one user name, and check the Generate an access key for each user checkbox if it is not already checked. Click Create.
   
7. Click Show User Security Credentials. Be sure to copy these credentials to some local textfile now. It is your only chance to do so. You can download the credentials to a local file or simply copy-and-paste. Click Close.
   
8. Add the credentials to the file config/app_environment_variables.rb. The correct format is like this, though your bucket name and key values will be different:

   ENV['S3_BUCKET_NAME']        = 'mybucketname'
   ENV['S3_ACCESS_KEY']         = 'ABCDEFGHIJKLMNOPQRST'
   ENV['S3_SECRET_KEY']         = 'a+/1SHdo8DbWX9/aBcDeFgHIjKLmnOpqrSTuVwx+'

9. Click Policies. If this is the first time you have used this part of the interface, there will be an introductory page: if so, click Get Started.
   
10. Click Create Policy.
    
11. Select Create Your Own Policy.
    
12. For Policy Name enter “Client”; for Policy Document enter the following (but replace YOUR_NAME_HERE with your name):

    {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Effect": "Allow",
    "Action": ["s3:PutObject", "s3:PutObjectAcl"],
    "Resource": "arn:aws:s3:::YOUR_NAME_HERE/temp/*"
    }
    ]
    }

    
13. Click Create Policy.
14. Again click Create Policy and select Create Your Own Policy. For Policy Name enter “Server”; for Policy Document enter the following (but replace YOUR_NAME_HERE with your name):

    {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Effect": "Allow",
    "Action": [
    "s3:DeleteObject",
    "s3:DeleteObjectVersion",
    "s3:GetObject",
    "s3:GetObjectAcl",
    "s3:PutObject",
    "s3:PutObjectAcl"
    ],
    "Resource": [
    "arn:aws:s3:::YOUR_NAME_HERE/*"
    ]
    }
    ]
    }

15. Your bucket should be ready to take uploads at this point.
16. The following steps are suggested (not required) in order to set objects in the temp/ directory to expire after a day. If you don’t do this, they will pile up.
17. Select Services > S3
    
18. In the list of buckets, click on the bucket you created earlier.
19. Click Properties and select Lifecycle from the accordion menu.
    
20. Click Add Rule.
21. Next to Apply the Rule to: select A Prefix and enter “temp/” in the input box. Click Configure Rule >
    
22. From the Action on Objects drop-down menu select Permanently Delete Only; set the Permanently Delete input setting to “1” days after the object's creation date. Click Review >
    
23. Review your rule and name it if you want to (it will be named “Rule for: temp/” by default), then click Create and Activate Rule.
24. Raise an issue if this doesn’t work.

Note that it might take some time for your bucket's name to be properly propagated through DNS. In the mean time you might get 307 redirect preflight error messages.