Rolling Windows and Sticky Bytes

<tagline type="clever"/>

Amazon Simple Storage Service, Heroku and Rails

I wasn’t able to find a concise walkthrough for getting access controlled download links for S3 resources up and kicking in a Heroku web application so I thought I would put something together.

First step — users and permissions!

You will need to create an IAM user to which you can grant permissions to generate pre-authenticated URLs for your bucket objects. I’m going to assume you already know what an S3 bucket is (or you know how to Google), that you’ve created a bucket and that you’ve populated it with a few non-public folders and files. When you create the new user account, be sure to generate an access key and record it somewhere safe — you’ll need to plug the Access Key Id and Secret Access Key into your application environment later.

After you create the user, you’ll have to assign permissions by using groups or roles or (if you only ever intend to have a single user account that your web app uses to interface with S3) by assigning permissions to the user directly. The permissions will look something like the following:

"Statement": [
    "Effect": "Allow",
      "Action": [
      "Resource": [

The permissions above grant the applicable user / group / role the ability to Get or List any object in the bucket. You can make the permissions more (or less) restrictive if appropriate. Keep in mind that because these permissions do not include the ListAllMyBuckets action, the user account will not be able to access the S3 console. This may make testing the permissions using a web browser challenging but you can always temporarily grant the account the ListAllMyBuckets permission or use an interactive ruby shell (irb) to instantiate a bucket object directly and test from there.

Before going further, you’ll need to make sure the aws-sdk gem is installed on the host you’ll be using for development / testing. Again, not going to cover this here.

Once you’ve assigned the permissions, fire up irb and test that your access controls are working correctly. Substitute your own access key information, bucket name, and object key, obviously. The object key is just the path to the file as it appears in the AWS S3 console. Keep in mind that objects aren’t actually stored hierarchically (i.e. in folders) on S3 but it’s a useful convention for making S3 objects people-friendly and URL-friendly.

s3 = :access_key_id => 'ABCDEFGHIJKLMNOPQRST', :secret_access_key => 'abcdefghijklmnopqrstuvwxyz1234567890ABCD')
bucket = s3.buckets['mybucket']
s3obj = bucket.objects['path/to/private/object.ext']
@download_url = photo.url_for(:read, :expires => 15.minutes)

This code will create a connection to S3, instantiate a ruby object for your bucket, select a (presumably private) bucket object and give you a download URL that can be used by anybody to read the object for 15 minutes. If you receive a forbidden message anywhere along the way, go back and recheck your permissions. For testing purposes, paste the URL that irb spits out into a browser with the cookies and cache cleared (an incognito window is good for this) to make sure you actually get the file back when you visit the URL.

Once you can safely retrieve URLs, you can punch similar code into your Rails controller to make these temporary object URLs available to your views. The current code does, however, present a problem — how do you keep your access key secure? Well, you’re going to want to use Heroku environment variables to store this information. Punch the following command into the command line on the box where you normally run your Heroku deployments to set the appropriate environment variables in your Heroku environment:

heroku config:add AWS_ACCESS_KEY_ID=ABCDEFGHIJKLMNOPQRST AWS_SECRET_ACCESS_KEY=abcdefghijklmnopqrstuvwxyz1234567890ABCD S3_BUCKET_NAME=mybucket

You can now replace the hard-coded values in your code with environment variables so you don’t end up checking your credentials into revision control accidentally:

s3 = :access_key_id => ENV['AWS_ACCESS_KEY_ID'], :secret_access_key => ENV['AWS_SECRET_ACCESS_KEY'])
bucket = s3.buckets[ENV['S3_BUCKET_NAME']]

One other useful trick — if you want to override the default browser behaviour for a file type and force the file to be downloaded instead, you can set the Content-Disposition header. This can be done in the Metadata section of the S3 console for a file:

Key: Content-Disposition
Value: attachment; filename=Downloaded Copy of object.ext

You can also set this header when you call url_for (which is probably more scalable):

@download_url = photo.url_for(:read, :expires => 15.minutes, :response_content_disposition => 'attachment; filename=Downloaded Copy of object.ext')

Setting the Content-Disposition header lets you do neat tricks like give bundles of files the same filename in S3 (e.g. but have them save to the user’s computer with a unique name (e.g. “An Album of Boring Family”).


No Comments »