Rolling Windows and Sticky Bytes

<tagline type="clever"/>

Officially Open Source

Well, I finally made my first contribution to the Open Source community with this pull request for the “axlsx” Ruby library, used to programatically generate Excel spreadsheets.

I’m grateful that my employer and my management team have allowed me to contribute to the project on company time.

1 Comment »

Debugging an Android Action Bar Menu Issue

I’ve been working through the University of Maryland Programming Mobile Applications for Android Handheld Systems course and recently ran into an issue during one of the labs. I could not figure out why one of my views was not calling its onClick handler. Turns out, the handler was being called but log messages were not being recorded in LogCat.

I eventually determined that the issue was that the location had not been set in the mock location provider included in the lab. For some reason, the action bar overflow button was not appearing, meaning I did not have access to the location provider settings that should have appeared under this button.

I eventually learned on the course forums that you can force the action bar overflow menu to appear by pressing the Page Up key on your keyboard. Once I was able to summon the menu, everything worked as expected.

 

No Comments »

Amazon Simple Storage Service, Heroku and Rails

I wasn’t able to find a concise walkthrough for getting access controlled download links for S3 resources up and kicking in a Heroku web application so I thought I would put something together.

First step — users and permissions!

You will need to create an IAM user to which you can grant permissions to generate pre-authenticated URLs for your bucket objects. I’m going to assume you already know what an S3 bucket is (or you know how to Google), that you’ve created a bucket and that you’ve populated it with a few non-public folders and files. When you create the new user account, be sure to generate an access key and record it somewhere safe — you’ll need to plug the Access Key Id and Secret Access Key into your application environment later.

After you create the user, you’ll have to assign permissions by using groups or roles or (if you only ever intend to have a single user account that your web app uses to interface with S3) by assigning permissions to the user directly. The permissions will look something like the following:

{
"Statement": [
  {
    "Effect": "Allow",
      "Action": [
        "s3:Get*",
        "s3:List*"
      ],
      "Resource": [
        "arn:aws:s3:::mybucket",
        "arn:aws:s3:::mybucket/*"
      ]
    }
  ]
}

The permissions above grant the applicable user / group / role the ability to Get or List any object in the bucket. You can make the permissions more (or less) restrictive if appropriate. Keep in mind that because these permissions do not include the ListAllMyBuckets action, the user account will not be able to access the S3 console. This may make testing the permissions using a web browser challenging but you can always temporarily grant the account the ListAllMyBuckets permission or use an interactive ruby shell (irb) to instantiate a bucket object directly and test from there.

Before going further, you’ll need to make sure the aws-sdk gem is installed on the host you’ll be using for development / testing. Again, not going to cover this here.

Once you’ve assigned the permissions, fire up irb and test that your access controls are working correctly. Substitute your own access key information, bucket name, and object key, obviously. The object key is just the path to the file as it appears in the AWS S3 console. Keep in mind that objects aren’t actually stored hierarchically (i.e. in folders) on S3 but it’s a useful convention for making S3 objects people-friendly and URL-friendly.

irb
s3 = AWS::S3.new( :access_key_id => 'ABCDEFGHIJKLMNOPQRST', :secret_access_key => 'abcdefghijklmnopqrstuvwxyz1234567890ABCD')
bucket = s3.buckets['mybucket']
s3obj = bucket.objects['path/to/private/object.ext']
@download_url = photo.url_for(:read, :expires => 15.minutes)
@download_url.to_s

This code will create a connection to S3, instantiate a ruby object for your bucket, select a (presumably private) bucket object and give you a download URL that can be used by anybody to read the object for 15 minutes. If you receive a forbidden message anywhere along the way, go back and recheck your permissions. For testing purposes, paste the URL that irb spits out into a browser with the cookies and cache cleared (an incognito window is good for this) to make sure you actually get the file back when you visit the URL.

Once you can safely retrieve URLs, you can punch similar code into your Rails controller to make these temporary object URLs available to your views. The current code does, however, present a problem — how do you keep your access key secure? Well, you’re going to want to use Heroku environment variables to store this information. Punch the following command into the command line on the box where you normally run your Heroku deployments to set the appropriate environment variables in your Heroku environment:

heroku config:add AWS_ACCESS_KEY_ID=ABCDEFGHIJKLMNOPQRST AWS_SECRET_ACCESS_KEY=abcdefghijklmnopqrstuvwxyz1234567890ABCD S3_BUCKET_NAME=mybucket

You can now replace the hard-coded values in your code with environment variables so you don’t end up checking your credentials into revision control accidentally:

s3 = AWS::S3.new( :access_key_id => ENV['AWS_ACCESS_KEY_ID'], :secret_access_key => ENV['AWS_SECRET_ACCESS_KEY'])
bucket = s3.buckets[ENV['S3_BUCKET_NAME']]

One other useful trick — if you want to override the default browser behaviour for a file type and force the file to be downloaded instead, you can set the Content-Disposition header. This can be done in the Metadata section of the S3 console for a file:

Key: Content-Disposition
Value: attachment; filename=Downloaded Copy of object.ext

You can also set this header when you call url_for (which is probably more scalable):

@download_url = photo.url_for(:read, :expires => 15.minutes, :response_content_disposition => 'attachment; filename=Downloaded Copy of object.ext')

Setting the Content-Disposition header lets you do neat tricks like give bundles of files the same filename in S3 (e.g. download.zip) but have them save to the user’s computer with a unique name (e.g. “An Album of Boring Family Photos.zip”).

References:
http://docs.aws.amazon.com/AWSRubySDK/latest/AWS/S3.html
http://docs.aws.amazon.com/AWSRubySDK/latest/AWS/S3/S3Object.html
https://devcenter.heroku.com/articles/s3
https://devcenter.heroku.com/articles/config-vars

No Comments »

Painless Thumbnail Creation

I recently had a need to create thumbnails and discovered that using off the shelf photo management software to generate thumbnails is not very practical since these applications tend to try to preserve the quality of the image regardless how small you make the image itself. The end result of my first attempt was a set of images that were 150px on their longest edge and still over 30KB each. For a thumbnail, that’s an awfully large file.

My wife has a copy of Photoshop so I fired that up and created an action to create a 150px thumbnail from an arbitrary image and used Photoshop automation to thumbnail them all in a batch. Here’s the rough process I followed:

  1. Make a copy of the images you want to thumbnail, just in case. Make a second backup and store it somewhere safe, just in case.
  2. Start up Photoshop and open up one of the images.
  3. Bring up the Actions panel if you don’t already have it open. Create a new action — it should start recording automatically.
  4. To size your image down, you’ll want to use the image fitter instead of the resize function. This is particularly important if you have a mix of portrait and landscape images since the resize function will only resize one dimension. If you resize a portrait image to 150px width, the height will be larger than 150px which is probably not what you want. To use the image fitter, select File => Automate => Fit Image. Set both dimensions to the desired longest edge (in my case, this was 150px) and click OK.
  5. Select File => Save for Web… and choose appropriate settings (I used JPEG High which gave me a thumbnail size of ~7.5KB with minimal quality loss). Save your thumbnail to a new location.
  6. Close the file. This is important because it keeps your action from opening up hundreds of files and leaving them open if you use it in Batch mode.
  7. Stop the action. You should now have an action you can use to generate one thumbnail.
  8. Finally, once you have your action set up, you can run it in Batch mode by selecting File => Automate => Batch… to run it against multiple files. Select the source directory and target directory. Make sure these are different or you may accidentally overwrite your originals (you made a backup… right?). In the Batch options, be sure to check the Override Action “Save As” Commands checkbox. The name is somewhat misleading — checking this checkbox prevents Photoshop from popping up the JPEG Options dialog and overwriting your tiny thumbnail file with a large one as part of the batch. If you leave this checkbox unchecked, the JPEG Options dialog will open for every image and your thumbnail will be overwritten with a larger file during the action.

The thumbnail run took about 5 minutes to process 150 images.

No Comments »

flash and flash.now

Are the flash messages in your Ruby on Rails application persisting through multiple pages in strange and exciting ways or failing to appear completely?

If so, you may be confusing flash[:warning] = "Some Message" with flash.now[:warning] = "Some Message"!

The key difference is that flash will persist to the next request before being cleared, where flash.now only persists until the end of the current request.

In practice, this means that your controller methods should use flash if your exit point is a redirect (since a new request will be created following the redirect) and should use flash.now if your exit point is a render (since flash.now will be presented to the user during the natural lifespan of the current request, then cleared).

Render — Wrong Way (flash will persist too long):

if song.valid?
  flash[:notice] = "Song saved successfully."
else
  flash[:warning] = "Unable to save song."
end
render :edit_song

Render — Right Way:

if song.valid?
  flash.now[:notice] = "Song saved successfully."
else
  flash.now[:warning] = "Unable to save song."
end
render :edit_song

Redirect — Wrong Way (flash will not appear):

if song.valid?
  flash.now[:notice] = "Song saved successfully."
else
  flash.now[:warning] = "Unable to save song."
end
redirect songs_url

Redirect — Right Way:

if song.valid?
  flash[:notice] = "Song saved successfully."
else
  flash[:warning] = "Unable to save song."
end
redirect songs_url

References: http://stackoverflow.com/questions/4613952/why-flash-message-wont-disappear

No Comments »

Magic Timestamps and Join Tables

…don’t mix.

If you’re creating a join table in Rails for a has_and_belongs_to_many relation, be sure that you don’t include t.timestamps to set up magic timestamps. When setting up the timestamps fields in the database (created_at, etc.), Rails establishes a NOT NULL constraint and since the timestamp magic doesn’t happen, adds across the join will fail.

No Comments »

Devise I18n Localization

Turns out that even if you install a translation for Devise, it only translates e-mail strings and flash messages — it does not include translations for the views (e.g. the Sign In page, the Register page, etc.). You have to use rails:generate to dump the built-in views to file so you can override them and replace all the strings with I18n t() calls yourself.

Not impressed right now.

I wonder if I should take some time to update the views in Devise itself, move the English translations into the locale file and submit a pull request…

No Comments »

Automating Elastic IP Assignment Redux

As a follow-up to my previous post on Automating Elastic IP Assignment, I went back and installed the package from my distro’s repository and updated my script with the new binary locations. I’m using Ubuntu on my development instance so this post will be very Ubuntu-centric.

The ec2-api-tools package is in the multiverse repository which is not enabled by default on Ubuntu so before I could get started, I had to enable the multiverse repository as documented in the Ubuntu documentation on Repositories:

  1. Edit /etc/apt/sources.list
  2. Uncomment the four lines for the multiverse repository
  3. Run apt-get update to fetch the repository index

Once I had the multiverse repository enabled, installing the package was as easy as apt-get install ec2-api-tools. The package payload is deployed into /usr.

The updated elastic-ip script is below.

/etc/init.d/elastic-ip

#!/bin/bash

# Set environment variables
export JAVA_HOME=/usr

export EC2_HOME=/usr
export EC2_BIN=$EC2_HOME/bin

export EC2_PRIVATE_KEY=/root/.ec2/keyfile.pem
export EC2_CERT=/root/.ec2/certfile.pem

export EC2_URL=https://ec2.us-east-1.amazonaws.com

# Associate the elastic IP with the instance
$EC2_BIN/ec2-associate-address -i i-deadbeef 10.99.99.5
No Comments »

Automating Elastic IP Assignment on Amazon EC2

For those unfamiliar, an Amazon Elastic IP is basically a static IP address for Amazon virtual machines (EC2 or VPC). It’s a dedicated IP for your instance so you don’t have to muck around with updating the IP address you want to connect to every time.

The only catch? If, like me, you shut down your dev machine when it’s not in use to avoid paying for idle CPU time, you will have noticed that the association between the instance and the Elastic IP evaporates on shutdown.

The solution in my case was to install the AWS API SDK on my instance (I put it into /usr/local/aws by hand, not realizing there’s a package for it in my Linux distro’s repository and was too lazy to go back and fix it), download my X.509 certificate to the instance and drop the following script into /etc/init.d:

/etc/init.d/elastic-ip

#!/bin/bash

# Set environment variables
export JAVA_HOME=/usr

export EC2_HOME=/usr/local/aws
export EC2_BIN=$EC2_HOME/bin

export EC2_PRIVATE_KEY=/root/.ec2/keyfile.pem
export EC2_CERT=/root/.ec2/certfile.pem

export EC2_URL=https://ec2.us-east-1.amazonaws.com

# Associate the elastic IP with the instance
$EC2_BIN/ec2-associate-address -i i-deadbeef 10.99.99.5

The script sets up the API environment, then calls ec2-associate-address to associate my instance with its elastic IP. Once the script was installed, I added a call to it in /etc/rc.local and now whenever my instance boots up, it automatically associates itself with its elastic IP.

Of course there are lots of ways you could implement equivalent functionality (I saw another post on the EC2 forums where they used route53 for dynamic DNS updates instead of using an Elastic IP) but this was the way I decided to approach it and it works well.

References:

https://forums.aws.amazon.com/message.jspa?messageID=229486

http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/SettingUp_CommandLine.html

http://docs.amazonwebservices.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-AssociateAddress.html

No Comments »

Devise Authentication, Internationalization and Mailed URIs

For those who aren’t familiar with it, Devise is an authentication framework for Ruby on Rails. I was initially hesitant to use a plug-in for authentication for a few reasons.

  1. I was worried that the various authentication frameworks represent a “kitchen sink” approach to authentication and I wasn’t sure if I needed all the features. More complexity means higher probability of coding errors.
  2. Rails 3 makes it easy to roll your own via ActiveModel::SecurePassword[1]. I had initially decided based on reading up on Rails authentication strategies that this would be the best approach since it would give me the most flexibility.

Eventually I changed my mind and went for Devise because of the 11 features provided by the Devise toolkit, my site required 5 of them (Database Authenticable, Token Authenticable, Recoverable, Rememberable, Validatable) and most of the rest would be “nice to have”. Rather than spending time and effort rolling my own version of each of these, it seemed prudent to re-use existing, well tested code.

Having made the leap to Devise, I figured it would be worth writing down my experience.

I was able to install the gem without issue by adding it to my Gemfile and using bundler in the usual way (as documented in the “Getting Started” section on the Devise github page). The “devise install” run went equally smoothly (n.b.: only because I followed the instructions closely).

At this point, the next step is to add Devise to your model and this is where I started to run into some difficulties. This wasn’t because of any problem with Devise itself but because in my original design, I had specified the database field for e-mail addresses as “email_address” instead of “email” (which is what Devise expects). When I created a migration to rename the database field, I found that the indexes on the table had names too long to rename!

Lesson Learned #1: Do not accept the generated index names for database migrations if you are indexing a lot of fields in a table.

After I determined that I would have to fix up the index names, I modified the migration for the database field rename so that it would drop the index, rename the email_address field to email, then re-add the index with a shorter name. This worked out reasonably well.

I had to futz with the generated model attributes a bit (editing some attr_accessible entries and customizing the devise features) but the modifications were minor.

The next step was to try to add an authentication call to a page. There are two (easy) ways to do this:

  1. Apply authentication to all methods in your controller. To do this, add before_filter :authenticate_user! to the controller (somewhere inside class UsersController < ApplicationController but outside of any individual controller method).
  2. Apply authentication to individual methods in your controller. To do this, add authenticate_user! to the controller method you want to protect.

Which strategy you use will depend on whether the whole object needs to be behind the “authentication wall” or if some actions on an object can be run by anyone (e.g. if your site has a “posts” controller, you may want the software to allow anybody to view a post but only allow authenticated users to comment on one).

Note that (as stated in the documentation), the name of the method will be authenticate_model!, so if you have devise on your users model, the call will be authenticate_user!, if you are using an “admins” model, it will be authenticate_admin!, if you are using a “commenters” model, it will be authenticate_commenter!, etc.

This bit me at least once during my experimentation, with the software throwing a missing method error.

Lesson Learned #2: Make sure you’ve specified the right model name in authenticate_model! calls

Once I had the dev site putting up an authentication wall, I moved on to getting a reset link sent out for a test user and getting the password set… and that’s when all hell broke loose.

I clicked the “reset password” link and the e-mail was sent. All good!

When I visited the generated reset link I was… forwarded to a login page. I spent a couple of hours trying to diagnose the problem, struggling with it until I checked the logs and remembered that my application is internationalized. This turned out to be important.

On each page visit, the site will do its best to make sure it serves you in the appropriate language. It does this by taking the following steps:

  1. Enumerate the available locales in the application
  2. Ask your web browser for your most preferred locale based on the list of available locales
  3. Set the locale to a) the locale specified on the URI (to ensure all URIs are RESTful), b) the preferred locale specified by your browser or if all else fails, c) the default locale
  4. If the locale has changed, redirect the browser to the current URI with the correct locale specified as a parameter

Naturally it’s important to make sure that the locale is set correctly on every internal link to avoid a constant barrage of phony redirects. Back in the early days of development on the application, I made sure this would happen by setting the default url options in my application controller to include the current locale like so:

def default_url_options(options={})
  { :locale => I18n.locale }
end

Essentially what was happening when a user was hitting the “reset” URI provided in the e-mail message was that they were reaching a page without a locale set and being redirected. Because the software strips single-use security tokens from redirect URIs, they were being forwarded to a login page instead of the password reset page.

So why didn’t the URIs being sent by Devise have the locale set? Essentially because ActionMailer (which is used to generate the mail messages) does not inherit from the application controller, so the default url options specified there do not apply.

Lesson Learned #3: You have to set the default locale for ActionMailer URIs if you have internationalized your application

To fix the locale problem in my e-mailed URIs, I had to add a line to set the locale to the ActionMailer settings for my environments:

config/environments/development.rb
config.action_mailer.default_url_options = { :host => 'dev.example.com:3000', :locale => I18n.locale }
config/environments/production.rb
config.action_mailer.default_url_options = { :host => 'example.com', :locale => I18n.locale }

Once I had set the ActionMailer default URL options and restarted the application, password reset links worked like a charm and I was able to reset the password for a test account and log into the site with it.

3 Comments »