Tag Archives: ruby

Learning Redis

This post is part of my weekly tech learning series, where I take one hour each week to try out a piece of technology that I’d like to learn. Make sure to read to the end, where I have a screencast overview of the final application.

This week I decided to learn a bit more about Redis. I’ve used Redis in Chirk HR already but only as part of a non-critical service to see how it performs under load (I’d like to screencast this later). Today I wanted to use Redis as an actual data store but first, what is Redis?

What is Redis?

Redis is an open source key-value store. I think of it as a light nosql database with a lot of flexibility.

Today’s application

My idea for today was to build my running journal but this time as a CRUD style web app. I want to stay close to the metal so I can really see how Redis works so I picked my second favorite web framework: Sinatra.

Basically I wanted to mimic a Rails scaffold inside Sinatra using Redis as the data storage. I’m also using the redis gem because it is very similar to the actual Redis API. This means I’ll be implementing the data layer myself, without an ORM.

(Normally I’d use ActiveRecord or Datamapper for production applications instance but I’m more concerned with learning and exploration with this.)

Since I will be doing a lot of data layer work I’m also opting for heavy TDD with this app, using minitest and autotest.

Here is the Gemfile I ended up with:

source :rubygems
 
gem "sinatra", :require => "sinatra/base"
gem "shotgun"
gem "redis"
 
group :test do
  gem "minitest"
  gem "rack-test"
  gem "autotest"
  gem "autotest-notification"
end

Use TDD for the application behavior

After getting everything hooked up I decided to split the tests into two files:

  1. test_redis_running.rb to test the RedisRunning Sinatra application at the integration layer. This would be the controller and view if you’re thinking about it as a MVC app.
  2. test_run.rb to test the Run class, which is used as the data layer between the application and Redis.

I like to start with integration testing and write pending tests for each major workflow or feature. I came up with the following integration tests to get started with:

  • test_get_root_response
  • test_get_root_list_runs
  • test_get_root_add_form_present
  • test_post_new_with_valid_run
  • test_post_new_with_invalid_run
  • test_get_edit_response
  • test_get_edit_show_form
  • test_put_update_with_valid_update
  • test_put_update_with_invalid_update
  • test_delete_response
  • test_delete_should_delete_record

This is very similar to Rails scaffold except I decided to put the new form on the list of runs (:index) and to skip the details of a run (:show). With such a small data model, a list provides a perfect summary.

The first test I wrote was to check that getting the index was successful, which isn’t about Redis at all so I won’t bore you with the details of it.

The second test on the other hand (test_get_root_list_runs), is the important one (and as you’ll see later is all that I had time for). Here I wanted to do a few things:

  1. Create a few runs and save them to Redis
  2. Load the index page
  3. Check that the runs were displayed

In order to implement this test I have to do a full cycle of saving and loading runs from Redis. This meant I’d need to build the data layer now.

Here is the full integration test.

  def test_get_root_list_runs
    run1 = Run.new(:date => "2012-09-28",
                   :distance => "3mi",
                   :duration => "30:00",
                   :pace => "10:00",
                   :comment => "A run")
    assert run1.save, "Save failed"
 
    run2 = Run.new(:date => "2012-09-29",
                   :distance => "3mi",
                   :duration => "27:00",
                   :pace => "9:00",
                   :comment => "Another run")
    assert run2.save, "Save failed"
 
    get '/'
 
    assert last_response.body.include?("A run")
    assert last_response.body.include?("Another run")
  end

RedisRunning Data Layer

The integration test needed three things from my Run class:

  1. Create an object with some attributes
  2. Save the object to Redis
  3. Load the saved object out of Redis

I started with the first, using standard Ruby accessors.

Run accessors and attributes

A run has five attributes that a user should be able to set:

  1. date
  2. distance
  3. duration
  4. pace
  5. comment

Additionally, I’m going to need some way to uniquely identify each run. In Postgres I’d just use an id field as a primary key but from what I see, Redis doesn’t have primary keys. Only keys.

At first I thought of using the date as the primary key, but many runners will run multiple times per day so that won’t work. I could use a combination of the date and another field like distance or duration but there is also a chance those will conflict also (e.g. two runs per day, around a 1 mile track at a specific pace).

Redis doesn’t have primary keys so you might need to roll your own.

So I’ll need to be creating my own unique ids for each run. I’ve used UUIDs before and they seem like a good fit here.

So using attr_accessor and initialize I can easily fulfill these requirements.

The test:

class RunTest < MiniTest::Unit::TestCase
  def test_has_attributes
    run = Run.new
 
    ['date','distance','duration','pace','comment'].each do |attribute|
      assert run.respond_to?(attribute), "Run does not respond to #{attribute}"
      assert run.respond_to?(attribute + "="), "Run does not respond to #{attribute}="
    end
 
  end
 
  def test_should_allow_setting_attributes_on_initialize
    run = Run.new(:date => '1', :distance => '2', :duration => '3', :pace => '4', :comment => '5')
 
    assert_equal '1', run.date
    assert_equal '2', run.distance
    assert_equal '3', run.duration
    assert_equal '4', run.pace
    assert_equal '5', run.comment
  end
end

The implementation:

class Run
  attr_accessor :id
  attr_accessor :date
  attr_accessor :distance
  attr_accessor :duration
  attr_accessor :pace
  attr_accessor :comment
 
  def initialize(attributes={})
    attributes.each do |attribute, value|
      setter = "#{attribute}="
      self.send(setter, value) if self.respond_to?(setter)
    end
  end
end

Generating unique ids with UUID

I’ve used some libraries to generate UUIDs in Ruby but I just found out that Ruby 1.9.3 now has UUID support in the standard library as part of SecureRandom. This makes the UUID code simple.

The test.

  def test_id_should_be_a_uuid
    run = Run.new
 
    assert run.id.is_a?(String)
    assert_equal 36, run.id.length
    assert_equal 4, run.id.count('-')
  end

The implementation.

# Way up top outside the Run class...
require 'securerandom'
 
# In the Run class...
  def id
    @id ||= SecureRandom.uuid
    @id
  end

By using ||= I can make sure that if a Run already has an id then I won’t generate it again (existing record).

I also wrote a brute force test to make sure Ruby’s UUIDs were unique and that I wasn’t misreading the docs. It’s not really production quality but good for exploration of a new API:

  def test_id_should_be_unique
    # Lets create a bunch of runs and check their uniqueness.
    # TODO: not the best solution but time is limited
    runs = 100.times.collect { Run.new.id }
 
    assert_equal 100, runs.uniq.length, "Duplicate ids found"
  end

Saving a run to Redis

Now that I have a data format in Ruby, I need to save it to Redis. Since Redis is a key value store, I decided to save my runs where the key is the id (uuid) and the data is a hash of strings. This way it would be easy to load the data back into a Ruby object.

Since the integration test uses the #save method, that’s what I need to implement first. After writing two quick tests I’m ready to figure out how to save to Redis:

  def test_save_with_valid_attributes
    run = Run.new(:date => "2012-09-28",
                  :distance => "3mi",
                  :duration => "30:00",
                  :pace => "10:00",
                  :comment => "A run")
 
    assert run.save, "Save failed"
 
    assert $db.keys.include?(run.id), "Run not found in Redis"
  end
 
  def test_save_with_mostly_empty_attributes
    run = Run.new(:date => "",
                  :distance => "",
                  :duration => "",
                  :pace => "",
                  :comment => "")
 
    assert run.save, "Save failed"
 
    assert $db.keys.include?(run.id), "Run not found in Redis"
  end

Redis uses a #set method to save data. It takes the key and value of the data and will return an “OK” when it succeeds.

One thing that tripped me up for a minute was that if I just sent a Ruby hash to Redis it would get saved but as the string interpolation of the hash like

'{"f"=>"g"}'

This wouldn’t work because I’d need to use eval to turn that back into a Ruby hash if I wanted to load it into a Run object.

So I turned to my old standby, JSON. By calling #to_json on the Ruby hash before sending it to Redis, I can be sure that I can unserialize it on the way back out. This is what Redis stored when I encoded the hash above into JSON.

'{"f":"g"}'

That figured out, the #save method became easy. Just use #set to save the JSON using a run’s uuid as the key.

  def save
    response = $db.set(id, {
                         "id" => id,
                         "date" => @date,
                         "distance" => @distance,
                         "duration" => @duration,
                         "pace" => @pace,
                         "comment" => @comment
                       }.to_json)
 
    return response == "OK"
  end

Why not use the native hash support in Redis?

Redis natively supports hashes with operations like hset, hget, and hkeys. I probably should have use them instead of JSON but I forget all about them until after I was done. Switching to a native Redis hash wouldn’t be difficult at all, because all of that logic is hidden inside two methods of the Run class. The Sinatra application itself doesn’t even know that Redis is used at all.

Loading a Ruby object out of Redis

The next step in the integration test is to load an object out of Redis and into Ruby, to be used by erb when creating the HTML page.

Mimicking ActiveRecord’s API again, I decided to name this method #find and only support finding a run by it’s uuid. The test is simple:

  1. Create a run object
  2. Save it to Redis
  3. Load the run object out of Redis
  4. Make sure its data is correct
  def test_find_existing_run
    run = Run.new(:date => "2012-09-28",
                  :distance => "3mi",
                  :duration => "30:00",
                  :pace => "10:00",
                  :comment => "A run")
 
    assert run.save, "Save failed"
 
    redis_run = Run.find(run.id)
    assert_equal run.id, redis_run.id
    assert_equal "2012-09-28", redis_run.date
    assert_equal "3mi", redis_run.distance
    assert_equal "30:00", redis_run.duration
    assert_equal "10:00", redis_run.pace
    assert_equal "A run", redis_run.comment
  end

Notice how I’m using an actual run object and its attributes this time instead of a raw hash from Redis. This is what the Run data model class is providing me.

Thinking about the #find method, it needs to do two things:

  1. Load the raw data from Redis, if it’s available
  2. Convert the raw data into a Run object

Loading the data is quite easy. Redis uses the #get method to fetch a single object, all you have to do is give it the key. If it finds an object, it is returned. If Redis doesn’t find the object, you’ll get a nil back. This makes it simple to use an if statement for control flow between these two cases.

Since the raw data is JSON, I needed to feed it through a JSON parser to convert it to a Ruby hash. This isn’t all I needed though, I wanted a Run class so I can interact with it right away. Since my earlier work created an initialize method that could set attributes based on a hash, I could easily send the data there and get a Run object back out.

In less words, here is the code:

  def self.find(uuid)
 
    record = $db.get(uuid)
 
    if record
      return new(JSON.parse(record))
    else
      return nil
    end
  end

Where are we now?

Going back up the stack to the integration test, the data layer (Run) works for initialing, saving, and loading a record. That means all I have to do is to load all of the Runs from Redis and show them on the index page.

But, I’m out of time now.

I already have an idea of how I’d go about listing the Runs, using Redis’s keys method which returns all keys and then using mget to get multiple keys in one call. After that it’s just a delete action and the HTML.

Summary

Even though I didn’t get any user interface written, I created most of the data model and had some good tests to make sure the data was behaving the way I wanted. Finishing up the application would be straight-forward and would just take some more time than I have available.

Like I said in the introduction, I have Redis running in production for Chirk HR while I test how easy it is to keep running. As of this writing, the Redis server has been online and receiving data for over 16 days and 16 hours with no downtime (probably longer, my monitoring system lost its network connection for a bit).

Given this exploration and my production test I’ve gotten very comfortable with Redis and will be adding it to my developer toolbox for future apps. I think it’s great when you need a simple and stable key/value store.

Screencast

Full code

Gemfile

source :rubygems
 
gem "sinatra", :require => "sinatra/base"
gem "shotgun"
gem "redis"
 
group :test do
  gem "minitest"
  gem "rack-test"
  gem "autotest"
  gem "autotest-notification"
end

RedisRunning Sinatra application

require 'rubygems'
require 'sinatra'
require 'securerandom'
require 'json'
 
$db = Redis.new
$db.select(1) # To get a different database
 
class Run
  attr_accessor :id
  attr_accessor :date
  attr_accessor :distance
  attr_accessor :duration
  attr_accessor :pace
  attr_accessor :comment
 
  def initialize(attributes={})
    attributes.each do |attribute, value|
      setter = "#{attribute}="
      self.send(setter, value) if self.respond_to?(setter)
    end
  end
 
  def id
    @id ||= SecureRandom.uuid
    @id
  end
 
  # Save a run to Redis
  def save
    response = $db.set(id, {
                         "id" => id,
                         "date" => @date,
                         "distance" => @distance,
                         "duration" => @duration,
                         "pace" => @pace,
                         "comment" => @comment
                       }.to_json)
 
    return response == "OK"
  end
 
  def self.find(uuid)
 
    record = $db.get(uuid)
 
    if record
      return new(JSON.parse(record))
    else
      return nil
    end
  end
end
 
 
class RedisRunning < Sinatra::Base
  get '/' do
    "Hi"
  end
end

Application tests

require 'rubygems'
require 'bundler'
 
Bundler.require
 
require './redis_running'
require 'minitest/autorun'
require 'rack/test'
 
ENV['RACK_ENV'] = 'test'
 
class RedisRunningTest < MiniTest::Unit::TestCase
  include Rack::Test::Methods
 
  def app
    RedisRunning
  end
 
  def setup
    # Select a test database and clean it
    $db.select(2)
    assert_equal "OK", $db.flushdb
    assert_equal [], $db.keys
  end
 
  def test_sanity
    assert_equal 2, 1 + 1
  end
 
  def test_get_root_response
    get '/'
 
    assert last_response.ok?
  end
 
  def test_get_root_list_runs
    run1 = Run.new(:date => "2012-09-28",
                   :distance => "3mi",
                   :duration => "30:00",
                   :pace => "10:00",
                   :comment => "A run")
    assert run1.save, "Save failed"
 
    run2 = Run.new(:date => "2012-09-29",
                   :distance => "3mi",
                   :duration => "27:00",
                   :pace => "9:00",
                   :comment => "Another run")
    assert run2.save, "Save failed"
 
    get '/'
 
    assert last_response.body.include?("A run")
    assert last_response.body.include?("Another run")
  end
 
  def test_get_root_add_form_present
    get '/'
 
    assert last_response.body.include?("form")
    assert last_response.body.include?("Add a new run")
  end
 
  def test_post_new_with_valid_run
  end
 
  def test_post_new_with_invalid_run
  end
 
  def test_get_edit_response
  end
 
  def test_get_edit_show_form
  end
 
  def test_put_update_with_valid_update
  end
 
  def test_put_update_with_invalid_update
  end
 
  def test_delete_response
  end
 
  def test_delete_should_delete_record
  end
end

Data storage tests

require 'rubygems'
require 'bundler'
 
Bundler.require
 
require './redis_running'
require 'minitest/autorun'
require 'rack/test'
 
ENV['RACK_ENV'] = 'test'
 
class RunTest < MiniTest::Unit::TestCase
  def setup
    # Select a test database and clean it
    $db.select(2)
    assert_equal "OK", $db.flushdb
    assert_equal [], $db.keys
  end
 
  def test_has_attributes
    run = Run.new
 
    ['date','distance','duration','pace','comment'].each do |attribute|
      assert run.respond_to?(attribute), "Run does not respond to #{attribute}"
      assert run.respond_to?(attribute + "="), "Run does not respond to #{attribute}="
    end
 
  end
 
  def test_should_allow_setting_attributes_on_initialize
    run = Run.new(:date => '1', :distance => '2', :duration => '3', :pace => '4', :comment => '5')
 
    assert_equal '1', run.date
    assert_equal '2', run.distance
    assert_equal '3', run.duration
    assert_equal '4', run.pace
    assert_equal '5', run.comment
  end
 
  def test_id_should_be_a_uuid
    run = Run.new
 
    assert run.id.is_a?(String)
    assert_equal 36, run.id.length
    assert_equal 4, run.id.count('-')
  end
 
  def test_id_should_be_unique
    # Lets create a bunch of runs and check their uniqueness.
    # TODO: not the best solution but time is limited
    runs = 100.times.collect { Run.new.id }
 
    assert_equal 100, runs.uniq.length, "Duplicate ids found"
  end
 
  def test_save_with_valid_attributes
    run = Run.new(:date => "2012-09-28",
                  :distance => "3mi",
                  :duration => "30:00",
                  :pace => "10:00",
                  :comment => "A run")
 
    assert run.save, "Save failed"
 
    assert $db.keys.include?(run.id), "Run not found in Redis"
  end
 
  def test_save_with_mostly_empty_attributes
    run = Run.new(:date => "",
                  :distance => "",
                  :duration => "",
                  :pace => "",
                  :comment => "")
 
    assert run.save, "Save failed"
 
    assert $db.keys.include?(run.id), "Run not found in Redis"
  end
 
  def test_find_existing_run
    run = Run.new(:date => "2012-09-28",
                  :distance => "3mi",
                  :duration => "30:00",
                  :pace => "10:00",
                  :comment => "A run")
 
    assert run.save, "Save failed"
 
    redis_run = Run.find(run.id)
    assert_equal run.id, redis_run.id
    assert_equal "2012-09-28", redis_run.date
    assert_equal "3mi", redis_run.distance
    assert_equal "30:00", redis_run.duration
    assert_equal "10:00", redis_run.pace
    assert_equal "A run", redis_run.comment
  end
 
end

Learning pjax – Tutorial and Screencast

This post is part of my weekly tech learning series, where I take one hour each week to try out a piece of technology that I’d like to learn.

This week I’m back to JavaScript, trying out the pjax library from Chris Wanstrath. pjax 1.0.0 was just released a few days ago and I’ve been itching to try it out so I took some time today to get a feel for it.

App

Since pjax is more of an infrastructure library, I decided not to try and build an actual application with it. It’s an enhancement to how webpages behave so modifying an existing application would be a better fit.

Server

For use as an example, I created a simple Sinatra app that would respond to two actions (index or goodbye). Each action uses a different template and content so I could tell the difference between the two. Super simple but enough to test out pjax.

# Non-pjax version of server
require 'rubygems'
require 'sinatra'
 
helpers do
  def title
    @title ||= "No title"
  end
end
 
get '/' do
  @title = 'Welcome'
  erb :index
end
 
get '/goodbye' do
  @title = 'Goodbye'
  erb :goodbye
end

Layout and views

The layout and the views are pretty simple. One thing to take note of because I’ll explain this later, the navigation is embedded in each view (nav element).

<!DOCTYPE html>
<!--[if lt IE 7]>      <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
<!--[if IE 7]>         <html class="no-js lt-ie9 lt-ie8"> <![endif]-->
<!--[if IE 8]>         <html class="no-js lt-ie9"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js"> <!--<![endif]-->
    <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
        <title><%= title %></title>
        <meta name="description" content="">
        <meta name="viewport" content="width=device-width">
 
        <!-- Place favicon.ico and apple-touch-icon.png in the root directory -->
 
        <link rel="stylesheet" href="css/normalize.css">
        <link rel="stylesheet" href="css/main.css">
        <link rel="stylesheet" href="css/app.css">
    </head>
    <body>
        <!-- Add your site or application content here -->
        <section id="main">
 
          <section id="content">
            <%= yield %>
          </section>
 
          <h2>Headers</h2>
          < id="headers"><%= headers.inspect %>>
          <h2>Status</h2>
          < id="status">
            <%= Time.now.to_s %>
            Randomized: <%= rand(1000) %>
          >
        </section>
 
        <!-- JS -->
        <script type="text/javascript" src="//ajax.googleapis.com/ajax/libs/jquery/1.8.1/jquery.min.js"></script>
        <script type="text/javascript" src="/js/app.js"></script>
    </body>
</html>
<%# index.erb %>
<nav>
  <a href="/"> Welcome</a>
  <a href="/goodbye">Goodbye</a>
</nav>
 
<h1>Application title</h1>
 
<p>Welcome to the application, have a nice day.</p>
<%# goodbye.erb %>
<nav>
  <a href="/"> Welcome</a>
  <a href="/goodbye">Goodbye</a>
</nav>
 
<h1>Sorry to see you go</h1>
 
<p>Goodbye</p>

Nothing too unusual. erb templates with a basic HTML5 layout. In the layout I have some debugging information to see the HTTP headers, current time, and a random number generated server side (an idea I got from Ryan Bates).

pjax time

Installing pjax is simple, just include its script file in the layout and make sure jQuery is loaded.

This is where I got confused and stalled a bit. pjax support three different ways to call it.

  1. Obtrusive by using a data-pjax attribute on links with a jQuery selector for the part of the page you want to replace (the container). e.g. <a href='/explore' data-pjax='#main'>Explore</a> will replace the main element.
  2. Semi-obtrusive by adding markup to links and then selecting them with jQuery. e.g. <a href='/explore' class='js-pjax'>Explore</a> and then selecting all of the .js-pjax elements.
  3. Unobtrusive by selecting the links directly.

There were two parts that confused me.

First, the examples of each of these covered the different ways to use pjax but they also added on additional things. So it wasn’t a direct comparison of A vs B vs C, more like A vs B+X vs C+Y where the X and Y features are optional. To help out the next person who comes along, here is an apples-to-apples comparison:

One. Functionally obtrusive, loading the href with ajax into data-pjax:

<a href='/explore' data-pjax='#main'>Explore</a>

$(document).pjax('a[data-pjax]')

Two. Slightly obtrusive, passing in a container.

<a href='/explore' class='js-pjax'>Explore</a>

$('#main').pjax('.js-pjax')

Three. Unobtrusive.

<div id='main'>
  <div class='tabs'>
    <a href='/explore'>Explore</a>
    <a href='/help'>Help</a>
  </div>
</div>

$('#main').pjax('a')

This is much clearer in my opinion and doesn’t confuse the comparisons with error handling or loading indicators.

The second confusing part is that by reading the documentation it sounds like all three of those ways are different but actually they are all the same way. Each one is selecting a container (#main or document) and then calling pjax on each element. The only differences are

  1. that if your container is anything except for the full document, then that element will be used as the target for the replacement. If the container is the document, then you must supply a replacement target in the data-pjax attribute.
  2. you can override a link’s target by using the data-pjax attribute.

So really, all you need is: $(your-replacement-target-selector).pjax(your-link-selector).

pjax with Sinatra

The nice thing about pjax is you can reuse most of the existing code on the server. pjax requests pass through the stack like any other request, except the browser replaces the content on the page instead of drawing a new page. This means you don’t need to duplicate server-side template on the client side or do any JSON encoding/decoding, which is nice.

The one change pjax recommends is to not render the layout on a pjax request. For non-Rails developers, this means you don’t want to render any of the response except for the container element (e.g. #main or #content in typically apps). This is how pjax can enhance performance:

  • you can skip layout rendering and its associated logic, and
  • the response HTML should be smaller which means less network usage and download time

The documentation for pjax shows what to change in Rails for pjax. pjax sets a “X-PJAX” HTTP header so it’s easy to check for that and toggle your layout on or off.

In my case I’m using Sinatra so the code is different than Rails, but still simple. First I created a helper method to check if this is a pjax request, is_pjax?. Then in each of my routes I changed the rendering to skip the layout for pjax requests. On a larger app I’d probably extract the layout logic to a helper method but for two actions I’m fine with some duplication.

helpers do
  # ...
 
  def is_pjax?
    #  headers['X-PJAX']
    env['HTTP_X_PJAX']
  end
 
end
 
get '/' do
  @title = 'Welcome'
  erb :index, :layout => !is_pjax?
end
 
get '/goodbye' do
  @title = 'Goodbye'
  erb :goodbye, :layout => !is_pjax?
end

From what I saw, you can use either Sintra’s headers method (commented out) or Rack’s env method. The beauty of Rack systems…

So now the server is configured for pjax, there are views that are working with regular request/response cycles, and jQuery and pjax are included on the page. Time to configure pjax itself.

Configuring pjax

The unobtrusive option seems the best for me so I went ahead and used the following JavaScript to configure pjax. #content is the container and the pjax enabled links are the nav links. pjax is automatically picking up the href and using that for where to get the data.

  $('#content').pjax('nav a')

Pretty simple right?

More complex pjax example

Since pjax is just using css selectors you can get pretty technical with it too. Given the HTML below:

<section id="content">
  <div id="magic">
    <a href="#">Naked link</a>
    <a href="#" class="js-pjax">Class link</a>
    <a href="#" id="id">Id link</a>
    <p>Some content here with an <a href="#">internal Naked link</a></p>
  </div>
</section>
  • To pjax everything: $("#content").pjax('a');
  • To pjax just the class link: $("#content").pjax('.js-pjax');
  • To pjax just the id link: $("#content").pjax('#id');
  • To pjax all of the link except for the internal naked link: $("#content").pjax('#magic > a');
  • To pjax just the two naked links: $("#content").pjax('a:contains(Naked)');
  • etc, etc, you get the point

From what I understand, you can also call pjax() on multiple elements so don’t feel like you need to build a complex selector to get everything at once.

Boxes, little boxes (what stalled me for a bit)

There was one gotcha that tripped me up for a bit until I understood how pjax worked. When using an element for the container (#content), only the elements inside of the container are used in the pjax selector. Lets modify the example from earlier, say you have a global navigation you want to use pjax with too.

<nav>
  <a href="/"> Welcome</a>
  <a href="/goodbye">Goodbye</a>
</nav>
 
<section id="content">
  <div id="magic">
    <a href="#">Naked link</a>
    <a href="#" class="js-pjax">Class link</a>
    <a href="#" id="id">Id link</a>
    <p>Some content here with an <a href="#">internal Naked link</a></p>
  </div>
</section>

In this case $("#content").pjax('a'); will only get you the links in the content, even though the selector is ‘a’, because pjax is scoping its find to the container.

This is why I had to put the navigation inside of each view. If it was outside of the replacement target then pjax wouldn’t find it.

Summary

pjax is a really good library. It feels a lot like convention over configuration. You link pages with links, so pjax will just use those hrefs you already have to ask for the ajax version. The majority of sites use a main element for the content, so you configure that container once and pjax replaces your content in there unless you override it.

Perhaps the part I like the most about pjax is that if it fails for whatever reason (e.g. server error, JavaScript error, network error) it will fallback to the browser’s default behavior which is to follow the link. This kind of error handling means that there is very little risk to using pjax in production, if fails and you are no worse off than you are now.

I’m definitely going to add pjax to my toolbox. What I might do is a blend of the default and data-pjax markups:

  • Use the default pjax markup for links inside a container: $('#content').pjax('a'); and then
  • Use the data-pjax markup for links outside of a container like a global navigation: $(document).pjax('a[data-pjax]') and set the data-pjax attribute to “#content”

Screencast

Now watch the screencast below to see a walk through of the code and see how pjax works in the browser.

(View it in full screen on YouTube)

Learning EventMachine

This post is part of my weekly tech learning series, where I take one hour each week to try out a piece of technology that I’d like to learn.

This week I decided to try out EventMachine: a fast, simple event-processing library for Ruby.

I have to say, I was pretty impressed. Both by its documentation and also how easy it was to get started.

Running Log

I decided to stick with my running log idea from last week. It’s a simple app idea that is different enough from the standard todo list examples used everywhere.

Getting started

Right away I found links to EventMachine’s wiki which was filled with an introduction and some code snippets. I did get stuck for a bit, but that is completely my fault and my own personal bias towards reading code instead of a description of that code.

Simple Prototype

To prevent getting stuck like last week with backbone.js I decided to try a quick prototype before I started on the app. That way there would be less code to debug if there was a problem.

But there was a problem.

I was trying to build a simple server that would accept input and return it reversed (e.g. “Hello” turns into “olleH”). I copied the first example from the code snippets, started the server, and tried to telnet to it:

$ telnet 127.0.0.1 8081
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused

Uh oh. I checked the server, it was still running, no exceptions raised. I tried telnet again, still nothing.

That’s when I pulled out my old sysadmin knowledge and ran `netstat -luntp`:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:17500           0.0.0.0:*               LISTEN      10838/dropbox
tcp        0      0 0.0.0.0:24800           0.0.0.0:*               LISTEN      20984/synergys
tcp        0      0 0.0.0.0:60198           0.0.0.0:*               LISTEN      8382/skype
# Truncated a bit

This command lists every process that is listening on a TCP or UDP socket, its interface address (localhost or ip addresses), as well as the process id and program name.

That’s when I noticed that my server wasn’t even listening on a port.

Going back to my code I started inspecting line by line to see what was wrong. I even tried to copy some other example EventMachine servers and ran them (most of them worked).

Finally I found my problem.

I used the first example from the Code Snippets which is under the Client Example.

After cursing I then realized that my intuition was correct at the beginning when I though EventMachine.connect was an odd method name for starting a server.

Lesson #1: Read ALL of the documentation, not just the code.

Lesson #2: If an API seems confusing, double check that you are not reading it incorrectly.

So one change from #connect to #start_server and my server booted, showed up in netstat, and I could telnet to it.

#!/usr/bin/env ruby
require 'rubygems'
require 'eventmachine'
 
class Reverser < EventMachine::Connection
  def post_init
    send_data ">> Ready\n"
  end
 
  def receive_data(data)
    send_data data.strip.reverse + "\n"
  end
 
end
 
EventMachine.run {
  EventMachine.start_server '127.0.0.1', 8081, Reverser
  puts "Running on port 8081"
}

Building the Running Log

Now that I’ve prototyped and got EventMachine running completely it was time to start on my Running Log app. I knew I wouldn’t have enough time to implement a complete CRUD type of system so I settled on the ability to add a run and then list all of the runs that have been added (the CR__ in CRUD).

Tests

Since I was working in Ruby, I also decided to use TDD to make sure I don’t misstep along the way. I’m running 1.9.3 so using minitest was the perfect fit. I also didn’t want to mess with multiple files so I embedded the test directly into the main server file. Not only will this keep the implementation and test in one place but it also serves as some basic internal documentation.

The only problem was that I usually used the __FILE__ == $0 trick to embed tests but that wouldn’t work with EventMachine because I was executing the file directly.

[Sidebar]

__FILE__ == $0 checks if the file is getting executed directly, such as ruby your_file.rb. The technique is to code your library like normal and wrap your tests in that check. So running the file directly will run the tests while using require 'your_file' will load the library without the tests.

class YourClass
end
 
if __FILE__ == $0
  # Your testing code
end

[End sidebar]

Instead I just decided to uses ARGVs so the server starts when the first argument is “start”, otherwise to run the test suite.

Since EventMachine lets you separate your implementation logic into a module, I was able to easily include that module in a test class and not have to worry about testing much of EventMachine itself. This made testing cycle a lot faster.

Running Log Methods

Since I was only concerned with two public APIs for the server I was able to keep the implementation pretty slim:

  • #receive_data is the main method EventMachine uses when it gets input. I used a classic condition in here to see which API the client was calling.
  • #list_runs is used to run a simple text list of the runs the server has saved.
  • #add_run_from_user is used to create a new run. I didn’t want to spend much time with a data format so I went with a simple line based CSV format. Making it line based makes it easy to enter into a telnet client.

You can see the implementations below, #add_run_from_user is the most complex because I’m doing the data parsing and validation inline. In actual production code I’d probably split it up to be clearer.

All in all there was about 50 lines of implementation code and close to 100 lines of test code which covered some of the common use cases.

Screencast

Below is a screencast of me walking through the code and how the app works.

Summary

Working with EventMachine was quite fun. I know I’m not really using it to it’s full power but it’s nice to know that there wouldn’t be that much else I’d have to do to scale it up in production.

My favorite part was how well it separated the network server part from the application code. You can see in my test, the RunLogWrapper class is not using EventMachine but manages to pull in my entire implementation for testing.

I could see EventMachine being used for running web services and acting as glue servers for other systems. I’m definitely happy I spent I hour to play with EventMachine and learn the basics.

Code

Here is the full code for the server along with the embedded tests.

#!/usr/bin/env ruby
require 'rubygems'
require 'eventmachine'
 
module RunLog
  def receive_data(data)
    if data.match(/list/i)
      list_runs
    else
      add_run_from_user(data)
    end
  end
 
  def list_runs
    output = runs.inject("") do |o, run|
      o += "Date: #{run[:date]} | Distance: #{run[:distance]} | Duration: #{run[:duration]} | Pace: #{run[:pace]} | Comment: #{run[:comment]}\n"
      o
    end
    send_data output
  end
 
  def add_run_from_user(data)
    run_data = {
      :date => "",
      :distance => "",
      :duration => "",
      :pace => "",
      :comment => ""
    }
    date, distance, duration, pace, comment = data.split(',')
    run_data[:date] = date.strip if date
    run_data[:distance] = distance.strip if distance
    run_data[:duration] = duration.strip if duration
    run_data[:pace] = pace.strip if pace
    run_data[:comment] = comment.strip if comment
 
    if run_data.any? {|key, value| !value.nil? && value != ""}
      add_run(run_data)
      send_data "OK\n"
    else
      send_data "INPUT ERROR\n"
    end
 
  end
 
  def runs
    @runs || []
  end
 
  def add_run(run)
    @runs ||= []
    @runs << run
  end
end
 
 
 
command = ARGV.shift
 
case command
when "start"
  EventMachine.run {
    EventMachine.start_server '127.0.0.1', 8081, RunLog
    puts "Running on port 8081"
  }
else
  require 'minitest/autorun'
 
  class RunLogWrapper
    include RunLog
 
    attr_reader :output_buffer
 
    # Used to stub EM's methods
    def send_data(*args)
      @output_buffer ||= ""
      @output_buffer << args.join("\n")
    end
 
  end
 
  class TestRunLog < MiniTest::Unit::TestCase
    def setup
      @runlog = RunLogWrapper.new
    end
 
    def test_sanity
      assert_equal 4, 2 + 2
    end
 
    def test_receive_data_with_list_command
      @runlog.add_run({
        :date => "2012-09-14",
        :distance => "3mi",
        :duration => "30:00",
        :pace => "10:00",
        :comment => "Nice and easy run"
      })
 
      @runlog.receive_data("list")
 
      assert_equal "Date: 2012-09-14 | Distance: 3mi | Duration: 30:00 | Pace: 10:00 | Comment: Nice and easy run\n", @runlog.output_buffer
    end
 
    def test_receive_data_with_good_input
      @runlog.receive_data("2012-09-14, 3mi, 30:00, 10:00, Nice and easy run")
 
      assert_equal 1, @runlog.runs.length
      expected_run = {
        :date => "2012-09-14",
        :distance => "3mi",
        :duration => "30:00",
        :pace => "10:00",
        :comment => "Nice and easy run"
      }
      assert_equal expected_run, @runlog.runs.first
 
    end
 
    def test_receive_data_with_empty_input
      @runlog.receive_data("\n")
 
      assert_equal 0, @runlog.runs.length
    end
 
    def test_receive_data_with_missing_fields
      @runlog.receive_data("2012-09-14, 3mi, 30:00")
 
      assert_equal 1, @runlog.runs.length
      expected_run = {
        :date => "2012-09-14",
        :distance => "3mi",
        :duration => "30:00",
        :pace => "",
        :comment => ""
      }
      assert_equal expected_run, @runlog.runs.first
 
    end
 
    def test_receive_data_with_malformed_input
      @runlog.receive_data("This input, isn't quite, right, but, it still is accepted, for now")
 
      assert_equal 1, @runlog.runs.length
      expected_run = {
        :date => "This input",
        :distance => "isn't quite",
        :duration => "right",
        :pace => "but",
        :comment => "it still is accepted"
      }
      assert_equal expected_run, @runlog.runs.first
 
    end
 
  end
 
end

Fog – Automating Cloud Servers

This post is part of my weekly tech learning series, where I take one hour each week to try out a piece of technology that I’d like to learn.

Today I decided to try something new. Since I’ve been working with JavaScript libraries for the past few weeks, I wanted to go in a completely different direction.

Fog

Today I decided to try out the fog Ruby gem. fog is a gem that abstracts about a dozen cloud services into a common API.

Since I mostly use VPSs for myself and my clients, I wanted to explore Amazon’s EC2 and the Rackspace Cloud. Both of these are covered under fog’s Compute section. My goal was to configure and boot an Ubuntu LTS server on each of these services.

(I’d also like to take a look at fog’s Storage section later. I use Amazon s3 for much of my hosting and having programmatic access to it would be useful).

Rackspace Cloud

Since I already have an account with Rackspace I figured I’d start there. fog’s tutorial included Rackspace so I walked through it to setup a new default server (Ubuntu 10.04 with 256MB of RAM). Changing the RAM looks easy as it’s configured by the flavor_id attribute. The OS can be changed by the image_id attribute but I couldn’t find an easy way to list all of the flavors. I’d prefer an Ubuntu 12.04 instance but for times sake I’ll just leave it as 10.04 for now.

One thing that wasn’t mentioned in the documentation is that fog is now using a ~/.fog file to store your credentials. That will let you enter everything there and keep authentication and keys out of your source code. From what I saw you can also configure it to use a different file too so you could have project-specific configurations, great for freelance Ruby developers.

Unfortunately I ran into a few errors trying to use Rackspace. First the bootstrap command used in the tutorial needed some additional parameters, specifically the :public_key_path and private_key_path. Otherwise you’ll get an error like:

“ArgumentError: public_key is required for this operation”

After adding the keys, I ran bootstrap again. This command is supposed to create a new server, boot it up, and configure ssh access by copying your ssh keys.

Unfortunately I ran into another error, this time it was “Net::SSH::Disconnect: disconnected: Too many authentication failures for root (2)”. This meant that Net::SSH couldn’t ssh into the server; so either the user, the password, or ssh keys were wrong. I tried switching the user to ‘ubuntu’ since Ubuntu doesn’t use the root user but that didn’t work either.

I suspect this problem might be with Net::SSH and my ssh keys. I create new keys for each client I work with and have a few old ones left around. Looking at my ~/.ssh/ directory I see 16 keys in there.

Turning off public key authentication for the server I was able to ssh in manually so I’m 80.111111% sure that this is a problem with my laptop.

Since the server was created, I decided to move onto EC2. I’ll be going back and debugging my ssh configuration later on, but I’d like to have a little bit of experience with EC2 before I run out of time.

AWS EC2

EC2 was new to me. I’ve used S3 for years but never had the need for EC2 computing yet. After configuring fog for Rackspace it was just a few changes in my code to switch over to EC2.

The default image is still Ubuntu 10.04 but with 613MB of RAM (t1.micro) instead, the smallest size EC2 allows.

I didn’t have any problems with EC2 once I got my SSH keys figured out.

AWS doesn’t support DSA keypairs

Summary

I was impressed by how well fog abstracted the differences between EC2 and Rackspace. The API is generic enough to do some scripting with, though I’d be careful doing a lot of automation around it otherwise you might end up with 100s of servers sitting around.

I don’t see myself using fog for my own servers anytime soon because I mostly work with 1-2 servers at a time and don’t need to scale that often. The largest configuration I’ve done was a four node web cluster: 1 proxy, 2 app servers, 1 database server on the Rackspace cloud.

There are a few places I can see fog helping me with though:

  • Storing files to S3, like from a command line or a bulk upload.
  • Synchronizing files between S3 and Rackspace Cloud Files.
  • Automating the creation of a temporary server, like for doing remote pair programming. This kind of server would be started, used, and then destroyed a few hours later.

I’ve even been toying around with the idea of doing all of my development on a remote server and just using my laptop (or even my ipad) as a dumb terminal. The cost would be minimal and I’d be able to use a larger server if I needed more power (e.g. a 30GB Rackspace server is only $1.20/hour, less than $10/day for an 8 hour workday).

Code

There isn’t much code to show for fog, especially since I did a lot of exploration using its console mode. Here are the server creation scripts I did use.

# rackspace_server.rb
require 'rubygems'
require 'fog'
 
connection = Fog::Compute.new({:provider => 'Rackspace'})
 
server = connection.servers.bootstrap(:public_key_path => '/home/edavis/.ssh/fog.pub',
                                      :private_key_path => '/home/edavis/.ssh/fog',
                                      :username => 'ubuntu')
# ec2_server.rb
require 'rubygems'
require 'fog'
 
 
connection = Fog::Compute.new({
                                :provider => 'AWS',
                                :aws_access_key_id => ENV['S3_ACCESS_KEY'],
                                :aws_secret_access_key => ENV['S3_SECRET_ACCESS_KEY']
                              })
 
server = connection.servers.bootstrap(:private_key_path => '~/.ssh/fog',
                                      :public_key_path => '~/.ssh/fog.pub',
                                      :username => 'ubuntu')

Like I said, pretty much identical.

Next Friday I’m planning to explore EventMachine and get my ruby event-processing on.

Weekly Tech Learning

Most web developers know that the tech industry moves at an accelerated pace, with hundreds (thousands?) of new technologies to try out that all promise to make our jobs [easier|faster|more efficient|stronger|more shiny].

The problem is time.

How do you know which technology is worth your time to learn? Which ones will actually deliver and make actual improvements?

I've spent the last seven years (2005) working in Ruby and Ruby on Rails as my primary development environment. I've used a smattering of different technologies but my default stack has changed very little over the years (I'd like to think that I've grown into the stack and am using it better than when I started).

Occasionally I'd try out something new which would have a huge payoff for me, like jQuery or Sinatra. Many other times the “new thing” wouldn't work or would only pay off in specific cases that I don't encounter. Even if there isn't an immediate benefit, I still need to make sure to set aside time to learn new tech.

To formalize this a bit and prevent it from getting put off for “later”, I've decided to follow what David Eisinger did. He picked 30 pieces of technology, spent one hour per day, and tried to actually use them. I'm going to do it a bit differently and do only one per week (maybe two) but I'd like to continue doing it over several months.

I've been asking around on Twitter and have a rough idea of the technologies I want to try out. If you have another, suggest it in the comments. I'm open to mostly anything; programming language, framework, library/plugin, etc.

  1. MongoDB
  2. RedisLearning Redis
  3. Resque
  4. Datamapper
  5. Riak
  6. Cassandra
  7. CoffeeScriptLearning CoffeeScript (with a dash of Node.js)
  8. Knockout.jsLearning Knockout.js
  9. Angular.js
  10. Backbone.jsLearning (and Failing at) Backbone.js
  11. Ember.js
  12. riot.jsriot.js, minimal MVP – Weekly Tech Learning
  13. Spine.js
  14. batman.js
  15. phantom.js
  16. casper.js
  17. Web sockets
  18. pjaxLearning pjax
  19. Socket.io
  20. Canvas element
  21. HTML5 geolocation
  22. CSS3 animations
  23. fog gem for EC2 and Rackspace cloud servers – Fog – Automating Cloud Servers
  24. VIM
  25. Padrino
  26. Node.jsLearning Node.js
  27. express (framework for node.js)
  28. tower.js (framework for node and express)
  29. flatiron.js (framework for node)
  30. derby.js
  31. meteor.js
  32. Iridium (tutorial) (JavaScript toolchain)
  33. handlebars.js
  34. RSpec 2
  35. EventMachineLearning EventMachine
  36. Goliath
  37. C
  38. Objective-C
  39. iOS development
  40. Phone Gap
  41. Lua
  42. Io
  43. Selenium
  44. Erlang
  45. spooky (Sinatra-like framework for Erlang)
  46. python
  47. django
  48. mongrel2
  49. Elastic Search
  50. WordPress plugins
  51. Compass
  52. MiddleMan (Static site generator)
  53. Renee
  54. fallen (daemon library for Ruby )
  55. knockback
  56. fnordmetric (Railscast)
  57. tmux
  58. R
  59. GoGo – Tech Learning
  60. QUnit – Tech Learning

I haven't completely thought out how this will work but at the very least I'll try to write up a summary of my experience with each technology here.