Learning Redis

This post is part of my weekly tech learning series, where I take one hour each week to try out a piece of technology that I’d like to learn. Make sure to read to the end, where I have a screencast overview of the final application.

This week I decided to learn a bit more about Redis. I’ve used Redis in Chirk HR already but only as part of a non-critical service to see how it performs under load (I’d like to screencast this later). Today I wanted to use Redis as an actual data store but first, what is Redis?

What is Redis?

Redis is an open source key-value store. I think of it as a light nosql database with a lot of flexibility.

Today’s application

My idea for today was to build my running journal but this time as a CRUD style web app. I want to stay close to the metal so I can really see how Redis works so I picked my second favorite web framework: Sinatra.

Basically I wanted to mimic a Rails scaffold inside Sinatra using Redis as the data storage. I’m also using the redis gem because it is very similar to the actual Redis API. This means I’ll be implementing the data layer myself, without an ORM.

(Normally I’d use ActiveRecord or Datamapper for production applications instance but I’m more concerned with learning and exploration with this.)

Since I will be doing a lot of data layer work I’m also opting for heavy TDD with this app, using minitest and autotest.

Here is the Gemfile I ended up with:

source :rubygems
 
gem "sinatra", :require => "sinatra/base"
gem "shotgun"
gem "redis"
 
group :test do
  gem "minitest"
  gem "rack-test"
  gem "autotest"
  gem "autotest-notification"
end

Use TDD for the application behavior

After getting everything hooked up I decided to split the tests into two files:

  1. test_redis_running.rb to test the RedisRunning Sinatra application at the integration layer. This would be the controller and view if you’re thinking about it as a MVC app.
  2. test_run.rb to test the Run class, which is used as the data layer between the application and Redis.

I like to start with integration testing and write pending tests for each major workflow or feature. I came up with the following integration tests to get started with:

  • test_get_root_response
  • test_get_root_list_runs
  • test_get_root_add_form_present
  • test_post_new_with_valid_run
  • test_post_new_with_invalid_run
  • test_get_edit_response
  • test_get_edit_show_form
  • test_put_update_with_valid_update
  • test_put_update_with_invalid_update
  • test_delete_response
  • test_delete_should_delete_record

This is very similar to Rails scaffold except I decided to put the new form on the list of runs (:index) and to skip the details of a run (:show). With such a small data model, a list provides a perfect summary.

The first test I wrote was to check that getting the index was successful, which isn’t about Redis at all so I won’t bore you with the details of it.

The second test on the other hand (test_get_root_list_runs), is the important one (and as you’ll see later is all that I had time for). Here I wanted to do a few things:

  1. Create a few runs and save them to Redis
  2. Load the index page
  3. Check that the runs were displayed

In order to implement this test I have to do a full cycle of saving and loading runs from Redis. This meant I’d need to build the data layer now.

Here is the full integration test.

  def test_get_root_list_runs
    run1 = Run.new(:date => "2012-09-28",
                   :distance => "3mi",
                   :duration => "30:00",
                   :pace => "10:00",
                   :comment => "A run")
    assert run1.save, "Save failed"
 
    run2 = Run.new(:date => "2012-09-29",
                   :distance => "3mi",
                   :duration => "27:00",
                   :pace => "9:00",
                   :comment => "Another run")
    assert run2.save, "Save failed"
 
    get '/'
 
    assert last_response.body.include?("A run")
    assert last_response.body.include?("Another run")
  end

RedisRunning Data Layer

The integration test needed three things from my Run class:

  1. Create an object with some attributes
  2. Save the object to Redis
  3. Load the saved object out of Redis

I started with the first, using standard Ruby accessors.

Run accessors and attributes

A run has five attributes that a user should be able to set:

  1. date
  2. distance
  3. duration
  4. pace
  5. comment

Additionally, I’m going to need some way to uniquely identify each run. In Postgres I’d just use an id field as a primary key but from what I see, Redis doesn’t have primary keys. Only keys.

At first I thought of using the date as the primary key, but many runners will run multiple times per day so that won’t work. I could use a combination of the date and another field like distance or duration but there is also a chance those will conflict also (e.g. two runs per day, around a 1 mile track at a specific pace).

Redis doesn’t have primary keys so you might need to roll your own.

So I’ll need to be creating my own unique ids for each run. I’ve used UUIDs before and they seem like a good fit here.

So using attr_accessor and initialize I can easily fulfill these requirements.

The test:

class RunTest < MiniTest::Unit::TestCase
  def test_has_attributes
    run = Run.new
 
    ['date','distance','duration','pace','comment'].each do |attribute|
      assert run.respond_to?(attribute), "Run does not respond to #{attribute}"
      assert run.respond_to?(attribute + "="), "Run does not respond to #{attribute}="
    end
 
  end
 
  def test_should_allow_setting_attributes_on_initialize
    run = Run.new(:date => '1', :distance => '2', :duration => '3', :pace => '4', :comment => '5')
 
    assert_equal '1', run.date
    assert_equal '2', run.distance
    assert_equal '3', run.duration
    assert_equal '4', run.pace
    assert_equal '5', run.comment
  end
end

The implementation:

class Run
  attr_accessor :id
  attr_accessor :date
  attr_accessor :distance
  attr_accessor :duration
  attr_accessor :pace
  attr_accessor :comment
 
  def initialize(attributes={})
    attributes.each do |attribute, value|
      setter = "#{attribute}="
      self.send(setter, value) if self.respond_to?(setter)
    end
  end
end

Generating unique ids with UUID

I’ve used some libraries to generate UUIDs in Ruby but I just found out that Ruby 1.9.3 now has UUID support in the standard library as part of SecureRandom. This makes the UUID code simple.

The test.

  def test_id_should_be_a_uuid
    run = Run.new
 
    assert run.id.is_a?(String)
    assert_equal 36, run.id.length
    assert_equal 4, run.id.count('-')
  end

The implementation.

# Way up top outside the Run class...
require 'securerandom'
 
# In the Run class...
  def id
    @id ||= SecureRandom.uuid
    @id
  end

By using ||= I can make sure that if a Run already has an id then I won’t generate it again (existing record).

I also wrote a brute force test to make sure Ruby’s UUIDs were unique and that I wasn’t misreading the docs. It’s not really production quality but good for exploration of a new API:

  def test_id_should_be_unique
    # Lets create a bunch of runs and check their uniqueness.
    # TODO: not the best solution but time is limited
    runs = 100.times.collect { Run.new.id }
 
    assert_equal 100, runs.uniq.length, "Duplicate ids found"
  end

Saving a run to Redis

Now that I have a data format in Ruby, I need to save it to Redis. Since Redis is a key value store, I decided to save my runs where the key is the id (uuid) and the data is a hash of strings. This way it would be easy to load the data back into a Ruby object.

Since the integration test uses the #save method, that’s what I need to implement first. After writing two quick tests I’m ready to figure out how to save to Redis:

  def test_save_with_valid_attributes
    run = Run.new(:date => "2012-09-28",
                  :distance => "3mi",
                  :duration => "30:00",
                  :pace => "10:00",
                  :comment => "A run")
 
    assert run.save, "Save failed"
 
    assert $db.keys.include?(run.id), "Run not found in Redis"
  end
 
  def test_save_with_mostly_empty_attributes
    run = Run.new(:date => "",
                  :distance => "",
                  :duration => "",
                  :pace => "",
                  :comment => "")
 
    assert run.save, "Save failed"
 
    assert $db.keys.include?(run.id), "Run not found in Redis"
  end

Redis uses a #set method to save data. It takes the key and value of the data and will return an “OK” when it succeeds.

One thing that tripped me up for a minute was that if I just sent a Ruby hash to Redis it would get saved but as the string interpolation of the hash like

'{"f"=>"g"}'

This wouldn’t work because I’d need to use eval to turn that back into a Ruby hash if I wanted to load it into a Run object.

So I turned to my old standby, JSON. By calling #to_json on the Ruby hash before sending it to Redis, I can be sure that I can unserialize it on the way back out. This is what Redis stored when I encoded the hash above into JSON.

'{"f":"g"}'

That figured out, the #save method became easy. Just use #set to save the JSON using a run’s uuid as the key.

  def save
    response = $db.set(id, {
                         "id" => id,
                         "date" => @date,
                         "distance" => @distance,
                         "duration" => @duration,
                         "pace" => @pace,
                         "comment" => @comment
                       }.to_json)
 
    return response == "OK"
  end

Why not use the native hash support in Redis?

Redis natively supports hashes with operations like hset, hget, and hkeys. I probably should have use them instead of JSON but I forget all about them until after I was done. Switching to a native Redis hash wouldn’t be difficult at all, because all of that logic is hidden inside two methods of the Run class. The Sinatra application itself doesn’t even know that Redis is used at all.

Loading a Ruby object out of Redis

The next step in the integration test is to load an object out of Redis and into Ruby, to be used by erb when creating the HTML page.

Mimicking ActiveRecord’s API again, I decided to name this method #find and only support finding a run by it’s uuid. The test is simple:

  1. Create a run object
  2. Save it to Redis
  3. Load the run object out of Redis
  4. Make sure its data is correct
  def test_find_existing_run
    run = Run.new(:date => "2012-09-28",
                  :distance => "3mi",
                  :duration => "30:00",
                  :pace => "10:00",
                  :comment => "A run")
 
    assert run.save, "Save failed"
 
    redis_run = Run.find(run.id)
    assert_equal run.id, redis_run.id
    assert_equal "2012-09-28", redis_run.date
    assert_equal "3mi", redis_run.distance
    assert_equal "30:00", redis_run.duration
    assert_equal "10:00", redis_run.pace
    assert_equal "A run", redis_run.comment
  end

Notice how I’m using an actual run object and its attributes this time instead of a raw hash from Redis. This is what the Run data model class is providing me.

Thinking about the #find method, it needs to do two things:

  1. Load the raw data from Redis, if it’s available
  2. Convert the raw data into a Run object

Loading the data is quite easy. Redis uses the #get method to fetch a single object, all you have to do is give it the key. If it finds an object, it is returned. If Redis doesn’t find the object, you’ll get a nil back. This makes it simple to use an if statement for control flow between these two cases.

Since the raw data is JSON, I needed to feed it through a JSON parser to convert it to a Ruby hash. This isn’t all I needed though, I wanted a Run class so I can interact with it right away. Since my earlier work created an initialize method that could set attributes based on a hash, I could easily send the data there and get a Run object back out.

In less words, here is the code:

  def self.find(uuid)
 
    record = $db.get(uuid)
 
    if record
      return new(JSON.parse(record))
    else
      return nil
    end
  end

Where are we now?

Going back up the stack to the integration test, the data layer (Run) works for initialing, saving, and loading a record. That means all I have to do is to load all of the Runs from Redis and show them on the index page.

But, I’m out of time now.

I already have an idea of how I’d go about listing the Runs, using Redis’s keys method which returns all keys and then using mget to get multiple keys in one call. After that it’s just a delete action and the HTML.

Summary

Even though I didn’t get any user interface written, I created most of the data model and had some good tests to make sure the data was behaving the way I wanted. Finishing up the application would be straight-forward and would just take some more time than I have available.

Like I said in the introduction, I have Redis running in production for Chirk HR while I test how easy it is to keep running. As of this writing, the Redis server has been online and receiving data for over 16 days and 16 hours with no downtime (probably longer, my monitoring system lost its network connection for a bit).

Given this exploration and my production test I’ve gotten very comfortable with Redis and will be adding it to my developer toolbox for future apps. I think it’s great when you need a simple and stable key/value store.

Screencast

Full code

Gemfile

source :rubygems
 
gem "sinatra", :require => "sinatra/base"
gem "shotgun"
gem "redis"
 
group :test do
  gem "minitest"
  gem "rack-test"
  gem "autotest"
  gem "autotest-notification"
end

RedisRunning Sinatra application

require 'rubygems'
require 'sinatra'
require 'securerandom'
require 'json'
 
$db = Redis.new
$db.select(1) # To get a different database
 
class Run
  attr_accessor :id
  attr_accessor :date
  attr_accessor :distance
  attr_accessor :duration
  attr_accessor :pace
  attr_accessor :comment
 
  def initialize(attributes={})
    attributes.each do |attribute, value|
      setter = "#{attribute}="
      self.send(setter, value) if self.respond_to?(setter)
    end
  end
 
  def id
    @id ||= SecureRandom.uuid
    @id
  end
 
  # Save a run to Redis
  def save
    response = $db.set(id, {
                         "id" => id,
                         "date" => @date,
                         "distance" => @distance,
                         "duration" => @duration,
                         "pace" => @pace,
                         "comment" => @comment
                       }.to_json)
 
    return response == "OK"
  end
 
  def self.find(uuid)
 
    record = $db.get(uuid)
 
    if record
      return new(JSON.parse(record))
    else
      return nil
    end
  end
end
 
 
class RedisRunning < Sinatra::Base
  get '/' do
    "Hi"
  end
end

Application tests

require 'rubygems'
require 'bundler'
 
Bundler.require
 
require './redis_running'
require 'minitest/autorun'
require 'rack/test'
 
ENV['RACK_ENV'] = 'test'
 
class RedisRunningTest < MiniTest::Unit::TestCase
  include Rack::Test::Methods
 
  def app
    RedisRunning
  end
 
  def setup
    # Select a test database and clean it
    $db.select(2)
    assert_equal "OK", $db.flushdb
    assert_equal [], $db.keys
  end
 
  def test_sanity
    assert_equal 2, 1 + 1
  end
 
  def test_get_root_response
    get '/'
 
    assert last_response.ok?
  end
 
  def test_get_root_list_runs
    run1 = Run.new(:date => "2012-09-28",
                   :distance => "3mi",
                   :duration => "30:00",
                   :pace => "10:00",
                   :comment => "A run")
    assert run1.save, "Save failed"
 
    run2 = Run.new(:date => "2012-09-29",
                   :distance => "3mi",
                   :duration => "27:00",
                   :pace => "9:00",
                   :comment => "Another run")
    assert run2.save, "Save failed"
 
    get '/'
 
    assert last_response.body.include?("A run")
    assert last_response.body.include?("Another run")
  end
 
  def test_get_root_add_form_present
    get '/'
 
    assert last_response.body.include?("form")
    assert last_response.body.include?("Add a new run")
  end
 
  def test_post_new_with_valid_run
  end
 
  def test_post_new_with_invalid_run
  end
 
  def test_get_edit_response
  end
 
  def test_get_edit_show_form
  end
 
  def test_put_update_with_valid_update
  end
 
  def test_put_update_with_invalid_update
  end
 
  def test_delete_response
  end
 
  def test_delete_should_delete_record
  end
end

Data storage tests

require 'rubygems'
require 'bundler'
 
Bundler.require
 
require './redis_running'
require 'minitest/autorun'
require 'rack/test'
 
ENV['RACK_ENV'] = 'test'
 
class RunTest < MiniTest::Unit::TestCase
  def setup
    # Select a test database and clean it
    $db.select(2)
    assert_equal "OK", $db.flushdb
    assert_equal [], $db.keys
  end
 
  def test_has_attributes
    run = Run.new
 
    ['date','distance','duration','pace','comment'].each do |attribute|
      assert run.respond_to?(attribute), "Run does not respond to #{attribute}"
      assert run.respond_to?(attribute + "="), "Run does not respond to #{attribute}="
    end
 
  end
 
  def test_should_allow_setting_attributes_on_initialize
    run = Run.new(:date => '1', :distance => '2', :duration => '3', :pace => '4', :comment => '5')
 
    assert_equal '1', run.date
    assert_equal '2', run.distance
    assert_equal '3', run.duration
    assert_equal '4', run.pace
    assert_equal '5', run.comment
  end
 
  def test_id_should_be_a_uuid
    run = Run.new
 
    assert run.id.is_a?(String)
    assert_equal 36, run.id.length
    assert_equal 4, run.id.count('-')
  end
 
  def test_id_should_be_unique
    # Lets create a bunch of runs and check their uniqueness.
    # TODO: not the best solution but time is limited
    runs = 100.times.collect { Run.new.id }
 
    assert_equal 100, runs.uniq.length, "Duplicate ids found"
  end
 
  def test_save_with_valid_attributes
    run = Run.new(:date => "2012-09-28",
                  :distance => "3mi",
                  :duration => "30:00",
                  :pace => "10:00",
                  :comment => "A run")
 
    assert run.save, "Save failed"
 
    assert $db.keys.include?(run.id), "Run not found in Redis"
  end
 
  def test_save_with_mostly_empty_attributes
    run = Run.new(:date => "",
                  :distance => "",
                  :duration => "",
                  :pace => "",
                  :comment => "")
 
    assert run.save, "Save failed"
 
    assert $db.keys.include?(run.id), "Run not found in Redis"
  end
 
  def test_find_existing_run
    run = Run.new(:date => "2012-09-28",
                  :distance => "3mi",
                  :duration => "30:00",
                  :pace => "10:00",
                  :comment => "A run")
 
    assert run.save, "Save failed"
 
    redis_run = Run.find(run.id)
    assert_equal run.id, redis_run.id
    assert_equal "2012-09-28", redis_run.date
    assert_equal "3mi", redis_run.distance
    assert_equal "30:00", redis_run.duration
    assert_equal "10:00", redis_run.pace
    assert_equal "A run", redis_run.comment
  end
 
end

One thought on “Learning Redis”

Comments are closed.