caboose

I sat down with David Chelimsky at Rubyconf today to talk about rSpec and an interesting topic came up.

In my mind, there are two reasons to use a mock object: first, when you’re developing TDD style, you physically don’t have the objects yet; and second, so that you can tightly focus your unit tests. Maybe, these two different purposes should use a different mechanism.

His question to me then was, “Do you replace your mocks with the real objects after you’ve implemented those objects?”. I guess I hadn’t thought about that before. Do you? If so, how do you handle the extra complexity, maintaining sane associations and valid data?

We’re launching a new service at work in the next week or so that involves me looking through a lot of job applications: resumes and sample code.

I’d like to tell people right now, upfront, if you’re applying for a Ruby or Rails job, for anyone, there are a few ways of ensuring you get called back. They’re probably fairly simple.

Send some sample code, maybe a link to a project on Github, or a snippet of work you’ve done. Make sure you send the tests for the code. Any tests would be good, and you get bonus points for good tests. If you don’t have any tests, write them.

Don’t worry too much about sending some crazy complex code. Maybe some polymorphic associations (models), some ajax (views), a knowledge of the whole stack (simple controllers), some nested resources. Write a simple todo list application.

It’s not just a silly philosophy. Writing tests – hell, submitting tests with your job application’s code – shows that you’ve actually thought about the code, and that it actually works. You’ve permutated and permeated through the logic, actually think about the various ramifications of the design decisions in the code itself.

Just the pure act of sending tests with your sample code will put you above 90% of applicants, I promise.

...for new projects.

We upgraded the gems for one of our client projects, and the auto-loading / config.gems managed to completely break all our other projects, requiring upgrades, which caused weird breakages in weird places in some of the specs.

The app would refuse to deploy (rake tmp:create failed, because lib/tasks/rspec.rake was being loaded, and spec wasn't installed on the server). The annoying thing was that just having whatever.11 installed (I don't know the exact version) broke older apps on whatever.4 or whatever.0.2. .. so those had to be upgraded too. We wasted a day or two (three, maybe four developers) which equates to several thousand dollars in wasteage. It was also really infuriating -- the culmination of a few years of frustration of rSpec's weirdnesses.

After that, I found that some of the specs had never run (who knows why). It stopped reading spec.opts and started doing some weirdness with pending options. Finally, Rick just snapped, threw out rSpec and his Model Stubbing library, and now we're playing with a combination of rr, context, and matchy, trying to get a feel for a decent workflow again. It's sad and maybe a bit exciting to be on the edge.

What are you testing with?

Say you have an email validation regex that you want to quickly spec against a bunch of possible addresses, good and bad.

Did you know rspec allows you to loop over arrays, dynamically creating describes and “it”s at will? I solved this tonight like so, with the help of cristi and some code of rick:

  [ "joe@foo.bar", "m@test.co.uk", "23@233.com", "joe.fake@gmail.com", "test+me@yeah-you-99.com"].each do |add|
    it "validates '#{add}' as a valid address" do
      @mail_account.email_address = add
      @mail_account.should be_valid
    end
  end

  [ "joe@foo", "joe", "joe@xa."].each do |bad|
    it "rejects obviously bad address '#{bad}'" do
      @mail_account.email_address = bad
      @mail_account.should_not be_valid
    end
  end

Dynamically creating each “it” block with a custom message so you can see the exact failure.

Over a year ago we published some benchmarks on how fast your computers were running the complete ActiveRecord test suite. I consider this to be a great test for the fastest platform for developing Rails. (Let’s ignore the speed of your IDE or pseudo-IDE—this one’s all about waiting for your autotest. This probably isn’t a good indicator of server status)

It’s time to run this test again. Why? Because I’m buying a new computer, and I want to be the most efficient with my money as possible. That means a macbook, rather than macbook-pro.

Check out Rails revision 8117 (trunk at this time), install sqlite if you haven’t already (macports: rb-sqlite), and run rake test_sqlite

Comment here with your platform, and the time reported. If you want to be more accurate, run it a few times. I’m not a professional statistician; don’t tell Zed Shaw about my shoddy procedure.

Factors that may influence your times: disk speed, processor speed, your ruby version, luck …?

Who	Hardware	Rake time (sec)	OS
—--	—-—-—-—-	——-	—-——
chrissturm	imac core2	18.88	leopard
octopod	mbp-sr	23.45	tiger
technomancy	mbp-sr	25.74	ubuntu gutsy
defiler	mb1	25.772	leopard
form	mb2 2.0	26.59	leopard
courtenay	macpro 2×2.6	28.49	tiger
mike	Athlon64/3000	34.63	xp
courtenay	Sempron64/2600	57.49	fc6
courtenay	powerbook 1.5	92.92	tiger

• Summary

From the looks of it, most current-level professional macs whether laptop or desktop run the benchmarks at within 15% of the same time. This probably isn’t too much of a surprise, since ActiveRecord won’t run on multiple processors; but it’s nice to know that if you’re only really doing rails on your laptop, a macbook is as good as anything out there.

The move to Intel has really helped Apple get a nice standard baseline for performance, that clearly smokes the ‘old’ PPCs.

In fact, ‘ol faithful, my previous fast-rails-box running linux on an amd-64, has dropped to very lowly status of 57 seconds. It’s time to retire my trusty powerbook. I spend more time waiting than coding.

Notes:

• mb1 : MacBook 1 (Core Duo)
• mb2 : MacBook 2 (Core 2 Duo)
• mb3 : MacBook 3 (Santa Rosa)
• mbp-sr : MacBookPro (Santa Rosa)

I’ve added a small patch to the form mutator, so that SELECT boxes will now be populated when spidering forms.

As always, point piston or script/plugin at svn://caboo.se/plugins/court3nay/spider_test

So, you're happily using mocks to remove the database from your skinny™ controller.

The code has been hacked on by about four different people and looks something like

describe CategoriesController, "showing a record" do

  before do
    @store = mock_model(Store, :categories => mock('categories proxy'))
    @product = mock_model(Product)
    @store.categories.stub!(:find_by_permalink).and_return @product
    @product.stub!(:name).and_return('foo')
  end

  it "should show successfully" do
    get :show
    response.should render_template('show')
  end

  it "should load one record" do
    @store.products.should_receive(:find_by_permalink).with('1').and_return @product
    get :show
  end

end

To be honest, it's pretty nasty, and with rSpec, if it feels nasty it's probably wrong. The controller is quite simple

class CategoriesController < ApplicationController

  before_filter :load_store

protected
  def load_store
    @store = Store.find(session[:store_id])
  end

public

  def show
    @category = @store.categories.find_by_permalink(params[:id])
  end

  def edit
    @category = @store.categories.find_by_permalink(params[:id])
  end

  def update
    @category = @store.categories.find_by_permalink(params[:id])
    @category.update_attributes(params[:category])
  end

end

Now, there are two ways of DRYing up this. They both involve a "find_category" method. The holy war involves whether you load the data in a before_filter or explicitly set @category in each action. I think the first is much cooler.

class CategoriesController < ApplicationController
  before_filter :find_category, :only => [ :show, :edit, :update ]

protected
  def store
    @store ||= Store.find(session[:store_id])
  end

  def find_category
    @category = store.categories.find_by_permalink(params[:id])
  end

public

  def show
  end

  def edit
  end

  def update
    @category.update_attributes(params[:category])
  end

end

In the new spec, we can do something like this:

describe CategoriesController, "showing a record" do

  before do
    controller.stub!(:find_store)
    controller.stub!(:find_category)
    controller.instance_variable_set(:@category, mock_model(Category)
  end

  it "should show successfully" do
    get :show
    response.should render_template('show')
  end

  it "should load one record" do
    controller.should_receive(:find_category)
    get :show
  end

end

describe CategoriesController, "finding a record" do
  before do
    @store = mock_model(Store)
    controller.stub!(:store).and_return(@store)
  end

  it "should find a record by permalink" do
    controller.stub!(:params).and_return({ :id => '1' })
    @store.should_receive(:find_by_permalink).with('1')

    controller.send(:find_category)
  end
end

First, we test the "should show.." logic. Then, in a different context, we test that the "find" works as advertised.

Got a better way?

I’m shuffling round the code a little to make it easier for others to extend.

You can now mutate existing form values, which will effectively fuzz your application and see where your error handling is broken. I haven’t worked out a way to easily toggle this, so it’s disabled by default. Read the source. Most people won’t want to fuzz until a later stage in testing. I’ve added a form value to every mutated form so you can tell if it was just given data (if the value was empty) or if it was mutated (existing value)

Forms now show a better error message, and also show the query. I’m working towards making it generate code you can just copy-paste into another integration test (lazy…)

You can ignore URLs and forms by regex also.

get ‘/’
spider(@response.body, ‘/’, 
       :ignore_urls => [‘/login’, %r{^.+logout}, %r{^.+delete.?}], 
       :ignore_forms => [])

Thanks again to everyone who sent in patches.

Do your specs always start with it "should" ?

Like

context "A list-item" do
  it "should denormalize task name" do
    @item = ListItem.new :task => mock_model(Task, :name => "TaskName")

Well, rather than moaning about changing the syntax, do it yourself! (Thanks to David Chelimsky in #rspec)

Put this in spec_helper

module Spec::DSL::BehaviourEval::ModuleMethods
  alias :should :it
end

Now you can refer to either "should", or "it".

context "A list-item" do
  should "denormalize task name" do

Win!

In followup to the previous article, another way I'm using rspec. This is a variation on the "fat model" idea, but I'm pushing it a little further.

In controllers, we frequently see code like this:

def index
  @people = Person.find(:all, :order => 'id desc', :conditions => ['activated=?', true])
end

def show
  @person = Person.find(:first, :conditions => ['id = ? and activated = ?', params[:id], true])
end

Now, to my eye, that looks like we're leaking commands from the database (which should be hidden away under the model). I even think that in many cases, the 'find' call should be a protected method to the model.

Exposing find and its parameters allows the coder to make database calls from the controller, which makes the spec incredibly brittle and rigid.

def index
  @people = Person.find_all_activated
end

def show
  @person = Person.find_activated( params[:id] )
end

The model will look something like:

def find_activated( person_id )
  find(:first, :conditions => ['id = ? and activated = ?', person_id, true]
end

We can now stub out the method happily. Also, this interface -- User.find_activated -- will never need to change.

it "should render show" do
  User.should_receive(:find_activated).and_return mock_model(User)
  get :show
  response.should be_success
end

You can also make that return a User.new object or whatever mock you want.

Once again the team has introducted a breaking change between versions. I'm holding off migrating up from 0.8.x until others solve all the issues that will arise. To be honest, it's the one thing that kept me from rspec in the past, and despite now using it in all my projects, I really hate that they keep changing the API. Hate hate. I may just wait til 1.0, when they change it all again.

Ruy Asan has hit and solved a few issues and gotchas in his own apps, so if you're feeling the pain of rspec 0.9, check out his migration pains post.

Got too much rspec setup code, or just want to DRY it out? Here's how I do it. Maybe there's a better way? (Fortunately, you can define as many setup methods as you like, and rspec will run them all. The trick is in getting a module to define the setup method for you)

module SetupCart
  def self.included(base)
    return unless base.class.name == 'Spec::Runner::ContextEvalModule'
    base.setup do 

      @cart = mock_model Cart, :to_param => '1aaBAX-34256'
      controller.stub!(:find_cart)
      controller.stub!(:set_person_var)
      assigns[:cart] = @cart

    end # setup
  end # included
end # module

and the spec

context "The cart controller" do
  include SetupCart
  setup do 
    @cart_items = mock "Association proxy"
    ...
  end

  ...
end

Now for the explanation: rspec will include the module twice, and the second time it does so "base" is Class. I didn't delve into the source to determine why.

Fixed a few errors and added features. See spider test for previous notes.

script/plugin install svn://caboo.se/plugins/court3nay/spider_test

Fixes:

• namespacing fixed (thanks, commenters)
• better showing of errors
• for the impatient: you can now hit ctrl-C (well, a few times) during the test and it’ll break out, but show the errors anyway.
• doesn’t try to follow page anchors (foo.html#monkeys)
• rescues from rails errors (:put isn’t working on rails 1.2.3 in integration tests for some people)

Ideas:

• Save the errors in a file?
• Better display of errors
• Ignore xml? Follow xml? Gah?
• Fuzzing
• Make it a Gem

For those of you still resolute in your test::unit ways, refusing, unable, or unwilling to join the dark side and move to the one true behavioral-based testing framework, here's a gateway into our world. Hopefully this guide will help you see ruby more as a message passing language, and will placate those monkeys in the trees screaming, "You don't need rpsec to mock and stub!"

I'm working on a project that uses test::unit and one of the tests looks something like this:

def test_should_not_get_edit_for_candidated_patch
  login_as :quentin
  get :edit, :id => 1
  assert_equal "Patch is no longer editable!", flash[:notice]
  assert_response :redirect
end

Should be fairly familiar to you all. Using authenticated system, and login_as sets the request.session[:user].

This is fine, but it requires you to load up the users.yml and fiddle around with various flags; you see, there's a test in the controller for current_user.administrator? but you wouldn't know that from looking at the test.

Here's the controller action we're trying to test:

def edit
  unless current_user.administrator? or (@patch.opened? and @patch.belongs_to?(current_user))
    flash[:notice] = "Patch is no longer editable!"
    redirect_to :action => 'show'
  end
end

The auth system does something like this in a filter to find the current user:

@current_user = User.find(session[:user]) 
return @current_user.is_a?User

It's a bit more complex than that, but basically, if the user can be found, load it up, otherwise return nil and redirect to a login page.

rewrite that test!

This is where a mock (with stubbed methods) comes in handy. First, set a dummy session variable, to keep auth_system happy (we're storing IDs not objects).

@request.session[:user] = 1

Now, the app will try to load a User object from the db, but we don't want that, do we? Why test the db from the controller? It should already be tested at the model. So, let's make a mock object

@user = mock('user')

and stub the find_by_id method, so the User model doesn't have to hit the database

User.expects(:find_by_id).with(1).returns(@user)

Or, if you're lazy like me

User.expects(:find_by_id).with(1).returns @user = mock('user')

Now, we have a virgin, impressionable object. Let's slap it round a bit.

@user.expects(:is_a?).with(User).returns(true)

Gasp! overriding the core ruby! This is getting fun..

@user.expects(:administrator?).returns(false)

Now, run that, and it warns that

#<Mock:user>.is_a? - expected calls: 1, actual calls: 4

This is because we're calling is_a?User multiple times (probably bad code in there, but regardless...)

@user.expects(:is_a?).with(User).at_least(1).returns(true)

Quick hack to tell it that the is_a?User message should happen at least once (duh!)

the code

Here's what we've built:

def test_should_not_get_edit_for_candidated_patch
  @request.session[:user] = 1
  User.expects(:find_by_id).with(1).returns @user = mock()
  @user.expects(:is_a?).with(User).at_least(1).returns true
  @user.expects(:administrator?).at_least(2).returns(false)

  get :edit, :id => 1
  assert_equal "Patch is no longer editable!", flash[:notice]
  assert_response :redirect
end

Sweet! But you're going to want that mock stuff in other methods, too, and it's not really relevant just to this test

def setup
  @request.session[:user] = 1
  User.expects(:find_by_id).with(1).returns @user = mock()
  @user.expects(:is_a?).with(User).at_least(1).returns true
end

def test_should_not_get_edit_for_candidated_patch
  @user.expects(:administrator?).at_least(2).returns(false)

  get :edit, :id => 1
  assert_equal "Patch is no longer editable!", flash[:notice]
  assert_response :redirect
end

So, now, obviously, we're testing the logic around a user being an administrator. No fixtures, no database required for the user. You know exactly what you're testing, and it's hell fast. Score!

I've been playing with rspec on a project and I gotta say, I'm in love. I'm working up to a full-blown article, but if you haven't checked it out, please do. Rspec really helps you see ruby as a message-based language. You're just passing messages around. So it's easy to set up an expectation, "this object should receive this message", and then fake out a response in order to test the logic in our application. For those of you who've been meaning to get around to it, here's the executive summary. (Go read the docs, they really are excellent) h2. behaviour-based Your specs set up expectations, like

   User.should_receive(:count) 

but can also return a value

   User.should_receive(:count).and_return(0)

This helps you test your logic, since you can determine any return value you like. In case you haven't got it, that's the same as doing

  User.stub!(:count).and_return(0)

except using should_receive is like an assertion, where it'll raise if the User class never receives that message. This means stub! is well suited to setup blocks. The counter-intuitive thing for you test::unit types is that you set up the expectations before calling the method.

context "The cart controller"
  setup do
    @user = mock("user")
    @carts = mock("carts")
    @cart = mock("cart")
  end

  specify "should render checkout with cart specified correctly" do
    @user.should_receive(:carts).and_return(@carts)
    @carts.should_receive(:find).with('1').and_return(@cart)
    @cart.should_receive(:waiting_for_address_info?).and_return(true)
    
    get :checkout, :id => 1
    response.should_be_success
  end
end

Nowhere here are we hitting the model, since that's been well tested earlier. We just ensure that the model and the association gets the correct messages, and that the logic in the controller is solid. The controller code looks something like

  def checkout
    @cart = params[:id] ? @person.carts.find(params[:id]) : @person.find_unfinished_cart  
    @cart.shipping_address ||= Address.new
    @cart.begin_checkout!
    redirect_to :action => 'address', :id => @cart unless (@cart.active? or @cart.waiting_for_address_info?)
  end

Oh, and one more tip. If you're doing something like

  redirect_to :id => @cart

You'll want to stub @cart.to_param to an ID.

  @cart = mock("cart")
  @cart.stub!(:to_param).and_return('1')