Keep performance in mind when using FactoryGirl in your test suite

Dave How-to, Testing, Web Design & Development 3

The latest major version of FactoryGirl came out last week, which coincided with an internal debate here on the best use of factories. Often, out of habit, when we’re writing tests, we reach for a Factory when we’re testing a new method. It’s convenient because it gives you some test data to work with, and you can easily set protected attributes without having to deal with mass assignment protection. After all, is there a big performance difference between these?:

Factory.build(:listing)

Listing.new

It depends, of course. What are you defining in your factory? Just a few attributes? A couple of associations? Here’s what the one in question looks like:

Factory.define :listing do |f|
  f.sequence(:title) { |n| "Super Listing #{n}" }
  f.display_price "15"
  f.description 'Check out my super listing. Buy buy buy!'
  f.association :account, :factory => :confirmed_account
  f.association :zip
  f.association :category
  f.listing_type { |a| ListingType.offline }
end

After we dive a bit deeper, now there’s no question; using a Factory is going to be more resource-intensive than building a blank ActiveRecord object. Not only are we doing a ‘Listing.new’ but we’re also doing a ‘Zip.new’ and an ‘Account.new’ and we’re keeping track of previously generated titles; there’s simply a lot more going on. But how much slower is it, really?

All of these fields are set primarily to get the model to pass validation. That’s what makes FactoryGirl such a powerful tool; you can type Factory(:listing) and magically have a listing in your database without having to worry about any of that underlying validation logic. It’s excellent for generating test data that you can use in your Cucumber scenarios or if you’re testing out database queries.

In our estimation, the main speed difference wasn’t between using a Factory and not using a Factory. The biggest win would be avoiding the database. That’s why we’d often opt for Factory.build, because it doesn’t save the object you’re creating. A perfect use case is for testing validations:

describe "validations" do
  it "is valid" do
    Factory.build(:listing).should be_valid
  end
 
  it "is invalid without a title" do
    Factory.build(:listing, title: '').should_not be_valid
  end
end

This test clearly shows that a fully fleshed out Listing is valid, but when you remove the title, it’s not valid. And since we’ve called Factory.build, there’s no database hit. Right?

Surprise!

That’s what I thought, anyway. It turns out that while Factory.build doesn’t save your Listing, it does save your associations; your Account, your Zip, your Category, and any associations that those factories may have. Those long test runs are starting to make a lot more sense now.

Enough with all of this conjecture; how about some benchmarks?

Benchmark.realtime { 100.times { Listing.new } }
=> 0.0148091316223145 seconds
 
Benchmark.realtime { 100.times { Factory.build(:listing) } }
=> 19.3972871303558 seconds
 
Benchmark.realtime { 100.times { Factory(:listing) } }
=> 38.2170720100403

Fleshing out the attributes using Factory.build in this case is 1300x slower than building an empty Listing, and using Factory (to persist the Listing itself) is a whopping 2500x slower. To be fair, with an empty Listing Factory, it’s not much worse than doing Listing.new:

Factory.define(:boring_listing, :class => 'Listing') {}
 
Benchmark.realtime { 100.times { Factory.build(:boring_listing) } }
=> 0.0174689292907715 seconds

So the culprit here is building and saving those associations.

How about our validation specs above? Is there a better way? We still want to be able to quickly flesh out a valid model without hitting the database. For that, we have a much better option, Factory.stub:

Benchmark.realtime { 100.times { Factory.stub(:listing) } }
=> 2.69829893112183 seconds

This does what we actually intended to do in the first place, in this case 7 times faster than using .build.

This is clearly not an exhaustive analysis, but it has taught us a few valuable lessons:

1. Use caution in utilizing Factories. If you can avoid it, do so and test your model directly and you will get drastic performance boosts. This should be easy for most unit tests that don’t need to touch the database.
2. If you need a fully fleshed out model, such as for testing validations, use Factory.stub (build_stubbed in FG 3.0)
3. Keep your base Factory definitions as lightweight as possible: just enough to get a record to save; which is many times more important if you’re defining an association.
4. Beware of Factory.build, for it is deceiving.

All that said, FactoryGirl is a truly handy tool that we use in just about all of our Rails test suites. We don’t suggest that it’s inefficient or broken, but rather that you should keep performance in mind when writing your specs.