Given I a have an account with email: "user@test.com" and password: "password"
When I go to the log in page
And I fill in "Email" with "user@test.com"
And I fill in "Password" with "password"
And I press "Log in"
Then I should see "Logged in successfully."
And I should not see "Log in"

It works and does the job but I feel that it isn’t enough. It’s fragile in several ways.

First, if any of your form field or button labels change or your flash message changes, you’ll need to update your feature even though the behaviour hasn’t actually changed! I totally understand why we write features like this – it’s convenient, it’s quick and easy.

Second, have we actually verified that the user has logged in? All we’ve really done is verify that we see a particular flash message and that we no longer see a piece of text on the page. We haven’t actually verified the behaviour under test here. Again, it’s convenience but it’s one that can cause you trouble.

I’ve had a feature like the above where the flash text I was looking for actually appeared in a login error message. For example:

Then I should see "successfully logged in"

My success flash message was “#{user.name} successfully logged in” and my feature passed. But it was a false positive. In fact the user had not successfully logged in at all. My error message was “#{user.name} was not successfully logged in” so when the login failed my step would still pass because the string is found on the page.

Okay, that’s easy to fix but it showed me just how easy it is for a feature to be fragile and too dependent on content and UI elements. Any rearrangement or inclusion of new content on a page can have a dramatic effect on your cucumber features. Worst case scenario is that you actually break functionality but your tests return a false positive.

To take it a step further, what if you had only implemented the flash handling and not the actual logging in of the user yet. You would get a passing feature.

My point is that the verification is not strong enough. You are reliant on an artefact of the logging in process. And this artefact is not part of the business rule.

The business rule here is that when a user logs in with the correct credentials they are successfully logged in. It is ancillary whether they see a flash message telling them they are logged in or if they can’t see the login button anymore. What you need to check is that the user is actually logged in.

So replace the last two lines in the example with:

Then I should be logged in
And I should be on the dashboard page

In your step definition you should use the logic that your system uses (or will use) to check if a user is logged in. For example:

Then /^I should be logged in$/ do
  # get your user first
  user.should be_logged_in
end

To take it further, you can remove the other content and UI references to make your feature more robust:

Given I a have an account with email: "user@test.com" and password: "password"
When I go to the log in page
And I fill in the email field with "user@test.com"
And I fill in the password field with "password"
And I press the login button
Then I should be logged in
And I should be on the dashboard page

I admit that this is more work so, like most things, you need to decide what’s the most appropriate for your context. If nothing else, give some extra thought to your Then steps – are they thorough enough? are they verifying the primary business behaviour?

Given 31 tasks exist  # pagination defaults to 30

The downside of course is speed. Creating a large number of complex objects can add a lot of extra wait time to your cucumber runs. In my case, although I only had a few scenarios that involved pagination, my per_page setting was 50 and the objects I was creating were complex. The end result was a extra couple of (unnecessary) minutes. It wasn’t too bad so I left myself a TODO to fix it one day and that day finally came.

What I wanted to do was to simply change the default per_page setting for pagination across my entire project when running my cucumber suite so that I could rewrite my steps like this:

Given 3 tasks exist  # pagination defaults to 2

This is how I set it all up:

• DRYed up my code by pulling out explicit settings for the per_page option and setting a project-wide per_page setting for will_paginate as a constant in my environment.rb file
• set a different value for this constant in my test.rb file and allow it to override the default setting above
• modify my cucumber scenarios to use the new (smaller) per_page value

More specifically, I added the following to the end of config/environment.rb:

unless defined? DEFAULT_PER_PAGE
  # see: http://groups.google.com/group/will_paginate/browse_thread/thread/eda47114e3127709  
  ActiveRecord::Base.instance_eval do     
    def per_page; 50; end   
  end
  DEFAULT_PER_PAGE = ActiveRecord::Base.per_page
end

Then in config/environments/test.rb:

ActiveRecord::Base.instance_eval do     
  def per_page; 2; end   
end
DEFAULT_PER_PAGE = ActiveRecord::Base.per_page

This solution was specific to my needs. In particular, I needed a DEFAULT_PER_PAGE constant within my code. This meant I could also use it as the conditional in my environment.rb to only set the override if it hadn’t already been set. One alternative was to take this code out of environment.rb and into the specific environment files eg development.rb, production.rb. But I also have a training.rb and a staging.rb so for me I preferred to put it into the generic environment file and change the setting specifically in the test environment.

Now, let me introduce you to capybara-envjs, a new gem that Joseph Wilk described in his talk at railsconf. It is a capybara driver for the envjs gem. Instead of using the selenium driver and launching a browser to perform the tests, it uses envjs, a simulated browser environment. That means it will be faster right? A quick experiment showed that it was. Another benefit is with running your javascript tests on a continuous integration server.

I’ve only just begun to play with this new setup but it looks promising. I’ve forked Tim’s original demo app and updated it to use capybara-envjs. To get it working, I added the following to cucumber’s env.rb file (in features/support):

Capybara.javascript_driver = :envjs

And then I replaced the capybara gem in the Gemfile with:

gem 'capybara-envjs', :require => 'capybara/envjs'

Easy! Clone it and give capybara-envjs a try.

This change prompted me to take a closer look at the options available for formatting the output of running cucumber features. To see the full list enter in your console:

cucumber --help

To change the format pass either of the following:

cucumber --format FORMAT
cucumber -f FORMAT

Here’s some of the format options:

• html – Generates an HTML version of your test results which you can save as a file and view in a browser. This is the one to use when running Cucumber inside Textmate.
• pdf – Generates a fancy formatted pdf of your test results which you can share with your clients if you wish or for archiving in your company’s records management app.
• pretty – This was originally the default format. It prints out each step of a scenario and colours it according to whether it passed, failed, was undefined or was skipped. This format provides you with the most information and is the one to use when working the red-green-refactor cycle.
• progress – This is the new default. It prints a dot for each step that passed, an F for one that failed, a U for an undefined step, and a hyphen for a skipped step.
• rerun – Prints out the failing features (with line numbers). This is now used in the default “smart” cucumber profile to setup an automatic rerun of failed scenarios. When you first use cucumber, it runs all features and scenarios. If any fail, a rerun.txt will be created that contains the output from the rerun format, a list of failed scenarios. Now when you run cucumber again, it will only run the failed scenarios instead of your entire collection of features.
• tag_cloud – Useful for showing you what tags you have used and how often.
• usage – Useful for revealing which step definitions have not been used and which are the slowest.

Finally, if none of these fit your needs, you can always write your own.

Given a company exists
And another company: "rubyflare" exists
And the following people exist:
  | name  | age | employed | company              |
  | Digby | 13  | false    | the first company    |
  | Ethyl | 27  | true     | company: "rubyflare" |
Then the following people should exist:
  | name  | age |
  | Digby | 13  |
  | Ethyl | 27  |
And the 2nd person should be one of company: "rubyflare"'s employees

Again, I’ll just point out that the above example required no custom step definitions. I didn’t have to write any step definitions! Pickle takes care of the grunt work leaving you free to focus on your domain specific scenario steps.

Other recent improvements with Pickle include:

• an email helper for mapping names to email addresses similar to paths
• its own separate configuration file (pickle.rb) – pickle no longer adds code to features/support/env.rb

I’ve also updated my pickle_example project to the latest version and have added the above example scenario. Feel free to clone it and use it to explore the benefits of using Pickle.

• you can now use negative numbers in numeric field values when creating objects, and
• you can use some extra default step definitions to verify the non-existence of objects

This means you can do things like this in your plain text cucumber steps:

Given a user exists with name: "Digby", bank_balance: -43.25

And another user exists with name: "Miranda", bank_balance: +86.50

Then a user should not exist with name: "Maya"
And the first user should not be one of the last user's debtors
And the last user should not be the first user's creditor

It feels good to contribute back to an open source project, even in a small way, and I encourage you all to try to do so as well.

Pickle makes life in the Cucumber world so much easier by providing you with convenient step definitions that take care of common object operations leaving you free to focus on the fun stuff. Combined with the pre-packaged Webrat steps you can dramatically reduce the number of step definitions you need to write.

Pickle allows you to easily create objects from your factories (Factory Girl), blueprints (Machinist) or ActiveRecord. Here’s a simple example:

Given a person exists

And another person exists with name: "Digby"

And another person exists with name: "Ethyl", age: 27, employed: true

Then a person should exist with name: "Ethyl"

And 3 people should exist

And the last person should be an adult
And the 2nd person should not be employed

Given a company exists
And another company: "rubyflare" exists

And a person exists with company: company "rubyflare"
And another person exists with company: the first company

Then the first person should be one of company: "rubyflare"'s employees

Given a person exists with email: "fake@address.com"
When I activate that person
Then 1 email should be delivered to fake@address.com

Given a company exists with name: "Ruby Flare"
When I go to the companies page
Then I should see "Listing companies"

When I go to the company's page
Then I should see "Ruby Flare"

When I go to the users page
Then I should see "Listing people"

When I go to the company's employees page
Then I should see "Listing employees"

When I activate that person

When /^I activate #{capture_model}$/ do |name|
  model(name).activate
end

When I activate the person
When I activate the 2nd person
When I activate the person: "maya"

So that’s a quick introduction to the joys of using Pickle with Cucumber. Download the gem, check out the readme and the code, and start using it in your Rails projects. You can also clone a copy of my pickle_example project and play around with it.

Pickle is an essential ingredient in my recipe for pure BDD joy. I hope you enjoy it too!

Update (4/11/09): Ryan Bates has done a screencast about Pickle! Go check it out.

Further to Brandon’s post, here’s how I’ve incorporated running search features with Cucumber…

What I wanted was to keep using the standard transaction approach for my normal Cucumber features but to switch to using database cleaner for sphinx features. Now, I haven’t been able to figure out how to turn on/off the use of transactional fixtures in Cucumber.

To enable it you use:

Cucumber::Rails.use_transactional_fixtures

But how do you turn it off?

My current solution is to pull this out of the main env.rb file and turn it on in a webrat.rb support file. This meant I needed to set up my Cucumber profiles to selectively load the appropriate support files (in cucumber.yml):

default: -t ~@sphinx -r features/support/env.rb -r features/support/webrat.rb -r features/step_definitions features
sphinx: -t @sphinx -r features/support/env.rb -r features/support/sphinx.rb -r features/step_definitions features

I then mark all sphinx features/scenarios with the cucumber tag @sphinx. Now I can run my sphinx features using:

cucumber -p sphinx

Finally, switching back and forth between development and testing was causing me grief in starting and stopping the search daemon. The simplest solution was to assign a different port for testing (in sphinx.yml):

development:
  port: 3312
test:
  port: 3313

Update (30/9/2009):

When I upgraded to the newest version of Cucumber (0.3.104) and ran script/generate cucumber I discovered this change in the env.rb:

# Whether or not to run each scenario within a database transaction.
#
# If you leave this to true, you can turn off traqnsactions on a
# per-scenario basis, simply tagging it with @no-txn
Cucumber::Rails::World.use_transactional_fixtures = true

This would eliminate the need to extract the use_transactional_fixtures setting from the env.rb. Definitely worth revisiting my strategy above now.

Update (15/10/2009):

By upgrading to the latest version of Cucumber (0.4.0) I have been able to undo most of the workaround mentioned in this post. I re-generated the env.rb file using ruby script/generate cucumber and I removed my cucumber profiles. If I want to run only sphinx scenarios I simply use cucumber -t @sphinx or to run non-sphinx scenarios I use cucumber -t ~@sphinx. I created a db_cleaner.rb file and I modified my sphinx.rb file removing all the database cleaner references. Finally, I add the @no-txn tag to all my scenarios which are already tagged with @sphinx. Separation of the database cleaner behaviour from the @sphinx tag means that the @no-txn stuff can be applied elsewhere as required eg selenium features.

Update (19/12/2009):

Be aware that the behaviour of tags changed with cucumber 0.4.3 to allow logical ORing and ANDing of tags. If you use tags in your profile definitions you might find you get different behaviour when you pass the @sphinx tag. My default profile is configured to exclude all scenarios and features marked with the @selenium tag. So when I run cucumber -t @sphinx cucumber will run all scenarios and features marked without @selenium OR with @sphinx. This results in all of my non-selenium scenarios running, not just the sphinx related ones. I still haven’t figured out a suitable fix for this however I just wanted to point this out.

If you don’t use profiles or don’t have tags specified in your profile then the above technique will continue to work as described above.

Update (02/06/2010):

As for version 1.3.2, Thinking Sphinx now provides a test helper for use with Cucumber. I simply add the following lines inside my features/support/sphinx.rb file:

require 'cucumber/thinking_sphinx/external_world'
Cucumber::ThinkingSphinx::ExternalWorld.new
ThinkingSphinx::Test.start_with_autostop

Now I can remove all the startup/setup code I had previously (as show in Brandon’s original post). I also no longer need an at_exit block since that is handled by the ThinkingSphinx::Test.start_with_autostop. Now my indexing step is simply:

ThinkingSphinx::Test.index

I have also found it necessary to add a sleep(0.25) after my indexing step to ensure that Sphinx has enough time to perform the reindex before Cucumber moves onto the next step.

View specs aren’t about verifying that your html is correct or that you have applied the correct css to the appropriate dom element. Like other specs, it’s about business requirements. View specs are only brittle when you tie them closely with the presentation of your data.

Meanwhile, I prefer to keep my Cucumber features and step definitions free of implementation details. I just want to verify that the behaviour works, that when I do a particular task I get the outcome I expect. How it happens or how it is presented isn’t really what I care about with Cucumber – it’s an integration test after all.

So, I use my view specs to verify low-level business requirements like “should have a search box”. I’d use Cucumber to verify that the search box works and I’d use view specs to ensure that the search element has a title, an input field and a submit button. The view spec verifies the nitty gritty detail. For example, as a designer I could label the search any way I wish. It’s just text after all. But the business may need to use specific terminology/wording for the title and for the form element labels.

This is where view specs can easily become brittle. If you write them such that they depend on html elements then you are tying them to the implement of the html. If you were ever to change your html, your specs would break even though the actual business requirements would still be met. My advice is to try and eliminate all reliance on the html implementation. Focus on the business requirements.

Here’s a simple example…

The business wants a title of “Candidate Search” for their search element. It would be easy to write:

response.should have_tag('h2', 'Candidate Search')

response.should include_text('Candidate Search')

The thing to always keep in mind, is not to go overboard. Keep focused on the business requirements and ask whether the tests are adding any business value. I don’t test all aspects of what my views present. Remember, you should try to keep logic out of your views. I test the bits that are important to the business. Yes, there will be crossover with Cucumber features so this is where your judgement comes in.

Here’s my final view spec (using webrat 0.5.1 matchers):

response.should have_selector('#candidate_search') do |search|
  search.should contain('Candidate Search')
  search.should have_selector('form', :action => candidates_path) do |form|
    form.should have_selector('[type=submit]')
  end
end

I believe software testing is essential. Behaviour Driven Development (BDD) is something that I’m learning and incorporating into my development lifecycle. I use Cucumber and RSpec with my Rails projects to implement new features and to verify their behaviour.

Aside from driving your design, validating your code, and giving you and your stakeholders confidence in your app, testing saves you money. That’s the bottom line. You might not think so. You might be able to get away with zero testing if you are extremely lucky or if you are a super coder who writes perfect software. Ultimately, the time you invest in writing tests for your code will save you time and money.

I recently made a small change to my SciNews web application without writing a test. Why? Because it was just a small change and I was in a rush. I had opened the app in my browser and tried it out and everything was fine and dandy – deploy!

A month went by without any problems. Admin users of the system weren’t experiencing any issues but when a client user tried to use the service – WHAM! BAM! ERROR!! So there I was, just before midnight, on the receiving end of a support call from the client trying to figure out what just went wrong.

I tried to fix it on the spot but I didn’t understand the cause. The only change I made was a minor one about a month ago and admins had been using the system fine during that time. But this was the first client access since then. Aha! But unfortunately, after 15 minutes on the phone with the client, I was unable to solve the problem. As I later discovered, the bug affected client roles, not admins. When I tested it in the browser, I had done so using my admin account. I never tried it as a client.

In the end I had to manually perform the service for the client on their behalf. But the system had failed them and worse, it was their first use of the online service. In order to save face, compensate them for their wasted time and frustration, and to keep them as clients, the service was provided gratis. My small little insignificant change had ended up costing my business $600 and for a small, fledging business, that’s a lot of money that we can’t afford to lose. For $600, I should’ve written a test! Add in time spent on the phone with the client, the embarrassment, and the potential damage to the service’s reputation, I should’ve definitely written a test.

Will this client come back and try our service again? I hope so, otherwise my little error may well end up costing heaps more in lost revenue. This client may have become a repeat customer, so we’re talking a loss of $600/mth. And what about word of mouth? They might have referred two new clients. Now we’re talking some serious lost revenue here all because of one small minor change. Hopefully I’ve done enough with that client for them to give the service a second chance. Whether they do or not, I’ve learnt my lesson – TATFT!