UPDATE (2012-01-22): I realized this morning that the credit I gave to Sam Livingston-Gray below may not have adequately shown how instrumental he was in getting this project off the ground; especially since much of his work was done in the private repository from which this was extracted. So, thanks, Sam. This might not have gone anywhere if you hadn’t worked to put the idea in practice in our application and helped everyone on our team learn how to use the approach. I made a few minor changes below to reflect this a bit better.
We’ve been using Cucumber for acceptance testing at Renewable Funding since back when it was still part of the RSpec project (indeed, since before we were even Renewable Funding). While we’ve always liked the ability to have plain-language feature documentation that we could automatically test against, after years of adding to and maintaining a fairly large set of Cucumber scenarios, the cost of that maintenance was starting to really slow us down. The test suite began to grow fragile, and it seemed like every time one of our UX designers changed anything about the application’s interface, the development team would spend a bunch of time just babysitting Cucumber tests to get them passing again.
Last year, as I was reading Jez Humble’s excellent Continuous Delivery book, I was inspired when I came across the section titled “The Application Driver Layer” (p. 198). This section describes an approach to acceptance testing where the specification and the test implementation are isolated from the details of the application’s user interface by inserting a layer between the two that uses good old OOP to abstract the user interface components. Martin Fowler describes it as the Window Driver pattern on his website.
I started a proof-of-concept implementation of this pattern last summer, then my coworker, Sam Livingston-Gray and I started pulling it into a new project at work. After Sam and the rest of the Renewable Funding team helped improve on my original attempt while putting it to use for the last six or so months, we extracted a library to make it easier for other Ruby developers to implement this pattern in their testing. I’m happy to introduce Kookaburra to the world.
The following comes from Kookaburra’s README as of version 0.7.1.
Kookaburra is a framework for implementing the Window Driver pattern in order to keep acceptance tests maintainable.
Kookaburra itself abstracts some common patterns for implementing the Window
Driver pattern for tests of Ruby web applications built on Rack. You will need
to tell Kookaburra which classes contain the specific Domain Driver
implementations for your application as well as which driver to use for running
the tests (currently only tested with Capybara). The details of setting up your
Domain Driver layer are discussed below, but in general you will need the
following in a locations such as
MyApplication with a module name suitable to your actual application:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
For RSpec integration tests, just add the following file to your project:
1 2 3 4 5
For Cucumber, add the following file to your project
1 2 3 4 5 6 7 8 9
This will cause the #api, #given and #ui methods will be available in your Cucumber step definitions.
Defining Your Testing DSL
Kookaburra attempts to extract some common patterns that make it easier to use the Window Driver pattern along with various Ruby testing frameworks, but you still need to define your own testing DSL. An acceptance testing stack using Kookaburra has the following four layers:
- The Business Specification Language (Cucumber scenarios and step definitions)
- The Domain Driver (Kookaburra::GivenDriver and Kookaburra::UIDriver)
- The Window Driver (Kookaburra::UIDriver::UIComponent)
- The Application Driver (Capybara and Rack::Test)
The Business Specification Language
The business specification language consists of the highest-level descriptions of a feature that are suitable for sharing with the non/less-technical stakeholders on a project.
Gherkin is the external DSL used by Cucumber for this purpose, and you might have the following scenario defined for an e-commerce application:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Note that the scenario is focused on business concepts versus interface details, i.e. you “choose to check out” rather than “click on the checkout button”. If for some reason your e-commerce system was going to be a terminal application rather than a web application, you would not need to change this scenario at all, because the actual business concepts described would not change.
The Domain Driver
The Domain Driver layer is where you build up an internal DSL that describes the
business concepts of your application at a fairly high level. It consists of
three top-level drivers: the
APIDriver (available via
#api) for interacting
with your application’s external API, the
GivenDriver (available via
which really just wraps the
APIDriver and is used to set up state for your
tests, and the UIDriver (available via
#ui) for describing the tasks that a
user can accomplish with the application.
Given the Cucumber scenario above, the step definitions call into the Domain Driver layer to interact with your application:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35
The step definitions contain neither explicitly shared state (instance variables) nor any logic branches; they are simply wrappers around calls into the Domain Driver layer. There are a couple of advantages to this approach. First, because step definitions are so simple, it isn’t necessary to force Very Specific Wording on the business analyst/product owner who is writing the specs. For instance, if she writes “I see a summary of my order” in another scenario, it’s not a big deal to have the following in your step definitions (as long as the author of the spec confirms that they really mean the same thing):
1 2 3 4 5 6 7
The step definitions are nothing more than a natural language reference to an action in the Domain Driver; there is no overwhelming maintenance cost to the slight duplication, and it opens up the capacity for more readable Gherkin specs. The fewer false road blocks you put between your product owner and a written specification, the easier it becomes to ensure her participation in this process.
The second advantage is that by pushing all of the complexity down into the Domain Driver, it’s now trivial to reuse the exact same code in developer-centric integration tests. This ensures you have parity between the way the automated acceptance tests run and any additional testing that the development team needs to add in. You could write the same test using just RSpec as follows:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Whether in Cucumber step definitions or developer integration tests, you will usually interact only with the GivenDriver and the UIDriver.
Kookaburra::TestData is the component via which the
GivenDriver and the
UIDriver share information. For instance, if you create a user account via the
GivenDriver, you would store the login credentials for that account in the
TestData instance, so the UIDriver knows what to use when you tell it to
#sign_in. This is what allows the Cucumber step definitions to remain free
from explicitly shared state.
TestData class can be configured to contain both collections of test data
as well as default data that can be used as a starting point for creating new
resources in the application. To configure
Kookaburra.test_data_setup with a block (usually in your
1 2 3 4 5 6 7 8 9 10 11 12
Then, in any context where you have an instance of
TestData (such as in
UIDriver), you can add/retrieve items to/from collections and
access default data:
1 2 3 4 5 6 7 8 9
1 2 3 4 5 6 7 8
Kookaburra::APIDriver is used to interact with an application’s external
web services API. You tell Kookaburra about your API by creating a subclass of
Kookaburra::APIDriver for your application:
1 2 3 4 5 6
Kookaburra::GivenDriver is used to create a particular “preexisting”
state within your application’s data and ensure you have a handle to that data
(when needed) prior to interacting with the UI. Like the
APIDriver, you will
create a subclass of
Kookaburra::GivenDriver in which you will create part of
the Domain Driver DSL for your application:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Kookaburra::UIDriver provides the necessary tools for driving your
application’s user interface using the Window Driver pattern. You will subclass
Kookaburra::UIDriver for your application and implement your testing DSL
within your subclass:
1 2 3 4 5 6 7 8 9 10 11
The Window Driver Layer
UIDriver provide a DSL that represents actions
your users can perform in your application, the Window Driver layer describes
the individual user interface components that the user interacts with to perform
these tasks. By describing each interface component using an OOP approach, it is
much easier to maintain your acceptance/integration tests, because the
implementation details of each component are captured in a single place. If/when
that implementation changes, you can—-for example—-fix every single test that
needs to log a user into the system just by updating the SignInScreen class.
You describe the various user interface components by sub-classing
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31