Webtest Tool Round-up (Round 1:WatiN)

This is the first article in a series of reviews. I plan to test a few Open Source automated Web-Testing tools. As I complete my tests I will publish the results to my blog.

The Test Procedure

The world has gone Ajax and Web 2.0 crazy. Any tool that doesn't handle dynamically changing DOMs is an instant failure. So, I decided to use one of the most Ajax intense applications I know of, Gmail. Gmail uses Ajax and Iframes indiscriminately. The developers at google seem to obfuscate ALL the HTML element IDs which means that my web-tests have to do some real back-flips to find the buttons, hyperlinks and textfields required to send messages. I have no doubt that Gmail will adequately stress the tools in this line-up.

Test Steps

1. Log in to Gmail
   
2. Send an Email
   
3. Check the same email
   
4. Apply a Label
   
5. Archive it
   
6. Check to see if it is in the label category.
   
7. Log Out
   

Round 1: WatiN (v1.2.1)

homepage | download

Test Date
17FEB08

Environment
Windows XP Home
Visual Studio 2005
.Net 2.0
IE 7

PROS
Easy Install
Integrates w/ Nunit
Good DSL
Integrates with most xUnit
Spreadsheet Tool
Recorder

CONS
IE Only until v2.0
Requires IE to be running.
The DSL could be more intuitive.

WatiN Project Overview
WatiN started life as a .Net clone of Watir. Rather than emulate a browser, WatiN uses the browser's own API to conduct web-tests. As of this writing, WatiN only supports IE7, but they promise that the 2.0 version will have FireFox support.

WatiN is technically just an API for controlling Internet Explorer, reading the DOM structure of web pages and triggering events on DOM elements. The library includes many utility methods (something of a DSL) that make scripting the tests ahead of time fairly trivial. Tests are written in you're favorite .Net language using your favorite .Net IDE.

For my tests I used the standard Visual Studio/Nunit stack that I use for unit testing. In short I created a class library of Nunit TestFixtures and used TestDriven.Net to start the tests.

Getting Started

When you download WatiN you have several options: a .Net 1.0 or .Net 2.0 installer or zip packages. I chose the .Net 2.0 installer. The install was pretty smooth. The installer, oddly enough, does not create any shortcuts for the documentation. Since WatiN is just an assembly and no shortcuts are created, the installer seems a bit superfluous.

WatiN has no command-line runner. As I mentioned above, WatiN is intended to provide an API/DSL for automating Internet Explorer. You write tests and run them using your favorite xUnit tools.

Before beginning the Gmail tests I worked through the "Getting Started" tutorial on the WatiN site. The tutorial walks you through creating a console executable that automates Internet Explorer. Unfortunately, I immediately ran into an error:

COMException: Retrieving the COM class factory for component with CLSID {0002DF01-0000-0000-C000-000000000046} failed due to the following error: 80040154.

After playing around with the code a bit I eventually discovered that IE must be running in order to get the console app to work properly. Once I started IE the Getting-Started app ran as expected. Internet Explorer opened and navigated to the appropriate pages. I could watch each click in the browser and it was kinda nice. Now I am ready to write some fixtures to pass my Gmail test!

Bring on the Gmail

I used FireBug's excellent HTML inspection tool to find the element IDs of each button and text field I would need. I wrote all the tests as Nunit TestFixtures. This was very easy since I already write TestFixtures for all my other projects. I used the test Setup method to launch an Instance of IE and closed IE in the TearDown method. WatiN provides pretty intuitive methods for controlling the browser.

The first thing that hits me while writing the fixture is the DSL. For the most part, I really like it. But, it could be re-factored a bit. In the example below, I wait for and then enter text into the email text box:

ie.TextField("Email").WaitUntilExists();

ie.TextField("Email").TypeText("******@gmail.com");

I would prefer to do something like this:

18 ie.TextField("Email").WaitUntilExists().TypeText("*****@gmail.com");

I also discovered that WatiN requires STA Threading. The web site doesn't hide this fact either. This required a small work around in Nunit. To guarantee STA Threading in Nunit (.Net 2.0) you must define it in your test assembly's config file:

  <configuration>
<configSections>
  <sectionGroup name="NUnit">
    <section name="TestRunner" type="System.Configuration.NameValueSectionHandler"/>
  </sectionGroup>
</configSections>
<NUnit>
  <TestRunner>
    <!-- Valid values are STA,MTA. Others ignored. -->
    <add key="ApartmentState" value="STA" />
 </TestRunner>
</NUnit>
</configuration>

The Gmail tests went surprisingly well. The browser opened up. The appropriate items were clicked, textfields were filled in and my emails were sent. As with most tests, I found a lot of overlap in the web-tests. The login and logout routines were used quite often as well as other routines. In order to simplify the tests and practice good "Once and Only Once", I re-factoring the logoff and login into separate utility methods.

Other WatiN Tools

WatiN also has two other contributed tool projects; Wax and WatiN Recorder. With Wax you can write simple test suites in an excel spreadsheet (think FIT but easier) and run them directly. WatiN Recorder is a fairly robust recording tool that will help when you are writing tests for legacy code.

Final Thoughts About WatiN

I found WatiN to be quite impressive. WatiN includes about as much DOM search functionality as you could ask for. You can find anything including elements embedded in IFrames. You can trigger any event you wish. The DSL is fairly intuitive but if need be you can drop into XPath style searches. Keep your tests simple by searching for element IDs. Write your tests first and design the UI that the tests are written for. You can check out their API to HTML mappings.

For me the fact that the tests are written in .Net/Visual Studio was very nice. This allows you to leverage Visual Studio's intellisense and the full power of the .Net framework. Your testers may not find it as easy to write Nunit tests. I would recommend using Wax tests in those cases. It doesn't support the most complex features of WatiN but I would suggest your site should stay away from an overly complex UI.

While the tutorial project was refreshing in the fact that it has no dependencies, other than WatiN. I worry that those developers who remain uninitiated into the world of TDD will read this as the accepted method fr running WatiN. I would prefer to see a tutorial that employs a unit testing tool.

Most shops will have many reasons for implementing an automated web-testing tool. For me, the most compelling reason is the value it brings to your user experience design. By writing automated tests first, you are forcing your UI design to be simple. In short, the UI you create, to pass the tests, will do the simplest thing that could possibly work. This allows you to deliver "any kind of value immediately". The cool stuff (web2.0) can be added later AFTER the necessary stuff is done.

Here is the source of my test project. Don't forget to install Nunit and Watin!

Round 2?

For Round 2 I plan to test Selenium! So stay tuned....