Sunday, February 24, 2008

Webtest Tool Round-up (Round 1:WatiN)

This is the first article in a series of reviews. I plan to test a few Open Source automated Web-Testing tools. As I complete my tests I will publish the results to my blog.

The Test Procedure

The world has gone Ajax and Web 2.0 crazy. Any tool that doesn't handle dynamically changing DOMs is an instant failure. So, I decided to use one of the most Ajax intense applications I know of, Gmail. Gmail uses Ajax and Iframes indiscriminately. The developers at google seem to obfuscate ALL the HTML element IDs which means that my web-tests have to do some real back-flips to find the buttons, hyperlinks and textfields required to send messages. I have no doubt that Gmail will adequately stress the tools in this line-up.

Test Steps

  1. Log in to Gmail
  2. Send an Email
  3. Check the same email
  4. Apply a Label
  5. Archive it
  6. Check to see if it is in the label category.
  7. Log Out

Round 1: WatiN (v1.2.1)

homepage | download

Test Date


Windows XP Home
Visual Studio 2005
.Net 2.0
IE 7

Easy Install
Integrates w/ Nunit
Good DSL
Integrates with most xUnit
Spreadsheet Tool

IE Only until v2.0
Requires IE to be running.
The DSL could be more intuitive.

WatiN Project Overview
WatiN started life as a .Net clone of Watir. Rather than emulate a browser, WatiN uses the browser's own API to conduct web-tests. As of this writing, WatiN only supports IE7, but they promise that the 2.0 version will have FireFox support.

WatiN is technically just an API for controlling Internet Explorer, reading the DOM structure of web pages and triggering events on DOM elements. The library includes many utility methods (something of a DSL) that make scripting the tests ahead of time fairly trivial. Tests are written in you're favorite .Net language using your favorite .Net IDE.

For my tests I used the standard Visual Studio/Nunit stack that I use for unit testing. In short I created a class library of Nunit TestFixtures and used TestDriven.Net to start the tests.

Getting Started

When you download WatiN you have several options: a .Net 1.0 or .Net 2.0 installer or zip packages. I chose the .Net 2.0 installer. The install was pretty smooth. The installer, oddly enough, does not create any shortcuts for the documentation. Since WatiN is just an assembly and no shortcuts are created, the installer seems a bit superfluous.

WatiN has no command-line runner. As I mentioned above, WatiN is intended to provide an API/DSL for automating Internet Explorer. You write tests and run them using your favorite xUnit tools.

Before beginning the Gmail tests I worked through the "Getting Started" tutorial on the WatiN site. The tutorial walks you through creating a console executable that automates Internet Explorer. Unfortunately, I immediately ran into an error:

COMException: Retrieving the COM class factory for component with CLSID {0002DF01-0000-0000-C000-000000000046} failed due to the following error: 80040154.

After playing around with the code a bit I eventually discovered that IE must be running in order to get the console app to work properly. Once I started IE the Getting-Started app ran as expected. Internet Explorer opened and navigated to the appropriate pages. I could watch each click in the browser and it was kinda nice. Now I am ready to write some fixtures to pass my Gmail test!

Bring on the Gmail

I used FireBug's excellent HTML inspection tool to find the element IDs of each button and text field I would need. I wrote all the tests as Nunit TestFixtures. This was very easy since I already write TestFixtures for all my other projects. I used the test Setup method to launch an Instance of IE and closed IE in the TearDown method. WatiN provides pretty intuitive methods for controlling the browser.

The first thing that hits me while writing the fixture is the DSL. For the most part, I really like it. But, it could be re-factored a bit. In the example below, I wait for and then enter text into the email text box:



I would prefer to do something like this:

18 ie.TextField("Email").WaitUntilExists().TypeText("*****");

I also discovered that WatiN requires STA Threading. The web site doesn't hide this fact either. This required a small work around in Nunit. To guarantee STA Threading in Nunit (.Net 2.0) you must define it in your test assembly's config file:
<sectionGroup name="NUnit">
<section name="TestRunner" type="System.Configuration.NameValueSectionHandler"/>
<!-- Valid values are STA,MTA. Others ignored. -->
<add key="ApartmentState" value="STA" />
The Gmail tests went surprisingly well. The browser opened up. The appropriate items were clicked, textfields were filled in and my emails were sent. As with most tests, I found a lot of overlap in the web-tests. The login and logout routines were used quite often as well as other routines. In order to simplify the tests and practice good "Once and Only Once", I re-factoring the logoff and login into separate utility methods.

Other WatiN Tools

WatiN also has two other contributed tool projects; Wax and WatiN Recorder. With Wax you can write simple test suites in an excel spreadsheet (think FIT but easier) and run them directly. WatiN Recorder is a fairly robust recording tool that will help when you are writing tests for legacy code.

Final Thoughts About WatiN

I found WatiN to be quite impressive. WatiN includes about as much DOM search functionality as you could ask for. You can find anything including elements embedded in IFrames. You can trigger any event you wish. The DSL is fairly intuitive but if need be you can drop into XPath style searches. Keep your tests simple by searching for element IDs. Write your tests first and design the UI that the tests are written for. You can check out their API to HTML mappings.

For me the fact that the tests are written in .Net/Visual Studio was very nice. This allows you to leverage Visual Studio's intellisense and the full power of the .Net framework. Your testers may not find it as easy to write Nunit tests. I would recommend using Wax tests in those cases. It doesn't support the most complex features of WatiN but I would suggest your site should stay away from an overly complex UI.

While the tutorial project was refreshing in the fact that it has no dependencies, other than WatiN. I worry that those developers who remain uninitiated into the world of TDD will read this as the accepted method fr running WatiN. I would prefer to see a tutorial that employs a unit testing tool.

Most shops will have many reasons for implementing an automated web-testing tool. For me, the most compelling reason is the value it brings to your user experience design. By writing automated tests first, you are forcing your UI design to be simple. In short, the UI you create, to pass the tests, will do the simplest thing that could possibly work. This allows you to deliver "any kind of value immediately". The cool stuff (web2.0) can be added later AFTER the necessary stuff is done.

Here is the source of my test project. Don't forget to install Nunit and Watin!

Round 2?

For Round 2 I plan to test Selenium! So stay tuned....


Unknown said...

Would you be interested to evaluate ?

I am going to look at your gmail test automation and write the same with

Frederic Torres
Web Testing with C# or VB.NET

Unknown said...

After reading your post I may move incisif to the front of the list. So far, I really like the syntax you use.

Jeroen said...

Thanks for your evaluation!

About the refactoring you are proposing. This is already in place. Instead of


Just use:


WatiN will call WaitUntilExists internally so no need to first call this method before using a property or method on an element (TextField in your example). This keeps the code is clean as possible, not bothering the test writer with waiting logic.

Thanks again for your review

Jeroen van Menen
Lead developer WatiN.

Unknown said...

Thanks for the comment and thanks for the great work with WatiN! The example I gave was a bad one. Let me propose a different example.

Look at the syntax for searching out a textfield:


I would suggest a more intuitive syntax would be:


Mike said...

There is an additional UI testing tool that you may find worth while to include in your test. Called SWAT (Simple Web Automation Toolkit), this tool allows users to test both Internet Explorer and Firefox with the same script. SWAT itself is a collection of C# libraries and can be run from any .net testing harness. Currently here at Ultimate we mainly use fitnesse which is the syntax the editor uses, but we have also used MbUnit to run tests. The download includes the SWAT libraries and an IDE we call the SWAT Editor. Inside the editor you will find an interface to create tests in the fitnesse syntax, a UI recorder and a SQL test creator. We are today releasing the 2.0 version which is a major upgrade over 1.1 and the reason I haven't emailed you prior. You can find information on the tool here ( Complete documentation and tutorials for SWAT are located in the sourceforge wiki. Even if you don't include it in your test we are always looking for ways to improve the tool so any feedback would be greatly appreciated.


Mike Longin
Product Development
Ultimate Software

Unknown said...

I downloaded your source sample of Gmail test.
But there is a problem with the source line:
Element sendBtn = CanvasFrame.Element("button", Find.ByText("Send"));
It is not compiled because this function get one parameter so I change to:
Div divSend = div.Div(Find.ByText("Send")); and write-
but the Click event wasnt occured and the the message wasnt send.What do you think?

Anonymous said...

What about Selemium?
Do you test it?

Unknown said...

Hi, I wish to be a regular contributor of your blog. I have read your blog. Your information is really useful for us. I did Software Testing Course in Chennai at Fita training and placement academy which offer best Software Testing Training with years of experienced professionals. This is really useful for me to make a bright career.

Irving Martin said...

Thanks for sharing this great content here I love this post very much. aio downloader android tutuapp andriod cartoon hd apk latest version

Take My Math Class For Me said...

I appreciate you taking the time and effort to put everything together. If you are reading this, you are fortunate to have access to our low-cost online class services. Writing is a challenging task! Thank you for sharing this information with us.