Webtest Tool Round-up (Round 1:WatiN)

This is the first article in a series of reviews. I plan to test a few Open Source automated Web-Testing tools. As I complete my tests I will publish the results to my blog.

The Test Procedure

The world has gone Ajax and Web 2.0 crazy. Any tool that doesn't handle dynamically changing DOMs is an instant failure. So, I decided to use one of the most Ajax intense applications I know of, Gmail. Gmail uses Ajax and Iframes indiscriminately. The developers at google seem to obfuscate ALL the HTML element IDs which means that my web-tests have to do some real back-flips to find the buttons, hyperlinks and textfields required to send messages. I have no doubt that Gmail will adequately stress the tools in this line-up.

Test Steps

1. Log in to Gmail
   
2. Send an Email
   
3. Check the same email
   
4. Apply a Label
   
5. Archive it
   
6. Check to see if it is in the label category.
   
7. Log Out
   

Round 1: WatiN (v1.2.1)

homepage | download

Test Date
17FEB08

Environment
Windows XP Home
Visual Studio 2005
.Net 2.0
IE 7

PROS
Easy Install
Integrates w/ Nunit
Good DSL
Integrates with most xUnit
Spreadsheet Tool
Recorder

CONS
IE Only until v2.0
Requires IE to be running.
The DSL could be more intuitive.

WatiN Project Overview
WatiN started life as a .Net clone of Watir. Rather than emulate a browser, WatiN uses the browser's own API to conduct web-tests. As of this writing, WatiN only supports IE7, but they promise that the 2.0 version will have FireFox support.

WatiN is technically just an API for controlling Internet Explorer, reading the DOM structure of web pages and triggering events on DOM elements. The library includes many utility methods (something of a DSL) that make scripting the tests ahead of time fairly trivial. Tests are written in you're favorite .Net language using your favorite .Net IDE.

For my tests I used the standard Visual Studio/Nunit stack that I use for unit testing. In short I created a class library of Nunit TestFixtures and used TestDriven.Net to start the tests.

Getting Started

When you download WatiN you have several options: a .Net 1.0 or .Net 2.0 installer or zip packages. I chose the .Net 2.0 installer. The install was pretty smooth. The installer, oddly enough, does not create any shortcuts for the documentation. Since WatiN is just an assembly and no shortcuts are created, the installer seems a bit superfluous.

WatiN has no command-line runner. As I mentioned above, WatiN is intended to provide an API/DSL for automating Internet Explorer. You write tests and run them using your favorite xUnit tools.

Before beginning the Gmail tests I worked through the "Getting Started" tutorial on the WatiN site. The tutorial walks you through creating a console executable that automates Internet Explorer. Unfortunately, I immediately ran into an error:

COMException: Retrieving the COM class factory for component with CLSID {0002DF01-0000-0000-C000-000000000046} failed due to the following error: 80040154.

After playing around with the code a bit I eventually discovered that IE must be running in order to get the console app to work properly. Once I started IE the Getting-Started app ran as expected. Internet Explorer opened and navigated to the appropriate pages. I could watch each click in the browser and it was kinda nice. Now I am ready to write some fixtures to pass my Gmail test!

Bring on the Gmail

I used FireBug's excellent HTML inspection tool to find the element IDs of each button and text field I would need. I wrote all the tests as Nunit TestFixtures. This was very easy since I already write TestFixtures for all my other projects. I used the test Setup method to launch an Instance of IE and closed IE in the TearDown method. WatiN provides pretty intuitive methods for controlling the browser.

The first thing that hits me while writing the fixture is the DSL. For the most part, I really like it. But, it could be re-factored a bit. In the example below, I wait for and then enter text into the email text box:

ie.TextField("Email").WaitUntilExists();

ie.TextField("Email").TypeText("******@gmail.com");

I would prefer to do something like this:

18 ie.TextField("Email").WaitUntilExists().TypeText("*****@gmail.com");

I also discovered that WatiN requires STA Threading. The web site doesn't hide this fact either. This required a small work around in Nunit. To guarantee STA Threading in Nunit (.Net 2.0) you must define it in your test assembly's config file:

  <configuration>
<configSections>
  <sectionGroup name="NUnit">
    <section name="TestRunner" type="System.Configuration.NameValueSectionHandler"/>
  </sectionGroup>
</configSections>
<NUnit>
  <TestRunner>
    <!-- Valid values are STA,MTA. Others ignored. -->
    <add key="ApartmentState" value="STA" />
 </TestRunner>
</NUnit>
</configuration>

The Gmail tests went surprisingly well. The browser opened up. The appropriate items were clicked, textfields were filled in and my emails were sent. As with most tests, I found a lot of overlap in the web-tests. The login and logout routines were used quite often as well as other routines. In order to simplify the tests and practice good "Once and Only Once", I re-factoring the logoff and login into separate utility methods.

Other WatiN Tools

WatiN also has two other contributed tool projects; Wax and WatiN Recorder. With Wax you can write simple test suites in an excel spreadsheet (think FIT but easier) and run them directly. WatiN Recorder is a fairly robust recording tool that will help when you are writing tests for legacy code.

Final Thoughts About WatiN

I found WatiN to be quite impressive. WatiN includes about as much DOM search functionality as you could ask for. You can find anything including elements embedded in IFrames. You can trigger any event you wish. The DSL is fairly intuitive but if need be you can drop into XPath style searches. Keep your tests simple by searching for element IDs. Write your tests first and design the UI that the tests are written for. You can check out their API to HTML mappings.

For me the fact that the tests are written in .Net/Visual Studio was very nice. This allows you to leverage Visual Studio's intellisense and the full power of the .Net framework. Your testers may not find it as easy to write Nunit tests. I would recommend using Wax tests in those cases. It doesn't support the most complex features of WatiN but I would suggest your site should stay away from an overly complex UI.

While the tutorial project was refreshing in the fact that it has no dependencies, other than WatiN. I worry that those developers who remain uninitiated into the world of TDD will read this as the accepted method fr running WatiN. I would prefer to see a tutorial that employs a unit testing tool.

Most shops will have many reasons for implementing an automated web-testing tool. For me, the most compelling reason is the value it brings to your user experience design. By writing automated tests first, you are forcing your UI design to be simple. In short, the UI you create, to pass the tests, will do the simplest thing that could possibly work. This allows you to deliver "any kind of value immediately". The cool stuff (web2.0) can be added later AFTER the necessary stuff is done.

Here is the source of my test project. Don't forget to install Nunit and Watin!

Round 2?

For Round 2 I plan to test Selenium! So stay tuned....

Microsoft MVC or Is the Monorail off it's track?

For those of you who didn't know, Microsoft has been working on it's own MVC framework. I haven't used it yet since I am in the middle of testing several Web-testing tools for an upcoming article. That said, I have done a little research on the web and here are a few things that Monorail users may want to know:

• Hammett, the founder of Castle::Monorail, has helped MS by making suggestions about their implementation. See his comment below:
  

  I'm not actively collaborating with MS regarding the MS MVC. I've offered help due to my expertise with complex web sites we have built using MonoRail, I was invited to fly to Redmond and analyze their product, and discuss things, make suggestions.

• It's been suggested (in comments) that Monorail may become a wrapper for the MS Framework, if the MS MVC framework progresses well.
• The MS Framework does support intellisense in it's views. Views are written in your .Net language of choice.
  
• ASP.Net controls are available for use in Views.
  
• The MS Framework does not forbid the use of ActiveRecord but with LINQ on the horizon developers may choose it. (I haven't played with the LINQ extensions yet)
  
• Monorail is more time tested (and just works)! Also, the MS framework is closed source.
• The MS MVC framework will allow you to use any xUnit framework and any Mocking framework you choose but, as of this writing the MS solution has nothing nearly as kewl as the Monorail BaseControllerTest.
• I can find nothing about IoC support in the MS offering.
• Some people believe the routing engine in the MS offering is easier to configure. I'll have to test this assumption.

I personally have no plan to switch to the Microsoft framework yet. I DO believe the MS Framework will introduce MS developers to how great MVC can be. I also believe that Microsoft will slowly position the MVC framework as it's preferred web-app architecture. The ASP.Net architecture, while successful, was a fundamentally flawed offering that should be abandoned quickly.

As Microsoft moves into this new territory we'll start to see a subtle change in how they do business. The .Net framework was their last major move ( just had it's 6 year anniversary). The latest signs of change are already out there; LINQ, extension methods, the MVC architecture. Things to look for now; Redesigning the Ajax.Net library, functional programming with F#, greater support for DSLs.

Some Interesting Links

• Code Rant: Writing Unit Tests with the MVC Framework
• Hammett's article about his visit to MS.
• Oren discusses protected methods in the MS MVC.
• Oren discusses the ControllerActionAtribute. This has alreay been removed by MS...
• Ryan doom asks which Framework to Go with...
• The Castle::Monorail site.
  

Let me know if you have any other questions. I'll try to answer them as I play with the new framework.

Getting the Data out of my Controller Tests

While working on PlanningPokerOnRails I started writing a unit test for my LoginController. The Monorail BaseControllerTest makes this easy. I am also using Rhino Mocks and Nunit. Here is the test:

The test is very simple. First, it sets a single recorded expectation that a Redirect will occur:

34 using (mocks.Record())

35 {

36 // First redirects to the index page

37 Response.Redirect("/Home/Index.rails");

38 }

Next, the playback is run and the code emulates a few things that users might do...like login with Request params for a ReturnUrl:

39 using (mocks.Playback())

40 {

41 Request.Params.Add("ReturnUrl", "/Home/Index.rails");

42 loginController.Login("dan", "danny");

43 }

Writing the Code to Pass this Test

Next I wrote the CardPlayer component (my user) and the LoginController. Seeking to do the simplest thing that could possibly work I merely had the CardPlayer inherit from the ActiveRecord base.

This made writing the controller easy, the ActiveRecord base has plenty of good methods for instantiating. I only have to call the FindOne method on the CardPlayer:

104 public static CardPlayer FindCardPlayer(string UserName,

105 string Password){

106

107 CardPlayer user = FindOne(NHibernate.Expression.Expression.And(

108 Expression.Eq("Username", UserName),

109 Expression.Eq("Password", Password)));

110 return user;

111 }

So why all the Problems?
I never intended to initialize ActiveRecord, in my tests assembly. I believe that test data should only be used for Acceptance Tests. It turns out that the test fails because no ActiveRecord config or initialization exists.


How do I solve this?

I forgot to implement good Dependency Injection. The way I would normally grab an instance of a component (like the CardPlayer) is through a service container (or repository). MVC architectures generally have two types of Model objects; Components (like the CardPlayer) and Services (like the CardPlayer Service I should have written). The services usually act like or wrap a repository of Components. In my opinion, ActiveReecord already provides the repository.

It turns out my solution is to write a CardPlayer Service container. This object will go into the IoC container (winsor) and be automatically injected into whatever controller requires it (like the LoginController). The service will use the ActiveRecordMediator to instantiate new CardPlayers and I can remove the ActiveRecordBase from my CardPlayer component.


Wait why does this help?

Well by forcing all instantiation to go through injected Services your tests can easily inject a Mock Service (I use Rhino Mocks) and tell the Mock to provide predefined instances of components. Since you have mocked the service layer you no longer need ActiveRecord in your tests!

Keeping the Data
Now that I've convinced you to write a service layer, I am going to tell you that you may not need it! While you can mock the Services going to your controllers you cannot easily get rid of ActiveRecord when testing the services. For most simple lookups a test may not be appropriate but in many cases your component lookup up and other operations will be non-trivial. For these cases you will want to test the service layer and you may have to initialize ActiveRecord.

You could decide to Mock the ActiveRecordMediator or you could use an approach similar to Oren's.

I'm Confused...

I admit that this is some advanced stuff. I promise to come back to this post and update it with more concise information.

Please leave comments on ways I can do this without a service layer. Thanks!

Is this a SourceForge Killer?

I doubt that the Google Code Projects Hosting will compete with SourceForge anytime soon but it does put Google one step closer to being the next SkyNet. I've been hosting one of my recent open source projects there and been quite pleased. Here are a few features of note:

• It's FREE! (So is SourceForge by the way)
• Provides a Subversion Repository with a really slick online browser. Check it out here.
• Most of the interface is a just a frontend to a smart wiki. So you can use wiki code just about anywhere.
  
• You can add users into different roles.
• The Issue Tracker is relatively complete and fairly customizable since it's a highly specialized wiki. It allows you to assign tasks and track the workflow in your own way.
  
• Plain old Wiki. For all your other needs there is a fairly robust wiki.
  
• Package Download system allows you to upload your latest builds, documentation or whatever.
  
• 100MB of storage for your source (you can request more).
• Google account logins are used. As a matter of fact an Google Account login can be used.
  
• Limited customization of the look and feel.
  

What has impressed me the most is it's barebones nature. No banner ads poping out. Searching is fast. Most of all, it seems to be growing in features. When I first started my project they had no source browser available. Now it has one of the best SVN browsers I have seen (FishEye would be better). I also feel like they are driven by user's needs. If enough people request a feature then they just build it in!

Finally, I really think that Google's approach to software development is very Agile. They always build the simplest thing that could possibly work and then add new features as users ask for them. We could learn a lot from this approach.

New Pattern: CPT (Can't Polish a Turd)

I recently read a hilarious article, "Asshole Driven Development." The author, Scott Berkun, takes a cynical look at the abundance of acronyms in the software engineering field. Along the way he uncovers a few insights that, while very cynical, ring true. In addition to Scott's funny acronyms many users have added additional goodness. One anonymous post read:

How about CPT (Can’t Polish a Turd) development, This is where management cannot understand the need to stop and fix/clean up/refactor or start again.

I saw this post and wondered if we hadn't worked together! I've polished many turds in my time, a few were my own, and let me tell you...it's not easy. The fact is, these turds never really shine. Sure, sure...wet em down a bit...put em in the right light and they seem to shine. Heck...we have well paid people (some brilliant) that spend most of their time putting turds, in the right light. But, in the end a turd by any other name is still crap.

So why all this infatuation, with turds? In my experience it can be broken down into several very bad reasons:

• Turd Love Affairs: This turd was once your next big thing. It would solve all the world's problems and make the company tons of cash. Somewhere along the line it got off track for various reasons (see the above article) and no one wants to admit it. These love affairs can be identified by an organization's death march development of a product that almost no one likes.
  
• WSIA (We Sold It Already): WSIA is a surprisingly common problem. It's in the field and someone has paid for it. Because it's a turd it takes all your manpower to constantly tweak and thus no developers are available to work on a new version or refactor the old turd. Companies with this problem are usually immature and don't adequately plan for the software's lifecycle. (BTW: XP solves this problem since it assumes the software is never done)
• Management Mistrusts the Engineers: I imagine them sitting around in a smoky room saying, "well those stupid nerds are whining about the XYZ again. Tell them we'll start on the new version in a month...that'll shut them up." Of course that month becomes 3 then 6 then the project is scrapped or the company goes out of business. "What!? no one wanted to buy a shiny new turd?!"

So, what can we do when we find ourselves in this situation? The first thing I always advise when someone feels workplace dissonance is to step back and remember, "everyone in the organization is probably doing what they think is the best thing for the product." This is important because we have to know that our peers and those above us are trying to make the project successful. People aren't always going to agree on the best plan of action.

The first thing you should do is try to understand why the organization is doing what it does....polishing turds. In most cases you just need to look at the project from another person's perspective. How long have they been working on this project? How emotionally attached are they? What fears do they have of losing customers? Once you understand the reasons for polishing turds you can begin to form arguments that support your course of action.

Finally, most turds started out as great ideas. Many of them can be fixed or re-written. Look for that original, good idea. In this way the true brilliance of the original great idea can shine through!

You can probably think of more reasons companies shine there crap...feel free to post a comment.