Agile Project Management Tools are doing it wrong

Introduction

I have noticed many bug tracking products attempting to add "Agile" project management to their offerings. I use the word "attempting" because I have yet to use one that really works. I also find it frustrating that many of these same products are setting aside useful features (like case commenting permissions in FogBugz) and enhancements in favor of failed attempts at making an agile PM tool. Here I'll try to outline, as I see them, the problems faced in trying to make an AGILE PM TOOL.

Users Believe that they want an online Card Board (or kanban)

Unfortunately, most of us have believed for a long time that a nice Trello style online board would be perfect for managing an Agile Project. So far, IMHO, none of the tools have gotten this right AND I doubt they ever will. The great thing about a card board or board full of sticky notes is that you can grab the card, pass it around, set them out on a table, return them to board or put them on another board very easily. All of this while having a full visceral interaction with the rest of the team.
The electronic card boards fail to allow uninhibited control of the cards. You have to click then drag a card somewhere AND unless you have a huge (45+ inches) touch screen monitor many of your cards end up falling off the screen or are unreadable. We always do our planning using physical cards. Even though the tool we use (FogBugz) has a so-called Agile Planner built in. It's just to unwieldy to use BUT we keep telling these companies that this board is what we want.

Suggestion #1: Stop trying to create online card boards and focus on managing the backlog (see below) or workflows (see below). When we ask for the card boards we really don't know what we're asking for.

Agile Projects all follow different workflows

Agile teams rarely have the same exact workflow. The workflows differ from team to team for good reasons. Differing product schedules, team size, organizational demands, etc. The tools that try to prescribe a single workflow, do so at the loss of potential market share. The tools that have a super flexible workflow editor don't always go far enough or it becomes difficult to see what the workflow really looks like. The tools that try to have a one size fits all approach tend to be berated for not really meeting anyone's needs.

Suggestion #2: Workflow is one place I think these tools can make big differences. If you are going to have a workflow editor then it needs to be robust enough to allow good workflows to be described. Managing the workflows is the absolute best use of these tools. It allows developers and users to easily move cases along the pipeline that your team has described. It does suggest "kanban" but as mentioned above it doesn't need to be visually like kanban, just conceptually. It's really just a pipeline approach.

The BACKLOG is what really needs to be managed

Some backlogs have tens of thousands of user stories in them. Managing that backlog effectively and allowing product owners to more easily garden is a missed opportunity in most of these tools. Once again no visual representation of cards is going to help here. It's too cumbersome and time consuming. The ability search and order stories in a backlog based on factors such as priority, area and cost are very important BUT also being able to tie these stories to epics on a road-map is even more important.

Suggestion #3: My backlog order rarely drives the priority of work. We follow a constantly changing road-map of EPICs based on current business needs. Being able to associate stories with Epics and then to a product road-map AND have that automatically fix the backlog order would be THE KILLER FEATURE! Being able to bring up a road-map of EPICS that I can modify then have the backlog automatically changed would be amazing.

Story Points versus Man Hours

Fogbugz just recently added story points as a second class citizen but still has no good burn down chart capability that uses story points. Most tools do have a burn down but the problem we always face is that some teams use hours while others prefer story points. To further compound this is how and when do you receive story points (or hours) for a story or task? Some teams allocate points once the developer has completed the work, other teams only allocate once a QA review has been done and still other teams use completely different methods. 

Suggestion #4: This is a point I am less confident about. My gut says this capability can be fleshed out if attention is given to it (see suggestion 1). This may be a matter of allowing point allocation to be defined as part of the workflow. I believe that would work but once again the workflow editor must give a solid representation of how that workflow will be enforced.

Final Thoughts

At the end of the day my preference is for a basic issue tracker that allows me to stay in continuity with our end users and allocate engineers/QA to stories. We've used the FogBugz API extensively to build our own burn downs and other information radiators. For us having the programmatic access to our stories is of greatest importance, along with the added workflow capabilities. Enhancements in the areas mentioned above would be of most value to us.

Ridding Your Garden of Bugs: Gardening the Backlog

I'm not sure if I agree with this "article" but, it is somewhat related to a discussion my boss (Dr. Rob Miller) and I were having earlier today. Ideally, we would like to focus every third (I just picked a number) release on "bug hunting" or finding and fixing bugs. Unfortunately, bugs are often a debt we live with for a long time, in the face of progress and new features. I have known few products that didn't have some glitch and the promise of new features often takes some of the edge off of these glitches.

Conversely, some bugs can leave users with such a bad taste that they never quite trust your software again. In the article above the author describes a bug that was reported fixed but still remains broken....after 2 years. Furthermore, as more bugs slip through the cracks a system can slowly become more and more unstable.

This suggests that, when planning product releases, stakeholders should balance the need for new features with the need for a stable system. I believe that, by better "gardening" a product's backlog, most products can avoid these pitfalls. So, here are a few points to consider when gardening your backlogs:

• Are stakeholders keeping and gardening personal backlogs? When I say, "personal backlog" I am referring to the features, enhancement and bugs that each stakeholder or program manager would like to see implemented. At every release planning meeting these "personal backlogs" should be gardened and brought to the table for consideration in the "product backlog".
  
• What have users been promised (explicitly or implicitly) and what is the impact of leaving those promises unfulfilled?
• Do you know how "real" users feel about your software? How do you collect this information? Sometimes, we get ONE very vocal user (see above). Sometimes we have a quiet majority. When should a user's/client's opinions impact the requirements for a release?
• When is a "bug hunting" expedition called for? Is there some threshold or event horizon? Do we have a good litmus test?
• Finally, is there a single decision maker (the one ringable neck) for the product backlog? Someone to take all of the personal backlogs and prioritize them. In XP this would be the On Site Customer and in SCRUM this would be the Product Owner. I suggest that it NEVER be your development team. This person maintains the backlog, the vision and the roadmap.
  

 It's not likely that you will squash all the bugs in any non-trivial system without writing some questionable code. So, you should take the time to test drive your software. Also, watch others "test drive" your software. This advice is not just for developers. In fact it's doubly important for non-developer stake holders and Decision Makers to "test-drive" the software. Kick the tires, check all the instruments. How's the suspension? Is it idling rough? Take it to a mechanic (engineer) and see if any maintenance is due (technology debt). Next time you prioritize your backlog you may have a different perspective.

Taking the time to garden your backlog will leave you smelling like roses. While just letting things grow out of control could have you digging in the weeds!

Writing User Stories and Pragmatic XP

Writing User Stories

Part 1 Part 2 Part 3

My previous post in this series generated a lot of commentary on the XP list. I honestly appreciate everyone's insight....it can only make me better.

My intent (with respect to this series) was to suggest other approaches to writing stories. The series is aimed mostly at helping users. I try to make these approaches simple. Some users will find a "by the numbers" approach easier to wrap their mind around. The 5Ws method may also provoke a deeper analysis of the story as users try to describe each W. I also believe that some developers and testers might appreciate a more predictable user story. In the end though, users can bring us any scribble. Be it on a card, a sticky note or a napkin and we will do our best to make it a reality.

The series was never meant to be a holistic commentary on writing stories. As a rule I try to avoid anything holistic. I do, these days, try to be pragmatic and a bit understated.

When I first started down the path of XP (uhg, sounds way too cult-like) I worked in a shop with no formal method for developing software. When I discovered XP, I wasn't looking to be agile....I was looking to be methodical. My first instinct was to codify as best I could a fully functional set of XP policies and procedures. What mostly drove my efforts was a need to justify our wacky ideas (ie pair programming) to what was and primarily still is a government contracting company.

It took me several years of struggling to realize that I missed the whole point of XP. Yes, XP does help you to be more methodical, but without the formalized process. XP's informal nature is what makes it "Agile". The practices are not rules but guidelines. The coach doesn't drive the bus (the team does) she simply points out the pot holes. Finally, this all works because each person values good software over the process. The point? We write and discriminate our user stories the same way.....with a lot of guidelines and very few rules.

I wish I could say that I gained all of this insight many years ago...but I didn't. I really only got it in the last two or three years. One day I had to stop talking and start listening. To that end, I can mostly thank the insights of two of the smartest people I know (Gareth and Jason) and the never-ending patience of my boss Rob.

In the words of the "Wyld Stallyns"...."Be excellent to each other!"

Programmer at War (in Iraq)

Most of those who read my blog are probably unaware that I am in Iraq. Normally, I am a Software Engineer/Chief Systems Architect for an Oklahoma based company. One weekend a month I am an Officer in the U.S. Army Reserves. Last summer I was deployed to Iraq.

I am a Platoon Leader in a Medium Truck Company. It's quite interesting how different my civilian and Army jobs are. In my civilian job we are always going forward with new developments. In my Army job, each mission tends to be organized similarly and entail doing the same task over and over again. Although, sometimes, the enemy has a say in that.

In some ways though, my jobs are very similar. There is a great deal to be learned from my experience in the Army. Being an "agilisto", I really value communication and people. In the Army, the most successful leaders learn to communicate very well and place the needs of their soldiers before all else.

On an agile project we plan and deliver value incrementally. This allows teams to embrace change. In the Army, it is understood that your enemy, the weather and the terain have the final say-so in all plans. Unfortunately, just as in the software engineering world, not everyone understands this concept and are unable to embrace change.

I dealt with this recently. We were planning for a mission. Our higher echelon wanted us to push out on a certain day. I felt we could do it. The Platoon Sergeants had a "feeling" that we could not push out until 2 days later. Usually , in the agile world, the nod would go to the Platoon Sergeants (the team).

99% of the time I would agree. Although, sometimes, we are presented with time sensitive requirements. I attended a workshop several years ago at AgileXP, that talked about Advanced Estimation. They suggested that, in some cases, you have to push the envelope to get requirements met on time. I felt this was a time to push the envelope.

Well, since I am the acting XO, I argued my point. Eventually, I was able to convince the Platoon Sergeants that we could "try" to meet the requested date. I suggested, we would keep our higher echelon (the customer) informed by passing up incremental updates and letting them know the risks upfront. As the date approaches, and things aren't looking good, we give them the option of changing the operational scope. If the scope is what they value most then we suggest they push the date to the right.

Ideally, we want to stay pragmatic and plan our time accordingly, but in a few cases we have no choice but to work out some solution. A common argument from the Platoon Sergeants was that our higher echelon would consider this a solid commitment and our hands would be tied if something went wrong. Surprisingly (not really) our higher echelon was very understanding and, like everyone in Army, was prepared for a contingency in which we were unable to meet the mission.

Writing User Stories

Writing user stories should be easy. The reality is that most users don't know how to properly write a user story. Suppose a client asks you to write a wire transfer subroutine for systems at two different banks. (below I will highlight proper Stories in green and improper stories in red)

The client says...Write a wire transfer subroutine for systems at two different banks.

While a need is expressed, I would submit that this is not a User Story. What's been stated here could be the beginning of a vision statement, perhaps for the next release or the next Iteration. Let's try to distill this a bit.

So the client reconsiders and says.....System A should use a web-service exposed by System B to electronically wire funds.

A common pitfall in writing User Stories......getting too technical. The statement above assumes that web-services will be used and it may be perfectly logical to assume that they will be used, but let's examine what a User Story is really intended for. First and foremost is the "USER";specifically, what actions a user will perform and what she will experience in return. The examples above don't expose any User Experience information. Second, is the "STORY"; how the user gets to this point, what the user expects to happen.

A user, wanting to transfer money to account XXXXX, logs into bank A's system She selects the wire transfer option. A message asks her to enter the account number and bank to transfer to (bank B). The system then asks the user for the amount to transfer. After submitting the amount the user may log into bank B and see the new wire transfer as a deposit.

This example gives us more insight into how our client intends to use the system. This provides a better foundation for acceptance tests. It also doesn't assume any particular technology. XP projects have an evolutionary design. So, whether or not the system uses web-services is unimportant.

With our story in hand we can begin to develop acceptance tests. Once acceptance test are written we can begin the process of costing and eventually developing the story.

Further Discussion
While users should avoid getting too technical, users should also feel free to express common UI elements. In most cases they have a good idea of how they want to do their work. If you have user experience people or just smart developers you can help suggest alternatives. The point is to get insight into how a user "uses" your software.

What you DO want to avoid are stories that specifically suggest infrastructure or service elements. Things that throw red flags for me are terms like "Web Service", "Array", "SQL Query", "Stored on the Hard Drive".

Checkout Part 2: Writing User Stories using the 5Ws.

Here are some other good references on writing User Stories:

• Writing Good User Stories
• Six Features of a Good User Story - INVEST Model
• C2's User Story Page
• User Stories Articles
• Gardening the Backlog (for stories)
  

Planning Poker on Rails

I'm starting a new Open Source project. It will be a Planning Poker game for distributed teams. Here's the blurb on the project site:

The planning poker on rails website will allow distributed teams to play planning poker. It will track user stories and the points assigned to each story. It should also allow for quick access to archived stories as references for future planning games.

Each player should be allowed to place a card (estimated effort). Once the cards are all placed the application will reveal the estimates to everyone. If no concensus is made then each player will have the floor to make comments. As each player makes his or her comments the application will move to the next player. Once all players have no more comments OR a player presses the VOTE NOW button. The computer will take another vote this continues until concensus is made OR all player agree to table the vote.

Before you google it...YES another project already does this. So, you may ask, "why re-invent something?" Well, here are a few reasons:

1. Of course I think I can do it better!
2. I don't think the other is open source....just free to use.
   
3. I really wanted to play with google's code projects.
4. I wanted a project for which I would be my own user.
5. Did I say I think I can do it better?
6. I wanted something that begged for Ajax and Prototype stuff.
7. I wanted a project that had users interacting in realtime over the web.
8. I really like using Planning Poker on my teams and I wanted something that was designed around "MY" rules for the game.
9. Just want to play with a few technologies.

Wana check it out? Go Here!

Wana help? Email me!

Wana check out the competition? Go Here...

Bashing Agile / XP

I just read an article that takes a very cynical view on XP and Agile methodologies. I wasn't allowed to comment without signing up for something so I will comment here.

Check out the post "On the trail with the Cowboy Coders."

I think the author misses the point of Test Driven Development (TDD) and the XP practices, as a whole. His suggestion that XP is "Cowboy Coding" couldn't be further from the truth. In fact (this is always said) XP takes a great deal of discipline. All of the practices and values must work together or you will never reap the benefits of XP. In particular, TDD is not merely a way of making sure that your code is tested. This is a common misconception amongst those who have been newly introduced to TDD.

The reality is that TDD serves as a design tool. Now, XP does "shun" up-front formal design. But XP does not ask us to do "no design". The design is expressed differently....the design evolves. Let's take a look at how TDD helps our designs evolve. First, users write Acceptance Tests. Properly written acceptance tests are the XP team's first expression of a system's design. These tests are used as a road map for what the user should experience when interacting with the new system. Second, we have unit tests. Unit tests stress those portions of the system that are invisible to the end user. The scope of Unit Tests should be determined by the team...for instance, we tend to write a test fixture for each discreet component of a system. Finally, we write tests BEFORE we write code. Once we have a failing test we begin writing code until the test passes. It sounds strange, until you realize that the tests are more than tests...they are the design document for our new system.

Ask any XP team, it takes a lot of discipline. Most traditional teams save the tests until the end. They see the tests as something that would be "nice to have" but not really necessary. In general they just never get written. This coupled with another reality, that most upfront designs go out the door after a week or two, makes a pretty strong argument for using TDD as a design tool.

Now, before I conclude my little rant, I must remind everyone that XP/TDD does not happen in a vacuum. All the practices need to be applied. Remember "evolutionary design" happens as you go! Tests should be written when you need them not just because you want a bunch of tests. So please don't run out and write a ton of tests for crap you'll be working on next year. I guarantee that your requirements will change and you will have wasted a lot of time!

What is Important (Warning: Disconjointed Thoughts abound)

Well, It's been a while since my last post...trust me I've been busy!

I've been reading quite a bit about Agile Developmet trends. Always ready to learn some new nugget of goodness. I always find myself asking, "What is really, really important to an agile project?" Is it simplicity? Test Driven Development? Pair Programming? Evolutionary Design? One of so many other values and practices?

Of course my first response is, "it's all important, XP requires a tempered application of ALL the values and practices."

We have so many around us that don't like this answer. They wonder, "where are the controls? What is the process?" My answer is, inevitably, "the team is the control and the process. They are guided by a coach and a good faith effort to provide regular software releases."

I must admit that, in business, we need to"know". Know when the software is going to be done. Know when the software is not going to be done. Here, I argue that most methodologies fail to effectively predict these things. So, why not select the methodology that wastes less time on fruitless planning and more time on fruitful development?


I am a huge fan of pair programming. Actually I am a huge fan of many of the more painful practices;TDD, Integrate Often, Simplest Thing That Could Possibly Work, etc. I think they force us to do the right thing. I've also found that most of the practices hinge on pairing. Navigators always seem to be sticklers for doing the right thing.

As usual my thoughts always return to simplicity. I like to think of simplicity as the "golden value". It's importance can never be overstated. Everything that goes wrong in software development can somehow be traced back to some complexity. Simplicity makes the code easier to write. Simple requirements require less time to design. Simple designs require less time to implement. Simple code is easier to integrate and maintain. Easier means faster and faster means a quicker and greater return on investment.

So what makes something simple? Is it simpler to just drop a datagrid on a webform or to use MonoRail views and Brail? What about IoC containers? Is dependency injection complex or simple? I bring up these points for a reason. I read a blog with a pretty half-hearted flame about these same things. I must tell you that I am easily side-tracked by these sorts of discussions. I want things to be simple! Shouldn't WebForms controls be simpler? Or why go through the through the trouble of implementing an IoC container? I'll try to answer....

I could just say, "these are the patterns we choose to use and it makes the code simpler to maintain and refactor (remember refactor mercilessly)." This works academically, but the real truth is that it "feels" better. The WebForms designer still has a cludgie feel to it. The code it generates is nasty. Also, the MVC framework takes away 90% of the design decisions and, as a framework, is simpler to wrap your mind around. The IoC containers simplify the bulk of the coding...the IoC container is complex...but it's done already! Finally, Dependency Injection really supports "once and only once" and allows you to design objects that do or express ONLY what they are meant to...that's simplicity.

My next post I hope to expand on some of the "big words" and ideas above. Please comment on what you believe is most important!

Tactical Scope Creep
In my experience, scope creep is one of the greatest threats to the completion of an iteration. I would even suggest that the most dangerous "scope creep" is the kind inflicted by engineers.

Most discussions of "scope creep" tend to focus on the customer inflicted variety. For agile teams, working iteratively, this sort of strategic creep has been managed (see embracing change). The creep we don't always catch is the creep at the tactical level.

Have you ever been faced with a simple story and somehow ended up making a Rube Goldberg machine? Damn the schedule and designed a complex framework in hopes of using it again later? Regretted wasting time on a complex API that did no more than paint your project into a corner?

In the end, all of these cases are examples of one thing...tactical scope creep. I myself have never fallen prey to these vices (yeah right). Seriously, I can't count the number of times I was going to write the next great framework that would solve ALL future problems......I wish someone had been there to say YAGNI!

So, what do we do? We apply the XP practices and principles.....duh. Remember, XP is a somewhat wholistic methodology. You MUST plan iteratively, have users write stories and play the planning game. Ok, disclaimer out of the way, here are a few XP principles/practices that will help you conquer tactical scope creep:

• Pair Programming
  No sh@t...it works. One navigator, one driver. Two minds are working on the code and the design. This creates a natural 2-way sounding board for new ideas (as opposed to having NO sounding board in normal programming). Bad ideas and transgressions of the other practices are identified and squashed quickly.
  
• YAGNI
  You Aren't Gonna Need It! This one is a no-brainer. If the story states "Users can write coments to blog entries", then write a solution that accomplishes this! Notice that this story doesn't suggest that users can view comments. It doesn't suggest that authors can manage comments either. SO...don't worry about it! Suggest that a user write stories for these other features. Writing very small stories is a good practice anyway. Small stories are easier to estimate and easier to focus on. Of course, differentiating between YAGNI and flexibility is no science.
  
• TSTTCPW
  The Simplest Thing That Could Possibly Work. This practice dives deeper into the tactical design than YAGNI does. TSTTCPW asks us to simply make a duck a duck. If writing a file to the disk is simpler than writing it to a database....I guess you should write it to the disk. Later if requirements change, and you have properly encapsulated responsibilities, you can refactor to a database. Of course, like most of the XP practices it's application is subject to team arbitration. If a pair can't agree then let the WHOLE TEAM (see below) decide if something really is TSTTCPW.
  
• Simple Design
  Over engineering a problem is the most frequent cause of tactical scope creep. If you can get away without a factory/service tier then do so. If you can use an O/R mapper (such as ActiveRecord) then do so. Stay away from glitzy user interfaces....wait until someone writes a story that requires a "glitzy" user interface.
• Whole Team
  Especially the idea of putting all engineers in one room. Let the team police itself. Inform the other pairs when you are about to make a complex design decision. If your argument sways ALL of them then go for it!
• Fail Fast
  Ok, this is actually a Test Driven Development (TDD) practice but it will help. Get something up and running quickly. Don't let yourself ponder the intricateness of the problem at hand. Over-engineering almost always starts in the "minds" of smart people.
  
• Small Frequent Releases
  Your users will love you AND you will love being loved! That killer new framework would have to be pretty damned good to give you the same validation as a satisfied user. More likely the time you waste designing the killer API will overshadow any benefit derived from it.

Above all, work with your users and testers constantly to improve your understanding of their needs. Also, avoid becoming a simplicity/YAGNI drone. XP is about putting good software before the methodology.

Now, some things can slip through the cracks. Most will be unimportant. Some will be VERY important. Here are some things to keep in mind when you are getting ready to drop the YAGNI bomb:

• Security
  This is the most serious potential problem. Security, especially on the low level, almost never makes it's way into a user story. I would suggest putting a security expert on your QA team. Have them write stories describing the security holes that require attention.
• Accesibilty / Multi-cultural / Usability
  Once again these can be handled by placing an expert on your QA team.

Another option may be to track a set of "perpetual stories".

Well, I hope I haven't wasted too much of your time. Please feel free to comment, complain or flame!

5 Levels of Planning

I just read an interesting article that breaks down the Agile planning process into 5 levels:
(This is slightly modified)

1. The Vision
2. The Roadmap
3. The Next Release
4. The Next Iteration
5. The Next Day

Essentially, each agile planning session occurs in one of these levels. With each level dependent on it's predecessor. Ideally, you need a vision before you have a road map, road map before you plan the next release...you get the idea.

This really made sense to me. For years now I've used a somewhat complicated model to describe what it is we do. This model, oddly enough, can be easily distilled into this 5 level process.

In the table below I have identified each level of planning with WHAT being the expected artifact and the WHO generally describing the participants:

Level	What?	Who?
The Vision	A Vision Statement	Chickens
The Roadmap	Roadmap (duh)	Chickens
The Next Release	Release Plan/Stories/	Chickens/Pigs
The Next Iteration	Iteration Plan	Pigs
The Next Day	Engineering Tasks	Pigs

In the first level, The Vision, your goal is to provide a vision statement. Each shop will decide it's own requirement but I general recommend a condensed vision statement that can be easily hung on the wall as an "Information Radiator" (Remember...big visible charts). The Vision should be agreed upon by all stakeholders (chickens) and written without concern for technology. It should also convey some sense of purpose, since it is intended to drive the development process.

Once you have a Vision your team of chickens is ready to start working on The Roadmap. The Roadmap should also be brief enough to work well as a Big Visible Chart. Ideally your developers will keep the roadmap hung up in the shop. As items on the Roadmap are completed they can be checked off. The Roadmap will list, in order of importance, the vision for future releases. I recommend a bulleted list with each release defined by no more than 2 or 3 sentences (1 is better). As with the vision the roadmap should have a sense of purpose and as each item on the roadmap is completed you'll likely want to revisit the vision (it may have changed).

With your Roadmap written you'll want to start planning your Next Release. Planning a release is no simple task. You'll want the input of all your chickens and pigs. I'd recommend, first, relooking/revamping the vision for the next release; something may have changed. During release planning the chickens should be presenting stories to the pigs. These stories fall into 3 categories; enhancements, new features, bugs. Chickens should only consider enhancements and new features that fit into the vision for this release. Any stories considered a bug may be considered, regardless of the vision. Beware what you call a bug....just because you "don't like the way something works", does not make it a bug.

During Release Planning you should play the planning game. Chickens write a story, pigs estimate, chickens rewrite.... The final result should be a set of estimated stories with user values and generally well described acceptance tests. This may take several days. Pigs should feel free to throw back any story that can't be estimated. Chickens should make sure to stay within the vision for this release (or revise it).

Once you've hammered out your Release Plan, you're ready to begin working on the Next Iteration. Your original estimates of a story should not change unless the story is changed. Tracking progress should be done via your velocity (that is outside the scope of this article). At the beginning of each Iteration you'll select a set of stories to complete. This should be a sort of "do or die trying" commitment on the part of the Pigs. If you've tracked your velocity properly then making a commitment should be easy. Put story cards somewhere visible or use a large chart to track progress. No story should be started without an acceptance test and no story is complete until the test passes.

As each iteration progresses you'll constantly be planning The Next Day's work. Each day engineers/developers should pair up and select a story to work on. Pigs will sometimes hold design sessions with CRCs or write engineering tasks to remind themselves what needs to be done next. A coach should be available to cheer them on as the stories slowly disappear and the new builds start to appear. Always make big visible charts and information radiators to track progress of tests, builds and stories.

As you get into a rhythm your team should start to move through these stages naturally. For many companies distractions are a way of life. Try your hardest to insulate developers from these distractions and you'll see your vision realized much faster. If distractions are unavoidable them just realize that your velocity will suffer. As long as distractions are common your velocity should adequately adjust for them.

Automate, you must!
Recently, at work, we rolled out a new version of one of our major web applications. Until recently rolling out changes was a major undertaking. Over the past few years it's evolved into a much more reliable system that takes about 11 minutes!



Here is a quick chronology of the evolution of our deployment system:

1. Pre-deployment system
   Believe it or not our earliest versions of this product (many years ago in a galaxy far far away...etc) had no deployment system, let alone automation. Unless you considered me or some other guy making a change and Ftping it to the live site for testing, a deployment system. These were carefree times with few users to be annoyed by the occasional bit of downtime. Also, we worked in mostly interpreted languages (ah the good old days!)
2. Let's get a testing server!
   As we grew and more users were around to pay attention to our untested new "features". We decided a test server was in order. The first iterations of the test server were basic configurations of the original application. Nothing was automated but at least things got tested...mostly.
3. Let's automate
   This was a though time. I made a lot of enemies during the year or so it took to automate deployments. No one wanted to spend the time required to automate the system. Even those who saw the value in it were afraid of those who did not. Luckily, my boss, a man with surprising good vision for these things, was a willing advocate for change. At this point I should mention that this was about the time I was placed in charge of several other software teams and I was proposing a sweeping change for them as well. Let's just say it was ugly.
4. Let's REALLY automate
   About the same time we were automating our builds and deployments using NAnt, I was really tied up in the school of thought that was trying to address where databases fit into the Agile methodology. Databases present many problems that aren't handled well by most agile practices. Historically you try to design databases that aren't going to be very volatile. Agile methods suggest that everything should be capable of some volatility! Refactor this, throw this other thing out...etc. So, my problem was this, how do we allow the database to be volatile and still keep data safe? Well, once again I shifted some our resources to non-moneymaking endeavors. We developed an in-house database installer system that automates the deployment or upgrade of our databases. It works quite well is likely one of the greatest timesavers we have!
5. Refactor the Automation
   Over the past few years now we've learned a few things. We've switched a few things around and reversed a few assumptions we've made in the past. Through perseverance we've become so confident in our tools that we have fully embraced automation. We intend to automate everything. We've written a few custom tasks for NAnt and discontinued the use of WIX for many things. Trying to use Windows Installer was painful and it wasn't always clear exactly what it was going to do. We decided it just wasn't worth the effort and stuck to a custom system. Another major shift for us was the realization that our build and deployment scripts should be able to run on any machine, not just the build server.

Along with automating the build and deployment we have automated the Unit Tests and Code Coverage. We are capable of automating the code documentation but frankly it takes way too long and the resulting documentation is somewhat daunting. I'm considering some ways of producing cookbook style documentation where you look up the task you're wanting to perform rather than looking up objects/methods.

We still need to automate our acceptance tests. This is something of a holy grail for me. We've played with FIT.Net but still haven't been able to fully commit to it. Either way we are on the right track and I think we'll only get better!

mmmm...code smell this has

"mmmm...code smell this has". I wish it was original but a co-worker of mine came up with it while we were making up a fictional Agile Jedi (yoda kinda). Some others were, "YAGNI this is" and "Refactor you must".
Having pulled our team, some kicking and screaming, into the Agile world. I have to wonder what has it gotten us? Also, what do we still need? Hopefully, over the next few posts, I can explore this. I know we're not perfect but I believe we're on the right track.


I'd like to start my blog by examining some of the lessons I have learned.

Lesson 1: You need a Coach!
This I can't stress enough! The agile approach depends so much on having a coach. Just like a viking war ship needs a guy beating a drum, an agile team needs a coach. The coach keeps the team on track. He Reinforces and evangelizes the agile (XP) practices. The coach is shield from scope creep (YAGNI). The coach is often the tie breaker. The coach even takes care of some of the more mundane aspects such as paperwork.

Lesson 2: Practice the Practices
After a year of forcing the issue on Test Driven Development (TDD), it hit us like a brick this really works! For us TDD became our design tool. User Stories and CRCs are what we work from. I'll cover the other practices later.

Lesson 3: Automate the crap out of everything!
We can't imagine not having an automated build and deployment system! In a later post I hope to discuss exactly how our build works but suffices to say it's cool. We can build a very complex web application, test it, update the database and deploy it to several load-balanced servers in about 11 minutes. Our CVS repository is tagged with every build and an archived package is kept. To accomplish this here are some of the tools we use: Luntbuild, NAnt, CVSNT, Psexec.

Lesson 4: Value the Values
Communication, Simplicity, Feedback, Courage. That's the subset we use. Above all we value the people over the process! With every new hire I take about 30 minutes to really explore what each value means. In later posts I intend to explore them here as well.

Lesson 5: Refactor
Get rid of redundancy! I really recommend Martin Fowler's book on refactoring! I also recommend Kerievsky's book "Refactoring to Patterns". The principle behind refactoring is to focus on getting something running THEN make it simpler cleaner, more readable. The point is that you must make time for the refactoring and it must happen often. We refactor our work everyday. Write something in the morning...refactor it in the afternoon.

Lesson 6: Keep User Stories simple
A user story should be something a User would write. Sometimes we HAVE to write the stories...it's a fact we haven't been able to get away from. IF an engineer HAS to write a story (man this really shouldn't happen) show it to someone who is not an engineer for a sanity check. Your user stories should not include words like web-service or phrases like "a class that can..." unless class is a classroom. Every story should have a user that wrote it. Every story should have a Acceptance Test as well.

Lesson 7: Use metaphors for your systems
This is an area we have been really bad...as the coach I blame myself. Your new software project shouldn't force users (and the team) to learn/invent a whole new set of jargon. The greatest (and most prolific) impediment I have seen to communication is jargon ambiguity. The project should have a common vocabulary that's easy to pick up. TRY VERY HARD TO MODEL IT AFTER SOMETHING COMMON IN THE REAL WORLD! You will save yourself a lot of confusion and lost hours.

Lesson 8: Don't over-engineer
Allow your project to evolve. Don't waste time on elaborate plans that will change anyway. Evolutionary design is the name of the game on agile teams. Have quick planning meetings every few weeks. Put the user stories away somewhere and work on/think about only one at a time. If you follow the practices (like Once and Only Once) you generally won't paint yourself into a corner and the time saved not future-proofing will make up for the few corners (very few) you find yourself in.

Lesson 9: Don't under-engineer
Yikes, am I bi-polar? Nope, just as you shouldn't over-engineer you should beware of under-engineering. Take time to spike solutions. Write tests up front and discuss all the stories with the team. Know responsibilities each story carries and how you will take care of them.

Lesson 10: Fail FAST
Fail Fast means write a failing test as soon as possible. Don't worry about the naming convention (that can be fixed when you refactor). Don't worry about inheritance or any other over-engineering concern. Just write a failing test as soon as possible. Self organizing teams are easily disorganized...fail fast will help keep a good pace. The team wants to be productive but sometimes the path is hidden. Chopping around in the bushes with a few failing tests will often reveal the hidden path.

Lesson 11: Embrace and Understand Pair Programming
On teams people will naturally pair. Let them find their own pairs. Don't continue to pair when it's being non-productive and always have a pair working on a single story. More than anything pairing (and a little coaching) refinforces courage.

Lesson 12: Be Courageous
Courage is very important. You can't be afraid to write tests first, to refactor or to ignore YAGNI. You can't be afraid to let your pair partner know when they are doing something the wrong way.

Lesson 13 (Added 01/29/08): Learn how to Write User Stories
I Actually discuss this here: Writing User Stories

I'm sure we've learned other lessons. On our team we make it rule that any story we can't fully cost is put on the back burner. It takes courage to tell a customer that you can't include a feature because the team can't agree on a solution. I say though, that in a world where so many software projects fail or go way over budget, it is better to know your risks earlier than later.