Automate, you must!
Recently, at work, we rolled out a new version of one of our major web applications. Until recently rolling out changes was a major undertaking. Over the past few years it's evolved into a much more reliable system that takes about 11 minutes!



Here is a quick chronology of the evolution of our deployment system:

1. Pre-deployment system
   Believe it or not our earliest versions of this product (many years ago in a galaxy far far away...etc) had no deployment system, let alone automation. Unless you considered me or some other guy making a change and Ftping it to the live site for testing, a deployment system. These were carefree times with few users to be annoyed by the occasional bit of downtime. Also, we worked in mostly interpreted languages (ah the good old days!)
2. Let's get a testing server!
   As we grew and more users were around to pay attention to our untested new "features". We decided a test server was in order. The first iterations of the test server were basic configurations of the original application. Nothing was automated but at least things got tested...mostly.
3. Let's automate
   This was a though time. I made a lot of enemies during the year or so it took to automate deployments. No one wanted to spend the time required to automate the system. Even those who saw the value in it were afraid of those who did not. Luckily, my boss, a man with surprising good vision for these things, was a willing advocate for change. At this point I should mention that this was about the time I was placed in charge of several other software teams and I was proposing a sweeping change for them as well. Let's just say it was ugly.
4. Let's REALLY automate
   About the same time we were automating our builds and deployments using NAnt, I was really tied up in the school of thought that was trying to address where databases fit into the Agile methodology. Databases present many problems that aren't handled well by most agile practices. Historically you try to design databases that aren't going to be very volatile. Agile methods suggest that everything should be capable of some volatility! Refactor this, throw this other thing out...etc. So, my problem was this, how do we allow the database to be volatile and still keep data safe? Well, once again I shifted some our resources to non-moneymaking endeavors. We developed an in-house database installer system that automates the deployment or upgrade of our databases. It works quite well is likely one of the greatest timesavers we have!
5. Refactor the Automation
   Over the past few years now we've learned a few things. We've switched a few things around and reversed a few assumptions we've made in the past. Through perseverance we've become so confident in our tools that we have fully embraced automation. We intend to automate everything. We've written a few custom tasks for NAnt and discontinued the use of WIX for many things. Trying to use Windows Installer was painful and it wasn't always clear exactly what it was going to do. We decided it just wasn't worth the effort and stuck to a custom system. Another major shift for us was the realization that our build and deployment scripts should be able to run on any machine, not just the build server.

Along with automating the build and deployment we have automated the Unit Tests and Code Coverage. We are capable of automating the code documentation but frankly it takes way too long and the resulting documentation is somewhat daunting. I'm considering some ways of producing cookbook style documentation where you look up the task you're wanting to perform rather than looking up objects/methods.

We still need to automate our acceptance tests. This is something of a holy grail for me. We've played with FIT.Net but still haven't been able to fully commit to it. Either way we are on the right track and I think we'll only get better!

mmmm...code smell this has

"mmmm...code smell this has". I wish it was original but a co-worker of mine came up with it while we were making up a fictional Agile Jedi (yoda kinda). Some others were, "YAGNI this is" and "Refactor you must".
Having pulled our team, some kicking and screaming, into the Agile world. I have to wonder what has it gotten us? Also, what do we still need? Hopefully, over the next few posts, I can explore this. I know we're not perfect but I believe we're on the right track.


I'd like to start my blog by examining some of the lessons I have learned.

Lesson 1: You need a Coach!
This I can't stress enough! The agile approach depends so much on having a coach. Just like a viking war ship needs a guy beating a drum, an agile team needs a coach. The coach keeps the team on track. He Reinforces and evangelizes the agile (XP) practices. The coach is shield from scope creep (YAGNI). The coach is often the tie breaker. The coach even takes care of some of the more mundane aspects such as paperwork.

Lesson 2: Practice the Practices
After a year of forcing the issue on Test Driven Development (TDD), it hit us like a brick this really works! For us TDD became our design tool. User Stories and CRCs are what we work from. I'll cover the other practices later.

Lesson 3: Automate the crap out of everything!
We can't imagine not having an automated build and deployment system! In a later post I hope to discuss exactly how our build works but suffices to say it's cool. We can build a very complex web application, test it, update the database and deploy it to several load-balanced servers in about 11 minutes. Our CVS repository is tagged with every build and an archived package is kept. To accomplish this here are some of the tools we use: Luntbuild, NAnt, CVSNT, Psexec.

Lesson 4: Value the Values
Communication, Simplicity, Feedback, Courage. That's the subset we use. Above all we value the people over the process! With every new hire I take about 30 minutes to really explore what each value means. In later posts I intend to explore them here as well.

Lesson 5: Refactor
Get rid of redundancy! I really recommend Martin Fowler's book on refactoring! I also recommend Kerievsky's book "Refactoring to Patterns". The principle behind refactoring is to focus on getting something running THEN make it simpler cleaner, more readable. The point is that you must make time for the refactoring and it must happen often. We refactor our work everyday. Write something in the morning...refactor it in the afternoon.

Lesson 6: Keep User Stories simple
A user story should be something a User would write. Sometimes we HAVE to write the stories...it's a fact we haven't been able to get away from. IF an engineer HAS to write a story (man this really shouldn't happen) show it to someone who is not an engineer for a sanity check. Your user stories should not include words like web-service or phrases like "a class that can..." unless class is a classroom. Every story should have a user that wrote it. Every story should have a Acceptance Test as well.

Lesson 7: Use metaphors for your systems
This is an area we have been really bad...as the coach I blame myself. Your new software project shouldn't force users (and the team) to learn/invent a whole new set of jargon. The greatest (and most prolific) impediment I have seen to communication is jargon ambiguity. The project should have a common vocabulary that's easy to pick up. TRY VERY HARD TO MODEL IT AFTER SOMETHING COMMON IN THE REAL WORLD! You will save yourself a lot of confusion and lost hours.

Lesson 8: Don't over-engineer
Allow your project to evolve. Don't waste time on elaborate plans that will change anyway. Evolutionary design is the name of the game on agile teams. Have quick planning meetings every few weeks. Put the user stories away somewhere and work on/think about only one at a time. If you follow the practices (like Once and Only Once) you generally won't paint yourself into a corner and the time saved not future-proofing will make up for the few corners (very few) you find yourself in.

Lesson 9: Don't under-engineer
Yikes, am I bi-polar? Nope, just as you shouldn't over-engineer you should beware of under-engineering. Take time to spike solutions. Write tests up front and discuss all the stories with the team. Know responsibilities each story carries and how you will take care of them.

Lesson 10: Fail FAST
Fail Fast means write a failing test as soon as possible. Don't worry about the naming convention (that can be fixed when you refactor). Don't worry about inheritance or any other over-engineering concern. Just write a failing test as soon as possible. Self organizing teams are easily disorganized...fail fast will help keep a good pace. The team wants to be productive but sometimes the path is hidden. Chopping around in the bushes with a few failing tests will often reveal the hidden path.

Lesson 11: Embrace and Understand Pair Programming
On teams people will naturally pair. Let them find their own pairs. Don't continue to pair when it's being non-productive and always have a pair working on a single story. More than anything pairing (and a little coaching) refinforces courage.

Lesson 12: Be Courageous
Courage is very important. You can't be afraid to write tests first, to refactor or to ignore YAGNI. You can't be afraid to let your pair partner know when they are doing something the wrong way.

Lesson 13 (Added 01/29/08): Learn how to Write User Stories
I Actually discuss this here: Writing User Stories

I'm sure we've learned other lessons. On our team we make it rule that any story we can't fully cost is put on the back burner. It takes courage to tell a customer that you can't include a feature because the team can't agree on a solution. I say though, that in a world where so many software projects fail or go way over budget, it is better to know your risks earlier than later.