Yet another software development disaster is headed for the digital trash heap of failed projects. This time, the casualty is software funded by the U. S. Census Bureau. The Associated Press reports failure to deliver usable software to census enumerators could add as much as $2 billion to the 2010 census. Worse, the AP reports “census officials are considering a return to using paper and pencil to count every man, woman and child in the nation.”
This is a spectacular train wreck that had doom written all over it from Day One. It’s a familiar, predictable pattern constantly repeated since the first clueless manager commanded “just make it user friendly”.
The goal was simple enough: create handheld computers for census field workers to interview citizens and to electronically communicate with a central system. What the Census Bureau got were expensive paperweights unable to transmit the large volumes of necessary data and too complex for census employees to comprehend during tests.
Census Director Steven Murdock blamed the failure on “communication problems” between the Census Bureau and the software contractor. But, the AP learned, “interviews, congressional testimony, and government reports” soundly spell out the failure more precisely: “census officials are being blamed for a poor job spelling out technical requirements.”
Gee, doesn’t that sound familiar? As Yogi Berra so eloquently put it: if you don’t know where you’re going, you might not get there. The Census Bureau failed to spell out what they wanted, so they didn’t get it.
Building software isn’t like shopping for it
How can there be such viral stupidity in corporate leadership that this failure mode burns up capital and burns out developers with such regularity?
Here’s an analogy that works for me. Many years ago, I bought a digital video camera. To do anything meaningful with it, I needed editing software. Several were available, so I browsed the stores and surfed the discussion groups to learn my options. Each came with a unique set of functionality and corresponding price tag.
Many video editing products came with cool features I hadn’t even thought about. Suddenly, visions of green screen magic with pan-and-zoom, slow-motion effects did a fade transition to my Best Director acceptance speech at the Academy Awards! When my decision was made, I plunked down my credit card and walked out of the store with a set of CDs rattling around inside a cardboard box. I couldn’t wait to see what it could do!
We’re all familiar with this process of buying software. Why is it so easy? Because somebody else did the requirements analysis for us! Someone else spelled out the workflow of the user interface. A subject matter expert carefully described what processes needed to be automated – and how. The product was carefully prototyped, evaluated, refactored, and packaged into a polished work of technical elegance. This process took months or years. But, all we have to do as consumers is waltz into Best Buy on our lunch break.
But, what if no video editing software had ever been written before? What if I was the first to identify the market and launch a little startup with the goal of making it big with the first video editing software targeting the consumer market?
Therein lies the greatest challenge in software development: building something that has never been built before, envisioning something never seen, describing an inspiration to someone else with sufficient clarity so they may bring it to reality.
Simply deciding on what features might be useful is hard enough. Creating an interface and an intuitive workflow requires diligence, collaboration, and lots and lots of prototyping.
Where the Census Bureau, and everyone else, gets it wrong
At some point, the Census Bureau approached its contractor and described their needs. I picture the conversation going something like this:
Census Bureau: Make us some software for our handheld computers.
Contractor: Okay. What sort of work do your field workers do? What are their most common tasks? What kind of data will they be collecting?
Census Bureau: I don’t have time to go over that. Just make sure it’s user friendly.
The Census Bureau thought they were strolling through Circuit City looking for “Intuit Census Taker 2010 Deluxe”. They hadn’t stopped to think about what they really needed, they just figured they’d know it when they saw it.
In reality, they weren’t going to find their software on some store shelf. After all, they needed software that had not been written before. They needed careful analysis of their requirements. They needed to identify who would use the software, what their users would be doing, and how they would be doing it. Requirements analysis is hard. It takes time. It is tedious.
This is not work that can be passed on to the developers. Developers are not the subject matter experts – they’re merely specialists in building dazzling creations meeting exacting specifications. Leaving them to guess at requirements guarantees they’ll solve the wrong problem.
How to fix it (and not fix it)
Imagine buying a custom home and telling the builder “build me a home for $250,000…and make it pretty”. Your builder can’t measure “pretty”. How does he know when he’s done? Requirements must be measurable and verifiable.
Your builder needs specific questions answered like “How many bedrooms?”, “One story or two?” Failure to answer fundamental questions ensures you’ll blow 250K on a house you don’t want. Answering these questions is your job – not the builder’s. Software is no different.
Poor requirements analysis is the most common thread in all failed projects. Who is responsible for providing requirements? The primary stakeholders paying for the work. Unfortunately, business process owners driving the software’s need see requirements definition as too burdensome – too much “busy work” – to task their own resources to address. Can’t the developers just write the software? I’ll see if I like when they’re done. That’s a costly cycle to rely on.
When projects fail, developers take the heat. Why? They have no direct reports. Everyone in the food chain has underlings to berate and scapegoat for their own failures.
So, an endless stream of passing fads gets hauled out to “correct” the waywardness of the development staff. These fads are called “process” and go by important and trendy sounding names like RUP, Extreme Programming, and Agile. One process after another is ditched for “better” processes that theoretically correct the cause of the previous digital disaster. When another failed product gets pitched into the bit bucket, the developers are erroneously accused of “not following the process.”
Don’t get me wrong. I don’t mean to disparage the litany of software processes that have come and gone over the years. But, doesn’t anybody wonder why we need so many? Doesn’t anybody question why new processes continually come along claiming to “fix” the shortcomings of all before it? Why is a workable software process so hard to find?
They’re all workable – they simply fail to address the root cause of the problem. The problem is leadership laziness.
When even halfway competent programmers are given a goal and sent on their way, you can expect them to deliver on all expectations. Things like process, skill, and experience can influence the cost and maintainability of the code. But, with clear requirements, you can always expect to get what you want.
So, business leaders, stop looking for the right “process”. Instead, roll up your sleeves and do a little hard work. Think about what you want. Spell it out. Be available for questions and clarifications. Don’t make the developers guess.
It’s time well spent. And, if your time is well spent, then so is your money.