Why is Software Development So Hard?
The recent fiasco over the launch of the healthcare.gov website (which is part of the US government’s effort to implement the Affordable Care Act) will almost certainly find its place in text books as a glaring example of yet another failure of a multi-million dollar software project to deliver on time and within budget.
As the industry and the associated technologies mature, will software development eventually become less of an uncertain endeavor, or will it continue to be complex forever? In the paragraphs that follow, I'll present some thoughts from different perspectives. You be the judge.
We Still Can't Get it Right
From requirement gathering to implementation and support, there's potential for problems lurking in every step of the way in the software development process. But what about all those tools and methodologies that were supposed to make life easier for us (developers) and for the end users?
From procedural and structured to object oriented, from waterfall to iterative and agile, from Booch and Rumbaugh to UML, and from Domain Driven Design (DDD) to Test-Driven Development (TDD), there are no shortages of tools and methodologies for software development. But unfortunately we still can’t get it right!
Obviously there are no simple answers to the question. While methodologies are good and even necessary, it definitely takes more than methodology to make projects successful. Here I’ll first present a few thoughts on why software development is so hard, and if there's light at the end of the tunnel.
I'm just a messenger. You're the judge.
I’m not referring to Six-Sigma’s Critical To Quality (CTQ). Rather these are the different variables management usually defines for software projects at the inception, where the letters stand for:
- C – Cost (from development perspective this usually equates to the number of developers on the team, or total number of man-hours allocated to the project)
- T – Time (total time allocated for the completion of the project, in weeks, months, years, etc.)
- Q – Quality desired (usually always high)
- S – Scope (list of features desired)
That’s all well and good. The problem occurs when management tries to control all of these variables without getting input or consensus from the development team. In this situation, usually the development team is given a fixed number of development resources (C), and told that the go-live has to be on a certain date (T), and the software has to have a certain set of features (S) and must be of high quality (Q). This is almost always a recipe for failure. Not having any control over the first three variables just described (C, T and S), because they are either too visible or just plain out of reach, what is a development team to do? What do you think will suffer the most? Exactly. It is the only remaining variable Q, or quality.
There is a better chance of avoiding this situation if the development team is made not just a part of the initial planning process, but also allowed to define at least one, if not more, of the variables other than Q. In my experience, in many cases it is usually possible to reduce the scope. There are often features one doesn’t absolutely have to have from the get-go. This allows one to have a phased approach to releasing the product.
In another scenario, management does involve the development team; listens to their assessment, and then begins to twist their arms until they agree to what management wants to hear. This is no better than not involving them at all.
Before I tell you why, let me refer to a quote by Lawrence Summers first. An economist by training, he was the president of Harvard University, and also served in the US government as an adviser on economic policies. He is also known for making some controversial remarks. But he once said "in the history of the world, no one has ever washed a rented car". What he was alluding to is that we only care about the things we own. By forcing the development team to agree to what management wants to hear, management is not only setting them up for failure, but also alienating them by making them feel that they are not part of the team (i.e., they don't have any ownership) and are here to just carry out orders, however unreasonable. What do you think it does to team morale?
Enabler or Differentiator?
When I first started looking for a development job, I spoke to a company that made electrical and hydraulic test equipment for the aerospace industry. During the interview they told me that they had several competitors. That “anyone can make these test equipments these days”. They also told me in no uncertain terms that they saw software as a differentiator, and not just an enabler. Today they are a leader in their industry and also own several subsidiaries.
It makes a big difference on how management sees the role of software for the organization. Are Amazon.com and FedEx retailer and logistics provider respectively, or are they software companies? The line is blurring every day. There are smartphones on the market that are technically superior to iPhone. Why aren’t those as desirable to the masses as the iPhone?
Behemoths have perished overnight for not realizing the importance of having a software strategy
The sooner an organization realizes this and creates a strategy around it and devotes its resources to unleash the potential of software to differentiate itself from its competitors, the better it’ll do in today’s marketplace. Behemoths have perished overnight who have failed to realize this (remember Borders?). (For non-US readers: Borders was one of the two leading book and music retailers in the USA, with about 400 stores and more than 10 thousand employees. It closed its door forever in 2011 after being in business for 40 years, supposedly because it did not have a timely web and eBook strategy).
Unnatural Construct and Implicit Context
To my knowledge there’s no programmer in the world that can write any significant amount of code without making any errors with 100% certainty. If an engineer built bridges the way most of us build software, how long do you think he’d be able to keep his job? Don’t get me wrong, I’m not saying or even implying that we are at fault. But we’ve all seen the statistics on the failure rates of software projects. There must be something fundamentally different about the software development process compared to other creative and intellectual endeavors.
Let’s face it, software is unnatural and the industry is in its infancy (which I’ll get to in the next section). We humans can easily get away with vague and fuzzy communication amongst ourselves. Humans are forgiving. A computer, from that perspective, is merciless. Imagine getting fired from work for not appropriately using commas in your sentences. But that is exactly what the computers do to us (i.e., to our programs) --- you misplace a comma; the computer kicks your program out with a nasty message! Imagine having to call your significant other (or a trusted friend) before saying every sentence at work, to have him/her validate that you’re saying the right thing. But that is what software development process requires us to do for our programs. Ideally, every line of software code should be tested. Even before we write the actual code, we’re being asked to write the code to test it (TDD).
Humans are also very much context aware. We almost always tailor our communications based on context, implicitly. In some cases we say something, but mean something else. But the person on the receiving end also interprets the message in the right context, and we all stay in harmony (most of the time). Our computers are still context insensitive --- they may be location-aware these days, but still context-agnostic. For humans it is hard and unnatural to be consistently that precise in communicating with computers (in designing and coding)!
It is still prohibitively expensive to guarantee near 100% accuracy in a software product
Because the barrier to entry is so low (anyone can learn to write code), we don’t always realize it --- but it really takes a special mindset and special skills, acquired through years of training and experience, to be able to write good software. Companies that view software as just an enabler and a tool for automation (as opposed to a differentiator), and think they can hire programmers off the street to write code to run their business, without much thought and upfront planning, in my opinion, will not go very far in the long run.
Particularly in outsourcing software development projects, the mistake many organizations make is not dedicating knowledge experts to work full time with the development team, for projects of any size. They appear to think we’re spending all this money on these contractors; it should be their responsibility to get it right. It just doesn’t work that way. Good contracting companies make sure they get the needed support from their clients.
We’ve been building houses, roads and bridges for thousands of years. Compared to that, the software industry is probably much less than a hundred years old. Even in my lifetime I’ve seen an explosion in the number of methodologies, development tools and programming languages. Despite that, it is still prohibitively expensive to guarantee near 100% accuracy in a software product, and is not usually done unless it has the potential to put human lives at risk (medical equipment, aerospace, etc.). Compared to many of the skills we’ve mastered, my hunch is that software development will remain on shaky grounds for years to come. It still has a long way to go, and we’re going through the growing pains of the spurt of growth our industry is experiencing.
We’ve all gone (or will go) through our awkward teenage years, but we, or at least most of us, eventually grow out of it. So will software development. In the mean time we just have to keep working at it.
The Future of Software Development:
Another School of Thought
Near the beginning of the last century futurists had predicted that by the turn of the century productivity improvement tools would advance so much that we'd only need to work four hours a day. The first part of the prediction turned out to be true, rather spectacularly, but if anything, we're working longer hours today! The reason, of course, is that our standards of living have also equally gone up over the last hundred years. Likewise, as the theory goes, software development will always remain complex, no matter what. As we'll see below, the argument is quite compelling.
Software development will always remain complex, no matter what
Edsger W. Dijkstra articulated this best in an article (The Humble Programmer) over 40 years ago: "... I would like to insert a warning to those who identify the difficulty of the programming task with the struggle against the inadequacies of our current tools, because they might conclude that, once our tools will be much more adequate, programming will no longer be a problem. Programming will remain very difficult, because once we have freed ourselves from the circumstantial cumbersomeness, we will find ourselves free to tackle the problems that are now well beyond our programming capacity."
I probably should have titled this article 'Why is Software Development So Error-Prone?', because I didn't really mean to imply that comparatively other disciplines are easy. So by 'hard' I really meant 'error-prone'. After all yours truly is just another person with imprecise and fuzzy thoughts!