Numerous studies attempt to quantify the losses caused by software defects. Take Toyota for instance: It just announced a recall for more than a half million 2010 hybrid vehicles. The reason? A change in "brake feeling" caused by faulty antilock braking software. And to date, there is still no fix for cars on the road yet. (That's in addition to its recalls for steering and accelerator problems.)
Bill Hoffman is vice president and CTO of Kitware.
One study estimated losses to the U.S. economy alone due to software defects at $60 billion annually. There are many instances of monumental software failures that have staggering losses up to and including the loss of human life. In 2010, a software problem caused bank cards to fail across Germany. It is also likely that Air France flight 447 was brought down by software that was not able to handle the extreme weather conditions. In 1999, the Mars Climate Orbiter was lost due to software confusing pounds with kilograms. One of the most notable instances of software doing bodily harm was the Therac-25 radiation treatment system. In this case, faulty software caused patients to be given massive overdoses of radiation, killing several of them.
Most developers are well aware of the problems caused by their profession, so why do they continue to write faulty software? Many software tools have been created to detect and combat software defects. However, in my experience these tools are only used by developers when they think there is a problem and not on a regular basis. I would say that the key to increasing software quality is to penetrate the work flow of the developer.
The Developer Mindset
Developers are often under great pressure to produce new features very rapidly, and I would argue that most of them actually like it that way. To a real geek, it can be exciting to create and deliver new features. Creating a new test or increasing code coverage is not as "sexy" as creating a new 3-D visualization. I would argue that the best way to improve software quality is to make the developers want to test, and to perform the tests automatically as part of the development process.
Going back to the Therac-25 example, the commission that studied that disaster found that " the primary reason should be attributed to the bad software design and development practices, and not explicitly to several coding errors that were found. In particular, the software was designed so that it was relatively impossible to test it in a clean automatic way." This drives home the point that for software testing to be effective, it must be automatic and part of the process. Since that accident, the FDA has put stricter approval processes in place for medical devices. But that hasn't stopped the problem. I would also argue that even strict "paperwork"-based processes will be worked around, dreaded and avoided by developers. To really make a difference, you have to get the geeks interested and put systems in place that are fun and easy to use, and that do not slow down development, but rather increase development speed.
In the late '90s while I was working at GE Research and Development, GE had a corporate-wide quality push based on the Six Sigma engineering process. The process is based on measurement of defects and is generally applicable in large-scale mass production. The idea is to only have defects in the sixth sigma of a bell curve or defects less than 3.4 per million opportunities for defect. The key to the process is coming up with things to measure.
With software this is a tricky problem. You can't really measure bugs, because although you suspect they exist, until you find them, you cannot count them. Once you find them, you can fix them, and then what is the point of counting them? So, several of the software developers at GE came up with software things that can be measured. For example, you can measure warnings, compile errors, test failures, timing changes, code coverage, memory defects found by dynamic analysis tools. The next thing to do was to create tools that would do the measurement automatically. The idea produced the concept of a software dashboard.
Not to tout ourselves too much, but since early 2000, my company has been creating a set of open source tools and refined a quality software development process. We have developed a very popular build tool called CMake that allows for cross-platform building of compiled software. CMake includes a tool called CTest that can drive automated builds, regression tests and diagnostic tools such as memory checkers and code coverage analyzers. Finally, a Web 2.0 application called CDash is used to communicate the testing results to the developers.
So help is out there; developers just have to seek it. Lightweight sets of tools and processes like ours can be quickly set up for a new project, allowing tests and diagnostic tools to run automatically. Having an automated process is key as most developers may not have time to run diagnostic tools, but they are unlikely to ignore reports sent to them automatically. With this approach, a single developer can set up the tools to run, with results sent to the entire team each time the code is changed. This gives developers immediate feedback on the work they perform. Once in place, this process engages developers and allows them to see problems as they are introduced into the code, making it much easier to fix them.