Battling Software Defects One Developer at a Time

Bill Hoffman

Numerous studies attempt to quantify the losses caused by software defects. Take Toyota for instance: It just announced a recall for more than a half million 2010 hybrid vehicles. The reason? A change in "brake feeling" caused by faulty antilock braking software. And to date, there is still no fix for cars on the road yet. (That's in addition to its recalls for steering and accelerator problems.)


Bill Hoffman is vice president and CTO of Kitware.

One study estimated losses to the U.S. economy alone due to software defects at $60 billion annually. There are many instances of monumental software failures that have staggering losses up to and including the loss of human life. In 2010, a software problem caused bank cards to fail across Germany. It is also likely that Air France flight 447 was brought down by software that was not able to handle the extreme weather conditions. In 1999, the Mars Climate Orbiter was lost due to software confusing pounds with kilograms. One of the most notable instances of software doing bodily harm was the Therac-25 radiation treatment system. In this case, faulty software caused patients to be given massive overdoses of radiation, killing several of them.

Most developers are well aware of the problems caused by their profession, so why do they continue to write faulty software? Many software tools have been created to detect and combat software defects. However, in my experience these tools are only used by developers when they think there is a problem and not on a regular basis. I would say that the key to increasing software quality is to penetrate the work flow of the developer.

The Developer Mindset

Developers are often under great pressure to produce new features very rapidly, and I would argue that most of them actually like it that way. To a real geek, it can be exciting to create and deliver new features. Creating a new test or increasing code coverage is not as "sexy" as creating a new 3-D visualization. I would argue that the best way to improve software quality is to make the developers want to test, and to perform the tests automatically as part of the development process.

Going back to the Therac-25 example, the commission that studied that disaster found that " the primary reason should be attributed to the bad software design and development practices, and not explicitly to several coding errors that were found. In particular, the software was designed so that it was relatively impossible to test it in a clean automatic way." This drives home the point that for software testing to be effective, it must be automatic and part of the process. Since that accident, the FDA has put stricter approval processes in place for medical devices. But that hasn't stopped the problem. I would also argue that even strict "paperwork"-based processes will be worked around, dreaded and avoided by developers. To really make a difference, you have to get the geeks interested and put systems in place that are fun and easy to use, and that do not slow down development, but rather increase development speed.

In the late '90s while I was working at GE Research and Development, GE had a corporate-wide quality push based on the Six Sigma engineering process. The process is based on measurement of defects and is generally applicable in large-scale mass production. The idea is to only have defects in the sixth sigma of a bell curve or defects less than 3.4 per million opportunities for defect. The key to the process is coming up with things to measure.


With software this is a tricky problem. You can't really measure bugs, because although you suspect they exist, until you find them, you cannot count them. Once you find them, you can fix them, and then what is the point of counting them? So, several of the software developers at GE came up with software things that can be measured. For example, you can measure warnings, compile errors, test failures, timing changes, code coverage, memory defects found by dynamic analysis tools. The next thing to do was to create tools that would do the measurement automatically. The idea produced the concept of a software dashboard.

Not to tout ourselves too much, but since early 2000, my company has been creating a set of open source tools and refined a quality software development process. We have developed a very popular build tool called CMake that allows for cross-platform building of compiled software. CMake includes a tool called CTest that can drive automated builds, regression tests and diagnostic tools such as memory checkers and code coverage analyzers. Finally, a Web 2.0 application called CDash is used to communicate the testing results to the developers.

So help is out there; developers just have to seek it. Lightweight sets of tools and processes like ours can be quickly set up for a new project, allowing tests and diagnostic tools to run automatically. Having an automated process is key as most developers may not have time to run diagnostic tools, but they are unlikely to ignore reports sent to them automatically. With this approach, a single developer can set up the tools to run, with results sent to the entire team each time the code is changed. This gives developers immediate feedback on the work they perform. Once in place, this process engages developers and allows them to see problems as they are introduced into the code, making it much easier to fix them.

Add Comment      Leave a comment on this blog post

Aug 1, 2010 7:42 AM jitkasem jitkasem  says:

I am software engineer and I have to write the source code for an embedded product.In my work I have to do a lot of documents due to the manager want to make sure that the written source code have no bug.

As I am programmer I have to do the detail design document and also test design document too. The test design document is used for describe how to test the features of the written code (or function). My manager told me to do the unit-test document to test the piece of source code inside the fucntion for ensure that it is correct.They want the final source code to pass the six sigma process. When the written software is passed the unit-test by programmer , the software QA will test the overall system again for ensure that the new written software don't have the side effect with other modules, they call this process "System test" and it is the last process for testing our product. If the product pass the "system test" then they will release the product to the market...

This is the overall of our work...

Aug 5, 2011 3:19 AM Paris Paris  says:

Thanks for writing this post. Software defects are headache for engineers and product managers everywhere. It has become an expectation that many defects will exist for a product. Your company has been doing the right thing by developing and refining the quality software process. Continuous improvement and innovation is CMMi Level 5 maturity

Good code quality and good testing are key to releasing a zero defect project.

My company continuously seeks to improve its software development practices. Quality is the number one thing you hear about around the company. One of our teams had zero defects for more than 70 percent of it's releases.


Post a comment





(Maximum characters: 1200). You have 1200 characters left.



Subscribe to our Newsletters

Sign up now and get the best business technology insights direct to your inbox.


Resource centers

Business Intelligence

Business performance information for strategic and operational decision-making


SOA uses interoperable services grouped around business processes to ease data integration

Data Warehousing

Data warehousing helps companies make sense of their operational data