DevSmart

Defect Prevention

Use Automated Defect Prevention as a software management strategy that increases application quality through process improvement

• By Adam Kolawa
• 12/15/2007

• Relief from having to orchestrate the build each day, then diagnose and resolve any problems that arise in this error-prone process.


• Confidence in knowing that an app is thoroughly tested each day.
  


• Collection of measurement data that facilitates control and improvement of software processes.

Production Line
The first step toward automation is to build a supporting infrastructure. Just like a manufacturing plant, you need an integrated production line with the following components:

• Source control system: A source control system (Microsoft Visual Studio Team System [VSTS] version control, Microsoft Visual SourceSafe) is a prerequisite for nightly builds. All of the files needed for the build should be in the source control system, including build files, scripts, etc.-not just the source that's being built.


• Nightly build system: This system builds an app by automatically executing the required build steps at the scheduled time without any human intervention. This way, any problems in the build are detected the next day. You can use MSBuild, NAnt or other scripts, along with tools like CruiseControl, or even the Scheduled Tasks feature available with Windows.


• Requirements management system (and/or bug tracking system): This is a repository for storing and tracking defects, requirements and feature requests (for example, the Work Item Tracking functionality in VSTS or Bugzilla). The items stored here can be correlated to the related source code and test cases, enabling accurate estimation of their true status.


• Regression testing system: Any tool or combination of tools that can automatically run the core of the existing tests on the entire code base on a regular basis. Its purpose is to identify if code modifications cause unexpected faults when modifying or extending code that previously functioned correctly. It should include any tools that facilitate change detection and thus help you build a robust regression test suite. This can include the many tools provided with VSTS (code analysis, unit testing, load testing) as well as third-party tools that plug into that environment.


• Reporting system: This system should be able to gather data from all these components (source control, nightly build, tests) and present it in a visual way to enable analysis of status and trends. This can be achieved by extending the VSTS reporting engine.

[click image for larger view]

Infrastructure for Automated Defect Prevention

ADP Tools
After you have this infrastructure for automation in place, you can shift your focus to defect prevention. Static analysis is one fully automated-and thus "cost-free"-way to prevent many classes of defects from occurring. Peer code review is another powerful defect prevention tool-though obviously less automated than static analysis.

Yet, the truth is that many defects cannot be prevented in the first place. Nevertheless, it is both possible and critical to identify such defects as they're introduced, then prevent them from reoccurring.

The primary way to accomplish this is to leverage a behavioral regression test suite: a baseline test suite that captures the project code's current functionality. The evolving code base is run against this test suite on a regular basis (preferably nightly), then test failures alert you when code modifications introduce functionality differences, which may or may not lead to defects.

To determine if a given functionality difference indicates a defect, the responsible developer evaluates the related code to determine if the change is actually the new expected behavior (for example, because the related requirement was recently modified). If so, he updates the test suite, and then the team will be alerted to any subsequent changes that might be introducing defects into this new expected functionality. If not, he fixes the code responsible for the defect, and the nightly regression test suite will alert the team if the same defect ever reoccurs.

This might sound trivial, but few teams implement it correctly. It's a shame, because working in this manner leads to amazing increases in team productivity and dramatically improves product quality. When defects are effectively controlled throughout the software dev lifecycle, the team isn't wasting time chasing after bugs. And there's a reduced risk that bugs will end up in the released or deployed software.