Dev Teams Hobbled by Poor Metrics, Study Suggests

Borland commissioned a study on the effectiveness of performance measurements in application development projects. The study surveyed 20 application development and project management organizations with revenues ranging from $1 billion to $5 billion.

The aim of the study was to investigate the question of "metrics" in software development. The study found big problems with how well the software development process is tracked and measured.

For instance, most application development metrics were collected manually by project managers or lead developers, according to the survey. In some cases, respondents indicated that manual collection of this information took up as much as a third of the manager's time.

The study, "Changing the Cost/Benefit Equation for Application Development Metrics," stated that collecting metrics is "expensive," regardless of whether the data are collected manually or automatically.

"Superficial metrics" often lead organizations astray, according to the report. An example of such poor measures is the use of "on-time, under-budget, and on-scope" metrics, which are typically collected at the end stage of a project. The report described these metrics as "unsuitable for application development," even though they are commonly used by application development professionals.

The ability to actually use the collected data was also considered to be problematic in the survey. Eight of 20 respondents were "unable to trend or aggregate the metrics," according to the study. Forrester Consulting cited inconsistent collection methods and the use of multiple tools and repositories as potential stumbling blocks in this area.

The report suggested two approaches as a way out of the metrics mess.

First, application development teams should use "iterative, incremental development processes," in which metrics are collected in intervals throughout the project, rather than at the end. An example given in the report is the use of six iterations in a project with 100 requirements. For instance, if just 20 requirements are completed by the third iteration, the project might be in trouble.

Second, application development teams need to practice a "disciplined estimation of business value" in their project estimates, the report recommended.

Metrics are typically used at three levels in projects, according to the study. Those levels include "portfolio metrics," which provide executives with an overall project view. There's also "in-flight metrics," which describe measurements taken during the project. Finally, there's "post-mortem project metrics," or information collected at the end of the project.

Organizations should maintain a "comprehensive metrics program" and include all three levels, according to the report. Still, the most useful metric is also the one most neglected by application developers -- namely, in-flight metrics.

"The lack of in-flight project metrics that really describe the work being performed on a project is a major fault of most application development metrics programs, and it's one that most shops aren't even aware of," the report stated.

The Forrester Consulting report on metrics is expected to available today for free to registered users at Borland's Web site.

About the Author

Kurt Mackie is senior news producer for 1105 Media's Converge360 group.

comments powered by Disqus


  • VS Code Now Has Apple Silicon Builds for Native Mac Development

    Goodbye Rosetta, hello M1. Visual Studio Code has been updated with new builds that let it run natively on machines with Apple Silicon (M1), the company's own ARM64 chips.

  • Visual Studio 2019 for Mac v8.9 Ships with .NET 6 Preview 1 Support

    During its Ignite 2021 online event for IT pros and developers this week, Microsoft shipped Visual Studio 2019 for Mac v8.9, arriving with out-of-the-box support for .NET 6 Preview 1, which the company also released recently.

  • Analyst: TypeScript Now Firmly in Top 10 Echelon (Ruby, Not So Much)

    RedMonk analyst Stephen O'Grady believes TypeScript has achieved the rare feat of firmly ensconcing itself into the top 10 echelon of his ranking, now questioning how high it might go.

  • Black White Wave IMage

    Neural Regression Using PyTorch: Training

    The goal of a regression problem is to predict a single numeric value, for example, predicting the annual revenue of a new restaurant based on variables such as menu prices, number of tables, location and so on.

Upcoming Events