Executive Summary: Independent Study of Software Reuse Using Frame Technology

Beginning in July 1994, and again in 1998, QSM Associates of Pittsfield, Massachussetts undertook an independent assessment of the development cost, schedule, and quality benefits achieved by implementing systematic software reuse for application development.

Fifteen projects collected independently from 9 different organizations were analyzed. All the organizations accomplished reuse through the application of frame technology as implemented in NETRON/CAP products (the predecessor of Netron Fusion™) from Netron Inc. The specific objective was to benchmark the bottom line productivity effects across the 9 organizations and compare them against industry, identifying what patterns were present, both individually by company and for the group as a whole.

In comparing data collected from these projects with that from QSM's industry-wide database of IS productivity statistics, conclusive evidence was found that software reuse as implemented in Netron's products and practices is a superior way to engineer business applications. Not only is productivity significantly impacted, but other measures of sound software engineering such as quality, reliability, cost, and effort showed large improvements.

Productivity Gains

Netron projects demonstrated over 9 times higher productivity versus the industry average in delivering systems that ranged from relatively small in scale to almost ten million lines of code. Even on initial projects, users averaged eight times higher productivity, including time spent absorbing the learning curve. Mature Netron users, possessing larger inventories of reusable code and greater experience, showed more than fifteen times productivity.

Quality Improvements

On average, it would take 12.7 months longer to reach the same level of quality realized in 5 months on Netron projects.

Reliability Increases

The average Mean Time to Failure (the amount of time a system runs before a defect is observed) on projects where this statistic was tracked was higher for Netron than for non-Netron projects. These above-average reliability ratings were accomplished within project schedules approximately one quarter of the industry average schedule.

Supporting Data

Supporting data for these conclusions based on an analysis of the 15 projects in this data sample follows:

- An average size project of 133,000 lines of code in this sample took approximately 5.3 months to build, versus an industry norm of 18 months. This represents a 70% cycle time reduction.

- The same average size project typically expended 27.9 person-months of effort, versus an industry norm of 179.5. This represents an 84% reduction in cost, amounting to a savings of approximately $1.38 million US.

- All 15 projects positioned as consistent high performers for schedule (on average, in the top 84th percentile) compared to QSM Industry Trendlines.

- All but 2 of the larger projects positioned in the top 84th percentile for effort (i.e., low cost performers) against the QSM Industry Trendlines. The remaining 2 applied large teams in an attempt to further compress schedule, and expended more effort.

- Nearly one-third (27%) of the projects were under budget and ahead of schedule. This contrasts with only 2% of industry exhibiting project underruns. Project overruns for this sample were nominal.

- Six of 7 projects that provided defect statistics nominally experienced average to lower than average defects found and fixed during testing.

- All but 2 projects utilized far fewer staff than industry average, applying 10 people or less per project.

- The average QSM Productivity Index (PI) for the sample, a metric of project efficiency and complexity, was 26.2, nine times higher than the industry average productivity index of 16.9 from QSM's database statistics. This high productivity is directly correlated to the high volume of functionality generated through reuse via Netron's frame technology.

- All but 2 projects produced high degrees of functionality per person-month, and all produced high degrees of functionality (and code) per month. Two projects expended higher effort due to schedule pressure (large teams), resulting in lower functionality per person-month.

Overview of Approach

To meet these objectives, the 9 organizations that elected to participate in this research study each contributed project data, using an identical set of guidelines that were developed and refined by QSM research over the last 15 years. These guidelines are embodied in QSM's industry data collection framework, and comprise the "front end" of the QSM Productivity Analysis Database System (PADS®). When complete, the data was sent directly to QSM Associates, Inc. for validation and analysis. None of the information gathered by QSM was disclosed in its raw form to Netron or any other party outside of QSM. Two of the 9 contributing organizations requested that their identity remain anonymous. The other 7 organizations which participated in and funded this study are:

To evaluate process productivity, the QSM Productivity Analysis Database System (PADS®), was used. PADS is an international database of more than 3,800 completed software projects collected worldwide (6/1/94 Baseline), and contains validated productivity statistics from hundreds of leading software producers throughout the industry from the U.S./North America, Europe, Japan, and Australia. PADS provided benchmark statistics against which the contributing organization's projects were compared.

All of the contributing organizations provided data for projects which were components of their strategic computer applications for business operations. Therefore, the project data was compared against the Business/MIS Systems application segment of the QSM database.

Independent Study Results - An Update

The study was updated in 1998 with 14 more frame-based projects. This was compared with an expanded baseline of over 5000 completed projects, including metrics from projects using Object-Oriented languages. The difference is still striking: OO projects had an average PI of 19.2. That's a pretty good improvement: compare this to the mostly non-OO average PI of 16.9 in 1994. However, the 29 frame-based projects are still holding their own with an average PI of 26.9 (compared to 26.2 in 1994).

As of this writing, it is the first known independent metrics assessment of the cost, schedule, and quality benefits of systematic software reuse for application development.

For a complete copy of the findings or more information about the study, call
Netron Inc., 416-636-8333, or QSM Associates Inc., 413-499-0988.