Volume 2 • Number 1
This article discusses practical experiences gained after three years of implementing a proven software measurement process that emphasizes the use of quantitative data to manage software projects. The authors have helped commercial businesses and government agencies implement the process known as Practical Software Measurement: A Foundation for Objective Project Management (PSM). A brief description of the process, project measurement roadblocks to avoid, and advice for institutionalizing project measurement are discussed.
Key words: decision making, information needs, process improvement, project analysis, software measurement
byBeth Layman, TeraQuest Metrics, Inc., and Sharon Rohde, Lockheed Martin Mission Systems
A software project managers worst nightmare is having his or her project canceled. Unfortunately, studies have shown that as many as one in 10 software projects are canceled, often due to excessive cost or schedule overruns, unmanageable scope creep, or unmet technical objectives. Many software organizations measure schedule delays in years, not months or weeks. The industry is aware of the many contributors to this software project crisis: unrealistic estimates, poor planning, poor risk management, lack of information to support decision making, and so on. What can the industry do about it (Layman and Rhode 1998)?
There is a growing awareness in the software industry that measurement plays an important role in solving these problems. Measurement, when integrated into the overall project management process, provides the information necessary to identify and manage the issues inherent in software projects. Measurement at the project level can be used to objectively validate estimates and plans, track progress, and even anticipate potential problems such as schedule slippage and cost overruns. The goal of project-level measurement is to provide project managers with sufficient insight into the project to support decision making and positively influence project outcomes.
A few years ago, the U.S. Department of Defense, as a major acquirer of software, identified this software project crisis as a major problem and recognized that better use of quantitative techniques were needed on its programs. It initiated a program called Practical Software Measurement. Practical Software Measurement: A Foundation for Objective Project Management (PSM) (McGarry et al. 1998) is one of the programs primary products. It is a guidebook that presents a systematic measurement approach and explains techniques for using measurement to make project decisions in time to affect the outcome of a software project. PSM is unique in that it was developed by a working group of measurement experts from both government and industry, and has received endorsements throughout the international software measurement community.
One of the authors of this article helped write the guidebook and both authors are qualified PSM trainers. Both authors have been working to transition the PSM guidance to software-intensive commercial information technology, government, and aerospace software projects. This article provides an overview of the PSM guidance and then describes the lessons the authors have learned through their experiences implementing PSM during the last three years. These lessons should be useful to anyone implementing PSM or any other project measurement approach.PSM OVERVIEW
PSM is a guidebook designed to help software project managers: 1) identify issues and objectives that are important to their projects success; 2) implement a measurement program focused on those issues; and 3) gain objective insight into those issues throughout the projects life cycle. PSM represents a practical, easy-to-use set of best practices for software measurement. Because PSM presents a flexible measurement process vs. a fixed set of software measures, it can be applied to virtually any software project. Information on how to obtain a complete copy of the PSM guidance is provided at the end of this article.
PSM characterizes the key elements or principles of a successful measurement program, then describes a comprehensive measurement process based on those principles. The process consists of three major activities, as shown in Figure 1. The first activity describes how to tailor the measurement program to address project-specific issues, risks, and objectives. The second activity describes a systematic process for applying, or using, measurement to gain insight into the projects issues and to aid in decision making. The third activity, implementing measurement, explains how to put measurement into practice within an organization. In addition to the process guidance, PSM includes detailed selection and specification guidelines for proven software measures, sample indicators, measurement case studies from real-life software projects, and guidance for putting measurement on contract.DEVELOPING A MEASUREMENT PLAN
During the tailoring phase, measurement requirements for the project are identified. PSMs issue-driven approach stipulates that the projects unique issues and objectives drive the identification of measurement requirements. This is because the purpose of measurement is, first and foremost, to help the project achieve its objectives, identify and track risks, satisfy constraints, and recognize problems early. PSM defines the following common types of project issues:
PSM emphasizes identifying project issues at the start of a project and then using the measurement process to provide insight into those issues. While some issues are common to most or all projects, each project typically has some unique issues. Examples of project-specific issues might be lack of available object-oriented expertise/resources or concerns about the implementation of a particular software package. Figure 2 shows an example of how project issues can be mapped to PSM common issues in order to further use the tailoring guidance to help identify useful measures and apply them.
Also, issue priority usually varies from project to project. Moreover, it is important to note that many of these issues are interrelated. For example, while an incremental development approach may help uncover or clarify requirements, it may also lead to more schedule slippage. Lack of available object-oriented expertise may not only contribute to additional costs and schedule delays but may also jeopardize software quality. The relationships between issues must be considered when prioritizing.
Once project-specific issues are identified and prioritized, measures can be selected that will provide insight into those issues. A measure is a quantification of an attribute of a software process or product. A variety of measures may be needed to provide insight into a single issue. Measurement selection will be driven by a number of factors including:
PSM provides a list of approximately 40 candidate measures. The measures identified in PSM are measures that have been used successfully by members of PSMs technical working group, which is composed of measurement practitioners from more than 40 different software producing organizations. While this list of measures is by no means complete, it represents a starting place for identifying and specifying measurement requirements. For each measure identified, helpful information is included in the guidance such as data items and useful attributes to collect; recommended unit of measure, collection level, and reporting level; applicability to various domains; sample indicators; and so on.
The measurement plan documents the measurement requirements for the project, starting with the projects issues and ending with a complete specification of each measure selected. It does not have to be a lengthy document, but should document the following:
Once a project gets under way, measurement data are regularly collected according to the measurement plan. Measurement tools, databases, and spreadsheets are often used to collect, store, and process raw measurement data. Once collected, the raw data are turned into information. As data are aggregated, compared, and analyzed within the context of recent project events, information emerges that can be used to help manage the project. PSM advocates using a flexible and dynamic analysis process that promotes the use of measurement as primarily an investigative activity. The key building blocks of this activity are indicators. An indicator is a measure or combination of measures that provides insight into a project issue. Indicators often compare actual project performance data to a plan or baseline and are often portrayed graphically.
One of the unique aspects of the PSM guidance is the detail provided regarding the measurement analysis process. PSM 3.1 describes three types of analysis that are performed on data:
PSM also describes a four-step process that can be applied whenever data are analyzed: 1) identify the problem; 2) assess problem impact; 3) project possible outcomes; and 4) evaluate alternatives.
Finally, the importance of understanding the relationships between project issues and the data that represent them is stressed. PSM prescribes using an analysis model (see Figure 3), which shows how some indicators can serve as leading indicators for a particular issue, because they provide insight into something that contributes to the emergence of the issue. The plusses and minuses show whether an increase in the contributor results in an increase (+) or decrease (-) in the resulting issue. For example, an increase in functional size (due to requirements growth) can result in an increase in product size, and an increase in product size can result in an increase in the effort required to complete the project. Therefore, requirements growth could be viewed as a leading indicator of effort overruns; this means that this relationship and the possible resulting outcomes should be considered during the analysis process.
The last step in applying measurement on a project is to actually use the insight gained from measurement analysis to make decisions. This involves communicating the results of the analysis (current problems, impact, outcomes, and alternatives) to the decision makers and taking action. PSM provides guidance on how to clearly communicate results and how to track the results of actions taken.
Because new issues and problems can emerge at any time throughout a software project, PSM advocates that the measurement process implemented be flexible and responsive to change as the project evolves. This means revisiting the tailoring activities and modifying measurement plans as needed throughout the project life cycle. The issues, measures, and analysis techniques must be changed over time to best meet the projects information needs.LESSON LEARNED IMPLEMENTING PSM
The authors have provided training, conducted measurement planning workshops, and assisted with full-scale implementation of PSM on a number of projectslarge and smallin both development and maintenance groups. The PSM process has received a very favorable response from the project teams they have worked with because the focus is on meeting their projects information needs vs. meeting some requirement to provide outsiders with project data.
The authors have also encountered a number of difficulties. These potential problems must be understood and resolved before the process can be successfully deployed. The authors experiences described in this section represent common or recurring implementation issues in the organizations they have consulted with.Lesson 1: There is often a disconnect between the measures currently collected and the issues real projects face.
One way the authors help organizations implement PSM is through one- to two-day facilitated workshops. Using PSMs issue-driven approach, they typically lead project teams through an issue identification process early in the workshop. Here, they use brainstorming and project team synergy to identify existing project issues and constraints, and potential issues/risks that may affect future phases. Next, the authors identify the type of information that would provide insight into the highest priority issues. Only then do they look at data currently collected and measures/metrics requirements, if any. A disconnect between what is currently being measured and what information is needed to help the project address their issues often becomes apparent at this stage.
Many of the organizations the authors consult with already have measurement initiatives under way. Typically, a measurement group has been established, and that group has developed a required list of metrics that all projects are required to produce on a regularly scheduled basis. Project staff members often say that the required metrics are of no value to them. Also, because their projects process does not support, as a natural by-product, the collection of the data or the analysis (which should be an integral part of the project management and decision-making process), they often just meet the requirements without concern for data validity or accuracy (that is, fudged numbers).
Often, PSMs flexible-at-the-project-level, issue-driven approach is counter to existing measurement practices. This usually becomes apparent during the workshopjust as projects are starting to see that measurement might be of value in managing their projects and making real-time decisions, the measurement group begins raising objections about losing control of the measurement process. Another concern is losing the ability to capture common measures and compare status and performance across projects. While this seems like a big stumbling block, it usually is not. The concerns are usually overcome with one or more of the following:
The disconnect between organizational and project needs can be seen in the following example of the often-required schedule data. While monthly Gantt or schedule variance charts are often required for each project, they provide little insight into the cause of schedule slips and often do not indicate a problem until it is of major proportion. Once projects identify the nature of the schedule issue, simple work unit progress charts, like the one shown in Figure 4, can be used to augment or even replace the Gantt chart. Project leads can use these progress charts to pinpoint schedule problems long before they appear on a Gantt chart.Lesson 2: Making people need measurement is the best first step toward institutionalizing it.
The authors have learned that the best philosophy to adopt is this: Rather than fighting with logic the many and varied objections project teams may raise against project measurement, get them hooked on measurement instead. Measurement planning workshops are a good way to do this. The authors isolate the project team for one to two days, immerse them in PSM, change their perception of measurement, help them realize their need for information, and make them feel their data deprivation. The authors show them how the information they need could be derived from their existing process, and show them how the information could be used during a project. This often transforms resistive types into chomping-at-the-bit advocates. A key to this transformation process is getting project teams to take ownership of their issues and recognize their responsibility for making visible the things that are happening on the projectthings both within and outside their control.
Once people feel they need specific data or indicators, they find ways to use them. After they begin using them, they often find ways to build on what they have in order to meet other needs and build more measurement into their process. To get this cycle going and achieve real institutionalization (for example, where measurement is a natural by-product of the process and represents the way we do things here), startup assistance and ongoing consulting services are usually needed. This is where the measurement group can be of service. Many measurement groups the authors have encountered have traditionally owned the analysis of measurement data. They collect the data from the project and produce various charts and graphs. They interpret the data and often report results within the organization. With PSM, measurements are analyzed and used by the project team. The authors work with these measurement groups to help them transition from measurement doers into measurement consultants. They offer their expertise to projects and help them build simple spreadsheets and collection systems to collect the data they need. They help project members learn how to generate graphs from spreadsheets. They share these simple tools across projects. And finally, they advise projects in the proper use of measurement and in this way, help establish true quantitative software project management within their organizations.
Many of the authors clients with maintenance or sustaining projects have the recurring information need to make visible the impact of a trickling resource drain. Typically the project staff is constantly being tagged for nonproject work, yet this drain is never assumed in project estimates nor is the impact of the drain quantified or even given visibility at the program management/customer level. A fixed staffing level is usually assumed. Project staff members identify the resource drain as a major issue, construct simple charts like the one in Figure 5 showing a gap between planned and actual staffing, and present it along with schedule data. Management and customers often say, WowI had no idea! and are willing to take corrective action to curb nonproject activities. A very simple measurement often helps solve a very big problem and gets project teams hooked on metrics.Lesson 3: The project culture will impact the implementation of measurement.
One of the biggest roadblocks to implementing effective project measurement is the prevailing project culture, which is difficult and slow to change. Unfortunately, some organizations still suffer from one or both of the following:
The bottom line is that both management and customers must be willing to listen and respond constructively to bad news. If this type of maturity is not present in an organizations current project culture and no attempt is being made to change it, measurement will only be useful within the projects limited scope of control.OTHER SUGGESTIONS FOR GETTING STARTED
Based on their experiences with implementing PSM to date, the authors put forth the following additional suggestions for getting started with project measurement:
The PSM guidance has served as a useful framework for introducing software organizations to the concept of an adaptable measurement approach that can be tailored to fit unique project needs. It has been used to establish measurement approaches on projects with little or no measurement, and has been used to hone comprehensive, formalized measurement programs. The authors hope that by introducing the PSM approach and that by sharing some of the implementation challenges they have encountered, readers will consider this approach to software project measurement and will be ready to deal with these common implementation difficulties.Obtaining a Copy of the PSM Guidebook
At the time of this writing, version 3.1 of the PSM Guidebook (April 13, 1998) was available on the PSM Web site at www.psmsc.com and version 4 was under development.
The authors of this article would like to acknowledge the other authors of PSM: John McGarry, Cheryl Jones, David Card, Betsy Bailey, Joseph Dean, Fred Hall, and George Stark; the PSM Support Center; and the many interesting (but unnamed!) projects and project team members whose experiences have served as input for the writing of this article.
Layman, Beth, and Sharon Rohde. 1998. Experiences implementing software measurement. Presented at Pacific Northwest Software Quality Conference and the 8th International Conference on Software Quality, Portland, Ore.
Layman, Beth. 1998. Lesson learned implementing a best practice: Practical software measurement. Presented at the Lockheed Martin Systems Engineering and Software Symposium, New Orleans, La.
McGarry, Jack et al. 1998. Practical software measurement: A foundation for objective project management, version 3.1. Available at www.psmsc.com.
Beth Layman is a senior associate at TeraQuest Metrics, Inc. She has more than 20 years of software industry experience with a system development background and specialization in quality and process management. She provides consulting support to TeraQuest clients in the areas of software measurement, process improvement, and quality management. Prior to joining TeraQuest, Layman worked for Lockheed Martin and served as research director at the Quality Assurance Institute.
Layman is a CMM-based lead assessor, one of the principal authors of Practical Software Measurement: A Foundation for Objective Project Management (PSM), and is an associate editor for Software Quality Professional. Layman can be contacted at TeraQuest Metrics, 5523 Cord Grass Ln., Melbourne Beach, FL 32951 or by e-mail at firstname.lastname@example.org.
Sharon Rohde is a senior consultant at Lockheed Martin Mission Systems. She has more than eight years of experience in software engineering, program management, and applied engineering research and development, with three years specifically in software and systems engineering processes and measurement. Other areas of expertise include software methodologies, tools, testing, reuse, quality, and risk management. Prior to joining Lockheed Martin, Rohde developed and implemented a measurement plan for a major government agency and contributed to the Practical Software Measurements Product Engineering Working Group. Rohde has authored journal articles dealing with systems engineering automation, risk management, and software reuse. She can be reached by e-mail at email@example.com.
(0) Member Reviews