If You’re Not Keeping Score, It’s Just Practice
by Ronald D. Snee
Despite the fundamental importance of measurement and measurement systems, statisticians and quality professionals engaged in process improvement and quality studies frequently find numerous gaps, including:
- No data being collected
- The wrong data being collected.
- Data being collected on process outcomes (Y’s) only and not on process variables (X’s).
- People who don’t know what should be measured.
In the absence of an adequate and appropriate measurement system, it’s virtually impossible to improve the performance of a manufacturing or business process, increase customer satisfaction or ensure the quality of product or service. Failing to measure is like failing to keep score in a football game––you’re not playing, you’re practicing. Translated into business terms: If you’re not measuring, you’re not competing.
Measuring badly can be even worse than not measuring at all because it creates a false sense of security that can lead to disastrous actions. For instance, poorly gauged artillery fire throughout the years has accidentally fallen on friendly troops during wartime. An adventure travel company a few years ago incorrectly measured the length of a bungee cord, which exceeded the altitude of the jump for a potential thrill-seeker.
In each of these situations, those involved would not have taken any action if had they known no measurement had been made. Given a measurement had been made, they assumed the measurement was accurate when it wasn’t and took an action resulting in a catastrophe.
To avert a grave error, an artillery commander who hadn’t measured the target distance in the first example would be unlikely to give the order to fire in the first place. In the second example, a tour guide who hadn’t measured the cord or the altitude would unlikely allow a customer to take the plunge.
Similarly, measuring the wrong things in business can lead to misdirected effort and valuable resources dedicated to the wrong goals, such as salespeople being measured by the number of deals they close rather than the dollar volume of those deals or baseball teams focused on less meaningful statistics rather than more telling figures.
For more than a century, many evaluators of baseball talent placed a high premium on batting average (BA, or hits divided by at-bats), until Bill James, the noted baseball historian and author, introduced his series of baseball abstracts.1 In these works, James demonstrated that on-base percentage (OBP, or hits plus walks plus hit-by-pitches divided by plate appearances minus sacrifice bunts) provided a much more reliable indicator of a player’s true value to a team.
For example, the Oakland Athletics teamwas one of the first teams to realize the superiority of OBP as a measure of a player’s productivity. The result? The Athletics has outperformed its much wealthier rivals—teams that may have spent more money on players with higher Bas—year after year by acquiring undervalued players at bargain prices.
Whether you’re failing to measure, measuring badly or measuring the wrong things—in operations or business processes—the ultimate consequences are the same: You are likely underperforming against the competition, as Oakland’s rivals did until they saw the light about OBP. Moreover, because competition today requires continuous improvement, inadequate measurement systems fail to provide the data required to make analysis and improvement possible. It’s a double whammy: You are not only currently underperforming against the competition, but also falling further behind.
When Frederick W. Taylor first introduced the concept of scientific management, it was just a hope. Today it is a reality. Leading companies have rigorously adopted statistical thinking and decision making techniques on an unprecedented scale. The days of managing by gut instinct are gone. Scientific management has moved from a skill that provides competitive advantage to a necessity for staying in the game. At the heart of the method that underpins scientific management lies measurement. It is, in effect, the “science” in scientific management.
Evaluating for the right metrics and creating measurement systems begin with an understanding that all work occurs in a system of interconnected processes and processes are subject to variation in their operation. Given those premises, we can first measure process performance—the outputs or the Y’s—to:
- Assess current performance levels.
- Compare current and past results to determine whether performance has changed.
- Determine whether the process needs to be adjusted (minor changes).
- Determine whether the process needs to be improved (major changes).
- Predict future performance.2
For example, consider a customer billing process in light of those uses of measurements. We might assess current performance by measuring the average number of days it takes to produce a bill. If we find it requires an average of eight days, we then might compare it to the average at a previous time. If it formerly required 16 days on average to get a bill out, then cycle time has been reduced by 50%.
On the other hand, some slight slippage in the average might indicate the process needs to be adjusted. An unacceptably high average or constantly deteriorating average might indicate the need for major change. Finally, studies might predict the removal of a bottleneck in the process could reduce the cycle time further.
What To Measure
When choosing which aspects of
process performance to measure, it is critical to pick outputs
that are linked to the bottom line, such as yield, defects, scrap
and waste, cost of poor quality, cycle time and productivity. In
a manufacturing environment, typical process performance metrics
might include those shown in Table 1.
For example, an individual manufacturing plant might measure performance by percentage of right first time, up time, production rate, on-time delivery, number of customer complaints, safety and cost.
- For nonmanufacturing processes, the most common measures of performance are:
- Accuracy as measured by correct financial figures, completeness of information or freedom from data errors.
- Cycle time as measured by how long it takes to accomplish a task, such as get out a bill, turn around a mortgage application or respond to customer requests for investment advice.
- Customer satisfaction as measured by degree of customers’ satisfaction and often determined through surveys.
- Cost as measured by materials, labor, equipment and other resources.3
Cost is a measurement that certainly is tied to the bottom line, but so are the other three measurements. An increase in accuracy, such as the accuracy of information on credit card bills, reduces costly rework like reissuing bills and increases customer satisfaction. Improvement in cycle time almost always improves customer satisfaction and increases productivity, thereby reducing internal costs. Improving customer satisfaction leads to improved customer retention, usually benefiting both the top and bottom lines.
A comprehensive performance
measurement system does not stop at simply measuring process
outputs (Y’s) and process variables (X’s). As Figure
1 depicts, it also can include measures that cover the three key
stakeholders in the process—suppliers, employees,
customers—as well as the performance of competitors. Key
measures of supplier performance, employee performance and
customer feedback, along with competitor benchmarks, then can be
used as a basis for determining where changes need to be made in
Deciding the X’s To Measure
Once the decision has been made about which Y’s to measure—the outputs that are critical to the customer and tied to the bottom line—you then can determine which X’s to measure. Some of these variables could be inputs from suppliers, such as variations in the quality of raw material arriving at a manufacturing plant or the accuracy of information coming to healthcare payers from providers. Some variables could be variations in employee behavior, such as different error rates among line operators.
Determining which variables to measure is much like conducting a design of experiments (DoE). Typically, three to six key variables drive a process. First, to arrive at those key variables from among a host of possibilities, brainstorm the potential X’s. Second, apply the Pareto principle to prioritize the X’s and identify candidates for the critical few with variations having the greatest impact on the outcomes of the process. Third, collect data and undertake correlation and regression studies to determine whether you have identified the critical X’s driving the process outputs, much as you would perform a multi-vari study.
Although multi-vari techniques can determine certain process variables are correlated with critical outputs, they cannot determine causation. An even better approach is to run screening DoE to identify the critical X’s. In the DoE, all the process variables are structured so it is easy to differentiate their effects on the outcomes. All the potential variables not being studied are strictly controlled so they do not affect the results of the experiment.
By running the experiments in random order, DoE also makes it virtually impossible for some unknown variable to change at exactly the same time as the variables in the DoE. As a result, the screening DoE yields high quality data that allows the cause and effect relationships between the variables being studied to be inferred with confidence. The cause and effect relationships tell us the most important variables to measure.
Maintaining a System
Neither processes nor business stand still. Over time, processes subtly and not so subtly change: New steps are introduced, new personnel introduce new variables, and changes in other processes affect an otherwise unchanged process nearby. Business objectives, customer expectations and competitive conditions also can change, perhaps rendering some Y’s obsolete or making some X’s less critical. Never assume yesterday’s measurement meets today’s conditions.
Measurement, then, is a process that must be managed and monitored continually as any other key process. Quarterly reviews will help you decide whether measurements can be added to the system or deleted. Also remember that taking measurements costs money. That’s why, in addition to assessing the relevance of a measurement, you also can perform a cost-benefit analysis of taking a particular measurement and making sure the return justifies the cost.
Although there’s concern about which measurements to take, it is important to note all process measurements entail two types of variation:
- Variation in the process itself.
- Variation created by the measurement process.
To fully understand process variation, we also must understand the magnitude of the measurement variation, which will help us better understand whether the process meets customer specifications. To maintain the integrity of the measurement system, we also must assess the stability, repeatability and reproducibility of measurements.4
Finally, it must be stressed that all measures must ultimately be tied to the highest level objectives and strategy of the organization. Since the early 1990s, many companies have attempted to make the connection through the balanced scorecard (BSC), a management—not measurement—system developed by Robert Kaplan and David Norton.5
The BSC was designed to incorporate a balance of financial, customer, process and learning metrics. Using this comprehensive set of measures, a company seeks to align its activities in all four areas around the company’s strategic goals and tracks performance to guide action.
Unfortunately, many companies—even those that employ the BSC—often fail to establish a clear and rigorous chain of connection and causation from process variables to process outcomes to financial and strategic outcomes. In fact, the BSC itself was first conceived because its originators understood financial measures are often lagging indicators that tell us about the past but not about the present or future.
This may explain why making the connection is so difficult and why so few companies do it successfully. Process output metrics (Y’s) provide leaders with indexes of business process performance.
Developing BSCs and linking them to process metrics is certainly worth the effort. Such integration among process measures and financial and strategic measures is invaluable for guiding improvement initiatives by:
- Identifying improvements likely to have the greatest business and financial impact.
- Narrowing the scope to highly targeted improvements that can be accomplished in short timeframes.
- Providing clear quantitative measures of business success as opposed to merely process success.
But the first step is to examine carefully what metrics you are using and resist the temptation to assume the metrics are necessarily the best ones simply because they’re there.
- Bill James, The Bill James Baseball Abstract, Ballantine Books, New York, 1982-88.
- Roger Hoerl and Ronald D. Snee, Statistical Thinking: Improving Business Performance, Duxbury, 2002.
- Ronald D. Snee and Roger Hoerl, Six Sigma Beyond the Factory Floor: Deployment Strategies for Financial Services, Health Care and the Rest of the Real Economy, Pearson Prentice Hall, 2005.
- Ronald D. Snee, “Are You Making Decisions in a Fog?” Quality Progress, December 2005, pp. 75-79.
- 5. Robert S. Kaplan and David P. Northon, The Balanced Scorecard—Translating Strategy Into Action, Harvard Business School Press, Cambridge, MA, 1996.
RONALD D. SNEE is principal of performance excellence at Tunnell Consulting in King of Prussia, PA. He has a doctorate in applied and mathematical statistics from Rutgers University in New Brunswick, NJ. Snee has received the Shewhart and Grant medals and is an ASQ Fellow.