3.4 PER MILLION
Sum It Up
Control charts for a composite summary of scorecard measures
by Joseph D. Conklin
The future of the quality profession is a perennially popular topic of conversation whenever my colleagues gather at the water cooler, the monthly section meeting or the annual ASQ conference. Practitioners without "quality" in their official titles periodically refresh, redirect and reenergize us. As long as the quality field rewards and recognizes these efforts, I am optimistic about our future.
The following case is an example. Irene from industrial engineering stopped by my office to chat after meeting with the company’s balanced scorecard1 team. Senior management, concerned about stagnating profits, was considering several remedial strategies, including chartering a team to look at balanced scorecards.
As a member of the team, the quality manager offered his department’s services for the cause. The team included a cross section of employees from the individual contributor, supervisory and executive levels. Irene was the designated member from industrial engineering. Because we had worked together in the past on manufacturing improvement projects, she thought of me first when thinking about what turned out to be an interesting question.
What to measure
While the team was open to recommending the services of an outside consultant, senior management asked the team to first see what it could accomplish with internal resources. In the first few meetings, the team naturally grappled with identifying the best measures to put on the scorecard. By the time Irene came to talk, the team was looking at seven measures:
- Net profits.
- External customer satisfaction as measured by an independent survey.
- Cost of quality as a percentage of gross sales (COQ%).2
- Total injuries per employee.
- Employee turnover as a percentage of total employees.
- Percentage of total employees certified in their jobs.
- Suggestions per employee.
Before addressing Irene’s question, I assumed the reasons for focusing on these measures were:
- They were already collected for other purposes, so we didn’t require a new system.
- They were the ones the company has the most experience measuring.
- The team felt the measures balanced financial, customer and employee perspectives.
The logic as presented seemed reasonable, so I asked Irene to clarify a few points:
"What does turnover include?" She said resignations, layoffs, retirements and involuntary terminations, but not internal transfers.
"What does total injuries include?" Minor, serious and fatal, she said. Fortunately, the company never had any in the last category. For the purpose of prevention, all injuries mattered, so the total was tracked.
"What’s counted as suggestions?" Ideas that have been implemented for at least six months with consistently positive results in terms of either cost or benefit, safety or employee satisfaction measures.
I thanked Irene for the explanations. She proceeded to her question by showing me the monthly values of the seven measures for the three most recent calendar years. You can find this data in Online Table 1.
She recalled from a prior conversation my point that control charts can handle measures outside of manufacturing. As an example, she had prepared the calculations (shown in Online Table 2 and graphed in Online Figure 1) for the individuals and moving range control charts3 of net profit, the first measure.
I encouraged her to analyze it as if it were one of any of the possible charts we might construct in the production department. With random movement inside the limits on the moving range chart, we turned our attention to the individuals chart. For 24 of the 36 months, net profit steadily increased. In the last 12 months, it stagnated.
She remembered some concerns about the role of the normal distribution assumption for individuals and moving range charts. That assumption matters most in the context of precise statements about process capability, I replied. Our focus is on the more fundamental issues of process stability and the search for nonrandom changes. In this case, the individuals and moving range charts are generally robust to the normality assumption.4
Irene was interested in looking at control charts to analyze the other six measures. She was open to an individuals and moving range chart for the customer satisfaction percentage and the COQ% measures. She thought the percent certified measure was a suitable candidate for a percent defective chart (p chart), while the remaining three fit the requirements for a defects per unit chart (u chart).5
I encouraged her to try individuals and moving range control charts for all of the measures. They can be useful to assess process stability, even for measures that might traditionally be placed on an attributes control chart. In some cases, especially for high-level business metrics, these control charts may be more appropriate.6 Using the same type of chart for all seven measures is also simple and convenient.
All into one
Irene expected the balanced scorecard team—now that it had made a first cut of the set of metrics—would next wonder about a way to summarize the metrics into one high-level measure for the entire company. She was interested in applying the z transform idea we had looked at when analyzing control charts in the production area. The idea is to have one chart accommodate several versions of the same product with differing means and possibly different variances.
The z transform originates in normal distribution theory in statistics. It converts any normally distributed variable to 1 with a mean of 0 and a standard deviation of 1. Statistical tables involving the normal distribution use the standard normal for reference.7
The equivalent application of this idea in the context of control charts is to create a new variable with a mean of 0 and a standard deviation of 1 by subtracting the mean of the data from each point and then dividing by some measure of the standard deviation.8 Using the control chart for profit as an example, the conversion proceeds as follows:
- Let X = a value for net profit.
- Let = the average value of the moving ranges for net profit.9
- Let s = measure of standard deviation = / 1.128.10
- Let bar = average of the values for net profit.
- Z = value of z transform = (X – ) / s.
The z transform calculations for net profit appear in Online Table 3.
Because Irene and I had individuals and moving range control charts in mind for all of the other measures, I encouraged Irene to apply the z transform calculations to the rest. The results appear in Online Table 4. For an overall control chart to summarize all seven measures, Irene and I created an and R chart using the seven values for each month as a rational subgroup. The supporting calculations appear in Online Table 5. The summary control charts are presented in Online Figure 2.
The range chart paints two different stories for the three-year period—a trend of declining variation in organizational performance during the first two years followed by a steep increase in variation during the last year. The chart suggests a declining trend in overall performance during the same period.11
For clues to explain the increased variation, Irene and I analyzed the control charts for the individual metrics. The net profit charts already have been addressed. The charts for the other six appear in Online Figures 3-8. Total injuries per employee, employee turnover as a percentage of total employees and the percentage of total employees certified in their jobs were plotted in control during the three-year period. We set those aside as likely contributors to the declining trend in the overall performance score.
The net profit, external customer satisfaction and suggestions per employee showed movements in the wrong direction for the most recent 12 months. Irene and I considered possible theories for the balanced scorecard team and senior management to investigate:
- Was the declining customer satisfaction reflecting a reliability problem? What story is the most recent warranty and return data telling?
- Was the declining customer satisfaction driven by a growing preference for a competitor’s product? How did the sales trend during the past year compare to our competitors’ products?
- Are we investing so much energy in scaling up production that the operators and engineers have insufficient time to propose and test improvements? Are we moving so fast that quality problems are slipping through undetected?
- Do our customers have issues with service and support after the sale?
- Are customers trying to use the product in ways we did not intend and are they disappointed when they can’t? If they are, does this suggest we should be offering new and different products?
- Is the potential market for this product smaller than we expected? Has the demand for it been met to the point that extreme price cutting is necessary to produce sales?
Irene had all the ingredients and a sample analysis for the balanced scorecard committee with respect to an overall performance measure. In addition to explaining this tool to the committee, I suggested the team should add more measures of customer satisfaction, such as on-time deliveries and cycle time for service and repair.
I congratulated her on trying to leverage appropriate quality tools, even if she didn’t carry a title that included the word "quality."
References and Notes
- For a comprehensive treatment of the balanced scorecard concept, see Praveen Gupta, Six Sigma Business Scorecard, second edition, McGraw-Hill, 2007.
- For a comprehensive treatment of cost of quality programs, see Jack Campanella, Principles of Quality Costs: Principles, Implementation and Use, third edition, ASQ Quality Press, 1999.
- For details of individuals and moving range control charts, see Eugene L. Grant and Richard S. Leavenworth, Statistical Quality Control, seventh edition, McGraw-Hill, 1996.
- For more details of the relationship between the normal distribution and control charts, see Donald J. Wheeler, Normality and the Process Behavior Chart, SPC Press, 2000.
- For an explanation of percentage defective and defects per unit charts with applications, see Grant and Leavenworth, Statistical Quality Control, reference 3.
- For more details of this point, see Donald J. Wheeler, Advanced Topics in Statistical Process Control: The Power of Shewhart’s Charts, SPC Press, second edition, 2004.
- For a discussion of the z transform, see Rudolph J. Freund and William J. Wilson Statistical Methods, second edition, Academic Press, 2002.
- For more details and examples of the points mentioned here, see the discussion of various z control charts in Donald J. Wheeler, Short Run SPC, SPC Press, 1991.
- For more details of the moving range, see Grant and Leavenworth, Statistical Quality Control, reference 3. The moving range, also known as the successive difference, is the absolute value of the difference between the current data point and the preceding one. This assumes that 1 is a rational subgroup size for the variable being charted. For high-level business metrics computed at regular intervals, this is typically a reasonable assumption. It should be verified whenever possible.
- The declining trend can be tested statistically by fitting a trend line through the overall scores for the past 12 months. For an example of testing a subset of data for a trend, see Joseph Conklin, "An Everyman’s Guide to Handling Data," Quality Progress, May 2010.
Joseph D. Conklin is a mathematical statistician at the U.S. Department of Energy in Washington, D.C. He earned a master’s degree in statistics from Virginia Tech and is a senior ASQ member. Conklin is also an ASQ-certified Six Sigma Black Belt, quality manager, quality engineer, quality auditor and reliability engineer.