Q: I recently reviewed customer satisfaction data with a manager. His satisfaction scores rose 1% from the prior period, so he was pleased. I noted the change probably did not represent a meaningful change in customer experience, and he responded, "But it’s statistically significant."
He was right. The problem was that it did not make a difference, and he was confusing "significant" with "meaningful." Can you explain the difference between the two?
A: While once reserved for scientists and researchers, the term "statistical significance" has become part of business lexicon. Unfortunately, among casual users of statistics, its meaning is often confused. Statistical significance does not tell you whether a change matters. It only tells you whether the change is real.
Part of the problem is with the word "significant." In the business world, it means important or impactful. A significant business deal probably has a high dollar value associated with it. A significant business setback is quite serious.
In statistical terms, significance also means important or noteworthy—but for a different reason. It is not a measure of the impact something has on something else. It is a measure of certainty. It indicates whether the difference between two numbers is real and is not simply a result of random chance or coincidence. Confusing these two interpretations leads to bad decisions and actions.
As a leader, both aspects of significance are important. It is important to determine whether a change actually matters. A 1% change in customer satisfaction probably does not mean much. But a one-tenth of a millimeter change could be the difference between a gear that works and one that does not.
The question of whether a change matters is based on context, not statistics. It is equally important to understand whether that change is real and whether it happened because of your actions. This is where statistics can help.
Often, leaders make the mistake of ignoring the context based on the statistic. Statistical significance is not the same as importance. Importance is a call a leader must make based on his or her understanding of the business. Do not get fooled by statistically significant findings that confirm—with certainty—that not much has changed.
Q: Why is the common term mean time between failures (MTBF) misunderstood? How can we effectively change the misuse of the metric? As quality professionals, how do we effectively influence the rest of the engineering and procurement community to fully specify reliability appropriately?
Los Gatos, CA
A: MTBF is an important but incomplete metric of the reliability of repairable systems. It is an estimate of the mean of observed durations of time between failures. It is often described with exponential or Weibull probability distribution for the steady state portion of the system life with a constant failure rate.
Larger MTBF means longer functioning of the system (product) between failures and repairs. It is achieved through improvements in product and process design, manufacturing, and preventive maintenance, which has its own metric: mean time of preventive maintenance (MTPM).
There are two common classes of failure: breakdowns and drifts of performance characteristics outside the specifications due to wear or changes of the environmental conditions.
Quality of raw materials and purchased components used in production are very important if you want to increase MTBF. You can also achieve it by using robust design (Taguchi method) that optimizes system parameters to make the system performance less sensitive to wear of various components.
MTBF is commonly confused with useful life. For example, a battery may have a useful life of four hours and an MTBF of 100,000 hours. These figures indicate that in a population of 1 million batteries, there will be approximately 10 battery failures every hour during a single battery’s four-hour lifespan.
Whenever a repairable system fails, it needs to be repaired. An important reliability metric of repairable systems is mean time to repair (MTTR), which should be as small as possible.
MTPM and MTTR both are reduced by design for maintainability and reparability, the latter of which may include built-in automatic test equipment to reduce time of diagnosing failures. These metrics can also be reduced by better organization of the processes of preventive maintenance and repairs—such as trained personnel and availability of spare parts.
A good metric of reliability that accounts for MTBF, MTPM and MTTR is system availability, which characterizes the probability of a system to be available or operational at a particular time. In a steady state, it is the proportion of time the system is operational. The latter is the ratio of the MTBF divided by the sum of MTBF, MTPM and MTTR.
It is not a simple matter to, as you phrased it, "fully specify reliability appropriately." But reliability costs can be minimized by optimal system design, production and organization of maintenance and repairs.
As far as your question about changing the misuse of these metrics, the best—and perhaps only route—is through education. ASQ’s Reliability Division (http://asq.org/reliability) may be a good place to start.
Roche Molecular Systems
Q: I’m on an ISO 22000 team, and people are arguing about preventive maintenance equipment (in clause 7.2.3). The standard says, "The organization must consider preventive maintenance equipment." Some people argued that preventive maintenance is not an obligation. But to me, that is an erroneous interpretation of ISO 22000 because when the word "must" appears, it is an obligation to comply, not a suggestion. Can you help us?
Buenos Aires, Argentina
A: The standard requires in clause 7.2.3 "the consideration of maintenance and preventive maintenance." You can argue that all that needs to be done is the consideration of maintenance and preventive maintenance programs.
If the food-safety team does not consider the maintenance and preventive maintenance programs part of the prerequisite programs, then the maintenance and preventive maintenance programs can be excluded from the prerequisite programs.
One critical fact going against this interpretation of ISO 22000 is that the Codex Good Hygiene Principles, PAS 220, ISO 22002-1, and the hazard analysis and critical control points (HACCP) system described by the National Advisory Committee on Microbiological Criteria Foods classify maintenance and preventive maintenance programs as part of the prerequisite programs for HACCP. Thus, it is my interpretation that maintenance and preventive maintenance programs are a mandatory part of the prerequisite programs.
This entire argument becomes moot if you are implementing a food-safety management system that complies with either FSSC 22000 (ISO 22000 and PAS 220) or Synergy 22000 (ISO 22000 and ISO 2202-1). In both cases, the maintenance and preventive maintenance programs are mandatory.
Surak and Associates