By Their Measures Shall Ye Know Them
Measurement results can cause undesired behaviors
by Philip Stein
Measurements are so commonplace and universal they sometimes seem to be part of the air. They're so usual that they're not noticed. Still, they play a profound role in our lives and the way we behave.
People are motivated by many and complex forces, some of which are subtle and difficult to recognize. Other motivations, though, are obvious and are often structured by circumstance or even by deliberate design. Motivation by measurement is one of the more obvious examples of this.
People's behavior can be controlled to some extent by incentive systems. Both positive and negative incentives play a role here, and measurement can be an important or even crucial incentive driver. When people are measured, they tend to operate in a way that optimizes the measurement result, even though to do so may be counterproductive or even silly. When this happens, the measurement system is distorted and doesn't drive the desired results.
This was exemplified during one of my recent airplane trips. At the scheduled departure time, the aircraft door was closed and the passenger ramp was pulled back a couple of feet. The luggage wasn't quite finished loading, so we waited a few more minutes and departed without further incident.
What wasn't apparent to most passengers was that by pulling the passenger ramp back, a switch was automatically thrown that caused a computer to record an on-time departure. In fact, we hadn't departed at all, but the measurement system created an incentive for the plane to appear to have departed.
A similar situation takes place in manufacturing. Figure 1 is a fictional histogram of the distribution of a critical measurement, one that causes product rejects if out of spec. The production characteristics are such that you would expect this measurement to be roughly normally (Gaussian) distributed.
While the distribution does look Gaussian, there's a chip taken out of it. The chip includes values just below the lower specification limit. It's very unlikely a situation like this could have happened in the normal course of manufacturing variation. What could account for this pattern?
This is a well-known situation called flinching. There is pressure to ship the product and, therefore, to accept the product at this inspection step. The inspection consists of a measurement, followed by a decision based on the result. Any manual inspection has a built in incentive, and in this case the incentive is to liberally interpret the measurement result in favor of product release.
The curve is shaped as it is in Figure 1 because any measurement value below, but close to, the lower product specification limit is interpreted as being at that lower limit. Values even lower are far enough away from being almost okay that rejects are inevitable. This results in the gap in the curve, where everything below but close is called good, and no product is judged as actually having a measurement value inside that region. Below that region, rejects are judged as such, and the histogram once again reflects the actual product distribution.
Another incident happened while I was writing this article. I ordered a piece of software online, and I requested guaranteed overnight shipping. I really needed it overnight, too, because I was about to leave on another trip. When the package didn't arrive by 5 p.m., I tracked it by computer. It arrived in my town at 8 a.m. and was on the truck at 9 a.m. In the end, I drove to the shipper's depot and picked it up. The measurement issue is that the shipper blamed "poor weather." There was no weather problem. In this case, the incentive system was to generate inaccurate data in order to make the measurement system show that the system had succeeded when it had actually failed.
An overall problem
These three examples should be enough to communicate the kind of problem being presented by this article. Here it is in the general case:
* You set up measurement systems to gather data. Analysis of the data is supposed to indicate how well the enterprise is working in some specific regard. Your intention is to better manage the organization by driving these measurements to their optimum values.
* The details are then communicated to staff who must do the work and "make the numbers" or work toward specific goals that cause the measurements to look good. Notice at this point the not-so-subtle change from a goal to optimize the organization to one to make the numbers look good.
* The staff proceeds to behave in the manner that will make the numbers look good. If the measurement and entire management systems have been exquisitely designed and implemented, this incentive system will drive behavior so the organization will operate correctly or according to its design.
* If making the numbers look good will cause counterproductive or even destructive behavior, the incentive system will cause your organization to suffer and possibly fail, because the staff will always do its best to satisfy the measurement requirements.
If your management system has not been perfectly designed and the incentive system drives dysfunctional behavior, your employees will know. This is one of the primary reasons for quality improvement systems, such as teams, quality circles and other participative approaches. In an open, supportive work atmosphere, your staff will be glad to tell you when systems aren't delivering the desired results. Your staff can help fix those systems, too.
As a measurement scientist or engineer, it's easy to figure out how to collect the data, calibrate the instrumentation and wire up the sensors. It's not so easy to decide how people will behave in response to what you have decided to measure. Unless you consider the people who use your measurement system and the implicit incentives built into the ways of collecting data, you will have a good chance of driving undesired, distorted behaviors. Be sure to carefully observe the results after you have put a system in place. The way your measurements are being used should tell the story.
PHILIP STEIN is a metrology and quality consultant in private practice in Pennington, NJ. He holds a master's degree in measurement science from The George Washington University in Washington, DC, and is an ASQ Fellow.