Drudgery to Strategy—A Statistical Metamorphosis
A strategy of experimentation can point you toward success
by Lynne B. Hare and Mark Vandeven
Think back to your Stats 101 course. You entered the first session laden with apprehension—induced by survivors’ horror stories—and your worst fears were confirmed. Early on, the professor said, "OK, boys and girls, today we’re going to discuss t-tests and confidence intervals." And you sat there thinking you’d rather visit the dentist.
Tools and occasional toy (artificial) problems characterize many introductory statistics courses. To be sure, the professors are gifted, enthusiastic lecturers; many have great humor and human kindness in their veins. But still, the class dealt with statistical tools suitable for fixed occasions.
At semester’s end, you knew about a bunch of tools you could resurrect if the right occasion ever arose.
It never did.
There’s no getting around the fact that statistics is a difficult subject. The thinking is different from that of many other disciplines: It acknowledges uncertainty, whereas others profess determinism. It runs counter to everything we are taught in algebra: "Solve for x, a fixed but unknown quantity."
Further, statistical applications require following mathematical formulas of which there are relatively few in political science, English, history or philosophy. Even pure mathematics claims closure (not always true), which is absent in statistics. The semester usually ends before the professor scratches the surface. And because the first semester lacks a happy ending, many students are reluctant to proceed to a second.
On the rise
Over the last two decades, we’ve noticed statistical stock is rising at some companies. They may be in the minority, but some top executives have begun to see statistics’ strategic value.
We believe this occurred because those executives witnessed very positive (read: dollars to the bottom line) results, especially when scientists and engineers used statistical methods to guide projects from highly uncertain project beginnings to solid, successful and sustainable products and processes. To their internal statisticians, executives are saying, "I want more of that. Make it happen."
"Egad," says the internal statistician. "Now what do I do? All that my colleagues learned from their statistics course was not to take another one. The only statistical methods they use come at the end of a project, when they compare the new prototype against the current product. And now I have to change the organizational culture so people not only use the methods, but also use them upstream in the development process for guidance. I’d rather visit the dentist and have root-canal surgery!"
Well, this is not an appeal for long lines at dentists’ offices. Positive results can emerge from some lessons learned, if only by osmosis, in Stats 101. An appeal to intuition reduces resistance to the notion that the t-test used to compare a sample mean to some hypothesized value is analogous to signal-to-noise, for example. Build on this to show that a reduction of noise makes it easier yet to hear the signal. Build on it further to show that the test can be expanded to compare two sample means as in Stats 101, chapter 3, but it’s still signal to noise.
It is a bit of a leap from there to compare more than two treatment means, but it can be shown that the guiding principle remains akin to signal to noise. Then, if you want to compare multiple treatment means, wouldn’t it be wise to economize by running half the experimental treatment combinations on one material and the other half on another? Doing that introduces a two-way classification painlessly—almost. The discussion with colleagues dredges up their unpleasant memories, true, but at least they form a base for intuitive appeal.
If you can study multiple treatments and multiple materials, why not add multiple speeds? Now you have a three-way classification. Conceivably, you could add other factors, such as ingredient levels, machines and locations. Whoa! Wait a minute and those experiments will get big in a hurry. Big experiments cost big bucks, and they are logistically hard to control. Fugetaboutit!
To the rescue come the two-level designs and their fractions. Don’t do multiple factor levels. Examine only two levels per factor, and make those levels the extremes. "Absurd," you say, "some of those levels could be best!"
"Right," we say in appropriately humble rejoinder, "but we don’t even know which factors are important right now. Let’s experiment to find which factors are most likely to drive success. We can hone in on levels in subsequent experiments."
This is the great leap or paradigm shift—we move from comparing treatment or material A and B to estimating something called an "effect." The question has changed from "What factor level is best?" to "Which factors are most important?"
"You mean I have to do more than one experiment?" you ask. Yes. The first one or two identify which factors or combinations of factors are important. The next experiments help identify best levels among the important factors. It is a strategy for experimentation, and it has a higher success rate than competitive approaches to experimentation such as spray and pray, try everything, and try your best hunch and hope you get lucky.
We see other advantages to the strategy of experimentation, especially as it is used to drive decision making upstream in the research process. For one, the competitive method of giving it your best shot and testing at the end simply tells you if you were successful. If you weren’t, you don’t know why.
The strategy of experimentation puts data behind your directional decisions so you know, early in the experimental process, what path to take; there are no blind alleys. You gain product and process knowledge along the way.
The strategy of experimentation also provides trade-offs. If practical constraints block the path to using one combination of factors and their levels for success, there may be another combination that comes close. The data point you in the direction toward success.
We’ve found that those who use the strategy are successful and they feel liberated. To go back to the old methods would be like having more oral surgery.
Lynne B. Hare is a statistical consultant. He holds a doctorate in statistics from Rutgers University in Camden, NJ, is past chairman of the ASQ Statistics Division and is a fellow of ASQ and the American Statistical Association.
Mark Vandeven is associate director of statistics at Colgate-Palmolive in Piscataway, NJ. He received a doctorate in decision sciences/statistics from Rensselaer Polytechnic Institute in Troy, NY. Vandeven is a member of ASQ.