Different Kinds Of Wrong
Understanding how statistical data collection and analyses get derailed can help avoid pitfalls
by Christine M. Anderson-Cook
In an episode of “The Big Bang Theory,” Sheldon Cooper and Stuart Bloom debate some aspects of superheroes, when Stuart says, “I’m afraid you couldn’t be more wrong.” Ever the stickler for correct use of terminology, Sheldon challenges Stuart with “More wrong? Wrong is an absolute state and not subject to gradation.” Stuart clarifies with “Of course it is. It is a little wrong to say a tomato is a vegetable; it is very wrong to say it is a suspension bridge.”1
When collecting or analyzing data, things often are imperfect, and we may worry about how statistical analyses, interpretation of results and conclusions affect decision making or problem solving. In addition to gradations of wrong, there also are different types of wrong. This column explores a few of the varieties I've encountered with an illustrative example, thoughts about potential impact on results and ideas for avoiding these pitfalls.
Using the wrong method
Statistical methods are designed for particular forms of data and to answer specific questions. If we observe pass-fail data and wish to model how the success rate varies as a function of one or more inputs, for example, logistic or probit regression is a method ideally suited for this scenario.
If, instead, the practitioner used regular linear regression to model the response, a variety of issues might result. First, it is possible with a linear regression model to predict outside of the commonsense bounds of [0%, 100%] for the success rate. Secondly, the uncertainty bounds for the intervals at various combinations of inputs also would not be suitable for the proportion that we are trying to estimate. So, using the wrong method can give unexpected results—with little warning of what to expect and how the method will fail.
Clearly, the strategy for solving this problem is finding the right method. For a wide class of problems, there are suitable statistical methods that are a good match for what is needed.
If you're a novice statistical user, it is helpful to consult with an expert to verify that an appropriate method has been selected. There also are excellent resources available in textbooks and online that describe what the question of interest is and requirements for the inputs and response for different analyses. For more experienced statistical users, it's worth cataloging what methods you know and the problems they're designed to solve.2
Using a method incorrectly
For years, I’ve had a running debate with a former colleague about why he prefers frequentist methods to Bayesian ones. His premise has always been that if you use a wildly wrong prior distribution, you will produce a silly answer. To me, this is not a useful starting point for comparing two methods. It is in effect the equivalent of saying, “I don’t use linear regression because if you use the formulas wrong, you get outrageous answers.”
We should not be evaluating the merits of an approach by how it performs when it is used incorrectly—I can just about guarantee that no method will survive that assessment. We should be looking at how to best implement methods to maximize their potential, then comparing them under those conditions. When a method is implemented incorrectly, we have no idea what to expect from the results.
To avoid being in this category of wrong, it can be helpful to use existing software packages or implementations, after carefully checking to make sure the fundamentals of your problem are well-suited to this method. To verify the execution of the method turned out well, do some simple sanity checks to make sure that the results are reasonable. With linear regression, for example, it is helpful to verify that the slope and intercept estimates from the model have the expected sign and magnitude based on the current understanding of the relationship.
Objective vs. subjective
Treating something that is objective as subjective—or vice versa—can cause a lot of trouble. Our modern political landscape is the poster child for treating facts as negotiable and saying that someone’s opinion is wrong.
To illustrate this with a less-controversial example, consider trying to choose the best supplier of a component for a product.3 With suitably defined metrics, we can objectively assess the cost and quality for parts from each supplier. Hence, in a comparison, we should be able to say whether one supplier’s component is cheaper or more expensive than another’s. Similarly, with a precise definition of quality, we should be able to say whether one component is better than another.
But if one person on the team thinks that lower cost is more important than better quality, this is a subjective assessment for which there likely is no single right or wrong answer. Understanding the difference between objective and subjective is highly beneficial for productive discussions and defensible decision making.4 Nothing distracts from the task at hand faster than calling someone’s priorities—which are based on their underlying values—wrong.
For objective aspects of a decision, we want to optimize and achieve the best possible value. For subjective aspects, we often want to determine whether there is a robust solution that honors the different preferences represented on the team. A key to avoiding this mistake is to think critically about what the basis for each judgment is, and to avoid inadvertently injecting your opinion or initial premises into the evaluation of choices.
Another version of error that we can make is to consider too limited a set of choices. We can sometimes try to make decisions more manageable by restricting the number of alternatives to consider. It seems much easier to choose between A and B, instead of entertaining a larger number of possibilities.
Taken to an extreme, presenting just a single choice makes the decision-making portion easy, but it certainly does not lead to making good decisions. Stephen Covey5 advocates for looking beyond obvious alternatives and seeking a win-win solution.
A helpful way to improve the quality of the choices available is to have a formal phase of developing alternative candidate solutions, including brainstorming and seeking to combine available solutions. The combining part of the process can be aided by highlighting the strengths of each solution for different aspects of the problem and trying to extract the essence of that strength to leverage as part of a new hybrid solution.
Wrong, but useful
Many know George E.P. Box’s quote, “All models are wrong, but some are useful.” This highlights the essence of Stuart’s earlier quote from “Big Bang”: If a model or approach is a good match in several critical aspects of the problem, minor imperfections might not render the approach unusable. For example, many statistical approaches are built on an independent and identically distributed (i.i.d.) assumption. There are many instances when either of the “i’s” aren't strictly true, but the statistical methods may be robust enough to overcome small failures in these assumptions.
In some cases, if the actual structure of the data cannot be precisely characterized, either because of its complexity or it is unknown, starting with a slightly flawed assumption might be the only expedient choice available. What is important, in these quite common cases, is to carefully use diagnostic checks to assess the degree to which the assumptions are violated and to remain aware of their potential impacts on the results.
So, there are different flavors of wrong, and despite what Sheldon thinks, there also are different gradations of wrong. Understanding how and where our statistical data collection and analyses can get derailed may allow us to avoid unnecessary pitfalls. This will enable us to solve our problems or make the required decisions with better awareness of how imperfections can affect our results.
Note and References
- The statement appeared in “The Big Bang Theory” episode, “The Hofstadter Isotope,” which first aired on April 13, 2009, during the show’s second season. Watch the video clip at www.youtube.com/watch?v=F_1zoX5Ax9U.
- Christine M. Anderson-Cook, “Taking Stock: Know What Statistical Tools You Have and How to Use Them,” Quality Progress, September 2011, pp. 60-61.
- Christine M. Anderson-Cook and Lu Lu, “Weighing Your Options: Decision making with the Pareto-front approach,” Quality Progress, October 2012, pp. 50-52.
- Christine M. Anderson-Cook, “The Inevitable Disagreement: Constructive Strategies to Move Beyond,” Quality Progress, October 2017, pp. 48-49.
- Stephen R. Covey, The Third Alternative: Solving Life’s Most Difficult Problems,” Free Press, 2012.
Christine M. Anderson-Cook is a research scientist in the Statistical Sciences Group at Los Alamos National Laboratory in Los Alamos, NM. She earned a doctorate in statistics from the University of Waterloo in Ontario, Canada. Anderson-Cook is a fellow of ASQ and the American Statistical Association. She is the 2018 recipient of the ASQ Shewhart Medal.