## 2020

STATISTICS ROUNDTABLE

# Let’s Be Realistic

## Strengthen decision making with formal structure

by Christine M. Anderson-Cook

Decision making is all about understanding and comparing trade-offs between choices: What benefits are possible, what are the risks and can they be mitigated, and what will each of the alternatives cost? Consider each of these aspects in the context of a few examples to gain some understanding about what they might involve and what role statistics might play in assessing alternatives in the presence of imperfect or incomplete knowledge.

On the positive side of any decision, there are desirable consequences that may be realized. You should consider not only what is possible, but also the probability of each of the outcomes. The types of benefits can be multifaceted and include economic advantage and improved satisfaction, safety or security. The risks also have a similar makeup: The various outcomes each have different severity of consequence and associated probability. The third aspect considers the costs of the alternatives: What will I need to give up—in terms of time and money—to select this choice?

Articulating and quantifying benefits, risks and costs are often difficult and may require a substantial investment of energy and thought. However, a thorough examination including these elements will lead to improved decision making. In addition to this quantification exercise, it is also important to consider the role of uncertainty associated with each aspect. A proper understanding of uncertainty can give you a solid foundation on which to build you decisions.

### Playing the slots

To get started, consider a simple example. You are in front of a $1 slot machine with a dollar bill in your hand. An inquiry at the casino reveals the expected return on this particular machine is $0.75, and you’re trying to decide whether to part with your money. In the short term, there are two alternatives: play the slot or walk away.

First, the cost is pretty clear: Playing the slot will be $1 and walking away is $0. For the second choice, the risk-reward portion is also quite simple: There is no financial risk because there is no financial outlay, and there also is no financial reward. Alternative No. 1 is a bit more complicated, even though you know that the expected return is $0.75. Clearly, if there was exactly a $0.75 return on every slot pull, no one would play. But the expected return is a summary of the long-term behavior. For this example, you only have a single dollar to spend, so the long-term behavior is relevant but incomplete.

There are many potential ways of getting this expected return. For example, different (very simple) slot machines could offer a 10% chance of winning $7.50 (expected return = $7.5 x 0.1 + $0 x 0.9), a 1% chance of winning $75, or a 0.001% chance of winning $75,000, and all have the same expected return. It’s helpful to combine the consequence of a result with the probability of it occurring to give:

Risk = consequence x probability.

Benefit = consequence x probability.

Probability is the overall chance of the event occurring. In other applications, this is sometimes broken down into the probability of a result on each trial multiplied by the total number of trials for high-volume processes. For this slot machine example, the quantification of risk (what you could lose) and benefit (what you could win) is straightforward with the consequence and the probability of that outcome being clearly specified.

### The role of intangibles

This approach makes events with different characteristics more comparable by putting them on a similar scale. In this example, everything is measured in dollars. However, it is also a blunt comparison that obscures other important aspects of the decision.

For most gamblers, a key difference between the three slot machines with potential payouts of $7.50, $75 and $75,000 is the emotional excitement that a win can produce. The thrill of winning $75,000 (even if the probability of it happening is very low) is enticing and sways many people toward parting with their $1, even when the long-term expected financial return favors walking away.

Quantifying the emotional benefit of "hitting the big one" is difficult. But for many people, it is a key part of the decision-making process. This highlights two additional aspects of decision making: the role of intangibles and how your ability to assess alternatives improves with more complete quantitative information.

Consider investment strategies. These represent complex choices with many alternatives: stocks or bonds, indexed or managed, pre-tax or post-tax, and many alternatives within each of these combinations. Cost can be captured by fees, taxes and the time investment needed to execute a strategy.

Clearly, uncertainty plays a central role in the decision-making process. One of the key trade-offs for many people is expected gain versus the potential risk (as measured by the probability and size of a substantial loss). Given the range of potential outcomes (large gains through large losses), how well can we summarize the anticipated behavior of a particular strategy through a distribution of outcomes with associated probabilities? That is truly the million-dollar question.

If you associate a realistic range of uncertainty with the consequence and the probability for any of the outcomes, you can obtain better quantification of what the alternatives might yield. This will allow you to compare typical (mean or median) and worst-case performance (as measured by a lower quantile) in your decision-making process.

Any insights possible to refine this distribution of outcomes can give you an advantage in understanding. The intangible aspect for this example might relate primarily to your comfort level with risk and uncertainty, and your ability to handle loss. For some, having an aggressive, potentially winning strategy might not be worth the stress of the fluctuations this strategy might experience.

A critical difference from the slot machine example is that investment strategies are not bound by a fixed set of rules. Understanding what happened with the slot machine in the past is a good indicator of what to expect in the future, but the rules of investing and future returns are far less certain and more dynamic. Will future returns resemble the past or has the system fundamentally changed?

The uncertainty associated with this is much more difficult to capture than that uncertainty associated with the slot machine. Understanding the framework in which the decision making is occurring can play an important role in quantifying the probability and range of outcomes.

### Assessment advice

Here are a few other common misconceptions to avoid:

**1. Omitting the choice of doing nothing
from the assessment of alternatives. **Evaluating the cost and outcome
of doing nothing can be helpful for realistic assessment of alternatives. Don’t
fall into the trap of thinking plan A and plan B both have some disadvantages
you don’t like, so choosing neither must be better. Sometimes, lack of action
can have more dire consequences than moving forward in any direction. Don’t
assume that passivity has no consequences.

**2. Assuming that all outcomes are equally
likely****. **Sadly, I have had a version of the following conversation
about gambling and lottery tickets with friends: "There are only two outcomes:
You win or you lose, so the probability of each one must be 50/50." The number
of outcomes is generally independent of the probability of any particular
outcome and only rarely do all outcomes have equal probability.

**3. Believing a bad outcome was the result
of poor decision making.** Sometimes, even
the best decision making can yield an undesirable
outcome. In the post-decision assessment, try to separate the outcome from the
process of making the decision. Sometimes, you are just unlucky, but this
should not negate the fact that you followed a sound process to make the
decision. The opposite also is possible: You can see a great outcome despite a
non-thoughtful process. But you shouldn’t always count on this working out.

These simple examples illustrate how decision making can be complex, and it is certainly true that many decisions are made in the presence of incomplete or imperfect information. So what are the take-away messages?

**Don’t allow yourself to become paralyzed
by a lack of information. **Gather and quantify whatever relevant
information is available to allow comparison between risk,
benefit and cost for as many of the available choices as possible. Including a
fair assessment of the uncertainty associated with each of the options allows
differences in the quality or availability of data to be more easily compared.

**Think of statistics and data as vehicles
to facilitate better decision making. **How
can we package information in a way that feeds the decision-making process? It
is easy to fall into the trap of focusing on evaluating statistical
significance for a standard hypothesis that doesn’t really focus on the key
issue. Don’t fall into the trap of answering the convenient, but wrong,
question when a graphical summary might be more beneficial to highlight the
practical importance of the differences between alternatives.

Think how much the different objectives are
valued, use tools such as the Pareto front^{1} to eliminate
non-competitive solutions from further consideration and explore different
graphical summaries^{2, 3} that help visualize the trade-offs between
the choices. Making a defensible and rational decision begins with good information,
and hinges on converting that information into a form that allows understanding
and translation to the decision-making framework.

**Because the risks and benefits of most
decisions are based on what will happen in the future, think hard about the assumptions
that go into quantifying the consequences and probabilities.** Being
too simplistic or na�ve about how to measure these aspects can lead to false
conclusions. A common mistake is not building in enough uncertainty about what
you know and what could change. It can be a helpful exercise to compare
alternatives with your original assumptions and repeat the process for
different versions of the assumptions to see what impact these have on your
final choice.

**Think broadly about what to include in
the risks, benefits and costs.** For example, focusing on just the
financial cost is often too limiting, and risks often have intangible aspects
to consider. If you formally define these categories too narrowly, you may find
the results of your decision making leave you with nagging doubts, as your
subconscious is focused on something important that was excluded in assessing
the trade-offs.

Just being aware of the decision-making process and being thoughtful about how to approach it with some formal structure will improve the quality of the decisions made. Formalizing the process, while enumerating and visualizing the alternatives, leads to better decisions.

### References

- Edward
M. Kasprzak and Kemper E. Lewis, "Pareto Analysis in Multiobjective Optimization Using the Collinearity
Theorem and Scaling Method,"
*Structural Multidisciplinary Optimization,*Vol. 22, 2001, pp. 208–218. - Lu
Lu, Christine M. Anderson-Cook and Timothy J.
Robinson, "Optimization of Designed Experiments Based on Multiple Criteria
Utilizing a Pareto Frontier,"
*Technometrics**,*Vol. 53, 2011, pp. 353-365. - Lu
Lu and Christine M. Anderson-Cook, "Rethinking the
Optimal Response Surface Design for a First-Order Model with Two-Factor
Interactions, When Protecting against Curvature,"
*Quality Engineering,*Vol. 24, 2012, pp. 404-422.

**Christine M. Anderson-Cook** is a research scientist at Los
Alamos National Laboratory in Los Alamos, NM. She earned a doctorate in
statistics from the University of Waterloo in Ontario. Anderson-Cook is a
fellow of the American Statistical Association and ASQ.

Featured advertisers