Making that elusive perfect decision: a primer
by Christine M. Anderson-Cook
We live in a complex world in which getting exactly what we want is often difficult or impossible. The notion of a trade-off is something we are forced to confront almost daily. In fact, Facebook CEO Sheryl Sandberg summarizes our experiences well: "The very concept of having it all flies in the face of the basic laws of economics and common sense because there’s always a trade-off."1
So how should you make decisions when balancing multiple objectives? In the March Statistics Roundtable column,2 I presented some ideas about how to formalize the decision-making process with a two-stage Pareto front approach.3,4 But this is predicated on knowing what objectives to focus on, how to translate them into quantitative measures and being able to identify the available choices. Now, let’s consider some of these preliminaries.
First, how do you identify and prioritize your objectives? Often, there are one or two primary or essential aspects—the bread and butter—of the decision. In a job search, this might be a good work environment and an adequate salary. In selecting a supplier, this might be superior quality at an affordable price. In looking for a mate, this might be compatible life view and priorities. These are the deal breakers, in which an acceptable solution must perform to at least a minimum standard to be viable. If you cannot satisfy these constraints, you might walk away from the decision or postpone committing to any of the available options.
A second tier of criteria consists of those attributes that are highly desirable, but clearly less critical than the first category of criteria—these are important, but not essential. In job searching, for example, these might include benefits, location and work environment. The final category includes tertiary considerations—characteristics that would be nice, but they’re not essential.
What is important when establishing the ground rules for your decision making is to be clear about which criteria fit in which categories. Why does this matter? Suppose you focus exclusively on one aspect and ignore all of the secondary and tertiary considerations. In this case, you will likely end up with a fabulous result on your one objective, but potentially terrible performance on one or more of the other aspects.
In selecting a supplier, for example, if you only focus on quality, you might pay an enormous premium for the very best quality, when a small step down in quality might yield a disproportionately large drop in cost.
The other extreme also can be detrimental. Suppose you include all primary, secondary and tertiary criteria formally into your decision-making process. This might yield:
- A solution that is mediocre on all objectives because of the large number of trade-offs that are being considered simultaneously.
- No solution.
Consider the scenario of finding a mate: If you have a vast laundry list of criteria that each eliminates a portion of the eligible population of mates, you may find the likelihood of your match existing becomes dismally small. Potentially, this is compounded by any inherent correlations between requirements that lower their chances of occurring simultaneously, such as finding a spouse who enjoys leisure and relaxation, while at the same time having a highly successful career.
Some might argue that choosing a mate has aspects that cannot be easily measured or compared. While this is undeniably true, there are some dimensions of the decision that can be organized and quantified to help shape thinking and formulate a viable strategy.
The proper balance requires you to have enough of the essential characteristics identified so you don’t oversimplify the problem, which will likely result in a loss of opportunities. On the other hand, you also don’t want the search to include extraneous criteria that are unnecessary. All of this requires a careful and thoughtful examination of your priorities, while being mindful that wanting too much (on any individual criterion or on how many criteria) has consequences.
It should be clear that having a larger pool of candidates is advantageous. In many situations, you may have a natural list of choices dictated by circumstances, such as "the number of jobs for which I am eligible or have offers." In other cases, there may be opportunities to expand the candidate list.
An obvious example is in the finding-a-mate scenario: There are numerous webpages and dating sites expounding the ability to connect you with more potential matches. Having more candidates is beneficial for finding one that best matches your desired balance of the competing trade-offs among criteria, but the extra time, effort or expense of expanding the search also should be considered.
In some cases, such as selecting a designed experiment, it may be possible to create a search algorithm that will enable a large number of possible candidates to be screened and sorted to identify promising candidates.5
Translating measures into scores
Another important consideration when balancing multiple objectives is how to translate a qualitative characteristic into a consistent numerical score that can be used with an optimization approach, such as the Pareto front. If you are focused on just a single objective, the key ideas for this translation include:
- Choosing a scale to provide an adequate spread of choices to distinguish between alternatives.
- Documenting specific characteristics or samples associated with particular score values.
- Spot-checking pairs of alternatives with similar values to judge their comparability.
- Spot-checking pairs with different values to verify an intuitive ordering based on the scaling.
In the job search scenario, for example, you may wish to establish a scoring system for characterizing "work environment." You may use a range of values from zero to 10 for each of these objectives, with larger values being preferred. After looking at all initially available candidates, identify a best or near-best candidate, and list some of the aspects that make it the best choice. Use this to define attributes of a best or near-best score. If you can imagine something being more ideal than what is currently available, you may want to assign a score of less than 10 to this category.
This process should be repeated for a worst or near-worst candidate, with the numerical value that it is assigned being sufficiently spread away from the best to allow adequate space to place other candidates. As more candidates are considered relative to one another, this should help formalize the definition of different categories, and provide a consistent and documented set of requirements for the different scores.
An additional aspect that must be included when dealing with several objectives simultaneously is to look at the comparability of similar scores across different qualitative criteria to judge how they compare. Suppose for the job search scenario, you wish to compare salary (a naturally quantitative summary measured in dollars) with scales that we have created for "work environment" and "benefits" (with scores constructed using the process described earlier).
It is helpful in evaluating trade-offs if you can compare the ranges of scores available for each objective. For example, the range of salaries might have a best value that you consider "excellent," while the worst value is "terrible." For "work environment," the range of scores might cover "excellent" to "moderate," while for "benefits," the range is "good" to "terrible." Clearly, getting an excellent score for all three objectives is not going to be possible, but your assessment of the spread of your choices and how close to ideal they are can be helpful when you think about combining them into an overall score.
While formalizing the process for comparing alternatives based on multiple objectives requires effort and a structured approach, the payoffs of better understanding the inherent trade-offs and being able to identify a justifiable, defendable final choice are worthwhile. The advantages are only amplified when there are several decision makers with potentially different priorities involved in the process.
References and Note
- Sheryl Sandberg, Lean In: Women, Work and the Will to Lead, Knopf, 2013.
- Christine M. Anderson-Cook, "Let’s Be Realistic," Quality Progress, Vol. 46, No. 3, March 2013, pp. 52-54.
- Lu Lu, Christine M. Anderson-Cook and Timothy J. Robinson, "Optimization of Designed Experiments Based on Multiple Criteria Utilizing a Pareto Frontier," Technometrics, Vol. 53, No. 4, 2011, pp. 353-365.
- Lu Lu and Christine M. Anderson-Cook, "Rethinking the Optimal Response Surface Design for a First-Order Model With Two-Factor Interactions, When Protecting Against Curvature," Quality Engineering, Vol. 24, No. 3, 2012, pp. 404-422.
- One possible algorithm for efficiently finding desirable designs is described in Lu Lu and Christine M. Anderson-Cook’s "Adapting the Hypervolume Quality Indicator to Quantify Trade-Offs and Search Efficiency for Multiple Criteria Decision-Making Using Pareto Fronts," Quality and Reliability Engineering International, Sept. 19, 2012, http://onlinelibrary.wiley.com/doi/10.1002/qre.1464/pdf.
Christine M. Anderson-Cook is a research scientist in the statistical sciences group at Los Alamos National Laboratory in Los Alamos, NM. She earned a doctorate in statistics from the University of Waterloo in Ontario. Anderson-Cook is a fellow of both ASQ and the American Statistical Association.