3.4 PER MILLION
Assessing the Effectiveness of Controls Under Uncertainty
by Joseph D. Conklin
Six Sigma program is about implementing effective controls. An interesting issue arises when a state-of-the-art system falls short of zero defects.
The word controls refers to any corrective or preventive action to reduce or eliminate a defect. In some systems, the causes and effects are so well understood that diligent implementation of the controls leads to near-perfect elimination of defects.
An example is the fire prevention and control principles I learned in the Navy. Fires require three ingredients: fuel, heat and oxygen. To prevent fires, you can segregate the fuel from the heat and oxygen by using approved storage containers. To control fires, depending on the fuel, you can apply foam to smother the oxygen or water to cool the fuel to below the point of combustion.
A system can often fall short of zero defects in service processes in which the human interaction components are difficult to understand and model.
What is a fair way to assess the effectiveness of a control when system knowledge is incomplete and uncertain? The absence of a defect for some extended period of time does not necessarily mean a control has reduced its probability to zero.
This article covers two statistical techniques—sequential sampling and logistic regression—that are options for assessing effectiveness in this situation.
Controls Over a System
There are two classes of controls: discrete and continuous.
A discrete control is the complete removal of an element from a system. Examples include: removing vendors from an approved purchasing list, eliminating certain operators from running a process and converting to a new material when updating a part design.
A continuous control does not completely remove a factor. The factor is allowed to operate at a lower or higher level depending on the results of the Six Sigma investigation. Examples of continuous controls include: temperature controls, changes in an allowable impurity percentage of incoming material and adjustments in the frequency of equipment maintenance.
With discrete controls, sequential sampling can be used to assess effectiveness under the following conditions:1
- The defects occur independently of one another. Because a defect has already occurred does not mean the next occurrence is more likely.2
- The probability of a defect in the absence of the control is known. Call this the status quo case. This might be measured by a daily defect rate.
- The desired defect probability is known, perhaps as a result of a benchmarking study.
- Both probabilities are measured using the same standards or procedures.
- The goal of the control is to attain the desired defect probability or at least improve over the status quo.
Discrete Controls Example
To illustrate, consider a computer services department that writes programs. Suppose the status quo probability of a defective program on any given day is 0.13. Let’s say the desired probability is 0.01. Our discrete control is a new strategy for writing programs. We count the number of programs written after implementation and cumulate the number of defective programs produced over the assessment period.
At the end, we want to conclude whether the defect probability is at the desired level or at the status quo. Regardless of the conclusion we make, there is a chance of being wrong. We must specify in advance the level of risk we are willing to accept. For this illustration, let’s say we are willing to accept a 0.10 (10%) risk of being wrong if we conclude the desired defect probability has been reached. Also, let’s say we are willing to accept a 0.05 (5%) risk of being wrong if we conclude we are still at the status quo.3 With these risks, we can use Table 1 to draw our conclusion. The details of constructing Table 1 are explained in “Steps for Constructing Table 1—Systematic Sampling Decision Table.”
The assessment period must cover at least 23 programs before we have any counts necessary to conclude we have attained the desired level. To conclude we definitely have attained the desired level, by some point the cumulative number of defective programs must be less than the corresponding cumulative count in the second column of Table 1.
Based on the assumptions made in this example, this does not become possible until 44 programs when the cumulative count increases from zero to one. If we can go at least 44 consecutive programs without any defects, we have evidence we are at the desired defect probability of 0.01.
Similarly, we can conclude we are at the status quo if, by some point, we have exceeded the corresponding cumulative count in the third column of Table 1. If at least one of the first three programs is defective, we have evidence of remaining at the status quo. The same can be said if at least two of the first four or three of the first 25 programs are defective.
The advantage of Table 1 (p. 65) is the possibility of reaching a conclusion before the scheduled end of the assessment. How many items should be included in the assessment? Within the desired risk levels of most practical applications, 30 is a reasonable minimum to expect.
The lower the risk level, the more items must be included. The total included must balance the need for reasonable risk against the desire for timely results. Ideally, a range of risks and item quantities should be considered with the goal of selecting one that represents the best compromise.
It is possible to complete the assessment without a firm conclusion because the cumulative count finishes between the limits of columns two and three. The practical implication of this result is we are somewhere between the old status quo and our goal, and more work is required to get there.
Continuous Control Conditions
The same conditions that support the application of sequential sampling for discrete controls also support the application of logistic regression for continuous ones. The formula for logistic regression is:
p(x) = ey (1 + ey),
in which p(x) is the probability of a defect.4 In the formula, y can be a response variable, but it typically represents a function that incorporates values of factors involved in continuous controls. In the formula, y can be a function of several factors: x1, x2, going up to xk. Linear functions of the x’s are popular. This leads to:
y = a0 + a1x1 + a2x2 + ... + akxk.
The coefficients a0, a1, a2, ..., ak can be calculated by linear regression techniques.5
For illustration, we consider a case of a single factor x1. The corresponding function for y is:
y = a0 + a1x1.
Suppose x1 refers to the number of hours taken by a loan department to complete applications from customers. The loan department suspects longer completion times are associated with a lower probability of approval. The department wants to quantify this relationship so it can set reasonable improvement goals for the loan approval process.
It consults its files on 1,000 recent applications and assembles the data for completion times, number of approvals and the finish rate, shown in Table 2. The finish rate is the number of approved applications divided by the total number of applications in a time period or group.
The 1,000 applications form 10 groups of 100 each. Group one’s applications took one hour to complete, group two’s applications took two hours and so on.
The column in Table 2 labeled “finish rate” estimates p(x) for each group of 100 loan applications. To get the appropriate expression for y so we can apply linear regression techniques to obtain estimates of a0 and a1, we must start with the formula p(x) = ey/(1 + ey) and apply the algebra at the beginning of “Steps for Application of Logistical Regress-ion for Continuous Control” (p. 65) to rewrite the equation so y is on the left hand side. When we do this, we find the equation is:
y = ln(p(x)) – ln(1 – p(x)).
Because the finish rate estimates p(x), we can substitute its values into the equation
y = ln(p(x)) – ln(1 – p(x))
as the next step. Doing this leads to the transformed values for finish rate shown in Table 3. They serve as values for y, the response variable, in a linear regression program. The values for the hours to complete each group of loan applications serve as values of x, our single predictor variable for y.
The regression program in the data analysis toolkit of Microsoft Excel gives us estimates for a0 and a1. The estimates can be read from the edited excerpt of the program reproduced at the bottom of “Steps for Application of Logistical Regression for Continuous Control.” The estimate for a0 is 2.666149. The estimate for a1 is –0.469446. So our prediction equation becomes
y = 2.666149 – (0.469446 * x1),
or in the terms of our example, predicted transformed finish rate = 2.666149 – (0.469446 * hours to complete loan application).
Reviewing the Residual
With this equation, we can construct a predicted transformed finish rate for each group of loan applications and compare it side by side with the actual transformed finish rate. The difference between these two quantities is called the residual. These values are brought together in Table 4. Before I discuss how to apply the prediction equation, it’s important to comment on how well the equation fits.
One way to assess the fit is to graph the actual transformed rate against the predicted transformed rate. This graph can be found in Figure 1. The actual and predicted curves track fairly closely and suggest the equation fits reasonably well. An additional way to assess the fit is to graph the residuals against the hours taken to complete each group of loans. This graph can be found in Figure 2 (p. 68).
The ideal appearance of a graph like Figure 2 is random scatter. Here we see some evidence of a cyclic pattern. The pattern is not sufficient to reject the equation completely, but it does suggest there should be efforts to refine the equation—perhaps by seeking and including better predictor variables. Iterative improvement is the name of the game here.6
The equation in its present form suggests a couple of things:
- There is evidence of a real relationship between hours to complete a loan and the finish rate.
- The coefficient for a1 is negative. Since a1 is the coefficient for the hours taken to complete the loan, we have evidence of an inverse relationship between hours and the finish rate. More hours to completion is associated with a lower finish rate.
To summarize the relationship between loan completion hours and the finish rate in the most compact form, we take the equation in terms of the transformed finish rate,
y = 2.666149 – (0.469446 * x1),
and rewrite it in the logistic regression form,
p(x) = ey/(1 + ey),
or in the terms of our example,
finish rate = ey/(1 + ey),
finish rate = e[2.666149 - (0.469446*hours to complete)] / (1 + e[2.666149 - (0.469446*hours to complete)]).
Table 5 shows the hours to completion and the finish rates we would expect from the logistic regression equation. This can be used for input in setting goals for the loan completion process. For instance, if the desired finish rate is 75% or better, the results of the logistic regression analysis suggest three hours is a good goal for loan completion time.
This observation should trigger discussion in three other areas:
- What constraints limit the ability to complete applications in three hours?
- What other factors besides the completion time affect the finish rate?
- Should a goal for completion time wait until these other factors are identified and measured?
With the tools to test the effectiveness of controls under uncertain conditions, the Six Sigma practitioner can better inform an organization about where to take its key products and processes.
REFERENCES AND NOTES
- Sequential sampling is a sampling procedure in which individual units or lots are tested one at a time and the cumulative number of defects is tallied until either an accept/reject decision is reached or the largest allowable amount of testing has occurred. Unlike with more traditional sampling schemes, the total number of units tested is not fixed in advance. For more information about sequential sampling, see How to Use Sequential Statistical Methods (Thomas McWilliams, ASQ Quality Press, 1989).
- It is possible to assess effectiveness when the defects do not occur independently of one another. The strategy requires some theory about how the occurrence of one defect affects the occurrence of the next one. The assistance of a technical expert should be considered in these cases. Some advanced knowledge of probability theory may be required.
- The decisions on risk levels in Six Sigma investigations should balance the costs of being wrong with the benefits of successfully improving the system. This is most successfully achieved by consensus and collaboration between customers, process owners, system experts and resources knowledgeable in statistical methods. Risk levels of 10% or less are common in process improvement efforts.
- Here, “e” stands for “exponential” and refers to the base of natural logarithms, approximately equal to 2.718282. The Microsoft Excel function EXP implements the exponential function, for example, raising e to some desired power.
- For more information on linear regression techniques, see Applied Regression Analysis, third edition (Norman R. Draper and Harry Smith, John Wiley, 1998).
- Reference 5, Applied Regression Analysis, offers a more in-depth treatment of equation assessment and residual analysis.
JOSEPH D. CONKLIN is a mathematical statistician at the U.S. Department of Energy in Washington, DC. He earned a master’s degree in statistics from Virginia Tech and is a senior member of ASQ. Conklin is also an ASQ certified quality manager, quality engineer, quality auditor and reliability engineer.