3.4 Per Million

DOE and Six Sigma

by Joseph D. Conklin

Design of experiments (DOE) is a powerful tool for improving processes as part of a Six Sigma program. It works best when we know which quality characteristic must be improved, that there are variables we can change that affect the quality characteristic, we have an agreed on measuring system for the variables and we can devote resources for manipulating the variables in an organized way.

Though DOE is typically presented using continuous data, there are ways to adapt it for use on discrete data. Continuous data are associated with measuring devices, such as thermometers, pressure gages and micrometers, and can be fractional--50.5 degrees, 13.92 kilopascals or 0.015 inches. The oldest DOE techniques were developed for those kinds of data.

But what happens when the quality characteristic we're working with produces discrete data--data associated with counts, such as the number of units in a lot failing a test? One strategy is to use approximate techniques, which take the original data and convert them so they can be analyzed with the more traditional techniques used for continuous data.

Approximate Strategy

To show how the approximate strategy might work, look at the data in Table 1 and assume the following:

  • Your product is a critical unit in an important subassembly for an original equipment manufacturer (OEM).
  • All units from a batch are tested before shipping to the OEM.
  • The critical quality characteristic is the fraction of each batch that passes.
  • Three vendors can sell you the raw material for your product.
  • Three types of raw material can be used to make your product.
  • Each vendor can process each material by one of two formulas--old or new.

The three vendors are Acme, Bigco and CanDo, and the three material types used by each vendor are ceramic, metal and plastic. The two pieces of discrete data available from product testing are the number of items going to test and the number of items passing. The results of product testing are expressed by dividing the number passed by the number tested to obtain the total fraction passed (TFP).

In DOE terms, there's one response variable (the TFP), and three independent variables (vendor, material and formula). Vendor and material exist at three levels, and the formula exists at two levels. Your goal is to find the combination of vendor, material and formula that gives the highest TFP.

A DOE project can help you reach your goal in this situation. You ask each vendor to ship a batch from each of two independent production runs for every combination of formula and material. Three materials x two formulas x two batches equals 12 batches from each vendor. With three vendors, there are 36 total batches to process and test. You process and test the batches in random order.1 After testing, you have the data in Table 1.2

To implement the approximate strategy, take the total fraction passed for each batch and convert it using the arc-sine root transformation. To convert the TFPs, compute the quantity [2 x sin-1(TFP1/2)] for each. Using the arc-sine root transformation is the recommended way to convert data in the form of fractions or percentages,3 because the converted data tend to be better aligned with the assumptions of traditional DOE analysis techniques for continuous variables. Using the arc-sine root transformation leads to the results in Table 2. Let's call the converted values TFP2.

Analysis of Variance

Using TFP2 as the response variable, you can use a statistical analysis package to construct an analysis of variance (ANOVA) table. You are looking for evidence that vendor, material, formula or some combination of the three has a significant impact on the TFP2, and by extension, the TFP (see Table 3).

An independent variable is significant if a change in its value is associated with a real change in the response variable. By "real," I mean a change too large to be considered coincidental. Table 3 (p. 68) has nine rows:

  1. Vendor.
  2. Material.
  3. Formula.
  4. Vendor x material.
  5. Vendor x formula.
  6. Material x formula.
  7. Vendor x material x formula.
  8. Error.
  9. Total.

The first three rows show the impact of the three independent variables on TFP2. Rows four, five, six and seven show the impact of possible interactions between the three independent variables.4 To call something an interaction is another way of stating a basic observation of real life--quite often a result requires two or more things to work together. For example, you need heat and oxygen to make fire, red and yellow to make orange and flour, water and yeast to make bread.

In DOE projects, there are two ways for an independent variable to affect the response variable. The variable can act on its own regardless of what the other variables do, or the impact of the variable can depend on other independent variables. When the impact of one variable depends on what another variable does, the two variables are said to interact.

A corresponding idea in business is reflected in the term "synergy"--the notion that, under the right conditions, two parts accomplish more together than the combined result of both parts acting separately. This is an example of a positive interaction. Negative interactions--cases where the cooperative result is less than the combination of the separate results--are also possible. To refer to an interaction of two or more independent variables, write out the names of the variables and connect them with multiplication signs.

Row eight captures the impact of other influences on TFP2 beyond those represented in the first seven rows. The types of influences that can be captured in row eight include:

  • The random variation of our various measuring processes.
  • Variables deliberately excluded from the experiment because their impact is known to be minor.5
  • Variables excluded from the experiment because you are unaware of them.6

Row nine is the total of the sums of squares in the first eight rows.7

The source column in Table 3 shows the names of the nine rows. The degrees of freedom column depends on the number of levels of the independent DOE variables. The more levels of variables included in the experiment, the higher the degrees of freedom will be. More degrees of freedom are better than fewer because having more supports a more detailed understanding of how the independent variables affect the response variable.

The sum of squares column is a function of how much the response variable varies from lowest to highest. The first eight rows in Table 3 show contributions to the overall sum of squares, and row nine shows the overall sum of squares. The more TFP2 varies, the higher the overall sum of squares will be.

The mean squares column is derived from the sum of squares and degrees of freedom columns. To obtain the mean square for a row, divide the sum of squares by its degrees of freedom. The F ratio column is derived from the mean squares column. Values for the F ratio apply only for rows one to seven in Table 3. To obtain the F ratio for one of these rows, divide the mean square for that row by the mean square for row eight.

The Pr > F column helps you decide whether changes in vendor, material or formula lead to real changes in TFP2.8 If they do, the value in the Pr > F column will tend toward zero. The possible values for Pr > F range from zero to one. There is no universal rule to say how close to zero Pr > F has to be before you conclude the changes in TFP2 are real effects of vendor, material or formula.

In general, the greater the consequences of a wrong decision, the closer to zero Pr > F has to be. Many possible cutoff values exist, but in practice, the cutoff value is almost always less than or equal to 0.10, so we will use 0.10 as the cutoff value for this example.9 If the value for Pr > F is less than 0.10 for a given row, you can conclude the term in that row identifies something that leads to a real change in TFP2.

Three values of Pr > F in Table 3 meet this condition: the rows for vendor, material and the interaction between vendor and material. When you change to a different vendor or material, you can expect real changes in the value of TFP2. Another way of expressing this is to say vendor, material and the interaction between vendor and material are statistically significant.

Rows four, five and six pertain to interactions between two independent variables and are referred to as two-way interactions. Row seven is an example of an interaction involving all three independent variables and is known as a three-way interaction. A three-way interaction is a higher order interaction than a two-way because it involves more variables.

The existence of statistically significant interactions strongly influences how you plan any follow-up DOE projects.10 First, you have to ask if any independent variables from the last DOE can be dropped. If an independent variable is involved in a statistically significant interaction, you should keep it in any follow-up efforts. The standard rule for interpreting an ANOVA table is to concentrate on the interactions that show the most statistical significance. In this example, there is only one to consider--the interaction between vendor and material.

The next question you need to ask is, "What combinations of vendor and material are best for TFP2?" To find out, compute the average TFP2 for each combination of vendor and material, use a multiple comparison procedure, and convert the results back to TFP by reversing the arc-sine root transformation.11 Whichever combinations you decide are best should be used and tested several times to make sure your conclusion is durable.


  1. To keep this example simple, assume vendor, material and formula are the important variables affecting TFP and the 36 batches reflect a statistically adequate sample size. A random order for processing and testing is important for minimizing the impact of unknown variables. Also assume the DOE is conducted along sound planning principles. See Statistics for Experimenters: An Introduction to Design, Data Analysis and Model Building by George Box, William Hunter and J. Stuart Hunter (John Wiley & Sons, 1978) for more information about sound planning.
  2. In real applications, each batch wouldn't contain the same number of units. Keeping the same number in each batch creates a balanced dataset, simplifies the analysis and makes the main points of the example easier to present. Computer programs such as PROC GLM in SAS's statistical package can be used to analyze datasets with different numbers of units in each batch. See SAS/STAT User's Guide, version 6, Vol. 2 (SAS Institute, 1989), for more information on PROC GLM.
  3. B.J. Winer, Statistical Principles in Experimental Design, second edition, McGraw-Hill, 1971, p. 400.
  4. Ibid.
  5. Excluding a variable should not be done lightly, but it is often necessary because of time and resource constraints.
  6. Part of sound planning for DOE projects is collecting suggestions for variables from the people and groups most responsible for or most affected by the process.
  7. This is true in our example because it uses an orthogonal design, which is easier to analyze than a nonorthogonal design. See "Efficient Experimental Design With Marketing Research Applications" by Warren F. Kuhfield (Journal of Marketing Research, November 1994, pp. 545-557) for a discussion of orthogonal and nonorthogonal designs.
  8. The "F" in F ratio and Pr > F refers to the F distribution, an important statistical distribution, particularly in DOE analysis. See An Introduction to Linear Statistical Models by F.A. Graybill (McGraw-Hill, 1961) for more information.
  9. The exact cutoff value should be determined in the planning stage after discussions take place between the people or groups who know the process and the people or groups who are helping to design the experiment.
  10. In practice, process improvement usually takes place after several DOEs, each one building on the things learned in the last one.
  11. B.J. Winer, Statistical Principles in Experimental Design, see reference 3.

JOSEPH D. CONKLIN is a mathematical statistician at the U.S. Department of Energy in Washington, DC. He earned a master's degree in statistics from Virginia Tech and is a Senior Member of ASQ. Conklin is also an ASQ certified quality manager, quality engineer, quality auditor and reliability engineer.

Average Rating


Out of 0 Ratings
Rate this article

Add Comments

View comments
Comments FAQ

Featured advertisers