Measuring Training's Effectiveness
Pre- and post-test scores help organizations monitor training initiatives
by Robert D. Zaciewski
Training plays an important role in keeping organizations competitive in the global marketplace. While most organizations support the use of training, many pay little attention to whether the training was effective.
Recently, quality system requirements have become a catalyst that directs organizations to measure the effectiveness of training. For example, QS-9000 states in element 4.18.1: "Training effectiveness shall be periodically reviewed." ISO 9001:2000 states in element 6.2.2: "Evaluate the effectiveness of the actions taken." Training effectiveness is also mentioned in element 4.18.2 of ISO/ Technical Specification (TS) 16949, the international version of QS-9000 for automotive suppliers.
In light of these changes, one organization I worked with conducted a statistical comparison of pretest and post-test scores. This comparison gave us a measurement indicating how effectively we conducted our internal quality auditing training.
The training course
The course consisted of two days of interactive instruction based on ISO 10011, the guidelines for auditing quality systems. A training team with two instructors facilitated the course for two days, and on the third day, the participants performed a practice internal quality audit using the organization's procedures/instructions.
Before the first day, we administered a 50-question multiple-choice test based on our course objectives. Participants were told not to guess if they didn't know the answers. This minimized the possibility of a participant's guessing a correct answer. At the end of the classroom instruction on the second day, participants were given the same test.
The co-instructor graded the pretest the first day. The test scores were plotted on a graph with a frequency distribution, and descriptive statistics were generated. The distribution was shown to the participants at the end of the day, and the instructor described the central tendency and range of the distribution of pretest scores. The class's goal was to reduce variation and move the distribution's central tendency closer to the nominal/target (50 correct answers).
The post-test was graded in class by the participants, and their scores were plotted on the graph. This allowed the participants to see how their scores improved, with corresponding improvements in the distribution's variation and closeness to the nominal value.
This analysis was later modified into a formal spreadsheet and corresponding graph (see Figure 1). The instructor team sent the spreadsheet and graph to the organization that asked us to administer the training course. This gave the organization a quality record that verified how it measured the effectiveness of its internal auditor training.
The answers to two important questions can determine if the training was effective:
1. Is the variation in the distribution of post-test scores less than that of the pretest scores? A decrease in range/standard deviation shows the training was effective.
2. Was the central tendency of the distribution of post-test scores closer to the target value (50 correct answers) than that of the pretest scores? A central tendency closer to the target value also shows the training was effective.
This method can be used as objective evidence during an audit of the quality management system. ISO/TS 16949 notes in element 4.18.2: "Training effectiveness may be practically reviewed by various methods, such as pre- and post-testing." QS-9000 states the same in element 4.18.1.
This pre- and post-test analysis was used by instructor teams for the continuous improvement of the internal auditing instructional process. Using statistical techniques in nontraditional applications will help these methods become more accepted in an organization.
ROBERT ZACIEWSKI is an independent quality consultant in Monroe, MI. He has a bachelor's degree in management and organizational development from Spring Arbor College in Spring Arbor, MI. Zaciewski is an ASQ certified quality engineer, certified quality auditor, certified quality technician and certified Quality Inspector. He is also an RAB certified quality management systems provisional auditor and an RAB certified quality management systems internal auditor.