ASQ's Certifications Should Be Accredited
I read the certification article in the April 2000 Quality Progress (John W. Moran and Patricia C. La Londe, "ASQ Certification Program Gains Wider Recognition," p. 29) with interest. Congratulations to the ALARIS authors for providing a comprehensive discussion. The article confirmed the high esteem ASQ certifications hold.
I was surprised that, unlike several other nationally and internationally recognized technical certifications, none of ASQ's certifications are accredited by an outside, recognized accrediting body for engineering or technical positions, such as the scientific specialty boards or the National Organization for Competency Assurance.
Accreditation for ASQ's certifications shouldn't be difficult to receive, and it assures that they meet national standards. After all, shouldn't ASQ practice quality assurance?
Article Doesn't Explain Organizational Benefits
I was disappointed in the lead article in the April 2000 Quality Progress (John W. Moran and Patricia C. La Londe, "ASQ Certification Program Gains Wider Recognition," p. 29). One of the difficulties in convincing top management to endorse certification programs such as these is to explain and quantify the benefits to the organization. The article did little to substantiate the bold headline on the cover of the issue.
Has the profitability of ALARIS (or the other 125 committed organizations) improved since mandating ASQ certification in their quality departments? Regarding examinees, has the number of exam applicants increased over time or reached a plateau? Has the number of job listings on the Web requiring certification increased on a percentage basis over the last three years?
Good managers will ask hard questions, and effective quality professionals need more than soft answers.
FRANK SLOUP, CQE, CQA
Sioux Center, IA
Certification Grading Criteria Are Subjective
I couldn't believe the article "ASQ Uses Cut Score Process" on page 36 of the April 2000 issue of Quality Progress (John W. Moran and Patricia C. La Londe). I found it confusing, subjective and totally embarrassing for ASQ members in the explanation of the way the ASQ certification examinations are graded.
One of the complaints I've heard numerous times during my 34 years as a quality professional is about the subjectivity and lack of standardized grading criteria in the quality organization. It's a lifelong struggle to get some respect in the quality profession and to be viewed as productive consultants and engineers rather than subjective critics. I see ASQ's article (and practice) about certification grading as undefined and subjective. How will our society ever earn any respect, and how can it defend any complaints brought against it with the nonsense I read in the article? Basically what the article reports is that a great deal of time was spent in developing opinionated approaches supported by debates, arguments and historical practice.
As one of the supporters of ISO 9000 and QS-9000, I call on ASQ for corrective action and ask ASQ what it's going to do to set up objective grading criteria? What is wrong with statistical process control formulae, design of experiments knowledge, questions related to the Baldridge Award and ISO 9000 guidelines, and other such black and white questions? ASQ can set up required bodies of knowledge, related training courses and associated certification tests. I see no need for "difficulty comparisons, cut points, judging differences of opinion by panel discussion, nor mean comparisons." Were the mean comparisons done statistically? What justification was used for throwing out scores on the high or low side of the mean? What is a borderline candidate? The article says that the number of answers a person has to get right depends on how hard the test is that the person is taking "in order to meet the standard." Which standard is it that is hard and you don't need to do well on?
The letter writers raise some good questions, and in some cases their comments suggest the need for further clarification.
In terms of accreditation for ASQ's certification programs, this seems to be a worthwhile idea and it is something that the ASQ Certification Board will be exploring.
We agree with Sloup about the importance of quantifying the benefits of ASQ certification, and we believe the article highlighted some of the ways of doing that. ASQ will continue to work to develop stronger measures in this area. But there are some limitations that we will encounter.
A key factor is that the practice of quality is changing, and ASQ's Certification Board is working appropriately to improve the program on a continuous basis. That means that some of the measures Sloup suggests would not necessarily yield the kind of evidence he is looking for. For example, as new certifications are developed we would expect the number of applicants to rise, but that in itself would not prove that certifications are more valuable.
In the case of ALARIS Medical Systems, the company is confident that it has seen improvement in the credibility and effectivness of its quality organization since requiring ASQ certifications. ALARIS believes that measurements in quality improvement can be used as indicators that we are making improvements.
The other issue raised has to do with the grading process for certification. Perito apparently perceives it as a means of allowing opinion and experiential bias to form the basis for establishing standards. Nothing could be further from the truth.
The grading process he refers to actually consists of several steps in the life of a certification exam:
1. Establishing a pass/fail cut point for the first time use of an exam (through a cut score study).
2. Ensuring that subsequent versions of that exam are comparable in every way before they are administered (through review by subject matter experts).
3. Ensuring that any measurable differences in difficulty between versions of the test are eliminated (through common-item equating and scaling), so that no candidates are adversely affected--or unfairly advantaged--simply because of the test version they took.
The purpose of a cut score study is to answer the basic question inherent to any pass/fail examination: "How many questions should a person have to answer correctly in order to be considered qualified?" If Perito's call for objective grading suggests that "100%" is the answer to that question, such an expectation would not be appropriate for any bona fide certification program, nor would it be in keeping with the "peer recognition" foundation on which the ASQ certification programs are based.
Moreover, once the cut score study is completed, a variety of statistical analyses are conducted on all aspects of the data, and the Certification Board considers those analyses when reviewing the recommended cut for each exam.
The questions used in the ASQ exams are developed in as straightforward and unambiguous a manner possible, as described elsewhere in the Quality Progress article. Many of the programs, though not all, include coverage on such topics as statistical process control (SPC) and design of experiments (DOE), as well as ISO standards and Baldrige criteria for performance excellence. These and other topics are included in the body of knowledge according to whether such topics show high importance ratings in the job analysis that is conducted for each of the ASQ exams. We do not test topics based on how easy they are to write questions for or because they offer black-and-white choices for answers.
Finally, the methodologies used in the cut score study process, the methods employed by the committees of volunteer subject matter experts, and the statistical analyses for scaling and equating conducted after each test administration all meet the guidelines described in the Standards for Educational and Psychological Testing, which is jointly published by the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education.
This publication outlines the accepted practices for all kinds of testing: licensure, certification, education and so on.
JOHN W. MORAN
PATRICIA C. LA LONDE
San Diego, CA
Is QS-9000 More Cost Effective Than ISO 9000?
In R. Dan Reid's column, "Why QS-9000 Was Developed and What's in Its Future" (April 2000, p. 115), it is obvious that he is a big fan of QS-9000, as well he should be. My only concern is his inflated cost advantages. His quote that the average implementation cost of QS-9000 is $120,000 while that of ISO 9000 is $156,000 does not equate.
QS-9000 is much more comprehensive than ISO 9000, with approximately 100 more "shalls," sector specific requirements and customer specific requirements. Additionally, roughly 50% of ISO 9000 registered companies are ISO 9002 certified. ISO 9002 does not have the design review element--perhaps the single most costly element to implement. Yet QS-9000 is cheaper to implement?
I can only think of two plausible explanations for the gross disparity. One is that most of the QS-9000 companies were previously ISO 9000 registered companies. Most of their costs were, therefore, already incurred. The second is the fact that all but $20,000 of the $120,000 was paid to the certification body. This is a strong indication that the survey respondents did not capture the true cost of implementation. I do not work for NASA, but would someone please explain to me why it is less costly to implement QS-9000?
STEVE BEUMER, CQE, CQA
Please note that the cost/benefit figures were quotes from independent surveys, not my own figures. The sources of the data were footnoted.
Still, Beumer suggests that the difference in costs reported was a "gross disparity." If you consider the size of the companies involved in terms of sales, this may not be a significant difference after all.
I think that the lower quoted cost for QS-9000 is due to the fact that domestic automotive suppliers have worked to individual and rigorous customer quality standards for many years prior to the adoption of QS-9000. By rights, automotive suppliers should have had a good quality system that included operating procedures, measurement systems and training programs prior to QS-9000 under the previous requirements manuals. Hence, the cost to implement QS-9000 should only have included the registration cost and any costs necessary to address any ISO 9000 base requirements. This is in contrast to nonautomotive companies or companies lower in the supply chain that may not have implemented as rigorous a quality system prior to their ISO 9000 certification.
Further, Beumer has the figures confused with the survey. The cost to the certification body was $20,000 or about 15% of the total reported cost. The rest were internal costs reported by the survey respondents.
I never quite understood the counting of the "shalls." The additional "shalls" in QS-9000, once satisfied, will in many cases satisfy the more general "shall" in the same clause, so they are not incremental. If you comply with the more rigorous requirement, you satisfy the more general requirement. This is the case for suppliers with multiple customers. If they implement a system to satisfy their most demanding customer, their other customers are generally satisfied by the same provision. What is the point of counting the "shalls"?
R. DAN REID
Don't Publish Letters Containing Political Views
Everyone wants to take pride in his or her profession and enjoy discussions with professional peers. For this to continue, it is important that these discussions take place on a professional level and not sink to name calling, mudslinging or partisan politics.
A.V. Feigenbaum and Donald S. Feigenbaum did not include any partisan politics in their article "New Quality for the 21st Century" (December 1999, p. 27), so it was very disappointing to see the letter in the April 2000 issue (Donald E. Klasing, "When Was There a Severe U.S. Economic Downturn?" p. 8). The letter had no actual value to the quality profession, just partisan politics.
Publishing Klasing's letter seems to be a continuation of a trend started a few years ago with the publication of an article by that well-known quality professional Newt Gingrich. Aside from ASQ, I belong to the American Statistical Association (ASA), and I can't remember a single letter or article in that association's publications that I thought contained partisan politics. None of its articles or letters stoop to an unprofessional level.
Perhaps Quality Progress' editorial board should contact ASA's editorial board to ask about its professional ethics and review policies. Or, may I suggest a simpler policy? If a letter appears to contain partisan politics, throw it away regardless of its value or insight to quality.
An important part of this magazine's mission statement is to encourage discussion about quality and performance excellence. For this reason, we exercise a light hand in determining what comments we will publish from readers.
We won't publish statements that are libelous or patently untrue. But otherwise we feel that members have a right to be heard. We also feel that to start setting arbitrary definitions of what is publishable or not publishable would serve to discourage reader input.
Our customer research shows that readers like the point/counterpoint aspect of our letters to the editor column and think that it is one of the best parts of the magazine.
In this particular case, I think Dr. Feigenbaum deftly handled the letter writer's complaint and made it clear that there was no partisan motivation to his article. Rather than detracting from the professional standards of the magazine, I think this approach served to emphasize that we do have very high standards and that our standards will stand up under all kinds of scrutiny.
All About System Auditing
In the column "All About Auditing" (J.P. Russell, May 2000, p. 96), the author raised a very interesting and insightful point pertaining to the similar approach the ISO 9001:2000 revisions and the current ISO 14001:1996 requirements take. Yet I must respectfully question the author's assertion that " ... the focus of an environmental system is to achieve compliance to statutory and regulatory requirements."
I have been involved in environmental management and regulatory affairs in the industry and am now an ISO 14001 environmental management system (EMS) auditor. I understand and accept the fact that there are organizations whose sole purpose in pursuing ISO 14001 registration is to fulfill certain customer or regulatory obligations. By and large, however, most forward thinking companies and management teams decide to implement an EMS to authenticate their genuine commitment to environmental protection, fulfill their social obligations and/or realize certain business profitability.
In short, the fundamental motivation and focus for implementing an EMS in many organizations are no longer bound by mere compliance to statutory and regulatory requirements, but are to reap benefit from a sustainable business management model. In theory, if a well-designed EMS is effectively implemented, any potential environmental excursions (noncompliance) would have been detected and corrected in situ, and thus the EMS would assure continual compliance to the statutory requirements and yield a more efficient and effective operation.
By design, a regulatory/compliance audit is only to look at the end result of a process. On the contrary, the primary objective of a third-party EMS audit is to focus on verification of the fact that an EMS is effectively implemented in an organization. Oftentimes it may be necessary for an EMS auditor to confirm effective implementation of the system. It is not the third-party auditor's job to determine if an environmental excursion occurred in the operation.
A system audit should not be confused with a regulatory compliance audit. No value assessment could be delivered in an audit if an EMS auditor considered the focus of an environmental system to be a means of achieving compliance to statutory and regulatory requirements. Consequently, the subscribing organization would not have any additional incentive to implement a management system over its environmental activities and operations.
It is especially important that an EMS auditor not only have thorough knowledge of ISO system requirements, but also have the appreciation of both a management system audit and a compliance/regulatory audit and of the difference between them. Then a complete, value-added assessment can be delivered to the client.
An incorrect Web address was given for Russell & Associates in the May 2000 issue. The correct address is www.JP-Russell.com. Russell, a quality management consultant, writes about auditing as one of the regular contributors to QP's "Standards Outlook" column.
In December 1999's "World View" column (Mahesh Chandra and Vasanth Adur, "India's Quality Movement," p. 88), five Indian companies were named as having met the Software Engineering Institute's Capability Maturity Model level 4 standard. We have learned that NIIT Ltd. achieved the rating after the article was written, but prior to publication.