Creating a better assessor for a better aerospace audit
by Dale K. Gordon
Ten years ago, I wrote an article on the certification and registration process, and the inherent need to do your homework in the selection of a certification and registration body (CRB) for ISO 9001 and other derivative certifications. This included the control or vetting of auditors so the organization receives viable and valuable work.
In that column, I wrote: "Several organizations that have supplemented ISO 9001 requirements (aerospace, automotive, telecommunications) have recognized that the value is not only in the standard, but also in its consistency of application. To have a low amount of variability in the audit of the standard, the audit variables must be controlled. To this end, many of the aforementioned organizations have mandated auditor knowledge, training and the process by which the audit is performed. Some have gone so far as to put strict controls on the auditor training, others on auditor qualifications and experience."1
The International Aerospace Quality Group (IAQG)—almost from its inception in 1998—was chartered to harmonize and institutionalize the process of achieving a single certification recognized by all members across the global aviation, space and defense industries based on common standards.
Like any good organization, IAQG is bound to continuously improve the process. And with the latest revision of the common standards in early 2009,2 it was also time to do a major update of the auditor authentication process.
Reasons for change
A large part of what necessitated this change was not only the revision of the standards, but also a complete revision of the assessment process and philosophy. Prior to this latest revision, the common method of assessment to the standards was a mandated checklist developed for use with each specific standard.3
The checklist approach, while effective for ensuring the totality of the standard was audited, appeared to limit the audit process by making it seem as if getting a check in the box was all that was needed to complete the assessment. It also did little to foster the concept that the effectiveness of the quality management system (QMS) should also be determined.
As a result, a new assessment process was developed along with the standards revision to be consistent with the International Organization for Standardization’s concept of process-based auditing and the revised standard. IAQG 9101D now covers all three IAQG aviation, space and defense QMS standards.
The checklist has been completely revised, significant information has been added in line with a process-based auditing method, and the audit itself has been divided into several stages to coincide with ISO 17021.4 The objective of the audit process is to evaluate whether:
- The processes within the QMS are identified and appropriately defined.
- Responsibilities are assigned.
- Procedures are implemented and maintained.
- The processes within the QMS are effective in achieving the desired results.
The audit stages and activities within the standard are depicted in Table 1.5
With these changes being implemented for more than 12,000 aerospace registrations and more than 1,200 auditors worldwide, IAQG needed to address the issue of auditor training.
As my 2001 column stated, "A good audit depends on one very important ingredient … a well-trained and competent auditor."6 IAQG said no certifications to the latest revisions were to be issued by any CRB until such time as the auditors were properly trained and re-authenticated based on the myriad changes to the assessment process.
IAQG always had requirements for the audit performance and auditor training. They were harmonized and codified into a series of standards to complement the assessment process.7
These standards covered how the audit was to be conducted, the oversight of the audit by IAQG, and the qualification and training requirements for auditors. With the latest revision of the standards, these aspects of the audit also have been significantly improved to coincide with the new assessment methods.
Because of the scope of the changes, the need for consistency in the assessment process and to ensure proper application, IAQG had several organizations submit proposals for training to the new auditing method and standards. The training would then be sourced to a single provider to develop the training materials, administer the training and evaluate auditor competency related to knowledge of the process and standards.
To reduce auditor variation, it was considered vital to control the material, the process and the method of training for existing and new auditors.
After a diligent selection process by select representatives of IAQG, Plexus International was chosen as the training provider. Plexus—having significant history with QS-9000 and similar standards—provided a level of confidence regarding the global management requirements of the training process. But IAQG remains intimately involved with decisions on pricing, contents of the training materials, trainer performance, examination materials, testing results and participant feedback.
Currently, the training is focused on updating existing auditors to the new requirements, which is why it is called Aerospace Auditor Transition Training (AATT). AATT also has a "train the trainer" component that allows member organizations to produce competent trainers.
AATT has three basic components to the training:
- Web-based online pretest and preparation course training (8-16 hours).
- Face-to-face classroom training (28-40 hours).
- Assessment and written exam.
AATT replaces the previous aerospace foundation course, which was offered by several different providers and mandated a 40-hour course on the aerospace industry.
Meet the requirements
For each of the components there are several requirements:
Online. There are three online modules—one for each of the aerospace QMS (AQMS) standards. Auditors must complete the online modules for the applicable standard prior to attending an instructor-led training.
In addition, auditors seeking authentication to the 9110 and 9120 standards must complete the 9100 online module and instructor-led training prior to attending training for the 9110 and 9120 standards.
Auditors who successfully pass the initial exam can choose whether to complete the online training for the applicable standard or go to the classroom component. Auditors who do not successfully pass the initial exam are required to complete the online training for the applicable standard.
Instructor-led training. This is required for auditors seeking 9100 and 9110 authentication. Training for 9120 does not include an instructor-led component. In addition, 9100 instructor-led training is a prerequisite for 9110 and 9120 training.
Train the trainer. Successful completion of the IAQG-sanctioned AATT course and the Plexus evaluator course is required for all trainers delivering instructor-led AATT for authentication of aerospace auditors.
The Plexus course is applicable for delivery of the 9100 and 9110 instructor-led training course. Training providers are required to use the trainer-candidate feedback report provided to them from the Plexus evaluator course as part of the training provider’s trainer-approval process.
The process—as rolled out in May 2010—states that:
- All currently authenticated aerospace auditors are required to complete the AATT to be authenticated for auditing of the 2009 version of the AQMS standards.
- AATT replaces the 9104/3-defined foundation course for authentication as an industry-controlled other-party auditor for the 2009 version of the AQMS standards.
- AATT is required to be included, in its entirety, in 9104/3-defined standard auditor courses. This means all individuals seeking authentication as aerospace auditors for the first time must complete this course.
- All training providers, including certification bodies, seeking to deliver the AATT for the purpose of training aerospace auditors shall be approved.
- All trainers conducting AATT shall successfully complete the course and the Plexus evaluator course.
- Two approved trainers are required to deliver AATT.
The understanding or knowledge of the standards is self-contained in the online training and examination. A participant in the classroom phase of the training focuses on the proper auditing methods and use of the standards to perform the audit.
Worth the effort
After almost a year of implementation, the results are impressive. Of the more than 1,600 attendees in the worldwide classroom courses, the pass rate is more than 70%. Only slightly less than 2%—35 of 1,622 participants as of this writing—taking the initial knowledge exam were able to opt out of the additional knowledge-based training.
An analysis of participants prior to and after the test revealed the knowledge level of the AQMS standards increased almost 30%. Considering that the changes in the standards were not considered highly significant, the pass rate shows the need for the training, and the knowledge-gain data show the effectiveness of the training.
The bottom line is that while there was a process and method to ensure a consistent and accurate assessment of the AQMS standards, without the constant reevaluation and examination of variations in the process, improvements are hard to come by.
The auditing process mandates a need for consistency and accuracy, and the process owner—IAQG—responded with improvements throughout. The standards in the automotive sector already have this same level of consistency for auditor training. Maybe we need it for all certifications to ensure we are getting the level of service for which we pay.
References and notes
- Dale K. Gordon, "Caveat Emptor," Quality Progress, August 2001, pp. 80-82.
- These standards include 9100—Quality management systems—Requirements for aviation, space and defense organizations; 9110—Quality management systems—Requirements for aviation and maintenance organizations; and 9120—quality management systems—Requirements for aviation, space and defense distributors.
- These standards include 9101—Quality management systems assessment; 9111—Quality management system assessment for maintenance organizations; and 9121—Quality system assessment applicable to stockist distributors.
- International Organization for Standardization, ISO/IEC 17021: 2006—Conformity assessment—Requirements for bodies providing audit and certification of management systems.
- SAE International, AS9101—Quality management systems—Requirements for aviation, space and defense organizations.
- Gordon, "Caveat Emptor," Quality Progress, see reference 1.
- These standards include 9104—Requirements for aerospace QMS certification and registrations programs; 9104/2—Requirements for oversight of aerospace QMS registration and certifications programs; and 9104/3—Requirements for aerospace auditor authentication.
Dale K. Gordon is group director of quality and compliance for Woodward Airframe Systems in Skokie, IL. He is an ASQ fellow, past chair of the Americas Aerospace Quality Group and one of the writers of the AS9100 aerospace series of standards. Gordon earned a bachelor’s degree in industrial engineering from General Motors Institute (now Kettering University) in Flint, MI, and an MBA from Butler University in Indianapolis.