Checking the Checkers
Reviewing and improving audit programs using ISO 19011 guidance
by J. P. Russell
ISO 19011:2011—Guidelines for auditing management systems provides guidance for managing audit programs. Clause 5.6 of the standard provides guidance on improving an audit program, just as other departments in an organization are expected to continue to improve.
The standard begins by stating that the "person responsible for managing the audit program [should] review the audit program to assess whether its objectives have been achieved."1 Clause 5.2—Establishing the audit program objectives provides a list of objectives the audit program can be based on, such as management requirements, characteristics of processes and supplier evaluations.
The audit program objectives should be aligned with the management system policies and objectives, and should consider the needs of customers and other interested parties.2 If the organization, for example, has objectives to improve safety, reduce working capital expenditures, and increase capacity and efficiency, the audit program objectives should reflect and support the same overall organization, division or department objectives.
As long as the objectives are measurable, you can determine whether they are met or achieved. The effectiveness3 of the program is determined by the extent to which planned activities are realized and planned results are achieved. There may be objectives related to the accomplishment of projects and others related to results or outputs. An important factor in determining whether an audit program is effective is the extent to which audit program objectives have been achieved.
The standard does not specify the frequency of the audit program review. Typically, programs are reviewed annually or semiannually, but other factors such as changes, complexity or risk could make it prudent to review the audit program more frequently. Perhaps review of progress of accomplishing audit program objectives should be at the same frequency as the organization progress reviews. It also should be clear that the person responsible for managing the audit program should be the person who conducts the review.
Next, the standard states that "lessons learned from the audit program review [should be used as] inputs for the audit program continual improvement process."4 That statement implies that one of the outputs of the audit program review should be lessons learned. The results of the review should include a record of lessons learned or however the organization wants to refer to them. Learnings could include things to do and not do, and would require making changes to the existing system if there was need for improvement.
Later in the standard, a section regarding completing the individual audit points out that lessons learned from the audit should be entered into the continual improvement program of the auditee organization. This is particularly important for internal audit programs. For individual audits (internal or external), opportunities for improvement might be considered lessons learned.
Next, the standard switches to a list of areas reviewers should consider as part of the review, which include:
1. Results and trends. Audit program management should monitor results of objective measurements and progress toward goals. Normally, there would be some type of analysis to identify positive and negative trends. For example, you might have a goal to lower travel costs X% by implementing an e-audit or remote audit program, or there could be a goal to improve consistency and efficiency of the audit team’s performance.
2. Conformity with audit program procedures. It makes sense that the audit program’s management should verify conformity to its policies, procedures and requirements because that is a core competency. This might be called "walk the talk."
3. Evolving needs and expectations of interested parties. This should include feedback from customers, auditors, auditees, clients, regulators and other interested parties. It could include individual audit surveys, feedback from department forums, top management feedback and regulatory agency comments.
I think you should have formal methods to collect comments and feedback that can be reviewed. There are still many organizations, however, that avoid feedback regarding their performance so as not to deal with performance issues, complaints or ineptness.
4. Audit program records. Areas in which information or data are recorded may include: audit reports, corrective actions and status reports, opportunities for improvement, findings, and individual auditor and audit team performance. Records may be needed to demonstrate implementation of projects such as training updates, expanding audit services and effectiveness of the supply chain management program.
5. Effectiveness of the measures to address risks. For this review, risks are limited to those associated with audit program management. This assumes you have identified audit program risks, assessed the risks and taken action to avoid the risk or treated the risk to minimize the probability of undesirable outcomes. As the reviewer, you are being asked to determine whether risk treatments continue to be effective. According to clause 5.3.4, audit program risk may be associated with: inadequate planning, auditor selection and competence, ineffective communications, improper control of records, and the suitability of measures and metrics to monitor performance.
6. Confidentiality and information security. This is limited to audit program issues and concerns. Confidentiality is about controlling access to information to a limited few and protecting information from public access—confidential documents and information must be safeguarded by audit program management at all times. This may involve limiting access, scrutinizing distribution, and deleting or destroying confidential information not needed for legal or verification purposes.
Modern trends to go paperless and provide information electronically may involve other security challenges to protect information and prevent unauthorized access. Before computers, security could be locked rooms or cabinets. With computers, we need internet security software, firewalls, encrypted messages and passwords to protect information.
The standard states that the "person responsible [for managing the audit program should] review the overall implementation of the program."5 The standard uses the word person, not persons, indicating there should be one person in charge of the audit program. There could be managers for different audit services (supply chain, internal audits, product audits, process audits, compliance audits and management audits) that report to one person.
The person responsible for managing the audit program also should "identify areas for improvement"6 based on review results and other inputs. The standard does not require a record, but I think it would be a good idea to include areas for improvement as part of the record of the review.
The next part of the same sentence states the "program should be amended if necessary."7 My thinking is that if there are areas that need improvement, the program will need to be amended, modified or changed. If you don’t change a system or process, you will continue to get the same results, but not all changes result in improvement. Changes must be evaluated and monitored over time.
The environment we work in continues to change. To ensure we are effective, we must stayed tuned to changes and adjust as necessary. The standard requires the person responsible for the audit program to "review the continual professional development of auditors,"8 in accordance with clauses 7.4, 7.5 and 7.6. Clause 7 addresses the competence and evaluation of management system auditors. Clauses 7.4, 7.5 and 7.6 specifically cover the selection of the appropriate auditor evaluation methods, conducting the auditor evaluation, and maintaining and improving auditor competence.
You should have your own methods and procedures for the continued professional development of auditors. It may include additional training on different audit techniques, preparing for certification such as the ASQ quality auditor certification exam or RABQSA certification training or exams.
To close the loop on accountability, the last requirement of this clause is to "report the results of the audit program review"9 to top management. Being accountable for your actions is a good thing. It keeps us aligned with priorities and avoids miscommunications.
Beyond ISO 19011, when reporting to top management, you also could include how the audit program has affected organization risks and how the program is contributing to organization performance. For many organizations, the audit program is the main tool used for oversight of management systems (quality, environmental and safety) and improvement. Management should be interested in how the audit program is performing and how it is benefiting the organization.
It is crucial to review the performance of the audit program. The results of the review should point out strengths and weaknesses. Take action to stay on track and make changes to improve the effectiveness of the program and continue to benefit the organization.
References and notes
- International Organization for Standardization, ISO 19011:2011—Guidelines for auditing management systems.
- Interested party is a person or group having interest in the performance or success of an organization, as stated in ISO 9000:2005—Quality management systems—Fundamentals and vocabulary, clause 3.3.7.
- Effectiveness is the extent to which planned activities are realized and planned results achieved, as defined in ISO 9000:2005—Quality management systems—Fundamentals and vocabulary, clause 3.2.14.
- International Organization for Standardization, ISO 19011:2011—Guidelines for auditing management systems, see reference 1.
- International Organization for Standardization, ISO 19011:2011—Guidelines for auditing management systems, clause 7.
J. P. Russell is the founder and managing director of QualityWBT Center for Education. He also is an ASQ fellow, ASQ-certified quality auditor, voting member of the American National Standards Institute/ASQ Z1 committee and member of the U.S. technical advisory group for International Organization for Standardization technical committee 176. Russell is a recipient of the Paul Gauthier Award from the ASQ Audit Division.