QUALITY IN THE FIRST PERSON
Tell Me What I Don’t Know
Making internal audits more meaningful by applying PDSA
by Larry M. Pope
A few years ago, I took over a compliance department with a management mandate to: "Tell me what I don’t know." The compliance department auditors had a reputation for producing audit reports that included a long list of nonconformances, most of which were superficial. Lack of differentiation between nonconformity and improvement suggestions can cause resentment among auditees.1
For example, auditors would list individual instances of documentation errors instead of identifying the bigger problem—a lack of understanding or inattention to detail in documenting process-related work. This resulted in the department fixing individual issues, rather than addressing the underlying source of the problem. I received direct feedback from department managers that there were diminishing returns on the amount of effort required to identify and correct the nonconformances.
I was faced with a dilemma: How can we make internal audits value added for the auditee while maintaining auditor confidence? The answer: Apply W. Edwards Deming’s modified Shewhart cycle, or plan-do-study-act (PDSA) to the problem.2
Step one: Plan
The expected output was already established by the senior management’s mandate: Tell me what I don’t know. This meant that I had to establish processes that would align the current auditors with the expected outcome. Because I was new to the department, I set up a training session with the auditors ostensibly to calibrate internal auditing to my preference—namely, finding problem areas not previously identified. I started with defining the purpose of an internal audit by providing management with the assurance that "procedures are adequate and utilized, and to provide for the early detection of a problem, which gives management the opportunity to identify root causes and take corrective action."3
With purpose established, I explained the importance of audit planning. The auditors were shown that areas of focus revealed themselves during a review of past audits, deviations and change controls. For example, past audits may not have covered engineering controls and drawings, thus highlighting an area of focus. I explained that some regulators actually take mechanical drawings and walk the facility to confirm that the drawings are up-to-date. Because the organization was regulated by the U.S. Food and Drug Administration (FDA), planning needed to also include a review of recent trends or focus areas. Making these areas and trends the focus of the audit would help find those issues which may not be readily apparent.
The final element of this step focused on the actual writing of nonconformances. This sounds simple; however, many auditors confuse nonconformances with areas for improvement. Auditors were trained that only issues in direct violation of a standard, regulation or organization procedure should be written as nonconformances, while any other observation was an area for improvement. To establish a nonconformance, auditors need to:
- Show a sincere interest in understanding the process and procedure before making any judgments. I explained that department managers’ impressions were that sometimes auditors had made up their minds on an observation before even coming to the department and gathering facts.
- Give the auditee an opportunity to review and explain the situation. I described events in which the FDA allows me a chance to explain and present evidence before making a decision on an observation.
- Taking careful notes (who, what, when, where and why). I related a perfect example from an audit report I received once. In the report, the auditor used vague references, and I could not find the objective evidence to support the observation. I was able to refute the observation, and the auditor retracted the observation because the notes were insufficient.
- Obtain concurrence (if possible) that the issue is a nonconformance. There is nothing worse than blindsiding an auditee with a nonconformance—it destroys any rapport that has been built with the auditee. Remember, I will likely have to audit this organization or department again someday.
Step two: Do
After training was complete, I worked directly with each auditor to plan their next audit. Audit plans were presented at departmental meetings so that the plan could be critiqued and suggestions for improvement provided. The audit plan would be provided to the department scheduled for an audit. During the execution of the audit, daily summaries were given to the department’s management and me. The purpose of these summaries was to:
- Provide feedback on potential non-conformances.
- Allow management to provide additional detail to mitigate the non-conformances.
- Maintain audit focus and avoid getting mired in minutia.
- Allow for the adjustment of the scope should a significant or serious issue be uncovered.
To improve the confidence of the auditors, I reached out to select departments and offered an extra set of eyes to examine the operations of their departments. The examinations would be used as a training exercise for the auditors, and no audit report would be written. During this phase, I think it is essential that the auditor is accompanied by a seasoned auditor. This seasoned auditor could provide immediate feedback and correction if the auditor missed something or stopped too soon.
These training exercises were a big success. The auditors gained experience and coaching prior to an actual audit situation. The departments became more confident that they were ready for an audit. Word of mouth spread that the extra set of eyes was beneficial, and it wasn’t long before other departments volunteered to be examined next. The organization had an abundance of trained auditors and departments who were confident that there were no hidden audit landmines.
Step three: Study
This phase starts with obtaining feedback from the audited department’s management. The simplest way to find out what people want from your service or product is to ask them. That’s what the hairdresser does when he asks, "How would you like your hair, sir?"
So meetings were set up with the audited departments to ask, "Was the audit of value? Did you learn something you didn’t already know?" Surprisingly, the responses were positive.
For example, in one interview, the department manager revealed that he learned that he needed to provide additional training for his employees on documentation practices. This feedback revealed that my efforts to revise the audit’s focus had begun to turn the perception of the audits from a nuisance to a necessity.
Step four: Act
PDSA is designed to be an iterative ongoing process. In this case, the first round was a success; however, an assessment was conducted to determine whether changes were needed prior to doing the next round of internal audits. My evaluation of the audit program determined that the audits were uncovering hidden issues. The organization was also about to introduce a new biological product to the facility, which would necessitate developing new auditing skills for the auditors—bringing us back to step one.
In conclusion, this process turned around the facility’s perception of the internal audit program and it began to add value. The mini-audits allowed the departments and my auditors to gain experience in the audit process. The facility underwent several regulatory inspections after this process was implemented. I am happy to report that I never again heard, "Why didn’t we know about this?"
- J. M. Askey and B.G. Dale, "Internal Quality Management Auditing: An Examination," Managerial Auditing Journal, Vol. 9, 1994, pp. 3-9.
- G. Langley, R. Moen, K. Nolen, T. Nolen, C. Norman and L. Provost, The Improvement Guide, second edition, Jossey-Bass, 2009, p. 24.
- J. P. Russell, The ASQ Auditing Handbook, third edition, ASQ Quality Press, 2005, p. 11.
Larry M. Pope is a senior quality manager for a pharmaceutical company in the San Francisco Bay area. He holds a bachelor’s degree in chemical engineering from the University of Tennessee in Knoxville and an MBA from the University of Phoenix. An ASQ senior member, he is an ASQ-certified quality auditor.