MEASURE FOR MEASURE
How to ensure your test and calibration supplier meets your needs
by Dilip Shah
My last column covered how to select a test and calibration supplier based on its accredited status.1 But what do you do when a supplier does not have accreditation to the required quality standard? How about when the particular test or calibration item is not covered under the scope of accreditation? What if you need to visit and evaluate the test and calibration supplier to verify customer requirements?
In this column, I’ll look at some of the ways to prepare for the assessment process, which can help evaluate the supplier in an objective manner. The emphasis in the following tips is on assessing key requirements for test and calibration, but the guidance provided may be used for assessing internal test and calibration facilities.
Too many test and calibration assessment audits are not performed well because the objective has not been defined or communicated with the supplier. Sometimes, the person performing the assessment does not have the right qualifications or a good checklist for the organization. To be sure, there are pitfalls at every turn. Hopefully, a few tips will help you avoid them.
Qualifications and preparation
The assessor must have knowledge of the test and calibration parameters and how to audit them. The assessor must be objective and unbiased. The assessor must follow rules of civility and treat the audited party with respect and be professional in his or her conduct while performing the assessment. The ASQ Quality Auditor certification is a good qualification to have for general auditing skills.
Lack of preparation is a major cause of failing to achieve assessment objectives. When preparing for an assessment, you must have knowledge of the standard being assessed or the customer’s requirements.
Some advance work may be required to review the supplier’s documentation, such as knowledge of the quality system, policies, procedures and work instructions. The required documentation should be requested in advance. The customer may need to sign binding nondisclosure or confidentiality agreements to obtain certain documentation from the supplier.
The checklist used should be specific to a standard, plus any other customer requirements. There are a number of reasons to use a checklist, the main one being to maintain the integrity of the audit.
The checklist helps the auditor by identifying and communicating the scope of the audit, providing a structured list of points to evaluate, guiding the course of the audit, keeping the pace of the audit and ensuring all aspects of the audit are assessed. But the checklist is only as good as the auditor’s preparation. An inadequate checklist with no preparation will do little to ensure the adequacy of the audit.
Preparation should also include communicating the assessment objectives with the laboratory so it is prepared to satisfy assessment inquiries. An audit should not be a surprise. A preliminary schedule shared with the supplier laboratory gives the laboratory an opportunity to prepare test and calibration processes for the assessment and have qualified personnel available to demonstrate methods and answer questions.
Performing the assessment
Most test and calibration activities or processes are witnessed by the auditor to determine conformance. When witnessing the process, verify training and competence of personnel conducting tests or calibrations. Training records may need to be examined. Some training records may reside in HR, so make sure you notify the department in advance if you need to retrieve them.
Make sure any environmental conditions critical to test or calibration activities are monitored with traceable equipment. Also, monitor at least the last year’s worth of environmental records for the area to verify the laboratory’s ability to maintain the appropriate environment, including temperature, relative humidity and vibration.
Check that procedures used are current, relevant and maintained under the laboratory’s document control system. Any supplementary documentation, such as manufacturers’ manuals and external standards, should also be examined for relevance. Any software that is used by the laboratory must be sufficiently validated.
Measurement uncertainty must be taken into account when the laboratory reports the measurement results for compliance. Standards such as ANSI/NCSLI Z540.3 also require determining probability of false accept (PFA) risk. If that is required, ensure the method the laboratory uses to determine PFA is acceptable and verified.
Determine who has the authority to reject or accept test results, as well as who can stop and resume workflow processes when a nonconformance occurs.
Make sure equipment used is traceable and calibrated on a predetermined schedule, taking into account frequency of use and other appropriate factors. Equipment calibration history and records of previous calibrations must be maintained. There should be a process to recall equipment for calibration and to quarantine any equipment due for calibration. Sample how the process is used with a few examples to verify effectiveness.
During the assessment, verify measurement (metrological) traceability hierarchy of the equipment and standards used. The traceability hierarchy should be satisfied to the international system of units up to the National Metrology Institute level or to the accredited laboratory that calibrated the equipment.
Any statistical sampling techniques used must be valid and appropriate for the test and calibration activity. If software is used, ensure it is sufficiently validated.
Note how processes are tracked during the test and calibration activity, including handling, conditioning and tracking items while in the laboratory’s possession. Is the laboratory environment suitable for the test and calibration activity? Are storage and staging areas sufficient? Is equipment handled properly and safely? What precautions are observed?
Note any best practices and preventive actions taken to ensure confidence in the lab’s capability to measure and report results. Typical examples of best practices are statistical process control, interlaboratory comparisons, and intermediate checks of standards and equipment used.
Study the sample test or calibration report generated to ensure it includes the right information, is unambiguous and meets customer or standards requirements. The sample report should include the relevant measurement results—including as-found and as-left data if appropriate—and associated measurement uncertainty information for satisfying measurement traceability requirements.
Any raw data used to report measurement results should be retained by the laboratory. If in doubt, examine other test and calibration reports generated by the lab, which should have a record retention system and processes in place. Sample a few examples of previous records.
Reporting the assessment
Assessment results should be reported in a clear, concise and objective manner. The report should state the facts backed by objective evidence.
Include all objective evidence sampled—for example, calibration report No. 1234 dated June 30, 2010, was sampled for equipment XYZ and recall record for equipment ABC to verify equipment recall process. State if processes or items observed were in conformance or why they were not in conformance, and reference the standard or requirement.
Any observations or opinions should be clearly stated as such. Observations and opinions may provide opportunities for improvement or preventive actions for the laboratory being audited.
Many audits place a passing criterion of at least 80% based on the checklist. If all items on the checklist are weighted the same, this may not be a good idea. For example, if everything was satisfactory except for lack of measurement traceability, then the whole system or process should fail. This is where the technical expertise and experience of the assessor is useful.
Passing should also depend on the criticality of the process being audited. It is better to accept the laboratory as the supplier after it has met all the requirements, including corrective actions for nonconformances, rather than just making sure it has a passing grade.
Closure and follow-through
Ensure that during closure, all items pertaining to the assessment are discussed. The laboratory must understand all items in the assessment report, including nonconformances. There should be no surprises at the closing meeting or thereafter.
Both parties should have a clear understanding of audit findings. If there are nonconformances, expectations for timely corrective actions need to be communicated to the laboratory, including any requirements for root cause analysis.
Ensure all corrective actions and the associated objective evidence satisfies the issues raised through the nonconformances cited. If a follow-up assessment is required, follow these same tips.
To develop a comprehensive test and calibration checklist, consult standards such as ISO 17025, ANSI/NCSLI Z540.3 and ISO 10012. That foundation, plus the guidance provided here, should result in a thorough assessment process.
- Dilip Shah, "Supplier Demand," Quality Progress, May 2010, pp. 48–50.
- American National Standards Institute and the National Conference of Standards Libraries International, ANSI/NCSLI Z540.3-2006—Requirements for the Calibration of Measuring and Test Equipment.
- International Organization for Standardization, ISO/IEC 17025:2005—General requirements for the competence of testing and calibration laboratories.
- International Organization for Standardization, ISO/IEC Guide 99:2007—International vocabulary of metrology—Basic and general concepts and associated terms.