MEASURE FOR MEASURE
Evaluating laboratory measurement performance
by Dilip Shah
Proficiency testing (PT) is defined as the "evaluation of participant performance against pre-established criteria by means of interlaboratory comparisons" by the ISO/IEC 17043 standard.
PT is often misunderstood when laboratories seeking initial ISO/IEC 17025 accreditation hear about it. Many think that its purpose is to test their employees’ skills. While proficiency is indirectly a result of the employees’ skills, training and other factors, PT assesses an individual laboratory’s measurement performance against the industry measurement performance. It identifies areas for improvement in measurement and techniques. It may identify best practices by comparing other laboratories in the same field of calibration or testing.
For the customers of laboratory services, PT:
- Establishes confidence and is a demonstration of accreditation.
- Helps the customer decide whether the laboratory meets its measurement, calibration and testing requirements.
- Acts as a measure for ensuring that the laboratory will continuously and consistently meet its quality requirements.
Continued, planned participation in PT is also a requirement by accrediting bodies for a laboratory to maintain accreditation. An example is clause 5.9.1(b) of ISO 17025, which requires laboratories to participate in PT or interlaboratory comparison schemes. There are many different PT schemes that a laboratory can participate in to assure the quality of its calibration and tests. One of the schemes for participation is outlined in Figure 1.
Acceptance of PT data is based on several factors. Statistical tests are one method to determine compliance and to form the basis for things such as data outliers. ISO/IEC 17043 Appendix B provides a limited discussion on statistical methods. It is important that a PT provider has a good statistical support base to ensure that the correct, unbiased assumption about data is made and reported.
At a minimum, the following statistical parameters should be considered when determining whether PT data are acceptable:
- Standard deviation.
- Range (range can be a good estimator of variability).
- Statistical significance using z, t or F tests.
- Measurement uncertainty.
The laboratory is asked to report its measurement result and the associated measurement uncertainty when participating in a commercial PT program. The PT provider issues a report with a calculated En number, which is based on the following formula and criteria for acceptance:
in which x is the participant laboratory’s measured result; X is the assigned value by a reference laboratory; U2LAB is the uncertainty of the participant’s result (k = 2); and U2ref is the uncertainty of the reference laboratory’s assigned value (k = 2).
Development of measurement uncertainty budgets and measurement uncertainty analysis of the measurement process is another important consideration when reporting the measurement data for PT.
Many laboratories fail the PT test participation (En value >1) because they may have made an error in calculating measurement uncertainty when they report the measurement results to their PT providers. These are some of the common errors that a participating laboratory may make when reporting the measurement uncertainty (U2LAB in the En equation):
- The specification of the blind artifact that the laboratory is measuring is taken into account.
- The uncertainty units are not in the same unit as that of the artifact value measured (such as reported in percentage, parts per million or milligrams instead of grams).
- The laboratory forgets to convert all uncertainty contributors to one standard deviation before combining with the root sum square method.
- The laboratory reports the measurement uncertainty that is in the laboratory’s scope of accreditation (the calibration and measurement capability, or CMC value) and does not take into account the uncertainty contribution of the blind artifact.
- The laboratory forgets to report the measurement uncertainty at 95% confidence interval (usually k = 2, but may vary depending on the effective degrees of freedom and referencing the student’s t-distribution).
- A single measurement is made and reported instead of making at least 10 measurements and reporting an average (and the associated repeatability standard deviation).
- The laboratory excludes the repeatability standard deviation in the uncertainty estimation (if a single measurement is made, it will not be possible to report it).
Laboratories should ensure their measurement processes are in statistical control before participating in the PT program. Using Shewhart X-bar and R control charts in the laboratory calibration and maintenance program is one way to do this in a preventive manner. Process control should cover operator training, controlled procedures and measuring equipment repeatability and reproducibility studies.
These are good laboratory practices that not only help affirm the PT measurement results, but ensure confidence when reporting results for the customer or if there is doubt on any measurement reported.
From the PT provider’s perspective, it is critical that the confidentiality of the laboratories be maintained when the PT data are reported publicly. The PT provider also should ensure and maintain neutrality and report data in an unbiased manner.
A reputable and technically knowledgeable PT provider will engage the laboratories on their reported measurement results to understand their technique and provide feedback in improving the technique for future participation without influencing the results in any way.
International Organization for Standardization and International Electrotechnical Commission, ISO/IEC 17043:2010—Conformity assessment—General requirements for proficiency testing.
International Organization for Standardization and International Electrotechnical Commission, ISO/IEC 17025:2005—General requirements for the competence of testing and calibration laboratories.
Dilip Shah is president of E = mc3 Solutions in Medina, OH. He is the past chair of ASQ’s Measurement Quality Division and past chair of Akron-Canton Section. Shah is also co-author of The Metrology Handbook (ASQ Quality Press, 2012), an ASQ-certified quality engineer, quality auditor, calibration technician and an ASQ fellow.