MEASURE FOR MEASURE
Matter of Life and Death
Lack of traceability can turn magic into mayhem
Is the headline an exaggeration or reality? I hope it’s an exaggeration. But it might not be far from the truth. In too many cases, lack of traceability is a daily occurrence. I’m not talking about the traceability of a batch or lot of product. I’m referring to the traceability of test instruments to a national standard, industry-accepted consensus standard or natural-occurring phenomena.
According to the International Vocabulary of Basic and General Terms in Metrology (VIM), traceability is a property of the result of a measurement and provides the ability to relate the measurement result to stated references through an unbroken chain of comparisons, each having stated uncertainties.1 This can be kind of a mouthful for those not making a living in the metrology or calibration fields. Let me give you an example and see if it helps.
An R&D group has found the secret to curing cancer. It can be found in a small tube that has been incubating in one specific water bath located in the lab of Dr. Jane Deaux. Everyone refers to that particular water bath as "where magic happens." It has been in use for decades, the water has been kept clean, and all filters have been changed or cleaned on a regular base (as far as anyone can remember). Every time it is used at the experiment’s design temperature of 42º C, magic continues to happen.
Two floors down and around the corner, the manufacturing staff is trying to duplicate the product. The magic, however, just isn’t happening. No matter what water bath the staff uses or how closely it follows the protocol, the staff cannot reproduce the product.
Several things are at play here: R&D’s water bath has never been calibrated, nor have any of its pipettes, balances or other test equipment. Why? Because they don’t fall under the company’s International Organization for Standardization (ISO) guidelines, or so R&D has led management to believe. All of manufacturing’s test instruments have recurring, traceable calibrations and meet all specifications. So, while manufacturing carefully measures, weighs and spins the ingredients, they just don’t have enough "magic" to get the job done.
As it turns out, the set of pipettes Dr. Deaux used to measure those special ingredients came with her when she left academia to delve into the commercial world of the life sciences. They were instrumental in helping her get a master’s degree and her doctorate.
It was obvious to the most casual observer that with those pipettes, a solid protocol and some miscellaneous test equipment, she could solve all the world’s problems. Too bad she did not know her pipettes required traceable calibration on a regular basis.
Over the years, the seals and o-rings were worn, and her actual results were off by a factor of three. Her pipettes and the uncalibrated balance she was using were out-of-tolerance on the high end (she was measuring too much of everything), and the water bath was on the low end (she was not heating to the specified temperature). Her use of those test instruments to repeat the study and measure her final product all resulted in a false positive.
Not a trace of doubt
The test equipment used in manufacturing, on the other hand, had been calibrated against working standards that were traceable to the National Institute of Standards and Technology (NIST) through an unbroken chain of comparisons, each having stated uncertainties. Here is how that works:
NIST uses its primary standard to calibrate an instrument for a temperature equipment manufacturer (let’s call them Acme Scientific Co., or ASC for short) and provides it with a calibration certificate that shows the stated uncertainties of the primary standard, as well as the uncertainties of the test instrument that is calibrated.
Keep in mind that calibration, as defined by the VIM, is a comparison of a standard with known uncertainties to an instrument of unknown uncertainties.2 Industry practice is for each standard to be at least four times more accurate than the test instrument being calibrated. This equates to a test uncertainty ration (TUR) of 4-to-1. When this ratio cannot be met, then uncertainty budgets are calculated to show the actual uncertainties and are recorded in the Certificate of Calibration.
ASC now has a standard (with known uncertainties) it can use to calibrate high-end reference or working standards sent to it from any company that wants traceable calibration. For example, ASC uses its standard in a controlled environment to compare the results obtained during the calibration of a biotech company’s (we’ll call them Biotech Science Inc. or BSI for short) temperature standard.
After the calibration by ASC, BSI now has a standard with known uncertainties that can be used to calibrate its own working standards. Every piece of test equipment that makes a quantitative measure in the manufacturing division of BSI receives a traceable calibration on a regular basis. Each calibration is performed using a standard operating procedure, and the results are recorded, along with the readings from the standard that is used.
If the test instrument is out of tolerance, it is adjusted, the final results are also recorded, and the technician performing the calibration signs off on everything. These records are reviewed and signed off by the calibration supervisor and kept on file, per ISO requirements.
The bottom line is this: No matter which pipette, balance or water bath being used during the manufacturing process at BSI, the results are always the same—within the window of the measurement process uncertainty—because the test equipment has traceable calibration to a national standard.
Such is not the case in Dr. Deaux’s laboratory. When it came to her cancer research, "magic" was happening on an irregular basis. By not insisting on the use of test equipment that had traceable calibrations, she could be depriving mankind of that elusive cure so many R&D departments are trying to find.
Learning a lesson
You can’t put too much blame on her for a couple of reasons: education and communication. Yes, she is highly educated, but not in the need to use traceable calibration. She probably never saw any signs of traceable calibration while getting her bachelor’s, master’s or doctorate degrees. As with most institutions, if the little red light comes on when you hit the "on" switch, the instrument must be working properly.
Nothing could be further from the truth. Everything wears out, degrades and changes value over time. That is part of the reason every piece of test equipment requires calibration on a recurring basis. In the words of a Russian proverb used often by the late President Ronald Reagan, "Trust, but verify."
No wonder calibration is a specific requirement for companies that are ISO compliant and regulated by the Food and Drug Administration (FDA). Calibration must be performed using calibration procedures and traceable standards, and the data recorded and filed. To quote the Code of Federal Regulations: "When accuracy and precision limits are not met, there shall be provisions for remedial action to establish the limits and to evaluate whether there was any adverse effect on the device’s quality."3
Granted, this comes from the FDA, and not everyone falls under its purview. But did you know the FDA regulates approximately 25% of the U.S. Gross National Product? In July 2007, that equated to about $3.5 trillion. None of that could have happened without the use of calibrated test equipment traceable to NIST or any other international standard.
So, the next time your experiment, product or manufacturing process doesn’t give you the results you’re looking for, I would ask you to consider whether your test equipment is providing you with the traceable measurements you require.
Some people still believe calibration is a necessary evil that must be tolerated to comply with ISO standards or FDA requirements. Eventually, they must realize that traceable calibration of their test instruments is a critical part of the foundation for building a quality system for reproducible results and enhancing the bottom line of their company.
William Thomson, 1st Baron Kelvin (also known as Lord Kelvin), is quoted as saying: "If you cannot measure it, you cannot improve it." One hundred and twenty-five years after he said them, Lord Kelvin’s words ring just as true.
Dr. Deaux and the aforementioned companies don’t exist—or do they? We all see the results of companies not using calibrated test equipment day after day. A quality calibration program doesn’t cost much to put in place and maintain. But the cost of not having one can be very expensive: in the loss of product, money and reputation—and possibly that ever-elusive cure for cancer.
- International Organization for Standardization, "International Vocabulary of Basic and General Terms in Metrology," section 2.24.
- Ibid., section 2.22.
- Food and Drug Administration, "Code of Federal Regulations," 21 CFR Part 820.72 (b).
Jay L. Bucher is president of Bucherview Metrology Services, LLC, a consulting company specializing in quality calibration systems. He is co-author and editor of The Metrology Handbook, and author of The Quality Calibration Handbook and Paperless Records—Designing and Creating Your Own Electronic Forms. He is a senior member of ASQ, the treasurer of the ASQ Measurement Quality Division and a certified calibration technician.