MEASURE FOR MEASURE
Four Steps to Ensure Measurement Data Quality
by Robert M. Graham
Part one of this column, published in July, posed four questions you should ask to ensure your measurements are providing data that are good enough for the given application:
- What do you measure that’s important?
- How good do your measurements need to be?
- How do you know they are good enough?
- In the future, when you need to, will you know what you measured and how?
The first two questions were discussed in the July column. The second two will be discussed here.
Question 3: How Do You Know the Measurements Are Good Enough?
Once you have determined how good the measurements need to be, you have to figure out whether the measurements will fulfill your minimum requirements.
Using calibrated test equipment is usually necessary but is never sufficient. Relying only on calibration may lull you into a false sense of confidence. In fact, in some cases calibration may not even be necessary. Each situation needs to be evaluated individually, using critical thinking.
With critical thinking, the decision maker is continually updating the decision process based on new data. The opposite is someone making a decision without considering all the alternatives or important factors and never questioning that decision again.
To determine whether calibration is necessary, compare the criticality of the measurement to its impact on the final result. For example, should a tire pressure gage be calibrated?
For most applications, such as checking the air pressure in your car’s tires before starting a trip, the answer is probably no. But for a high performance race car, where a difference of even a few pounds per square inch can affect the car’s handling and top speed, calibration is a very good idea.
For military or commercial aircraft, calibration is definitely required. But the end use might not be the only determining factor.
Another example: A customer requested that a mass measurement be performed on a large 20-kilogram (44-pound) iron ram that was in poor condition. It was rusted, dirty and encrusted with salt. The ram was used to crush salt samples during the construction and validation of a long-term storage facility for low level nuclear waste.
The laboratory is able to calibrate a 20-kilogram mass standard to an uncertainty level of less than 10 milligrams. However, the user needs to know the mass value only to an uncertainty of 5% (1 kilogram, or 2.2 pounds).
Normally, the calibration laboratory would refuse this kind of calibration because an uncalibrated household scale would be sufficient to ensure adequate performance. However, the legal implications, as well as the environmental, safety and health requirements, mandated a traceable calibration. Again, performance requirements may not be the most important driving factor.
Question 4: In the Future, Will You Know What You Measured and How?
Properly answering this question can save a lot of time and energy in the future. Documentation is the key: By recording the facts that influenced the decision making process right away, you don’t have to try to reconstruct them if questions come up long after you’ve forgotten what was done originally.
The documentation does not need to be complicated. As long as the process is followed consistently, reconstructing the past thought processes should be simple. Documentation is also helpful if the original decision maker is not available and someone else needs to determine why certain decisions were made.
At Sandia National Laboratories, we have
developed a measurement assurance process (MAP) to help us and
our customers implement this four-question process to document
measurement decisions. An example MAP form is shown in Table
The form has three major sections, representing the first three questions. The fourth question is satisfied by filling out the form and filing it in a safe place. Each major section is divided into two or three subsections to record detailed information for each important measurement.
The first column under question one lists the important measurement to be made. The second column lists the measurement concerns (MC), or why the measurement is considered important. For this form, the six MCs that cover most situations are listed on the bottom of the form. Other concerns can be added directly to the form if necessary.
The first column under question two shows the expected quantity to be measured. Don’t forget to include units—if a column shows a temperature measurement of 150, does that mean degrees Fahrenheit, degrees Celsius or Kelvin? There are significant differences among the three, and the correct unit might not be obvious to someone unfamiliar with the measurement process.
The second column under question two shows the desired tolerance or uncertainty, and the third column shows the measurement and test equipment (M&TE) that will be used to make the measurement.
Under question three, the first column lists detailed specifications of the M&TE, and the last two columns detail the decision factors. The test accuracy ratio is the ratio of required tolerance to the equipment accuracy. In many cases, if this ratio is greater than or equal to 4.0, the M&TE is considered adequate for the measurement. Otherwise a more rigorous uncertainty analysis might be required.
Typically, we fill out one or more forms for each major measurement process, using one row for each individual measurement made during that process. Let’s take an example: heat treating a part.
We place the part in an oven at 200° C (plus or minus 5° C) for 30 minutes (plus or minus 1 minute) and then remove it and allow it to air cool. We must make two measurements: temperature and time.
If we use a type K thermocouple and readout for
the temperature measurement and a stopwatch for the time
measurement, the form will look similar to Table 2. The first
line is used for the temperature measurement and the second line
for the time measurement.
As Table 2 (p. 83) shows, the decision is that calibration is required for the temperature measuring M&TE but not for the stopwatch. If an auditor or assessor questions why the stopwatch is not calibrated, the form will show how we reached the decision through an analysis of the process and critical thinking.
Let’s look at another example of how to employ this process with critical thinking. We are going to conduct a rocket flight test and collect data during the flight on several different parameters at many different locations in the rocket body and the payload.
There are four major measurements to be made: vibration, shock, temperature and strain. There will be preflight tests under laboratory conditions before the actual flight test is conducted, so two separate situations need to be analyzed, evaluated and documented on two forms.
Let’s examine the laboratory tests first
and fill out the MAP form (Table 3). The first measurement is
vibration. The primary measurement concern is performance, so we
put a one in that column. The expected range of vibration is
approximately 10 gravities from 30 hertz to 1,000 hertz; an
accuracy of 15% is acceptable. The semiconductor strain-gage
accelerometers, when calibrated with their associated
electronics, have an uncertainty of 3%, so the test accuracy
ratio is five.
Shock will be measured with a similar device. The expected range is 1,000 gravities, and the acceptable accuracy is 40%. Because the devices can be calibrated to 8% at 1,000 gravities, we can maintain a test accuracy ratio of five.
However, during the laboratory test phase we do not expect the unit to actually experience any shocks, so we simply will verify the devices are operational before the lab test.
The next parameter to consider is temperature. We have 115 thermistors to monitor the temperature variations in the rocket body and payload assembly. The expected temperature range is -55° C to 125° C during flight, with a desired uncertainty of 1.5° C.
Because it is expensive and time consuming to get all 115 thermistors calibrated over a wide temperature range, we decided to have a 10% sample calibrated. If any of them are found to be bad, we can take another sample to see whether it is an isolated event or the entire lot should be calibrated or rejected and replaced.
Finally, the strain gage will not be used to record data. It only provides a go/no-go indication of whether the rocket body has been overstressed. There are no specifications, so the unit will not need to be calibrated.
Once the laboratory tests are finished and the
measurement scheme validated, it is time for the flight test.
Since the conditions are different, a new MAP form is generated
(see Table 4). We will make the same measurements; they’ll
just cover a different operating environment.
The vibration measurements are the same as those on the original form, as are the conclusions. Next, because the rocket will be flight tested, it will experience shock conditions during its launch and trajectory, so we will have the shock transducers fully calibrated along with the associated electronics. We also want to ensure all the thermistors are giving valid data, so we will commit the time and money to have all 115 thermistors calibrated over the full temperature range.
Finally, the strain gage still does not need to be calibrated, but this time we will verify the signal is present before launch. Even though the measurements are identical in both cases, critical thinking shows that since the conditions have changed, the conclusions also have changed.
This is why it is important to revisit your measurement decisions when something changes—this is critical thinking. Changes in the measurement conditions, test equipment or procedures are all reasons to recheck that the original decisions are still valid and change them if they are not.
Again, document your conclusions, even if you make no changes to the original measurement decisions. This ensures you (or someone else) will know the decisions were reconsidered when something that affected the measurement decisions changed.
This form can be used to make noncalibration related measurement decisions too. The process is what is important, not the actual form.
First, decide what you need to measure.
Customer satisfaction, response time, total number of
customers—the list is endless and limited only by your
requirements. Table 5 shows two more examples of how the form can
be modified and used.
Valid and Worthwhile
When answered and properly documented, these four questions can help ensure your measurements are valid and worthwhile. Identifying the important measurements that are made, determining the level of accuracy required and then using the proper tools to make the measurements will yield valid, useful results.
Remember, critical thinking is the key to success.
ROBERT M. GRAHAM is a principal metrologist at the Primary Standards Laboratory of Sandia National Laboratories in Albuquerque, NM. Sandia is a multiprogram laboratory operated by Sandia Corp., a Lockheed Martin company, for the United States Department of Energy under contract DE-AC04-94AL85000. Graham has a master’s degree in technical management from the University of Phoenix in Albuquerque. He is a member of ASQ’s Measurement Quality Division and is a certified calibration technician.