The Need To Use Standard Terminology
by Dilip Shah

s most readers know, Phil Stein, the originator of this column, died unexpectedly on June 24, 2004. Phil was a good friend and mentor to me as I assumed an active leadership role in the Measurement Quality Division (MQD). He was one of the founding members of the MQD and was active in the division at all levels.

Phil will be missed by us all, but his contribution to metrology education will live on. In this spirit, the MQD has assumed the responsibility of continuing Phil’s passion for measurement education in this column. We will feature guest contributors from the division and other invited subject matter experts. As chair of the division, I thought it fitting to write the first column.

Tolerance and Resolution

In my daily association with clients and customers, I respond to many questions related to metrology, measurement uncertainty and statistical applications. Some questions are easily answered while others take more time. I recently received the following query via e-mail: What should be the tolerance of a digital caliper with a resolution of 0.0000 inches? Is it +/-0.0001?

If one worked in the field of metrology or calibration, it would be easy to answer this question. Realizing many people managing a calibration or metrology department may not have a hands-on technical background, I asked this individual why she wanted to know.

She said her facility was to undergo an ISO 9001 reassessment audit and she did not know how to address this question when the assessor asked about it the last time. She was in charge of maintaining calibration equipment and records and wanted to make sure she knew the correct response. The auditor wanted to hear an actual number for tolerance instead of, “I do not know.”

“Fair enough,” I wrote back. So let us try to answer her query and educate ourselves as well.

I decided to call this individual and ask, “When you mention a digital micrometer with a resolution of 0.0000 inches, do you mean 0.0001 inch or 0.0005 inch?”

“A digital micrometer has a resolution of 0.0005 inches,” she said.

“So, the tolerance cannot be 0.0001 inch,” I replied.

Standard Definitions

OK, so now we know the micrometer can only resolve in 0.0005 inch increments. “Resolution” is defined by the International Vocabulary of Basic and General Terms in Metrology (VIM) as “the smallest difference between indications of a displaying device that can be meaningfully distinguished.”1 VIM also notes that for a digital measuring device, this is the change in the indication when the least significant digit changes by one step.2

“Tolerance” is a design feature that defines limits within which a quality characteristic is supposed to be on individual parts. A tolerance has to balance the perfection desired by the designer with what is economically achievable from the reality of a manufacturing process.3

“Specifications” define the expected performance limits of a large group of substantially identical finished products,4 which may possibly include all units of the digital micrometer in question here. Specification of an instrument can be used to estimate uncertainty of the measurement being made.

I then asked, “What does the manufacturer specify for this model of micrometer?”

“The manufacturer specifies an accuracy of +/-0.0005 inches,” she replied. “That is all the information provided in the catalog.”

“Accuracy” is defined as the qualitative indication of the ability of a measuring instrument to give responses close to the true value of the parameter being measured. Accuracy is a design specification and may be verified during calibration.5

The least significant digit on the micrometer will display either a 0 or 5. So the accuracy statement of +/- 0.0005 inches makes sense. The indicated reading on the micrometer can be read between +/-0.0005 inches of the actual (true) value. For example, an indicated measurement of 0.9000 inches can be anywhere between 0.8995 and 0.9005 inches.

Do Not Assume

We did not discuss the other factors that would have defined the tolerance on this particular piece of measuring equipment, such as method, environment, calibration and measurement uncertainty. We decided the conservative tolerance on this micrometer will be anywhere between +/-0.001 and +/-0.0005 inches (not 0.0001 inches) until we can quantify all the parameters effectively.

We also came to the conclusion that this is a very simplified solution as the instrument in question just measures one parameter (dimension) and a specified range (0 to 1.0000 inches). If the analysis was to quantify the tolerance of a digital multimeter with a capability of measuring more than one parameter and multiple ranges, our task would be complicated because we would have to list the tolerance of more than one parameter, all the ranges and all the factors influencing the measurement.

As quality and metrology professionals, we need to better educate each other and communicate with each other using standard terminology. Too often, we assume a measuring instrument that can display (resolve) a certain number of decimal points is accurate to the same number of decimal places. Resolution does not automatically guarantee the same level of accuracy.

I got this individual to start thinking about various terms related to measurement and referencing the published data on the micrometer. That is a good start. Now we need to learn why making just one measurement is not a good idea. But that is a discussion for another column.

Note: Phil Stein devoted his July 2002 “Measure for Measure” column “Choose Your Words Carefully”
(p. 101) to specification and tolerances, and he devoted his July 2001 column “All You Ever Wanted To Know About Resolution” (p. 141) to resolution. The columns are available to all ASQ student, associate and regular members at www.asq.org (in the Quality Progress back issues archive) and serve as an interesting and educational primer.


1. International Vocabulary of Basic and General Terms in Metrology, second edition, International Organization for Standardization, 1993.

2. Jay Bucher, The Metrology Handbook, ASQ Quality Press, 2004.

3. Ibid.

4. Ibid.

5. Ibid.


DILIP SHAH is the president of E = mc3 Solution, in Wadsworth, OH, a consulting practice that assists clients in the area of metrology, training and ISO 17025 laboratory accreditation. He is the current chair of the Measurement Quality Division and co-author of The Metrology Handbook published by ASQ Quality Press.

Be able to provide auditors with specific answers.


84 I NOVEMBER 2004 I www.asq.org



If you would like to comment on this article, please post your remarks on the Quality Progress Discussion Board at www.asq.org, or e-mail them to editor@asq.org.


Average Rating


Out of 0 Ratings
Rate this article

Add Comments

View comments
Comments FAQ

Featured advertisers