Volume 7 · Issue 10 · October 2002
The ISO 9001:2000 PSI IDEAS Q&A: Question
The ISO 9001:2000 Product Support Initiative (PSI) was developed
in 2001 by the US Technical Advisory Group (TAG) to ISO Technical
Committee (TC) 176 to assist US organizations in understanding
ISO 9001:2000, implementing quality management systems (QMSs)
and continually improving those systems. As part of the PSI,
the IDEAS (Information, Discussion, Examples, Analysis and
Sources) program is designed to provide help to organizations
Under the leadership of Jeanne Ketola, CEO of Pathway Consulting,
Inc., and an active participant of the US TAG, IDEAS has launched
a question and answer process aimed at providing a variety
of perspectives on requirements of ISO 9001:2000 and the challenges
they present to organizations using the standard. Each question
will be distributed by THE OUTLOOKs Senior Editor
to experts from the user community, the standards field and
the registrar sector who have volunteered to provide their
perspectives in answer to a given question. In addition, Jeanne
Ketola and Morgan Hall, who is the US representative to the
international ISO 9001:2000 interpretations pilot project,
will have the opportunity to comment on the question and answers
The answers will be compiled and edited to ensure consistency
of language usage but not necessarily of content and will
be published monthly in THE OUTLOOK and will be posted
to a page on the US Standards Group web site (http://standardsgroup.asq.org)
dedicated to the PSI soon thereafter. It is important to note
that these answers do not represent interpretations, sanctioned
or otherwise, by ISO/TC 176 or the US TAG, and are only the
experts perspectives of ISO 9001:2000 based on the question
and their experience and understanding.
Below is the fifth question in the series and the answers
received from four experts.
Continually improving the effectiveness of the QMS is the
focus of Subclause 8.5.1, as discussed in Question 4 in
September 2002. Effectiveness, which is largely associated
with continual improvement, is also related to other QMS
activities (e.g., evaluating the effectiveness of actions
taken under Subclause 6.2.2, Competence, Awareness and Training,
conducting internal audits to ensure the QMS is effectively
implemented and maintained). Also, Clause 0.2, IntroductionProcess
Approach, states that ISO 9001:2000 "promotes the adoption
of a process approach when developing, implementing and
improving the effectiveness" of a QMS and that, "When
used within a quality management system, such an approach
emphasizes the importance of
c) obtaining results
of process performance and effectiveness
But what is meant by "effectiveness"? Specifically,
what are some examples of how an organization will determine
whether or not its QMS is effective? What types of changes
may need to be made in an organization to ensure that the
effectiveness of the QMS is continually improved over time?
What types of process performance measures might an organization
need to establish to help it determine effectiveness?
Further, how will registrar auditors determine that an
organization has continually improved the QMSs effectiveness
over time? What types of objective evidence will they be
looking for? If many continual improvement activities are
taking place, what evidence would prove that the effectiveness
of the QMS is actually improving?
Answer From User Expert
James August, CQA, Quality Assurance Manager, American
Biltrite Inc., Moorestown, NJ
The word "effectiveness" is used in several places
in ISO 9001:2000. In the Introduction, Clause 0.2 refers to
"effectiveness" as noted above. While the standard
often uses the words "effective" and "effectiveness",
there are two subclauses containing requirements where having
a common understanding of "effectiveness" is especially
- Subclause 6.2.2 requires an organization to evaluate the
effectiveness of actions taken to ensure the necessary competence
of workers affecting product quality.
- Subclause 8.5.1, Continual Improvement, requires an organization
to continually improve the effectiveness of its QMS.
These were highlighted in the question. In addition, at American
Biltrite, we identified that Clause 5.1, Management Commitment,
specifies that top management is to provide evidence of its
commitment to this task of improving QMS effectiveness. Picking
up from here, Subclause 5.6.3, Management ReviewReview
Output, requires such evidence to be an output from management
reviews of the QMS.
So, what is meant by "effectiveness"? Effectiveness
can be defined as a measure that demonstrates that a process
yields, or is capable of yielding, positive results. ISO
9000:2000, Quality management systemsFundamentals and
vocabulary, defines "effectiveness" in 3.2.14
as the "extent to which planned activities are realized
and planned results achieved". Thus, the meaning of the
requirement in Subclause 6.2.2 becomes clear: organizations
must evaluate how well planned hiring, training and other
processes close the gaps in employee competency needs for
those workers who affect product quality.
But what Subclause 8.5.1 requires is not as easy to understand.
Unlike with employee competency, we found that identifying
"effectiveness gaps" in the QMS can be most difficult.
Certainly, internal and external auditors can identify gaps
between the QMS and what ISO 9001 requires, but it is more
difficult to identify gaps in the QMS that lead to less than
the best organizational performance. To do that, you need
a better understanding of how the QMS is constructed and what
function it plays in the organization.
Any organization must have a business management system to
continue operating. The business management system is the
assemblage of skilled people causing the execution of specific
procedures (e.g., disseminating information, assigning tasks,
setting priorities) by allocating the organizations
strategic and tactical resources so as to maintain and improve
business process performance. Business management systems
include performance measures to assure that objectives are
met; many of these measures are financial.
The QMS, however, has a more specific definition. From an
ISO 9000:2000 perspective, as noted in Clause 2.1, Rationale
for Quality Management Systems, QMSs are meant to "analyse
customer requirements, define the processes that contribute
to the achievement of a product which is acceptable to the
customer, and keep these processes under control
the probability of enhancing customer satisfaction and the
satisfaction of other interested parties." It is therefore
possible to have two separate management systems with extensive
areas of overlapping responsibility, within one organization,
which is how many American companies were set up under MIL-I-45208,
MIL-Q-9858 and the 1987 and 1994 editions of ISO 9001/2/3.
Thus, the basis for evaluating QMS effectiveness will revolve
around how well the QMS is integrated into the organizations
An organization implementing its first QMS to ISO 9001:2000s
requirements can meet them by incorporating the QMS elements
directly into its business management system. In fact, Clause
0.4, Compatibility With Other Management Systems, of ISO 9004:2000
states that an organization can "integrate its own quality
management system with related management systems".
Responsibility for and activities related to measuring and
improving "quality" performance will ideally be
spread among the functions of the organization. The QMS objectives
need to be objectives for the functional organization, distinguishable
only by their close alignment with the quality policy. Measures
of QMS effectiveness for such a well-integrated system will
be found among the organizations performance measuresproductivity,
efficiency and quality indicatorsthat provide the underpinning
for its financial and other business measures. And, in this
integrated system, there is evidence of the effective QMS
at all levels of the organization.
With an integrated system, auditors can expect to find Baldrige-like
alignment from customer requirements to product and process
improvements to production cost and capacity estimates to
sales objectives and profit projections. Improving the effectiveness
of the QMS means improving the organizations business
results based on integrating the voice of the customer throughout
the organization. At the highest level, an organization that
continually improves customer satisfaction may be the best
indicator that the effectiveness of its QMS is continually
For an organization where the QMS has evolved as a separate
system, evaluating QMS effectiveness becomes more complex.
In this case, the quality objectives need to parallel business
plans and likely will be based on a separate quality policy
Since the intent of ISO 9001:2000 is to build the voice of
the customer into the business processes, early measures of
the effectiveness of the ISO 9001:2000-conforming QMS must
take into account the extent to which this integration has
occurred. Besides traditional measures (e.g., defect rates,
audit compliance, process efficiency), continual improvement
of the QMSs effectiveness will be evidenced by the increasing
integration of the QMS processes with business processes.
As the QMS matures, the results of these efforts will be seen
throughout the organization, culminating in improving customer
American Biltrite, Inc., developed its QMS in 1993, combining
elements of Philip B. Crosbys philosophy with the requirements
of ISO 9001:1987. American Biltrite transitioned to ISO 9001:2000
over a 20-month period from December 2000 to July 2002. We
started with a review of the new requirements for top management
and elected to accelerate the integration of the QMS with
our business management system. The leadership team modeled
its periodic QMS reviews along the outline of ISO 9001:2000
with particular attention to the requirements of Clause 5.6.
First, our team made resources available to provide additional
auditor and process training. Second, quality tools were introduced
to top management in the evaluation of organizational performance.
Third, the annual business plan format was expanded to include
explicit quality objectives alongside the business objectives.
Each of these actions demonstrated the evolution of the QMS
toward the customer-focused business system envisioned in
ISO 9001:2000. They also enabled our organization to demonstrate
that our processes yield positive results.
In the final analysis, improving QMS effectiveness has to
mean continually changing the processes of your organization
so as to increase customer satisfaction by integrating continual
improvement approaches into all aspects of your activities.
"Like all systems, it either improves or becomes less
effective. It does not remain static for long." (Source:
Maintaining the Benefits and Continual Improvement,
ISO web site page.)
In the best systems, it will not be possible to separate improvement
of QMS effectiveness from improvement of business performance.
Answer From User Expert
Richard T. Vinton, Manager, ISO/Division Auditing, Command,
Control, Communications and Information Systems, Raytheon
This was actually an easy aspect of ISO 9001:2000 for Raytheon
to comply with. We found that the key is being able to compare
the "past state" of the QMS with the "current
state" regardless of what process/activity you are evaluating.
To be able to compare, you need metrics that must be meaningful
to the organizational goals or objectives, not just a number
thats easily reported to management.
By tying your metrics directly to your organizations
goals, you now establish a method by which to monitor continual
improvement of both the organization as well as the effectiveness
of its QMS. By plotting historical data and a goal in a histogram
format, you can quickly understand whether there is improvement
or not. In either case, a strong QMS will identify reasons
for movement in a negative or positive direction and will
specifically address actions required to fill the gap between
the current state of the QMS and the goal your organization
is striving to achieve.
At Raytheon, continual improvement is a culture, and we are
using various Six Sigma tools to analyze and achieve our goals.
Our process, Raytheon 6 Sigma, has 6 steps:
- Visualizea gap exists between the current and desired
state of a process
- Commitobtain management commitment of necessary
- Prioritizedetermine improvement activities based
upon facts and data
- Characterizedefine the existing process and plan
- Improvedesign and implement improvements
- Achievedeliver measurable results that link the
objective to results and actions.
Once the identified objective is achieved, the process starts
over again. The backbone of our system are the metrics, those
measurable results. However, it is Step 6linking the
objective to the results and actionsthat provide the
measure of QMS effectiveness for us. This is a culture that
is used at all levels of the organization and includes our
customers as key participants.
When assessing the QMS, our registrars auditors will
compare several management reviews over X period of time,
gain an understanding of the actions assigned and look for
the effective implementation of the actions (Does an issue
reappear?). They also look at the internal audit results.
Since Raytheon holds a multisite certificate, we utilize a
core group of auditors whose only responsibility is to assess
the entire organization (all sites) annually. These results
are also reported during management review as a year-to-year
comparison of the organization and by site and by the requirements
of a clause/subclause. By slicing and comparing the data (audit
results), we are able to understand the state of the QMS and
allocate resources appropriately.
In closing, to understand the effectiveness of a process,
the proper measurement must be identified for the desired
result, identify gaps, take action and then monitor the results.
Answer From Standards Expert
Dr. Lawrence A. Wilson., Principal of Lawrence A. Wilson
& Associates and US delegate to ISO/TC 176 for drafting
of ISO 9001:2000 and ISO 9004:2000
Effective adj; effectively adv;
Effectiveness = the extent to which planned activities
are realized and achieve planned results. (Derived from
3.2.14 of ISO 9000:2000.)
The management of an organization must determine the nature,
content and processes of a QMS that will be suitable and effective
for its organization to achieve the planned results necessary
for the success of the organization. Management must then
totally and effectively implement such a QMS and all its processes.
Once the system is fully implemented, management must maintain
and measure the effectiveness of the QMS in achieving the
"Planned activities" involve first identifying
and defining customer and organizational requirements and
then determining those activities and processes that are capable
of being used to achieve the expected realization or transformation
of requirements into expected results. The term "extent"
in the ISO 9000:2000 definition implies that the planned activities,
realization processes and achievement of the planned results
may have some degree of variance from perfection.
Measurement of the level of effectiveness of any QMS can
be achieved, in part, by analyzing the processes/results of
the design and planning reviews, extent of process and procedural
variance, level of product and service nonconformity, audits
of process and product, success of corrective action and preventive
action, management reviews, monitoring of customer satisfaction
and service and field performance feedback.
Improvement of the effectiveness of a QMS in achieving the
desired planned results can be accomplished by reducing or
eliminating the causes of variances found during analysis.
Increased effectiveness can be realized by:
- Quantum changes identified and implemented as being essential
for improvement (i.e., new equipment, facilities, processes,
- Continual improvement where ongoing incremental improvements
are identified and implemented (e.g., changes to procedures,
training, tools, methods and materials).
QMS effectiveness should be continually increased when the
performance of the system and its processes is measured, the
data collected from the measurements are analyzed and an improvement
is defined, implemented and verified. Such continual process
improvements can be verified by the positive changes in the
Improvement in processes may also be achieved by other than
variance reduction, such as through the reduction of resources
required per unit of product, known as improved efficiency.
Although ISO 9001:2000 does not address efficiency, many organizations
also gain improvement by reducing per unit material, labor
and time, all of which amount to reducing per unit cost.
A reduction in cost, however, must not result in a reduction
in customer satisfaction or product and service performance.
When done correctly, to the benefit of both the customer and
the organization, improvements in both effectiveness and efficiency
can be achieved to their mutual satisfaction (see ISO 9004:2000).
Proof that the organization is actually pursuing continual
improvement is the responsibility of the organizations
management. Records of the improvement activities mentioned
above should be regarded as essential by prudent management.
Performance improvement must be a basic management objective,
and upon which they are judged.
Answer From Registrar Expert
Kevin Beard, Program Director, NQA, Inc
There are many different approaches used by organizations
to define, develop and implement the structure of their QMSs,
including process mapping, documentation, training, etc. But
when running a business and serving customer needs, "effectiveness"
of the QMS is the bottom line. We all have probably heard
the criticism of ISO 9001:1994, that it promoted "consistency"
within a QMS, which gave organizations the option to produce
bad product "with consistency".
With ISO 9001:2000, the QMS is not merely a set of procedures
that strive to produce consistency within a QMS, it is a management
system that focuses on producing information regarding the
effectiveness of the processes. The system also allows management
to understand where problems exist and where there are risks
to meeting customer needs. It is a management system where
information is used to improve the effectiveness of an organizations
Effectiveness does not focus on whether a product meets a
requirement. Effectiveness focuses on whether processes are
achieving planned results and if QMS initiatives/activities
are meeting expectations. The concept of effectiveness asks
that managers develop performance measurement strategies and
use the information to improve the organizations processes.
The key to effectiveness determination is information. The
structured acquisition of information allows managers and
process owners to analyze the performance of the various QMS
processes of an organization to determine if a process meets
their expectations and to identify process weaknesses requiring
While ISO 9001:1994 specified the concept of effectiveness
in only a few places, an organization may have to think hard
about how effectiveness applies to its management system strategies.
In ISO 9001:2000, the focus on effectiveness is disseminated
throughout various clauses/subclauses. To satisfy this, the
biggest change an organization may need to make is to have
management develop a strategy to address how all these disseminated
facets of effectiveness work together and how information
from the many sources within its QMS can be best used to determine
the health of the QMS.
An organization should strive to understand how its operations
are structured and how effectiveness can be applied to the
organizations various process categories:
- Realization Processes (the end-to-end production processcontract
- Subprocesses (steps within the Realization Processespurchasing,
- Support Processes (HR, infrastructure, quality assurance,
- Management Processes (business strategy, budgeting, resource
Once its process structure is understood, the organization
should identify information sources and their relationship
to each other, as well as impacts on other parts of the QMS.
The organization can then determine which performance measures
are important, not only for determining the effectiveness
of that individual process, but for contributing to the determination
of overall management system effectiveness.
Within ISO 9001:2000, effectiveness is not limited to one
part of an organizations QMS. Rather, it relates to
almost all aspects of the organizations business approach.
While many organizations focus on the effectiveness of the
Processes and Subprocesses that deliver the product, equal
attention should be placed on the overhead costs and effectiveness
of the QMS support processes. In breaking down an organization,
costs are evaluated in many ways (i.e., sales/marketing costs,
product delivery costs, overhead costs, etc.).
With ISO 9001 focused on the management system of the "organization",
quite often the cost for implementing the system falls into
the overhead column of the profit and loss statement. Much
of the cost associated with a QMS does not directly contribute
to the delivery of product, and understanding this highlights
the importance of establishing effectiveness measures across
the organizations QMS. Management needs to pay attention
to the effectiveness of infrastructure processes (IT, preventive
maintenance, etc.), training, internal audits, corrective
action procedures, etc.
As for process performance measures, Clause 8.2, Monitoring
and Measurement, of ISO 9001:2000 establishes requirements
for four QMS "areas"monitoring of customer
satisfaction, internal auditing of the QMS and monitoring
and measurement of QMS processes and of product. Each of these
areas has a unique focus within an organizations QMS,
and each can contribute to the evaluation of the QMSs
When it comes to the measurement of Realization Processes,
organizations quite often gather only information to evaluate
the effectiveness of a subprocess (i.e., purchasing, manufacturing
steps, etc.). However, an organization can make significant
determinations of QMS effectiveness when it starts to relate
performance measures information across its interrelated subprocesses.
For example, the impact of a final product inspection problem
is not limited to failure to ensure that good product gets
delivered to customers. Information on a problem found with
the final inspection process could also be used to determine
the effectiveness of the in-process inspections, receiving
inspection, supplier control approach, design review process,
etc. This analogy can be applied to warranty returns and other
significant events within an organizations realization
While much has been said about customer satisfaction, internal
audits and product monitoring and measuring, "QMS process
measures" (Subclause 8.2.3) frequently make the most
significant contribution to determining QMS effectiveness.
Subclause 8.2.3 seems like a catchall requirement in ISO 9001:2000,
but the intent is to ensure that organizations look beyond
individual product inspection data for performance measures
to information and data from all sources within the QMS.
An example can be found in the area of support processes,
where many of these QMS functions are treated as overhead
(i.e., training, internal audits, etc.). In this case, organizations
should strive to ensure that overhead-related QMS functions
are effective and that nonproduct-related monitoring systems
(e.g., internal audits, customer satisfaction) are providing
value-added information to management regarding the effectiveness
of the QMS.
Let me give you a case in point. As an auditor, I find that
one of the most frequent causes of nonconformities is employee
error, with the most frequent corrective action being to retrain
employees. However, if retraining is the most common response
dictated by an organizations corrective action procedure,
what does this say about the effectiveness of its training
approach? Many examples of this type can be used not only
to determine the ability or effectiveness of the corrective
action procedure and internal auditing program to reduce repeat
problems and provide system-based feedback to management,
but to determine the effectiveness of the organizations
infrastructure-oriented groups as well.
How will registrar auditors determine if an organization
has continually improved its QMS effectiveness over time?
If an organization has developed good QMS (performance measurement)
strategies, it should be a fairly straightforward process
for an auditor to find the objective evidence of effectiveness.
This should be evident in the quality records and metrics
maintained by the organization. Whether the organization uses
metrics, meeting minutes or other data gathering/evaluation
methods, the key realization and QMS process records will
be viewed to see how the organization over time monitors its
own effectiveness and what improvement initiatives are taken
based on the evaluation results.
While the focus of mid-level managers may be on the effectiveness
of their Subprocesses or Support Processes, top management
must keep an eye on the overall effectiveness of the QMS processes.
Auditors can interview top management to evaluate the organizations
strategies towards effectiveness, improvement and initiative
alignment. Then the various levels of management can be evaluated
to assure alignment of improvement initiatives, which contribute
to the overall improvement strategies with regards to QMS
Return to Top