Volume 7 · Issue 10 · October 2002

Contents

The ISO 9001:2000 PSI IDEAS Q&A: Question 5

The ISO 9001:2000 Product Support Initiative (PSI) was developed in 2001 by the US Technical Advisory Group (TAG) to ISO Technical Committee (TC) 176 to assist US organizations in understanding ISO 9001:2000, implementing quality management systems (QMSs) and continually improving those systems. As part of the PSI, the IDEAS (Information, Discussion, Examples, Analysis and Sources) program is designed to provide help to organizations during implementation.

Under the leadership of Jeanne Ketola, CEO of Pathway Consulting, Inc., and an active participant of the US TAG, IDEAS has launched a question and answer process aimed at providing a variety of perspectives on requirements of ISO 9001:2000 and the challenges they present to organizations using the standard. Each question will be distributed by THE OUTLOOK’s Senior Editor to experts from the user community, the standards field and the registrar sector who have volunteered to provide their perspectives in answer to a given question. In addition, Jeanne Ketola and Morgan Hall, who is the US representative to the international ISO 9001:2000 interpretations pilot project, will have the opportunity to comment on the question and answers given.

The answers will be compiled and edited to ensure consistency of language usage but not necessarily of content and will be published monthly in THE OUTLOOK and will be posted to a page on the US Standards Group web site (http://standardsgroup.asq.org) dedicated to the PSI soon thereafter. It is important to note that these answers do not represent interpretations, sanctioned or otherwise, by ISO/TC 176 or the US TAG, and are only the experts’ perspectives of ISO 9001:2000 based on the question and their experience and understanding.

Below is the fifth question in the series and the answers received from four experts.

Question 5.
Continually improving the effectiveness of the QMS is the focus of Subclause 8.5.1, as discussed in Question 4 in September 2002. Effectiveness, which is largely associated with continual improvement, is also related to other QMS activities (e.g., evaluating the effectiveness of actions taken under Subclause 6.2.2, Competence, Awareness and Training, conducting internal audits to ensure the QMS is effectively implemented and maintained). Also, Clause 0.2, Introduction—Process Approach, states that ISO 9001:2000 "promotes the adoption of a process approach when developing, implementing and improving the effectiveness" of a QMS and that, "When used within a quality management system, such an approach emphasizes the importance of… c) obtaining results of process performance and effectiveness…."

But what is meant by "effectiveness"? Specifically, what are some examples of how an organization will determine whether or not its QMS is effective? What types of changes may need to be made in an organization to ensure that the effectiveness of the QMS is continually improved over time? What types of process performance measures might an organization need to establish to help it determine effectiveness?

Further, how will registrar auditors determine that an organization has continually improved the QMS’s effectiveness over time? What types of objective evidence will they be looking for? If many continual improvement activities are taking place, what evidence would prove that the effectiveness of the QMS is actually improving?

Answer From User Expert…
James August, CQA, Quality Assurance Manager, American Biltrite Inc., Moorestown, NJ

The word "effectiveness" is used in several places in ISO 9001:2000. In the Introduction, Clause 0.2 refers to "effectiveness" as noted above. While the standard often uses the words "effective" and "effectiveness", there are two subclauses containing requirements where having a common understanding of "effectiveness" is especially important:

• Subclause 6.2.2 requires an organization to evaluate the effectiveness of actions taken to ensure the necessary competence of workers affecting product quality.
• Subclause 8.5.1, Continual Improvement, requires an organization to continually improve the effectiveness of its QMS.

These were highlighted in the question. In addition, at American Biltrite, we identified that Clause 5.1, Management Commitment, specifies that top management is to provide evidence of its commitment to this task of improving QMS effectiveness. Picking up from here, Subclause 5.6.3, Management Review—Review Output, requires such evidence to be an output from management reviews of the QMS.

So, what is meant by "effectiveness"? Effectiveness can be defined as a measure that demonstrates that a process yields, or is capable of yielding, positive results. ISO 9000:2000, Quality management systems—Fundamentals and vocabulary, defines "effectiveness" in 3.2.14 as the "extent to which planned activities are realized and planned results achieved". Thus, the meaning of the requirement in Subclause 6.2.2 becomes clear: organizations must evaluate how well planned hiring, training and other processes close the gaps in employee competency needs for those workers who affect product quality.

But what Subclause 8.5.1 requires is not as easy to understand. Unlike with employee competency, we found that identifying "effectiveness gaps" in the QMS can be most difficult. Certainly, internal and external auditors can identify gaps between the QMS and what ISO 9001 requires, but it is more difficult to identify gaps in the QMS that lead to less than the best organizational performance. To do that, you need a better understanding of how the QMS is constructed and what function it plays in the organization.

Any organization must have a business management system to continue operating. The business management system is the assemblage of skilled people causing the execution of specific procedures (e.g., disseminating information, assigning tasks, setting priorities) by allocating the organization’s strategic and tactical resources so as to maintain and improve business process performance. Business management systems include performance measures to assure that objectives are met; many of these measures are financial.

The QMS, however, has a more specific definition. From an ISO 9000:2000 perspective, as noted in Clause 2.1, Rationale for Quality Management Systems, QMSs are meant to "analyse customer requirements, define the processes that contribute to the achievement of a product which is acceptable to the customer, and keep these processes under control…to increase the probability of enhancing customer satisfaction and the satisfaction of other interested parties." It is therefore possible to have two separate management systems with extensive areas of overlapping responsibility, within one organization, which is how many American companies were set up under MIL-I-45208, MIL-Q-9858 and the 1987 and 1994 editions of ISO 9001/2/3. Thus, the basis for evaluating QMS effectiveness will revolve around how well the QMS is integrated into the organization’s business system.

An organization implementing its first QMS to ISO 9001:2000’s requirements can meet them by incorporating the QMS elements directly into its business management system. In fact, Clause 0.4, Compatibility With Other Management Systems, of ISO 9004:2000 states that an organization can "integrate its own quality management system with related management systems".

Responsibility for and activities related to measuring and improving "quality" performance will ideally be spread among the functions of the organization. The QMS objectives need to be objectives for the functional organization, distinguishable only by their close alignment with the quality policy. Measures of QMS effectiveness for such a well-integrated system will be found among the organization’s performance measures—productivity, efficiency and quality indicators—that provide the underpinning for its financial and other business measures. And, in this integrated system, there is evidence of the effective QMS at all levels of the organization.

With an integrated system, auditors can expect to find Baldrige-like alignment from customer requirements to product and process improvements to production cost and capacity estimates to sales objectives and profit projections. Improving the effectiveness of the QMS means improving the organization’s business results based on integrating the voice of the customer throughout the organization. At the highest level, an organization that continually improves customer satisfaction may be the best indicator that the effectiveness of its QMS is continually improving.

For an organization where the QMS has evolved as a separate system, evaluating QMS effectiveness becomes more complex. In this case, the quality objectives need to parallel business plans and likely will be based on a separate quality policy statement.

Since the intent of ISO 9001:2000 is to build the voice of the customer into the business processes, early measures of the effectiveness of the ISO 9001:2000-conforming QMS must take into account the extent to which this integration has occurred. Besides traditional measures (e.g., defect rates, audit compliance, process efficiency), continual improvement of the QMS’s effectiveness will be evidenced by the increasing integration of the QMS processes with business processes. As the QMS matures, the results of these efforts will be seen throughout the organization, culminating in improving customer satisfaction.

American Biltrite, Inc., developed its QMS in 1993, combining elements of Philip B. Crosby’s philosophy with the requirements of ISO 9001:1987. American Biltrite transitioned to ISO 9001:2000 over a 20-month period from December 2000 to July 2002. We started with a review of the new requirements for top management and elected to accelerate the integration of the QMS with our business management system. The leadership team modeled its periodic QMS reviews along the outline of ISO 9001:2000 with particular attention to the requirements of Clause 5.6.

First, our team made resources available to provide additional auditor and process training. Second, quality tools were introduced to top management in the evaluation of organizational performance. Third, the annual business plan format was expanded to include explicit quality objectives alongside the business objectives. Each of these actions demonstrated the evolution of the QMS toward the customer-focused business system envisioned in ISO 9001:2000. They also enabled our organization to demonstrate that our processes yield positive results.

In the final analysis, improving QMS effectiveness has to mean continually changing the processes of your organization so as to increase customer satisfaction by integrating continual improvement approaches into all aspects of your activities. "Like all systems, it either improves or becomes less effective. It does not remain static for long." (Source: Maintaining the Benefits and Continual Improvement, ISO web site page.) In the best systems, it will not be possible to separate improvement of QMS effectiveness from improvement of business performance.

Answer From User Expert…
Richard T. Vinton, Manager, ISO/Division Auditing, Command, Control, Communications and Information Systems, Raytheon

This was actually an easy aspect of ISO 9001:2000 for Raytheon to comply with. We found that the key is being able to compare the "past state" of the QMS with the "current state" regardless of what process/activity you are evaluating. To be able to compare, you need metrics that must be meaningful to the organizational goals or objectives, not just a number that’s easily reported to management.

By tying your metrics directly to your organization’s goals, you now establish a method by which to monitor continual improvement of both the organization as well as the effectiveness of its QMS. By plotting historical data and a goal in a histogram format, you can quickly understand whether there is improvement or not. In either case, a strong QMS will identify reasons for movement in a negative or positive direction and will specifically address actions required to fill the gap between the current state of the QMS and the goal your organization is striving to achieve.

At Raytheon, continual improvement is a culture, and we are using various Six Sigma tools to analyze and achieve our goals. Our process, Raytheon 6 Sigma, has 6 steps:

1. Visualize—a gap exists between the current and desired state of a process
2. Commit—obtain management commitment of necessary resources
3. Prioritize—determine improvement activities based upon facts and data
4. Characterize—define the existing process and plan improvements
5. Improve—design and implement improvements
6. Achieve—deliver measurable results that link the objective to results and actions.

Once the identified objective is achieved, the process starts over again. The backbone of our system are the metrics, those measurable results. However, it is Step 6—linking the objective to the results and actions—that provide the measure of QMS effectiveness for us. This is a culture that is used at all levels of the organization and includes our customers as key participants.

When assessing the QMS, our registrar’s auditors will compare several management reviews over X period of time, gain an understanding of the actions assigned and look for the effective implementation of the actions (Does an issue reappear?). They also look at the internal audit results. Since Raytheon holds a multisite certificate, we utilize a core group of auditors whose only responsibility is to assess the entire organization (all sites) annually. These results are also reported during management review as a year-to-year comparison of the organization and by site and by the requirements of a clause/subclause. By slicing and comparing the data (audit results), we are able to understand the state of the QMS and allocate resources appropriately.

In closing, to understand the effectiveness of a process, the proper measurement must be identified for the desired result, identify gaps, take action and then monitor the results.

Answer From Standards Expert…
Dr. Lawrence A. Wilson., Principal of Lawrence A. Wilson & Associates and US delegate to ISO/TC 176 for drafting of ISO 9001:2000 and ISO 9004:2000

Effective adj; effectively adv; effectiveness noun

Effectiveness = the extent to which planned activities are realized and achieve planned results. (Derived from 3.2.14 of ISO 9000:2000.)

The management of an organization must determine the nature, content and processes of a QMS that will be suitable and effective for its organization to achieve the planned results necessary for the success of the organization. Management must then totally and effectively implement such a QMS and all its processes. Once the system is fully implemented, management must maintain and measure the effectiveness of the QMS in achieving the expected/planned results.

"Planned activities" involve first identifying and defining customer and organizational requirements and then determining those activities and processes that are capable of being used to achieve the expected realization or transformation of requirements into expected results. The term "extent" in the ISO 9000:2000 definition implies that the planned activities, realization processes and achievement of the planned results may have some degree of variance from perfection.

Measurement of the level of effectiveness of any QMS can be achieved, in part, by analyzing the processes/results of the design and planning reviews, extent of process and procedural variance, level of product and service nonconformity, audits of process and product, success of corrective action and preventive action, management reviews, monitoring of customer satisfaction and service and field performance feedback.

Improvement of the effectiveness of a QMS in achieving the desired planned results can be accomplished by reducing or eliminating the causes of variances found during analysis. Increased effectiveness can be realized by:

1. Quantum changes identified and implemented as being essential for improvement (i.e., new equipment, facilities, processes, etc.)
2. Continual improvement where ongoing incremental improvements are identified and implemented (e.g., changes to procedures, training, tools, methods and materials).

QMS effectiveness should be continually increased when the performance of the system and its processes is measured, the data collected from the measurements are analyzed and an improvement is defined, implemented and verified. Such continual process improvements can be verified by the positive changes in the data collected.

Improvement in processes may also be achieved by other than variance reduction, such as through the reduction of resources required per unit of product, known as improved efficiency. Although ISO 9001:2000 does not address efficiency, many organizations also gain improvement by reducing per unit material, labor and time, all of which amount to reducing per unit cost.

A reduction in cost, however, must not result in a reduction in customer satisfaction or product and service performance. When done correctly, to the benefit of both the customer and the organization, improvements in both effectiveness and efficiency can be achieved to their mutual satisfaction (see ISO 9004:2000).

Proof that the organization is actually pursuing continual improvement is the responsibility of the organization’s management. Records of the improvement activities mentioned above should be regarded as essential by prudent management. Performance improvement must be a basic management objective, and upon which they are judged.

Answer From Registrar Expert…
Kevin Beard, Program Director, NQA, Inc

There are many different approaches used by organizations to define, develop and implement the structure of their QMSs, including process mapping, documentation, training, etc. But when running a business and serving customer needs, "effectiveness" of the QMS is the bottom line. We all have probably heard the criticism of ISO 9001:1994, that it promoted "consistency" within a QMS, which gave organizations the option to produce bad product "with consistency".

With ISO 9001:2000, the QMS is not merely a set of procedures that strive to produce consistency within a QMS, it is a management system that focuses on producing information regarding the effectiveness of the processes. The system also allows management to understand where problems exist and where there are risks to meeting customer needs. It is a management system where information is used to improve the effectiveness of an organization’s processes.

Effectiveness does not focus on whether a product meets a requirement. Effectiveness focuses on whether processes are achieving planned results and if QMS initiatives/activities are meeting expectations. The concept of effectiveness asks that managers develop performance measurement strategies and use the information to improve the organization’s processes. The key to effectiveness determination is information. The structured acquisition of information allows managers and process owners to analyze the performance of the various QMS processes of an organization to determine if a process meets their expectations and to identify process weaknesses requiring improvement initiatives.

While ISO 9001:1994 specified the concept of effectiveness in only a few places, an organization may have to think hard about how effectiveness applies to its management system strategies. In ISO 9001:2000, the focus on effectiveness is disseminated throughout various clauses/subclauses. To satisfy this, the biggest change an organization may need to make is to have management develop a strategy to address how all these disseminated facets of effectiveness work together and how information from the many sources within its QMS can be best used to determine the health of the QMS.

An organization should strive to understand how its operations are structured and how effectiveness can be applied to the organization’s various process categories:

• Realization Processes (the end-to-end production process—contract through delivery)
• Subprocesses (steps within the Realization Processes—purchasing, engineering, etc.)
• Support Processes (HR, infrastructure, quality assurance, etc.)
• Management Processes (business strategy, budgeting, resource allocation, etc.).

Once its process structure is understood, the organization should identify information sources and their relationship to each other, as well as impacts on other parts of the QMS. The organization can then determine which performance measures are important, not only for determining the effectiveness of that individual process, but for contributing to the determination of overall management system effectiveness.

Within ISO 9001:2000, effectiveness is not limited to one part of an organization’s QMS. Rather, it relates to almost all aspects of the organization’s business approach. While many organizations focus on the effectiveness of the Processes and Subprocesses that deliver the product, equal attention should be placed on the overhead costs and effectiveness of the QMS support processes. In breaking down an organization, costs are evaluated in many ways (i.e., sales/marketing costs, product delivery costs, overhead costs, etc.).

With ISO 9001 focused on the management system of the "organization", quite often the cost for implementing the system falls into the overhead column of the profit and loss statement. Much of the cost associated with a QMS does not directly contribute to the delivery of product, and understanding this highlights the importance of establishing effectiveness measures across the organization’s QMS. Management needs to pay attention to the effectiveness of infrastructure processes (IT, preventive maintenance, etc.), training, internal audits, corrective action procedures, etc.

As for process performance measures, Clause 8.2, Monitoring and Measurement, of ISO 9001:2000 establishes requirements for four QMS "areas"—monitoring of customer satisfaction, internal auditing of the QMS and monitoring and measurement of QMS processes and of product. Each of these areas has a unique focus within an organization’s QMS, and each can contribute to the evaluation of the QMS’s effectiveness.

When it comes to the measurement of Realization Processes, organizations quite often gather only information to evaluate the effectiveness of a subprocess (i.e., purchasing, manufacturing steps, etc.). However, an organization can make significant determinations of QMS effectiveness when it starts to relate performance measures information across its interrelated subprocesses. For example, the impact of a final product inspection problem is not limited to failure to ensure that good product gets delivered to customers. Information on a problem found with the final inspection process could also be used to determine the effectiveness of the in-process inspections, receiving inspection, supplier control approach, design review process, etc. This analogy can be applied to warranty returns and other significant events within an organization’s realization processes.

While much has been said about customer satisfaction, internal audits and product monitoring and measuring, "QMS process measures" (Subclause 8.2.3) frequently make the most significant contribution to determining QMS effectiveness. Subclause 8.2.3 seems like a catchall requirement in ISO 9001:2000, but the intent is to ensure that organizations look beyond individual product inspection data for performance measures to information and data from all sources within the QMS.

An example can be found in the area of support processes, where many of these QMS functions are treated as overhead (i.e., training, internal audits, etc.). In this case, organizations should strive to ensure that overhead-related QMS functions are effective and that nonproduct-related monitoring systems (e.g., internal audits, customer satisfaction) are providing value-added information to management regarding the effectiveness of the QMS.

Let me give you a case in point. As an auditor, I find that one of the most frequent causes of nonconformities is employee error, with the most frequent corrective action being to retrain employees. However, if retraining is the most common response dictated by an organization’s corrective action procedure, what does this say about the effectiveness of its training approach? Many examples of this type can be used not only to determine the ability or effectiveness of the corrective action procedure and internal auditing program to reduce repeat problems and provide system-based feedback to management, but to determine the effectiveness of the organization’s infrastructure-oriented groups as well.

How will registrar auditors determine if an organization has continually improved its QMS effectiveness over time? If an organization has developed good QMS (performance measurement) strategies, it should be a fairly straightforward process for an auditor to find the objective evidence of effectiveness. This should be evident in the quality records and metrics maintained by the organization. Whether the organization uses metrics, meeting minutes or other data gathering/evaluation methods, the key realization and QMS process records will be viewed to see how the organization over time monitors its own effectiveness and what improvement initiatives are taken based on the evaluation results.

While the focus of mid-level managers may be on the effectiveness of their Subprocesses or Support Processes, top management must keep an eye on the overall effectiveness of the QMS processes. Auditors can interview top management to evaluate the organization’s strategies towards effectiveness, improvement and initiative alignment. Then the various levels of management can be evaluated to assure alignment of improvement initiatives, which contribute to the overall improvement strategies with regards to QMS effectiveness.

Return to Top