This month's question
I have been tasked with evaluating and grading how efficient and necessary my organization’s quality department is. Do you have any advice on what criteria and tools to use to perform this task? How do you determine whether the quality program is efficient or “working”?
My answer focuses on evaluating an organization’s overall quality program, of which the quality functional group or department is a part.
Success of the department itself can be measured by the influence the department has on the rest of the organization—in particular, how well the entire organization buys into quality and improvement thinking. In this way, all employees essentially are an extension of the quality department. The quality department is not seen as a silo, but as employing experts and facilitators for implementing quality systems, tools, techniques and best practices everywhere.
To evaluate the efficiency or effectiveness of the quality program, there must be something against which to compare current performance. It is best practice to set quality objectives or metrics at the highest level in the organization. Like any objective, those characterizing quality should be SMART—strategic, measurable, attainable, realistic and time-based. They also should be diverse enough to cover all the traditional areas of cost of quality (COQ) measurement: prevention, appraisal, internal failure and external failure. If an organization is mature in its implementation of COQ, a numerical reduction in the COQ itself could be an overall quality objective.
Another input to setting quality objectives may be the categories of the balanced scorecard: financial, customer, internal business process, and learning and growth. In addition, in regulated industries such as medical, food and aerospace, a measure of compliance to standards and regulations should be included. In fact, many quality system standards contain statements that require setting quality objectives and ensuring the quality system achieves its intended results, which of course requires monitoring performance versus the objectives.
ISO 9001:2015 explicitly endorses the traditional plan-do-check-act method, taking the evaluation of performance versus objectives all the way through the continuous process improvement loop.
Considering all these inputs, here is a sample of areas to cover through quality objectives. Additional or different areas of focus may apply, depending on the organization. Objectives should be written in a clear format, such as “Improve performance in ‘Z’ from ‘X’ to ‘Y’ by ‘date.’” The specifics will contain increasing detail as the objectives are deployed to the lower levels of the organization.
- Financial metrics, including aspects of appraisal COQ.
- Overall COQ.
- Costs associated with inspection and test (a specific measure of the quality function or department).
- Revenue from new products.
- Costs associated with ship holds or backorders.
- Accurate, on-time shipments.
- Customer complaints about products or services.
- Recalls or near misses with respect to field corrective actions.
- Timeliness of any activity (actual work task time and calendar time).
- 100% complete and accurate execution of any activity.
- In operations, overall equipment effectiveness, which includes a measure of yield or process/product defects per million.
- Product nonconformances.
- Process nonconformances, which may be identified via an internal audit program and may include any business process, not just manufacturing processes.
- Implementation of a new or improved system by a certain date (such as change management, labeling, human resources information and supplier monitoring).
- Corrective and preventive action projects—timeliness, no repeat issues on solved problems.
- On-time completion of training.
- Employee retention.
- Survey results on employee engagement, satisfaction or buying into the culture.
- Compliance with a new or revised standard or regulation by a certain date.
- External audit nonconformances.
- Warning letters or other actions taken against the organization.
- Timeliness of corrective and preventive actions to audit observations.
An important quality system process may be the subject of many interrelated quality objectives. For example, customer complaints can be numerically counted as an external failure measure; timeliness of response back to the customer counted as an internal business process measure; complaint process internal audit nonconformances counted as a compliance measure; and on-time training completion by complaint department personnel monitored as a learning and growth measure.
After quality objectives for the organization are set, they can be cascaded down to the business units, to each functional group and eventually to individuals via the performance management process. The effectiveness of the quality system is tracked via management reviews, again at various levels in the organization escalated from site or facility to business unit to top-level executive.
Results versus objectives for each process can be represented on scorecards and dashboards, and graded by a green-yellow-red system. Green means the process is performing fine versus objectives, yellow designates concern and management awareness is needed, and red indicates correction and management input are required.
Finally, there is a trend for quality system standards to now require applying a risk-based approach to the control of the appropriate processes needed for the quality management system (ISO 13485:2016, for example). Historically, a desired outcome of management review has been to record a confirmation of the suitability, adequacy and effectiveness of the quality system and its processes. The confirmation would be based primarily on performance versus objectives. However, the intent of the risk-based approach goes beyond, extending to the task of using quality data to monitor whether any new risks have been created or any existing risks have been modified. There are all kinds of practical implications.
Organizations should be ready to modify their quality objectives depending on the results of risk analyses. Unacceptable or increasing quality system risks may require new initiatives, objectives or risk controls to adequately maintain the quality system going forward.
This response was written by Scott Laman, senior manager, quality engineering and risk management, Teleflex, Reading, PA.