QuEST Forum’s TL 9000 Measurements Handbook Release 4.0
by Sandford Liebesman
The Quality Excellence for Suppliers of Telecommunications (QuEST) Forum is the global society dedicated to the quality of telecom products and services. The forum has developed and maintained a single common set of telecom quality management system requirements and measurements built on existing industry standards and practices.
New versions of the TL 9000 requirements and measurements handbooks were issued in 2006.1,2An earlier Quality Progress column described the changes in the requirements handbook,3while this column will deal with the measurements one.
The goal of the measurements system is to help the telecom industry provide faster, better and more cost effective services to its customers. The TL 9000 measurements are sent to the QuEST Forum database by organizations that comply with the standard.
The measurements system facilitates analysis against industry benchmarks and provides the basis for objective product or supplier evaluations. The result is an improvement in the suppliers’ processes, which makes the industry more efficient.
The QuEST Forum continually updates its handbooks, with Release 4.0 being the third revision. Release 3.0 was issued in 2001 after ISO 9001 was revised.4 Release 3.5 came out in 2003 to simplify some rules related to how measurements are counted and to expand outage and software measurements, eliminate complexities of dual reporting and add product categories.5
A major goal of Release 4.0 was to expand measurement examples to cover more diverse product categories. Examples and appendixes were moved to the forum website6 to improve the forum’s support of update flexibility. New examples can be quickly added based on frequently asked questions or data submission failures, and they can support product or service alerts.
Other significant changes cover general measurement requirements and common, outage, hardware, software and service quality measurements.
July 1, 2007, was the start date for use of Release 4.0. Changes in outage and software measurements are particularly significant. To comply, organizations should understand the complexities and detailed descriptions of the requirements.
General Measurement Requirements
Release 4.0 expands the TL 9000 administrative responsibilities to provide internet accessible systems for documenting TL 9000 registrations. This includes the registration steps; status of all registrations; implementation statistics updates on registration requirements; information on the QuEST Executive Board, workgroup leadership and member organizations; and contact information for accreditation bodies and registrars.
The systems also cover management of registration parameters; management and tracking of submission history; accreditation bodies and registrars’ approval and review of client information; and use of techniques to obtain the required information from the data.
Release 4.0 specifies changes for the registration management system. In addition, the release delineates the processes for the delivery of the data submission receipts and performance reports and for specification of the customer base, which must include all customers for which data are available for accurate measurements. The information on the customer base must be retained for potential data resubmission. The monthly submissions must be made within seven weeks of the monthly close.
Some specific changes of the data submission process allow the aggregating of data within only a single product category, require data for new releases by the time of general availability and require data for new products within six months of general availability.
Changes to Four Common Measurements
Changes were made to the four common measurements that apply to all products and services: number of problem reports, problem report fix response time, overdue problem report fix responsiveness and on-time delivery.
Problem reports are classified as critical, major or minor:
- Critical reports occur when the conditions severely affect the primary functionality of the product and require immediate corrective action.
- Major problem reports occur when a product is usable but a condition exists that seriously degrades the product operation, maintenance or administration and requires attention during predefined standard hours to resolve the situation.
- Minor problems are of lesser severity and do little or nothing to impair the function of the system.
The descriptions of outage measurements are organized into three distinct sections: service impact outage measurement, network element im-pact outage measurements (identified as SONE), and engineering or installation caused outage measurements.
Detailed changes to the counting rules include combining total and partial outages for SONE and lowering the generic threshold to 15 seconds. Outage measurements are classified into product attributable, customer or external attributable.
Product attributable outages are triggered by one or more of the following:
- The design, hardware, software, components or other parts of the system.
- Scheduled outage necessitated by the design of the system.
- Support activities performed or prescribed by an organization, including documentation, training, engineering, ordering, installation, maintenance, technical assistance, and software or hardware change actions.
- Procedural errors caused by the organization.
- Failure of the system to provide the necessary information to conduct a conclusive root cause determination.
Customer attributable outages are due to the following customer equipment or support activities:
- Customer procedural errors.
- Office environment problems due to power, grounding, temperature, humidity or security problems.
- Refusal or neglect of the customer to provide access to the necessary information for the organization to conduct root cause determination.
External attributable outages are caused by natural disasters such as tornadoes or floods. Other outages might be caused by third parties not associated with the customer or organization, such as commercial power failures or third-party contractors not working on behalf of the organization or customer.
Only minor changes have been made to hardware measurements, which include an early return index, one-year return rate, long-term return rate and the normalized return rate of units during the one-year return rate period. These measurements apply only to any system comprised of field replaceable units (FRUs), a system which itself is an FRU or the individual FRUs.
Software measurements track the effectiveness of an organization’s software fix process by measuring the ratio of defective fixes to the number of fixes delivered.
The measurements also track maintenance efforts associated with the deployed software by measuring the incidence of customer-found software problems associated with the products developed by the organization. This section of the handbook had a major rewrite.
Software measurements provide an understanding of the quality of software releases and the quality of software fixes, installation and maintenance efforts of software releases, and the risk of introducing a software fault via a defective fix.
The measurements and maintenance cover the activities to correct defects in generally available releases and are independent of the delivery administrative unit (patch, file, maintenance release, dot release or fix release). The software measures count the quality of independent fixes within the administration unit but not the quality of the administrative unit.
The treatment of firmware (a computer program that is also a vital part of a piece of hardware) is based on how it is maintained in the field. Firmware that can be changed by means of a download is treated as software, and all applicable software measurements apply. If changes require rotation or replacement of hardware, use of specialized equipment for field installation of the new firmware or returning the circuit pack to a repair facility, the firmware is treated as hardware, and the software measurements do not apply.
The first measurement is software fix quality (SFQ), which is used to assess the effectiveness of an organization’s software fix processes. It counts the number of updates and patches as fixes. SFQ is the percentage of software fixes determined to be defective. This measure is used to evaluate the percentage of defective fixes with a goal of minimizing customer risk of failure when introducing fixes to an in-service generic software release. Software fix quality equals the number of defective fixes divided by the total number of applied fixes.
The second measurement is software problem report measurement (SPR). Its purpose is to track the software problems that are found and reported by customers.
The problem reports included in SPR are a proper subset of those in the number of problem reports measurement, but the measurements are tracked and reported separately to focus effort on addressing the software component of these problem reports.
SPR is the measurement used to evaluate the number of customer originated software problem reports and indicates the software quality of the product delivered during the operating life cycle of the product.
SPR is intended to stimulate ongoing improvements and reduce the number of software problem reports, associated costs and potential revenue losses.
Service Quality Measurements
Service quality (SQ) is a measure of the performance of service transactions. SQ is based on the ratio of the number of defective service transactions to the total number of service transactions.
Common measurements and outage measurements also contain measurements associated with service, namely problem reports, fix response time, overdue problem reports, on-time service delivery and engineering/installation caused outages. SQ is the ratio of the number of defective service transactions to the total number of service transactions.
- TL 9000 Quality Management System Requirements Handbook Release 4.0, QuEST Forum, 2006.
- TL 9000 Quality Management System Measurements Handbook Release 4.0, QuEST Forum, 2006.
- Sandford Liebesman and John Walz, “New Version of TL 9000 Released,” Quality Progress, March 2007, pp. 74-76.
- TL 9000 Quality Management System Measurements Handbook Release 3.0, QuEST Forum, 2001.
- TL 9000 Quality Management System Measure-ments Handbook, Release 3.5, QuEST Forum, 2003.
- QuEST Forum, www.questforum.org.
SANDFORD LIEBESMAN had more than 30 years of experience in quality at Bell Laboratories, Lucent Technologies and Bellcore (Telcordia) before becoming a consultant. He is an author of the books TL 9000, Release 3.0: A Guide to Measuring Excellence in Telecommunications, second edition, and Using ISO 9000 to Improve Business Processes. Liebesman, a fellow of ASQ, is a member of ISO technical committee 176 and the ANSI Z-1 committee on quality assurance.