TL 9000An Update On the Metrics Process
Validation of training, registration and metrics completed
by Sandford Liebesman
A great deal has happened since Gene Hutchison and I described the launching of the telecommunications industry quality initiative, TL 9000, in the December 1998 issue of Quality Progress.1
TL 9000 is a sector specific derivative of the ISO 9000 quality management standard. It was developed and is managed by the Quality Excellence for Suppliers of Telecommunications (QuEST) Forum, which represents both service providers and suppliers. The QuEST Forum Web site is at www.questforum.org.
Recent progress includes the following:
* Sanctioned training courses were developed by the two authorized trainers, STAT-A-MATRIX and Excel Partnership Inc.
* An accreditation/registration process was developed with the help of the Registrar Accreditation Board, the Standards Council of Canada and 12 registrars.
* A metrics gathering process was developed and tested to ensure feasibility.
* A pilot project was completed to validate the training, registration and metrics processes. Eleven companies completed the registration process and received 16 certificates.
The metrics process
The unique difference between TL 9000 and current ISO 9001 practice is TL 9000's use of metrics in continuous quality improvement. These metrics are used to determine the quality level of products and services. Examples are circuit pack return rate, number of problem reports and software update quality.
Extending ISO 9001 in this direction is expected to result in an industry drive to improve overall quality. Certainly, when suppliers know where their products stand relative to those of their competitors, and their customers know the quality level of best in class, there will be a drive by all to improve.
The metrics implementation process
The supplier gathers the data on a monthly basis and sends three months of data to the University of Texas at Dallas (UTD) for inclusion in the industry database. A double blind system ensures that the data are secure and not available to competitors and other unauthorized persons.
UTD was selected because of its longstanding role in telecommunications engineering education. It's expected that UTD will also provide insight into benchmarking industry quality problems and identifying quality trends and solutions to problems.
The process works as follows:
* When a supplier has gathered the data, a notification is sent to QuEST Forum's administrator, ASQ, which provides encryption keys back to the supplier to be used in sending the data to the industry database.
* The supplier encrypts the data and sends them to the data repository at UTD. Personnel transfer the data from the receiving server into the industry database. At least three individuals must be present during the transfer to ensure security of the data.
UTD has complied with BS 7799,4 the information security management requirements document developed by the British Standards Institute (BSI). UTD is the first U.S. organization to be certified to this document by BSI.
The statistics published by UTD will identify the mean (or median) standard deviation (or range) and best in class for each product category that has at least five inputs. The data will be published on the QuEST Forum Web site at www.questforum.org.
Eleven metrics in four categories have been approved. The categories and metrics are as follows:
1. Hardware, software and services: five metrics including number of problem reports, problem report fix response time, overdue fix responsiveness, system outage measurement and on time delivery.
2. Hardware only: return rates.
3. Software only: four metrics including quality of emergency fixes between software releases (corrective patches), quality of emergency feature improvements (feature patches), quality of software updates and failures of scheduled new software releases (aborts).
4. Services only: service quality.
A major issue during the development of these metrics was the identification of normalization factors, which allow comparison of similar products with different characteristics. For example, for switching systems, the normalization factor for return rate is "returns per 10,000 terminations per year." Returns are units that are removed and returned for repair or disposition or components that are removed and replaced. Thus products with a large number of circuits attached per circuit pack are measured fairly against those with fewer circuits attached.
Another issue was a result of a requirement from some service providers that the suppliers send them reliability and quality measurements for telecommunications systems (RQMS) metrics as defined in the Telcordia Technologies document GR-929-CORE.5 In these cases the supplier may provide the RQMS metrics in place of the TL 9000 metrics.
Other metrics and indicators are under consideration but have not been approved. Indicators are a separate category of measures that are used to flag potential cost, schedule, productivity and quality issues, but are not reportable to UTD.6
The service providers have an important role with respect to certain metrics. They must provide data necessary to calculate them. If the data are not forthcoming, the supplier is excused from providing the associated metric. For example, service providers must report problems and normalizing information to suppliers so that suppliers can calculate the number of problem reports by product category and severity.
The registrar has an important role in assuring compliance to the metrics requirements. "This includes assuring data validity and integrity, verifying [that] the benchmark metrics have been reported within the required time frame and [assuring] that any metric-specific nonconformance is resolved in a timely manner."7
In addition, because of the requirement that "objectives for quality shall include targets for the TL 9000 metrics defined in the TL 9000 Quality System Metrics handbook,"8 the registrar has a role in assuring this use of the metrics. Finally, the registrar must assure that the supplier collects and analyzes customer satisfaction, field performance and service performance data, some of which are metrics data.9
Product and service categories
The following are the product and service categories defined for the metrics implementation:
* Switching--broadly defined to include packet or circuit switched architectures.
* Signaling--five basic categories of signals including supervisory, information, address, control and alerting.
* Transmission--connects the customer to the central office, providing access to the interoffice network.
* Operations and maintenance--management, upkeep, diagnosis and repair equipment.
* Common systems--shared equipment to support network elements.
* Customer premises--equipment installed beyond the network demarcation point.
* Services--includes installation, engineering, maintenance, repair, call center and other support services.
* Supply chain--includes components, contract manufacturers and original equipment manufacturer suppliers.
One QuEST Forum initiative was the creation of a pilot program to validate TL 9000 and its associated activities. The goals of the program were to provide feedback on the bottom-line value of the 83 TL 9000 requirements that are not in ISO 9001, the processes that form the metrics reporting system, the processes used to manage the registrar accreditation and supplier assessment/certification system, and the sanctioned training courses.
Feedback was accomplished through weekly conference calls, monthly meetings, documented lessons learned, case studies and commonly asked questions. The conference call minutes and commonly asked questions were posted on the members only section of the Quest Forum Web site.
Sixteen organizations from 11 companies participated in the program in four categories:
* Hardware, software and services: Fujitsu Network Communications, Lucent Technologies (Illinois, the Netherlands and Poland), Marconi Communications (Access) and Nortel Networks.
* Hardware only: ADTRAN, Marconi Communications (Power), NEC America, Nortel Networks, Pirelli Cables and Systems LLC, Siecor (North Carolina), Siecor (Texas) and Tellabs (Texas).
* Software only: Motorola, Nortel Networks and SBC (California).
* Services only: Nortel Networks.
Each pilot organization had to train its auditors and implementers, perform a gap analysis, develop processes for collection of the metrics and submit the metrics data to UTD. The organizations that completed the process and were certified were not allowed to advertise the fact until the end of January 2000. They were honored at the January 2000 Forum meeting in Dallas.
The QuEST Forum authorized the following ongoing activities:
* The requirements are being aligned with ISO 9001:2000.
* The metrics are being reviewed for possible changes and additions.
* The Business Excellence Acceleration Model is being developed to provide industry guidelines for continuous improvement through linkages to awards criteria such as the Malcolm Baldrige National Quality Award.
* The QuEST Forum is actively increasing its global membership.
Material in this article was excerpted from the forthcoming Quality Press book TL 9000: A Guide to Measuring Excellence in Telecommunications by Alka Jarvis, Ashok Dandekar and the author of this article. ASQ Quality Press plans a December publication date for the book.
1. Eugene Hutchison and Sandford Liebesman, "Telecom Industry Forum Working Toward TL 9000, an ISO 9000-Based Standard," Quality Progress, December 1998, pp. 14-19.
2. The QuEST Forum, TL 9000 Quality System Requirements, Book 1, Release 2.5 (Milwaukee: Quality Press, 1999).
3. The QuEST Forum, TL 9000 Quality System Metrics, Book 2, Release 2.5 (Milwaukee: Quality Press, 1999).
4. British Standards Institute, BS 7799: 1999, Information Security Management, Part 1: Code of Practice for Information Security Management and Part 2: Specification for Information Security Management (London: British Standards Institute, 1999).
5. Telcordia Technologies, GR-929-CORE, Reliability and Quality Measurements for Telecommunications Systems (RQMS), December 1998.
6. Henry Malec, "TL 9000 Database Repository and Metrics," The Informed Outlook, June 1999, p. 4.
8. The QuEST Forum, TL 9000 Quality System Requirements (see reference 2), p. 4-1.
9. Ibid, pp. 4-40 and 4-41.
SANDFORD LIEBESMAN is ISO manager for quality and customer satisfaction, Lucent Technologies, Murray Hill, NJ. He earned a doctorate in operations research from New York University. Liebesman is a Registrar Accreditation Board certified lead auditor and author of Using ISO 9000 To Improve Business Processes, published by the AT&T Customer Information Center.