Weighing cost of quality
Q: In agile systems development pursuits, are the appraisal and failure costs of a greater magnitude or proportion of the overall cost of quality (CoQ) than in nonagile pursuits? In agile development pursuits, what are the preferred proportions of prevention costs vs. appraisal costs vs. failure costs relative to the overall cost of quality?
Wallace J. Robinson
A: It’s been known since the founding of scrum and extreme programming—later called "agile methods"—that these methods have a lower CoQ.
In simplest terms, CoQ is the total costs of achieving conformance to requirements for a product or service across its entire life cycle (from concept to retirement). There are generally four major quality engineering activity expenses that contribute to CoQ:
- Prevention costs: the cost of preventing defect introduction before development (training and root cause analysis).
- Appraisal costs: the cost of independently evaluating products and services before delivery (inspections and testing).
- Internal failure costs: the cost of fixing defective products and services before delivery (rework, retesting and scrap).
- External failure costs: the cost of fixing defective products and services after delivery (warranty, repair and recall).
The expenses associated with each type of cost grow in order of magnitude from stage to stage. For instance, prevention costs $1 per defect. Then, appraisal costs $10 per defect, internal failure costs $100 per defect and external failure costs $1,000 per defect. External failure costs may be as high as $10,000-$50,000 per defect. These basic economic ratios have been understood since the 1970s and were used to justify the creation, proliferation, use and regulation of traditional, linear systems and software engineering methods.
Quality engineering economists often estimated the total number of defects created during the development of an average product or service (such as 10,000 defects). Then, they could estimate the cost of quality, based on the ratio of investment among these four categories. That is, emphasis on earlier quality engineering activities reduced CoQ, while emphasis on later quality engineering activities increased CoQ. Therefore, the emphasis was on the earliest possible quality engineering activities (defect prevention activities), although a balanced portfolio using all four types of quality activities and cost was often sought. In spite of these models, most firms invested the majority of their activities in the latter two categories (failure costs).
Developing a culture of quality across organizations is quite difficult and expensive. This is often referred to as world-class quality, and few organizations have achieved it. It requires a handsome investment of resources over a long period of time. It costs millions of dollars over decades in conventional techniques. Today, some firms focus on the last category (external failure costs), and economic models show it may be more cost effective to recall a product than to "do it right the first time," as W. Edwards Deming suggested.
Organizational change is required to institutionalize early quality engineering activities or form a quality culture. Organizational change is quite difficult and involves shifting deeply held psychological beliefs and human behaviors. It requires decades of time, numerous initiatives and millions of dollars to achieve success. The level of difficulty increases exponentially with complex activities, organizations or antithetical behaviors. The slightest change to organizations can be egregiously difficult.
Proponents of traditional methods like to point out that academic textbook approaches emphasize the use of a balanced portfolio of quality engineering activities. In fact, traditionalists tend to refer to the earliest and lowest-cost defect prevention activities as the answer to the quality dilemma. Unfortunately, instituting these practices is difficult, expensive and time consuming. Traditional methods come with portfolios of thousand-page models recommending hundreds of activities, metrics and artifacts.
Agile methods aren’t as well entrenched, don’t come with dozens of thousand-page manuals, and have few activities and artifacts. Their creators understood why traditional projects failed and incorporated practices to overcome these problems. Agile methods dramatically reduce scope and sharply align it with business objectives. They develop a small, highly prioritized list of the top 20% of customer needs to achieve 80% of the business value. They’re implemented in small batches until tacit knowledge is uncovered. Buyers and suppliers communicate, cooperate and collaborate in an open, empowered and transparent way using lightweight, disciplined processes and artifacts. Thousands of automated tests are run every few minutes using free and open-source tools. Frequent defect prevention is performed on a daily, weekly and monthly basis.
More specifically, there are 18 practices that help make agile CoQ lower than traditional methods:
- Business alignment. One of the highest priorities in agile methods is to align agile projects directly with business or mission objectives. Agile projects do not exist in a vacuum as autonomous, self-serving infrastructure groups, or for the greater good. Objectives are mapped directly to portfolios, epics, features, themes, user stories and technical tasks. Business or market value is attached directly to product and service requirements. Agile teams are right-sized to satisfy specific organizational needs and priorities, and teams can track the organizational value they create on a task-by-task basis using earned business value. Agile team members know exactly how their daily tasks align to organizational objectives and the value they create every day. This is an explicit form of defect prevention and waste reduction that does not exist in traditional methods and projects.
- Requirements prioritization. A major practice in agile methods is to identify, focus on and deliver only the highest priority business needs, objectives and requirements. That is, agile teams recognize that only a small fraction of product and service requirements are needed and have any business value at all. Therefore, agile teams identify a small set of high value-adding requirements, quantify the value of these requirements in terms of business and mission value, and deliver these first. More than 80% of the business value is achieved by implementing less than 20% of the highest priority requirements. In doing so, agile teams can help organizations rapidly increase their mission and business effectiveness, efficiency, market share, revenues, profits and other objectives. This is a major form of defect prevention not typically performed in traditional methods.
- Customer collaboration. A key practice in agile methods is early, continuous and close customer collaboration. Buyers and suppliers work together as one team throughout development to ensure conformance to requirements. Business and mission objectives are shared, along with responsibility for success and failure. Communication is based on rich, high-context, face-to-face communications, rather than lawyers, contracts, liaisons, specifications, documents and other inefficient and error-prone means. Customer needs exist in the form of tacit knowledge. Agile teams discover needs with rich communications, quickly develop solutions and demonstrate physical models to create business value. This is a powerful form of low-cost defect prevention using real solutions that do not exist in traditional methods based on contracts, processes and documentation.
- Small scope and batches. A major practice in agile methods is to reduce scope, requirements, team and batch size. These are critical success factors that have been known for decades. More than 80% of requirements are often unneeded, wasteful and result in defects. Large batches increase risks, costs and timelines. They increase communication paths, queues and delays and proliferate defects. Technologies become obsolete, and the few projects that are completed are often merely discarded. Agile methods reduce the size of the scope, timelines and costs, develop value-adding solutions in smaller increments to satisfy customer needs, and produce vastly smaller numbers of defects. Reducing scope and batch size alone is a major defect prevention practice that doesn't exist in traditional methods, which aim to scale up to the largest scope, cost, timeline, risk and complexity possible.
- Early, fast and frequent customer feedback. Agile methods focus on early, fast and frequent customer feedback throughout development. This occurs from the beginning to the end. This is the only way to solicit customer needs, test and validate your solutions, achieve business and mission value, and, ultimately, satisfy customers. This solves problems by teasing out hidden, inexpressible customer needs that exist as tacit knowledge, quickly manifesting and validating smaller batches of inexpensive work in process as value-adding solutions, and improving products and processes in hyper-fast cycles to rapidly converge upon customer solutions. This uses early and continuous communications to rapidly realize high business value. This is a defect prevention practice not found in traditional methods that maximize CoQ during long periods of time.
- High-context communications. Agile methods rely on rich, high-context communications throughout a project, whereas traditional methods often discourage these communications. This exacerbates poor performance because customer needs exist as tacit knowledge that emerges through human communications. In turn, inordinate levels of scope, time and cost risk, along with poor quality, result. Rich, high-context communications in agile methods is a key defect prevention practice often legally prohibited in traditional methods.
- Transparency and openness. Agile methods promote transparency and openness through cooperation, collaboration and communication. It also comes from small teams working together on a daily basis. Agile methods rely on face-to-face communications, as well as honesty, empowerment and information sharing. This enables everyone to sync and tailor their behaviors, activities and alignment to achieve business or mission value. Information is shared on walls, white boards, wikis, workflow tools and electronic artifacts. Nothing is hidden. There is no division of labor and data are not hidden in desk drawers, complex schedules, processes and documentation, or withheld by cliques of managers and developers. This is a major defect prevention activity.
- Small, disciplined processes. Agile methods use small, disciplined and structured processes. These include eliciting and documenting requirements and acceptance criteria, evaluating risk and business value, prioritizing and scheduling, day-to-day coordination, customer demonstrations and process improvement. This also includes configuration management, quality assurance, testing and other support disciplines, such as architecture, documentation, user experience and security engineering. Automated system administration, delivery and deployment are also important. There is a process for all activities, and most of them have been automated using low cost, free and open-source software. All of these practices are key defect-prevention activities improving productivity and quality by 10, 100 or even 200 times more than traditional methods.
- Small cross-functional teams. Agile methods rely on using one or more small cross-functional teams. Agile projects focus on smaller, high-priority scope statements, requirements, architectures, designs and solutions. They also exploit smaller batch sizes, and early and continuous validation of requirements throughout the day, week, iteration, release and project. Therefore, using smaller teams goes hand-in-hand with agile methods more than any other paradigm based on larger processes, projects and complexity. There is more face-to-face communication, trust, cohesion, teamwork, cooperation and collaboration. Silos and barriers between organizational functions are dissolved, integrated and automated. This combines to dramatically increase productivity, quality, customer satisfaction and business value. This is a key defect prevention activity.
- Teamwork and cooperation. Agile methods also rely on teamwork and cooperation to rapidly create high-quality, value-adding solutions. This is important on technology-intensive projects, which are complex by nature and require advanced skills, education and experience. Teamwork also applies to routine administrative or operations projects, as well. Trivial problems are complex and may take months or years for individuals to solve alone. Moreover, individualized solutions may not be optimal, efficient or effective, and may even be defect-prone. Combine experts and novices together, and complex problems become simple and efficient, and high-quality solutions can be created in a matter of minutes or hours. Even a mediocre team can do great things. This is a key defect prevention activity.
- Small, emergent design. Agile methods focus on small, emergent architectures, designs and solutions. Agile teams create just-enough design to solve a small batch of high-priority, value-adding customer needs. Agile teams develop, validate and deliver functional and operational products and services rapidly and efficiently. Most customer requirements exist as hidden, inexpressible tacit knowledge. Small emergent designs enable agile teams to deliver solutions to customers in a matter of hours, days or weeks. Early conceptual designs help developers rapidly converge on valid solutions. Smaller designs enhance system performance, have fewer defects and are more reliable. It is a key defect prevention activity not found in traditional methods that aim for more expensive, wasteful and unreliable designs with a higher CoQ.
- Flexible technologies. Agile methods emphasize use of flexible technologies to rapidly deliver value-adding solutions. Teams identify, exploit and use proven higher-level abstractions instead of creating unique, one-off unprecedented designs, one line of code at a time. These abstractions may be proven design patterns, frameworks, free and open-source software, commercial solutions, web services, service-oriented architectures and cloud services. This also may include other preexisting, dynamically reconfigurable and highly tailorable e-commerce components. This applies to collaboration, project management, development and solutions. This enables new employees to achieve high levels of productivity and quality. This is a major defect prevention activity not found in traditional methods, which require advanced skill, education and experience.
- Frequent multitier improvement cycles. Agile methods are based on multitier improvement cycles, which rapidly boost productivity and quality. Requirements are developed in small teams that make improvements every few hours. Daily meetings identify high-priority issues for immediate resolution, which is the second major improvement loop. Customer demonstrations and internal retrospectives are held every one or two weeks. This is the third and fourth major improvement loop. Post-mortems are held for 90-day releases and project end. This is the fifth and sixth level of improvement. These improvement cycles are multiplicative and help rapidly boost productivity and quality. Thousands or millions of automated tests also are run each day leading to more improvements. These are major defect prevention activities simply not found in longer-term, traditional methods.
- Fully automated low-cost environment. Agile methods use low-cost development environments, tools and automation. These may be free and open source, or commercial tools. There are integrated environments with project management, workflow, development and testing tools and support. Microsoft Visual Studio for .NET or Eclipse for free and open-source software-based Java development are examples. This includes collaboration tools for virtual teams, wikis for information sharing and version control tools. Build, integration, unit test, acceptance test, graphical user interface, and security testing tools are helpful. Static code analysis plug-ins can be used to run millions of low-cost automated inspections. All of this combines to help teams achieve high levels of productivity and quality. These are important defect prevention activities not found in traditional methods.
- Fully automated configuration management (CM), build and testing. Agile methods use fully automated CM, builds and tests. They use lightweight, structured and disciplined processes. They are based on open and transparent communication. Artifacts belong to the entire team. They are kept in an automated environment including version control. This is egoless programming at its finest, and artifacts are seen by everyone. When the code is checked in, it is automatically compiled, built, tested and delivered within seconds. Millions of tests are automatically executed in seconds at a fraction of traditional CoQ. This is a major defect prevention practice in agile methods, whereas traditional methods often involve performing a few manual code inspections every year and a late manual "big bang" integration testing every five years to kill projects.
- Early and often continuous integration. Agile methods use continuous integration early and often. Teams implement code in small batches to reduce feedback cycles and tease out tacit customer needs. This is not just a one-time automated testing event, but it is done every few minutes from the first to last project hour. Teams pull a high-priority customer need with the highest business value from the requirements queue. They create a small design, implement it and check it into version control. Automated tools take over to compile, build and integrate the code, and run millions of tests. Developers know the results within seconds, and customers can instantly validate solutions. No more late, big-bang integration to kill the project. This is a major defect prevention activity shortening feedback and improvement cycles to a tiny fraction of traditional CoQ.
- Fully automated, multilevel testing. Agile methods use fully automated, multilevel testing—not just unit testing. As customer needs are identified, acceptance test criteria also are developed to satisfy customer expectations. Developers must use their experience to identify unstated performance, security, usability, quality, reliability, availability and other non-functional requirements. Independent test, quality, reliability, user experience, certification, accreditation and regulatory representatives should identify their criteria and tests for developers to use. As developers implement requirements and check in source code, millions of automated tests representing all of these disciplines must be executed. This is a key defect prevention activity within agile methods, and failure data can be fed back into the process and classified within seconds to instantly improve the process.
- Fully automated deployment. Agile methods use fully automated deployment. Static and dynamic tests from multiple levels, disciplines and regulatory bodies are run, and electronic documentation is generated. Images are created with deployment packages and manuals in the form of source code comments and other electronic online help. Code is deployed to IT operations groups that must run enterprises. Code may be deployed instantly to field devices, such as spacecraft, aircraft, weapons, automobiles, ships, autonomous aircraft, personal computers, laptops, networks, smartphones, wearable devices, medical devices, businesses, factories and any imaginable equipment. This is a key defect prevention activity in agile methods, which speeds delivery from years and decades to hours, minutes and seconds at a fraction of traditional CoQ.
Agile methods have few downsides. They have the repeatability, definition, quantification and optimization of traditional methods without a library of manuals, process areas and artifacts. They eliminate most manual processes by running automated tests at a fraction of traditional CoQ and don’t require years of expensive training. Done right, they eliminate debilitating big-bang organizational change. They are lean, just-in-time, have low work in progress, focus on high-priority customer needs, and can be used to meet scope, time and cost constraints.
Even novices using agile methods experience great improvement in terms of time, cost, quality, risk reduction and customer satisfaction. The rewards are greater if you invest more time implementing most agile practices. Projects that use more of these agile practices experience 100 to 200 times the level of productivity and quality improvement typically found by using traditional methods. Employee morale increases, expensive attrition comes to a screeching halt, and customer satisfaction, trust, loyalty and retention exponentially increase, along with buyer and supplier business value.
Remember that it is possible to fail with agile methods. Ignoring advanced practices and principles such as fully automated testing, continuous integration and continuous delivery is the fastest way to fail. Remember that agile methods are a mindset, value system and set of behaviors similar to Deming’s plan-do-check-act cycle.
Sticking to basic backlogs, iteration plans, daily standups, demonstrations and retrospectives instead of advanced agile principles is also a quick way to fail. You also can revert to debilitating traditionalism because you don’t know how to get coaching to achieve productivity and quality levels at 150 times the level of traditional methods at a fraction of the CoQ.
Agile methods consist of a plethora of sophisticated human behaviors as opposed to voluminous guidelines consisting of dozens of process areas and work product artifacts. This can pose difficulties for proponents of agile methods because accepting, exhibiting and practicing small numbers of agile behaviors may work sometimes, but not other times. They also pose a significant dilemma as adding larger numbers of agile behaviors increases the time, cost, complexity and risk associated with organizational change.
Are agile methods—with 18 inexpensive defect prevention-focused practices—a "deliver it now, fix it later" approach with a high CoQ? Or, do complexity inducing, queue freezing manual, appraisal-based traditional methods have a higher CoQ? Some early adopters used basic agile practices, saved up all testing for yearly releases, and created a ton of technical debt and defect backlogs. More agile groups are using advanced agile principles to ensure the lowest possible CoQ in seconds at the fraction of traditional CoQ. In essence, you can empty the defect backlogs with more sophisticated agile practices.
David F. Rico
Software process and
quality improvement expert
Employee qualification audits
Q: I am a quality assurance good practices auditor being told I cannot perform an employee qualification audit because curriculum vitae (CV) and résumés, job descriptions and training records are confidential. How do I prove the organization has qualified personnel? Is this also true for an internal or first-party audit to verify the qualifications of employees to ensure they meet U.S. Food and Drug Administration and International Conference on Harmonization (FDA/ICH) guidelines?
From ASQ’s Ask the Experts blog,
A: Maintaining confidentiality can be a major concern for employees, organizations and auditors. For this reason, the review of employee files containing private data, such as social security numbers, banking, personal contact or other sensitive information, should be avoided if possible. This not only maintains employee privacy, but also reduces the auditor’s exposure to potential liabilities.
So how can the auditor verify employee qualifications and experience? Remember that there is no requirement for an auditor to review job applications, CV and résumés, or other confidential information.
It’s the organization’s responsibility to provide the auditor with objective evidence showing it has established job descriptions for employees performing work activities that affect the quality of the product or services to be provided to the customer.1 This includes providing evidence that the employee’s qualifications, skills, education and any applicable certifications have been verified to meet job description requirements, or the need for training has been established to ensure job description requirements are met.2
Similarly, FDA/ICH audit criteria, which includes 21 CFR Part 820, requires manufacturers to have established procedures and documented training to ensure the competency of personnel.3 The organization is responsible for to ensuring the competency of personnel (education, background, training and experience), not the auditor. The auditor’s role is specific to verifying by review of objective evidence, observation and auditee interviews that requirements, as defined by 21 CFR, Part 820 or other applicable audit criteria, are met.
The auditee is not obligated to provide an auditor access to résumés, CVs or documents that contain confidential information. Some auditees may opt to provide an auditor with a record of the auditee’s review and approval of an employee, such as a qualification checklist or supporting training records as evidence that an employee was found to meet defined job description requirements.
Job descriptions and training records may be considered proprietary, but they are seldom considered private or confidential because they don’t contain personal information. Some organizations may require a nondisclosure agreement be signed to protect propriety information, such as engineering data, drawings or training materials related to product realization processes.
A record of an organization’s review and verification of employee qualifications should be readily available. Likewise, training and applicable certification records should be available to provide objective evidence that qualification and competency requirements have been met.4,5
Aston Technical Consulting Services
- International Organization for Standardization, ISO 9001:2008—Quality management systems—Requirements, clause 6.2.1.
- International Organization for Standardization, ISO 9001:2008—Quality management systems—Requirements, clause 6.2.2, sub., a, b and c.
- U.S. Food and Drug Administration, Code of Federal Regulations Title 21 (21 CFR), Sub. Part B 820.25, Personnel.
- International Organization for Standardization, ISO 9001:2008—Quality management systems—Requirements, clause 6.2.2, sub., e.
- U.S. Food and Drug Administration, 21 CFR, Part 820.25, Personnel.