Game of Chance
Managing and reporting the risks associated with auditing
by J.P. Russell
Risk is a popular word being added to the standards and procedures lexicon, but it’s a massive topic that can be confusing. ISO 19011:2011, for example, states that management system auditors should understand risk associated with auditing. The standard includes audit program risk and auditing risk.1
ASQ’s 2004 certified quality auditor body of knowledge (CQA BoK) included the topics of evaluating risk that are associated with:
- Management and the organization as part of the audit purpose or objective.
- Managing the audit program.
- Collecting audit evidence.
The 2012 CQA BoK expands and clarifies risk to include:
- How the audit program affects an organization’s risk.
- How the audit organization’s risk can influence the number and frequency of audits performed.
- The use of risk management tools, such as failure mode and effects analysis, hazard analysis and critical control points, critical to quality analysis and health hazard analysis.2
This topic can get even more complicated when you try to put risk in a box—similar to what is done with document control, corrective action programs or purchasing controls.
What people fail to realize is that risk is the bull in the china shop. It’s free to roam wherever it has access and can cause mayhem any time.
It can ruin product on the shelves and cause an organization to suffer loss of income. If it makes it to the street, it could injure other people or destroy their property.
What is it?
Risk is all around and is part of everyday business. It has been defined as the possibility of loss, injury, disadvantage or destruction, or the product of the amount that may be lost and the probability of losing it.3
For example, in marbles, if I make a shot, I can add a marble to my bag. If I miss the shot, I could lose five marbles. So what’s the risk? I think I have a 90% chance of making the shot—or a 10% chance of missing it. The product of the amount that may be lost (five marbles) and probability of losing it (10%) equals 0.5—half a marble.
Based on this, I should go ahead and take the risk to add another marble to my bag. But if my shot is more risky and has only a 30% chance of success, I risk 3.5 marbles instead of half a marble. In that case, I may decide to play it safe and not take the shot or opt for an approach that will obstruct my opponent.
In business school, I took a class on risk that described it as I did with the marbles, except we were dealing with business ventures and the marbles were money. Due to a lack of statistics or adequate information, we brainstormed the probability of failure and estimated the resources—or revenue—that could be lost.
Risk has long been a word used by the insurance industry and lending institutions. The higher the probability of loss, the higher the premium or interest is on a loan. Higher rates compensate for the higher probability of loss.
The same is true of an organization that decides on a threshold of 15% return on investment (ROI) for a venture of negligible or managed risk, yet it might be willing to make a riskier investment if the ROI is 25%.
The standard writers from the International Organization for Standardization (ISO) have come up with a different definition for risk: the effect of uncertainty on objectives.4
According to the first two notes in the standard, an effect is a deviation from the expected positive or negative, and objectives can have different aspects (such as financial, health and safety, and environmental goals) and can apply at different levels (such as strategic, organizationwide, project, product and process).5
One of the weaknesses of this definition is that organizations don’t have explicit objectives for every risk. It’s akin to quality being conformance to customer-stated requirements, except not all product or service requirements are specified.
The ISO definition of risk should be expanded to include the effect of uncertainty on objectives, including objectives that aren’t stated but are necessary for the organization to be sustainable and to survive.
Risk management vs. reporting
One reason the topic of risk is confusing is that people intermingle the need to manage risk with the need to monitor or report risk. ISO 31000 states that the risk management process involves:
- Establishing context (scope and objectives).
- Identifying risk.
- Analyzing risk.
- Evaluating risk.
- Treating risk.6
If you’re asked to manage risk, you need to follow risk management process steps or a similar model. If you’re asked to monitor or report risk, you need to be able to recognize it or know it when you see it.
In many cases, auditors and others are asked to monitor or report what they observe relative to risk treatments or the context of risky processes or activities, with risk being the possibility of loss, injury, disadvantage or destruction, or the effect of uncertainty on objectives.
If you’re asked to report risk, the reporting could occur during any of the risk assessment steps (identification, analysis and evaluation).7 The reporting may be based on intuitive assessment, such as a finding that could result in loss of license, certification or a customer order. Auditors also may be directed to report that risk treatments are implemented and effective.
An audit is a service performed by auditors that may be internal or external to the organization being audited. ISO 19011:2011 specifically lists audit performance risks that should be addressed. The first risk to address involves preparing the audit plan (clause 126.96.36.199).8 The audit team leader should be aware of the risk to the auditee organization created by the audit.
For example, the presence of audit team members could influence health, safety, environmental or quality controls. A member of the team could be sick, get injured, pollute the environment or interfere with an inspection. There may not be a specific audit objective that involves avoiding injury, but it’s a risk that should be considered depending on the auditee site and requirements.
If you’re asked to analyze and evaluate risk associated with the audit, you may want to consider the aspects related to the product or service, as well as causal factors such as people, equipment, environment, materials, methods and measurements.
Auditors, for example, might contaminate a clean room, ruin a circuit board with a static spark or void a calibration due to equipment damage. To identify potential risk, I would suggest you first consider the aspects that can create risk relative to the environment: the nature of the organization.
Any methods that will be used to mitigate or treat risk should be discussed in the audit’s opening meeting and included in the audit plan. You can include managed risk in the audit plan under managed risk or another suitable title. I’ve done this in the past, but I may have used titles such as special requirements, or special topics or issues.
One of the biggest risks when conducting an audit is the risk associated with sampling. Samples may not be representative of the population from which they are selected, and thus any conclusions based on the sample would be wrong.
I won’t explore consumer or producer risk in this column, but sampling error is an identified aspect of risk that must be addressed. External auditors are more likely than internal auditors to experience sampling error, but it’s important for all auditors to be vigilant and attentive listeners for any indication a sample may be skewed.
Perhaps the auditee changed processes 45 days ago, the form you have is just for special orders, or the records you selected are for a service that is no longer provided. In these cases, you’re being asked to monitor and report risk, but they wouldn’t necessarily be included in the audit plan because it could identify the exact samples you will be reviewing.
Do not delay
As part of the audit performance, any evidence collected that suggests an immediate and significant risk—effect of uncertainty on objectives—to the auditee should be reported without delay to the auditee and, as appropriate, to the audit client.9
Reporting something that suggests an immediate and significant risk is subjective, which is why auditors are frequently asked to use their judgment. If the risk doesn’t have a significant impact on the organization, the auditee will let you know. This type of activity isn’t managing risk; it’s reporting risk.
The 2012 CQA BoK indicates that audit results may be classified by the level of risk. This may be as simple as reporting results as major or minor, rating nonconformities on a scale of 1-10, or comparing them relative to the business bottom line or budget. In this case, auditors are being asked to assess their observations and report audit findings based on relative risk.
Auditing for risk
Some audit programs or objectives include risk. Organizations may conduct risk audits while they conduct compliance audits. At other times, the identification of risky processes or activities beyond conformity or compliance to requirements is added to the purpose of the audit.
Conducting a risk audit may involve collecting evidence to verify known risk is being controlled and that risk treatment plans are effective. The objective or purpose of the audit would be to start with a list of identified and treated risks, and then verify the controls are effective—similar to an auditor verifying that corrective actions have been implemented and are effective.
Risk treatments must be verified in the short and long term, and when there are changes to processes related or linked to identified risk. Some standards, such as ISO 22000 for food safety, have plans to treat or mitigate significant hazards and risk. Auditors can verify or validate those plans.
Process audits are an effective approach for identifying new risk. Auditors conducting process audits are more familiar with the process, and would be able to spot and identify processes or events that could be a significant risk to the organization. System and product audits, however, do not exclude auditors from identifying processes or events that could be significant risk.
Auditors are not charged with conducting a formal risk management assessment; they are merely making observations that there might be an aspect of risk that needs formal evaluation. Later, however, an auditor may be assigned to a team that conducts a formal risk management analysis.
For example, an auditor may observe that the ink is smearing on a product label with return instructions. This may be a performance issue in which the product is not being returned in an efficient manner, or it could be a potential risk to the organization if product is put into a landfill in lieu of proper instructions.
As I mentioned at the outset, risk is a complicated topic—complicated enough that it’s difficult to cover in just one column. So look for a continuation of this discussion in next month’s Standards Outlook.
- International Organization for Standardization, ISO 19011:2011—Guidelines for auditing management systems.
- ASQ, "Quality Auditor Certification–Body of Knowledge," http://prdweb.asq.org/certification/control/quality-auditor/bok.
- Merriam-Webster, "Risk," www.merriam-webster.com/dictionary/risk.
- International Organization for Standardization, ISO Guide 73:2009—Risk management—vocabulary, clause 1.1.
- International Organization for Standardization, ISO 31000:2009—Risk management—Principles and guidelines, clause 5.1.
- Ibid, clause 5.4.
- International Organization for Standardization, ISO 19011:2011—Guidelines for auditing management systems.
- Ibid, clause 6.4.4.
J.P. Russell is president of J.P. Russell & Associates in Gulf Breeze, FL. He also is managing director of and provides training for QualityWBT Center for Education. Russell is an ASQ fellow, ASQ-certified quality auditor, voting member of the American National Standards Institute/ASQ Z1 committee and member of the U.S. technical advisory group for International Organization for Standardization technical committee 176. He is the editor of The ASQ Auditing Handbook, third edition, and author of several ASQ Quality Press books, including The Process Auditing and Techniques Guide, second edition.