In our last two articles, Part 5: Assigning Harms and Severity and Part 6: Estimating Probability of Occurrence, we discussed approaches to develop values for these two elements of risk and the challenge of completing these assessments/estimates. This article takes the next step to address the question: “Are we, as a company, willing to accept the risks associated with our device given the benefits that a patient is likely to receive?” This article (Part A) provides a review of the standard approach. A second article (Part B) will provide our thoughts on the challenges.
If you have missed any of our previous articles in this series, you can find them on our website here.
Determining Risk Acceptability – The Standard Approach
Before we jump into the challenges of determining risk acceptability, it’s important to first make sure that we have a common understanding of the key elements of the standard approach as described in ISO 14971:2019, Medical Devices – Application of Risk Management to Medical Devices (we’ll just refer to it as “the standard”).
As a first step, it’s important to be clear on two terms: “risk evaluation” and “risk acceptability.” The standard defines risk evaluation as the “process of comparing the estimated risk against given risk criteria to determine the acceptability of the risk.” So, the process involves taking the “estimated risks” (i.e., typically the results of FMEA-style analysis where you estimate the risk of a Hazardous Situation based on Severity and the Probability of Occurrence of Harm) and evaluating those risks to determine whether they are acceptable or if risk reduction is required. Two tools described in the standard to complete this work are “risk acceptability criteria” and the “risk evaluation matrix.”
The standard calls for the risk analysis team to establish the risk acceptability criteria well before the actual risk analysis activities begin. These criteria must be established in the Risk Management Plan (either directly or by reference) and that plan must be reviewed and approved by company management.
Your company’s Risk Management Procedure needs to include your company’s policy for establishing risk acceptability criteria. By “policy,” the standard means that your company needs to document the purpose, scope, considerations for determining acceptable risk, approaches to risk control, and requirements for approval and review of the risk acceptability criteria. The guidance document (TIR 24971) provides examples of each of these elements. For example, the policy should: identify that intent of the criteria is to ensure a high level of safety consistent with customer expectations (purpose); apply to all medical devices manufactured by your company (scope); take regulations, technical state-of-the-art, and customer concerns into account (considerations); work to reduce risks as far as possible (approach); and identify authorities for approving the policy (approval).
Once the plan is approved, the risk analysis team may begin its work with a clear understanding of the level of risk that management considers to be acceptable.
Using the criteria for estimating Severity and the Probability of Occurrence of Harm established previously, the risk acceptability criteria can take several forms. The two most common approaches are 1) to use Risk Priority Numbers (RPNs), and 2) to develop a risk evaluation matrix.
- Risk Priority Number (RPN): Using the RPN approach, the risk analysis team multiplies the assigned “Severity” and “Probability of Occurrence of Harm” values. The product of these two values is the RPN (S x O = RPN). [Note: when Detection is included in the analysis, the RPN is the product of Severity, Occurrence, and Detection (S x O x D = RPN).] The acceptability of the resulting RPN is based on pre-established cutoff values, such as:
- RPN < 4 = Acceptable: Additional controls can be considered but are not required
- RPN ≥ 4, but < 15 = Acceptable – Risk Reduced As Far As Possible (AFAP): Additional controls would not be technically practical (i.e., additional controls would have a negative effect on the balance between benefit and risk)
- RPN ≥ 15 = Unacceptable: Design changes are needed to mitigate risk, or a benefit-risk analysis is needed to further assess benefit-risk.
If one were to plot the RPN acceptability criteria in a risk evaluation matrix , the result would be a symmetrical pattern. It’s symmetrical because Severity and Occurrence have the same weight. As a result, a risk with a Severity of 5 and an Occurrence of 3 (RPN = 15) is placed in the same risk category as a risk with a Severity of 3 and an Occurrence of 5 (RPN = 15). See below.
- Risk Acceptability Matrix: Over time, many manufacturers found that they were not comfortable with the assumption that Severity and Occurrence have equal weight, arguing that risks with a high Severity are of greater concern than risks with high Occurrence. As a result, most manufacturers now use “asymmetrical” risk acceptability matrices to better reflect their tolerance for risk (see example below).
By taking this “asymmetrical” approach, companies are able to more clearly articulate their acceptance of different types of risk. In the example provided above, a risk with a Severity of 5 and an Occurrence of 2 is considered to be “Unacceptable” while a risk with a Severity of 2 and an Occurrence of 5 is “AFAP.” This approach would also allow a manufacturer to identify any risk with a Severity of 5 as “Unacceptable” regardless of the probability of occurrence if that was appropriate for their product.
There are no prescribed RPN cutoff values or Risk Acceptability Matrix “patterns.” Each manufacturer is able to structure these risk prioritization tools in any way that best fits the nature and expected use of their device. In fact, the standard (Annex A) states that: “Although there has been significant debate over what constitutes an acceptable level of risk, this document does not specify acceptability levels. Specifying a universal level for acceptable risk could be inappropriate” because of the wide range of medical devices and situations.
Following this standard approach, the acceptability of each individual risk identified in your Hazard Analysis or FMEA is judged using the established criteria. This evaluation is typically conducted early in the Design Control process (before the start of Detailed Design) so that there is time to implement risk controls needed to reduce identified risks before getting too deep into the design process. Needed risk controls can take the form of design changes, protective measures, and/or information to the user (e.g., instructions, Warnings and Cautions) in that order of priority.
Then, as part of the approval of the final drawing package, the risk analysis team reassesses the risks following the implementation and verification of risk controls. The risk analysis team uses these “post- mitigation” risk values to determine the acceptability of each individual risk (i.e., each line-item in your FMEA).
This resulting level of risk is called the “residual risk” for that line-item. If after implementing planned risk controls and verifying effectiveness, if you find that the residual risk is not acceptable and that further risk control is not technically practical (i.e., it will have a negative impact on the products benefits and risks), the manufacturer must conduct a benefit-risk analysis to more thoroughly assess whether the medical benefits of the device outweigh the residual risk.
Finally, once all controls have been implemented and verified, and the acceptability of each individual residual risk has been determined (either through a residual risk evaluation or a benefit-risk analysis), the risk analysis team must conduct an “overall residual risk evaluation.” This evaluation is also based on the criteria defined in the Risk Management Plan. If the overall residual risk is not acceptable, the manufacturer must conduct a benefit-risk analysis to determine whether the medical benefits of the intended use of the device outweigh the overall residual risk. If so, the manufacturer should be ready to place the product on the market (or at least prepare a submission for the FDA or a Notified Body). If not, it’s back to the drawing board.
Manufacturers tend to use the same criteria for overall residual risk as they use for each individual risk. That is, if all individual risks are acceptable (i.e., either Acceptable or AFAP) the overall residual risk is acceptable. But if any one residual risk is unacceptable, the overall residual risk is unacceptable.
So, in a nutshell, if the residual risk of each individual risk is acceptable according to the management-approved acceptability criteria, the overall residual risk of the device is acceptable and requires no further review. If the overall residual risk is not acceptable, a benefit-risk analysis is required. The role of the benefit-risk analysis is to assess situations where “unacceptable” risks cannot be further mitigated , but the device (as designed) could provide significant benefits to patients. For example, if a company manufactures a life-sustaining device that is only effective for half of all patients, use of the device may still be warranted (even though half of all patients are expected to die) if there are no better treatment alternatives.
Assessing Risk Acceptability – The Challenges
For those of you who work with the Risk Management Standard on a regular basis, the approach for determining risk acceptability described above should seem pretty familiar. For others, hopefully you now have a general understanding of the standard approach. While this approach establishes a solid framework for building your internal risk acceptability processes, there are a few aspects of the approach that remain a bit murky and require some interpretation.
The most challenging issues to deal with when applying this approach are:
- The role of Benefits in the Risk Acceptability Criteria
- Independent Risks vs. Overall Risk
- Residual Risk Evaluation vs. Risk-Benefit Analysis
Next Steps
In our next article “Determining Risk Acceptability – Part B,” we’ll dig deep into the three challenges described above.